Build a Neural Network from Scratch

Build a digit classifier using only NumPy, no deep learning frameworks.

Introduction

30 Second Summary

Every time you deposit a check by snapping a photo or unlock your phone with a handwritten passcode, software is reading handwriting and converting it into numbers. That process starts with a neural network learning what each digit looks like.

In this project, you will build a neural network from scratch using only Python and NumPy to classify handwritten digits. No deep learning frameworks. Just pure math and code.

What You'll Build

A feedforward neural network that downloads 70,000 handwritten digit images, learns to recognize them through training, and predicts which digit (0-9) each image shows.

By the end of this project, you'll have:

  • A working 2-layer neural network built entirely from scratch with NumPy that classifies handwritten digits.
  • Training and test accuracy and loss plots that visualize how the network learns over time.
  • A visual grid of predictions on real handwritten digit images, showing which ones your model gets right and wrong.
  • Upgraded your model with softmax and cross-entropy loss for a significant accuracy boost (Secret Mission).

Do I need to know calculus or linear algebra?

You need solid Python fundamentals like functions, loops, and arrays. Basic familiarity with matrix multiplication and derivatives is helpful but not required. The project walks you through the math step by step.

Set Up Your Project

Before you can build a neural network that recognizes handwritten digits, you need a workspace with the right tools in place.

In this step, you'll create your project folder and Python file, verify that your packages are installed, and add the imports your code depends on.

In this step, get ready to:
  • Create a project folder and Python file.
  • Verify that your Python packages are installed.
  • Add the initial imports to your Python file.
Create your project folder and Python file

Everything you build in this project lives inside a single Python file. Let's create the folder and file now.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir digit-classifier
cd digit-classifier
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the digit-classifier folder on your Desktop, then click Open.

Now create the Python file you'll work in for the entire project.

  • In VS Code's Explorer sidebar on the left, hover over the digit-classifier folder name.
  • Click the New File icon (a page with a plus sign).
  • Type neural_network.py and press Enter.

You should see neural_network.py appear in the Explorer sidebar and open as an empty file in the editor.

  • Confirm the file exists by switching back to your terminal and running:
ls

You should see neural_network.py in the output.

Don't see the file?

Make sure your terminal is inside the digit-classifier folder. Run pwd (macOS) or cd (Windows) to check your current directory. If you're not in digit-classifier, run cd ~/Desktop/digit-classifier again.

Help me find my project folder and file.

Verify your Python packages are installed

This project uses three Python packages: NumPy for math and array operations, Matplotlib for plotting, and requests for downloading data.

  • Check that all three packages are installed by running this command in your terminal:
python -c "import numpy; import matplotlib; import requests; print('All packages ready!')"

I see 'All packages ready!'

All three packages are installed and ready to go. Move on to the next substep.

I see an ImportError

The error message tells you which package is missing. Install all three at once by running:

pip install numpy matplotlib requests

After the installation finishes, re-run the verification command above. You should see All packages ready! this time.

Still seeing an error?

  • Try pip3 instead of pip if you have multiple Python versions installed.
  • On macOS or Linux, you may need to use python3 instead of python in the verification command.

Help me install my Python packages.

Add the initial imports

Your project needs five imports that handle everything from downloading data to performing matrix math and generating plots.

  • In neural_network.py, add the following imports at the top of the file:
import numpy as np              # Math and array operations
import requests                 # HTTP requests for downloading data
import gzip                     # Decompress .gz files
import os                       # File and directory operations
import matplotlib.pyplot as plt # Plotting and visualization

What do these imports do?

  • numpy is the core library for this project. Every matrix multiplication, weight update, and activation function uses NumPy arrays.
  • requests downloads the MNIST dataset files from the internet.
  • gzip decompresses the dataset files, which are stored in compressed .gz format.
  • os handles file paths and directory creation so the code works on both macOS and Windows.
  • matplotlib.pyplot generates the visualizations of your training progress and predictions.
  • Save neural_network.py by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • In your terminal, run the file to confirm there are no errors:
python neural_network.py

You should see no output and no errors. A clean run means all five imports loaded successfully.

Seeing a ModuleNotFoundError?

  • Check that you installed the packages in the previous substep. The error message names the missing module.
  • Make sure you are running python from the same Python installation where you installed the packages. Try python3 neural_network.py if python points to a different version.
  • Check for typos in your import lines. Each import must match exactly what is shown in the code block above.

Help me fix my import errors.

Awesome, I've got everything!

Double check you've saved your file before moving on.

I'd like to double check the full code

Here is what your neural_network.py should look like so far.

import numpy as np
import requests
import gzip
import os
import matplotlib.pyplot as plt

Your workspace is ready. Next up, you'll download 70,000 handwritten digit images and take your first look at the data.

Download and Explore the MNIST Dataset

Your project is set up and ready to go. Now you need data for your neural network to learn from.

The MNIST dataset is a collection of 70,000 handwritten digit images, each labeled with the digit it represents (0 through 9). In this step, you will download it, parse the raw binary files, and visualize some samples to make sure everything loaded correctly.

In this step, get ready to:
  • Download the MNIST dataset files from a public mirror.
  • Parse the compressed binary files into NumPy arrays.
  • Visualize sample digits to confirm the data loaded correctly.
Download the MNIST data files

The MNIST dataset is stored as 4 compressed binary files: training images (60,000), test images (10,000), and their corresponding labels. You will download them from a public S3 mirror.

  • In neural_network.py, add the following code below your import statements to define the file names and download them:
# Define the 4 MNIST data files to download
data_sources = {
    "training_images": "train-images-idx3-ubyte.gz",
    "test_images": "t10k-images-idx3-ubyte.gz",
    "training_labels": "train-labels-idx1-ubyte.gz",
    "test_labels": "t10k-labels-idx1-ubyte.gz",
}

# Create a local directory to store the data
data_dir = "mnist_data"
os.makedirs(data_dir, exist_ok=True)

# OSSCI mirror URL for MNIST files
base_url = "https://ossci-datasets.s3.amazonaws.com/mnist/"

# Download each file only if it doesn't already exist
for fname in data_sources.values():
    fpath = os.path.join(data_dir, fname)
    if not os.path.exists(fpath):
        print("Downloading " + fname + "...")
        resp = requests.get(base_url + fname, stream=True)
        resp.raise_for_status()
        with open(fpath, "wb") as f:
            for chunk in resp.iter_content(chunk_size=128):
                f.write(chunk)
        print("Done.")

What does this code do?

  • data_sources maps short names to the 4 gzip-compressed MNIST files.
  • The code creates a local mnist_data folder to cache the downloaded files.
  • The download loop checks if each file already exists before downloading. Re-running the script will skip files you already have.
  • stream=True downloads in small chunks to handle the large files efficiently.
  • Save neural_network.py.
  • Run the script by running:
python neural_network.py

You should see Downloading train-images-idx3-ubyte.gz... followed by Done. for each of the 4 files.

Not seeing download messages?

  • Make sure you are connected to the internet. The files are downloaded from an AWS S3 mirror.
  • If you see a ModuleNotFoundError for requests, install it by running pip install requests in your terminal.
  • If you see a ConnectionError, a VPN or firewall may be blocking access to the S3 URL. Try disabling your VPN temporarily.
  • Help me debug my MNIST download error.
Parse the binary files into NumPy arrays

The downloaded files are compressed binary data, not a format like CSV or JSON you can open directly. You need to decompress them and parse the raw bytes into NumPy arrays your code can work with.

  • Add this code below your download loop in neural_network.py:
mnist_dataset = {}

# Parse image files: skip 16-byte header, reshape each image to a 784-element vector
for key in ("training_images", "test_images"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(
            f.read(), np.uint8, offset=16
        ).reshape(-1, 28 * 28)

# Parse label files: skip 8-byte header
for key in ("training_labels", "test_labels"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(f.read(), np.uint8, offset=8)

# Assign to convenient variable names
x_train = mnist_dataset["training_images"]
y_train = mnist_dataset["training_labels"]
x_test = mnist_dataset["test_images"]
y_test = mnist_dataset["test_labels"]

# Print shapes to verify the data loaded correctly
print("Training images shape:", x_train.shape)
print("Training labels shape:", y_train.shape)
print("Test images shape:", x_test.shape)
print("Test labels shape:", y_test.shape)

What does this code do?

  • np.frombuffer reads the raw bytes from each file and converts them into a NumPy array.
  • Image files have a 16-byte header, and label files have an 8-byte header. The offset parameter tells NumPy to skip past these headers to reach the actual data.
  • Each 28x28 pixel image is reshaped into a flat 784-element vector. This is the format the neural network will expect as input.
  • The final result is 4 arrays: x_train (60,000 images), y_train (60,000 labels), x_test (10,000 images), and y_test (10,000 labels).
  • Save neural_network.py and run it again:
python neural_network.py

You should see the following shape printouts after the download messages:

Training images shape: (60000, 784)
Training labels shape: (60000,)
Test images shape: (10000, 784)
Test labels shape: (10000,)

What do these shapes mean?

(60000, 784) means 60,000 images, each represented as a row of 784 pixel values (28 x 28 flattened). (60000,) means 60,000 labels, one integer per image telling you which digit (0-9) it is.

Seeing an error instead of shapes?

  • If you see FileNotFoundError, the download step may not have completed. Delete the mnist_data folder and re-run the script to trigger a fresh download.
  • If you see BadGzipFile, a file may have been partially downloaded. Delete the corrupted file from mnist_data/ and re-run.
  • Help me debug my MNIST parsing error.
Visualize sample digits

Numbers in an array are hard to interpret on their own. Plotting a few images confirms the data is real handwritten digits and not garbled data.

  • Add the following code at the bottom of neural_network.py to plot 5 sample digits and save the result as an image:
# Plot 5 sample training images side by side
fig, axes = plt.subplots(1, 5, figsize=(12, 3))
for i, ax in enumerate(axes):
    # Reshape the flat 784-element vector back to 28x28 for display
    ax.imshow(x_train[i].reshape(28, 28), cmap="gray")
    ax.set_title("Label: " + str(y_train[i]))
    ax.axis("off")
plt.suptitle("Sample MNIST Digits")
plt.tight_layout()
plt.savefig("sample_digits.png", dpi=100)
plt.show()

What does this code do?

  • reshape(28, 28) converts each flat 784-element vector back to its original 28x28 pixel grid for display.
  • cmap="gray" renders the pixels in grayscale, matching the original image format.
  • plt.savefig saves the plot to your project folder as sample_digits.png so you have a permanent copy.
  • Save neural_network.py and run it again:
python neural_network.py

A Matplotlib window opens showing 5 handwritten digits with their labels above each one. You should also see sample_digits.png saved in your project folder.

Plot window not appearing?

  • Some environments (like remote servers or WSL) do not have a display for Matplotlib to render to. Even if the window does not appear, sample_digits.png should still be saved in your project folder. Open it manually to confirm.
  • If you see NameError: name 'x_train' is not defined, make sure the parsing code from the previous substep is above this visualization code in your file.
  • Help me fix my Matplotlib plot issue.

Awesome, I've got everything!

Great. Double check you have saved neural_network.py before moving on.

I'd like to double check the full code

Here is the complete neural_network.py file so far. Make sure your code matches this exactly.

import numpy as np
import requests
import gzip
import os
import matplotlib.pyplot as plt

# =============================================
# Download and Explore the MNIST Dataset
# =============================================

data_sources = {
    "training_images": "train-images-idx3-ubyte.gz",
    "test_images": "t10k-images-idx3-ubyte.gz",
    "training_labels": "train-labels-idx1-ubyte.gz",
    "test_labels": "t10k-labels-idx1-ubyte.gz",
}

data_dir = "mnist_data"
os.makedirs(data_dir, exist_ok=True)

base_url = "https://ossci-datasets.s3.amazonaws.com/mnist/"

for fname in data_sources.values():
    fpath = os.path.join(data_dir, fname)
    if not os.path.exists(fpath):
        print("Downloading " + fname + "...")
        resp = requests.get(base_url + fname, stream=True)
        resp.raise_for_status()
        with open(fpath, "wb") as f:
            for chunk in resp.iter_content(chunk_size=128):
                f.write(chunk)
        print("Done.")

mnist_dataset = {}

for key in ("training_images", "test_images"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(
            f.read(), np.uint8, offset=16
        ).reshape(-1, 28 * 28)

for key in ("training_labels", "test_labels"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(f.read(), np.uint8, offset=8)

x_train = mnist_dataset["training_images"]
y_train = mnist_dataset["training_labels"]
x_test = mnist_dataset["test_images"]
y_test = mnist_dataset["test_labels"]

print("Training images shape:", x_train.shape)
print("Training labels shape:", y_train.shape)
print("Test images shape:", x_test.shape)
print("Test labels shape:", y_test.shape)

fig, axes = plt.subplots(1, 5, figsize=(12, 3))
for i, ax in enumerate(axes):
    ax.imshow(x_train[i].reshape(28, 28), cmap="gray")
    ax.set_title("Label: " + str(y_train[i]))
    ax.axis("off")
plt.suptitle("Sample MNIST Digits")
plt.tight_layout()
plt.savefig("sample_digits.png", dpi=100)
plt.show()

Your dataset is loaded and you have confirmed the images are real handwritten digits. Next up, you will preprocess this raw data to get it ready for your neural network.

Preprocess the Data

Your MNIST dataset is downloaded, parsed, and visualized. You have 60,000 training images and 10,000 test images sitting in NumPy arrays, ready to go.

Raw pixel data is not ideal for a neural network. The values range from 0 to 255, and labels are single integers like 5. In this step, you will transform the data into a format that helps the network learn efficiently.

In this step, get ready to:
  • Normalize pixel values from [0, 255] to [0.0, 1.0] and select a training subset.
  • One-hot encode the digit labels into 10-element vectors.
Normalize and subset the image data

Each pixel in an MNIST image is an integer between 0 (black) and 255 (white). Neural networks learn more effectively when input values are small and consistently scaled. Dividing every pixel by 255 rescales the data to the range [0.0, 1.0].

Training on all 60,000 images one at a time would take many minutes. You will use a subset of 1,000 images to keep training under a minute. You can increase this number later for better accuracy.

  • In neural_network.py, add the following code below the sample digits visualization block:
# Use a subset of 1,000 images for fast training
training_sample, test_sample = 1000, 1000

# Normalize pixel values from [0, 255] to [0.0, 1.0]
training_images = x_train[:training_sample] / 255.0
test_images = x_test[:test_sample] / 255.0

print("Training images dtype:", training_images.dtype)
print("Pixel value range:", training_images.min(), "to", training_images.max())

What does this code do?

  • training_sample and test_sample control how many images you use. Set to 1,000 each for fast iteration.
  • Dividing by 255.0 converts every pixel from an integer (0-255) to a float (0.0-1.0). This is called normalization.
  • The print statements confirm the data type changed to float64 and the values now fall between 0.0 and 1.0.
  • Save neural_network.py.
  • Run the script by running:
python neural_network.py
  • Close the sample digits plot window when it appears (this is from the previous step). After the earlier output, you should see two new lines showing float64 as the dtype and pixel values ranging from 0.0 to 1.0.

Not seeing the new output?

  • Make sure you added the new code below the plt.show() line from the sample digits visualization, not inside it.
  • The sample digits plot window blocks execution until you close it. Close the plot window and the remaining output will appear in your terminal.
  • help me debug why my normalization output is not showing

Why normalize to [0.0, 1.0]?

Large input values (like 255) produce large outputs in each layer, which can cause the weight updates during training to overshoot. Small, consistent values keep the math stable and help the network converge faster.

One-hot encode the labels

Your labels are currently single integers like 5 or 3. The neural network's output layer has 10 neurons, one for each digit (0 through 9). To compare the network's output against the correct answer, you need the labels in the same shape.

One-hot encoding converts each label into a 10-element vector. For example, the digit 5 becomes [0, 0, 0, 0, 0, 1, 0, 0, 0, 0]. The 1.0 sits at the index matching the digit.

  • In neural_network.py, add the following code below the normalization block you just wrote:
def one_hot_encode(labels, num_classes=10):
    # Compare each label to every class index using broadcasting
    one_hot = labels[..., None] == np.arange(num_classes)[None]
    return one_hot.astype(np.float64)

# Encode training and test labels
training_labels = one_hot_encode(y_train[:training_sample])
test_labels = one_hot_encode(y_test[:test_sample])

print("Training labels shape:", training_labels.shape)
print("Example label for digit", y_train[0], ":", training_labels[0])

What does this code do?

  • one_hot_encode() uses NumPy broadcasting to compare each label against every class index (0 through 9). The result is a boolean array that gets converted to floats.
  • training_labels and test_labels now have the shape (1000, 10). Each row is a 10-element vector with a single 1.0 at the correct digit's position.
  • The print statement shows an example encoded label so you can verify the format matches what the network's 10-neuron output layer expects.
  • Save neural_network.py and run the script again:
python neural_network.py
  • Close the sample digits plot window. You should see Training labels shape: (1000, 10) in the output, followed by an example label array showing a 1. at the position matching the first training digit's value.

Seeing a shape mismatch or error?

  • Check that you are slicing with y_train[:training_sample] (using y_train, the original labels) and not training_images (the normalized images).
  • Make sure the one_hot_encode function is defined before the lines that call it. Python reads top to bottom, so the function must come first.
  • help me debug my one-hot encoding output

Awesome, I've got everything!

Double check you've saved neural_network.py before moving on.

I'd like to double check the full code

Here is the complete neural_network.py file so far, including all code from previous steps.

import numpy as np
import requests
import gzip
import os
import matplotlib.pyplot as plt

# =============================================
# Download and Explore the MNIST Dataset
# =============================================

data_sources = {
    "training_images": "train-images-idx3-ubyte.gz",
    "test_images": "t10k-images-idx3-ubyte.gz",
    "training_labels": "train-labels-idx1-ubyte.gz",
    "test_labels": "t10k-labels-idx1-ubyte.gz",
}

data_dir = "mnist_data"
os.makedirs(data_dir, exist_ok=True)

base_url = "https://ossci-datasets.s3.amazonaws.com/mnist/"

for fname in data_sources.values():
    fpath = os.path.join(data_dir, fname)
    if not os.path.exists(fpath):
        print("Downloading " + fname + "...")
        resp = requests.get(base_url + fname, stream=True)
        resp.raise_for_status()
        with open(fpath, "wb") as f:
            for chunk in resp.iter_content(chunk_size=128):
                f.write(chunk)
        print("Done.")

mnist_dataset = {}

for key in ("training_images", "test_images"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(
            f.read(), np.uint8, offset=16
        ).reshape(-1, 28 * 28)

for key in ("training_labels", "test_labels"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(f.read(), np.uint8, offset=8)

x_train = mnist_dataset["training_images"]
y_train = mnist_dataset["training_labels"]
x_test = mnist_dataset["test_images"]
y_test = mnist_dataset["test_labels"]

print("Training images shape:", x_train.shape)
print("Training labels shape:", y_train.shape)
print("Test images shape:", x_test.shape)
print("Test labels shape:", y_test.shape)

fig, axes = plt.subplots(1, 5, figsize=(12, 3))
for i, ax in enumerate(axes):
    ax.imshow(x_train[i].reshape(28, 28), cmap="gray")
    ax.set_title("Label: " + str(y_train[i]))
    ax.axis("off")
plt.suptitle("Sample MNIST Digits")
plt.tight_layout()
plt.savefig("sample_digits.png", dpi=100)
plt.show()

# =============================================
# Preprocess the Data
# =============================================

training_sample, test_sample = 1000, 1000
training_images = x_train[:training_sample] / 255.0
test_images = x_test[:test_sample] / 255.0

print("Training images dtype:", training_images.dtype)
print("Pixel value range:", training_images.min(), "to", training_images.max())


def one_hot_encode(labels, num_classes=10):
    one_hot = labels[..., None] == np.arange(num_classes)[None]
    return one_hot.astype(np.float64)


training_labels = one_hot_encode(y_train[:training_sample])
test_labels = one_hot_encode(y_test[:test_sample])

print("Training labels shape:", training_labels.shape)
print("Example label for digit", y_train[0], ":", training_labels[0])

Your data is clean, normalized, and shaped for a neural network. Next up, you will build the actual network architecture and watch it make its first (terrible) prediction.

Build the Neural Network Architecture

Your data is preprocessed and ready to flow through a neural network. Each image is a 784-element vector of normalized pixel values, and each label is a 10-element one-hot vector that the network can compare its output against.

Now it's time to build the network itself. You'll define the activation functions that let the network learn non-linear patterns, initialize the weight matrices that connect each layer, and run a test forward pass to confirm data flows through correctly.

In this step, get ready to:
  • Define the ReLU activation function and its derivative.
  • Initialize the weight matrices connecting the input, hidden, and output layers.
  • Test a forward pass to verify data flows through the network.
Define the ReLU activation function and its derivative

A neural network needs an activation function to learn non-linear patterns. Without one, stacking layers of matrix multiplications would collapse into a single linear transformation, no matter how many layers you add.

ReLU (Rectified Linear Unit) is one of the simplest and most widely used activation functions. It passes positive values through unchanged and replaces anything negative with zero.

  • In neural_network.py, add the ReLU function and its derivative below the preprocessing code by copying this code:
# Returns the input if positive, 0 otherwise
def relu(x):
    return (x >= 0) * x

# Returns 1 for positive inputs, 0 otherwise (used during backpropagation)
def relu_derivative(output):
    return output >= 0

What does this code do?

  • relu(x) uses a clever trick: (x >= 0) produces True (1) or False (0). Multiplying by x keeps positive values and zeros out negatives.
  • relu_derivative(output) returns 1 where the output was positive and 0 where it was zero. The network uses this during backpropagation to figure out which neurons contributed to the output.

You won't see a change yet. The next chunk uses these functions in a forward pass.

Initialize the weight matrices

Weights are the numbers the network actually learns. Each weight controls how strongly one neuron's output influences the next neuron's input. Before training, you initialize them with small random values so the network starts with a blank slate.

  • Add the weight initialization code below the ReLU functions in neural_network.py:
# Set a fixed seed for reproducible results
seed = 884736743
rng = np.random.default_rng(seed)

# Define the size of each layer
pixels_per_image = 784
hidden_size = 100
num_labels = 10

# Initialize weights with small random values in range [-0.1, 0.1]
weights_1 = 0.2 * rng.random((pixels_per_image, hidden_size)) - 0.1
weights_2 = 0.2 * rng.random((hidden_size, num_labels)) - 0.1

print("Weights 1 shape:", weights_1.shape)
print("Weights 2 shape:", weights_2.shape)

What does this code do?

  • np.random.default_rng(seed) creates a random number generator with a fixed seed. This means you get the same "random" weights every time you run the script, making results reproducible.
  • weights_1 is a 784x100 matrix. It connects every pixel in the input image to every neuron in the hidden layer. That's 78,400 individual weights.
  • weights_2 is a 100x10 matrix. It connects every hidden neuron to each of the 10 output neurons (one per digit 0-9).
  • The formula 0.2 * rng.random(...) - 0.1 generates values in the range [-0.1, 0.1]. Starting small prevents any single neuron from dominating early in training.
  • Save neural_network.py and run it in your terminal:
python neural_network.py
  • You should see the weight shapes printed at the bottom of the output:
Weights 1 shape: (784, 100)
Weights 2 shape: (100, 10)

What do these shapes mean?

(784, 100) means 784 inputs (one per pixel) each connected to 100 hidden neurons. (100, 10) means 100 hidden neurons each connected to 10 outputs (one per digit). Together, these two matrices define a 3-layer network: input, hidden, and output.

Seeing an error when running the script?

  • If you see NameError: name 'np' is not defined, make sure your imports from Step 1 are still at the top of the file.
  • If you see a SyntaxError, check that the function definitions from the previous chunk are complete and properly indented.

Help me debug an error when initializing my neural network weights in NumPy.

Test a single forward pass

Forward propagation is the process of feeding an input through the network to produce an output. Data flows in one direction: from the input layer, through the hidden layer (where ReLU is applied), to the output layer.

  • Add this forward pass test below the weight initialization code in neural_network.py:
# Take the first training image as a test input
sample = training_images[0]

# Forward pass: input -> hidden layer (with ReLU) -> output layer
hidden = relu(np.dot(sample, weights_1))
output = np.dot(hidden, weights_2)

print("Forward pass output shape:", output.shape)
print("Predicted digit:", np.argmax(output), "| Actual digit:", y_train[0])

What does this code do?

  • np.dot(sample, weights_1) multiplies the 784-pixel input by the 784x100 weight matrix, producing 100 values for the hidden layer.
  • relu(...) zeroes out any negative values in the hidden layer, introducing the non-linearity the network needs to learn complex patterns.
  • np.dot(hidden, weights_2) multiplies the 100 hidden values by the 100x10 weight matrix, producing 10 output scores (one per digit).
  • np.argmax(output) picks the digit with the highest score as the prediction. With random weights, this prediction is essentially a guess.
  • Save neural_network.py and run it again:
python neural_network.py
  • At the bottom of the output, you should see the forward pass results:
Forward pass output shape: (10,)
Predicted digit: 4 | Actual digit: 5

The predicted digit will likely be wrong. That's expected. The weights are random, so the network is just guessing. Training in the next step will fix that.

Seeing a shape mismatch or dimension error?

  • A ValueError: shapes not aligned means the matrix dimensions don't match. Check that pixels_per_image is 784, hidden_size is 100, and num_labels is 10.
  • If you see NameError: name 'relu' is not defined, make sure the relu function definition appears above this code in the file.

Help me debug my neural network forward pass.

Awesome, I've got everything!

Great. Double check you've saved neural_network.py before moving on.

I'd like to double check the full code

Here is the complete neural_network.py up to this point. Compare your file against this:

import numpy as np
import requests
import gzip
import os
import matplotlib.pyplot as plt

# =============================================
# Download and Explore the MNIST Dataset
# =============================================

data_sources = {
    "training_images": "train-images-idx3-ubyte.gz",
    "test_images": "t10k-images-idx3-ubyte.gz",
    "training_labels": "train-labels-idx1-ubyte.gz",
    "test_labels": "t10k-labels-idx1-ubyte.gz",
}

data_dir = "mnist_data"
os.makedirs(data_dir, exist_ok=True)

base_url = "https://ossci-datasets.s3.amazonaws.com/mnist/"

for fname in data_sources.values():
    fpath = os.path.join(data_dir, fname)
    if not os.path.exists(fpath):
        print("Downloading " + fname + "...")
        resp = requests.get(base_url + fname, stream=True)
        resp.raise_for_status()
        with open(fpath, "wb") as f:
            for chunk in resp.iter_content(chunk_size=128):
                f.write(chunk)
        print("Done.")

mnist_dataset = {}

for key in ("training_images", "test_images"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(
            f.read(), np.uint8, offset=16
        ).reshape(-1, 28 * 28)

for key in ("training_labels", "test_labels"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(f.read(), np.uint8, offset=8)

x_train = mnist_dataset["training_images"]
y_train = mnist_dataset["training_labels"]
x_test = mnist_dataset["test_images"]
y_test = mnist_dataset["test_labels"]

print("Training images shape:", x_train.shape)
print("Training labels shape:", y_train.shape)
print("Test images shape:", x_test.shape)
print("Test labels shape:", y_test.shape)

fig, axes = plt.subplots(1, 5, figsize=(12, 3))
for i, ax in enumerate(axes):
    ax.imshow(x_train[i].reshape(28, 28), cmap="gray")
    ax.set_title("Label: " + str(y_train[i]))
    ax.axis("off")
plt.suptitle("Sample MNIST Digits")
plt.tight_layout()
plt.savefig("sample_digits.png", dpi=100)
plt.show()

# =============================================
# Preprocess the Data
# =============================================

training_sample, test_sample = 1000, 1000
training_images = x_train[:training_sample] / 255.0
test_images = x_test[:test_sample] / 255.0

print("Training images dtype:", training_images.dtype)
print("Pixel value range:", training_images.min(), "to", training_images.max())


def one_hot_encode(labels, num_classes=10):
    one_hot = labels[..., None] == np.arange(num_classes)[None]
    return one_hot.astype(np.float64)


training_labels = one_hot_encode(y_train[:training_sample])
test_labels = one_hot_encode(y_test[:test_sample])

print("Training labels shape:", training_labels.shape)
print("Example label for digit", y_train[0], ":", training_labels[0])

# =============================================
# Build the Neural Network Architecture
# =============================================


def relu(x):
    return (x >= 0) * x


def relu_derivative(output):
    return output >= 0


seed = 884736743
rng = np.random.default_rng(seed)

pixels_per_image = 784
hidden_size = 100
num_labels = 10

weights_1 = 0.2 * rng.random((pixels_per_image, hidden_size)) - 0.1
weights_2 = 0.2 * rng.random((hidden_size, num_labels)) - 0.1

print("Weights 1 shape:", weights_1.shape)
print("Weights 2 shape:", weights_2.shape)

sample = training_images[0]
hidden = relu(np.dot(sample, weights_1))
output = np.dot(hidden, weights_2)

print("Forward pass output shape:", output.shape)
print("Predicted digit:", np.argmax(output), "| Actual digit:", y_train[0])

Your neural network architecture is in place. You have activation functions, weight matrices, and a working forward pass. Next up, you'll teach it to actually learn by implementing backpropagation and training it on your digit images.

Train the Network with Backpropagation

In the previous step, you built your network's architecture and ran a single forward propagation pass. The output was random because the weights have never been adjusted. Now it is time to make this network actually learn.

Training a neural network means showing it examples, measuring how wrong its predictions are, and then nudging the weights to reduce that error. The algorithm that computes those nudges is called backpropagation. You will implement this entire process by hand in this step.

In this step, get ready to:
  • Set hyperparameters that control how the network learns.
  • Implement the full training loop with forward propagation, dropout, backpropagation, and weight updates.
  • Evaluate the model on test data after each training pass.
Set the training hyperparameters

Before the training loop runs, you need to define a few values that control how the network learns. These are called hyperparameters because they sit above the parameters (weights) that the network learns on its own.

  • In neural_network.py, add this code below the forward pass test from the previous step:
learning_rate = 0.005
epochs = 20

store_training_loss = []
store_training_accuracy = []
store_test_loss = []
store_test_accuracy = []

What do these values control?

  • learning_rate controls how much the weights change with each update. Too high and the network overshoots. Too low and it barely learns.
  • epochs is the number of complete passes through all the training data. 20 passes gives the network enough repetition to learn meaningful patterns.
  • The four store_ lists track loss and accuracy after each epoch so you can plot the learning curves later.

You won't see any output yet. The next chunk builds the training loop that uses these values.

Write the training loop

This is the core of the entire project. The training loop feeds every image through the network, measures the error, then uses backpropagation to figure out how much each weight contributed to that error.

It then applies gradient descent to nudge each weight in the direction that reduces the error. This process repeats for every image in every epoch.

  • Add the training loop below your hyperparameters in neural_network.py:
for epoch in range(epochs):
    training_loss = 0.0
    training_correct = 0

    for i in range(len(training_images)):
        # Forward propagation
        layer_0 = training_images[i]
        layer_1 = relu(np.dot(layer_0, weights_1))

        # Apply dropout
        dropout_mask = rng.integers(low=0, high=2, size=layer_1.shape)
        layer_1 *= dropout_mask * 2

        layer_2 = np.dot(layer_1, weights_2)

        # Calculate loss
        training_loss += np.sum((training_labels[i] - layer_2) ** 2)
        training_correct += int(
            np.argmax(layer_2) == np.argmax(training_labels[i])
        )

        # Backpropagation
        layer_2_delta = training_labels[i] - layer_2
        layer_1_delta = (
            np.dot(weights_2, layer_2_delta) * relu_derivative(layer_1)
        )
        layer_1_delta *= dropout_mask

        # Update weights
        weights_1 += learning_rate * np.outer(layer_0, layer_1_delta)
        weights_2 += learning_rate * np.outer(layer_1, layer_2_delta)

    store_training_loss.append(training_loss / len(training_images))
    store_training_accuracy.append(training_correct / len(training_images))

What does this code do?

Each iteration of the inner loop processes one training image through the full learning cycle:

  • Forward propagation: layer_0 is the raw image (784 pixels). layer_1 multiplies it by weights_1 and applies ReLU to activate the 100 hidden neurons. layer_2 multiplies the hidden layer by weights_2 to produce 10 output scores.
  • Dropout: dropout randomly zeroes out about half the hidden neurons each pass. This prevents overfitting by forcing the network to learn with different combinations of neurons. The * 2 compensates for the missing neurons so the total signal stays balanced.
  • Loss: The squared difference between the prediction and the correct label measures how wrong the network is. training_correct counts how many predictions matched the actual digit.
  • Backpropagation: layer_2_delta is the output error. layer_1_delta pushes that error backward through weights_2 to figure out how much each hidden neuron contributed. The dropout mask is reapplied so only the neurons that were active get blame.
  • Weight updates: np.outer computes the outer product of the activations and the deltas, giving the exact adjustment for each weight. The learning_rate scales these adjustments down so changes are gradual.

Why use dropout during training?

Without dropout, a few neurons can become overly specialized and memorize the training data instead of learning general patterns. Dropout forces the network to spread knowledge across many neurons, which makes it more robust when it encounters new images it has never seen before.

You won't see a change yet. The next chunk adds test evaluation and a print statement so you can watch the network learn.

Add test set evaluation to the training loop

Training loss and accuracy tell you how well the network fits the data it is learning from. Test accuracy tells you how well it generalizes to images it has never seen. You need both to know if the network is truly learning.

  • In neural_network.py, find this line at the bottom of the for epoch loop:
    store_training_accuracy.append(training_correct / len(training_images))
  • Add the following code directly below that line, at the same indentation level (one level inside the for epoch loop):
    # Evaluate on test set (no dropout)
    results = relu(test_images @ weights_1) @ weights_2
    test_loss = np.sum((test_labels - results) ** 2)
    test_correct = np.sum(
        np.argmax(results, axis=1) == np.argmax(test_labels, axis=1)
    )

    store_test_loss.append(test_loss / len(test_images))
    store_test_accuracy.append(test_correct / len(test_images))

    print(
        f"Epoch {epoch:2d} | "
        f"Train Loss: {training_loss / len(training_images):.3f} | "
        f"Train Acc: {training_correct / len(training_images):.3f} | "
        f"Test Loss: {test_loss / len(test_images):.3f} | "
        f"Test Acc: {test_correct / len(test_images):.3f}"
    )

What does this code do?

  • Unlike the training loop which processes one image at a time, the test evaluation uses vectorized matrix operations (the @ operator) to evaluate all 1,000 test images at once. This is faster and cleaner.
  • No dropout is applied during evaluation. Dropout is a training-only technique. During testing you want the full network making predictions.
  • np.argmax(results, axis=1) picks the highest-scoring digit for each test image, and np.argmax(test_labels, axis=1) picks the actual digit from the one-hot label. Comparing them gives the accuracy.
  • The print statement outputs a summary line for each epoch so you can watch the network improve in real time.
  • Save neural_network.py.
  • Run the script in your terminal:
python neural_network.py
  • You should see 20 lines of output, one per epoch. Watch the Train Loss decrease and Train Acc increase as the network learns. Test Acc should reach approximately 0.78 by the final epoch.

What should I see?

Early epochs will have high loss and low accuracy. By epoch 19, you should see training and test accuracy around 0.78 to 0.80. The sample images plot from Step 2 will also pop up before training begins.

Training seems stuck or accuracy stays low?

  • Check that the test evaluation code (the block starting with results = relu(test_images @ weights_1) @ weights_2) is indented at one level inside the for epoch loop, not inside the for i loop. Wrong indentation means it runs per image instead of per epoch.
  • Make sure the weight initialization from Step 4 still uses 0.2 * rng.random(...) - 0.1. Large initial weights can cause the network to saturate and stop learning.
  • Verify that your normalization step divides by 255.0 (from Step 3). Without normalization, pixel values in the range 0-255 will overwhelm the small weights.
  • help me debug my neural network training loop

Awesome, I've got everything!

Double check you have saved neural_network.py and that your training output shows 20 epoch lines.

I'd like to double check the full code

Here is the complete neural_network.py up to and including this step:

import numpy as np
import requests
import gzip
import os
import matplotlib.pyplot as plt

# =============================================
# Download and Explore the MNIST Dataset
# =============================================

data_sources = {
    "training_images": "train-images-idx3-ubyte.gz",
    "test_images": "t10k-images-idx3-ubyte.gz",
    "training_labels": "train-labels-idx1-ubyte.gz",
    "test_labels": "t10k-labels-idx1-ubyte.gz",
}

data_dir = "mnist_data"
os.makedirs(data_dir, exist_ok=True)

base_url = "https://ossci-datasets.s3.amazonaws.com/mnist/"

for fname in data_sources.values():
    fpath = os.path.join(data_dir, fname)
    if not os.path.exists(fpath):
        print("Downloading " + fname + "...")
        resp = requests.get(base_url + fname, stream=True)
        resp.raise_for_status()
        with open(fpath, "wb") as f:
            for chunk in resp.iter_content(chunk_size=128):
                f.write(chunk)
        print("Done.")

mnist_dataset = {}

for key in ("training_images", "test_images"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(
            f.read(), np.uint8, offset=16
        ).reshape(-1, 28 * 28)

for key in ("training_labels", "test_labels"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(f.read(), np.uint8, offset=8)

x_train = mnist_dataset["training_images"]
y_train = mnist_dataset["training_labels"]
x_test = mnist_dataset["test_images"]
y_test = mnist_dataset["test_labels"]

print("Training images shape:", x_train.shape)
print("Training labels shape:", y_train.shape)
print("Test images shape:", x_test.shape)
print("Test labels shape:", y_test.shape)

fig, axes = plt.subplots(1, 5, figsize=(12, 3))
for i, ax in enumerate(axes):
    ax.imshow(x_train[i].reshape(28, 28), cmap="gray")
    ax.set_title("Label: " + str(y_train[i]))
    ax.axis("off")
plt.suptitle("Sample MNIST Digits")
plt.tight_layout()
plt.savefig("sample_digits.png", dpi=100)
plt.show()

# =============================================
# Preprocess the Data
# =============================================

training_sample, test_sample = 1000, 1000
training_images = x_train[:training_sample] / 255.0
test_images = x_test[:test_sample] / 255.0

print("Training images dtype:", training_images.dtype)
print("Pixel value range:", training_images.min(), "to", training_images.max())


def one_hot_encode(labels, num_classes=10):
    one_hot = labels[..., None] == np.arange(num_classes)[None]
    return one_hot.astype(np.float64)


training_labels = one_hot_encode(y_train[:training_sample])
test_labels = one_hot_encode(y_test[:test_sample])

print("Training labels shape:", training_labels.shape)
print("Example label for digit", y_train[0], ":", training_labels[0])

# =============================================
# Build the Neural Network Architecture
# =============================================


def relu(x):
    return (x >= 0) * x


def relu_derivative(output):
    return output >= 0


seed = 884736743
rng = np.random.default_rng(seed)

pixels_per_image = 784
hidden_size = 100
num_labels = 10

weights_1 = 0.2 * rng.random((pixels_per_image, hidden_size)) - 0.1
weights_2 = 0.2 * rng.random((hidden_size, num_labels)) - 0.1

print("Weights 1 shape:", weights_1.shape)
print("Weights 2 shape:", weights_2.shape)

sample = training_images[0]
hidden = relu(np.dot(sample, weights_1))
output = np.dot(hidden, weights_2)

print("Forward pass output shape:", output.shape)
print("Predicted digit:", np.argmax(output), "| Actual digit:", y_train[0])

# =============================================
# Train the Network with Backpropagation
# =============================================

learning_rate = 0.005
epochs = 20

store_training_loss = []
store_training_accuracy = []
store_test_loss = []
store_test_accuracy = []

for epoch in range(epochs):
    training_loss = 0.0
    training_correct = 0

    for i in range(len(training_images)):
        # Forward propagation
        layer_0 = training_images[i]
        layer_1 = relu(np.dot(layer_0, weights_1))

        # Apply dropout
        dropout_mask = rng.integers(low=0, high=2, size=layer_1.shape)
        layer_1 *= dropout_mask * 2

        layer_2 = np.dot(layer_1, weights_2)

        # Calculate loss
        training_loss += np.sum((training_labels[i] - layer_2) ** 2)
        training_correct += int(
            np.argmax(layer_2) == np.argmax(training_labels[i])
        )

        # Backpropagation
        layer_2_delta = training_labels[i] - layer_2
        layer_1_delta = (
            np.dot(weights_2, layer_2_delta) * relu_derivative(layer_1)
        )
        layer_1_delta *= dropout_mask

        # Update weights
        weights_1 += learning_rate * np.outer(layer_0, layer_1_delta)
        weights_2 += learning_rate * np.outer(layer_1, layer_2_delta)

    store_training_loss.append(training_loss / len(training_images))
    store_training_accuracy.append(training_correct / len(training_images))

    # Evaluate on test set (no dropout)
    results = relu(test_images @ weights_1) @ weights_2
    test_loss = np.sum((test_labels - results) ** 2)
    test_correct = np.sum(
        np.argmax(results, axis=1) == np.argmax(test_labels, axis=1)
    )

    store_test_loss.append(test_loss / len(test_images))
    store_test_accuracy.append(test_correct / len(test_images))

    print(
        f"Epoch {epoch:2d} | "
        f"Train Loss: {training_loss / len(training_images):.3f} | "
        f"Train Acc: {training_correct / len(training_images):.3f} | "
        f"Test Loss: {test_loss / len(test_images):.3f} | "
        f"Test Acc: {test_correct / len(test_images):.3f}"
    )

Your network is learning. You can see loss dropping and accuracy climbing with each epoch. Next up, you will visualize these training curves as plots and see the model's actual predictions on handwritten digits.

Evaluate and Visualize Results

Your neural network just spent 20 epochs learning to recognize handwritten digits. You watched the loss and accuracy numbers scroll by in the terminal, but raw numbers only tell part of the story.

In this step, you will plot the training curves to see how the network improved over time. Then you will visualize actual predictions on test images to see which digits it gets right and which ones trip it up.

In this step, get ready to:
  • Plot training and test loss/accuracy curves and save them as an image.
  • Visualize the model's predictions on real test images with color-coded correctness.
Plot the training curves

Training curves are line charts that show how loss and accuracy changed across each epoch. They are the standard way to diagnose whether a model is learning, stalling, or overfitting.

  • In neural_network.py, add the following code at the very bottom of the file, below the training loop. This creates two side-by-side plots for loss and accuracy:
# =============================================
# Evaluate and Visualize Results
# =============================================

# Create a figure with two side-by-side plots
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
epochs_range = np.arange(1, epochs + 1)

# Plot training and test loss on the left
ax1.plot(epochs_range, store_training_loss, label="Training Loss")
ax1.plot(epochs_range, store_test_loss, label="Test Loss")
ax1.set_title("Loss Over Epochs")
ax1.set_xlabel("Epoch")
ax1.set_ylabel("Loss")
ax1.legend()

# Plot training and test accuracy on the right
ax2.plot(epochs_range, store_training_accuracy, label="Training Accuracy")
ax2.plot(epochs_range, store_test_accuracy, label="Test Accuracy")
ax2.set_title("Accuracy Over Epochs")
ax2.set_xlabel("Epoch")
ax2.set_ylabel("Accuracy")
ax2.legend()

# Save the figure to disk and display it
plt.tight_layout()
plt.savefig("training_results.png", dpi=100)
plt.show()

What does this code do?

  • plt.subplots(1, 2) creates a figure with two charts side by side. The left chart shows loss, the right shows accuracy.
  • Each chart plots two lines. One for training data and one for test data. Comparing them tells you whether the model generalizes or just memorizes.
  • plt.savefig("training_results.png") saves the figure as an image file in your project folder so you can reference it later.
  • Save neural_network.py and run it in your terminal:
python neural_network.py

After training completes, a plot window opens showing two charts. You should see the loss curve dropping from left to right and the accuracy curve climbing from left to right.

  • Confirm that a file called training_results.png was saved to your project folder by running:
ls

You should see training_results.png listed alongside neural_network.py and sample_digits.png.

Plot window not showing up?

  • Some environments (remote servers, WSL) do not support interactive plot windows. The image file training_results.png is still saved even if the window does not appear. Open the file directly from your file browser.
  • Make sure this new code block is placed outside the training loop (no indentation). If it is indented inside the for epoch loop, it will run every epoch instead of once at the end.
  • Still stuck? Help me debug why my Matplotlib plot is not displaying.

What should healthy training curves look like?

A healthy model shows loss decreasing and accuracy increasing over epochs. Both the training and test curves should follow similar trends. If the training accuracy is much higher than test accuracy, the model is overfitting (memorizing training data instead of learning general patterns).

With 1,000 training samples and 20 epochs, expect a final test accuracy of roughly 78-80%. The curves may look a bit noisy because of dropout randomness.

Visualize the model's predictions on test images

Accuracy numbers tell you how often the model is correct overall. Seeing individual predictions tells you which digits it struggles with.

  • In neural_network.py, add this code at the very bottom of the file, below the training curves plot. It creates a 2x5 grid of test images with color-coded prediction labels:
# Create a 2x5 grid to display 10 test predictions
fig, axes = plt.subplots(2, 5, figsize=(14, 6))
for i, ax in enumerate(axes.flat):
    image = test_images[i]
    # Run a forward pass on this single test image
    prediction = np.argmax(relu(np.dot(image, weights_1)) @ weights_2)
    actual = np.argmax(test_labels[i])
    ax.imshow(image.reshape(28, 28), cmap="gray")
    # Color the title green for correct, red for incorrect
    color = "green" if prediction == actual else "red"
    ax.set_title(f"Pred: {prediction} | Real: {actual}", color=color)
    ax.axis("off")

plt.suptitle("Model Predictions on Test Images")
plt.tight_layout()
plt.savefig("predictions.png", dpi=100)
plt.show()

What does this code do?

  • For each of the 10 test images, the code runs a full forward pass through the trained network: input to hidden layer (with ReLU), then hidden to output.
  • np.argmax picks the digit with the highest output score as the prediction.
  • Green titles mean the model predicted correctly. Red titles mean it got the digit wrong. This lets you spot patterns in the model's mistakes at a glance.
  • Save neural_network.py and run it again:
python neural_network.py

After training finishes and the training curves plot closes, a second plot window opens showing a 2x5 grid of handwritten digits. Most titles should be green, with a few red ones where the model got confused.

  • Confirm that predictions.png was saved by running:
ls

You should see predictions.png alongside training_results.png and sample_digits.png.

Seeing all red titles?

  • Check that the predictions code is placed after the training loop, not before it. The network needs to be trained before predictions make sense.
  • Verify that your training loop completed all 20 epochs. If it stopped early due to an error, the weights are not fully trained.
  • Still seeing issues? Help me figure out why my model predictions are all wrong.

Awesome, I've got everything!

Double check that you have saved neural_network.py and that all three image files (sample_digits.png, training_results.png, predictions.png) exist in your project folder.

I'd like to double check the full code

Here is the complete neural_network.py file with all code from every step.

import numpy as np
import requests
import gzip
import os
import matplotlib.pyplot as plt

# =============================================
# Download and Explore the MNIST Dataset
# =============================================

data_sources = {
    "training_images": "train-images-idx3-ubyte.gz",
    "test_images": "t10k-images-idx3-ubyte.gz",
    "training_labels": "train-labels-idx1-ubyte.gz",
    "test_labels": "t10k-labels-idx1-ubyte.gz",
}

data_dir = "mnist_data"
os.makedirs(data_dir, exist_ok=True)

base_url = "https://ossci-datasets.s3.amazonaws.com/mnist/"

for fname in data_sources.values():
    fpath = os.path.join(data_dir, fname)
    if not os.path.exists(fpath):
        print("Downloading " + fname + "...")
        resp = requests.get(base_url + fname, stream=True)
        resp.raise_for_status()
        with open(fpath, "wb") as f:
            for chunk in resp.iter_content(chunk_size=128):
                f.write(chunk)
        print("Done.")

mnist_dataset = {}

for key in ("training_images", "test_images"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(
            f.read(), np.uint8, offset=16
        ).reshape(-1, 28 * 28)

for key in ("training_labels", "test_labels"):
    with gzip.open(os.path.join(data_dir, data_sources[key]), "rb") as f:
        mnist_dataset[key] = np.frombuffer(f.read(), np.uint8, offset=8)

x_train = mnist_dataset["training_images"]
y_train = mnist_dataset["training_labels"]
x_test = mnist_dataset["test_images"]
y_test = mnist_dataset["test_labels"]

print("Training images shape:", x_train.shape)
print("Training labels shape:", y_train.shape)
print("Test images shape:", x_test.shape)
print("Test labels shape:", y_test.shape)

fig, axes = plt.subplots(1, 5, figsize=(12, 3))
for i, ax in enumerate(axes):
    ax.imshow(x_train[i].reshape(28, 28), cmap="gray")
    ax.set_title("Label: " + str(y_train[i]))
    ax.axis("off")
plt.suptitle("Sample MNIST Digits")
plt.tight_layout()
plt.savefig("sample_digits.png", dpi=100)
plt.show()

# =============================================
# Preprocess the Data
# =============================================

training_sample, test_sample = 1000, 1000
training_images = x_train[:training_sample] / 255.0
test_images = x_test[:test_sample] / 255.0

print("Training images dtype:", training_images.dtype)
print("Pixel value range:", training_images.min(), "to", training_images.max())


def one_hot_encode(labels, num_classes=10):
    one_hot = labels[..., None] == np.arange(num_classes)[None]
    return one_hot.astype(np.float64)


training_labels = one_hot_encode(y_train[:training_sample])
test_labels = one_hot_encode(y_test[:test_sample])

print("Training labels shape:", training_labels.shape)
print("Example label for digit", y_train[0], ":", training_labels[0])

# =============================================
# Build the Neural Network Architecture
# =============================================


def relu(x):
    return (x >= 0) * x


def relu_derivative(output):
    return output >= 0


seed = 884736743
rng = np.random.default_rng(seed)

pixels_per_image = 784
hidden_size = 100
num_labels = 10

weights_1 = 0.2 * rng.random((pixels_per_image, hidden_size)) - 0.1
weights_2 = 0.2 * rng.random((hidden_size, num_labels)) - 0.1

print("Weights 1 shape:", weights_1.shape)
print("Weights 2 shape:", weights_2.shape)

sample = training_images[0]
hidden = relu(np.dot(sample, weights_1))
output = np.dot(hidden, weights_2)

print("Forward pass output shape:", output.shape)
print("Predicted digit:", np.argmax(output), "| Actual digit:", y_train[0])

# =============================================
# Train the Network with Backpropagation
# =============================================

learning_rate = 0.005
epochs = 20

store_training_loss = []
store_training_accuracy = []
store_test_loss = []
store_test_accuracy = []

for epoch in range(epochs):
    training_loss = 0.0
    training_correct = 0

    for i in range(len(training_images)):
        # Forward propagation
        layer_0 = training_images[i]
        layer_1 = relu(np.dot(layer_0, weights_1))

        # Apply dropout
        dropout_mask = rng.integers(low=0, high=2, size=layer_1.shape)
        layer_1 *= dropout_mask * 2

        layer_2 = np.dot(layer_1, weights_2)

        # Calculate loss
        training_loss += np.sum((training_labels[i] - layer_2) ** 2)
        training_correct += int(
            np.argmax(layer_2) == np.argmax(training_labels[i])
        )

        # Backpropagation
        layer_2_delta = training_labels[i] - layer_2
        layer_1_delta = (
            np.dot(weights_2, layer_2_delta) * relu_derivative(layer_1)
        )
        layer_1_delta *= dropout_mask

        # Update weights
        weights_1 += learning_rate * np.outer(layer_0, layer_1_delta)
        weights_2 += learning_rate * np.outer(layer_1, layer_2_delta)

    store_training_loss.append(training_loss / len(training_images))
    store_training_accuracy.append(training_correct / len(training_images))

    # Evaluate on test set (no dropout)
    results = relu(test_images @ weights_1) @ weights_2
    test_loss = np.sum((test_labels - results) ** 2)
    test_correct = np.sum(
        np.argmax(results, axis=1) == np.argmax(test_labels, axis=1)
    )

    store_test_loss.append(test_loss / len(test_images))
    store_test_accuracy.append(test_correct / len(test_images))

    print(
        f"Epoch {epoch:2d} | "
        f"Train Loss: {training_loss / len(training_images):.3f} | "
        f"Train Acc: {training_correct / len(training_images):.3f} | "
        f"Test Loss: {test_loss / len(test_images):.3f} | "
        f"Test Acc: {test_correct / len(test_images):.3f}"
    )

# =============================================
# Evaluate and Visualize Results
# =============================================

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
epochs_range = np.arange(1, epochs + 1)

ax1.plot(epochs_range, store_training_loss, label="Training Loss")
ax1.plot(epochs_range, store_test_loss, label="Test Loss")
ax1.set_title("Loss Over Epochs")
ax1.set_xlabel("Epoch")
ax1.set_ylabel("Loss")
ax1.legend()

ax2.plot(epochs_range, store_training_accuracy, label="Training Accuracy")
ax2.plot(epochs_range, store_test_accuracy, label="Test Accuracy")
ax2.set_title("Accuracy Over Epochs")
ax2.set_xlabel("Epoch")
ax2.set_ylabel("Accuracy")
ax2.legend()

plt.tight_layout()
plt.savefig("training_results.png", dpi=100)
plt.show()

fig, axes = plt.subplots(2, 5, figsize=(14, 6))
for i, ax in enumerate(axes.flat):
    image = test_images[i]
    prediction = np.argmax(relu(np.dot(image, weights_1)) @ weights_2)
    actual = np.argmax(test_labels[i])
    ax.imshow(image.reshape(28, 28), cmap="gray")
    color = "green" if prediction == actual else "red"
    ax.set_title(f"Pred: {prediction} | Real: {actual}", color=color)
    ax.axis("off")

plt.suptitle("Model Predictions on Test Images")
plt.tight_layout()
plt.savefig("predictions.png", dpi=100)
plt.show()

Secret mission

Upgrade to Softmax + Cross-Entropy Loss

Your neural network works, but its squared error loss function is holding it back. Can you upgrade to the industry-standard softmax + cross-entropy approach and unlock a meaningful accuracy boost?

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • neural_network.py Python file with the full neural network implementation.
  • mnist_data/ directory containing the downloaded MNIST dataset files (approximately 11 MB).
  • sample_digits.png showing sample MNIST digit images.
  • training_results.png showing training and test loss/accuracy curves.
  • predictions.png showing the model's predictions on test images.
  • Python packages: numpy, matplotlib, and requests installed in your Python environment.

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with different hyperparameters and training sample sizes.

  • Your project folder, MNIST data, and saved images will remain on your computer.
  • The Python packages will stay available for future projects.
  • There are no running processes or ongoing costs to worry about.

Pause - I'll come back to this later

Since this project runs locally with no background processes, there is nothing to pause. Your files will be waiting for you when you come back.

  • If you want to free up disk space, you can delete the mnist_data/ folder. The script will re-download the data automatically when you run it again.
  • Keep neural_network.py and the generated images if you want to reference your work later.

Delete - I don't want to use this again

Remove all project resources and start fresh.

  • Delete your entire project folder, which includes neural_network.py, the mnist_data/ directory, and all saved images.
  • Optionally uninstall the Python packages if you no longer need them by running:
pip uninstall numpy matplotlib requests

Pro tip: keep NumPy and Matplotlib

NumPy and Matplotlib are used in many Python projects. You will likely need them again, so consider keeping them installed even if you delete the project files.

Mission accomplished

That's a Wrap!

Congratulations! You just built a neural network from nothing but math and Python. No TensorFlow. No PyTorch. Just NumPy, raw matrix operations, and your own implementation of every core algorithm that powers modern deep learning.

Here's what you accomplished:

  • Downloaded and preprocessed the MNIST dataset of 70,000 handwritten digit images, normalizing pixel values and one-hot encoding labels for neural network training.
  • Built a 2-layer feedforward neural network from scratch with ReLU activation and dropout regularization, using only NumPy for all matrix operations.
  • Implemented forward propagation, backpropagation, and gradient descent by hand, then trained the model and visualized its learning progress with accuracy and loss curves.
  • Upgraded to softmax + cross-entropy loss for a significant accuracy boost over the basic squared error approach.

Ready to quiz yourself?