Build a Neural Network from Scratch

Build a digit-recognizing neural network using only NumPy, no frameworks.

Introduction

30 Second Summary

Every time your phone unlocks with your face or your email filters out spam, a neural network is making that decision. These networks learn from examples the same way you do: by making mistakes, getting feedback, and adjusting.

In this project, you will build a neural network entirely from scratch using Python and NumPy. Your network will learn to recognize handwritten digits, going from random guessing to over 95% accuracy.

What You'll Build

Run your finished script on real handwritten digit images and see your network identify each one, then display a color-coded grid proving it got over 95% of them right.

By the end of this project, you'll have:

  • A from-scratch neural network that takes a handwritten digit image and correctly identifies it over 95% of the time.
  • A visible training journey where accuracy climbs from 10% (random guessing) to 95%+ through forward propagation, backpropagation, and gradient descent.
  • Visualizations of sample digits as pixel grids and a color-coded prediction grid with green (correct) and red (wrong) labels showing exactly where your network succeeds and fails.
  • Secret Mission: Implement dropout regularization to improve your network's generalization on unseen digits.

Are there any prerequisites?

This project assumes you're comfortable writing and running Python scripts. You'll need Python and VS Code installed on your machine.

Step 1 walks you through verifying your Python version and installing the project's dependencies.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Python Environment

You've committed to building a neural network from scratch. Before writing any network code, you need the right tools installed and a project file ready to go.

In this step, you'll confirm your Python version, install three essential libraries, and create the single file that will hold your entire neural network.

In this step, get ready to:
  • Confirm Python 3.12+ is installed on your machine.
  • Install NumPy, matplotlib, and scikit-learn.
  • Create your project file and verify all imports load.
Confirm your Python version

NumPy 2.5.x requires Python 3.12 or higher. If your Python version is too old, NumPy will fail to install.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Open a terminal inside VS Code by pressing Ctrl+` (the backtick key, usually above Tab) on both macOS and Windows.
  • Check your Python version by running this command:
python --version

✔️ I see version 3.12 or higher

Your Python version is compatible. Move on to the next substep.

ⓧ I see an older version

NumPy 2.5.2 requires Python 3.12 or higher. You need to install a newer version.

  • Download the latest Python from the official downloads page.
  • Run the installer and follow the prompts.
  • After installation, close and reopen your VS Code terminal by pressing Ctrl+` twice.
  • Run python3 --version to confirm the new version is 3.12 or higher.

macOS users

macOS may ship with an older Python that shadows the version you just installed. If python --version still shows the old version, use python3 and pip3 for every command in this project.

ⓧ Command not found

  • Try running python3 --version instead.

If that shows 3.12 or higher, use python3 and pip3 for all commands in this project.

If neither command works, download Python from the official downloads page. Run the installer, restart VS Code, and try the version check again.

Install the project dependencies

Three libraries power different parts of the project. NumPy handles matrix math. matplotlib generates visualizations. scikit-learn loads the handwritten digit dataset.

  • Install all three libraries by running this command:
pip install numpy==2.5.2 matplotlib==3.11.1 scikit-learn==1.9.0

You should see pip download each package and print confirmation that the packages were installed.

What do these libraries do?

  • NumPy provides fast array and matrix operations. Every weight, bias, and activation in your neural network is a NumPy array.
  • matplotlib creates charts and image displays. You'll use it to visualize handwritten digits and plot your network's predictions.
  • scikit-learn is a machine learning library, but you're only using it to download the MNIST handwritten digit dataset. All the actual neural network code is yours.

Seeing 'pip: command not found'?

  • Try running the same command with pip3 instead of pip. If you use python3, the matching installer is usually pip3.
  • If you see a permissions error, add --user to the end of the install command to install packages for your user account only.

Help me fix my pip installation for Python 3.12+.

Create your project file and verify imports
  • Move to your Desktop and create your project folder by running these commands:
cd ~/Desktop
mkdir neural-network
cd neural-network
  • Confirm you're inside the new folder by running:
ls

The output is empty because the folder is brand new. This confirms it was created and you're inside it.

  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the neural-network folder, and click Open.

Seeing a trust dialog?

VS Code may ask Do you trust the authors of the files in this folder? Click Yes, I trust the authors to continue. This dialog appears whenever VS Code opens a new folder for the first time.

  • In the Explorer sidebar on the left, hover over the neural-network folder name and click the New File icon (a page with a plus sign).
  • Name the file neural_network.py and press Enter.
  • Paste the following imports into the file:
import numpy as np
from sklearn.datasets import fetch_openml
import matplotlib.pyplot as plt

What do these imports do?

  • import numpy as np loads NumPy and gives it the shorthand np so you can write np.array() instead of numpy.array().
  • from sklearn.datasets import fetch_openml imports the function that downloads the MNIST handwritten digit dataset.
  • import matplotlib.pyplot as plt loads the plotting module so you can display digit images and prediction charts.
  • Save neural_network.py.

All three libraries are installed and your imports are in place. What do you think will happen when you try to import all three at once?

  • If your terminal closed when the folder opened, reopen it by pressing Ctrl+`.
  • Test your setup by running this command:
python -c "import numpy; import matplotlib; import sklearn; print('All imports OK')"

You should see All imports OK printed in your terminal. This confirms all three libraries are installed and loadable by Python.

Seeing a ModuleNotFoundError?

  • Try running the command with python3 instead of python. If pip installed packages for a different Python version than python points to, the imports will fail.
  • Make sure you used the same Python version for both installing packages and running the command. If you installed with pip3, run with python3.

Why am I getting a ModuleNotFoundError even though I installed the packages?

✔️ Awesome, I've got everything!

Make sure you've saved neural_network.py. You're all set for the next step.

ⓧ I'd like to double check the full code

Your neural_network.py file should contain exactly this:

import numpy as np
from sklearn.datasets import fetch_openml
import matplotlib.pyplot as plt

Your environment is locked in. Next up, you'll load real handwritten digit images and see exactly what your neural network will be working with.

Load and Visualize the Handwritten Digits

Your Python environment is ready with all three libraries installed. Now it is time to meet the data your neural network will learn from.

Before building any network logic, you need to understand what a handwritten digit looks like to a computer. Each image is just a flat grid of 784 numbers representing pixel brightnesses. In this step, you will load those images, inspect their raw numbers, and visualize them so the data stops being abstract.

In this step, get ready to:
  • Load the MNIST handwritten digit dataset and split it into training and test sets.
  • Print the data shapes and raw pixel values to understand the structure.
  • Visualize sample digits as grayscale images and save the figure.
Write the data loading functions

The MNIST dataset contains 70,000 handwritten digit images. Each image is 28x28 pixels, flattened into a row of 784 numbers. Your network will need these images split into a training set (60,000 images to learn from) and a test set (10,000 images to evaluate on).

  • In neural_network.py, add the following two functions below your import statements:
# --- Data Loading ---

def load_mnist():
    """Load MNIST dataset and split into train/test sets."""
    X, y = fetch_openml("mnist_784", version=1, return_X_y=True, as_frame=False)
    y = y.astype(int)
    X = X / 255.0

    X_train, X_test = X[:60000], X[60000:]
    y_train, y_test = y[:60000], y[60000:]
    return X_train, y_train, X_test, y_test


def one_hot_encode(y, num_classes=10):
    """Convert labels to one-hot encoded vectors."""
    one_hot = np.zeros((y.shape[0], num_classes))
    one_hot[np.arange(y.shape[0]), y] = 1
    return one_hot

What does this code do?

load_mnist() downloads 70,000 digit images using scikit-learn's fetch_openml. It converts the labels from strings to integers with .astype(int) (a known gotcha, since fetch_openml returns labels as strings like "7" instead of the number 7).

X = X / 255.0 normalizes pixel values from the 0-255 range down to 0-1. Neural networks learn faster when input values are small and uniform.

one_hot_encode() converts a label like 7 into a vector of 10 values where position 7 is 1 and the rest are 0. You will need this format later when computing loss.

You will not see a result from these functions yet. The next chunk calls them and prints the output.

Load the data and inspect its structure

With the functions defined, you need a main block that actually calls them and prints the data shapes. This confirms you have 60,000 training images and 10,000 test images, each with 784 pixel values.

  • Add the following main block at the bottom of neural_network.py, below your two functions:
# --- Main ---

if __name__ == "__main__":
    # Load data
    print("Loading MNIST dataset...")
    X_train, y_train, X_test, y_test = load_mnist()
    y_train_oh = one_hot_encode(y_train)

    print(f"Training set: {X_train.shape[0]} images, {X_train.shape[1]} pixels each")
    print(f"Test set: {X_test.shape[0]} images")
    print(f"Pixel value range: {X_train.min()} to {X_train.max()}")
    print(f"Sample raw pixels (first 20): {X_train[0][:20]}")

What does this code do?

  • if __name__ == "__main__": ensures this block only runs when you execute the file directly, not when it is imported by another script.
  • load_mnist() downloads and splits the data. The first time you run this, scikit-learn downloads the dataset from the internet and caches it locally.
  • The print statements confirm the data shapes and show the first 20 raw pixel values from a single image. These are the actual numbers your network will process.

First download takes a moment

The first time fetch_openml runs, it downloads the MNIST dataset (~50MB) from the internet. Subsequent runs use the locally cached copy and load much faster.

  • Save neural_network.py.
  • Run your script by entering this command in the terminal:
python neural_network.py

You should see output showing 60,000 training images and 10,000 test images, each with 784 pixels. The pixel value range should be 0.0 to 1.0 (confirming normalization worked), and the raw pixel sample shows a mix of zeros and decimal values.

Seeing an error when loading the dataset?

  • If you see a network error, check your internet connection. The first run must download the dataset.
  • If you see a NameError for fetch_openml, make sure the import from sklearn.datasets import fetch_openml is at the top of your file.
  • If pixel values show numbers above 1.0, check that X = X / 255.0 is inside your load_mnist() function.

Help me debug my MNIST data loading error in neural_network.py

Visualize sample digits

The printed numbers confirm the data loaded correctly, but 784 decimal values are hard to interpret. Reshaping those 784 numbers back into a 28x28 grid and displaying them as an image reveals the actual handwritten digit your network will learn to recognize.

  • In neural_network.py, add the following visualization code at the bottom of your if __name__ == "__main__": block, below the last print statement:
    # Visualize sample digits
    fig, axes = plt.subplots(1, 5, figsize=(12, 3))
    for i, ax in enumerate(axes):
        ax.imshow(X_train[i].reshape(28, 28), cmap="gray")
        ax.set_title(f"Label: {y_train[i]}")
        ax.axis("off")
    plt.suptitle("Sample MNIST Digits")
    plt.tight_layout()
    plt.savefig("sample_digits.png")
    plt.show()
    print("Saved sample_digits.png\n")

What does this code do?

  • plt.subplots(1, 5) creates a row of 5 side-by-side image slots.
  • .reshape(28, 28) folds the flat 784-number array back into a 28x28 pixel grid so matplotlib can display it as an image.
  • cmap="gray" renders the image in grayscale. Each pixel's brightness corresponds to the 0-1 value your network sees.
  • plt.savefig("sample_digits.png") saves the figure to your project folder before plt.show() opens the interactive window.
  • Save neural_network.py.

Before you run the script again, what do you think the 5 images will look like? Will they be clean and uniform, or messy and varied?

  • Run the script again by entering this command:
python neural_network.py

A window pops up showing 5 handwritten digit images rendered in grayscale. Each image has a label showing the actual digit it represents. The digits look hand-drawn and imperfect. That is real human handwriting, and your network will learn to read it.

Close the matplotlib window to let the script finish. You should also see Saved sample_digits.png printed in the terminal, and a new sample_digits.png file in your project folder.

Not seeing the image window?

  • Make sure the visualization code is indented inside the if __name__ == "__main__": block. Python is whitespace-sensitive, so incorrect indentation means the code never runs.
  • If the window appeared behind other windows, check your taskbar or dock for a Python or matplotlib icon.
  • If the script hangs without showing a window, your matplotlib backend may not support interactive display. Check that sample_digits.png was saved to your project folder. The saved file confirms the code worked.

Help me debug why my matplotlib visualization is not showing up

✔️ Awesome, I've got everything!

Great. Double check that you have saved neural_network.py and that sample_digits.png exists in your project folder.

ⓧ I'd like to double check the full code

Scroll up and confirm your neural_network.py has each of these sections in order:

  • The three import lines at the top (numpy, fetch_openml, matplotlib.pyplot).
  • The load_mnist() function that fetches, normalizes, and splits the data.
  • The one_hot_encode() function that converts labels to one-hot vectors.
  • The if __name__ == "__main__": block with data loading, print statements, and the visualization code.

You now have real handwritten digits loaded, normalized, and visualized. Your network has its data. Next up, you will build the forward pass that turns those 784 pixel values into a prediction of which digit the image shows.

Build the Forward Pass and See It Fail

You have loaded the MNIST dataset and visualized real handwritten digits. You know that each image is just a grid of 784 numbers representing pixel brightness. Now the question is: how does a neural network turn those numbers into a prediction?

The forward pass is the answer. It is the sequence of math that transforms raw pixel values into a probability for each digit (0 through 9). In this step, you will build a complete forward pass, run it on 10,000 test images, and see what happens when the network has never learned anything.

In this step, get ready to:
  • Initialize random weights and biases for a 2-layer network.
  • Implement ReLU and softmax activation functions.
  • Write the forward pass and measure initial accuracy on test data.
Initialize weights and activation functions

A neural network stores everything it "knows" in weights and biases. Before training, these values are completely random. You also need activation functions that decide which neurons fire and how the network converts raw scores into probabilities.

  • In neural_network.py, find the line # --- Main ---. Add the following code above it:
# --- Weight Initialization (naive) ---

def initialize_weights(input_size, hidden_size, output_size):
    """Initialize weights with small random values and biases to zero."""
    np.random.seed(42)
    params = {
        "W1": np.random.randn(input_size, hidden_size) * 0.01,
        "b1": np.zeros((1, hidden_size)),
        "W2": np.random.randn(hidden_size, output_size) * 0.01,
        "b2": np.zeros((1, output_size)),
    }
    return params

# --- Activation Functions ---

def relu(Z):
    """ReLU activation: returns max(0, Z)."""
    return np.maximum(0, Z)

def relu_derivative(Z):
    """Derivative of ReLU: 1 where Z > 0, else 0."""
    return (Z > 0).astype(float)

def softmax(Z):
    """Softmax activation with numerical stability."""
    exp_Z = np.exp(Z - np.max(Z, axis=1, keepdims=True))
    return exp_Z / np.sum(exp_Z, axis=1, keepdims=True)

What does this code do?

initialize_weights creates four arrays: two weight matrices (W1 and W2) and two bias vectors (b1 and b2). Multiplying by 0.01 keeps the initial weight values small so signals don't explode as they pass through layers.

ReLU (Rectified Linear Unit) is the simplest activation function. It lets positive values pass through and blocks negatives. relu_derivative returns 1 where the input was positive and 0 elsewhere. You will use it during backpropagation in the next step.

Softmax converts a vector of raw scores into probabilities that sum to 1. Subtracting the max value before exponentiating prevents numerical overflow without changing the result.

  • Save neural_network.py. These building blocks come alive in the next chunk when you chain them into the forward pass.
Build the forward pass and accuracy functions

The forward pass chains the building blocks together into a pipeline. Data flows in as 784 pixel values, gets multiplied by weights, pushed through activation functions, and comes out as 10 probabilities.

  • In neural_network.py, find the line # --- Main --- again. Add the following code above it, below the softmax function:
# --- Forward Pass ---

def forward(X, params):
    """Compute forward pass through the network."""
    Z1 = X @ params["W1"] + params["b1"]
    A1 = relu(Z1)
    Z2 = A1 @ params["W2"] + params["b2"]
    A2 = softmax(Z2)
    cache = {"Z1": Z1, "A1": A1, "Z2": Z2, "A2": A2}
    return A2, cache

# --- Prediction and Accuracy ---

def predict(X, params):
    """Return predicted class labels."""
    A2, _ = forward(X, params)
    return np.argmax(A2, axis=1)

def accuracy(X, y, params):
    """Compute classification accuracy."""
    predictions = predict(X, params)
    return np.mean(predictions == y)

What does the forward pass do?

  • Z1 = X @ W1 + b1 multiplies the 784 pixel values by the first weight matrix and adds a bias. The @ operator is NumPy's matrix multiplication.
  • A1 = relu(Z1) applies the ReLU activation, zeroing out negative values. This creates the 128 hidden neuron activations.
  • Z2 = A1 @ W2 + b2 multiplies the hidden activations by the second weight matrix to produce 10 raw scores (one per digit).
  • A2 = softmax(Z2) converts those 10 raw scores into probabilities. The highest probability is the network's guess.
  • predict() picks the digit with the highest probability using np.argmax. accuracy() compares predictions to actual labels and returns the fraction that match.
  • Save neural_network.py. The forward pass pipeline is complete. The next chunk puts it to work on real test data.
Run the network and see it guess randomly

The network has all its architecture in place. It can take in an image and output a prediction. Time to see how it performs with completely random, untrained weights.

  • In neural_network.py, scroll to the bottom of the if __name__ == "__main__": block. Find the line print("Saved sample_digits.png\n"). Add the following code below it:
    # Initialize network and check initial accuracy
    print("Initializing network...")
    params = initialize_weights(784, 128, 10)

    initial_acc = accuracy(X_test, y_test, params)
    print(f"Initial accuracy (random weights): {initial_acc:.4f}")
    print("(Expected ~10% since there are 10 digits and we are guessing randomly)\n")

What does this code do?

initialize_weights(784, 128, 10) creates a network with 784 inputs (one per pixel), 128 hidden neurons, and 10 outputs (one per digit class). Then accuracy() runs all 10,000 test images through the forward pass and compares the predictions to the real labels.

  • Save neural_network.py.

Before you run this, make a prediction: what accuracy do you expect from a network with completely random weights? There are 10 possible digit classes (0 through 9).

  • Run the script by executing this command in your terminal:
python neural_network.py

After the sample digits window appears and you close it, look at the terminal output. The initial accuracy prints as approximately 0.10. With 10 possible digits, 10% is pure random guessing. The architecture is in place, but the network has learned absolutely nothing.

Why is ~10% accuracy expected?

If you randomly guess which digit an image shows, you have a 1-in-10 chance of being right (10 classes, each equally likely). That is exactly what the network is doing. Its weights are random noise, so its "predictions" are meaningless.

This is the designed starting point. The forward pass turns inputs into outputs, but without a way to learn from mistakes, the network cannot improve. That is what you will build next.

Not seeing the accuracy output?

  • Make sure you saved neural_network.py after adding the new code. Unsaved changes will not run.
  • Check that the new lines are inside the if __name__ == "__main__": block with 4 spaces of indentation. If they are at the top level (no indent), they will run but may error because X_test and y_test are not defined there.
  • The sample digits window must be closed before the script continues. If the script seems stuck, close the matplotlib window.

Help me debug my forward pass output.

✔️ Awesome, I've got everything!

Great. Double check you have saved neural_network.py and that your terminal printed an initial accuracy of approximately 0.10.

ⓧ I'd like to double check the full code

Scroll up through your neural_network.py and confirm you have each of these sections in order:

  • Imports at the top: numpy, fetch_openml, and matplotlib.pyplot.
  • Data Loading: load_mnist() and one_hot_encode() functions.
  • Weight Initialization (naive): initialize_weights() using np.random.randn(...) * 0.01.
  • Activation Functions: relu(), relu_derivative(), and softmax().
  • Forward Pass: forward() function returning predictions and a cache.
  • Prediction and Accuracy: predict() and accuracy() functions.
  • Main block: loads data, visualizes digits, initializes the network with initialize_weights(784, 128, 10), and prints initial accuracy.

The forward pass works, but 10% accuracy is no better than a coin flip across 10 digits. The network has no way to correct its mistakes. In the next step, you will give it exactly that: a loss function to measure how wrong it is, and backpropagation to figure out which weights caused the error.

Teach the Network to Learn

In the last step, you proved something important. A neural network with random weights is no smarter than a coin flip. It scored about 10% accuracy on 10 digit classes, which is pure guessing.

The network guesses randomly because it has never been corrected. To fix this, you need two things: a way to measure how wrong the network is, and a way to figure out which weights caused the mistake. Together, these let the network learn from its errors.

In this step, get ready to:
  • Implement cross-entropy loss to measure prediction errors.
  • Write the backpropagation algorithm to compute gradients.
  • Run a training loop and watch the network learn for the first time.
Implement the cross-entropy loss function

Your network currently produces a prediction for each digit (a probability from 0 to 1 for each of the 10 classes). You need a way to score how far off those predictions are from the correct answer.

This is what a loss function does. Cross-entropy loss penalizes confident wrong answers much more heavily than uncertain ones. If your network says "I'm 99% sure this is a 3" and it's actually a 7, the loss is massive.

What about one-hot encoding?

You already have a one_hot_encode() function from Step 2. It converts a label like "7" into a vector of 10 values where position 7 is 1 and the rest are 0. This matches the shape of your network's softmax output, so the loss function can compare them directly.

  • In neural_network.py, find the # --- Prediction and Accuracy --- section comment.
  • Add the cross-entropy loss function above that section by copying this code:
# --- Loss Function ---

def cross_entropy_loss(predictions, targets):
    """Compute cross-entropy loss."""
    m = targets.shape[0]
    loss = -np.sum(targets * np.log(predictions + 1e-8)) / m
    return loss

What does this code do?

Cross-entropy compares the network's predicted probabilities against the one-hot encoded target. When the prediction matches the target, np.log of a value near 1 is close to 0, so the loss is small. When the prediction is wrong, the log of a value near 0 produces a huge negative number, making the loss large.

The 1e-8 prevents taking the log of zero, which would crash the program. Dividing by m averages the loss across all samples so the value stays comparable regardless of dataset size.

Implement backpropagation

Now you can measure how wrong the network is. The next question is: which weights caused the mistake? Backpropagation answers this by working backwards through the network, using the chain rule from calculus to trace each weight's contribution to the final error.

  • Add the backward pass function directly below cross_entropy_loss() (and still above the # --- Prediction and Accuracy --- section) by copying this code:
# --- Backward Pass ---

def backward(X, targets, params, cache):
    """Compute gradients using backpropagation."""
    m = X.shape[0]

    # Output layer gradients
    dZ2 = cache["A2"] - targets
    dW2 = (cache["A1"].T @ dZ2) / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m

    # Hidden layer gradients
    dA1 = dZ2 @ params["W2"].T
    dZ1 = dA1 * relu_derivative(cache["Z1"])
    dW1 = (X.T @ dZ1) / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m

    grads = {"dW1": dW1, "db1": db1, "dW2": dW2, "db2": db2}
    return grads

What does this code do?

The function computes a gradient for every weight and bias. A gradient tells you how much the loss would change if you nudged that weight up or down slightly.

  • dZ2 = cache["A2"] - targets computes the output layer error. The gradient of softmax combined with cross-entropy simplifies to just (prediction minus target).
  • dA1 = dZ2 @ params["W2"].T sends the error signal backward through the second layer's weights to the hidden layer.
  • dZ1 = dA1 * relu_derivative(cache["Z1"]) masks the gradient by the ReLU derivative. Neurons that were "off" (output was 0) get zero gradient, so only active neurons get updated.
  • Save neural_network.py.
  • Run the script to confirm your new functions load without errors:
python neural_network.py

You should see the same output as before: dataset loading, sample digit visualization, and initial accuracy of about 0.10. The output has not changed because the new functions are defined but not called yet. A clean run confirms they have no syntax errors.

Seeing a NameError or SyntaxError?

  • Check that cross_entropy_loss() and backward() are at the top level of the file (not indented inside another function or the if __name__ block).
  • Make sure the function signatures match exactly, including the cache parameter in backward().
  • Verify that the two new function sections are placed between forward() and predict().

Why is my script crashing after adding cross_entropy_loss and backward?

Complete the training loop

You can measure errors and compute gradients. The final piece is a function that actually nudges the weights in the right direction. This is gradient descent: subtract a small fraction of each gradient from each weight, so the loss shrinks a little with every update.

  • Add the parameter update function directly below backward() (still above # --- Prediction and Accuracy ---) by copying this code:
# --- Parameter Updates ---

def update_params(params, grads, learning_rate):
    """Update weights and biases using gradient descent."""
    params["W1"] -= learning_rate * grads["dW1"]
    params["b1"] -= learning_rate * grads["db1"]
    params["W2"] -= learning_rate * grads["dW2"]
    params["b2"] -= learning_rate * grads["db2"]
    return params

What does this code do?

Each line subtracts the gradient (scaled by a learning rate) from its corresponding weight or bias. The learning rate controls how big each adjustment is. Too large and the network overshoots. Too small and it barely learns.

Now you have all the pieces: forward pass, loss, backpropagation, and weight updates. Time to wire them together in a training loop that repeats 10 times so you can watch the network learn.

  • Scroll down to the bottom of your if __name__ == "__main__": block.
  • Find the last two lines that print the initial accuracy message:
    print(f"Initial accuracy (random weights): {initial_acc:.4f}")
    print("(Expected ~10% since there are 10 digits and we are guessing randomly)\n")
  • Add the training loop below those lines by copying this code:
    # Run 10 manual training iterations
    print("Running 10 training iterations...")
    learning_rate = 0.1
    for i in range(10):
        predictions, cache = forward(X_train, params)
        loss = cross_entropy_loss(predictions, y_train_oh)
        grads = backward(X_train, y_train_oh, params, cache)
        params = update_params(params, grads, learning_rate)
        print(f"Iteration {i + 1}/10 - Loss: {loss:.4f}")

What does this training loop do?

Each of the 10 iterations runs the complete learning cycle on all 60,000 training images at once:

  • forward() turns images into predictions.
  • cross_entropy_loss() scores how wrong those predictions are.
  • backward() figures out which weights caused the errors.
  • update_params() nudges every weight to reduce the error.

The learning rate of 0.1 controls the size of each adjustment. The loss printed after each iteration should get smaller, proving the network is learning.

  • Save neural_network.py.

Before you run the script, think about what you expect to happen. The network scored 10% accuracy with random weights. After 10 rounds of measuring mistakes and adjusting weights, do you think the loss will go up or down?

  • Run the script:
python neural_network.py

After the dataset loads and the sample digits display, you should see the loss decrease with each iteration. Your network is learning for the first time.

What should I see?

The 10 training iterations should print lines like this, with the loss getting smaller each time:

Running 10 training iterations...
Iteration 1/10 - Loss: 2.3026
Iteration 2/10 - Loss: 2.2753
...
Iteration 10/10 - Loss: 1.5xxx

Your exact numbers may differ slightly, but the trend should be clearly downward. Each iteration makes the predictions a little less wrong.

Loss not decreasing or seeing NaN?

  • If the loss stays flat, check that update_params() uses -= (subtract-assign), not +=. Adding gradients makes the loss worse, not better.
  • If you see NaN values, make sure the 1e-8 is present inside np.log(predictions + 1e-8) in your loss function. Without it, log(0) produces infinity.
  • Check that you are passing y_train_oh (the one-hot encoded version) to cross_entropy_loss() and backward(), not y_train.

Why is my training loop producing NaN or not decreasing the loss?

✔️ Awesome, I've got everything!

Great. Double check you have saved neural_network.py and that your loss is decreasing across all 10 iterations.

ⓧ I'd like to double check the full code

Your neural_network.py file is getting long, so scroll through and confirm these sections appear in this exact order:

  • Imports (numpy, sklearn, matplotlib) at the top.
  • # --- Data Loading --- with load_mnist() and one_hot_encode().
  • # --- Weight Initialization (naive) --- with initialize_weights().
  • # --- Activation Functions --- with relu(), relu_derivative(), and softmax().
  • # --- Forward Pass --- with forward().
  • # --- Loss Function --- with cross_entropy_loss() (new in this step).
  • # --- Backward Pass --- with backward() (new in this step).
  • # --- Parameter Updates --- with update_params() (new in this step).
  • # --- Prediction and Accuracy --- with predict() and accuracy().
  • if __name__ == "__main__": block ending with the 10-iteration training loop.

Your network is learning. Each iteration, the loss drops a little as the weights get nudged toward better predictions. But 10 iterations is not enough to learn 60,000 images well. In the next step, you will build a full training loop, fix the weight initialization, and push accuracy past 95%.

Train to 95%+ Accuracy

Your network learned to reduce its loss over ten iterations. That proved the forward pass, backpropagation, and weight updates all work together. But ten iterations on 60,000 images barely scratches the surface.

To reach 95%+ accuracy, you need two things: smarter weight initialization and a proper training loop that runs for many more epochs. The current initialization (multiplying random values by 0.01) makes signals shrink as they pass through layers, strangling the learning process. In this step, you will fix that and train for real.

In this step, get ready to:
  • Replace naive weight initialization with He initialization to keep signals flowing through layers.
  • Build a full training loop that runs for 100 epochs and tracks accuracy on both training and test data.
  • Visualize your network's predictions on real handwritten digit images with color-coded results.
Upgrade to He initialization

Right now, every weight in your network starts as a tiny random number scaled by 0.01. When signals pass through multiple layers of tiny weights, they shrink toward zero. Gradients shrink too, which means the network barely learns anything per iteration.

What is He initialization?

He initialization scales each weight by np.sqrt(2.0 / fan_in), where fan_in is the number of inputs feeding into that layer. This keeps the variance of activations stable as signals flow forward through ReLU layers.

Stable variance means gradients stay large enough to actually update weights meaningfully. The network learns faster because each iteration makes a real difference.

  • In neural_network.py, find the initialize_weights() function near the top of your file. It currently looks like this:
# --- Weight Initialization (naive) ---

def initialize_weights(input_size, hidden_size, output_size):
    """Initialize weights with small random values and biases to zero."""
    np.random.seed(42)
    params = {
        "W1": np.random.randn(input_size, hidden_size) * 0.01,
        "b1": np.zeros((1, hidden_size)),
        "W2": np.random.randn(hidden_size, output_size) * 0.01,
        "b2": np.zeros((1, output_size)),
    }
    return params
  • Replace the entire function (including the section comment above it) with this updated version:
# --- Weight Initialization ---

def initialize_weights(input_size, hidden_size, output_size):
    """Initialize weights using He initialization and biases to zero."""
    np.random.seed(42)
    params = {
        "W1": np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size),
        "b1": np.zeros((1, hidden_size)),
        "W2": np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size),
        "b2": np.zeros((1, output_size)),
    }
    return params

What changed?

The only difference is how the weights are scaled. Instead of * 0.01, each weight matrix is now scaled by np.sqrt(2.0 / fan_in).

For W1 with 784 inputs, that is roughly 0.05. For W2 with 128 inputs, that is roughly 0.125. Larger layers get smaller initial weights, smaller layers get bigger ones. This balance is what keeps signals stable.

  • Now scroll down to the __main__ block and find the line that says:
    print("Initializing network...")
  • Replace it with this line so the output reflects the change you just made:
    print("Initializing network with He initialization...")
  • Save neural_network.py and run your script by running this command:
python neural_network.py
  • Close the sample digits plot window when it appears. You should see the 10 training iterations run, but the loss values should be noticeably lower than before.

Why is the loss lower?

He initialization gives the network a much better starting point. The weights are sized so that signals neither explode nor vanish as they flow through layers. Each training iteration now makes a bigger, more meaningful update.

Loss values look the same as before?

Make sure you replaced both * 0.01 lines in the function. One is for W1 and the other is for W2. If either still uses 0.01, the network will not benefit from He initialization.

Help me debug my He initialization code.

Build the training loop

The 10-iteration loop from the previous step was a proof of concept. To reach 95%+ accuracy, you need to train for 100 epochs and track how accuracy improves on both the training set and the test set.

  • In neural_network.py, find the accuracy() function. Add this new train() function below it, just above the # --- Main --- comment:
# --- Training Loop ---

def train(X_train, y_train_oh, y_train, X_test, y_test, params, epochs, learning_rate):
    """Train the neural network."""
    for epoch in range(epochs):
        predictions, cache = forward(X_train, params)
        loss = cross_entropy_loss(predictions, y_train_oh)
        grads = backward(X_train, y_train_oh, params, cache)
        params = update_params(params, grads, learning_rate)

        if (epoch + 1) % 10 == 0:
            train_acc = accuracy(X_train, y_train, params)
            test_acc = accuracy(X_test, y_test, params)
            print(
                f"Epoch {epoch + 1}/{epochs} - Loss: {loss:.4f} "
                f"- Train Acc: {train_acc:.4f} - Test Acc: {test_acc:.4f}"
            )

    return params

What does this function do?

Each epoch runs the full pipeline you built: forward pass, loss, backward pass, and weight update. After every 10 epochs, it pauses to measure accuracy on both the training data and the test data.

Tracking test accuracy separately is important. Training accuracy tells you if the network can memorize its data. Test accuracy tells you if it can generalize to digits it has never seen.

Now you need to replace the old 10-iteration loop in your __main__ block with a call to this new function.

  • Scroll down to the __main__ block and find this code at the bottom of the file:
    # Run 10 manual training iterations
    print("Running 10 training iterations...")
    learning_rate = 0.1
    for i in range(10):
        predictions, cache = forward(X_train, params)
        loss = cross_entropy_loss(predictions, y_train_oh)
        grads = backward(X_train, y_train_oh, params, cache)
        params = update_params(params, grads, learning_rate)
        print(f"Iteration {i + 1}/10 - Loss: {loss:.4f}")
  • Delete those lines entirely and replace them with this code:
    # Train the network
    print("Training...")
    params = train(X_train, y_train_oh, y_train, X_test, y_test, params, epochs=100, learning_rate=0.1)

    # Final results
    final_acc = accuracy(X_test, y_test, params)
    print(f"\nFinal test accuracy: {final_acc:.4f}")

What changed here?

Instead of a manual loop, you are calling your train() function with 100 epochs and a learning rate of 0.1. The function handles the forward-backward-update cycle internally and prints progress every 10 epochs.

After training finishes, you compute the final test accuracy separately and print it on its own line. This is the number that tells you how well your network performs on digits it has never trained on.

Before you run this, what accuracy do you think the network will reach after 100 epochs with He initialization? Remember, 10% is random guessing.

  • Save neural_network.py and run the script by running this command:
python neural_network.py
  • Close the sample digits plot window when it appears. Training will take roughly 30 to 60 seconds depending on your machine. Watch the accuracy climb every 10 epochs.

You should see the final line print a test accuracy of 0.95 or above. Your network just went from random guessing to correctly identifying 95 out of every 100 handwritten digits.

Accuracy stuck below 90%?

Check that your initialize_weights() function uses np.sqrt(2.0 / input_size) for W1 and np.sqrt(2.0 / hidden_size) for W2. Also confirm you set epochs=100 in the train() call. With only 10 epochs, the network does not have enough time to converge.

Help me debug my training accuracy.

Visualize your network's predictions

A 95% accuracy number is impressive, but seeing the network's predictions on actual digit images makes the result feel real. You will create a grid that shows what the network predicts versus the true label, with green for correct and red for wrong.

  • In neural_network.py, add this visualization code at the very end of the __main__ block, below the print(f"\nFinal test accuracy: {final_acc:.4f}") line:
    # Visualize predictions
    np.random.seed(0)
    fig, axes = plt.subplots(2, 5, figsize=(12, 6))
    for ax in axes.flat:
        idx = np.random.randint(0, X_test.shape[0])
        ax.imshow(X_test[idx].reshape(28, 28), cmap="gray")
        pred = predict(X_test[idx : idx + 1], params)[0]
        actual = y_test[idx]
        color = "green" if pred == actual else "red"
        ax.set_title(f"Pred: {pred} | Real: {actual}", color=color)
        ax.axis("off")
    plt.suptitle("Network Predictions (Green=Correct, Red=Wrong)")
    plt.tight_layout()
    plt.savefig("predictions.png")
    plt.show()
    print("Saved predictions.png")

What does this code do?

  • np.random.seed(0) fixes the random selection so you see the same 10 images each time you run the script.
  • plt.subplots(2, 5) creates a 2-row, 5-column grid. For each cell, a random test image is drawn and fed through predict().
  • Each title shows the predicted digit and the actual digit. Green means the network got it right. Red means it got it wrong.
  • Save neural_network.py and run the script by running this command:
python neural_network.py
  • Close the sample digits window and wait for training to finish. After the final accuracy prints, a second window will appear with your 2x5 predictions grid.
  • You should see mostly green labels. Any red labels show digits the network got wrong. Close the window and confirm you see Saved predictions.png printed in the terminal.

Not seeing the predictions window?

Make sure you closed the first plot window (the sample digits). matplotlib waits for you to close each plt.show() window before continuing the script. The predictions grid only appears after training finishes and the first window is closed.

Help me fix my matplotlib visualization.

✔️ Awesome, I've got everything!

Great. Double check you have saved neural_network.py and that predictions.png is in your project folder.

ⓧ I'd like to double check the full code

Your neural_network.py is too long to show in one block. Scroll up and confirm each of these pieces is present, in order:

  • Imports: numpy, fetch_openml, matplotlib.pyplot.
  • load_mnist() and one_hot_encode() functions.
  • initialize_weights() using He initialization (np.sqrt(2.0 / input_size) and np.sqrt(2.0 / hidden_size)).
  • relu(), relu_derivative(), and softmax() activation functions.
  • forward(), cross_entropy_loss(), backward(), and update_params().
  • predict() and accuracy() helper functions.
  • train() function with 100-epoch loop and accuracy tracking every 10 epochs.
  • __main__ block: loads data, visualizes samples, initializes with He, trains, prints final accuracy, and saves predictions.png.

Your neural network can now recognize handwritten digits with over 95% accuracy. Every piece of the learning pipeline (forward pass, loss, backpropagation, weight updates) was written by you from scratch. Next up, the Secret Mission challenges you to make the network even more robust.

Secret mission

Add Dropout Regularization

Your network hits 95%+ accuracy, but can it generalize even better? Dropout forces neurons to learn redundant representations by randomly turning some off during training. Add inverted dropout to your forward and backward passes, then compare the results.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder on your Desktop containing neural_network.py, sample_digits.png, and predictions.png.
  • Scikit-learn cached MNIST data at ~/scikit_learn_data/ (approximately 50MB).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your neural network or want to try different architectures.

  • Your neural_network.py file and generated images stay on your Desktop. No ongoing cost.
  • The cached MNIST data at ~/scikit_learn_data/ stays on your machine. Future runs will skip the download step.

Pause - I'll come back to this later

Free up disk space by removing the cached MNIST data while keeping your project files intact.

  • Delete the scikit-learn data cache. On macOS or Linux, run:
rm -rf ~/scikit_learn_data
  • On Windows, open PowerShell and run:
Remove-Item -Recurse -Force "$env:USERPROFILE\scikit_learn_data"
  • Your project folder stays on your Desktop. When you run the script again, it will re-download the MNIST data automatically.

Delete - I don't want to use this again

Remove all project files and cached data from your machine. Nothing external needs cleaning up.

  • Delete the scikit-learn data cache. On macOS or Linux, run:
rm -rf ~/scikit_learn_data
  • On Windows, open PowerShell and run:
Remove-Item -Recurse -Force "$env:USERPROFILE\scikit_learn_data"
  • Delete your project folder from the Desktop. On macOS or Linux, run:
cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]
  • On Windows, open PowerShell and run:
cd $env:USERPROFILE\Desktop
Remove-Item -Recurse -Force [[PROJECT_FOLDER="your project folder name"]]

Nice Work!

Nice Work!

You just built a neural network from scratch that recognizes handwritten digits with over 95% accuracy. Every piece of the learning pipeline was written by you, with nothing hidden behind a framework.

You've learned how to:

  • Build a 2-layer neural network from scratch using only NumPy, implementing forward propagation, cross-entropy loss, backpropagation, and gradient descent.
  • Load and visualize real handwritten digit images from the MNIST dataset, and understand how machines convert images into arrays of numbers that a network can process.
  • Train the network to 95%+ accuracy on 10,000 unseen test images, and understand why He initialization outperforms naive random initialization for ReLU networks.
  • Secret Mission: Added dropout regularization to prevent overfitting, modifying both the forward and backward passes to handle inverted dropout.

Ready to quiz yourself?