Build a Neural Network from Scratch

Build a neural network in Python that recognizes handwritten digits.

Introduction

30 Second Summary

Every time your phone recognizes your face or your email filters out spam, a neural network is making that decision behind the scenes. These networks learn from examples, much like how you learned to read handwriting by seeing thousands of letters growing up.

In this project, you will build a neural network entirely from scratch using Python and NumPy. Your network will look at images of handwritten numbers and correctly predict which digit they are.

What You'll Build

A fully functional neural network that recognizes handwritten digits (0-9) from the MNIST dataset, implemented entirely from scratch with no deep learning frameworks.

By the end of this project, you'll have:

  • A working neural network that classifies handwritten digits with ~95% accuracy, built using only NumPy.
  • A training visualization showing the network's loss decreasing over epochs as it learns from 60,000 images.
  • A prediction display showing test images alongside the network's guesses, proving it works on data it has never seen before.
  • Secret Mission: Add a second hidden layer and compare deep vs shallow network performance.

Are there any prerequisites?

This project assumes you're comfortable with Python basics like functions, classes, and loops. You should have a basic sense of what a matrix or array is, but no advanced math is needed.

Set Up Your Environment

Before your neural network can learn anything, you need a clean workspace with the right Python libraries installed.

In this step, you'll create a dedicated project folder, set up a virtual environment to keep your dependencies isolated, and install the three libraries that power your network.

In this step, get ready to:
  • Create a project folder and virtual environment.
  • Install NumPy, Matplotlib, and scikit-learn.
  • Create your main Python file and verify the imports work.
Create a project folder and virtual environment

A virtual environment is an isolated space where Python stores the libraries for a specific project. It prevents conflicts between projects that need different library versions.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder, move into it, and create a virtual environment by running these commands:
mkdir neural-network
cd neural-network
python -m venv venv
  • Confirm the folder was created by running:
ls

You should see a venv folder in the output.

  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

After activation, your terminal prompt changes to show (venv) at the beginning. This confirms you are working inside the virtual environment.

Don't see (venv) in your prompt?

Make sure you ran the activation command for your operating system. macOS uses source venv/bin/activate and Windows uses venv\Scripts\activate.

On Windows, if you see an error about execution policies, run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned first, then try the activation command again.

Help me troubleshoot my virtual environment activation

Install the project dependencies

Your neural network needs three libraries. NumPy handles all the matrix math. Matplotlib draws training charts. scikit-learn provides easy access to the MNIST handwritten digits dataset.

  • Install all three libraries by running this command:
pip install numpy matplotlib scikit-learn

What does this command do?

The pip install command downloads and installs Python packages from the Python Package Index (PyPI). Because your virtual environment is active, these libraries install only inside the venv folder, not globally on your system.

  • numpy provides fast array and matrix operations that make neural network math efficient.
  • matplotlib creates plots and charts so you can visualize your network's learning progress.
  • scikit-learn gives you one-line access to the MNIST dataset of 70,000 handwritten digit images.

The installation takes a minute or two. When it finishes, you should see a Successfully installed message listing the packages.

Seeing an installation error?

If you see a permissions error, make sure (venv) appears in your terminal prompt. If it does not, go back and run the activation command for your OS.

If you see an error about python not being found, try replacing pip with pip3 in the command.

Help me fix my pip install error

Create the main file and verify imports

Now that your libraries are installed, you need a Python file to write your neural network code in.

  • In your code editor, open the neural-network folder on your Desktop.
  • Create a new file called neural_network.py in the root of the neural-network folder.
  • Add these imports at the top of neural_network.py:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml

What do these imports do?

  • import numpy as np loads NumPy and gives it the shorthand np so you can write np.array() instead of numpy.array().
  • import matplotlib.pyplot as plt loads the plotting module so you can create charts with plt.plot().
  • from sklearn.datasets import fetch_openml imports a single function that downloads the MNIST digit images directly from OpenML.
  • Save neural_network.py.
  • Switch back to your terminal and verify the imports work by running:
python neural_network.py

You should see no output at all. A blank return to the prompt means all three libraries imported successfully.

Seeing a ModuleNotFoundError?

This means Python cannot find one of the libraries you installed. Check that (venv) still appears in your terminal prompt. If it disappeared, activate the virtual environment again before running the file.

If your prompt shows (venv) but you still see the error, try running pip install numpy matplotlib scikit-learn again to make sure the packages installed correctly.

On macOS, if python does not work, try python3 neural_network.py instead.

Help me fix my ModuleNotFoundError

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is what your neural_network.py should look like at this point:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml

Your environment is ready and all three libraries are installed. Next up, you'll load the MNIST dataset and prepare 70,000 handwritten digit images for your neural network to learn from.

Load and Prepare the MNIST Data

Your environment is set up and your imports are confirmed working. Now it's time to give your neural network something to learn from.

In this step, you'll download the MNIST dataset of 70,000 handwritten digit images and prepare it for training. This means normalizing pixel values, splitting into training and test sets, and encoding labels so the network can understand them.

In this step, get ready to:
  • Download the MNIST dataset and split it into training and test sets.
  • Normalize pixel values and one-hot encode labels for the network.
  • Visualize sample images to confirm the data loaded correctly.
Write the data loading function

Your neural network needs data to learn from. The MNIST dataset contains 70,000 images of handwritten digits (0-9), each represented as a 28x28 pixel grid flattened into 784 numbers.

  • In neural_network.py, add the following function below your import statements:
# --- Data Loading and Preparation ---

def load_data():
    """Load MNIST dataset and prepare for training."""
    print("Loading MNIST dataset (this may take a moment on first run)...")
    mnist = fetch_openml('mnist_784', version=1, as_frame=False)
    X, y = mnist.data, mnist.target.astype(int)

    # Normalize pixel values to 0-1
    X = X / 255.0

    # Split into train and test
    X_train, X_test = X[:60000], X[60000:]
    y_train, y_test = y[:60000], y[60000:]

    return X_train, y_train, X_test, y_test

What does this code do?

This function handles three critical data preparation steps in one place:

  • fetch_openml('mnist_784', version=1, as_frame=False) downloads the full MNIST dataset from OpenML. The as_frame=False flag returns raw NumPy arrays instead of a pandas DataFrame.
  • X / 255.0 normalizes pixel values from a 0-255 range down to 0-1. Neural networks learn faster when input values are small.
  • The slicing splits the data into 60,000 training samples and 10,000 test samples. The test set stays untouched until evaluation so you get an honest accuracy measurement.

Why normalize to 0-1?

Raw pixel values range from 0 to 255. Large input values cause large weight updates during training, making the network unstable. Dividing by 255.0 squeezes everything into a 0-1 range, which keeps gradient updates small and smooth.

You won't see output yet. The next chunk adds the encoding function, and then you'll test everything together.

Write the one-hot encoding function

Your network's output layer has 10 neurons, one for each digit. To compare predictions against actual labels, you need labels in the same shape. One-hot encoding converts a label like "3" into a vector of 10 zeros with a 1 at index 3.

  • Add this function below load_data() in neural_network.py:
def one_hot_encode(y, num_classes=10):
    """Convert labels to one-hot encoded vectors."""
    one_hot = np.zeros((y.shape[0], num_classes))
    one_hot[np.arange(y.shape[0]), y] = 1
    return one_hot

What does this code do?

np.zeros((y.shape[0], num_classes)) creates a matrix of zeros with one row per sample and 10 columns (one per digit class).

The line one_hot[np.arange(y.shape[0]), y] = 1 uses NumPy fancy indexing to place a 1 at the correct column for each row. If a sample's label is 7, column 7 of that row becomes 1.

Now let's test both functions to make sure the data loads and encodes correctly.

Verify the data with sample images

Before building the network, you should confirm the data looks right. You'll load the dataset, print its shape, and display a few handwritten digit images.

  • Add this temporary test code at the bottom of neural_network.py, below the one_hot_encode function:
# --- Temporary test code (delete after verifying) ---

X_train, y_train, X_test, y_test = load_data()
y_train_encoded = one_hot_encode(y_train)

print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"Image size: 28x28 = {X_train.shape[1]} pixels")

# Visualize sample images
fig, axes = plt.subplots(2, 5, figsize=(12, 6))
for i, ax in enumerate(axes.flat):
    ax.imshow(X_train[i].reshape(28, 28), cmap='gray')
    ax.set_title(f'Label: {y_train[i]}')
    ax.axis('off')
plt.tight_layout()
plt.show()

What does this test code do?

  • It calls both functions you just wrote to load and encode the data.
  • The print statements confirm you have 60,000 training samples, 10,000 test samples, and 784 pixels per image.
  • The visualization grid shows 10 sample images reshaped from flat 784-length arrays back into 28x28 images so you can see actual handwritten digits.
  • Save neural_network.py.
  • Run the file by entering this command in your terminal:
python neural_network.py

The first run downloads the MNIST dataset, which may take 30-60 seconds depending on your internet connection. After that, you should see the training set size printed and a window pop up with a grid of handwritten digit images.

Not seeing the image window?

  • If you see the print output but no image window, your Matplotlib backend may not support display. Try adding plt.savefig('test.png') before plt.show() and open the saved image manually.
  • If you get a connection error, check your internet connection. The dataset only needs to download once and is cached locally after that.
  • If you see NameError, make sure the test code is placed below both function definitions, not inside them.

Help me troubleshoot this error

  • Now that you've confirmed the data is loading correctly, delete the temporary test code block at the bottom of your file (everything below the one_hot_encode function).

Why delete the test code?

You'll rebuild the main execution block properly in a later step. Keeping temporary test code around leads to duplicate runs and confusing output as you add more functionality.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file after removing the temporary test code.

ⓧ I'd like to double check the full code

Here's what neural_network.py should look like at this point:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml


# --- Data Loading and Preparation ---

def load_data():
    """Load MNIST dataset and prepare for training."""
    print("Loading MNIST dataset (this may take a moment on first run)...")
    mnist = fetch_openml('mnist_784', version=1, as_frame=False)
    X, y = mnist.data, mnist.target.astype(int)

    # Normalize pixel values to 0-1
    X = X / 255.0

    # Split into train and test
    X_train, X_test = X[:60000], X[60000:]
    y_train, y_test = y[:60000], y[60000:]

    return X_train, y_train, X_test, y_test


def one_hot_encode(y, num_classes=10):
    """Convert labels to one-hot encoded vectors."""
    one_hot = np.zeros((y.shape[0], num_classes))
    one_hot[np.arange(y.shape[0]), y] = 1
    return one_hot

Your data pipeline is ready. Next up, you'll build the neural network class that takes this prepared data and learns to recognize digits from it.

Build the Neural Network Class

Your data is loaded, normalized, and split into training and test sets. Now you need something that can actually learn from that data.

A neural network is a series of layers where data flows in, gets transformed by weights and activation functions, and comes out as a prediction. In this step, you will build the NeuralNetwork class that defines this structure and can make a forward pass through the network.

In this step, get ready to:
  • Define the network architecture with weights and biases.
  • Implement the sigmoid and softmax activation functions.
  • Write the forward pass that turns pixel inputs into digit predictions.
Define the network architecture

A neural network needs weights (the values it adjusts during learning) and biases (offsets that give the network more flexibility). Your network has three layers: 784 input neurons (one per pixel in a 28x28 image), 128 hidden neurons, and 10 output neurons (one per digit 0-9).

  • In neural_network.py, find the one_hot_encode function and add the following code below it:
# --- Neural Network ---

class NeuralNetwork:
    def __init__(self, input_size=784, hidden_size=128, output_size=10):
        """Initialize network with He initialization."""
        self.W1 = np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size)
        self.b1 = np.zeros((1, hidden_size))
        self.W2 = np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size)
        self.b2 = np.zeros((1, output_size))

What does this code do?

This creates the skeleton of your neural network. The __init__ method sets up four key components:

  • W1 is a matrix of 784x128 random values. Each value is a "weight" connecting an input pixel to a hidden neuron.
  • b1 is a row of 128 zeros. These biases give each hidden neuron an adjustable starting offset.
  • W2 and b2 do the same thing for the connection between the hidden layer and the 10 output neurons.

The np.sqrt(2.0 / fan_in) scaling is called He initialization. It keeps the initial weights small enough that signals don't explode or vanish as they travel through layers.

  • Save neural_network.py.
  • Run your file to confirm there are no syntax errors:
python neural_network.py

You should see no output and no errors. The class is defined but nothing calls it yet.

Seeing a syntax error?

  • Make sure the class NeuralNetwork: line is at the top level of the file (no indentation), not inside another function.
  • Check that self.W1, self.b1, self.W2, and self.b2 are all indented inside the __init__ method (two levels of indentation).
  • help me debug a syntax error in my NeuralNetwork class
Implement the activation functions

Activation functions control what signal passes through each neuron. Without them, your network would just be multiplying matrices together, which can only learn simple linear patterns.

  • Inside the NeuralNetwork class, add these three methods directly below the __init__ method:
    def sigmoid(self, z):
        """Sigmoid activation function."""
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

    def sigmoid_derivative(self, a):
        """Derivative of sigmoid given the sigmoid output."""
        return a * (1 - a)

    def softmax(self, z):
        """Softmax activation for output layer."""
        exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
        return exp_z / np.sum(exp_z, axis=1, keepdims=True)

What do these activation functions do?

  • sigmoid squashes any input value into a range between 0 and 1. This is what the hidden layer uses to decide how much signal to pass forward. The np.clip prevents overflow errors with very large or very small numbers.
  • sigmoid_derivative calculates how sensitive the sigmoid output is to changes in its input. You will need this later for backpropagation.
  • softmax converts raw output scores into probabilities that sum to 1. The digit with the highest probability is the network's prediction.
  • Save neural_network.py.

You won't see a change yet. The next chunk wires these activation functions into a full forward pass.

Write the forward pass

The forward pass is how data flows through the network from input to prediction. Each pixel value gets multiplied by weights, added to biases, then squeezed through an activation function.

  • Still inside the NeuralNetwork class, add this method below softmax:
    def forward(self, X):
        """Forward pass through the network."""
        self.z1 = X @ self.W1 + self.b1
        self.a1 = self.sigmoid(self.z1)
        self.z2 = self.a1 @ self.W2 + self.b2
        self.a2 = self.softmax(self.z2)
        return self.a2

What does the forward pass do?

The @ operator is matrix multiplication. Here is what happens at each line:

  • z1 multiplies the input pixels by the first set of weights and adds biases. This is the raw signal going into the hidden layer.
  • a1 applies sigmoid to squash those raw values between 0 and 1. These are the hidden layer's activations.
  • z2 takes the hidden activations, multiplies by the second set of weights, and adds biases.
  • a2 applies softmax to produce 10 probabilities (one per digit). The highest probability is the network's guess.

All intermediate values (z1, a1, z2, a2) are stored on self because backpropagation needs them to calculate how to adjust the weights.

  • Save neural_network.py.
  • Test the forward pass by running this command in your terminal:
python -c "from neural_network import NeuralNetwork, load_data; X, _, _, _ = load_data(); nn = NeuralNetwork(); out = nn.forward(X[0:1]); print('Output shape:', out.shape); print('Probabilities:', out.round(4)); print('Sum:', out.sum().round(4))"

You should see an output shape of (1, 10) and a sum of 1.0. The 10 probability values represent the network's initial (untrained) guess for each digit.

Seeing an import error or wrong shape?

  • Make sure your terminal is in the neural-network folder where neural_network.py lives.
  • If you see ModuleNotFoundError, confirm your virtual environment is still activated.
  • Check that forward is indented inside the NeuralNetwork class, not at the file's top level.
  • help me fix my forward pass test

✔️ Awesome, I've got everything!

Great. Double check you have saved neural_network.py with all the changes from this step.

ⓧ I'd like to double check the full code

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml


# --- Data Loading and Preparation ---

def load_data():
    """Load MNIST dataset and prepare for training."""
    print("Loading MNIST dataset (this may take a moment on first run)...")
    mnist = fetch_openml('mnist_784', version=1, as_frame=False)
    X, y = mnist.data, mnist.target.astype(int)

    # Normalize pixel values to 0-1
    X = X / 255.0

    # Split into train and test
    X_train, X_test = X[:60000], X[60000:]
    y_train, y_test = y[:60000], y[60000:]

    return X_train, y_train, X_test, y_test


def one_hot_encode(y, num_classes=10):
    """Convert labels to one-hot encoded vectors."""
    one_hot = np.zeros((y.shape[0], num_classes))
    one_hot[np.arange(y.shape[0]), y] = 1
    return one_hot


# --- Neural Network ---

class NeuralNetwork:
    def __init__(self, input_size=784, hidden_size=128, output_size=10):
        """Initialize network with He initialization."""
        self.W1 = np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size)
        self.b1 = np.zeros((1, hidden_size))
        self.W2 = np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size)
        self.b2 = np.zeros((1, output_size))

    def sigmoid(self, z):
        """Sigmoid activation function."""
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

    def sigmoid_derivative(self, a):
        """Derivative of sigmoid given the sigmoid output."""
        return a * (1 - a)

    def softmax(self, z):
        """Softmax activation for output layer."""
        exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
        return exp_z / np.sum(exp_z, axis=1, keepdims=True)

    def forward(self, X):
        """Forward pass through the network."""
        self.z1 = X @ self.W1 + self.b1
        self.a1 = self.sigmoid(self.z1)
        self.z2 = self.a1 @ self.W2 + self.b2
        self.a2 = self.softmax(self.z2)
        return self.a2

Your neural network can now take a 784-pixel image and produce a prediction. The problem is those predictions are random right now. Next up, you will teach the network to learn from its mistakes using backpropagation.

Implement Backpropagation and Training

Your network can make predictions through the forward pass, but right now those predictions are random. It has no way to learn from its mistakes.

In this step, you will build the learning engine. You will add a loss function to measure how wrong the network's predictions are, a backpropagation algorithm to figure out which weights caused the errors, and a training loop that repeatedly adjusts those weights using gradient descent.

In this step, get ready to:
  • Implement a loss function to measure prediction errors.
  • Write the backpropagation algorithm to calculate weight adjustments.
  • Build a training loop that improves the network over many passes.
Implement the cross-entropy loss function

Before the network can learn, it needs a way to measure how wrong its predictions are. A loss function takes the network's predicted probabilities and the actual correct answer, then returns a single number representing the total error.

Cross-entropy loss is the standard choice for classification problems. It penalizes confident wrong answers more heavily than uncertain ones.

  • In neural_network.py, find the forward method inside the NeuralNetwork class.
  • Add the following method directly below forward:
    def cross_entropy_loss(self, y_pred, y_true):
        """Compute cross-entropy loss."""
        m = y_true.shape[0]
        loss = -np.sum(y_true * np.log(y_pred + 1e-8)) / m
        return loss

What does this code do?

This method compares the network's predicted probabilities against the actual labels.

  • m = y_true.shape[0] gets the number of samples in the batch so you can average the loss.
  • np.log(y_pred + 1e-8) takes the natural log of predictions. The tiny 1e-8 prevents a math error from log(0).
  • The negative sign makes the loss positive. A perfect prediction produces a loss near 0, while a terrible prediction produces a large number.

You won't see a change yet. The next method uses this loss value to figure out how to adjust the weights.

Write the backward method for backpropagation

Backpropagation is the algorithm that figures out which weights caused the error and how much to adjust each one. It works backwards from the output layer to the input layer, using the chain rule from calculus to distribute blame for the error across all connections.

The sigmoid_derivative method you already wrote in the previous step plays a key role here. It tells the algorithm how sensitive each hidden neuron's output was to its input.

  • Add the backward method directly below cross_entropy_loss in your NeuralNetwork class:
    def backward(self, X, y_true, learning_rate=0.1):
        """Backpropagation and weight update."""
        m = X.shape[0]

        # Output layer error
        dz2 = self.a2 - y_true

        # Gradients for W2 and b2
        dW2 = (self.a1.T @ dz2) / m
        db2 = np.sum(dz2, axis=0, keepdims=True) / m

        # Hidden layer error
        da1 = dz2 @ self.W2.T
        dz1 = da1 * self.sigmoid_derivative(self.a1)

        # Gradients for W1 and b1
        dW1 = (X.T @ dz1) / m
        db1 = np.sum(dz1, axis=0, keepdims=True) / m

        # Update weights
        self.W2 -= learning_rate * dW2
        self.b2 -= learning_rate * db2
        self.W1 -= learning_rate * dW1
        self.b1 -= learning_rate * db1

What does this code do?

This method propagates the error backwards through the network in three stages.

  • dz2 = self.a2 - y_true computes the output layer error. This is the difference between what the network predicted and what the answer actually was.
  • dW2 and dW1 are the gradients. They tell you the direction and magnitude each weight needs to change to reduce the loss.
  • da1 = dz2 @ self.W2.T sends the error backwards through the second weight matrix to reach the hidden layer. This is the "chain rule" in action.
  • The final section subtracts learning_rate * gradient from each weight. A larger learning rate means bigger steps toward the solution.

Why subtract the gradient?

The gradient points in the direction of increasing loss. Since you want to decrease loss, you move in the opposite direction by subtracting. This is the core idea of gradient descent.

You won't see a change yet. The training loop in the next chunk calls both forward and backward together.

Build the training loop

A single forward-backward pass adjusts the weights once. To actually learn, the network needs to repeat this process thousands of times across the entire dataset. The training loop handles this repetition.

Instead of feeding all 60,000 images at once, you will split the data into mini-batches of 64 images. This makes training faster and more stable.

  • Add the train method directly below backward in your NeuralNetwork class:
    def train(self, X_train, y_train_encoded, epochs=20, learning_rate=0.1, batch_size=64):
        """Train the network using mini-batch gradient descent."""
        losses = []
        m = X_train.shape[0]

        for epoch in range(epochs):
            # Shuffle training data
            indices = np.random.permutation(m)
            X_shuffled = X_train[indices]
            y_shuffled = y_train_encoded[indices]

            epoch_loss = 0
            num_batches = 0

            for i in range(0, m, batch_size):
                X_batch = X_shuffled[i:i+batch_size]
                y_batch = y_shuffled[i:i+batch_size]

                # Forward pass
                output = self.forward(X_batch)

                # Compute loss
                epoch_loss += self.cross_entropy_loss(output, y_batch)
                num_batches += 1

                # Backward pass and update
                self.backward(X_batch, y_batch, learning_rate)

            avg_loss = epoch_loss / num_batches
            losses.append(avg_loss)
            print(f"Epoch {epoch + 1}/{epochs} - Loss: {avg_loss:.4f}")

        return losses

What does this code do?

This method orchestrates the full training process.

  • np.random.permutation(m) shuffles the training data at the start of each epoch. This prevents the network from memorizing the order of images.
  • The inner loop slices the shuffled data into mini-batches of 64 samples each. For each batch, it runs forward, computes loss, and runs backward.
  • avg_loss tracks the average loss per epoch. A decreasing loss means the network is learning.
  • The method returns the full list of losses so you can plot them later.

What is an epoch?

One epoch is a complete pass through the entire training dataset. If you train for 20 epochs, the network sees every image 20 times. Each time it sees the data, it adjusts weights a little further toward the correct answers.

  • Save neural_network.py.
  • Test that training works by running this command in your terminal:
python -c "
from neural_network import *
X_train, y_train, _, _ = load_data()
nn = NeuralNetwork()
losses = nn.train(X_train[:1000], one_hot_encode(y_train[:1000]), epochs=3, learning_rate=0.1, batch_size=64)
"

You should see the MNIST loading message followed by three lines showing the loss decreasing from one epoch to the next.

Seeing an error or the loss isn't decreasing?

  • If you see IndentationError, make sure all three new methods are indented inside the NeuralNetwork class (one level of indentation from the class body).
  • If you see AttributeError: 'NeuralNetwork' object has no attribute 'a2', the backward method depends on values stored by forward. Make sure your train method calls self.forward(X_batch) before self.backward(X_batch, y_batch, learning_rate).
  • If the loss stays flat or increases, double-check the subtraction in the weight update section. The lines should use -= (subtract), not += (add).

Help me debug my neural network training loop

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before continuing.

ⓧ I'd like to double check the full code

Here is the complete neural_network.py file so far:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml


# --- Data Loading and Preparation ---

def load_data():
    """Load MNIST dataset and prepare for training."""
    print("Loading MNIST dataset (this may take a moment on first run)...")
    mnist = fetch_openml('mnist_784', version=1, as_frame=False)
    X, y = mnist.data, mnist.target.astype(int)

    # Normalize pixel values to 0-1
    X = X / 255.0

    # Split into train and test
    X_train, X_test = X[:60000], X[60000:]
    y_train, y_test = y[:60000], y[60000:]

    return X_train, y_train, X_test, y_test


def one_hot_encode(y, num_classes=10):
    """Convert labels to one-hot encoded vectors."""
    one_hot = np.zeros((y.shape[0], num_classes))
    one_hot[np.arange(y.shape[0]), y] = 1
    return one_hot


# --- Neural Network ---

class NeuralNetwork:
    def __init__(self, input_size=784, hidden_size=128, output_size=10):
        """Initialize network with He initialization."""
        self.W1 = np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size)
        self.b1 = np.zeros((1, hidden_size))
        self.W2 = np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size)
        self.b2 = np.zeros((1, output_size))

    def sigmoid(self, z):
        """Sigmoid activation function."""
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

    def sigmoid_derivative(self, a):
        """Derivative of sigmoid given the sigmoid output."""
        return a * (1 - a)

    def softmax(self, z):
        """Softmax activation for output layer."""
        exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
        return exp_z / np.sum(exp_z, axis=1, keepdims=True)

    def forward(self, X):
        """Forward pass through the network."""
        self.z1 = X @ self.W1 + self.b1
        self.a1 = self.sigmoid(self.z1)
        self.z2 = self.a1 @ self.W2 + self.b2
        self.a2 = self.softmax(self.z2)
        return self.a2

    def cross_entropy_loss(self, y_pred, y_true):
        """Compute cross-entropy loss."""
        m = y_true.shape[0]
        loss = -np.sum(y_true * np.log(y_pred + 1e-8)) / m
        return loss

    def backward(self, X, y_true, learning_rate=0.1):
        """Backpropagation and weight update."""
        m = X.shape[0]

        # Output layer error
        dz2 = self.a2 - y_true

        # Gradients for W2 and b2
        dW2 = (self.a1.T @ dz2) / m
        db2 = np.sum(dz2, axis=0, keepdims=True) / m

        # Hidden layer error
        da1 = dz2 @ self.W2.T
        dz1 = da1 * self.sigmoid_derivative(self.a1)

        # Gradients for W1 and b1
        dW1 = (X.T @ dz1) / m
        db1 = np.sum(dz1, axis=0, keepdims=True) / m

        # Update weights
        self.W2 -= learning_rate * dW2
        self.b2 -= learning_rate * db2
        self.W1 -= learning_rate * dW1
        self.b1 -= learning_rate * db1

    def train(self, X_train, y_train_encoded, epochs=20, learning_rate=0.1, batch_size=64):
        """Train the network using mini-batch gradient descent."""
        losses = []
        m = X_train.shape[0]

        for epoch in range(epochs):
            # Shuffle training data
            indices = np.random.permutation(m)
            X_shuffled = X_train[indices]
            y_shuffled = y_train_encoded[indices]

            epoch_loss = 0
            num_batches = 0

            for i in range(0, m, batch_size):
                X_batch = X_shuffled[i:i+batch_size]
                y_batch = y_shuffled[i:i+batch_size]

                # Forward pass
                output = self.forward(X_batch)

                # Compute loss
                epoch_loss += self.cross_entropy_loss(output, y_batch)
                num_batches += 1

                # Backward pass and update
                self.backward(X_batch, y_batch, learning_rate)

            avg_loss = epoch_loss / num_batches
            losses.append(avg_loss)
            print(f"Epoch {epoch + 1}/{epochs} - Loss: {avg_loss:.4f}")

        return losses

Your network can now learn from its mistakes. Next up, you will train it on the full 60,000 images and measure how accurately it can recognize digits it has never seen before.

Train the Network and Evaluate Results

Your neural network can now learn from its mistakes through backpropagation. The entire training pipeline is in place: forward pass, loss calculation, gradient computation, and weight updates.

In this step, you will train the network on 60,000 handwritten digit images and measure how well it performs on 10,000 images it has never seen. You will also create visualizations that show the network's learning progress and its predictions on real digit images.

In this step, get ready to:
  • Add prediction and evaluation methods to your network.
  • Create visualization functions for training loss and sample predictions.
  • Train the network for 20 epochs and confirm it achieves above 90% accuracy.
Add predict and evaluate methods

The train method teaches the network, but you still need a way to ask it for predictions and measure its accuracy on unseen data.

  • In neural_network.py, find the end of the train method (after return losses).
  • Add the following two methods to the NeuralNetwork class, inside the class body at the same indentation level as train:
    def predict(self, X):
        """Predict class labels for input data."""
        output = self.forward(X)
        return np.argmax(output, axis=1)

    def evaluate(self, X_test, y_test):
        """Evaluate accuracy on test data."""
        predictions = self.predict(X_test)
        accuracy = np.mean(predictions == y_test)
        return accuracy

What does this code do?

  • predict runs a forward pass, then uses np.argmax to pick the digit (0-9) with the highest probability.
  • evaluate calls predict on the entire test set, then compares each prediction to the true label to calculate the percentage of correct answers.

You won't see a change yet. The next chunk uses these methods when training completes.

Create the visualization functions

Numbers alone can tell you the network is learning, but a chart makes the learning curve unmistakable. You will write two functions: one that plots loss over time, and one that displays sample predictions alongside the actual digit images.

  • Below the NeuralNetwork class (outside and below it, not indented inside the class), add a comment header and the training loss plot function:
# --- Visualization ---

def plot_training_loss(losses):
    """Plot the training loss over epochs."""
    plt.figure(figsize=(10, 6))
    plt.plot(range(1, len(losses) + 1), losses, 'b-', linewidth=2)
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.title('Training Loss Over Time')
    plt.grid(True, alpha=0.3)
    plt.savefig('training_loss.png', dpi=100, bbox_inches='tight')
    plt.show()
    print("Training loss plot saved as 'training_loss.png'")

What does this function do?

This function takes the list of loss values (one per epoch) and draws a line chart. A healthy training curve slopes downward, meaning the network makes fewer mistakes over time. The chart is saved as training_loss.png so you have a permanent record.

  • Directly below plot_training_loss, add the predictions visualization function:
def plot_predictions(nn, X_test, y_test, num_samples=10):
    """Show sample predictions with images."""
    indices = np.random.choice(len(X_test), num_samples, replace=False)
    predictions = nn.predict(X_test[indices])

    fig, axes = plt.subplots(2, 5, figsize=(12, 6))
    for i, ax in enumerate(axes.flat):
        ax.imshow(X_test[indices[i]].reshape(28, 28), cmap='gray')
        actual = y_test[indices[i]]
        predicted = predictions[i]
        color = 'green' if actual == predicted else 'red'
        ax.set_title(f'Pred: {predicted} | Actual: {actual}', color=color)
        ax.axis('off')

    plt.tight_layout()
    plt.savefig('predictions.png', dpi=100, bbox_inches='tight')
    plt.show()
    print("Predictions plot saved as 'predictions.png'")

What does this function do?

This picks 10 random test images, runs them through your trained network, and displays a 2x5 grid. Each image shows the network's prediction and the actual label. Correct predictions appear in green, mistakes in red.

You won't see output yet. The main block you add next calls both visualization functions after training completes.

Write the main block and run training

The main block ties everything together: load the data, create the network, train for 20 epochs, evaluate on the test set, and generate visualizations.

  • At the very bottom of neural_network.py, below the plot_predictions function, add the main execution block:
# --- Main ---

if __name__ == '__main__':
    # Load data
    X_train, y_train, X_test, y_test = load_data()
    y_train_encoded = one_hot_encode(y_train)

    print(f"Training set: {X_train.shape[0]} samples")
    print(f"Test set: {X_test.shape[0]} samples")
    print(f"Image size: 28x28 = {X_train.shape[1]} pixels")
    print()

    # Create and train network
    nn = NeuralNetwork(input_size=784, hidden_size=128, output_size=10)

    print("Training neural network...")
    print("-" * 40)
    losses = nn.train(X_train, y_train_encoded, epochs=20, learning_rate=0.1, batch_size=64)

    # Evaluate
    print("-" * 40)
    accuracy = nn.evaluate(X_test, y_test)
    print(f"Test Accuracy: {accuracy * 100:.2f}%")

    # Visualize
    plot_training_loss(losses)
    plot_predictions(nn, X_test, y_test)

What does this code do?

  • load_data() downloads MNIST and splits it into 60,000 training images and 10,000 test images.
  • one_hot_encode(y_train) converts each label (like 3) into a vector of 10 values with a 1 at position 3.
  • The network trains for 20 epochs with a learning rate of 0.1 and batch size of 64. Each epoch processes all 60,000 images in shuffled mini-batches.
  • After training, evaluate tests the network on 10,000 images it never trained on. This measures real-world performance.
  • Save neural_network.py.
  • Run your neural network by executing this command:
python neural_network.py

What should I see?

The first run downloads MNIST, which may take a moment. Then you will see 20 lines of output like Epoch 1/20 - Loss: 0.XXXX with the loss decreasing each epoch. Training takes 1-2 minutes on most machines.

After training finishes, you will see a test accuracy line (target: ~95%). Two chart windows will pop up showing your loss curve and sample predictions. Both charts are also saved as training_loss.png and predictions.png in your project folder.

Training seems stuck or accuracy is low?

  • If loss is not decreasing, double-check that the backward method subtracts the gradient (uses -= not +=) when updating weights.
  • If you get a shape mismatch error, confirm that one_hot_encode is called on y_train and that you pass y_train_encoded (not y_train) to the train method.
  • If accuracy is below 80%, check that pixel values are normalized (divided by 255.0) in load_data.

Help me debug my neural network training

✔️ Awesome, I've got everything!

Great. Double check you have saved your file and that training completed successfully with accuracy above 90%.

ⓧ I'd like to double check the full code

Here is the complete neural_network.py file at this point:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml


# --- Data Loading and Preparation ---

def load_data():
    """Load MNIST dataset and prepare for training."""
    print("Loading MNIST dataset (this may take a moment on first run)...")
    mnist = fetch_openml('mnist_784', version=1, as_frame=False)
    X, y = mnist.data, mnist.target.astype(int)

    # Normalize pixel values to 0-1
    X = X / 255.0

    # Split into train and test
    X_train, X_test = X[:60000], X[60000:]
    y_train, y_test = y[:60000], y[60000:]

    return X_train, y_train, X_test, y_test


def one_hot_encode(y, num_classes=10):
    """Convert labels to one-hot encoded vectors."""
    one_hot = np.zeros((y.shape[0], num_classes))
    one_hot[np.arange(y.shape[0]), y] = 1
    return one_hot


# --- Neural Network ---

class NeuralNetwork:
    def __init__(self, input_size=784, hidden_size=128, output_size=10):
        """Initialize network with He initialization."""
        self.W1 = np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size)
        self.b1 = np.zeros((1, hidden_size))
        self.W2 = np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size)
        self.b2 = np.zeros((1, output_size))

    def sigmoid(self, z):
        """Sigmoid activation function."""
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

    def sigmoid_derivative(self, a):
        """Derivative of sigmoid given the sigmoid output."""
        return a * (1 - a)

    def softmax(self, z):
        """Softmax activation for output layer."""
        exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
        return exp_z / np.sum(exp_z, axis=1, keepdims=True)

    def forward(self, X):
        """Forward pass through the network."""
        self.z1 = X @ self.W1 + self.b1
        self.a1 = self.sigmoid(self.z1)
        self.z2 = self.a1 @ self.W2 + self.b2
        self.a2 = self.softmax(self.z2)
        return self.a2

    def cross_entropy_loss(self, y_pred, y_true):
        """Compute cross-entropy loss."""
        m = y_true.shape[0]
        loss = -np.sum(y_true * np.log(y_pred + 1e-8)) / m
        return loss

    def backward(self, X, y_true, learning_rate=0.1):
        """Backpropagation and weight update."""
        m = X.shape[0]

        # Output layer error
        dz2 = self.a2 - y_true

        # Gradients for W2 and b2
        dW2 = (self.a1.T @ dz2) / m
        db2 = np.sum(dz2, axis=0, keepdims=True) / m

        # Hidden layer error
        da1 = dz2 @ self.W2.T
        dz1 = da1 * self.sigmoid_derivative(self.a1)

        # Gradients for W1 and b1
        dW1 = (X.T @ dz1) / m
        db1 = np.sum(dz1, axis=0, keepdims=True) / m

        # Update weights
        self.W2 -= learning_rate * dW2
        self.b2 -= learning_rate * db2
        self.W1 -= learning_rate * dW1
        self.b1 -= learning_rate * db1

    def train(self, X_train, y_train_encoded, epochs=20, learning_rate=0.1, batch_size=64):
        """Train the network using mini-batch gradient descent."""
        losses = []
        m = X_train.shape[0]

        for epoch in range(epochs):
            # Shuffle training data
            indices = np.random.permutation(m)
            X_shuffled = X_train[indices]
            y_shuffled = y_train_encoded[indices]

            epoch_loss = 0
            num_batches = 0

            for i in range(0, m, batch_size):
                X_batch = X_shuffled[i:i+batch_size]
                y_batch = y_shuffled[i:i+batch_size]

                # Forward pass
                output = self.forward(X_batch)

                # Compute loss
                epoch_loss += self.cross_entropy_loss(output, y_batch)
                num_batches += 1

                # Backward pass and update
                self.backward(X_batch, y_batch, learning_rate)

            avg_loss = epoch_loss / num_batches
            losses.append(avg_loss)
            print(f"Epoch {epoch + 1}/{epochs} - Loss: {avg_loss:.4f}")

        return losses

    def predict(self, X):
        """Predict class labels for input data."""
        output = self.forward(X)
        return np.argmax(output, axis=1)

    def evaluate(self, X_test, y_test):
        """Evaluate accuracy on test data."""
        predictions = self.predict(X_test)
        accuracy = np.mean(predictions == y_test)
        return accuracy


# --- Visualization ---

def plot_training_loss(losses):
    """Plot the training loss over epochs."""
    plt.figure(figsize=(10, 6))
    plt.plot(range(1, len(losses) + 1), losses, 'b-', linewidth=2)
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.title('Training Loss Over Time')
    plt.grid(True, alpha=0.3)
    plt.savefig('training_loss.png', dpi=100, bbox_inches='tight')
    plt.show()
    print("Training loss plot saved as 'training_loss.png'")


def plot_predictions(nn, X_test, y_test, num_samples=10):
    """Show sample predictions with images."""
    indices = np.random.choice(len(X_test), num_samples, replace=False)
    predictions = nn.predict(X_test[indices])

    fig, axes = plt.subplots(2, 5, figsize=(12, 6))
    for i, ax in enumerate(axes.flat):
        ax.imshow(X_test[indices[i]].reshape(28, 28), cmap='gray')
        actual = y_test[indices[i]]
        predicted = predictions[i]
        color = 'green' if actual == predicted else 'red'
        ax.set_title(f'Pred: {predicted} | Actual: {actual}', color=color)
        ax.axis('off')

    plt.tight_layout()
    plt.savefig('predictions.png', dpi=100, bbox_inches='tight')
    plt.show()
    print("Predictions plot saved as 'predictions.png'")


# --- Main ---

if __name__ == '__main__':
    # Load data
    X_train, y_train, X_test, y_test = load_data()
    y_train_encoded = one_hot_encode(y_train)

    print(f"Training set: {X_train.shape[0]} samples")
    print(f"Test set: {X_test.shape[0]} samples")
    print(f"Image size: 28x28 = {X_train.shape[1]} pixels")
    print()

    # Create and train network
    nn = NeuralNetwork(input_size=784, hidden_size=128, output_size=10)

    print("Training neural network...")
    print("-" * 40)
    losses = nn.train(X_train, y_train_encoded, epochs=20, learning_rate=0.1, batch_size=64)

    # Evaluate
    print("-" * 40)
    accuracy = nn.evaluate(X_test, y_test)
    print(f"Test Accuracy: {accuracy * 100:.2f}%")

    # Visualize
    plot_training_loss(losses)
    plot_predictions(nn, X_test, y_test)

Secret mission

Add a Second Hidden Layer

Your single-layer network hits ~95% accuracy. Can a deeper network do better? Add a second hidden layer, propagate gradients through three weight matrices, and compare the two architectures head-to-head.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your project files, pause your environment, or delete everything.

Resources you used:

  • neural-network project folder on your Desktop (contains neural_network.py, virtual environment, and generated images).
  • Cached MNIST dataset in ~/scikit_learn_data (downloaded by scikit-learn on first run).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your neural network or want to reference the code later.

  • Your neural-network folder stays on your Desktop with all code and generated plots.
  • The cached MNIST data in ~/scikit_learn_data stays available for faster loading next time.
  • The virtual environment stays inactive until you activate it again.

Pause - I'll come back to this later

Deactivate the virtual environment to free up your terminal session. Your files stay intact for when you return.

  • Deactivate your virtual environment by running:
deactivate

When you come back, navigate to your project folder and reactivate the environment.

Delete - I don't want to use this again

Remove all project files, the virtual environment, and the cached dataset to reclaim disk space.

  • Deactivate the virtual environment first (if it is still active):
deactivate
  • Delete the project folder by running:

macOS

cd ~/Desktop
rm -rf neural-network

Windows

cd ~\Desktop
rmdir /s /q neural-network
  • Delete the cached MNIST dataset by running:

macOS

rm -rf ~/scikit_learn_data

Windows

rmdir /s /q %USERPROFILE%\scikit_learn_data
  • Confirm both are gone by checking your Desktop and home directory. The neural-network folder and scikit_learn_data folder should no longer appear.

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully functional neural network from scratch using nothing but NumPy, and it can recognize handwritten digits with ~95% accuracy.

You've learned how to:

  • Implement forward propagation and backpropagation from scratch, turning raw pixel data into digit predictions and teaching the network to learn from its mistakes.
  • Train a neural network on 60,000 images using mini-batch gradient descent and evaluate it against 10,000 unseen test images to confirm ~95% accuracy.
  • Visualize the training loss curve and sample predictions to see exactly how your network improves over time.
  • Secret Mission: Added a second hidden layer to create a deeper network and compared its performance against the original architecture.

Ready to quiz yourself?