Build a Neural Network From Scratch

Build a digit classifier neural network from scratch using only NumPy.

Introduction

30 Second Summary

Every time you unlock your phone with face recognition or get a product recommendation online, a neural network is making predictions behind the scenes. These networks power modern AI, but how do they actually learn?

In this project, you will build a neural network entirely from scratch using Python and NumPy. Your network will learn to recognize handwritten digits (0-9) with over 95% accuracy, and you will implement every piece of the math that makes it work.

What You'll Build

A 3-layer neural network that classifies 8x8 pixel handwritten digit images through forward propagation, backpropagation, and gradient descent.

By the end of this project, you'll have:

  • A complete neural network implemented from scratch in NumPy that classifies handwritten digits at 95%+ accuracy.
  • A working backpropagation algorithm that computes gradients using the chain rule, implemented line by line.
  • Training visualizations (a loss curve and prediction grid) proving your network learned from data.
  • Secret Mission: Upgrade to mini-batch gradient descent with He initialization for faster, more stable training.

Are there any prerequisites?

You should be comfortable writing Python functions and working with loops. Basic algebra helps too. You do not need to know linear algebra or calculus in advance. This project teaches the relevant math (matrix multiplication, chain rule) as you build.

Set Up Your Environment

Before you can build a neural network, you need the right tools in place. NumPy handles the matrix math that powers every layer of your network.

In this step, you will confirm your Python version, create an isolated project environment, and install the three libraries your neural network depends on.

In this step, get ready to:
  • Confirm Python 3.12+ is installed and create a project folder.
  • Install NumPy, matplotlib, and scikit-learn.
  • Create your main project file and verify imports.
Verify your Python version and create a project folder

This project requires Python 3.12+ because NumPy 2.5.0 only supports Python 3.12 and above.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Open the integrated terminal by pressing Ctrl+` (the backtick key above Tab on your keyboard).

Check your Python version by running the command for your operating system:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.12 or higher

You are all set. Move on to the next instruction below.

ⓧ I see an older version

Your system has an older Python that does not meet the 3.12+ requirement.

On macOS, the default python3 command often points to an older version bundled with the OS. You need to install a newer version alongside it.

  • Visit python.org/downloads and download the latest Python 3.12+ installer for your operating system.
  • Run the installer and follow the on-screen prompts.
  • On macOS, after installation, use python3.12 --version (the versioned binary) to confirm. If you continue using the plain python3 command, it may still resolve to the older version.
  • On Windows, check the box Add python.exe to PATH during installation, then reopen your terminal and run python --version again.

ⓧ Command not found

Python is not installed on your machine yet.

  • Visit python.org/downloads and download the latest Python 3.12+ installer for your operating system.
  • On Windows, check the box Add python.exe to PATH during installation.
  • On macOS, run the .pkg installer. After installation, confirm with python3 --version in a fresh terminal.
  • Close and reopen the VS Code terminal, then run the version check command again.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir nn-from-scratch
cd nn-from-scratch
  • Create a virtual environment by running the command for your operating system:

macOS

python3 -m venv venv

Windows

python -m venv venv

What is a virtual environment?

A virtual environment is an isolated Python installation specific to your project. Packages you install inside it stay separate from your system Python, preventing version conflicts between different projects.

  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the environment is active.

Don't see (venv) in your prompt?

On Windows, if you get a permissions error, you may need to run this command first:

Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned

Then try the activation command again. On macOS, make sure you used source before venv/bin/activate (not just the path alone).

Help me activate my virtual environment.

Install the project dependencies

Your neural network needs three libraries: NumPy for matrix math, matplotlib for plotting training progress, and scikit-learn for the digit image dataset.

  • Install all three packages by running this command:
pip install numpy==2.5.0 matplotlib==3.11.0 scikit-learn==1.9.0

What does this command do?

The == syntax pins each package to an exact version. This ensures your project behaves identically to this tutorial, regardless of when you run it.

  • numpy==2.5.0 provides fast array and matrix operations.
  • matplotlib==3.11.0 generates charts for your training loss curve and prediction grid.
  • scikit-learn==1.9.0 includes the handwritten digits dataset you will train on.
  • Verify the packages installed correctly by running:
python -c "import numpy; import matplotlib; import sklearn; print('All packages installed')"

You should see All packages installed printed in your terminal with no errors.

Seeing an import error?

Make sure (venv) appears in your terminal prompt. If it does not, you are installing packages into your system Python instead of the virtual environment. Re-run the activation command from the previous substep.

Help me fix my package installation.

Create your project file and verify imports

All of your neural network code will live in a single file. Creating it now and verifying the imports confirms your environment is fully working.

  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the nn-from-scratch folder, then click Open.
  • In the Explorer sidebar on the left, hover over the folder name and click the new file icon.
  • Name the file neural_network.py and press Enter.
  • Add the following imports and verification print to neural_network.py:
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
print("Setup complete!")

What does this code do?

  • numpy as np gives you fast array operations for matrix multiplication in your neural network layers.
  • load_digits loads 1,797 images of handwritten digits (0-9) that your network will learn to classify.
  • train_test_split divides your data into training and test sets so you can measure real accuracy.
  • matplotlib.pyplot as plt generates charts to visualize how your network learns over time.
  • Save neural_network.py by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Run the file in your terminal:
python neural_network.py

You should see Setup complete! printed with no errors.

Seeing a ModuleNotFoundError?

This means your terminal is not using the virtual environment where the packages are installed. Check that (venv) appears in your prompt. If you opened a new terminal in VS Code, you need to activate the environment again.

Also confirm you are running python (not python3) inside the activated venv. The virtual environment always maps python to the correct version.

Help me fix my ModuleNotFoundError.

✔️ Awesome, I've got everything!

Great. Double check you have saved neural_network.py and that it printed Setup complete! without errors.

ⓧ I'd like to double check the full code

Here is the complete neural_network.py file at this point:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
print("Setup complete!")

Your environment is ready. Next up, you will load handwritten digit images and prepare them for your neural network to learn from.

Load and Preprocess the Digits Dataset

Your neural network needs data to learn from. Before it can recognize digits, you need to feed it labeled examples of handwritten digits so it can discover the patterns that distinguish a 3 from an 8.

In this step, you will load a dataset of 1,797 handwritten digit images from scikit-learn, preprocess the pixel data into a format a neural network can consume, and split it into training and test sets.

In this step, get ready to:
  • Load and explore the scikit-learn digits dataset (1,797 images of 8x8 pixels).
  • Normalize pixel values and one-hot encode labels for the neural network.
  • Split the data into training and test sets and verify the shapes.
Understand the digits dataset

The load_digits() function from scikit-learn provides a classic machine learning dataset: images of handwritten digits (0 through 9).

What is the digits dataset?

Each image is 8x8 pixels, where each pixel is an integer from 0 to 16 representing intensity. The original 32x32 bitmaps were divided into 4x4 blocks, and each block's pixel values were summed to produce these 8x8 representations.

The dataset contains 1,797 samples across 10 classes (digits 0-9). Each sample is stored as a flat array of 64 values (the 8x8 grid flattened into a single row). This flat format is exactly what a neural network expects as input.

Your neural network cannot work with raw pixel integers or plain digit labels directly. It needs two transformations:

  • Pixel values normalized to the range [0, 1] so that large numbers do not dominate the math.
  • Labels converted from a single number (like 3) into a vector of probabilities (like [0,0,0,1,0,0,0,0,0,0]). This is called one-hot encoding.
Write the data preprocessing function
  • In neural_network.py, find the line that says:
print("Setup complete!")
  • Delete that line. You no longer need it since you will replace it with your data loading function.
  • Below the imports, add the load_and_preprocess_data() function by pasting the following code:
# --- Data Loading and Preprocessing ---
def load_and_preprocess_data():
    # Load the 8x8 digit images from scikit-learn
    digits = load_digits()
    # Normalize pixel values from 0-16 to 0-1
    X = digits.data / 16.0

    # One-hot encode the labels (e.g., digit 3 becomes [0,0,0,1,0,0,0,0,0,0])
    y = digits.target
    num_classes = 10
    y_onehot = np.zeros((y.shape[0], num_classes))
    y_onehot[np.arange(y.shape[0]), y] = 1

    # Split into 80% training and 20% test
    X_train, X_test, y_train, y_test = train_test_split(
        X, y_onehot, test_size=0.2, random_state=42
    )
    return X_train, X_test, y_train, y_test

What does this code do?

  • load_digits() loads 1,797 images as a flat array of 64 pixel values each, plus their labels (0-9).
  • digits.data / 16.0 scales every pixel from the 0-16 integer range down to 0-1. Neural networks train better when input values are small and uniform.
  • The one-hot encoding block creates a matrix of zeros, then places a 1 at the correct class index for each sample. This gives the network a probability-shaped target to aim at.
  • train_test_split holds back 20% of the data (360 samples) as a test set the network never trains on. This lets you measure real accuracy.
Add the main block and run the script

Now you need a way to actually call the function and see what the data looks like. The if __name__ == "__main__" block runs when you execute the script directly.

  • Below the load_and_preprocess_data() function, add this main block:
if __name__ == "__main__":
    print("Loading data...")
    X_train, X_test, y_train, y_test = load_and_preprocess_data()
    print(f"Training samples: {X_train.shape[0]}, Test samples: {X_test.shape[0]}")
    print(f"Input features: {X_train.shape[1]}, Output classes: {y_train.shape[1]}")

What does this main block do?

It calls your preprocessing function, then prints the shape of each dataset split. You will see 1,437 training samples and 360 test samples, each with 64 input features and 10 output classes.

  • Save neural_network.py.
  • Run the script by executing this command in your terminal:
python neural_network.py

You should see this output:

Loading data...
Training samples: 1437, Test samples: 360
Input features: 64, Output classes: 10

Why these numbers?

The dataset has 1,797 total samples. An 80/20 split gives 1,437 for training and 360 for testing. Each sample has 64 features (the 8x8 pixel grid flattened). The output has 10 classes because there are 10 possible digits (0-9).

Seeing an error?

  • If you see ModuleNotFoundError, make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • If you see IndentationError, double-check that all lines inside the function are indented with 4 spaces.
  • If you see SyntaxError, make sure the f-string curly braces {X_train.shape[0]} are copied exactly.

Help me debug my neural_network.py script. I'm getting an error when running it after adding the load_and_preprocess_data function.

✔️ Awesome, I've got everything!

Great. Double check you have saved neural_network.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete neural_network.py file at this point:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt


# --- Data Loading and Preprocessing ---
def load_and_preprocess_data():
    digits = load_digits()
    X = digits.data / 16.0

    y = digits.target
    num_classes = 10
    y_onehot = np.zeros((y.shape[0], num_classes))
    y_onehot[np.arange(y.shape[0]), y] = 1

    X_train, X_test, y_train, y_test = train_test_split(
        X, y_onehot, test_size=0.2, random_state=42
    )
    return X_train, X_test, y_train, y_test


if __name__ == "__main__":
    print("Loading data...")
    X_train, X_test, y_train, y_test = load_and_preprocess_data()
    print(f"Training samples: {X_train.shape[0]}, Test samples: {X_test.shape[0]}")
    print(f"Input features: {X_train.shape[1]}, Output classes: {y_train.shape[1]}")

Your data pipeline is ready. You have 1,437 labeled digit images prepared for training and 360 held out for testing. Next up, you will build the forward propagation engine that transforms these 64-pixel inputs into digit predictions.

Implement Forward Propagation

Your data is loaded, normalized, and split into training and test sets. Now it's time for the exciting part: building the machinery that turns pixel values into predictions.

Forward propagation is how a neural network transforms raw inputs into a prediction. You'll build the mathematical pipeline that takes 64 pixel values and produces 10 probability scores, one for each digit.

In this step, get ready to:
  • Implement activation functions (ReLU and Softmax) that introduce non-linearity.
  • Initialize the network's weights and biases with the correct shapes.
  • Build the forward pass that chains matrix multiplications and activations into a prediction.
Implement activation functions

A neural network without activation functions is just a chain of linear equations. No matter how many layers you stack, the result is always a straight line. Activation functions break this linearity so the network can learn complex patterns like curves and edges in digit images.

  • In neural_network.py, find the end of the load_and_preprocess_data() function (the line return X_train, X_test, y_train, y_test).
  • Add the following activation functions between that function and the if __name__ block:

# --- Activation Functions ---
def relu(z):
    return np.maximum(0, z)


def relu_derivative(z):
    return (z > 0).astype(float)


def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)

What do these functions do?

  • relu(z) returns the input if it's positive, or zero if it's negative. This is the ReLU activation. It lets positive signals pass through while blocking negatives.
  • relu_derivative(z) returns 1 where the input is positive and 0 where it's negative. You'll need this during backpropagation to calculate gradients.
  • softmax(z) converts raw scores into probabilities that sum to 1. Subtracting np.max(z, axis=1, keepdims=True) before the exponential prevents numerical overflow without changing the result.

Why subtract the max in Softmax?

The exponential function grows extremely fast. If your raw scores are large numbers (like 500 or 1000), np.exp(500) overflows to infinity. Subtracting the maximum value from each row shifts everything down so the largest value becomes 0. This is the numerically stable version that all production neural networks use.

  • Save neural_network.py.
  • Run the file to confirm no syntax errors by running:
python neural_network.py

You should see the same data loading output as before. No errors means your activation functions are syntactically correct.

Seeing a syntax error?

  • Check that you placed the new code between the end of load_and_preprocess_data() and the if __name__ block. There should be two blank lines separating each function definition.
  • Make sure the function definitions start at column 0 (no indentation). They are top-level functions, not nested inside another function.

Help me debug my activation functions in neural_network.py

Initialize weights and biases

Your network needs parameters to learn. Weights control how strongly each input influences each neuron. Biases give neurons the ability to activate even when all inputs are zero.

The architecture you're building has 64 input neurons (one per pixel), 128 hidden neurons with ReLU, and 10 output neurons with Softmax (one per digit class).

  • Add the weight initialization function directly below your softmax function:

# --- Weight Initialization ---
def initialize_weights(input_size, hidden_size, output_size):
    np.random.seed(42)
    W1 = np.random.randn(input_size, hidden_size) * 0.01
    b1 = np.zeros((1, hidden_size))
    W2 = np.random.randn(hidden_size, output_size) * 0.01
    b2 = np.zeros((1, output_size))
    return W1, b1, W2, b2

What does this code do?

  • np.random.seed(42) makes the random initialization reproducible. You'll get the same starting weights every time you run the script.
  • W1 has shape (64, 128). It connects every input pixel to every hidden neuron. Multiplying by 0.01 keeps initial values small so gradients flow smoothly during training.
  • b1 has shape (1, 128). Biases start at zero.
  • W2 has shape (128, 10). It connects each hidden neuron to each of the 10 output classes.

Why multiply by 0.01?

If weights start too large, the activation functions saturate (output near 0 or 1 everywhere) and gradients vanish. If weights are all identical, every neuron learns the same thing. Small random values break this symmetry while keeping gradients healthy.

  • Save neural_network.py.
  • Run python neural_network.py to confirm no errors.

You should see the same data loading output with no new errors.

Seeing a NameError or IndentationError?

  • Confirm that initialize_weights is placed after the softmax function and before the if __name__ block. Two blank lines should separate it from the function above.
  • Check that all lines inside the function are indented by exactly 4 spaces.

Help me fix my initialize_weights function

Implement the forward pass function

The forward pass chains everything together. Data flows from input through the hidden layer to the output, transforming at each step via matrix multiplication and activation functions.

  • Add the forward propagation function directly below initialize_weights:

# --- Forward Propagation ---
def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1
    A1 = relu(Z1)
    Z2 = A1 @ W2 + b2
    A2 = softmax(Z2)
    cache = (Z1, A1, Z2, A2)
    return A2, cache

What does each line compute?

  • Z1 = X @ W1 + b1 multiplies the input matrix by the first weight matrix and adds the bias. The @ operator performs matrix multiplication.
  • A1 = relu(Z1) applies the ReLU activation to introduce non-linearity.
  • Z2 = A1 @ W2 + b2 passes the hidden layer output through the second set of weights.
  • A2 = softmax(Z2) converts raw scores into probabilities. The output is 10 numbers that sum to 1.
  • The cache stores intermediate values (Z1, A1, Z2, A2) that backpropagation needs to compute gradients.
  • Save neural_network.py.
  • Verify that the forward pass produces the correct output shape by running:
python -c "from neural_network import *; X_train, X_test, y_train, y_test = load_and_preprocess_data(); W1, b1, W2, b2 = initialize_weights(64, 128, 10); A2, _ = forward(X_train, W1, b1, W2, b2); print(f'Output shape: {A2.shape}'); print(f'Row sum: {A2[0].sum():.4f}')"

You should see Output shape: (1437, 10) and Row sum: 1.0000. This confirms your network produces one probability distribution per training sample, with all 10 class probabilities summing to 1.

Row sum not equal to 1.0?

  • Check your softmax function. Make sure you're using axis=1 and keepdims=True in both the np.max and np.sum calls. Without these, the subtraction and division broadcast incorrectly.
  • If you see a shape mismatch error, confirm that W1 has shape (64, 128) and W2 has shape (128, 10). The inner dimensions must match for matrix multiplication to work.

Help me debug my forward propagation output

✔️ Awesome, I've got everything!

Great. Double check you've saved neural_network.py before moving on.

ⓧ I'd like to double check the full code

Here's the complete neural_network.py file at this point:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt


# --- Data Loading and Preprocessing ---
def load_and_preprocess_data():
    digits = load_digits()
    X = digits.data / 16.0

    y = digits.target
    num_classes = 10
    y_onehot = np.zeros((y.shape[0], num_classes))
    y_onehot[np.arange(y.shape[0]), y] = 1

    X_train, X_test, y_train, y_test = train_test_split(
        X, y_onehot, test_size=0.2, random_state=42
    )
    return X_train, X_test, y_train, y_test


# --- Activation Functions ---
def relu(z):
    return np.maximum(0, z)


def relu_derivative(z):
    return (z > 0).astype(float)


def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)


# --- Weight Initialization ---
def initialize_weights(input_size, hidden_size, output_size):
    np.random.seed(42)
    W1 = np.random.randn(input_size, hidden_size) * 0.01
    b1 = np.zeros((1, hidden_size))
    W2 = np.random.randn(hidden_size, output_size) * 0.01
    b2 = np.zeros((1, output_size))
    return W1, b1, W2, b2


# --- Forward Propagation ---
def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1
    A1 = relu(Z1)
    Z2 = A1 @ W2 + b2
    A2 = softmax(Z2)
    cache = (Z1, A1, Z2, A2)
    return A2, cache


if __name__ == "__main__":
    print("Loading data...")
    X_train, X_test, y_train, y_test = load_and_preprocess_data()
    print(f"Training samples: {X_train.shape[0]}, Test samples: {X_test.shape[0]}")
    print(f"Input features: {X_train.shape[1]}, Output classes: {y_train.shape[1]}")

Your network can now make predictions. They're random predictions (the weights are tiny random numbers), but the mathematical pipeline is complete. Next up, you'll implement backpropagation to teach the network how to improve those predictions by learning from its mistakes.

Implement Backpropagation

Your forward pass turns pixel data into class probabilities. But right now, those probabilities are random guesses. The network needs a way to measure how wrong it is and figure out which weights caused the error.

Backpropagation is the algorithm that computes gradients by applying the chain rule backward through the network. Each gradient tells you exactly how much a specific weight contributed to the overall error. In this step, you will implement this gradient computation line by line.

In this step, get ready to:
  • Implement a cross-entropy loss function that measures prediction error.
  • Code the full backward pass using the chain rule.
  • Understand the gradient descent update rule that will train the network.
Implement the cross-entropy loss function

Before computing gradients, the network needs a single number that captures how wrong its predictions are. Cross-entropy loss measures the distance between predicted probabilities and the true labels. A prediction of 90% confidence on the correct class has low loss. A prediction of 1% confidence on the correct class has very high loss.

  • In neural_network.py, find the softmax function.
  • Add the following loss function between softmax and the # --- Weight Initialization --- section:
# --- Loss Function ---
def cross_entropy_loss(y_pred, y_true):
    m = y_true.shape[0]
    log_probs = -np.log(y_pred[np.arange(m), np.argmax(y_true, axis=1)] + 1e-15)
    return np.mean(log_probs)

What does this code do?

This function picks out the predicted probability for the correct class of each sample, then computes the negative log of those probabilities.

  • np.argmax(y_true, axis=1) converts one-hot labels back to class indices (e.g., [0,0,0,1,0,...] becomes 3).
  • y_pred[np.arange(m), ...] uses fancy indexing to grab the predicted probability for each sample's true class.
  • The + 1e-15 prevents log(0) which would produce negative infinity.
  • np.mean(log_probs) averages across all samples to get a single loss value.
  • Save neural_network.py.

Seeing a syntax error?

  • Make sure the function is placed after softmax and before initialize_weights. Python reads top to bottom, so order matters.
  • Check that np.arange(m) uses parentheses, not square brackets.

Help me debug my cross-entropy loss function

Code the backward pass

The backward pass is where the chain rule does its work. Starting from the output layer error, it flows gradients backward through each layer, computing how much each weight contributed to the total loss.

A beautiful mathematical shortcut makes this elegant. When you combine softmax with cross-entropy loss, the output gradient simplifies to just the predicted probabilities minus the true labels.

  • In neural_network.py, scroll down to the end of the forward function.
  • Add the backward propagation function between forward and the if __name__ == "__main__": block:
# --- Backward Propagation ---
def backward(X, y_true, cache, W1, b1, W2, b2):
    m = X.shape[0]
    Z1, A1, Z2, A2 = cache

    dZ2 = A2 - y_true
    dW2 = (A1.T @ dZ2) / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m

    dA1 = dZ2 @ W2.T
    dZ1 = dA1 * relu_derivative(Z1)
    dW1 = (X.T @ dZ1) / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m

    return dW1, db1, dW2, db2

What does this code do?

Every line is one step of the chain rule applied backward through the network:

  • dZ2 = A2 - y_true computes the output layer error. This elegant formula comes from the softmax + cross-entropy derivative.
  • dW2 and db2 are the gradients for the output layer weights and biases. Dividing by m averages across all samples.
  • dA1 = dZ2 @ W2.T propagates the error backward through the weight matrix to the hidden layer.
  • dZ1 = dA1 * relu_derivative(Z1) applies the chain rule through ReLU. Neurons that were "off" (Z1 <= 0) get zero gradient.
  • dW1 and db1 are the gradients for the input layer weights and biases.
  • Save neural_network.py.
  • Run the file to confirm there are no syntax errors:
python neural_network.py

You should see the same data loading output as before with no errors. The new functions are defined but not called yet.

Seeing a NameError for relu_derivative?

  • The backward function calls relu_derivative(Z1), which must be defined earlier in the file. Make sure your activation functions section is still intact above.
  • Check that you have def relu_derivative(z): defined in the # --- Activation Functions --- section.

Help me fix my backward function

✔️ Awesome, I've got everything!

Great. Double check you have saved neural_network.py and that it runs without errors.

ⓧ I'd like to double check the full code

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt


# --- Data Loading and Preprocessing ---
def load_and_preprocess_data():
    digits = load_digits()
    X = digits.data / 16.0

    y = digits.target
    num_classes = 10
    y_onehot = np.zeros((y.shape[0], num_classes))
    y_onehot[np.arange(y.shape[0]), y] = 1

    X_train, X_test, y_train, y_test = train_test_split(
        X, y_onehot, test_size=0.2, random_state=42
    )
    return X_train, X_test, y_train, y_test


# --- Activation Functions ---
def relu(z):
    return np.maximum(0, z)


def relu_derivative(z):
    return (z > 0).astype(float)


def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)


# --- Loss Function ---
def cross_entropy_loss(y_pred, y_true):
    m = y_true.shape[0]
    log_probs = -np.log(y_pred[np.arange(m), np.argmax(y_true, axis=1)] + 1e-15)
    return np.mean(log_probs)


# --- Weight Initialization ---
def initialize_weights(input_size, hidden_size, output_size):
    np.random.seed(42)
    W1 = np.random.randn(input_size, hidden_size) * 0.01
    b1 = np.zeros((1, hidden_size))
    W2 = np.random.randn(hidden_size, output_size) * 0.01
    b2 = np.zeros((1, output_size))
    return W1, b1, W2, b2


# --- Forward Propagation ---
def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1
    A1 = relu(Z1)
    Z2 = A1 @ W2 + b2
    A2 = softmax(Z2)
    cache = (Z1, A1, Z2, A2)
    return A2, cache


# --- Backward Propagation ---
def backward(X, y_true, cache, W1, b1, W2, b2):
    m = X.shape[0]
    Z1, A1, Z2, A2 = cache

    dZ2 = A2 - y_true
    dW2 = (A1.T @ dZ2) / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m

    dA1 = dZ2 @ W2.T
    dZ1 = dA1 * relu_derivative(Z1)
    dW1 = (X.T @ dZ1) / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m

    return dW1, db1, dW2, db2


if __name__ == "__main__":
    print("Loading data...")
    X_train, X_test, y_train, y_test = load_and_preprocess_data()
    print(f"Training samples: {X_train.shape[0]}, Test samples: {X_test.shape[0]}")
    print(f"Input features: {X_train.shape[1]}, Output classes: {y_train.shape[1]}")
Understand the gradient descent update rule

Your backward function returns four gradients: dW1, db1, dW2, db2. Each gradient has the exact same shape as its corresponding weight or bias matrix. This is essential because gradient descent subtracts the gradient from the weight element by element.

The update rule is simple: W = W - learning_rate * dW. Each weight gets nudged in the direction that reduces the loss. A learning rate of 0.5 works well for full-batch gradient descent on this small dataset because the gradients are smooth and stable.

Why does the learning rate matter?

The learning rate controls how big each step is. Too large and the network overshoots the optimal weights, causing loss to spike. Too small and training takes thousands of epochs to converge.

With 1,437 training samples processed at once (full-batch), the gradient signal is very clean. This allows a relatively large learning rate of 0.5. In the Secret Mission, you will switch to mini-batches where gradients are noisier, requiring a smaller learning rate of 0.1.

You will implement the actual update loop in the next step. The key insight is that each gradient must have the same shape as its parameter. For example, dW1 is (64, 128) matching W1, and dW2 is (128, 10) matching W2.

Your loss function and backward pass are ready. Next up, you will wire everything together into a training loop that repeatedly runs forward, measures loss, computes gradients, and updates the weights until accuracy climbs above 95%.

Train the Network and Visualize Results

You have all the building blocks in place. The forward pass computes predictions, the loss function measures how wrong those predictions are, and the backward pass computes gradients telling each weight how to improve.

Now you will connect these pieces into a training loop that runs gradient descent for 500 epochs. Each epoch runs the full cycle: predict, measure error, compute gradients, nudge weights. By the end, your network will classify handwritten digits at 95%+ accuracy.

In this step, get ready to:
  • Build the training loop that ties forward pass, loss, and backpropagation together.
  • Create visualization functions for the loss curve and prediction grid.
  • Run the complete program and watch accuracy climb from 10% to 95%+.
Build the training loop

The training loop is where learning actually happens. Each iteration (called an epoch) pushes the entire training set through the network, measures the total error, then nudges every weight in the direction that reduces that error.

  • In neural_network.py, find the backward function.
  • Add the following train function below it, before the if __name__ block:
# --- Training ---
def train(X_train, y_train, X_test, y_test, epochs=500, learning_rate=0.5):
    input_size = 64
    hidden_size = 128
    output_size = 10

    W1, b1, W2, b2 = initialize_weights(input_size, hidden_size, output_size)

    train_losses = []

    for epoch in range(epochs):
        # Forward pass: compute predictions
        y_pred, cache = forward(X_train, W1, b1, W2, b2)

        # Compute loss: how wrong are we?
        loss = cross_entropy_loss(y_pred, y_train)
        train_losses.append(loss)

        # Backward pass: compute gradients
        dW1, db1, dW2, db2 = backward(X_train, y_train, cache, W1, b1, W2, b2)

        # Gradient descent: nudge weights to reduce loss
        W1 -= learning_rate * dW1
        b1 -= learning_rate * db1
        W2 -= learning_rate * dW2
        b2 -= learning_rate * db2

        # Print progress every 50 epochs
        if epoch % 50 == 0:
            test_pred, _ = forward(X_test, W1, b1, W2, b2)
            accuracy = np.mean(
                np.argmax(test_pred, axis=1) == np.argmax(y_test, axis=1)
            )
            print(f"Epoch {epoch:4d} | Loss: {loss:.4f} | Test Accuracy: {accuracy:.4f}")

    return W1, b1, W2, b2, train_losses

What does this code do?

This function orchestrates the entire learning process:

  • It initializes random weights, then loops 500 times through the same training data.
  • Each epoch runs forward pass, loss computation, backward pass, and weight updates in sequence.
  • The update rule W1 -= learning_rate * dW1 subtracts a fraction of the gradient from each weight. This is gradient descent: move downhill on the loss surface.
  • Every 50 epochs, it evaluates on the test set so you can watch accuracy improve over time.
  • Save neural_network.py.
  • You won't see a change yet. The next substep adds the visualization functions, and then you will run the full program.
Add the visualization functions

Numbers alone don't tell the full story. A loss curve shows whether the network is learning smoothly, and a prediction grid lets you see the network's guesses on real digit images.

  • Below the train function you just added (still above the if __name__ block), add these two visualization functions:
# --- Visualization ---
def visualize_predictions(X_test, y_test, W1, b1, W2, b2):
    # Run test images through the trained network
    predictions, _ = forward(X_test, W1, b1, W2, b2)
    pred_labels = np.argmax(predictions, axis=1)
    true_labels = np.argmax(y_test, axis=1)

    # Display a 2x5 grid of predictions vs true labels
    fig, axes = plt.subplots(2, 5, figsize=(12, 5))
    for i, ax in enumerate(axes.flat):
        image = X_test[i].reshape(8, 8)
        ax.imshow(image, cmap="gray_r")
        ax.set_title(f"Pred: {pred_labels[i]}, True: {true_labels[i]}")
        ax.axis("off")

    plt.tight_layout()
    plt.savefig("predictions.png")
    plt.show()
    print("Predictions saved to predictions.png")


def plot_training_loss(train_losses):
    # Plot how loss decreased over training
    plt.figure(figsize=(10, 5))
    plt.plot(train_losses)
    plt.xlabel("Epoch")
    plt.ylabel("Cross-Entropy Loss")
    plt.title("Training Loss Over Time")
    plt.grid(True)
    plt.savefig("training_loss.png")
    plt.show()
    print("Training loss plot saved to training_loss.png")

What do these functions do?

visualize_predictions runs 10 test images through the trained network and displays them in a 2x5 grid. Each image shows both the network's prediction and the true label, so you can see where it gets things right (and wrong).

plot_training_loss plots the loss value at every epoch. A healthy training curve drops steeply at first then flattens near zero. Both functions save their output as PNG files for reference.

  • Save neural_network.py.
  • You won't see output yet. The next substep updates the main block to call everything together.
Update main and run the program

The main block ties everything together: load data, train, report accuracy, and produce visualizations.

  • In neural_network.py, find the existing if __name__ == "__main__": block at the bottom of the file.
  • Replace the entire block (from if __name__ to the last line of the file) with this updated version:
# --- Main ---
if __name__ == "__main__":
    print("Loading data...")
    X_train, X_test, y_train, y_test = load_and_preprocess_data()
    print(f"Training samples: {X_train.shape[0]}, Test samples: {X_test.shape[0]}")
    print(f"Input features: {X_train.shape[1]}, Output classes: {y_train.shape[1]}")

    print("\nTraining neural network...")
    W1, b1, W2, b2, train_losses = train(
        X_train, y_train, X_test, y_test, epochs=500, learning_rate=0.5
    )

    print("\nFinal Results:")
    final_pred, _ = forward(X_test, W1, b1, W2, b2)
    final_accuracy = np.mean(
        np.argmax(final_pred, axis=1) == np.argmax(y_test, axis=1)
    )
    print(f"Final Test Accuracy: {final_accuracy:.4f}")

    print("\nVisualizing results...")
    plot_training_loss(train_losses)
    visualize_predictions(X_test, y_test, W1, b1, W2, b2)

What does this main block do?

  • Loads and preprocesses the digit images into training and test sets.
  • Calls train() which runs 500 epochs of gradient descent and returns the learned weights.
  • Computes and prints the final test accuracy using the trained weights.
  • Generates both plots (loss curve and prediction grid) and saves them as PNG files.
  • Save neural_network.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file, then continue below to run it.

ⓧ I'd like to double check the full code

Here is the complete neural_network.py file. Compare yours against this:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt


# --- Data Loading and Preprocessing ---
def load_and_preprocess_data():
    digits = load_digits()
    X = digits.data / 16.0

    y = digits.target
    num_classes = 10
    y_onehot = np.zeros((y.shape[0], num_classes))
    y_onehot[np.arange(y.shape[0]), y] = 1

    X_train, X_test, y_train, y_test = train_test_split(
        X, y_onehot, test_size=0.2, random_state=42
    )
    return X_train, X_test, y_train, y_test


# --- Activation Functions ---
def relu(z):
    return np.maximum(0, z)


def relu_derivative(z):
    return (z > 0).astype(float)


def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)


# --- Loss Function ---
def cross_entropy_loss(y_pred, y_true):
    m = y_true.shape[0]
    log_probs = -np.log(y_pred[np.arange(m), np.argmax(y_true, axis=1)] + 1e-15)
    return np.mean(log_probs)


# --- Weight Initialization ---
def initialize_weights(input_size, hidden_size, output_size):
    np.random.seed(42)
    W1 = np.random.randn(input_size, hidden_size) * 0.01
    b1 = np.zeros((1, hidden_size))
    W2 = np.random.randn(hidden_size, output_size) * 0.01
    b2 = np.zeros((1, output_size))
    return W1, b1, W2, b2


# --- Forward Propagation ---
def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1
    A1 = relu(Z1)
    Z2 = A1 @ W2 + b2
    A2 = softmax(Z2)
    cache = (Z1, A1, Z2, A2)
    return A2, cache


# --- Backward Propagation ---
def backward(X, y_true, cache, W1, b1, W2, b2):
    m = X.shape[0]
    Z1, A1, Z2, A2 = cache

    dZ2 = A2 - y_true
    dW2 = (A1.T @ dZ2) / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m

    dA1 = dZ2 @ W2.T
    dZ1 = dA1 * relu_derivative(Z1)
    dW1 = (X.T @ dZ1) / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m

    return dW1, db1, dW2, db2


# --- Training ---
def train(X_train, y_train, X_test, y_test, epochs=500, learning_rate=0.5):
    input_size = 64
    hidden_size = 128
    output_size = 10

    W1, b1, W2, b2 = initialize_weights(input_size, hidden_size, output_size)

    train_losses = []

    for epoch in range(epochs):
        y_pred, cache = forward(X_train, W1, b1, W2, b2)

        loss = cross_entropy_loss(y_pred, y_train)
        train_losses.append(loss)

        dW1, db1, dW2, db2 = backward(X_train, y_train, cache, W1, b1, W2, b2)

        W1 -= learning_rate * dW1
        b1 -= learning_rate * db1
        W2 -= learning_rate * dW2
        b2 -= learning_rate * db2

        if epoch % 50 == 0:
            test_pred, _ = forward(X_test, W1, b1, W2, b2)
            accuracy = np.mean(
                np.argmax(test_pred, axis=1) == np.argmax(y_test, axis=1)
            )
            print(f"Epoch {epoch:4d} | Loss: {loss:.4f} | Test Accuracy: {accuracy:.4f}")

    return W1, b1, W2, b2, train_losses


# --- Visualization ---
def visualize_predictions(X_test, y_test, W1, b1, W2, b2):
    predictions, _ = forward(X_test, W1, b1, W2, b2)
    pred_labels = np.argmax(predictions, axis=1)
    true_labels = np.argmax(y_test, axis=1)

    fig, axes = plt.subplots(2, 5, figsize=(12, 5))
    for i, ax in enumerate(axes.flat):
        image = X_test[i].reshape(8, 8)
        ax.imshow(image, cmap="gray_r")
        ax.set_title(f"Pred: {pred_labels[i]}, True: {true_labels[i]}")
        ax.axis("off")

    plt.tight_layout()
    plt.savefig("predictions.png")
    plt.show()
    print("Predictions saved to predictions.png")


def plot_training_loss(train_losses):
    plt.figure(figsize=(10, 5))
    plt.plot(train_losses)
    plt.xlabel("Epoch")
    plt.ylabel("Cross-Entropy Loss")
    plt.title("Training Loss Over Time")
    plt.grid(True)
    plt.savefig("training_loss.png")
    plt.show()
    print("Training loss plot saved to training_loss.png")


# --- Main ---
if __name__ == "__main__":
    print("Loading data...")
    X_train, X_test, y_train, y_test = load_and_preprocess_data()
    print(f"Training samples: {X_train.shape[0]}, Test samples: {X_test.shape[0]}")
    print(f"Input features: {X_train.shape[1]}, Output classes: {y_train.shape[1]}")

    print("\nTraining neural network...")
    W1, b1, W2, b2, train_losses = train(
        X_train, y_train, X_test, y_test, epochs=500, learning_rate=0.5
    )

    print("\nFinal Results:")
    final_pred, _ = forward(X_test, W1, b1, W2, b2)
    final_accuracy = np.mean(
        np.argmax(final_pred, axis=1) == np.argmax(y_test, axis=1)
    )
    print(f"Final Test Accuracy: {final_accuracy:.4f}")

    print("\nVisualizing results...")
    plot_training_loss(train_losses)
    visualize_predictions(X_test, y_test, W1, b1, W2, b2)
  • Run your neural network by executing this command in your terminal:
python neural_network.py

Two plot windows will pop up one at a time. Close each plot window to continue execution.

You should see training progress printed in the terminal. The loss starts around 2.3 (random guessing across 10 classes) and drops steadily. Test accuracy climbs from roughly 10% at epoch 0 to 95%+ by epoch 450.

What should I see?

Your terminal output will look something like this:

Epoch    0 | Loss: 2.3026 | Test Accuracy: 0.0917
Epoch   50 | Loss: 0.3510 | Test Accuracy: 0.9111
Epoch  100 | Loss: 0.1542 | Test Accuracy: 0.9444
...
Epoch  450 | Loss: 0.0100 | Test Accuracy: 0.9694
Final Test Accuracy: 0.9694

Your exact numbers may differ slightly, but the pattern is the same: loss drops rapidly and accuracy rises above 0.95. You will also see training_loss.png and predictions.png saved in your nn-from-scratch folder.

Seeing an error or low accuracy?

  • If you get a NameError, check that the train function is defined before the if __name__ block, not inside it.
  • If accuracy stays at 10%, make sure your backward function returns dW1, db1, dW2, db2 in that exact order and that the weight update lines use -= (not +=).
  • If the plot window freezes your terminal, close it by clicking the X on the plot window. The script continues after each window closes.

Why is my neural network not learning?

  • Confirm that training_loss.png and predictions.png exist in your nn-from-scratch folder by running:
ls *.png

You should see predictions.png and training_loss.png in the output.

Your neural network just taught itself to read handwritten digits. Every component, from the matrix multiplications in the forward pass to the chain rule derivatives in backpropagation, was code you wrote by hand. This is the same fundamental loop that powers every deep learning system in production today.

Secret mission

Mini-Batch Gradient Descent with He Initialization

Your network trains on all 1,437 samples at once. Real neural networks never do this. Can you upgrade to mini-batch gradient descent and He initialization for faster, more stable training?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your files for future reference or delete them to free up disk space.

Resources you used:

  • neural_network.py (your neural network implementation, local file).
  • training_loss.png (training loss plot, local file).
  • predictions.png (predictions visualization, local file).
  • venv folder (virtual environment with NumPy, matplotlib, and scikit-learn installed).

Keep everything running

No action needed. Choose this if you want to keep experimenting with your neural network or reference the code later.

  • Your nn-from-scratch folder stays on your Desktop.
  • The virtual environment uses around 200-400 MB of disk space for the installed packages.
  • No cloud resources or subscriptions to worry about.

Pause - I'll come back to this later

Deactivate your virtual environment to free up your terminal session. Your files remain on disk for when you return.

  • Deactivate your virtual environment by running:
deactivate
  • When you come back, reactivate it with venv\Scripts\activate (Windows) or source venv/bin/activate (macOS/Linux) from inside the nn-from-scratch folder.

Delete - I don't want to use this again

Remove all project files and the virtual environment entirely. This frees up all disk space used by the project.

  • First, deactivate your virtual environment if it is still active:
deactivate
  • Navigate to your Desktop:
cd ~/Desktop
  • Delete the entire project folder by running this command on macOS/Linux:
rm -rf nn-from-scratch
  • On Windows, run this instead:
rmdir /s /q nn-from-scratch
  • Confirm the folder is gone by running:
ls
  • You should no longer see nn-from-scratch in the output.

Nice Work!

Nice Work!

You just built a neural network from absolute scratch using nothing but NumPy and math. Your network classifies handwritten digits at 95%+ accuracy, proving you understand the core engine that powers every deep learning system from GPT to image classifiers.

You've learned how to:

  • Implement forward propagation as matrix multiplication, transforming raw pixel data into class probabilities through linear transformations and activation functions.
  • Derive and code backpropagation using the chain rule, computing exact gradients layer by layer to tell the network how to improve.
  • Train a network with gradient descent and visualize the learning process with loss curves and prediction grids, watching accuracy climb from 10% to 95%+.
  • Secret Mission: Upgraded to mini-batch gradient descent with He initialization for faster convergence and production-ready training dynamics.

Ready to quiz yourself?