Build a Neural Network From Scratch

Build a digit-recognizing neural network using only NumPy, no frameworks.

Introduction

30 Second Summary

Every AI system, from ChatGPT to self-driving cars, learns through the same core mechanism. It makes a guess, measures how wrong it was, and adjusts itself to be less wrong next time.

In this project, you will build a neural network entirely from scratch that recognizes handwritten digits. You will write every piece of the learning mechanism yourself, giving you a deep understanding of how all AI actually works.

What You'll Build

A handwritten digit classifier that takes an 8x8 pixel image of a number (0-9) and correctly identifies which digit it is, built using only NumPy (a math library).

By the end of this project, you'll have:

  • A working neural network that achieves over 90% accuracy on digit recognition, proving you understand the core mechanism that powers all modern AI.
  • A clear mental model of how every AI system learns: forward propagation, loss functions, backpropagation, and gradient descent.
  • A Google Colab notebook you can revisit, modify, and experiment with to deepen your understanding.
  • Secret Mission: Add a second hidden layer and compare how "deeper" networks perform differently.

Are there any prerequisites?

You only need a Google account. Google Colab provides Python, NumPy, and all other tools pre-installed in your browser, so there is nothing to download or configure.

Set Up Your Workspace in Google Colab

Every neural network needs data to learn from and a workspace to run in. In this step, you will open Google Colab (a free, browser-based coding environment), load a dataset of handwritten digit images, and prepare that data for your neural network to consume.

By the end of this step, your data will be cleaned, formatted, and split into training and test sets. That gives your neural network everything it needs to start learning in the next step.

In this step, get ready to:
  • Open Google Colab and create a new notebook.
  • Load and visualize the handwritten digits dataset.
  • Prepare the data for neural network training.
Open Google Colab and create a new notebook

Google Colab is a free online notebook environment from Google. It comes with Python and popular data science libraries pre-installed, so you can start coding immediately with zero setup.

  • Open your browser and navigate to colab.research.google.com.
  • Sign in with your Google account if prompted.
  • Click New notebook in the pop-up window that appears.
  • Click the filename at the top left (it says Untitled0.ipynb by default) and rename it to Neural Network From Scratch.

What is Google Colab?

Google Colab is a hosted Jupyter Notebook service that runs Python code in your browser. It provides free access to computing resources and comes with libraries like NumPy, scikit-learn, and matplotlib already installed.

You write code in "cells" and run each cell individually by clicking the play button or pressing Shift+Enter.

Verify your environment and load the dataset

Before writing any neural network logic, you need to confirm that the required libraries are available and load the data your network will learn from. The sklearn digits dataset contains 1,797 images of handwritten digits, each 8x8 pixels.

  • Import the required libraries and load the digits dataset by pasting this code into the first cell and pressing Shift+Enter to run it:
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# =============================================================
# STEP 1: Load and prepare the data
# =============================================================

# Load the digits dataset
digits = load_digits()
X = digits.data  # Shape: (1797, 64)
y = digits.target  # Shape: (1797,)

What does this code do?

  • The first four lines import the libraries you need: NumPy for math, load_digits for the dataset, train_test_split for dividing data, and matplotlib for plotting.
  • load_digits() loads 1,797 handwritten digit images. Each image is 8x8 pixels, flattened into 64 numbers.
  • X holds the pixel data (shape 1797 x 64) and y holds the digit labels (0 through 9).
  • The cell should run without any errors, confirming all libraries are available.

Seeing an ImportError?

If you see an error mentioning a missing module, make sure you are running this in Google Colab (not a local Python installation). Colab has NumPy, scikit-learn, and matplotlib pre-installed.

If you accidentally opened a local Jupyter notebook, close it and return to colab.research.google.com to start fresh. Help me fix my import error in Google Colab

Now visualize a few sample digits to see what your network will learn to recognize.

  • Create a new code cell by clicking the + Code button at the top, then paste this code and press Shift+Enter:
# Visualize some sample digits
fig, axes = plt.subplots(1, 5, figsize=(10, 3))
for i, ax in enumerate(axes):
    ax.imshow(digits.images[i], cmap='gray')
    ax.set_title(f"Label: {y[i]}")
    ax.axis('off')
plt.suptitle("Sample Handwritten Digits")
plt.show()

What does this code do?

  • plt.subplots(1, 5) creates a row of 5 image slots side by side.
  • ax.imshow() displays each 8x8 digit image in grayscale.
  • Each image gets a title showing its label (the actual digit it represents).
  • You should see 5 small grayscale images of handwritten digits with their labels displayed above each one.

No images appearing?

Make sure you ran the first cell (imports and data loading) before this one. Colab runs cells independently, so the second cell needs the variables created by the first.

If you see a blank output, try clicking Runtime in the top menu, then Run all to re-run both cells in order. Help me display digit images in Colab

Prepare the data for the neural network

Raw pixel values and integer labels are not ideal for a neural network. You need to normalize the numbers and convert labels into a format the network can work with.

  • Normalize the pixel values and one-hot encode the labels by adding a new code cell with this code:
# Normalize pixel values to 0-1 range
X = X / 16.0

# One-hot encode the labels
def one_hot_encode(labels, num_classes=10):
    encoded = np.zeros((len(labels), num_classes))
    for i, label in enumerate(labels):
        encoded[i, label] = 1.0
    return encoded

y_encoded = one_hot_encode(y)

What does this code do?

  • X = X / 16.0 scales pixel values from the 0-16 range down to 0-1. Smaller, consistent numbers help the network learn faster and more stably.
  • one_hot_encode() converts each digit label into a vector of 10 numbers. For example, the digit 3 becomes [0,0,0,1,0,0,0,0,0,0].
  • This format lets the network output a probability for each digit class, making it easy to measure how wrong its guesses are.
  • Run the cell. You won't see any output, which is expected. The data is transformed in memory.

Why normalize to 0-1?

Neural networks learn by adjusting small numerical weights. If input values are large (like 0-16), the math during training can become unstable or very slow.

Scaling everything to 0-1 keeps the numbers small and consistent, which helps gradient descent (the learning algorithm) converge smoothly.

  • Split the data into training and test sets by adding a final code cell:
# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y_encoded, test_size=0.2, random_state=42
)

print(f"Training samples: {X_train.shape[0]}")
print(f"Test samples: {X_test.shape[0]}")
print(f"Features per sample: {X_train.shape[1]}")

What does this code do?

  • train_test_split() randomly divides your data into two groups: 80% for training and 20% for testing.
  • The training set is what the network learns from. The test set is held back to measure how well the network handles data it has never seen.
  • random_state=42 ensures the split is the same every time you run it, making your results reproducible.
  • Run the cell. You should see this output:
Training samples: 1437
Test samples: 360
Features per sample: 64

Seeing a NameError?

If you see an error like NameError: name 'X' is not defined, it means the previous cells did not run successfully.

Click Runtime in the top menu, then Run all to execute all cells from the beginning. Each cell depends on the variables created by the cells above it. Help me fix my NameError in Colab

✔️ Awesome, I've got everything!

Great. Double check that all four cells ran without errors and that your final output shows 1,437 training samples and 360 test samples.

ⓧ I'd like to double check the full code

Here is the complete code across all four cells in this step. Compare it against your notebook:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# =============================================================
# STEP 1: Load and prepare the data
# =============================================================

# Load the digits dataset
digits = load_digits()
X = digits.data  # Shape: (1797, 64)
y = digits.target  # Shape: (1797,)

# Visualize some sample digits
fig, axes = plt.subplots(1, 5, figsize=(10, 3))
for i, ax in enumerate(axes):
    ax.imshow(digits.images[i], cmap='gray')
    ax.set_title(f"Label: {y[i]}")
    ax.axis('off')
plt.suptitle("Sample Handwritten Digits")
plt.show()

# Normalize pixel values to 0-1 range
X = X / 16.0

# One-hot encode the labels
def one_hot_encode(labels, num_classes=10):
    encoded = np.zeros((len(labels), num_classes))
    for i, label in enumerate(labels):
        encoded[i, label] = 1.0
    return encoded

y_encoded = one_hot_encode(y)

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y_encoded, test_size=0.2, random_state=42
)

print(f"Training samples: {X_train.shape[0]}")
print(f"Test samples: {X_test.shape[0]}")
print(f"Features per sample: {X_train.shape[1]}")

Your workspace is ready and your data is prepped. Next up, you will build the actual neuron math that powers your neural network.

Build the Neuron Math

Your data is loaded, normalized, and split into training and test sets. Now it's time to build the core building blocks that make a neural network actually compute things: the math.

Every neural network is built from neurons that take numbers in, transform them through mathematical functions, and pass the results forward. In this step, you'll implement those transformations yourself using only NumPy.

In this step, get ready to:
  • Implement activation functions (ReLU and softmax).
  • Initialize the network's weights and biases.
  • Build the forward pass function that makes predictions.
Understand and implement activation functions

Activation functions decide what signal a neuron passes forward. Without them, stacking multiple layers would be pointless because the math would collapse into a single linear operation.

You'll implement two activation functions. ReLU passes positive values through and blocks negatives (acting like a gate). Softmax converts raw scores into probabilities that sum to 1, which is how the network expresses its confidence about each digit.

  • Add a new code cell by clicking + Code at the top of your notebook, then paste the following code:
# =============================================================
# STEP 2: Build the neuron math
# =============================================================

# Activation functions
def relu(z):
    return np.maximum(0, z)

def relu_derivative(z):
    return (z > 0).astype(float)

def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)

What does this code do?

  • relu(z) returns the input if it's positive, or zero if it's negative. This simple gate lets the network learn non-linear patterns.
  • relu_derivative(z) returns 1 for positive values and 0 for negatives. The network needs this during backpropagation (coming in Step 3) to know which neurons contributed to the output.
  • softmax(z) takes a row of raw scores and squishes them into probabilities that add up to 1. The subtraction of np.max(z) prevents overflow when exponentiating large numbers.
  • Run the cell by pressing Shift+Enter.
  • You should see no output or errors. The functions are now defined and available for the rest of your notebook.

How can I verify these functions work?

After running the cell, you can test ReLU in a new cell by typing print(relu(np.array([[-1, 0, 3, -2, 5]]))) and running it. You should see [[0 0 3 0 5]]. Negative values become zero while positive values pass through unchanged.

Seeing a NameError?

  • Make sure you ran the Step 1 cell first. The activation functions use np (NumPy), which was imported in that earlier cell.
  • Check that your cells ran in order from top to bottom. Colab cells must be executed sequentially for variables and imports to be available.

Help me fix my NameError in Google Colab

Initialize the network's weights and biases

Weights are the "knowledge" of the network. Before training, they start as random numbers. The specific way you randomize them matters for whether the network can learn effectively.

Your network architecture has three layers: 64 inputs (one per pixel), 128 hidden neurons (where learning happens), and 10 outputs (one per digit). The weights connect these layers together.

  • Add a new code cell below your activation functions cell, then paste this code:
# Initialize weights and biases
np.random.seed(42)

input_size = 64
hidden_size = 128
output_size = 10

# He initialization
W1 = np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size)
b1 = np.zeros((1, hidden_size))
W2 = np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size)
b2 = np.zeros((1, output_size))

What does this code do?

  • np.random.seed(42) locks the random number generator so you get the same results every time you run the notebook. This makes debugging easier.
  • input_size, hidden_size, and output_size define the shape of each layer: 64 pixels in, 128 hidden neurons, 10 digit classes out.
  • He initialization scales random weights by sqrt(2/n) where n is the number of inputs to that layer. This prevents signals from exploding or vanishing as they pass through layers.
  • Biases (b1 and b2) start at zero. They give each neuron a baseline activation level that the network adjusts during training.

Why do we use random initialization?

If all weights started at the same value (like zero), every neuron would compute the exact same thing. They would all learn the same pattern, making the entire hidden layer useless. Random values break this symmetry so each neuron can specialize in detecting different features of the digit images.

  • Run the cell by pressing Shift+Enter.
  • You should see no output. The weights and biases are now stored in memory, ready for the forward pass.

Seeing unexpected output?

  • If you see array values printed, you may have accidentally left out a line or added a bare variable name at the end of the cell. Colab prints the last expression in a cell if it's not assigned to a variable.
  • If you see a shape error, double-check that input_size = 64, hidden_size = 128, and output_size = 10 are defined exactly as shown.

Help me debug my weight initialization code

Implement the forward pass

The forward pass is the network making a prediction. Data flows from input through the hidden layer to the output. Each step is matrix multiplication followed by an activation function.

  • Add a new code cell below your weight initialization cell, then paste this code:
# Forward pass
def forward(X, W1, b1, W2, b2):
    Z1 = X.dot(W1) + b1
    A1 = relu(Z1)
    Z2 = A1.dot(W2) + b2
    A2 = softmax(Z2)
    cache = (Z1, A1, Z2, A2)
    return A2, cache

What does this code do?

The forward pass moves data through two layers, step by step:

  • Hidden layer: Z1 = X.dot(W1) + b1 multiplies each input by its weights and adds the bias. Then A1 = relu(Z1) applies ReLU to block negative values.
  • Output layer: Z2 = A1.dot(W2) + b2 multiplies hidden activations by the second set of weights. Then A2 = softmax(Z2) converts the raw scores into 10 probabilities (one per digit).
  • The cache stores all the intermediate values. Backpropagation will need them in Step 3 to figure out how to update the weights.
  • Run the cell by pressing Shift+Enter.
  • You should see no output. The function is now defined and ready to make predictions once training begins in the next step.

Seeing a NameError for relu or softmax?

  • Make sure your activation functions cell (the first cell in this step) ran successfully before this one. The forward function calls relu() and softmax(), so they must be defined first.
  • If you restarted your runtime, you need to re-run all previous cells from the top. Click Runtime in the menu bar, then Run all.

Help me fix my forward pass function

✔️ Awesome, I've got everything!

Great. Double check you've run all your cells in order and have no errors showing.

ⓧ I'd like to double check the full code

Here's everything your notebook should contain up to this point. Compare your cells against this:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# =============================================================
# STEP 1: Load and prepare the data
# =============================================================

# Load the digits dataset
digits = load_digits()
X = digits.data  # Shape: (1797, 64)
y = digits.target  # Shape: (1797,)

# Visualize some sample digits
fig, axes = plt.subplots(1, 5, figsize=(10, 3))
for i, ax in enumerate(axes):
    ax.imshow(digits.images[i], cmap='gray')
    ax.set_title(f"Label: {y[i]}")
    ax.axis('off')
plt.suptitle("Sample Handwritten Digits")
plt.show()

# Normalize pixel values to 0-1 range
X = X / 16.0

# One-hot encode the labels
def one_hot_encode(labels, num_classes=10):
    encoded = np.zeros((len(labels), num_classes))
    for i, label in enumerate(labels):
        encoded[i, label] = 1.0
    return encoded

y_encoded = one_hot_encode(y)

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y_encoded, test_size=0.2, random_state=42
)

print(f"Training samples: {X_train.shape[0]}")
print(f"Test samples: {X_test.shape[0]}")
print(f"Features per sample: {X_train.shape[1]}")

# =============================================================
# STEP 2: Build the neuron math
# =============================================================

# Activation functions
def relu(z):
    return np.maximum(0, z)

def relu_derivative(z):
    return (z > 0).astype(float)

def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)

# Initialize weights and biases
np.random.seed(42)

input_size = 64
hidden_size = 128
output_size = 10

# He initialization
W1 = np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size)
b1 = np.zeros((1, hidden_size))
W2 = np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size)
b2 = np.zeros((1, output_size))

# Forward pass
def forward(X, W1, b1, W2, b2):
    Z1 = X.dot(W1) + b1
    A1 = relu(Z1)
    Z2 = A1.dot(W2) + b2
    A2 = softmax(Z2)
    cache = (Z1, A1, Z2, A2)
    return A2, cache

Your network now has all the math it needs to make predictions. Next up, you'll teach it to learn from its mistakes using loss functions and backpropagation.

Teach the Network to Learn

Your network can now make predictions through the forward pass, but those predictions start completely random. To improve, the network needs a way to measure its mistakes and figure out which weights caused them.

In this step, you'll implement the three mechanisms that make learning possible. A loss function measures errors, backpropagation traces those errors back to their source, and gradient descent nudges the weights in the right direction.

In this step, get ready to:
  • Implement the cross-entropy loss function to measure prediction errors.
  • Implement backpropagation to calculate weight gradients.
  • Implement gradient descent to update the weights.
Implement the cross-entropy loss function

The network needs a way to quantify how wrong its predictions are. Cross-entropy is the standard loss function for classification problems. It penalizes confident wrong answers much more heavily than uncertain ones.

  • Create a new code cell in your notebook and add the following code:
# =============================================================
# STEP 3: Teach the network to learn
# =============================================================

# Cross-entropy loss
def compute_loss(y_true, y_pred):
    m = y_true.shape[0]
    # Clip to avoid log(0)
    y_pred_clipped = np.clip(y_pred, 1e-12, 1.0 - 1e-12)
    loss = -np.sum(y_true * np.log(y_pred_clipped)) / m
    return loss

What does this code do?

  • m = y_true.shape[0] gets the number of training samples so we can average the loss.
  • np.clip(y_pred, 1e-12, 1.0 - 1e-12) prevents the log function from receiving exactly 0, which would produce infinity.
  • The formula -np.sum(y_true * np.log(y_pred_clipped)) / m is the cross-entropy equation. It only measures the log-probability at the correct class (because y_true is one-hot, so all other terms are zeroed out).
  • Run the cell by pressing the play button or Shift+Enter.
  • Test the loss function by adding a new cell below and running this verification code:
# Test the loss function
test_predictions, _ = forward(X_train, W1, b1, W2, b2)
print(f"Loss with untrained network: {compute_loss(y_train, test_predictions):.4f}")

# A perfect prediction should give loss near 0
perfect = y_train.copy()
print(f"Loss with perfect predictions: {compute_loss(y_train, perfect):.4f}")

You should see the untrained network's loss is high (around 2.3 or above), while the perfect predictions give a loss near 0.

Seeing an error when running compute_loss?

  • Make sure you ran all the previous cells first (Step 1 and Step 2 code). The variables X_train, y_train, W1, b1, W2, and b2 must exist in memory.
  • Check that the function name is spelled exactly compute_loss with an underscore.

Why is my compute_loss function returning an error or unexpected values?

Implement backpropagation

Now that the network can measure its mistakes, it needs to figure out which weights caused those mistakes. Backpropagation answers this question by tracing the error backward through the network using the chain rule from calculus.

Each gradient tells you how much a specific weight contributed to the error. A large gradient means that weight had a big impact on the mistake.

  • Create a new code cell and add the backpropagation function:
# Backpropagation
def backward(X, y_true, cache, W1, W2):
    Z1, A1, Z2, A2 = cache
    m = X.shape[0]

    # Output layer gradient
    dZ2 = A2 - y_true
    dW2 = A1.T.dot(dZ2) / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m

    # Hidden layer gradient
    dA1 = dZ2.dot(W2.T)
    dZ1 = dA1 * relu_derivative(Z1)
    dW1 = X.T.dot(dZ1) / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m

    return dW1, db1, dW2, db2

What does this code do?

The function works backward through the network in two stages:

  • Output layer gradient: dZ2 = A2 - y_true calculates how far off each prediction was. Then dW2 and db2 measure how much the output weights and biases contributed.
  • Hidden layer gradient: The error is propagated backward through W2 to the hidden layer. The relu_derivative(Z1) ensures that only neurons that were active during the forward pass receive blame.
  • Run the cell by pressing Shift+Enter.

You won't see any output yet. The gradients will be used in the next function to actually update the weights.

Seeing a NameError for relu_derivative?

  • The relu_derivative function was defined in Step 2. Make sure you ran that cell before running this one.
  • If your Colab runtime disconnected, click Runtime in the top menu, then Run all to re-execute every cell from the top.

Help me debug my backpropagation function in my neural network notebook.

Implement the weight update with gradient descent

Backpropagation told you which direction each weight should move. Gradient descent takes a small step in that direction. The learning rate controls how big each step is.

  • Create a new code cell and add the weight update function:
# Weight update
def update_weights(W1, b1, W2, b2, dW1, db1, dW2, db2, learning_rate):
    W1 = W1 - learning_rate * dW1
    b1 = b1 - learning_rate * db1
    W2 = W2 - learning_rate * dW2
    b2 = b2 - learning_rate * db2
    return W1, b1, W2, b2

What does this code do?

Each line subtracts a fraction of the gradient from the corresponding weight or bias. The learning_rate multiplier controls step size. Too large and the network overshoots the best values. Too small and training takes forever.

This single function call makes the network slightly less wrong. Repeat it thousands of times and the network becomes highly accurate.

  • Run the cell by pressing Shift+Enter.
  • Verify all three learning functions work together by adding a new cell and running this test:
# Test one complete learning cycle
predictions, cache = forward(X_train, W1, b1, W2, b2)
loss_before = compute_loss(y_train, predictions)

dW1, db1, dW2, db2 = backward(X_train, y_train, cache, W1, W2)
W1, b1, W2, b2 = update_weights(W1, b1, W2, b2, dW1, db1, dW2, db2, 0.1)

predictions_after, _ = forward(X_train, W1, b1, W2, b2)
loss_after = compute_loss(y_train, predictions_after)

print(f"Loss before update: {loss_before:.4f}")
print(f"Loss after update: {loss_after:.4f}")
print(f"Improvement: {loss_before - loss_after:.4f}")

You should see that the loss decreased after the update. Even a single learning cycle makes the network slightly better at predicting digits.

Loss increased instead of decreased?

  • Check that your backward function returns gradients in the correct order: dW1, db1, dW2, db2.
  • Make sure the subtraction in update_weights uses a minus sign. Adding gradients would make the network worse.
  • If you ran the test cell multiple times, the weights have already been updated. Use Runtime then Run all to reset from the beginning.

Help me figure out why my weight update is making the loss go up instead of down.

Why does this matter?

You just implemented the core learning mechanism that powers every AI system. From ChatGPT to self-driving cars, the fundamental loop is the same: make a prediction, measure the error, calculate gradients, update weights.

Frameworks like PyTorch and TensorFlow automate these steps. But now you understand exactly what they are doing under the hood.

✔️ Awesome, I've got everything!

Great. Make sure all your cells are saved before moving on.

ⓧ I'd like to double check the full code

Here is the complete code your notebook should contain up to this point:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# =============================================================
# STEP 1: Load and prepare the data
# =============================================================

# Load the digits dataset
digits = load_digits()
X = digits.data  # Shape: (1797, 64)
y = digits.target  # Shape: (1797,)

# Visualize some sample digits
fig, axes = plt.subplots(1, 5, figsize=(10, 3))
for i, ax in enumerate(axes):
    ax.imshow(digits.images[i], cmap='gray')
    ax.set_title(f"Label: {y[i]}")
    ax.axis('off')
plt.suptitle("Sample Handwritten Digits")
plt.show()

# Normalize pixel values to 0-1 range
X = X / 16.0

# One-hot encode the labels
def one_hot_encode(labels, num_classes=10):
    encoded = np.zeros((len(labels), num_classes))
    for i, label in enumerate(labels):
        encoded[i, label] = 1.0
    return encoded

y_encoded = one_hot_encode(y)

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y_encoded, test_size=0.2, random_state=42
)

print(f"Training samples: {X_train.shape[0]}")
print(f"Test samples: {X_test.shape[0]}")
print(f"Features per sample: {X_train.shape[1]}")

# =============================================================
# STEP 2: Build the neuron math
# =============================================================

# Activation functions
def relu(z):
    return np.maximum(0, z)

def relu_derivative(z):
    return (z > 0).astype(float)

def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)

# Initialize weights and biases
np.random.seed(42)

input_size = 64
hidden_size = 128
output_size = 10

# He initialization
W1 = np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size)
b1 = np.zeros((1, hidden_size))
W2 = np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size)
b2 = np.zeros((1, output_size))

# Forward pass
def forward(X, W1, b1, W2, b2):
    Z1 = X.dot(W1) + b1
    A1 = relu(Z1)
    Z2 = A1.dot(W2) + b2
    A2 = softmax(Z2)
    cache = (Z1, A1, Z2, A2)
    return A2, cache

# =============================================================
# STEP 3: Teach the network to learn
# =============================================================

# Cross-entropy loss
def compute_loss(y_true, y_pred):
    m = y_true.shape[0]
    # Clip to avoid log(0)
    y_pred_clipped = np.clip(y_pred, 1e-12, 1.0 - 1e-12)
    loss = -np.sum(y_true * np.log(y_pred_clipped)) / m
    return loss

# Backpropagation
def backward(X, y_true, cache, W1, W2):
    Z1, A1, Z2, A2 = cache
    m = X.shape[0]

    # Output layer gradient
    dZ2 = A2 - y_true
    dW2 = A1.T.dot(dZ2) / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m

    # Hidden layer gradient
    dA1 = dZ2.dot(W2.T)
    dZ1 = dA1 * relu_derivative(Z1)
    dW1 = X.T.dot(dZ1) / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m

    return dW1, db1, dW2, db2

# Weight update
def update_weights(W1, b1, W2, b2, dW1, db1, dW2, db2, learning_rate):
    W1 = W1 - learning_rate * dW1
    b1 = b1 - learning_rate * db1
    W2 = W2 - learning_rate * dW2
    b2 = b2 - learning_rate * db2
    return W1, b1, W2, b2

Your network now has everything it needs to learn. Next up, you'll wire these functions together into a training loop and watch the network get smarter with each repetition.

Train the Network and Watch It Learn

In the last step, you built all the learning machinery your network needs: a loss function to measure mistakes, backpropagation to trace where errors came from, and gradient descent to fix them. Your network now knows how to learn.

Now it is time to put these pieces together into a training loop that repeats the learn cycle thousands of times. You will watch the network go from random guessing to over 90% accuracy on handwritten digits.

In this step, get ready to:
  • Build the training loop that combines all learning steps into one repeating cycle.
  • Track the loss over time and visualize the network's learning curve.
  • Test the trained network on images it has never seen before.
Build the training loop

A training loop repeats four actions in sequence: make a prediction (forward pass), measure how wrong it was (loss), figure out which weights caused the error (backpropagation), and nudge those weights in a better direction (gradient descent). One pass through this cycle is called an epoch.

Your network will repeat this cycle 1,000 times. Each repetition makes it slightly less wrong.

  • Add a new code cell in your notebook by clicking + Code in the toolbar.
  • Paste the following training loop code into the new cell:
# =============================================================
# STEP 4: Train the network and watch it learn
# =============================================================

# Training loop
learning_rate = 0.1
epochs = 1000
losses = []

for epoch in range(epochs):
    # Forward pass
    predictions, cache = forward(X_train, W1, b1, W2, b2)

    # Compute loss
    loss = compute_loss(y_train, predictions)
    losses.append(loss)

    # Backward pass
    dW1, db1, dW2, db2 = backward(X_train, y_train, cache, W1, W2)

    # Update weights
    W1, b1, W2, b2 = update_weights(W1, b1, W2, b2, dW1, db1, dW2, db2, learning_rate)

    # Print progress
    if (epoch + 1) % 100 == 0:
        print(f"Epoch {epoch + 1}/{epochs}, Loss: {loss:.4f}")

What does this code do?

The learning_rate = 0.1 controls how big of a step the network takes when updating weights. Too large and it overshoots. Too small and it learns painfully slowly.

Inside the loop, each epoch runs the full cycle: forward pass to predict, loss calculation to measure error, backward pass to find gradients, and weight update to improve. The losses.append(loss) line records the loss at every epoch so you can visualize the learning curve later.

The print statement fires every 100 epochs so you can watch the loss decrease without flooding the output.

  • Run the cell by pressing Shift+Enter.
  • Watch the output. You should see 10 lines printed, one every 100 epochs, with the loss value decreasing from around 2-3 down to below 0.1.

Not seeing the loss decrease?

If the loss stays the same or increases, make sure you ran all previous cells in order. Colab cells must run sequentially because each one depends on variables defined earlier.

Try clicking Runtime in the top menu, then Run all to re-run everything from the start.

help me debug my training loop that isn't decreasing the loss

Track and visualize the learning progress

The printed numbers confirm the network is learning, but a chart makes the progress much easier to interpret. A loss curve shows how quickly the network improves and when it plateaus.

  • Add a new code cell and paste this code to plot the loss curve:
# Plot the loss curve
plt.figure(figsize=(10, 5))
plt.plot(losses)
plt.title("Training Loss Over Time")
plt.xlabel("Epoch")
plt.ylabel("Cross-Entropy Loss")
plt.grid(True)
plt.show()

What does this chart show?

The x-axis is the epoch number (0 to 1000). The y-axis is the cross-entropy loss. A steep drop at the start means the network learned quickly from its initial random state. The curve flattening out means the network has squeezed most of the improvement it can from this architecture.

  • Run the cell. You should see a curve that drops steeply in the first 100-200 epochs, then gradually levels off.

Now let's put a concrete number on how well the network learned its training data.

  • Add a new code cell and paste this code to calculate the training accuracy:
# Calculate training accuracy
train_predictions, _ = forward(X_train, W1, b1, W2, b2)
train_pred_labels = np.argmax(train_predictions, axis=1)
train_true_labels = np.argmax(y_train, axis=1)
train_accuracy = np.mean(train_pred_labels == train_true_labels)
print(f"\nTraining Accuracy: {train_accuracy * 100:.2f}%")

What does this code do?

The np.argmax function finds which of the 10 output probabilities is highest. That index IS the network's guess for which digit it sees.

Comparing predicted labels to true labels and taking the mean gives the percentage of correct guesses. Training accuracy should be well above 90%.

  • Run the cell. You should see a training accuracy above 90% printed in the output.

Accuracy below 90%?

If your training accuracy is significantly lower than expected, the most likely cause is that the training loop cell did not finish running. Check that you see all 10 printed epoch lines (Epoch 100 through Epoch 1000) in the training loop output above.

my neural network training accuracy is low, help me debug

Test on unseen data

Training accuracy tells you how well the network memorized its practice data. The real test is whether it can recognize digits it has never seen before. This is what the test set is for.

  • Add a new code cell and paste this code to calculate the test accuracy:
# Test on unseen data
test_predictions, _ = forward(X_test, W1, b1, W2, b2)
test_pred_labels = np.argmax(test_predictions, axis=1)
test_true_labels = np.argmax(y_test, axis=1)
test_accuracy = np.mean(test_pred_labels == test_true_labels)
print(f"Test Accuracy: {test_accuracy * 100:.2f}%")

Why test on separate data?

A network that only performs well on data it trained on has just memorized answers. Testing on unseen data proves it actually learned the underlying patterns of what digits look like. This ability to perform on new data is called generalization.

  • Run the cell. You should see a test accuracy above 90%, proving your network generalizes well to new digits.

Numbers are great, but seeing the network's predictions on actual digit images makes it feel real. Let's visualize 10 test predictions side by side.

  • Add a new code cell and paste this visualization code:
# Visualize some predictions
fig, axes = plt.subplots(2, 5, figsize=(12, 5))
for i, ax in enumerate(axes.flat):
    idx = i
    img = X_test[idx].reshape(8, 8)
    ax.imshow(img, cmap='gray')
    predicted = test_pred_labels[idx]
    actual = test_true_labels[idx]
    color = 'green' if predicted == actual else 'red'
    ax.set_title(f"Pred: {predicted}\nTrue: {actual}", color=color)
    ax.axis('off')
plt.suptitle("Network Predictions on Test Data (Green=Correct, Red=Wrong)")
plt.tight_layout()
plt.show()

What does this visualization show?

Each subplot shows one test digit image with the network's prediction and the true label. Green titles mean the network got it right. Red titles mean it guessed wrong.

Most titles should be green. The few red ones often make sense if you look closely. The digit images are only 8x8 pixels, so some are genuinely ambiguous even to a human.

  • Run the cell. You should see a grid of 10 digit images with color-coded predictions above each one.

Seeing a NameError for test_pred_labels?

This means the test accuracy cell above did not run successfully. The visualization depends on test_pred_labels and test_true_labels from that cell. Run the test accuracy cell first, then re-run this one.

help me fix a NameError in my Colab notebook

✔️ Awesome, I've got everything!

Great. Make sure all your cells are saved. Your notebook now contains a fully trained neural network that recognizes handwritten digits.

ⓧ I'd like to double check the full code

Here is the complete notebook code from start to finish. Compare your cells against this:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# =============================================================
# STEP 1: Load and prepare the data
# =============================================================

# Load the digits dataset
digits = load_digits()
X = digits.data  # Shape: (1797, 64)
y = digits.target  # Shape: (1797,)

# Visualize some sample digits
fig, axes = plt.subplots(1, 5, figsize=(10, 3))
for i, ax in enumerate(axes):
    ax.imshow(digits.images[i], cmap='gray')
    ax.set_title(f"Label: {y[i]}")
    ax.axis('off')
plt.suptitle("Sample Handwritten Digits")
plt.show()

# Normalize pixel values to 0-1 range
X = X / 16.0

# One-hot encode the labels
def one_hot_encode(labels, num_classes=10):
    encoded = np.zeros((len(labels), num_classes))
    for i, label in enumerate(labels):
        encoded[i, label] = 1.0
    return encoded

y_encoded = one_hot_encode(y)

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y_encoded, test_size=0.2, random_state=42
)

print(f"Training samples: {X_train.shape[0]}")
print(f"Test samples: {X_test.shape[0]}")
print(f"Features per sample: {X_train.shape[1]}")

# =============================================================
# STEP 2: Build the neuron math
# =============================================================

# Activation functions
def relu(z):
    return np.maximum(0, z)

def relu_derivative(z):
    return (z > 0).astype(float)

def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)

# Initialize weights and biases
np.random.seed(42)

input_size = 64
hidden_size = 128
output_size = 10

# He initialization
W1 = np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size)
b1 = np.zeros((1, hidden_size))
W2 = np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size)
b2 = np.zeros((1, output_size))

# Forward pass
def forward(X, W1, b1, W2, b2):
    Z1 = X.dot(W1) + b1
    A1 = relu(Z1)
    Z2 = A1.dot(W2) + b2
    A2 = softmax(Z2)
    cache = (Z1, A1, Z2, A2)
    return A2, cache

# =============================================================
# STEP 3: Teach the network to learn
# =============================================================

# Cross-entropy loss
def compute_loss(y_true, y_pred):
    m = y_true.shape[0]
    # Clip to avoid log(0)
    y_pred_clipped = np.clip(y_pred, 1e-12, 1.0 - 1e-12)
    loss = -np.sum(y_true * np.log(y_pred_clipped)) / m
    return loss

# Backpropagation
def backward(X, y_true, cache, W1, W2):
    Z1, A1, Z2, A2 = cache
    m = X.shape[0]

    # Output layer gradient
    dZ2 = A2 - y_true
    dW2 = A1.T.dot(dZ2) / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m

    # Hidden layer gradient
    dA1 = dZ2.dot(W2.T)
    dZ1 = dA1 * relu_derivative(Z1)
    dW1 = X.T.dot(dZ1) / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m

    return dW1, db1, dW2, db2

# Weight update
def update_weights(W1, b1, W2, b2, dW1, db1, dW2, db2, learning_rate):
    W1 = W1 - learning_rate * dW1
    b1 = b1 - learning_rate * db1
    W2 = W2 - learning_rate * dW2
    b2 = b2 - learning_rate * db2
    return W1, b1, W2, b2

# =============================================================
# STEP 4: Train the network and watch it learn
# =============================================================

# Training loop
learning_rate = 0.1
epochs = 1000
losses = []

for epoch in range(epochs):
    # Forward pass
    predictions, cache = forward(X_train, W1, b1, W2, b2)

    # Compute loss
    loss = compute_loss(y_train, predictions)
    losses.append(loss)

    # Backward pass
    dW1, db1, dW2, db2 = backward(X_train, y_train, cache, W1, W2)

    # Update weights
    W1, b1, W2, b2 = update_weights(W1, b1, W2, b2, dW1, db1, dW2, db2, learning_rate)

    # Print progress
    if (epoch + 1) % 100 == 0:
        print(f"Epoch {epoch + 1}/{epochs}, Loss: {loss:.4f}")

# Plot the loss curve
plt.figure(figsize=(10, 5))
plt.plot(losses)
plt.title("Training Loss Over Time")
plt.xlabel("Epoch")
plt.ylabel("Cross-Entropy Loss")
plt.grid(True)
plt.show()

# Calculate training accuracy
train_predictions, _ = forward(X_train, W1, b1, W2, b2)
train_pred_labels = np.argmax(train_predictions, axis=1)
train_true_labels = np.argmax(y_train, axis=1)
train_accuracy = np.mean(train_pred_labels == train_true_labels)
print(f"\nTraining Accuracy: {train_accuracy * 100:.2f}%")

# Test on unseen data
test_predictions, _ = forward(X_test, W1, b1, W2, b2)
test_pred_labels = np.argmax(test_predictions, axis=1)
test_true_labels = np.argmax(y_test, axis=1)
test_accuracy = np.mean(test_pred_labels == test_true_labels)
print(f"Test Accuracy: {test_accuracy * 100:.2f}%")

# Visualize some predictions
fig, axes = plt.subplots(2, 5, figsize=(12, 5))
for i, ax in enumerate(axes.flat):
    idx = i
    img = X_test[idx].reshape(8, 8)
    ax.imshow(img, cmap='gray')
    predicted = test_pred_labels[idx]
    actual = test_true_labels[idx]
    color = 'green' if predicted == actual else 'red'
    ax.set_title(f"Pred: {predicted}\nTrue: {actual}", color=color)
    ax.axis('off')
plt.suptitle("Network Predictions on Test Data (Green=Correct, Red=Wrong)")
plt.tight_layout()
plt.show()

Secret mission

Go Deeper

Your single-layer network already classifies digits with over 90% accuracy. But what happens when you stack another hidden layer on top? Deeper networks can learn more complex patterns. In this mission, you will add a second hidden layer and discover how network depth changes learning.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely in Google Colab, so there are no ongoing costs. Your notebook is saved to your Google Drive and uses no persistent compute.

Resources you used:

  • Google Colab notebook(s) saved to your Google Drive.

Keep everything running

No action needed. Choose this if you want to revisit your neural network, experiment with different architectures, or continue learning.

  • Your notebook stays in Google Drive indefinitely at no cost.
  • The Colab runtime shuts down automatically when you close the browser tab. No compute resources persist.
  • You can reopen the notebook anytime from drive.google.com and re-run the cells to train your network again.

Pause - I'll come back to this later

There is nothing to pause for this project. Google Colab runtimes shut down automatically when you close the tab or after a period of inactivity.

  • Close the Colab browser tab. The runtime stops on its own.
  • Your notebook remains saved in Google Drive. All your code and outputs are preserved.
  • When you return, open the notebook and click **Runtime**, then **Run all** to re-execute everything.

Delete - I don't want to use this again

Remove your notebook from Google Drive entirely. This cannot be undone after you empty the trash.

  • Open drive.google.com in your browser.
  • In the search bar at the top, type the name of your notebook (e.g. Neural Network From Scratch).
  • Right-click the notebook file and select **Move to trash**.
  • If you created a second notebook for the Secret Mission, repeat the same steps for that file.
  • To permanently delete, click **Trash** in the left sidebar, then click **Empty trash** at the top right.

Nice Work!

Nice Work!

You just built a neural network entirely from scratch. No frameworks, no magic. Every weight update, every gradient calculation, every prediction your network made came from code you wrote and understood.

You've learned how to:

  • Build a neural network from scratch using only NumPy, with no frameworks hiding the logic.
  • Implement the core components of all AI: forward propagation, loss functions, backpropagation, and gradient descent.
  • Train the network on real handwritten digit images and achieve over 90% accuracy on data it had never seen before.
  • Visualize how a neural network learns over time by plotting the loss curve and watching it decrease.
  • Secret Mission: Added a second hidden layer and observed how network depth affects learning.

Ready to quiz yourself?