Build a Neural Network from Scratch

Build a NumPy neural network that classifies a spiral dataset with 95%+ accuracy.

Introduction

30 Second Summary

Every time your phone recognizes a face or a streaming service recommends a show, a neural network is making decisions behind the scenes. But how does a pile of math actually "learn" to classify things it has never seen before?

In this project, you will build a neural network entirely from scratch using Python and NumPy. You'll train it on a tricky spiral dataset and create visualizations that reveal exactly how the network learns at each stage.

What You'll Build

A fully-functional neural network in a Jupyter Lab notebook that classifies a 3-class spiral dataset, with rich visualizations proving why each component exists.

By the end of this project, you'll have:

  • A spiral dataset generator that produces 3 interleaving classes of 2D points for training.
  • A multi-layer neural network built from pure NumPy with ReLU hidden layers, softmax output, and backpropagation.
  • A suite of visualizations including loss curves, decision boundaries, activation function comparisons, and hyperparameter experiments that show what the network does at each training stage.
  • Secret Mission: Animate the decision boundary evolving frame-by-frame across training epochs to watch the network "think" in real-time.

Are there any prerequisites?

No prior machine learning experience is needed. This guide includes setup for all required tools (Python, NumPy, Matplotlib, and Jupyter Lab). Basic familiarity with Python syntax is helpful but not required.

Before We Start

You're about to build a neural network from scratch. No TensorFlow, no PyTorch. Just NumPy, math, and your own understanding of how machines learn.

Before diving into the math, you need a workspace. In this step, you'll open Jupyter Lab and configure your notebook with the libraries you'll use throughout this project.

In this step, get ready to:
  • Open Jupyter Lab and create a new notebook.
  • Import NumPy, Matplotlib, and confirm your environment is working.
Open Jupyter Lab and create your notebook

Jupyter Lab is an interactive coding environment where you can write code in cells and run them one at a time. This makes it perfect for experimenting with neural networks because you can see results immediately after each piece.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows) and typing Terminal, then press Enter.
  • Launch Jupyter Lab by running this command:
jupyter lab

Jupyter Lab opens in your web browser automatically. You should see a Launcher tab with options for creating new files.

  • In the Launcher, click Python 3 under the Notebook section.
  • A new untitled notebook opens in a new tab.
  • In the left sidebar file browser, right-click Untitled.ipynb and select Rename.
  • Type neural_network_from_scratch.ipynb and press Enter.

Don't see the Launcher?

If Jupyter Lab opened without a Launcher tab, click the + button at the top of the file browser in the left sidebar to open a new Launcher.

help me open Jupyter Lab and see the Launcher

Import your core libraries

Every neural network you'll build in this project relies on two libraries. NumPy handles all the matrix math (the backbone of neural networks). Matplotlib creates the visualizations that show you what's happening inside the network.

  • Click into the first empty cell in your notebook and type the following code:
# Cell 1: Imports and Setup
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap

np.random.seed(42)

What does this code do?

  • import numpy as np imports NumPy and gives it the short alias np so you can type less throughout the project.
  • import matplotlib.pyplot as plt imports the plotting interface from Matplotlib.
  • from matplotlib.colors import ListedColormap imports a tool for creating custom color maps for your visualizations.
  • np.random.seed(42) locks the random number generator so your results are reproducible every time you run the notebook.
  • Run the cell by pressing Shift+Enter.

If everything worked, the cell runs silently with no output and a new empty cell appears below.

  • Confirm your library versions by typing this in the new cell and pressing Shift+Enter:
print(np.__version__)
print(plt.matplotlib.__version__)

You should see two version numbers printed, confirming both libraries loaded successfully.

Seeing a ModuleNotFoundError?

This means the library isn't installed in the Python environment Jupyter Lab is using. Open a new terminal in Jupyter Lab (File > New > Terminal) and run pip install numpy matplotlib.

help me fix a ModuleNotFoundError when importing numpy or matplotlib in Jupyter Lab

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook with Cmd+S (macOS) or Ctrl+S (Windows).

ⓧ I'd like to double check the full code

Your first cell should look exactly like this:

# Cell 1: Imports and Setup
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap

np.random.seed(42)

Your workspace is ready. Next up, you'll generate the spiral dataset that will challenge your neural network to learn curved, non-linear boundaries.

Generate and Visualize the Spiral Dataset

Your notebook is ready with NumPy and Matplotlib imported. Now you need training data for your neural network to learn from.

In this step, you will generate a spiral dataset with three interleaving classes of 2D points. This dataset is specifically designed to be impossible for a simple linear model to solve, which motivates every piece of neural network architecture you will build in the next steps.

In this step, get ready to:
  • Define a spiral data generator function that produces three classes of points.
  • Visualize the spirals to see why this dataset demands a non-linear solution.
Create the spiral data generator

A spiral dataset creates three groups of points that wind around each other like arms of a galaxy. Each class spirals outward from the center, making it impossible to draw a single straight line that separates them.

  • Create a new code cell below Cell 1 and enter this function definition:
# Cell 2: Generate Spiral Dataset
def generate_spiral_data(points_per_class=100, num_classes=3):
    """Generate a spiral dataset with N points per class."""
    # Create empty arrays to hold coordinates and labels
    X = np.zeros((points_per_class * num_classes, 2))
    y = np.zeros(points_per_class * num_classes, dtype=int)
    
    for class_idx in range(num_classes):
        start_idx = points_per_class * class_idx
        end_idx = points_per_class * (class_idx + 1)
        
        # r increases linearly: points start near center, end at edge
        r = np.linspace(0.0, 1, points_per_class)
        # theta sweeps each class through a unique angular range, with noise
        theta = np.linspace(
            class_idx * 4, (class_idx + 1) * 4, points_per_class
        ) + np.random.randn(points_per_class) * 0.2
        
        # Convert polar (r, theta) to Cartesian (x, y)
        X[start_idx:end_idx, 0] = r * np.sin(theta)
        X[start_idx:end_idx, 1] = r * np.cos(theta)
        y[start_idx:end_idx] = class_idx
    
    return X, y

# Generate the dataset
X, y = generate_spiral_data()

What does this code do?

The function creates 100 points for each of 3 classes (300 points total). Each class gets its own angular range so the spirals don't overlap perfectly.

`r = np.linspace(0.0, 1, points_per_class)` makes points start near the origin and move outward. `np.random.randn(points_per_class) * 0.2` adds Gaussian noise so the spirals are fuzzy and realistic, not perfectly smooth.

The result is X with shape (300, 2) holding the x/y coordinates, and y with shape (300,) holding class labels 0, 1, or 2.

You won't see any output yet. The next section adds a scatter plot to make this data visible.

Visualize the spiral data

Seeing the raw data is essential before building a model. A scatter plot reveals the pattern your neural network must learn to separate.

  • Below the line X, y = generate_spiral_data(), add this visualization code in the same cell:
# Visualize the spiral data
plt.figure(figsize=(8, 8))
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20)
plt.title('Spiral Dataset - 3 Classes')
plt.xlabel('x1')
plt.ylabel('x2')
plt.axis('equal')
plt.show()

What does this visualization show?

`plt.scatter` plots each point's x1 and x2 coordinates. The c=y argument colors each point by its class label, and ListedColormap assigns red, teal, and blue to classes 0, 1, and 2.

The key insight is this: no single straight line can separate the three spirals. Even stacking multiple linear layers without an activation function collapses to one linear transformation (W2 @ (W1 @ X) = (W2 @ W1) @ X). This is why neural networks need non-linear activation functions, which you will build in the next step.

  • Run the cell by pressing Shift+Enter.
  • You should see a square scatter plot with three colorful spirals (red, teal, and blue) interleaving around the center.

Don't see the spiral plot?

  • Make sure you ran Cell 1 first. Cell 2 depends on numpy, matplotlib, and ListedColormap being imported.
  • If you see a NameError for np or plt, go back to Cell 1 and press Shift+Enter to re-run it.
  • Check that the function code and visualization are both in the same cell, with the visualization below the X, y = generate_spiral_data() line.

Why can't a linear model solve this spiral?

Why spirals instead of simpler data?

Many tutorials use linearly separable data (two blobs that a single line can divide). That hides the real value of neural networks.

Spirals force non-linearity. Every component you build from here on (ReLU, backpropagation, He initialization) exists because simple lines fail on this exact shape. The spiral is your visual proof of WHY each formula matters.

✔️ Awesome, I've got everything!

Great. Double check that you've saved your notebook and the spiral scatter plot is visible in the cell output.

ⓧ I'd like to double check the full code

Here is the complete code for your notebook so far (Cell 1 and Cell 2):

# Cell 1: Imports and Setup
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap

np.random.seed(42)

# Cell 2: Generate Spiral Dataset
def generate_spiral_data(points_per_class=100, num_classes=3):
    """Generate a spiral dataset with N points per class."""
    X = np.zeros((points_per_class * num_classes, 2))
    y = np.zeros(points_per_class * num_classes, dtype=int)
    
    for class_idx in range(num_classes):
        start_idx = points_per_class * class_idx
        end_idx = points_per_class * (class_idx + 1)
        
        r = np.linspace(0.0, 1, points_per_class)
        theta = np.linspace(
            class_idx * 4, (class_idx + 1) * 4, points_per_class
        ) + np.random.randn(points_per_class) * 0.2
        
        X[start_idx:end_idx, 0] = r * np.sin(theta)
        X[start_idx:end_idx, 1] = r * np.cos(theta)
        y[start_idx:end_idx] = class_idx
    
    return X, y

X, y = generate_spiral_data()

plt.figure(figsize=(8, 8))
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20)
plt.title('Spiral Dataset - 3 Classes')
plt.xlabel('x1')
plt.ylabel('x2')
plt.axis('equal')
plt.show()

Your spiral data is ready. Next up, you will prove exactly why a linear model fails on this data, and build the activation functions that fix it.

Prove Why Non-Linearity Matters

Your spiral dataset revealed the challenge: three interleaving classes of points that no single straight line can separate. The question now is whether a basic model can learn these curved boundaries on its own.

In this step, you'll prove that a linear model fundamentally cannot solve this problem. Then you'll explore the activation functions that give neural networks the power to learn curves instead of just straight lines.

In this step, get ready to:
  • Train a linear model and visualize its straight-line decision boundaries.
  • Compare sigmoid, tanh, and ReLU activation functions and their derivatives.
  • Understand why ReLU is the preferred choice for hidden layers.
Build and train a linear-only model

The simplest possible classifier multiplies inputs by weights and adds a bias: Z = X @ W + b. There is no activation function bending the output.

Mathematically, stacking multiple linear layers collapses into a single linear transformation. W2 @ (W1 @ X) is equivalent to (W2 @ W1) @ X, which is just another matrix multiply.

  • Create a new cell below Cell 2. Add this code to start Cell 3:
# Cell 3: Demonstrate Linear Model Failure
def linear_forward(X, W, b):
    """A model with NO activation function."""
    return X @ W + b

def softmax(Z):
    """Numerically stable softmax."""
    exp_Z = np.exp(Z - np.max(Z, axis=1, keepdims=True))
    return exp_Z / np.sum(exp_Z, axis=1, keepdims=True)

# Train a linear model briefly
W_linear = np.random.randn(2, 3) * 0.01
b_linear = np.zeros((1, 3))

num_samples = X.shape[0]
y_one_hot = np.zeros((num_samples, 3))
y_one_hot[np.arange(num_samples), y] = 1

for i in range(1000):
    scores = linear_forward(X, W_linear, b_linear)
    probs = softmax(scores)
    
    dscores = (probs - y_one_hot) / num_samples
    dW = X.T @ dscores
    db = np.sum(dscores, axis=0, keepdims=True)
    
    W_linear -= 1.0 * dW
    b_linear -= 1.0 * db

What does this code do?

  • linear_forward computes raw scores with no activation. It is a pure matrix multiply plus bias.
  • softmax converts raw scores into probabilities that sum to 1 across classes. The np.max subtraction prevents numerical overflow from large exponentials.
  • The training loop runs 1000 iterations of gradient descent: compute predictions, measure error, nudge weights to reduce that error.

You won't see any output yet. The next chunk adds the visualization that reveals what this model learned.

  • Continue in the same cell by adding this code below what you just typed:
# Plot linear decision boundary
def plot_decision_boundary(X, y, predict_fn, title):
    """Plot the decision boundary of a classifier."""
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    
    xx, yy = np.meshgrid(
        np.linspace(x_min, x_max, 200),
        np.linspace(y_min, y_max, 200)
    )
    grid_points = np.c_[xx.ravel(), yy.ravel()]
    Z = predict_fn(grid_points)
    Z = Z.reshape(xx.shape)
    
    plt.figure(figsize=(8, 8))
    plt.contourf(xx, yy, Z, alpha=0.3, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']))
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20, edgecolors='k', linewidth=0.5)
    plt.title(title)
    plt.xlabel('x1')
    plt.ylabel('x2')
    plt.axis('equal')
    plt.show()

def linear_predict(X_input):
    scores = linear_forward(X_input, W_linear, b_linear)
    return np.argmax(scores, axis=1)

plot_decision_boundary(X, y, linear_predict, 'Linear Model: Can Only Draw Straight Lines')

What does this code do?

  • plot_decision_boundary creates a dense grid of points across the input space, predicts a class for each point, and colors the regions. This shows you the boundaries the model uses to separate classes.
  • linear_predict feeds any input through the trained linear model and returns the predicted class (0, 1, or 2).
  • The final line calls the function to render the decision boundary plot.
  • Run the cell by pressing Shift+Enter.
  • You should see a plot titled Linear Model: Can Only Draw Straight Lines with colored regions divided by straight lines that clearly miss the spiral pattern.

Don't see the decision boundary plot?

  • Make sure you ran Cell 1 and Cell 2 first. The linear model depends on X and y being defined from Cell 2.
  • Check for indentation errors inside the for loop. Every line inside the loop must be indented one level.
  • If you see NameError: name 'ListedColormap' is not defined, make sure Cell 1 imported it from matplotlib.colors.

Help me debug my linear model decision boundary plot

Why does the linear model fail?

Look at the plot. The decision boundaries are perfectly straight lines. No matter how long you train, a linear model can only carve the plane into wedge-shaped regions.

Spirals are curves. Straight lines cannot trace curves. This is why activation functions exist: they introduce non-linearity so the network can bend its boundaries to fit complex shapes.

Compare activation functions visually

Three popular activation functions exist: sigmoid, tanh, and ReLU. The key difference is what happens to gradients as they flow backward through layers during training.

Sigmoid's maximum derivative is 0.25. Each layer multiplies gradients by at most 0.25 during backpropagation. With several layers, gradients shrink toward zero and the network stops learning. ReLU's derivative is either 0 or 1, so gradients pass through active neurons unchanged.

  • Create a new cell below Cell 3. Add this code for Cell 4:
# Cell 4: Activation Function Comparison
fig, axes = plt.subplots(2, 3, figsize=(15, 8))

z = np.linspace(-5, 5, 200)

# Sigmoid
sigmoid_out = 1 / (1 + np.exp(-z))
sigmoid_deriv = sigmoid_out * (1 - sigmoid_out)
axes[0, 0].plot(z, sigmoid_out, 'b-', linewidth=2)
axes[0, 0].set_title('Sigmoid')
axes[0, 0].set_ylabel('Output')
axes[0, 0].grid(True, alpha=0.3)
axes[1, 0].plot(z, sigmoid_deriv, 'r-', linewidth=2)
axes[1, 0].set_title('Sigmoid Derivative (max=0.25!)')
axes[1, 0].set_ylabel('Gradient')
axes[1, 0].set_xlabel('Input z')
axes[1, 0].grid(True, alpha=0.3)

# Tanh
tanh_out = np.tanh(z)
tanh_deriv = 1 - tanh_out**2
axes[0, 1].plot(z, tanh_out, 'b-', linewidth=2)
axes[0, 1].set_title('Tanh')
axes[0, 1].grid(True, alpha=0.3)
axes[1, 1].plot(z, tanh_deriv, 'r-', linewidth=2)
axes[1, 1].set_title('Tanh Derivative (max=1.0)')
axes[1, 1].set_xlabel('Input z')
axes[1, 1].grid(True, alpha=0.3)

# ReLU
relu_out = np.maximum(0, z)
relu_deriv = (z > 0).astype(float)
axes[0, 2].plot(z, relu_out, 'b-', linewidth=2)
axes[0, 2].set_title('ReLU')
axes[0, 2].grid(True, alpha=0.3)
axes[1, 2].plot(z, relu_deriv, 'r-', linewidth=2)
axes[1, 2].set_title('ReLU Derivative (0 or 1)')
axes[1, 2].set_xlabel('Input z')
axes[1, 2].grid(True, alpha=0.3)

plt.suptitle('Why ReLU? Sigmoid/Tanh derivatives shrink gradients. ReLU passes them through unchanged.', fontsize=12, y=1.02)
plt.tight_layout()
plt.show()

What does this visualization show?

  • The top row shows each function's output shape. Sigmoid squishes values between 0 and 1. Tanh squishes between -1 and 1. ReLU passes positives through unchanged and zeros out negatives.
  • The bottom row shows each function's derivative. This is what matters for learning. Sigmoid's derivative peaks at 0.25, meaning each layer shrinks gradients by 75% at best.
  • ReLU's derivative is a flat 1 for positive inputs. Gradients flow backward through the network at full strength, letting deep layers learn just as fast as shallow ones.
  • Run the cell by pressing Shift+Enter.
  • You should see a 2x3 grid of plots. The top row shows the three activation functions. The bottom row shows their derivatives. Notice how ReLU's derivative is a clean step function compared to the squished curves of sigmoid and tanh.

Plots overlapping or cut off?

  • If subplot titles are overlapping, make sure you included plt.tight_layout() before plt.show().
  • If the suptitle is cut off at the top, the y=1.02 parameter pushes it above the figure. This is expected behavior. Scroll up slightly if needed.

Help me fix my activation function comparison plot

Why ReLU for hidden layers?

The vanishing gradient problem killed early deep networks. Sigmoid and tanh squish their derivatives so small that after a few layers, the gradient signal disappears.

ReLU solves this with elegant simplicity: if the input is positive, pass the gradient through at full strength. If negative, block it completely. This on/off behavior keeps learning signals alive across many layers.

✔️ Awesome, I've got everything!

Great. Make sure both Cell 3 and Cell 4 are saved and showing their plots correctly.

ⓧ I'd like to double check the full code

# Cell 1: Imports and Setup
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap

np.random.seed(42)

# Cell 2: Generate Spiral Dataset
def generate_spiral_data(points_per_class=100, num_classes=3):
    """Generate a spiral dataset with N points per class."""
    X = np.zeros((points_per_class * num_classes, 2))
    y = np.zeros(points_per_class * num_classes, dtype=int)
    
    for class_idx in range(num_classes):
        start_idx = points_per_class * class_idx
        end_idx = points_per_class * (class_idx + 1)
        
        r = np.linspace(0.0, 1, points_per_class)
        theta = np.linspace(
            class_idx * 4, (class_idx + 1) * 4, points_per_class
        ) + np.random.randn(points_per_class) * 0.2
        
        X[start_idx:end_idx, 0] = r * np.sin(theta)
        X[start_idx:end_idx, 1] = r * np.cos(theta)
        y[start_idx:end_idx] = class_idx
    
    return X, y

X, y = generate_spiral_data()

plt.figure(figsize=(8, 8))
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20)
plt.title('Spiral Dataset - 3 Classes')
plt.xlabel('x1')
plt.ylabel('x2')
plt.axis('equal')
plt.show()

# Cell 3: Demonstrate Linear Model Failure
def linear_forward(X, W, b):
    """A model with NO activation function."""
    return X @ W + b

def softmax(Z):
    """Numerically stable softmax."""
    exp_Z = np.exp(Z - np.max(Z, axis=1, keepdims=True))
    return exp_Z / np.sum(exp_Z, axis=1, keepdims=True)

W_linear = np.random.randn(2, 3) * 0.01
b_linear = np.zeros((1, 3))

num_samples = X.shape[0]
y_one_hot = np.zeros((num_samples, 3))
y_one_hot[np.arange(num_samples), y] = 1

for i in range(1000):
    scores = linear_forward(X, W_linear, b_linear)
    probs = softmax(scores)
    
    dscores = (probs - y_one_hot) / num_samples
    dW = X.T @ dscores
    db = np.sum(dscores, axis=0, keepdims=True)
    
    W_linear -= 1.0 * dW
    b_linear -= 1.0 * db

def plot_decision_boundary(X, y, predict_fn, title):
    """Plot the decision boundary of a classifier."""
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    
    xx, yy = np.meshgrid(
        np.linspace(x_min, x_max, 200),
        np.linspace(y_min, y_max, 200)
    )
    grid_points = np.c_[xx.ravel(), yy.ravel()]
    Z = predict_fn(grid_points)
    Z = Z.reshape(xx.shape)
    
    plt.figure(figsize=(8, 8))
    plt.contourf(xx, yy, Z, alpha=0.3, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']))
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20, edgecolors='k', linewidth=0.5)
    plt.title(title)
    plt.xlabel('x1')
    plt.ylabel('x2')
    plt.axis('equal')
    plt.show()

def linear_predict(X_input):
    scores = linear_forward(X_input, W_linear, b_linear)
    return np.argmax(scores, axis=1)

plot_decision_boundary(X, y, linear_predict, 'Linear Model: Can Only Draw Straight Lines')

# Cell 4: Activation Function Comparison
fig, axes = plt.subplots(2, 3, figsize=(15, 8))

z = np.linspace(-5, 5, 200)

sigmoid_out = 1 / (1 + np.exp(-z))
sigmoid_deriv = sigmoid_out * (1 - sigmoid_out)
axes[0, 0].plot(z, sigmoid_out, 'b-', linewidth=2)
axes[0, 0].set_title('Sigmoid')
axes[0, 0].set_ylabel('Output')
axes[0, 0].grid(True, alpha=0.3)
axes[1, 0].plot(z, sigmoid_deriv, 'r-', linewidth=2)
axes[1, 0].set_title('Sigmoid Derivative (max=0.25!)')
axes[1, 0].set_ylabel('Gradient')
axes[1, 0].set_xlabel('Input z')
axes[1, 0].grid(True, alpha=0.3)

tanh_out = np.tanh(z)
tanh_deriv = 1 - tanh_out**2
axes[0, 1].plot(z, tanh_out, 'b-', linewidth=2)
axes[0, 1].set_title('Tanh')
axes[0, 1].grid(True, alpha=0.3)
axes[1, 1].plot(z, tanh_deriv, 'r-', linewidth=2)
axes[1, 1].set_title('Tanh Derivative (max=1.0)')
axes[1, 1].set_xlabel('Input z')
axes[1, 1].grid(True, alpha=0.3)

relu_out = np.maximum(0, z)
relu_deriv = (z > 0).astype(float)
axes[0, 2].plot(z, relu_out, 'b-', linewidth=2)
axes[0, 2].set_title('ReLU')
axes[0, 2].grid(True, alpha=0.3)
axes[1, 2].plot(z, relu_deriv, 'r-', linewidth=2)
axes[1, 2].set_title('ReLU Derivative (0 or 1)')
axes[1, 2].set_xlabel('Input z')
axes[1, 2].grid(True, alpha=0.3)

plt.suptitle('Why ReLU? Sigmoid/Tanh derivatives shrink gradients. ReLU passes them through unchanged.', fontsize=12, y=1.02)
plt.tight_layout()
plt.show()

You now have visual proof that linear models are fundamentally limited. Next up, you'll build a full neural network with ReLU activations and see those straight lines transform into the smooth curves your spiral data demands.

Build the Neural Network

You proved that a linear model cannot separate spirals. You also saw that ReLU passes gradients through unchanged, making it ideal for hidden layers.

Now it is time to assemble the actual neural network. You will implement every function the network needs to learn: a forward pass that produces predictions, a loss function that measures errors, and backpropagation that computes how to fix those errors.

In this step, get ready to:
  • Implement the neural network's forward and backward pass functions.
  • Visualize why He weight initialization prevents dead neurons.
  • Compare cross-entropy loss against MSE to see why it learns faster.
Implement the neural network functions

The network needs six functions working together. Activation and initialization set up the network's structure. The forward pass moves data through it. Loss, backpropagation, and parameter updates form the learning engine.

  • Create a new cell below Cell 4 in your notebook.
  • Add a comment and the first two functions (ReLU activation and He parameter initialization) by pasting this code:
# Cell 5: Neural Network Implementation
def relu(Z):
    """ReLU activation: max(0, Z)."""
    return np.maximum(0, Z)

def initialize_parameters(input_size, hidden_size, output_size):
    """He initialization for weights, zeros for biases."""
    params = {
        'W1': np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size),
        'b1': np.zeros((1, hidden_size)),
        'W2': np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size),
        'b2': np.zeros((1, output_size))
    }
    return params

What does this code do?

relu(Z) is a reusable version of the ReLU activation. It returns the input if positive, or zero otherwise.

initialize_parameters uses He initialization. The formula np.sqrt(2.0 / fan_in) scales weights so that the variance of activations stays stable across layers. Without this scaling, activations either explode or collapse to zero as data passes through the network.

  • Below the previous code in the same cell, add the forward pass function:
def forward(X, params):
    """Forward pass: input -> hidden (ReLU) -> output (softmax)."""
    Z1 = X @ params['W1'] + params['b1']
    A1 = relu(Z1)
    Z2 = A1 @ params['W2'] + params['b2']
    A2 = softmax(Z2)

    cache = {'Z1': Z1, 'A1': A1, 'Z2': Z2, 'A2': A2}
    return A2, cache

What is the forward pass doing?

Data flows through two layers. The hidden layer computes Z1 = X @ W1 + b1 (a linear transformation), then applies ReLU to get A1. The output layer transforms A1 into raw scores, then softmax converts those scores into probabilities that sum to 1 across all 3 classes.

The cache dictionary stores intermediate values. Backpropagation needs these to compute gradients without redoing the forward pass.

  • Below the forward function in the same cell, add the loss function, backpropagation, and parameter update functions:
def compute_loss(A2, Y_one_hot):
    """Cross-entropy loss."""
    m = Y_one_hot.shape[0]
    loss = -np.sum(Y_one_hot * np.log(A2 + 1e-8)) / m
    return loss

def backward(X, Y_one_hot, params, cache):
    """Backpropagation: compute gradients for all parameters."""
    m = X.shape[0]
    A1 = cache['A1']
    A2 = cache['A2']
    Z1 = cache['Z1']

    # Output layer gradients
    dZ2 = A2 - Y_one_hot
    dW2 = A1.T @ dZ2 / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m

    # Hidden layer gradients
    dA1 = dZ2 @ params['W2'].T
    dZ1 = dA1 * (Z1 > 0)  # ReLU derivative
    dW1 = X.T @ dZ1 / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m

    grads = {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}
    return grads

def update_parameters(params, grads, learning_rate):
    """Gradient descent update."""
    params['W1'] -= learning_rate * grads['dW1']
    params['b1'] -= learning_rate * grads['db1']
    params['W2'] -= learning_rate * grads['dW2']
    params['b2'] -= learning_rate * grads['db2']
    return params

How does backpropagation work here?

compute_loss uses cross-entropy loss. The 1e-8 prevents taking the log of zero, which would produce NaN.

backward applies the chain rule layer by layer. The output gradient dZ2 = A2 - Y_one_hot is beautifully simple because softmax + cross-entropy simplifies to "prediction minus truth." Then it propagates backward through W2 and the ReLU derivative (Z1 > 0) to reach W1.

update_parameters applies gradient descent: each parameter moves in the opposite direction of its gradient, scaled by the learning rate.

  • Run Cell 5 by pressing Shift+Enter.
  • The cell produces no output. It only defines functions. If you see no error, the functions are ready to use.

Seeing a NameError or SyntaxError?

  • If you see NameError: name 'np' is not defined, go back and re-run Cell 1 (the imports cell) first.
  • If you see NameError: name 'softmax' is not defined, re-run Cell 3 where softmax is defined.
  • Check that your indentation uses consistent spaces (not a mix of tabs and spaces) inside each function body.

Help me debug my Cell 5 error

✔️ Awesome, I've got everything!

Great. Make sure you have saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here is the complete Cell 5. Make sure your code matches this exactly:

# Cell 5: Neural Network Implementation
def relu(Z):
    """ReLU activation: max(0, Z)."""
    return np.maximum(0, Z)

def initialize_parameters(input_size, hidden_size, output_size):
    """He initialization for weights, zeros for biases."""
    params = {
        'W1': np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size),
        'b1': np.zeros((1, hidden_size)),
        'W2': np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size),
        'b2': np.zeros((1, output_size))
    }
    return params

def forward(X, params):
    """Forward pass: input -> hidden (ReLU) -> output (softmax)."""
    Z1 = X @ params['W1'] + params['b1']
    A1 = relu(Z1)
    Z2 = A1 @ params['W2'] + params['b2']
    A2 = softmax(Z2)
    cache = {'Z1': Z1, 'A1': A1, 'Z2': Z2, 'A2': A2}
    return A2, cache

def compute_loss(A2, Y_one_hot):
    """Cross-entropy loss."""
    m = Y_one_hot.shape[0]
    loss = -np.sum(Y_one_hot * np.log(A2 + 1e-8)) / m
    return loss

def backward(X, Y_one_hot, params, cache):
    """Backpropagation: compute gradients for all parameters."""
    m = X.shape[0]
    A1 = cache['A1']
    A2 = cache['A2']
    Z1 = cache['Z1']
    dZ2 = A2 - Y_one_hot
    dW2 = A1.T @ dZ2 / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m
    dA1 = dZ2 @ params['W2'].T
    dZ1 = dA1 * (Z1 > 0)
    dW1 = X.T @ dZ1 / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m
    grads = {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}
    return grads

def update_parameters(params, grads, learning_rate):
    """Gradient descent update."""
    params['W1'] -= learning_rate * grads['dW1']
    params['b1'] -= learning_rate * grads['db1']
    params['W2'] -= learning_rate * grads['dW2']
    params['b2'] -= learning_rate * grads['db2']
    return params
Visualize why weight initialization matters

Choosing the right initial weight values determines whether your network can learn at all. If weights start too small, activations collapse to near-zero and gradients vanish. He initialization solves this by scaling weights to keep activations healthy.

  • Create a new cell below Cell 5.
  • Compare tiny random weights against He initialization by pasting this code:
# Cell 6: Visualize He Initialization vs Tiny Weights
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Tiny weights
params_tiny = {
    'W1': np.random.randn(2, 100) * 0.01,
    'b1': np.zeros((1, 100)),
    'W2': np.random.randn(100, 3) * 0.01,
    'b2': np.zeros((1, 3))
}
_, cache_tiny = forward(X, params_tiny)
axes[0].hist(cache_tiny['A1'].ravel(), bins=50, color='red', alpha=0.7)
axes[0].set_title('Tiny Init (0.01): Hidden Activations Near Zero')
axes[0].set_xlabel('Activation Value')
axes[0].set_ylabel('Frequency')

# He initialization
params_he = initialize_parameters(2, 100, 3)
_, cache_he = forward(X, params_he)
axes[1].hist(cache_he['A1'].ravel(), bins=50, color='green', alpha=0.7)
axes[1].set_title('He Init: Healthy Spread of Activations')
axes[1].set_xlabel('Activation Value')
axes[1].set_ylabel('Frequency')

plt.suptitle('Why He Initialization? Tiny weights produce dead neurons. He keeps signals alive.', fontsize=12)
plt.tight_layout()
plt.show()

What do these histograms show?

The left histogram shows what happens with tiny weights (0.01 * randn): almost all hidden activations are crammed near zero. When activations are near-zero, the gradients flowing backward are also near-zero. The network cannot learn.

The right histogram shows He initialization: activations spread out across a healthy range. This means gradients have a strong signal to work with during backpropagation.

  • Run Cell 6 by pressing Shift+Enter.
  • You should see two side-by-side histograms. The left one (red) is tightly clustered near zero. The right one (green) is spread out across a wide range of positive values.

Histograms look identical or empty?

  • Make sure Cell 5 ran successfully before Cell 6. Cell 6 uses the forward and initialize_parameters functions defined there.
  • Confirm you are using cache_tiny['A1'] (the hidden layer activations), not cache_tiny['A2'] (the output probabilities).

Help me fix my initialization visualization

Compare cross-entropy loss against MSE

For classification tasks, the choice of loss function matters more than you might expect. Mean Squared Error (MSE) is the default for regression, but it produces weak gradients when the model is confidently wrong. Cross-entropy punishes confident mistakes far more harshly, which means the network corrects big errors faster.

  • Create a new cell below Cell 6.
  • Plot both loss functions side by side to see the difference visually by pasting this code:
# Cell 7: Cross-Entropy vs MSE Comparison
fig, ax = plt.subplots(1, 1, figsize=(10, 6))

predicted_prob = np.linspace(0.01, 0.99, 100)
true_label = 1.0

ce_loss = -true_label * np.log(predicted_prob)
mse_loss = (true_label - predicted_prob) ** 2

ce_grad = -true_label / predicted_prob
mse_grad = -2 * (true_label - predicted_prob)

ax.plot(predicted_prob, ce_loss, 'b-', linewidth=2, label='Cross-Entropy Loss')
ax.plot(predicted_prob, mse_loss, 'r--', linewidth=2, label='MSE Loss')
ax.set_xlabel('Predicted Probability for True Class')
ax.set_ylabel('Loss')
ax.set_title('Why Cross-Entropy? It penalizes confident wrong predictions much more harshly.')
ax.legend(fontsize=12)
ax.grid(True, alpha=0.3)
ax.set_xlim(0, 1)
ax.set_ylim(0, 5)
plt.tight_layout()
plt.show()

What does this plot reveal?

The x-axis shows how confident the model is in the correct class (1.0 = perfect, 0.01 = completely wrong). Look at the left side of the chart where the model is confidently wrong.

Cross-entropy (blue) shoots up steeply, creating a massive gradient that forces rapid correction. MSE (red dashed) rises gently, producing a weak gradient. In practice, this means a cross-entropy-trained network fixes its worst mistakes much faster.

  • Run Cell 7 by pressing Shift+Enter.
  • You should see a single plot with two curves. The blue cross-entropy line rises steeply on the left side (near 0.0 predicted probability), while the red MSE line stays relatively flat.

Plot looks empty or curves are flat?

  • Verify that predicted_prob starts at 0.01, not 0.0. Starting at zero causes np.log(0) which produces infinity.
  • Check that true_label = 1.0. If it is 0.0, the cross-entropy formula will produce all zeros.

Help me fix my loss comparison plot

Your neural network is fully built. Every function needed for learning is in place. Next, you will wire them together into a training loop and watch the network learn to classify spirals in real time.

Train the Network and Evaluate Results

You have all the components of a neural network sitting in your notebook. forward pushes data through the network, backward computes gradients, and update_parameters adjusts the weights.

Now it is time to wire these pieces into a training loop that runs for hundreds of epochs and actually learns the spiral pattern. You will also build visualizations that prove the network is learning and show exactly what it has figured out.

In this step, get ready to:
  • Build the full training loop that combines all components and records loss, accuracy, and parameter snapshots.
  • Plot learning curves showing loss and accuracy over epochs.
  • Visualize the final decision boundary to see the curved regions the network learned.
Implement the full training loop

A training loop repeats the same cycle over and over: forward pass, compute loss, backpropagation, update weights. Each repetition is called an epoch. The more epochs you run, the more chances the network has to correct its mistakes.

  • Create a new cell in your notebook (Cell 8) and add the training function by pasting this code:
# Cell 8: Training Loop
def train(X, y, hidden_size=100, learning_rate=1.0, epochs=1000, verbose=True):
    num_classes = len(np.unique(y))
    m = X.shape[0]
    Y_one_hot = np.zeros((m, num_classes))
    Y_one_hot[np.arange(m), y] = 1
    params = initialize_parameters(X.shape[1], hidden_size, num_classes)
    history = {'loss': [], 'accuracy': [], 'params_snapshots': []}
    for epoch in range(epochs):
        A2, cache = forward(X, params)
        loss = compute_loss(A2, Y_one_hot)
        predictions = np.argmax(A2, axis=1)
        accuracy = np.mean(predictions == y)
        history['loss'].append(loss)
        history['accuracy'].append(accuracy)
        if epoch % 20 == 0:
            snapshot = {k: v.copy() for k, v in params.items()}
            history['params_snapshots'].append((epoch, snapshot))
        grads = backward(X, Y_one_hot, params, cache)
        params = update_parameters(params, grads, learning_rate)
        if verbose and epoch % 100 == 0:
            print(f"Epoch {epoch:4d} | Loss: {loss:.4f} | Accuracy: {accuracy:.4f}")
    return params, history

params_trained, history = train(X, y, hidden_size=100, learning_rate=1.0, epochs=1000)

What does this code do?

The train function orchestrates the full learning cycle. Here is what each section handles:

  • It one-hot encodes the labels and initializes parameters with He initialization.
  • Each epoch runs a forward pass, computes loss and accuracy, then runs backpropagation and updates weights.
  • Every 20 epochs, it saves a snapshot of the parameters. You will use these snapshots in the Secret Mission later.
  • The function returns the final trained parameters and the full history dictionary.
  • Run the cell by pressing Shift+Enter.

You should see output printed every 100 epochs showing the loss decreasing and accuracy increasing. The final accuracy should be above 95%.

What do the three phases mean?

Watch the printed output carefully. You will notice three distinct phases in the numbers:

  • Phase 1 (early epochs): accuracy hovers around 33%. The network is randomly guessing across 3 classes.
  • Phase 2 (middle epochs): accuracy jumps rapidly. The gradients are large and the network is finding the spiral structure.
  • Phase 3 (late epochs): accuracy plateaus near 95%+. The gradients shrink as the network converges on its best solution.

Accuracy stuck below 50%?

Make sure you ran all previous cells in order (Cells 1 through 7) before running Cell 8. The train function depends on initialize_parameters, forward, compute_loss, backward, and update_parameters all being defined in earlier cells.

Help me debug my neural network training loop

Plot the learning curves

The printed numbers confirm the network is learning, but a chart tells a much richer story. A dual-axis plot lets you see loss and accuracy on the same timeline.

  • Create a new cell (Cell 9) and plot the training history by pasting this code:
# Cell 9: Plot Learning Curves
fig, ax1 = plt.subplots(figsize=(12, 6))
color_loss = '#FF6B6B'
color_acc = '#4ECDC4'
ax1.set_xlabel('Epoch')
ax1.set_ylabel('Loss', color=color_loss)
ax1.plot(history['loss'], color=color_loss, linewidth=2, label='Loss')
ax1.tick_params(axis='y', labelcolor=color_loss)
ax2 = ax1.twinx()
ax2.set_ylabel('Accuracy', color=color_acc)
ax2.plot(history['accuracy'], color=color_acc, linewidth=2, label='Accuracy')
ax2.tick_params(axis='y', labelcolor=color_acc)
ax2.set_ylim(0, 1.05)
plt.title('Training Progress: Loss Decreases as Accuracy Increases')
fig.tight_layout()
plt.show()

What does this code do?

  • ax1.twinx() creates a second y-axis on the right side of the same chart, so loss and accuracy share the same x-axis (epochs) but have independent scales.
  • The red line (loss) should fall from left to right. The teal line (accuracy) should rise. When both curves flatten out, the network has converged.
  • Run the cell with Shift+Enter.

You should see a chart with a red loss curve dropping steeply in the first 200 epochs, then flattening. The teal accuracy curve mirrors it, climbing quickly and plateauing above 0.95.

Chart looks blank or shows a single flat line?

Check that you ran Cell 8 first. The history variable must exist with populated 'loss' and 'accuracy' lists before this cell can plot them.

Help me fix my learning curve plot

Plot the final decision boundary

The learning curve proves the network improved over time. Now you will see exactly what it learned by coloring every point in 2D space according to the network's prediction. This creates a decision boundary map showing the regions the network assigns to each class.

  • Create a new cell (Cell 10) and visualize the boundary by pasting this code:
# Cell 10: Plot Final Decision Boundary
def nn_predict(X_input):
    A2, _ = forward(X_input, params_trained)
    return np.argmax(A2, axis=1)

plot_decision_boundary(X, y, nn_predict, f'Neural Network Decision Boundary (Accuracy: {history["accuracy"][-1]:.1%})')

What does this code do?

  • nn_predict is a thin wrapper that feeds any set of 2D points through the trained network and returns class predictions.
  • plot_decision_boundary (defined back in Cell 3) creates a dense grid of 200x200 points, predicts a class for each one, and fills regions with color. Your original spiral data points are overlaid on top.
  • Compare this to the linear model's boundary from Step 2. The linear model could only draw straight lines. Your neural network draws smooth curves that wrap around each spiral arm.
  • Run the cell with Shift+Enter.

You should see a colored plot where each region matches one of the three spiral classes. The boundary curves smoothly between them, and the title displays your final accuracy (95% or higher).

Why can the network draw curves?

Each hidden neuron creates one linear boundary (a straight line in 2D). ReLU combines these into piecewise-linear regions. With 100 hidden neurons, the network has 100 straight-line pieces that together approximate smooth curves. More neurons means finer approximations.

Decision boundary looks like random patches?

Make sure params_trained came from a successful run of Cell 8. If you re-ran Cell 8 without np.random.seed(42) at the top of the notebook, re-run all cells from Cell 1 onward to reset the random state.

Help me fix my decision boundary plot

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook with all 10 cells.

ⓧ I'd like to double check the full code

Here is the complete notebook up to this point. Compare your cells against this:

# Cell 1: Imports and Setup
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap

np.random.seed(42)

# Cell 2: Generate Spiral Dataset
def generate_spiral_data(points_per_class=100, num_classes=3):
    X = np.zeros((points_per_class * num_classes, 2))
    y = np.zeros(points_per_class * num_classes, dtype=int)
    for class_idx in range(num_classes):
        start_idx = points_per_class * class_idx
        end_idx = points_per_class * (class_idx + 1)
        r = np.linspace(0.0, 1, points_per_class)
        theta = np.linspace(class_idx * 4, (class_idx + 1) * 4, points_per_class) + np.random.randn(points_per_class) * 0.2
        X[start_idx:end_idx, 0] = r * np.sin(theta)
        X[start_idx:end_idx, 1] = r * np.cos(theta)
        y[start_idx:end_idx] = class_idx
    return X, y

X, y = generate_spiral_data()

plt.figure(figsize=(8, 8))
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20)
plt.title('Spiral Dataset - 3 Classes')
plt.xlabel('x1')
plt.ylabel('x2')
plt.axis('equal')
plt.show()

# Cell 3: Demonstrate Linear Model Failure
def linear_forward(X, W, b):
    return X @ W + b

def softmax(Z):
    exp_Z = np.exp(Z - np.max(Z, axis=1, keepdims=True))
    return exp_Z / np.sum(exp_Z, axis=1, keepdims=True)

W_linear = np.random.randn(2, 3) * 0.01
b_linear = np.zeros((1, 3))
num_samples = X.shape[0]
y_one_hot = np.zeros((num_samples, 3))
y_one_hot[np.arange(num_samples), y] = 1

for i in range(1000):
    scores = linear_forward(X, W_linear, b_linear)
    probs = softmax(scores)
    dscores = (probs - y_one_hot) / num_samples
    dW = X.T @ dscores
    db = np.sum(dscores, axis=0, keepdims=True)
    W_linear -= 1.0 * dW
    b_linear -= 1.0 * db

def plot_decision_boundary(X, y, predict_fn, title):
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200), np.linspace(y_min, y_max, 200))
    grid_points = np.c_[xx.ravel(), yy.ravel()]
    Z = predict_fn(grid_points)
    Z = Z.reshape(xx.shape)
    plt.figure(figsize=(8, 8))
    plt.contourf(xx, yy, Z, alpha=0.3, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']))
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20, edgecolors='k', linewidth=0.5)
    plt.title(title)
    plt.xlabel('x1')
    plt.ylabel('x2')
    plt.axis('equal')
    plt.show()

def linear_predict(X_input):
    scores = linear_forward(X_input, W_linear, b_linear)
    return np.argmax(scores, axis=1)

plot_decision_boundary(X, y, linear_predict, 'Linear Model: Can Only Draw Straight Lines')

# Cell 4: Activation Function Comparison
fig, axes = plt.subplots(2, 3, figsize=(15, 8))
z = np.linspace(-5, 5, 200)
sigmoid_out = 1 / (1 + np.exp(-z))
sigmoid_deriv = sigmoid_out * (1 - sigmoid_out)
axes[0, 0].plot(z, sigmoid_out, 'b-', linewidth=2)
axes[0, 0].set_title('Sigmoid')
axes[0, 0].set_ylabel('Output')
axes[0, 0].grid(True, alpha=0.3)
axes[1, 0].plot(z, sigmoid_deriv, 'r-', linewidth=2)
axes[1, 0].set_title('Sigmoid Derivative (max=0.25!)')
axes[1, 0].set_ylabel('Gradient')
axes[1, 0].set_xlabel('Input z')
axes[1, 0].grid(True, alpha=0.3)
tanh_out = np.tanh(z)
tanh_deriv = 1 - tanh_out**2
axes[0, 1].plot(z, tanh_out, 'b-', linewidth=2)
axes[0, 1].set_title('Tanh')
axes[0, 1].grid(True, alpha=0.3)
axes[1, 1].plot(z, tanh_deriv, 'r-', linewidth=2)
axes[1, 1].set_title('Tanh Derivative (max=1.0)')
axes[1, 1].set_xlabel('Input z')
axes[1, 1].grid(True, alpha=0.3)
relu_out = np.maximum(0, z)
relu_deriv = (z > 0).astype(float)
axes[0, 2].plot(z, relu_out, 'b-', linewidth=2)
axes[0, 2].set_title('ReLU')
axes[0, 2].grid(True, alpha=0.3)
axes[1, 2].plot(z, relu_deriv, 'r-', linewidth=2)
axes[1, 2].set_title('ReLU Derivative (0 or 1)')
axes[1, 2].set_xlabel('Input z')
axes[1, 2].grid(True, alpha=0.3)
plt.suptitle('Why ReLU? Sigmoid/Tanh derivatives shrink gradients. ReLU passes them through unchanged.', fontsize=12, y=1.02)
plt.tight_layout()
plt.show()

# Cell 5: Neural Network Implementation
def relu(Z):
    return np.maximum(0, Z)

def initialize_parameters(input_size, hidden_size, output_size):
    params = {
        'W1': np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size),
        'b1': np.zeros((1, hidden_size)),
        'W2': np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size),
        'b2': np.zeros((1, output_size))
    }
    return params

def forward(X, params):
    Z1 = X @ params['W1'] + params['b1']
    A1 = relu(Z1)
    Z2 = A1 @ params['W2'] + params['b2']
    A2 = softmax(Z2)
    cache = {'Z1': Z1, 'A1': A1, 'Z2': Z2, 'A2': A2}
    return A2, cache

def compute_loss(A2, Y_one_hot):
    m = Y_one_hot.shape[0]
    loss = -np.sum(Y_one_hot * np.log(A2 + 1e-8)) / m
    return loss

def backward(X, Y_one_hot, params, cache):
    m = X.shape[0]
    A1 = cache['A1']
    A2 = cache['A2']
    Z1 = cache['Z1']
    dZ2 = A2 - Y_one_hot
    dW2 = A1.T @ dZ2 / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m
    dA1 = dZ2 @ params['W2'].T
    dZ1 = dA1 * (Z1 > 0)
    dW1 = X.T @ dZ1 / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m
    grads = {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}
    return grads

def update_parameters(params, grads, learning_rate):
    params['W1'] -= learning_rate * grads['dW1']
    params['b1'] -= learning_rate * grads['db1']
    params['W2'] -= learning_rate * grads['dW2']
    params['b2'] -= learning_rate * grads['db2']
    return params

# Cell 6: Visualize He Initialization vs Tiny Weights
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
params_tiny = {
    'W1': np.random.randn(2, 100) * 0.01,
    'b1': np.zeros((1, 100)),
    'W2': np.random.randn(100, 3) * 0.01,
    'b2': np.zeros((1, 3))
}
_, cache_tiny = forward(X, params_tiny)
axes[0].hist(cache_tiny['A1'].ravel(), bins=50, color='red', alpha=0.7)
axes[0].set_title('Tiny Init (0.01): Hidden Activations Near Zero')
axes[0].set_xlabel('Activation Value')
axes[0].set_ylabel('Frequency')
params_he = initialize_parameters(2, 100, 3)
_, cache_he = forward(X, params_he)
axes[1].hist(cache_he['A1'].ravel(), bins=50, color='green', alpha=0.7)
axes[1].set_title('He Init: Healthy Spread of Activations')
axes[1].set_xlabel('Activation Value')
axes[1].set_ylabel('Frequency')
plt.suptitle('Why He Initialization? Tiny weights produce dead neurons. He keeps signals alive.', fontsize=12)
plt.tight_layout()
plt.show()

# Cell 7: Cross-Entropy vs MSE Comparison
fig, ax = plt.subplots(1, 1, figsize=(10, 6))
predicted_prob = np.linspace(0.01, 0.99, 100)
true_label = 1.0
ce_loss = -true_label * np.log(predicted_prob)
mse_loss = (true_label - predicted_prob) ** 2
ce_grad = -true_label / predicted_prob
mse_grad = -2 * (true_label - predicted_prob)
ax.plot(predicted_prob, ce_loss, 'b-', linewidth=2, label='Cross-Entropy Loss')
ax.plot(predicted_prob, mse_loss, 'r--', linewidth=2, label='MSE Loss')
ax.set_xlabel('Predicted Probability for True Class')
ax.set_ylabel('Loss')
ax.set_title('Why Cross-Entropy? It penalizes confident wrong predictions much more harshly.')
ax.legend(fontsize=12)
ax.grid(True, alpha=0.3)
ax.set_xlim(0, 1)
ax.set_ylim(0, 5)
plt.tight_layout()
plt.show()

# Cell 8: Training Loop
def train(X, y, hidden_size=100, learning_rate=1.0, epochs=1000, verbose=True):
    num_classes = len(np.unique(y))
    m = X.shape[0]
    Y_one_hot = np.zeros((m, num_classes))
    Y_one_hot[np.arange(m), y] = 1
    params = initialize_parameters(X.shape[1], hidden_size, num_classes)
    history = {'loss': [], 'accuracy': [], 'params_snapshots': []}
    for epoch in range(epochs):
        A2, cache = forward(X, params)
        loss = compute_loss(A2, Y_one_hot)
        predictions = np.argmax(A2, axis=1)
        accuracy = np.mean(predictions == y)
        history['loss'].append(loss)
        history['accuracy'].append(accuracy)
        if epoch % 20 == 0:
            snapshot = {k: v.copy() for k, v in params.items()}
            history['params_snapshots'].append((epoch, snapshot))
        grads = backward(X, Y_one_hot, params, cache)
        params = update_parameters(params, grads, learning_rate)
        if verbose and epoch % 100 == 0:
            print(f"Epoch {epoch:4d} | Loss: {loss:.4f} | Accuracy: {accuracy:.4f}")
    return params, history

params_trained, history = train(X, y, hidden_size=100, learning_rate=1.0, epochs=1000)

# Cell 9: Plot Learning Curves
fig, ax1 = plt.subplots(figsize=(12, 6))
color_loss = '#FF6B6B'
color_acc = '#4ECDC4'
ax1.set_xlabel('Epoch')
ax1.set_ylabel('Loss', color=color_loss)
ax1.plot(history['loss'], color=color_loss, linewidth=2, label='Loss')
ax1.tick_params(axis='y', labelcolor=color_loss)
ax2 = ax1.twinx()
ax2.set_ylabel('Accuracy', color=color_acc)
ax2.plot(history['accuracy'], color=color_acc, linewidth=2, label='Accuracy')
ax2.tick_params(axis='y', labelcolor=color_acc)
ax2.set_ylim(0, 1.05)
plt.title('Training Progress: Loss Decreases as Accuracy Increases')
fig.tight_layout()
plt.show()

# Cell 10: Plot Final Decision Boundary
def nn_predict(X_input):
    A2, _ = forward(X_input, params_trained)
    return np.argmax(A2, axis=1)

plot_decision_boundary(X, y, nn_predict, f'Neural Network Decision Boundary (Accuracy: {history["accuracy"][-1]:.1%})')

Your network achieves 95%+ accuracy on a problem that a linear model could never solve. Next, you will explore how hyperparameters like learning rate and network width control the quality of these boundaries.

Explore Hyperparameters with Visual Experiments

Your neural network is trained and achieving over 95% accuracy on the spiral dataset. The decision boundary plot proves it learned the curved separations between classes.

But how sensitive is this result to the choices you made? In this step, you will run controlled experiments to see exactly how the learning rate and network width affect training. The goal is a visual reference that builds your intuition for tuning these parameters on future problems.

In this step, get ready to:
  • Compare three learning rates and observe how each affects the loss curve.
  • Compare three network widths and observe how each affects the decision boundary shape.
  • Build a visual summary showing how each hyperparameter controls learning behavior.
Compare learning rates side by side

The learning rate controls how large each parameter update is during gradient descent. Too small, and the network takes forever to converge. Too large, and it overshoots the minimum and becomes unstable.

There is no universal formula for the "right" learning rate. These plots will teach you the intuition to recognize each failure mode at a glance.

  • Create a new cell below Cell 10 in your notebook and paste the following code:
# Cell 11: Learning Rate Comparison
fig, axes = plt.subplots(1, 3, figsize=(18, 5))
learning_rates = [0.1, 1.0, 5.0]

for idx, lr in enumerate(learning_rates):
    np.random.seed(42)
    _, hist = train(X, y, hidden_size=100, learning_rate=lr, epochs=1000, verbose=False)
    axes[idx].plot(hist['loss'], linewidth=2)
    axes[idx].set_title(f'Learning Rate = {lr}')
    axes[idx].set_xlabel('Epoch')
    axes[idx].set_ylabel('Loss')
    axes[idx].set_ylim(0, max(3.0, max(hist['loss'][:100])))
    axes[idx].grid(True, alpha=0.3)

plt.suptitle('Learning Rate: Too low = slow. Too high = unstable. Just right = fast convergence.', fontsize=12)
plt.tight_layout()
plt.show()

What does this code do?

This cell trains three identical networks from the same starting point (using np.random.seed(42) for reproducibility), changing only the learning rate each time.

  • learning_rate=0.1 takes tiny steps. The loss curve drops slowly and may not fully converge in 1000 epochs.
  • learning_rate=1.0 is the balanced rate you used in the previous step. The loss drops quickly and flattens out.
  • learning_rate=5.0 takes huge steps. The loss curve bounces wildly or explodes, showing the network is overshooting.
  • Run the cell by pressing Shift+Enter.

You should see a 1x3 grid of loss curves. The left plot (0.1) should show a slow, steady decline. The middle plot (1.0) should show fast convergence. The right plot (5.0) should show erratic behavior or divergence.

All three curves look the same?

Make sure you have np.random.seed(42) inside the loop, not outside it. Each training run needs the same starting weights for a fair comparison.

Also check that verbose=False is set. If you see printed output flooding the cell, the plot may be pushed out of view.

Help me debug my learning rate comparison plot.

Compare network widths side by side

Each hidden neuron learns one linear boundary in the input space. ReLU combines them into piecewise-linear regions. More neurons means more pieces, which means smoother curves that better match the spiral shape.

  • Create a new cell below Cell 11 and paste the following code:
# Cell 12: Network Width Comparison
fig, axes = plt.subplots(1, 3, figsize=(18, 6))
hidden_sizes = [10, 50, 200]

for idx, hs in enumerate(hidden_sizes):
    np.random.seed(42)
    params_exp, _ = train(X, y, hidden_size=hs, learning_rate=1.0, epochs=1000, verbose=False)
    
    # Decision boundary
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200), np.linspace(y_min, y_max, 200))
    grid_points = np.c_[xx.ravel(), yy.ravel()]
    
    A2_grid, _ = forward(grid_points, params_exp)
    Z_grid = np.argmax(A2_grid, axis=1).reshape(xx.shape)
    
    axes[idx].contourf(xx, yy, Z_grid, alpha=0.3, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']))
    axes[idx].scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20, edgecolors='k', linewidth=0.5)
    axes[idx].set_title(f'Hidden Neurons: {hs}')
    axes[idx].axis('equal')

plt.suptitle('Network Width: More neurons = smoother decision boundaries', fontsize=12)
plt.tight_layout()
plt.show()

What does this code do?

This cell trains three networks with different hidden layer sizes (10, 50, and 200 neurons), then plots the decision boundary each one learned.

  • hidden_size=10 gives the network only 10 linear pieces to work with. The boundaries look jagged and miss parts of the spiral.
  • hidden_size=50 provides enough pieces to approximate the curves reasonably well.
  • hidden_size=200 produces smooth, precise boundaries that hug the spiral structure closely.

For each network, the code creates a mesh grid over the input space, runs the forward pass on every grid point, and colors each region by predicted class. The spiral data points are overlaid on top.

  • Run the cell by pressing Shift+Enter.

You should see a 1x3 grid of decision boundary plots. The left plot (10 neurons) should have rough, angular boundaries. The right plot (200 neurons) should have smooth, flowing curves that closely follow each spiral arm.

Seeing identical decision boundaries?

Check that the hidden_size=hs parameter is inside the train() call. If you accidentally hardcoded a value, all three plots will look the same.

Also confirm that params_exp (not params_trained) is being passed to forward() when computing the grid predictions.

Help me debug my network width comparison.

Review your summary dashboard

Scroll through Cells 11 and 12 together. These six plots form your hyperparameter summary dashboard. The top row (Cell 11) shows how the learning rate controls the speed and stability of training. The bottom row (Cell 12) shows how network width controls the complexity of learned boundaries.

Why does this matter for real projects?

In practice, you cannot visualize decision boundaries for high-dimensional data. But the loss curve shape is always available. Recognizing a "too slow" vs "too fast" loss curve from this experiment means you can diagnose learning rate problems on any dataset, even when you cannot see the boundaries directly.

The width experiment teaches a transferable principle: model capacity must match problem complexity. A 10-neuron network is too simple for spirals, just like a small model would be too simple for image recognition. A 200-neuron network handles spirals easily, but for harder problems you would need even more capacity (or more layers).

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook with all 12 cells.

ⓧ I'd like to double check the full code

Here is the complete notebook with all cells from start to finish. Compare your code against this reference.

# Cell 1: Imports and Setup
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap

np.random.seed(42)

# Cell 2: Generate Spiral Dataset
def generate_spiral_data(points_per_class=100, num_classes=3):
    X = np.zeros((points_per_class * num_classes, 2))
    y = np.zeros(points_per_class * num_classes, dtype=int)
    for class_idx in range(num_classes):
        start_idx = points_per_class * class_idx
        end_idx = points_per_class * (class_idx + 1)
        r = np.linspace(0.0, 1, points_per_class)
        theta = np.linspace(class_idx * 4, (class_idx + 1) * 4, points_per_class) + np.random.randn(points_per_class) * 0.2
        X[start_idx:end_idx, 0] = r * np.sin(theta)
        X[start_idx:end_idx, 1] = r * np.cos(theta)
        y[start_idx:end_idx] = class_idx
    return X, y

X, y = generate_spiral_data()

plt.figure(figsize=(8, 8))
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20)
plt.title('Spiral Dataset - 3 Classes')
plt.xlabel('x1')
plt.ylabel('x2')
plt.axis('equal')
plt.show()

# Cell 3: Demonstrate Linear Model Failure
def linear_forward(X, W, b):
    return X @ W + b

def softmax(Z):
    exp_Z = np.exp(Z - np.max(Z, axis=1, keepdims=True))
    return exp_Z / np.sum(exp_Z, axis=1, keepdims=True)

W_linear = np.random.randn(2, 3) * 0.01
b_linear = np.zeros((1, 3))
num_samples = X.shape[0]
y_one_hot = np.zeros((num_samples, 3))
y_one_hot[np.arange(num_samples), y] = 1

for i in range(1000):
    scores = linear_forward(X, W_linear, b_linear)
    probs = softmax(scores)
    dscores = (probs - y_one_hot) / num_samples
    dW = X.T @ dscores
    db = np.sum(dscores, axis=0, keepdims=True)
    W_linear -= 1.0 * dW
    b_linear -= 1.0 * db

def plot_decision_boundary(X, y, predict_fn, title):
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200), np.linspace(y_min, y_max, 200))
    grid_points = np.c_[xx.ravel(), yy.ravel()]
    Z = predict_fn(grid_points)
    Z = Z.reshape(xx.shape)
    plt.figure(figsize=(8, 8))
    plt.contourf(xx, yy, Z, alpha=0.3, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']))
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20, edgecolors='k', linewidth=0.5)
    plt.title(title)
    plt.xlabel('x1')
    plt.ylabel('x2')
    plt.axis('equal')
    plt.show()

def linear_predict(X_input):
    scores = linear_forward(X_input, W_linear, b_linear)
    return np.argmax(scores, axis=1)

plot_decision_boundary(X, y, linear_predict, 'Linear Model: Can Only Draw Straight Lines')

# Cell 4: Activation Function Comparison
fig, axes = plt.subplots(2, 3, figsize=(15, 8))
z = np.linspace(-5, 5, 200)
sigmoid_out = 1 / (1 + np.exp(-z))
sigmoid_deriv = sigmoid_out * (1 - sigmoid_out)
axes[0, 0].plot(z, sigmoid_out, 'b-', linewidth=2)
axes[0, 0].set_title('Sigmoid')
axes[0, 0].set_ylabel('Output')
axes[0, 0].grid(True, alpha=0.3)
axes[1, 0].plot(z, sigmoid_deriv, 'r-', linewidth=2)
axes[1, 0].set_title('Sigmoid Derivative (max=0.25!)')
axes[1, 0].set_ylabel('Gradient')
axes[1, 0].set_xlabel('Input z')
axes[1, 0].grid(True, alpha=0.3)
tanh_out = np.tanh(z)
tanh_deriv = 1 - tanh_out**2
axes[0, 1].plot(z, tanh_out, 'b-', linewidth=2)
axes[0, 1].set_title('Tanh')
axes[0, 1].grid(True, alpha=0.3)
axes[1, 1].plot(z, tanh_deriv, 'r-', linewidth=2)
axes[1, 1].set_title('Tanh Derivative (max=1.0)')
axes[1, 1].set_xlabel('Input z')
axes[1, 1].grid(True, alpha=0.3)
relu_out = np.maximum(0, z)
relu_deriv = (z > 0).astype(float)
axes[0, 2].plot(z, relu_out, 'b-', linewidth=2)
axes[0, 2].set_title('ReLU')
axes[0, 2].grid(True, alpha=0.3)
axes[1, 2].plot(z, relu_deriv, 'r-', linewidth=2)
axes[1, 2].set_title('ReLU Derivative (0 or 1)')
axes[1, 2].set_xlabel('Input z')
axes[1, 2].grid(True, alpha=0.3)
plt.suptitle('Why ReLU? Sigmoid/Tanh derivatives shrink gradients. ReLU passes them through unchanged.', fontsize=12, y=1.02)
plt.tight_layout()
plt.show()

# Cell 5: Neural Network Implementation
def relu(Z):
    return np.maximum(0, Z)

def initialize_parameters(input_size, hidden_size, output_size):
    params = {
        'W1': np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size),
        'b1': np.zeros((1, hidden_size)),
        'W2': np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size),
        'b2': np.zeros((1, output_size))
    }
    return params

def forward(X, params):
    Z1 = X @ params['W1'] + params['b1']
    A1 = relu(Z1)
    Z2 = A1 @ params['W2'] + params['b2']
    A2 = softmax(Z2)
    cache = {'Z1': Z1, 'A1': A1, 'Z2': Z2, 'A2': A2}
    return A2, cache

def compute_loss(A2, Y_one_hot):
    m = Y_one_hot.shape[0]
    loss = -np.sum(Y_one_hot * np.log(A2 + 1e-8)) / m
    return loss

def backward(X, Y_one_hot, params, cache):
    m = X.shape[0]
    A1 = cache['A1']
    A2 = cache['A2']
    Z1 = cache['Z1']
    dZ2 = A2 - Y_one_hot
    dW2 = A1.T @ dZ2 / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m
    dA1 = dZ2 @ params['W2'].T
    dZ1 = dA1 * (Z1 > 0)
    dW1 = X.T @ dZ1 / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m
    grads = {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}
    return grads

def update_parameters(params, grads, learning_rate):
    params['W1'] -= learning_rate * grads['dW1']
    params['b1'] -= learning_rate * grads['db1']
    params['W2'] -= learning_rate * grads['dW2']
    params['b2'] -= learning_rate * grads['db2']
    return params

# Cell 6: Visualize He Initialization vs Tiny Weights
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
params_tiny = {
    'W1': np.random.randn(2, 100) * 0.01,
    'b1': np.zeros((1, 100)),
    'W2': np.random.randn(100, 3) * 0.01,
    'b2': np.zeros((1, 3))
}
_, cache_tiny = forward(X, params_tiny)
axes[0].hist(cache_tiny['A1'].ravel(), bins=50, color='red', alpha=0.7)
axes[0].set_title('Tiny Init (0.01): Hidden Activations Near Zero')
axes[0].set_xlabel('Activation Value')
axes[0].set_ylabel('Frequency')
params_he = initialize_parameters(2, 100, 3)
_, cache_he = forward(X, params_he)
axes[1].hist(cache_he['A1'].ravel(), bins=50, color='green', alpha=0.7)
axes[1].set_title('He Init: Healthy Spread of Activations')
axes[1].set_xlabel('Activation Value')
axes[1].set_ylabel('Frequency')
plt.suptitle('Why He Initialization? Tiny weights produce dead neurons. He keeps signals alive.', fontsize=12)
plt.tight_layout()
plt.show()

# Cell 7: Cross-Entropy vs MSE Comparison
fig, ax = plt.subplots(1, 1, figsize=(10, 6))
predicted_prob = np.linspace(0.01, 0.99, 100)
true_label = 1.0
ce_loss = -true_label * np.log(predicted_prob)
mse_loss = (true_label - predicted_prob) ** 2
ce_grad = -true_label / predicted_prob
mse_grad = -2 * (true_label - predicted_prob)
ax.plot(predicted_prob, ce_loss, 'b-', linewidth=2, label='Cross-Entropy Loss')
ax.plot(predicted_prob, mse_loss, 'r--', linewidth=2, label='MSE Loss')
ax.set_xlabel('Predicted Probability for True Class')
ax.set_ylabel('Loss')
ax.set_title('Why Cross-Entropy? It penalizes confident wrong predictions much more harshly.')
ax.legend(fontsize=12)
ax.grid(True, alpha=0.3)
ax.set_xlim(0, 1)
ax.set_ylim(0, 5)
plt.tight_layout()
plt.show()

# Cell 8: Training Loop
def train(X, y, hidden_size=100, learning_rate=1.0, epochs=1000, verbose=True):
    num_classes = len(np.unique(y))
    m = X.shape[0]
    Y_one_hot = np.zeros((m, num_classes))
    Y_one_hot[np.arange(m), y] = 1
    params = initialize_parameters(X.shape[1], hidden_size, num_classes)
    history = {'loss': [], 'accuracy': [], 'params_snapshots': []}
    for epoch in range(epochs):
        A2, cache = forward(X, params)
        loss = compute_loss(A2, Y_one_hot)
        predictions = np.argmax(A2, axis=1)
        accuracy = np.mean(predictions == y)
        history['loss'].append(loss)
        history['accuracy'].append(accuracy)
        if epoch % 20 == 0:
            snapshot = {k: v.copy() for k, v in params.items()}
            history['params_snapshots'].append((epoch, snapshot))
        grads = backward(X, Y_one_hot, params, cache)
        params = update_parameters(params, grads, learning_rate)
        if verbose and epoch % 100 == 0:
            print(f"Epoch {epoch:4d} | Loss: {loss:.4f} | Accuracy: {accuracy:.4f}")
    return params, history

params_trained, history = train(X, y, hidden_size=100, learning_rate=1.0, epochs=1000)

# Cell 9: Plot Learning Curves
fig, ax1 = plt.subplots(figsize=(12, 6))
color_loss = '#FF6B6B'
color_acc = '#4ECDC4'
ax1.set_xlabel('Epoch')
ax1.set_ylabel('Loss', color=color_loss)
ax1.plot(history['loss'], color=color_loss, linewidth=2, label='Loss')
ax1.tick_params(axis='y', labelcolor=color_loss)
ax2 = ax1.twinx()
ax2.set_ylabel('Accuracy', color=color_acc)
ax2.plot(history['accuracy'], color=color_acc, linewidth=2, label='Accuracy')
ax2.tick_params(axis='y', labelcolor=color_acc)
ax2.set_ylim(0, 1.05)
plt.title('Training Progress: Loss Decreases as Accuracy Increases')
fig.tight_layout()
plt.show()

# Cell 10: Plot Final Decision Boundary
def nn_predict(X_input):
    A2, _ = forward(X_input, params_trained)
    return np.argmax(A2, axis=1)

plot_decision_boundary(X, y, nn_predict, f'Neural Network Decision Boundary (Accuracy: {history["accuracy"][-1]:.1%})')

# Cell 11: Learning Rate Comparison
fig, axes = plt.subplots(1, 3, figsize=(18, 5))
learning_rates = [0.1, 1.0, 5.0]

for idx, lr in enumerate(learning_rates):
    np.random.seed(42)
    _, hist = train(X, y, hidden_size=100, learning_rate=lr, epochs=1000, verbose=False)
    axes[idx].plot(hist['loss'], linewidth=2)
    axes[idx].set_title(f'Learning Rate = {lr}')
    axes[idx].set_xlabel('Epoch')
    axes[idx].set_ylabel('Loss')
    axes[idx].set_ylim(0, max(3.0, max(hist['loss'][:100])))
    axes[idx].grid(True, alpha=0.3)

plt.suptitle('Learning Rate: Too low = slow. Too high = unstable. Just right = fast convergence.', fontsize=12)
plt.tight_layout()
plt.show()

# Cell 12: Network Width Comparison
fig, axes = plt.subplots(1, 3, figsize=(18, 6))
hidden_sizes = [10, 50, 200]

for idx, hs in enumerate(hidden_sizes):
    np.random.seed(42)
    params_exp, _ = train(X, y, hidden_size=hs, learning_rate=1.0, epochs=1000, verbose=False)
    
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200), np.linspace(y_min, y_max, 200))
    grid_points = np.c_[xx.ravel(), yy.ravel()]
    
    A2_grid, _ = forward(grid_points, params_exp)
    Z_grid = np.argmax(A2_grid, axis=1).reshape(xx.shape)
    
    axes[idx].contourf(xx, yy, Z_grid, alpha=0.3, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']))
    axes[idx].scatter(X[:, 0], X[:, 1], c=y, cmap=ListedColormap(['#FF6B6B', '#4ECDC4', '#45B7D1']), s=20, edgecolors='k', linewidth=0.5)
    axes[idx].set_title(f'Hidden Neurons: {hs}')
    axes[idx].axis('equal')

plt.suptitle('Network Width: More neurons = smoother decision boundaries', fontsize=12)
plt.tight_layout()
plt.show()

You now have a complete visual reference showing how hyperparameters control your neural network's learning. Next up is the Secret Mission, where you will animate the decision boundary evolving frame-by-frame across training epochs to watch the network "think" in real time.

Secret mission

Animate the Decision Boundary

Your static decision boundary plot shows the end result, but what about the journey? In this secret mission, you'll animate the boundary evolving frame-by-frame across training epochs to watch the network "think" in real-time.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your files for future experiments, close Jupyter Lab to free up memory, or delete everything entirely.

Resources you used:

  • neural_network_from_scratch.ipynb (Jupyter notebook with all 13 cells).
  • decision_boundary_evolution.gif (animation file saved locally).
  • Pillow package (installed for GIF export in the secret mission).

Keep everything running

No action needed. Choose this if you want to keep experimenting with hyperparameters, adding layers, or trying new datasets.

  • Your notebook and GIF remain in the working directory where you launched Jupyter Lab.
  • Jupyter Lab continues running in your terminal until you stop it.
  • Pillow stays installed in your Python environment.

Pause - I'll come back to this later

Shut down Jupyter Lab to free up memory. Your notebook and files stay on disk for next time.

  • Switch to the terminal where Jupyter Lab is running.
  • Press Ctrl+C to stop the Jupyter Lab server.
  • Confirm the shutdown when prompted.
  • Your notebook auto-saves, so all cells and outputs are preserved for when you return.

Delete - I don't want to use this again

Remove all project files and uninstall packages you added during this project.

  • Switch to the terminal where Jupyter Lab is running and press Ctrl+C to stop the server.
  • Delete the notebook and animation file by running:
rm neural_network_from_scratch.ipynb decision_boundary_evolution.gif
  • Uninstall the Pillow package (added for the secret mission GIF export) by running:
pip uninstall Pillow -y
  • Confirm deletion by running:
ls

You should no longer see neural_network_from_scratch.ipynb or decision_boundary_evolution.gif in the output.

Nice Work!

Nice Work!

Nice work! You just built a complete neural network from scratch using nothing but NumPy and math. Your notebook now contains a working classifier that achieves 95%+ accuracy on a 3-class spiral dataset, backed by visualizations that prove you understand every component.

You've learned how to:

  • Build a multi-layer neural network from scratch with forward propagation, backpropagation, and gradient descent, training it to classify a complex spiral dataset.
  • Prove visually why each design choice matters by showing a linear model failing, comparing activation function derivatives, and contrasting He initialization against tiny random weights.
  • Create decision boundary visualizations and hyperparameter comparison dashboards that reveal how learning rate and network width control the network's behavior.
  • Secret Mission: Animated the decision boundary evolving across training epochs to watch the network learn in real-time.

Ready to quiz yourself?