Build a Neural Network from Scratch

Implement a neural network using only NumPy to classify non-linear data.

Introduction

30 Second Summary

Every time you ask ChatGPT a question or unlock your phone with your face, a neural network is doing math behind the scenes. The core of that math is surprisingly simple: multiply, add, check how wrong you are, adjust, repeat.

In this project, you will build a neural network from scratch using only Python and NumPy. You will implement every piece of the math yourself, from matrix multiplication to gradient descent, with no frameworks hiding the details.

What You'll Build

A Python script that generates a circular dataset, attempts to classify it with a single neuron (which fails), then builds and trains a two-layer neural network that succeeds with 95%+ accuracy.

By the end of this project, you'll have:

  • A dataset generator that creates non-linearly separable circular data and a single neuron classifier that demonstrates its limitations.
  • A two-layer neural network built entirely from scratch with NumPy that classifies non-linear data with 95%+ accuracy.
  • A decision boundary visualization showing exactly how your network separates the two classes of data.
  • Secret Mission: Generate a spiral dataset and build a deeper 3-layer network with ReLU activation to classify it.

Are there any prerequisites?

You need basic Python knowledge (variables, functions, loops). No prior machine learning or advanced math experience is required. You will also need Python 3.11+ and NumPy installed, which you will verify in Step 1.

Before We Start

You're about to build a neural network from scratch to understand the math that powers AI. By the end, you'll be able to explain gradient descent, backpropagation, and matrix operations with confidence.

In this step, you'll verify your tools are ready and create the project file you'll work in for the rest of this project.

In this step, get ready to:
  • Verify Python and NumPy are installed and working.
  • Install Matplotlib for visualizations.
  • Create your project file with the required imports.
Verify Python and NumPy

This project requires Python 3.11+ and NumPy (a math library for working with matrices and vectors). Let's confirm both are available on your machine.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal (macOS) or PowerShell (Windows), and pressing Enter.
  • Check your Python version by running this command:
python3 --version

✔️ I see version 3.11 or higher

You should see something like Python 3.11.x or higher. Python is ready to go.

ⓧ Command not found

This means Python is not in your system PATH.

  • On macOS, install Python via Homebrew by running brew install python in your terminal.
  • On Windows, download the installer from python.org and make sure to check Add Python to PATH during installation.
  • After installing, close and reopen your terminal, then try the command again.
  • Verify NumPy is installed by running this command:
python3 -c "import numpy; print(numpy.__version__)"

You should see a version number like 2.4.6 printed.

Seeing ModuleNotFoundError?

If you see an error mentioning that NumPy cannot be found, install it by running:

pip3 install numpy

Then try the verify command again. Help me troubleshoot my NumPy installation.

Install Matplotlib

Matplotlib is the visualization library you'll use to plot your data and decision boundaries.

  • Install Matplotlib by running this command:
pip3 install matplotlib
  • Verify the installation by running:
python3 -c "import matplotlib; print(matplotlib.__version__)"

You should see a version number like 3.10.9 printed.

Install command failed?

Make sure you're using pip3 (not pip) to match your python3 installation. If you see a permissions error, try adding --user at the end of the install command.

Help me fix my Matplotlib installation.

Create your project file

Everything you build in this project lives in a single Python file. Let's create it and add the two libraries you'll use throughout.

  • Open your code editor by pressing Cmd+Space (macOS) or the Windows key (Windows), typing VS Code, and pressing Enter.
  • Create a new file called neural_network.py by pressing Cmd+N (macOS) or Ctrl+N (Windows), then save it with Cmd+S (macOS) or Ctrl+S (Windows).
  • Name the file neural_network.py and save it to your Desktop.
  • Add the following imports to the file:
import numpy as np
import matplotlib.pyplot as plt

What do these imports do?

  • import numpy as np gives you access to NumPy's math functions (matrix multiplication, random numbers, etc.) using the shorthand np.
  • import matplotlib.pyplot as plt gives you access to Matplotlib's plotting tools using the shorthand plt.
  • Save neural_network.py.
  • In your terminal, navigate to your Desktop and run the file to confirm no errors:
cd ~/Desktop
python3 neural_network.py

If everything is set up correctly, the command completes with no output and no errors.

Seeing an error when running the file?

If you see ModuleNotFoundError, one of the imports failed. Double-check that you ran the install and verify steps above for both NumPy and Matplotlib.

If you see SyntaxError, check that your import lines match the code block above exactly.

Help me debug my neural_network.py file.

✔️ Awesome, I've got everything!

Great. Double-check you've saved your file.

ⓧ I'd like to double check the full code

import numpy as np
import matplotlib.pyplot as plt

Your environment is ready. Next up, you'll generate a dataset and see why a straight line can't solve every classification problem.

Create Your Dataset

Your environment is ready. Now it's time to create the data your neural network will learn from.

Every neural network needs a problem to solve. You'll generate a dataset with two classes arranged in a circular pattern: an inner circle and an outer ring. The critical insight is that no straight line can separate these two groups. This is exactly the kind of problem that exposes the limitations of simple models and motivates building a neural network.

In this step, get ready to:
  • Generate a circular dataset using NumPy vectors and matrices.
  • Understand the shape of your data in terms of linear algebra.
  • Visualize the dataset to see why a straight line cannot separate it.
Generate circular data

Your dataset will have 300 points split into two classes. Class 0 lives in an inner circle (small radius). Class 1 lives in an outer ring (large radius). You'll use polar coordinates (radius and angle) to generate each point, then convert to x,y coordinates.

  • In neural_network.py, add the following code below your existing imports:
# ============================================================
# PART 1: Create Your Dataset
# ============================================================

np.random.seed(42)

# Generate a circular dataset (non-linearly separable)
num_samples = 300

# Inner circle (class 0)
r_inner = np.random.uniform(0, 1, num_samples // 2)
theta_inner = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_inner = r_inner * np.cos(theta_inner)
y_inner = r_inner * np.sin(theta_inner)

# Outer ring (class 1)
r_outer = np.random.uniform(1.5, 2.5, num_samples // 2)
theta_outer = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_outer = r_outer * np.cos(theta_outer)
y_outer = r_outer * np.sin(theta_outer)

# Combine into dataset
X = np.vstack([
    np.column_stack([x_inner, y_inner]),
    np.column_stack([x_outer, y_outer])
])
y = np.array([0] * (num_samples // 2) + [1] * (num_samples // 2)).reshape(-1, 1)

# Shuffle
shuffle_idx = np.random.permutation(num_samples)
X = X[shuffle_idx]
y = y[shuffle_idx]

print(f"Dataset shape: {X.shape}")
print(f"Labels shape: {y.shape}")
print(f"First data point: {X[0]} -> Label: {y[0][0]}")

What does this code do?

This code generates 300 data points arranged in two circular patterns using polar coordinates.

  • np.random.seed(42) locks the random number generator so you get the same dataset every time you run the script.
  • np.random.uniform generates random radii and angles. The inner circle has radius 0 to 1. The outer ring has radius 1.5 to 2.5.
  • np.cos and np.sin convert polar coordinates (radius, angle) into x,y positions on a flat plane.
  • np.vstack and np.column_stack combine all the points into a single matrix X where each row is one data point.
  • np.random.permutation shuffles the data so class 0 and class 1 points are mixed together, preventing the network from memorizing order.
  • Save neural_network.py.
  • Run the file by executing this command in your terminal:
python3 neural_network.py
  • You should see output showing Dataset shape: (300, 2) and Labels shape: (300, 1) printed in your terminal.

Seeing a NameError or ModuleNotFoundError?

  • Make sure your import numpy as np line is still at the top of the file. The new code depends on it.
  • Check that you added the new code BELOW the imports, not inside or above them.
  • If you see an IndentationError, make sure none of the lines are accidentally indented. All this code should be at the top level (no leading spaces).

Why is my dataset code throwing errors?

Understand the shapes

Look at the output your script just printed. The shapes tell you everything about how your data is organized.

What do these shapes mean?

X has shape (300, 2). This is a matrix with 300 rows and 2 columns. Each row is a single data point represented as a 2D vector (an x-coordinate and a y-coordinate).

y has shape (300, 1). Each entry is either 0 (inner circle) or 1 (outer ring). This is the label the network will try to predict.

This IS linear algebra. Your data lives in a 2D vector space. The neural network's job is to learn a boundary through that space that separates the two classes.

Visualize the challenge

Numbers alone don't reveal the problem. Plotting the data makes the challenge obvious: you cannot draw a single straight line to separate the inner from the outer.

  • In neural_network.py, add this visualization code below the print statements:
# Visualize
plt.figure(figsize=(8, 8))
plt.scatter(X[y.flatten() == 0, 0], X[y.flatten() == 0, 1],
            c='blue', label='Class 0 (inner)', alpha=0.6)
plt.scatter(X[y.flatten() == 1, 0], X[y.flatten() == 1, 1],
            c='red', label='Class 1 (outer)', alpha=0.6)
plt.xlabel('Feature 1 (x)')
plt.ylabel('Feature 2 (y)')
plt.title('Circular Dataset: Can You Draw a Straight Line to Separate These?')
plt.legend()
plt.axis('equal')
plt.grid(True, alpha=0.3)
plt.show()

What does this code do?

  • plt.scatter draws dots at each data point's x,y location. The first call plots class 0 in blue, the second plots class 1 in red.
  • y.flatten() == 0 creates a filter that selects only the rows belonging to class 0. This is NumPy boolean indexing.
  • plt.axis('equal') ensures the x and y axes use the same scale, so circles actually look like circles instead of ovals.
  • Save neural_network.py and run it again:
python3 neural_network.py
  • A scatter plot window should appear showing blue dots clustered in a small circle at the center and red dots forming a ring around them.
  • Close the plot window to let the script finish.

Why can't a straight line separate this?

Look at the plot. The blue points are surrounded on all sides by red points. No matter how you angle a straight line, it will always misclassify a large chunk of one class.

This is called a non-linearly separable problem. A single neuron can only learn straight-line boundaries. To classify this data correctly, you need a neural network that can learn curved boundaries.

Plot window not appearing?

  • If you're running inside VS Code's integrated terminal, Matplotlib may need an interactive backend. Try running the script from a standalone terminal instead.
  • Make sure your import matplotlib.pyplot as plt line is at the top of the file.

Help me troubleshoot my Matplotlib plot

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete neural_network.py so far:

import numpy as np
import matplotlib.pyplot as plt

# ============================================================
# PART 1: Create Your Dataset
# ============================================================

np.random.seed(42)

# Generate a circular dataset (non-linearly separable)
num_samples = 300

# Inner circle (class 0)
r_inner = np.random.uniform(0, 1, num_samples // 2)
theta_inner = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_inner = r_inner * np.cos(theta_inner)
y_inner = r_inner * np.sin(theta_inner)

# Outer ring (class 1)
r_outer = np.random.uniform(1.5, 2.5, num_samples // 2)
theta_outer = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_outer = r_outer * np.cos(theta_outer)
y_outer = r_outer * np.sin(theta_outer)

# Combine into dataset
X = np.vstack([
    np.column_stack([x_inner, y_inner]),
    np.column_stack([x_outer, y_outer])
])
y = np.array([0] * (num_samples // 2) + [1] * (num_samples // 2)).reshape(-1, 1)

# Shuffle
shuffle_idx = np.random.permutation(num_samples)
X = X[shuffle_idx]
y = y[shuffle_idx]

print(f"Dataset shape: {X.shape}")
print(f"Labels shape: {y.shape}")
print(f"First data point: {X[0]} -> Label: {y[0][0]}")

# Visualize
plt.figure(figsize=(8, 8))
plt.scatter(X[y.flatten() == 0, 0], X[y.flatten() == 0, 1],
            c='blue', label='Class 0 (inner)', alpha=0.6)
plt.scatter(X[y.flatten() == 1, 0], X[y.flatten() == 1, 1],
            c='red', label='Class 1 (outer)', alpha=0.6)
plt.xlabel('Feature 1 (x)')
plt.ylabel('Feature 2 (y)')
plt.title('Circular Dataset: Can You Draw a Straight Line to Separate These?')
plt.legend()
plt.axis('equal')
plt.grid(True, alpha=0.3)
plt.show()

You now have a dataset that no straight line can classify correctly. Next, you'll build a single neuron and watch it fail on this exact problem. That failure is what makes the full neural network's success so satisfying.

Build and Train a Single Neuron

Your dataset is ready and you can see the problem: inner points and outer points form two circles that no straight line can separate. Now you need a model that can try.

In this step, you'll build the simplest possible model: a single neuron. It learns by computing predictions, measuring how wrong they are, and nudging its weights in the right direction. This process is called gradient descent. By the end, you'll see exactly why one neuron is not enough for circular data.

In this step, get ready to:
  • Implement the sigmoid function and initialize weights for a single neuron.
  • Train the neuron using gradient descent for 500 epochs.
  • Observe that a single neuron can only achieve ~50% accuracy on circular data.
Implement the forward pass

A single neuron takes in data, multiplies it by weights, adds a bias, and squashes the result through an activation function. The activation function you'll use is called sigmoid, which converts any number into a value between 0 and 1 (perfect for binary classification).

  • In neural_network.py, scroll to the bottom of the file (below the plt.show() line from Part 1).
  • Add the sigmoid function and weight initialization by copying this code:
# ============================================================
# PART 2: Build and Train a Single Neuron
# ============================================================

def sigmoid(z):
    """Squashes any number to range (0, 1)"""
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

# Initialize weights
w = np.random.randn(2, 1) * 0.01
b = 0.0

What does this code do?

The sigmoid function takes any number and outputs a value between 0 and 1. Large positive inputs produce values close to 1. Large negative inputs produce values close to 0. The np.clip prevents overflow errors from extremely large numbers.

The weights w are a 2x1 matrix (one weight per input feature). They start as tiny random numbers so the neuron begins without bias toward either class. The bias b starts at 0.

What is the forward pass?

The forward pass is the computation that turns input data into a prediction. For a single neuron: multiply the data by weights (matrix multiplication), add the bias, then squash through sigmoid. This is the core of linear algebra applied to machine learning.

  • Save neural_network.py.
  • You won't see a change yet. The next chunk adds the training loop that puts this neuron to work.
Implement loss, gradients, and the training loop

The neuron needs to learn. That means measuring how wrong it is (the loss function) and figuring out which direction to adjust its weights (the gradient). Then it repeats this process hundreds of times.

  • Add the training loop below your weight initialization by copying this code:
# Training loop for single neuron
learning_rate = 0.5
losses_single = []

for epoch in range(500):
    # Forward pass: z = X @ w + b, a = sigmoid(z)
    z = np.dot(X, w) + b
    a = sigmoid(z)

    # Compute loss (binary cross-entropy)
    epsilon = 1e-8
    loss = -np.mean(y * np.log(a + epsilon) + (1 - y) * np.log(1 - a + epsilon))
    losses_single.append(loss)

    # Backward pass (compute gradients)
    m = X.shape[0]
    dz = a - y
    dw = (1 / m) * np.dot(X.T, dz)
    db = (1 / m) * np.sum(dz)

    # Update weights (gradient descent!)
    w = w - learning_rate * dw
    b = b - learning_rate * db

What does this code do?

Each epoch runs four operations in sequence:

  • Forward pass: z = np.dot(X, w) + b multiplies every data point by the weights (matrix multiplication). Then sigmoid(z) squashes the result into a prediction between 0 and 1.
  • Loss: binary cross-entropy measures how far each prediction is from the true label. The epsilon prevents taking the log of zero.
  • Gradients: dw and db tell you how much to change each weight to reduce the loss. This is calculus (derivatives).
  • Update: subtract the gradient times the learning rate from each weight. This is gradient descent. The neuron gets slightly less wrong each time.

Why does dz = a - y work as the gradient?

When you combine binary cross-entropy loss with sigmoid activation, the math simplifies beautifully. The derivative of the loss with respect to z reduces to just the prediction minus the true label (a - y). If the prediction is too high, the gradient is positive (push weights down). If too low, it's negative (push weights up).

  • Save neural_network.py.
  • You won't see output yet. The next chunk adds the accuracy check and loss plot.
Check accuracy and plot the loss curve

After 500 epochs of training, you need to see how well the neuron actually performs. You'll compute accuracy and plot how the loss changed over time.

  • Add the accuracy check and loss plot below the training loop:
# Check accuracy
predictions = (sigmoid(np.dot(X, w) + b) >= 0.5).astype(int)
accuracy_single = np.mean(predictions == y) * 100
print(f"\nSingle Neuron Accuracy: {accuracy_single:.1f}%")
print("A single neuron can only draw a STRAIGHT line. It cannot separate circles!")

# Plot loss
plt.plot(losses_single)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title(f'Single Neuron Training (Accuracy: {accuracy_single:.1f}%)')
plt.show()

What does this code do?

The accuracy line runs a final forward pass, rounds predictions to 0 or 1 (using the 0.5 threshold), and compares them against the true labels. The losses_single list recorded the loss at every epoch, so plotting it shows whether the neuron was learning at all.

  • Save neural_network.py.
  • Run the script by executing this command in your terminal:
python3 neural_network.py
  • You should see the dataset scatter plot (from Part 1), then a printed accuracy around 50%, and finally a loss curve plot.
  • The terminal output should include Single Neuron Accuracy: followed by a number near 50%.

Why is 50% accuracy terrible?

Your dataset has two classes with equal numbers of points. Random guessing gets 50%. So the neuron learned basically nothing useful. The reason: a single neuron can only draw a straight line through the data. No straight line can separate an inner circle from an outer ring.

This is the fundamental limitation that motivates the next step. You need multiple neurons working together to learn curved boundaries.

Not seeing output or getting errors?

  • If you see a NameError for X or y, make sure Part 1 (the dataset code) is still at the top of your file.
  • If the plot window blocks the script, close it to let the next section run.
  • If accuracy is exactly 0% or 100%, check that your indentation inside the for loop is consistent (all lines indented with 4 spaces).

Ask for help: My single neuron training is giving unexpected results. Can you help me debug?

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the full neural_network.py up to this point:

import numpy as np
import matplotlib.pyplot as plt

# ============================================================
# PART 1: Create Your Dataset
# ============================================================

np.random.seed(42)

# Generate a circular dataset (non-linearly separable)
num_samples = 300

# Inner circle (class 0)
r_inner = np.random.uniform(0, 1, num_samples // 2)
theta_inner = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_inner = r_inner * np.cos(theta_inner)
y_inner = r_inner * np.sin(theta_inner)

# Outer ring (class 1)
r_outer = np.random.uniform(1.5, 2.5, num_samples // 2)
theta_outer = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_outer = r_outer * np.cos(theta_outer)
y_outer = r_outer * np.sin(theta_outer)

# Combine into dataset
X = np.vstack([
    np.column_stack([x_inner, y_inner]),
    np.column_stack([x_outer, y_outer])
])
y = np.array([0] * (num_samples // 2) + [1] * (num_samples // 2)).reshape(-1, 1)

# Shuffle
shuffle_idx = np.random.permutation(num_samples)
X = X[shuffle_idx]
y = y[shuffle_idx]

print(f"Dataset shape: {X.shape}")
print(f"Labels shape: {y.shape}")
print(f"First data point: {X[0]} -> Label: {y[0][0]}")

# Visualize
plt.figure(figsize=(8, 8))
plt.scatter(X[y.flatten() == 0, 0], X[y.flatten() == 0, 1],
            c='blue', label='Class 0 (inner)', alpha=0.6)
plt.scatter(X[y.flatten() == 1, 0], X[y.flatten() == 1, 1],
            c='red', label='Class 1 (outer)', alpha=0.6)
plt.xlabel('Feature 1 (x)')
plt.ylabel('Feature 2 (y)')
plt.title('Circular Dataset: Can You Draw a Straight Line to Separate These?')
plt.legend()
plt.axis('equal')
plt.grid(True, alpha=0.3)
plt.show()

# ============================================================
# PART 2: Build and Train a Single Neuron
# ============================================================

def sigmoid(z):
    """Squashes any number to range (0, 1)"""
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

# Initialize weights
w = np.random.randn(2, 1) * 0.01
b = 0.0

# Training loop for single neuron
learning_rate = 0.5
losses_single = []

for epoch in range(500):
    # Forward pass: z = X @ w + b, a = sigmoid(z)
    z = np.dot(X, w) + b
    a = sigmoid(z)

    # Compute loss (binary cross-entropy)
    epsilon = 1e-8
    loss = -np.mean(y * np.log(a + epsilon) + (1 - y) * np.log(1 - a + epsilon))
    losses_single.append(loss)

    # Backward pass (compute gradients)
    m = X.shape[0]
    dz = a - y
    dw = (1 / m) * np.dot(X.T, dz)
    db = (1 / m) * np.sum(dz)

    # Update weights (gradient descent!)
    w = w - learning_rate * dw
    b = b - learning_rate * db

# Check accuracy
predictions = (sigmoid(np.dot(X, w) + b) >= 0.5).astype(int)
accuracy_single = np.mean(predictions == y) * 100
print(f"\nSingle Neuron Accuracy: {accuracy_single:.1f}%")
print("A single neuron can only draw a STRAIGHT line. It cannot separate circles!")

# Plot loss
plt.plot(losses_single)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title(f'Single Neuron Training (Accuracy: {accuracy_single:.1f}%)')
plt.show()

You've just proven that a single neuron hits a hard wall on non-linear data. Next up, you'll build a multi-layer neural network that breaks through this limitation by stacking neurons together.

Build a Two-Layer Neural Network

Your single neuron hit a wall. With only ~50% accuracy, it proved that a straight line cannot separate circular data. The problem is clear: you need a model that can learn curved boundaries.

The solution is a two-layer neural network. By adding a hidden layer between input and output, the network can transform your data into a new space where circles become separable. In this step, you will build all the functions that make this possible: initialization, forward propagation, and backpropagation.

In this step, get ready to:
  • Design and initialize a network with 2 inputs, 8 hidden neurons, and 1 output.
  • Implement forward propagation with matrix multiplication and activation functions.
  • Implement backpropagation using the chain rule to compute gradients for every weight.
Design the network architecture

A single neuron has one set of weights connecting input directly to output. A two-layer network adds a hidden layer in between. This hidden layer transforms the input into a new representation where the data becomes linearly separable.

  • In neural_network.py, scroll to the bottom of your file (below the single neuron loss plot).
  • Add the following code to create an initialization function for the network:
# ============================================================
# PART 3: Build a Two-Layer Neural Network
# ============================================================

def initialize_network(input_size, hidden_size, output_size):
    """Initialize weights with small random values"""
    np.random.seed(42)
    params = {
        'W1': np.random.randn(input_size, hidden_size) * 0.5,
        'b1': np.zeros((1, hidden_size)),
        'W2': np.random.randn(hidden_size, output_size) * 0.5,
        'b2': np.zeros((1, output_size))
    }
    return params

What does this code do?

This function creates four weight matrices and stores them in a dictionary called params.

  • W1 has shape (2, 8). It connects 2 inputs to 8 hidden neurons. Each column is one neuron's weights.
  • b1 has shape (1, 8). One bias per hidden neuron.
  • W2 has shape (8, 1). It connects 8 hidden neurons to 1 output.
  • b2 has shape (1, 1). One bias for the output neuron.
  • Multiplying by 0.5 keeps the initial weights small, which helps training converge smoothly.

You won't see a change yet. The next chunk builds the forward pass that uses these weights.

Why 8 hidden neurons?

Eight neurons give the network enough capacity to learn a circular boundary without overfitting. Each hidden neuron learns to detect a different feature of the input space. Together, they carve out the curved region that separates inner points from outer points.

Implement forward propagation

Forward propagation pushes data through the network layer by layer. Each layer performs a matrix multiplication, adds a bias, then applies a non-linear activation function. This is the network making a prediction.

  • Add the forward function below initialize_network:
def forward(X, params):
    """Forward pass through 2-layer network"""
    # Hidden layer
    z1 = np.dot(X, params['W1']) + params['b1']
    a1 = np.tanh(z1)

    # Output layer
    z2 = np.dot(a1, params['W2']) + params['b2']
    a2 = sigmoid(z2)

    cache = {'z1': z1, 'a1': a1, 'z2': z2, 'a2': a2}
    return a2, cache

What does this code do?

Data flows through two layers of computation:

  • z1 = X @ W1 + b1 multiplies each data point by the hidden weights. This is linear algebra in action.
  • a1 = tanh(z1) applies the tanh activation, squashing values to the range (-1, 1). This non-linearity is what lets the network learn curves.
  • z2 = a1 @ W2 + b2 multiplies the hidden layer output by the second set of weights.
  • a2 = sigmoid(z2) squashes the final output to (0, 1) for a probability prediction.
  • The cache dictionary saves intermediate values. Backpropagation needs these to compute gradients.

Why tanh for the hidden layer?

Tanh outputs are centered around zero, which helps the next layer learn faster. It also has a clean derivative: 1 - a1². You will use that derivative in the backpropagation step.

You won't see output yet. The next chunk adds the loss and backward functions that complete the network.

Implement backpropagation

Backpropagation is the chain rule from calculus applied layer by layer in reverse. It answers: how much did each weight contribute to the error? You need two pieces: a loss function to measure the error, and a backward pass to distribute blame.

  • Add the compute_loss and backward functions below forward:
def compute_loss(y_true, y_pred):
    """Binary cross-entropy loss"""
    epsilon = 1e-8
    return -np.mean(y_true * np.log(y_pred + epsilon) +
                    (1 - y_true) * np.log(1 - y_pred + epsilon))

def backward(X, y, params, cache):
    """Backward pass: chain rule in action"""
    m = X.shape[0]
    a1 = cache['a1']
    a2 = cache['a2']

    # Output layer gradients
    dz2 = a2 - y
    dW2 = (1 / m) * np.dot(a1.T, dz2)
    db2 = (1 / m) * np.sum(dz2, axis=0, keepdims=True)

    # Hidden layer gradients (this is backpropagation!)
    da1 = np.dot(dz2, params['W2'].T)
    dz1 = da1 * (1 - a1 ** 2)  # tanh derivative
    dW1 = (1 / m) * np.dot(X.T, dz1)
    db1 = (1 / m) * np.sum(dz1, axis=0, keepdims=True)

    return {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}

What does this code do?

compute_loss measures how wrong the network's predictions are using binary cross-entropy. The epsilon prevents dividing by zero inside the logarithm.

backward computes gradients by working backward through the chain rule:

  • dz2 = a2 - y is the output error. How far off was the prediction?
  • dW2 and db2 tell you how to adjust the output layer weights.
  • da1 = dz2 @ W2.T propagates the error backward through the output weights. This is the chain rule: multiply by the derivative of each step.
  • dz1 = da1 * (1 - a1 ** 2) applies the tanh derivative. This tells you how much each hidden neuron contributed to the error.
  • dW1 and db1 tell you how to adjust the hidden layer weights.
  • Save neural_network.py.
  • Run the file to confirm there are no syntax errors by running:
python3 neural_network.py

You should see the same dataset plot, single neuron output, and loss curve from before. No new output appears because the functions are defined but not called yet.

Seeing a syntax error?

  • Check that all parentheses and brackets are matched. The params dictionary in initialize_network has nested braces that are easy to miss.
  • Make sure compute_loss has its return statement on the correct indentation level. The expression spans two lines.
  • Verify that backward references cache['a1'] and cache['a2'] with matching quote styles.

Help me debug a syntax error in my neural network functions.

✔️ Awesome, I've got everything!

Great. Double check that you have saved your file.

ⓧ I'd like to double check the full code

Here is the complete neural_network.py file up to this point:

import numpy as np
import matplotlib.pyplot as plt

# ============================================================
# PART 1: Create Your Dataset
# ============================================================

np.random.seed(42)

# Generate a circular dataset (non-linearly separable)
num_samples = 300

# Inner circle (class 0)
r_inner = np.random.uniform(0, 1, num_samples // 2)
theta_inner = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_inner = r_inner * np.cos(theta_inner)
y_inner = r_inner * np.sin(theta_inner)

# Outer ring (class 1)
r_outer = np.random.uniform(1.5, 2.5, num_samples // 2)
theta_outer = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_outer = r_outer * np.cos(theta_outer)
y_outer = r_outer * np.sin(theta_outer)

# Combine into dataset
X = np.vstack([
    np.column_stack([x_inner, y_inner]),
    np.column_stack([x_outer, y_outer])
])
y = np.array([0] * (num_samples // 2) + [1] * (num_samples // 2)).reshape(-1, 1)

# Shuffle
shuffle_idx = np.random.permutation(num_samples)
X = X[shuffle_idx]
y = y[shuffle_idx]

print(f"Dataset shape: {X.shape}")
print(f"Labels shape: {y.shape}")
print(f"First data point: {X[0]} -> Label: {y[0][0]}")

# Visualize
plt.figure(figsize=(8, 8))
plt.scatter(X[y.flatten() == 0, 0], X[y.flatten() == 0, 1],
            c='blue', label='Class 0 (inner)', alpha=0.6)
plt.scatter(X[y.flatten() == 1, 0], X[y.flatten() == 1, 1],
            c='red', label='Class 1 (outer)', alpha=0.6)
plt.xlabel('Feature 1 (x)')
plt.ylabel('Feature 2 (y)')
plt.title('Circular Dataset: Can You Draw a Straight Line to Separate These?')
plt.legend()
plt.axis('equal')
plt.grid(True, alpha=0.3)
plt.show()

# ============================================================
# PART 2: Build and Train a Single Neuron
# ============================================================

def sigmoid(z):
    """Squashes any number to range (0, 1)"""
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

# Initialize weights
w = np.random.randn(2, 1) * 0.01
b = 0.0

# Training loop for single neuron
learning_rate = 0.5
losses_single = []

for epoch in range(500):
    # Forward pass: z = X @ w + b, a = sigmoid(z)
    z = np.dot(X, w) + b
    a = sigmoid(z)

    # Compute loss (binary cross-entropy)
    epsilon = 1e-8
    loss = -np.mean(y * np.log(a + epsilon) + (1 - y) * np.log(1 - a + epsilon))
    losses_single.append(loss)

    # Backward pass (compute gradients)
    m = X.shape[0]
    dz = a - y
    dw = (1 / m) * np.dot(X.T, dz)
    db = (1 / m) * np.sum(dz)

    # Update weights (gradient descent!)
    w = w - learning_rate * dw
    b = b - learning_rate * db

# Check accuracy
predictions = (sigmoid(np.dot(X, w) + b) >= 0.5).astype(int)
accuracy_single = np.mean(predictions == y) * 100
print(f"\nSingle Neuron Accuracy: {accuracy_single:.1f}%")
print("A single neuron can only draw a STRAIGHT line. It cannot separate circles!")

# Plot loss
plt.plot(losses_single)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title(f'Single Neuron Training (Accuracy: {accuracy_single:.1f}%)')
plt.show()

# ============================================================
# PART 3: Build a Two-Layer Neural Network
# ============================================================

def initialize_network(input_size, hidden_size, output_size):
    """Initialize weights with small random values"""
    np.random.seed(42)
    params = {
        'W1': np.random.randn(input_size, hidden_size) * 0.5,
        'b1': np.zeros((1, hidden_size)),
        'W2': np.random.randn(hidden_size, output_size) * 0.5,
        'b2': np.zeros((1, output_size))
    }
    return params

def forward(X, params):
    """Forward pass through 2-layer network"""
    # Hidden layer
    z1 = np.dot(X, params['W1']) + params['b1']
    a1 = np.tanh(z1)

    # Output layer
    z2 = np.dot(a1, params['W2']) + params['b2']
    a2 = sigmoid(z2)

    cache = {'z1': z1, 'a1': a1, 'z2': z2, 'a2': a2}
    return a2, cache

def compute_loss(y_true, y_pred):
    """Binary cross-entropy loss"""
    epsilon = 1e-8
    return -np.mean(y_true * np.log(y_pred + epsilon) +
                    (1 - y_true) * np.log(1 - y_pred + epsilon))

def backward(X, y, params, cache):
    """Backward pass: chain rule in action"""
    m = X.shape[0]
    a1 = cache['a1']
    a2 = cache['a2']

    # Output layer gradients
    dz2 = a2 - y
    dW2 = (1 / m) * np.dot(a1.T, dz2)
    db2 = (1 / m) * np.sum(dz2, axis=0, keepdims=True)

    # Hidden layer gradients (this is backpropagation!)
    da1 = np.dot(dz2, params['W2'].T)
    dz1 = da1 * (1 - a1 ** 2)  # tanh derivative
    dW1 = (1 / m) * np.dot(X.T, dz1)
    db1 = (1 / m) * np.sum(dz1, axis=0, keepdims=True)

    return {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}

Your neural network now has all the building blocks: initialization, forward pass, loss measurement, and backward pass. Next up, you will wire these functions into a training loop and watch the network learn to draw a curved decision boundary.

Train and Visualize Your Neural Network

You've built all the machinery of a neural network: forward propagation, loss calculation, and backpropagation.

Now it's time to put these pieces together into a training loop and see your network learn. You'll watch the accuracy climb from coin-flip level (~50%) all the way to 95%+, then visualize the exact decision boundary your network discovered.

In this step, get ready to:
  • Train the neural network with a 2000-epoch gradient descent loop.
  • Plot the decision boundary to see how your network separates the circular data.
  • Experiment with the hidden layer size to see how network capacity affects learning.
Write the training loop

The training loop is where all the math you've implemented comes together. Each iteration runs forward propagation, computes the loss, runs backpropagation for gradients, then updates every weight in the direction that reduces the loss.

  • In neural_network.py, add the following code below the backward function:
# ============================================================
# PART 4: Train and Visualize
# ============================================================

# Train the network
params = initialize_network(input_size=2, hidden_size=8, output_size=1)
learning_rate = 1.0
losses_network = []

for epoch in range(2000):
    # Forward
    y_pred, cache = forward(X, params)

    # Loss
    loss = compute_loss(y, y_pred)
    losses_network.append(loss)

    # Backward
    grads = backward(X, y, params, cache)

    # Update
    params['W1'] -= learning_rate * grads['dW1']
    params['b1'] -= learning_rate * grads['db1']
    params['W2'] -= learning_rate * grads['dW2']
    params['b2'] -= learning_rate * grads['db2']

    if epoch % 400 == 0:
        accuracy = np.mean((y_pred >= 0.5).astype(int) == y) * 100
        print(f"Epoch {epoch:4d} | Loss: {loss:.4f} | Accuracy: {accuracy:.1f}%")

# Final accuracy
y_pred, _ = forward(X, params)
accuracy_network = np.mean((y_pred >= 0.5).astype(int) == y) * 100
print(f"\nNeural Network Accuracy: {accuracy_network:.1f}%")
print(f"Single Neuron was: {accuracy_single:.1f}%")

What does this code do?

This is the full training loop. Each of the 2000 epochs follows the same 4-step cycle:

  • forward(X, params) pushes the data through both layers to get predictions.
  • compute_loss(y, y_pred) measures how wrong those predictions are.
  • backward(X, y, params, cache) uses the chain rule to compute which direction each weight should move.
  • The update step subtracts learning_rate * gradient from each weight, nudging it toward lower loss.

The learning rate of 1.0 controls how big each step is. Every 400 epochs, the loop prints its progress so you can watch accuracy climb.

  • Save neural_network.py and run it in your terminal:
python3 neural_network.py
  • Look at the terminal output after the training completes. You should see the accuracy climb from around 50% at epoch 0 to 95%+ by epoch 1600.

Why does it start at ~50%?

With random weights, the network is essentially guessing. Since your dataset has two equal classes, random guessing gives about 50% accuracy. The fact that it climbs to 95%+ means the network learned the circular pattern purely through gradient descent.

Accuracy stuck near 50%?

  • Make sure learning_rate is set to 1.0 (not 0.5 from the single neuron section).
  • Check that your initialize_network function uses hidden_size=8 and the weight scale is 0.5.
  • Verify you're updating all four parameters: W1, b1, W2, and b2.

Why is my neural network accuracy stuck at 50%?

Plot the training loss curve

A loss curve shows you whether the network is actually learning over time. A downward trend means gradient descent is working.

  • Add this code directly below the final accuracy print statements:
# Plot training loss
plt.plot(losses_network)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title(f'Neural Network Training (Accuracy: {accuracy_network:.1f}%)')
plt.show()

What does this code do?

This plots the losses_network list as a line chart. The x-axis is the epoch number and the y-axis is the loss value. You should see a curve that drops steeply at first, then flattens out as the network converges.

  • Save neural_network.py and run it again.
  • Close each plot window as it appears. After the training output, you should see a loss curve that drops steeply in the first few hundred epochs, then flattens near zero.
Visualize the decision boundary

The decision boundary is the most powerful visualization in this project. It shows you exactly how the network "thinks" by coloring every point in the space according to the network's prediction.

  • Add the plot_decision_boundary function below the loss plot code, then call it:
# Visualize decision boundary
def plot_decision_boundary(X, y, params):
    """See what the neural network learned"""
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5

    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                         np.linspace(y_min, y_max, 200))

    grid_points = np.c_[xx.ravel(), yy.ravel()]
    predictions, _ = forward(grid_points, params)
    predictions = predictions.reshape(xx.shape)

    plt.figure(figsize=(8, 8))
    plt.contourf(xx, yy, predictions, levels=50, cmap='RdBu_r', alpha=0.6)
    plt.scatter(X[y.flatten() == 0, 0], X[y.flatten() == 0, 1],
                c='blue', edgecolors='black', linewidth=0.5, label='Class 0')
    plt.scatter(X[y.flatten() == 1, 0], X[y.flatten() == 1, 1],
                c='red', edgecolors='black', linewidth=0.5, label='Class 1')
    plt.xlabel('Feature 1')
    plt.ylabel('Feature 2')
    plt.title('Neural Network Decision Boundary')
    plt.legend()
    plt.colorbar(label='Prediction Confidence')
    plt.axis('equal')
    plt.show()

plot_decision_boundary(X, y, params)

What does this code do?

The function creates a 200x200 grid of points covering the entire data space. It runs every single grid point through the trained network to get a prediction (0 to 1). Then it colors the grid with plt.contourf to create a smooth heatmap.

Blue regions are where the network predicts class 0. Red regions predict class 1. The transition between them IS the decision boundary. Your network learned a curved, roughly circular boundary without anyone telling it that circles exist.

  • Save neural_network.py and run it one more time.
  • Close the earlier plots as they appear. The final plot is the decision boundary. You should see a blue inner region surrounded by a red outer region, with your data points overlaid on top.

Decision boundary looks like random noise?

  • Make sure plot_decision_boundary(X, y, params) is called AFTER the training loop finishes, not before.
  • Check that the params variable passed to the function is the same one updated during training.

Why does my decision boundary look wrong?

Experiment with hidden size

The number of hidden neurons controls how complex a boundary the network can learn. More neurons mean more capacity to represent intricate shapes.

  • In neural_network.py, find this line:
params = initialize_network(input_size=2, hidden_size=8, output_size=1)
  • Change hidden_size=8 to hidden_size=2, then save and run your script again.
params = initialize_network(input_size=2, hidden_size=2, output_size=1)
  • Look at the decision boundary. With only 2 hidden neurons, the boundary is much simpler and accuracy drops.
  • Now change it to hidden_size=16 and run again. The boundary becomes smoother and more circular.

Why does hidden size matter?

Each hidden neuron can learn one "feature" of the data. With 2 neurons, the network can only combine two simple curves. With 8 or 16, it can piece together a much more complex boundary. This is exactly why modern AI uses millions of neurons.

  • Set hidden_size back to 8 before moving on.

✔️ Awesome, I've got everything!

Great. Make sure you've saved neural_network.py with hidden_size=8 restored.

ⓧ I'd like to double check the full code

import numpy as np
import matplotlib.pyplot as plt

# ============================================================
# PART 1: Create Your Dataset
# ============================================================

np.random.seed(42)

# Generate a circular dataset (non-linearly separable)
num_samples = 300

# Inner circle (class 0)
r_inner = np.random.uniform(0, 1, num_samples // 2)
theta_inner = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_inner = r_inner * np.cos(theta_inner)
y_inner = r_inner * np.sin(theta_inner)

# Outer ring (class 1)
r_outer = np.random.uniform(1.5, 2.5, num_samples // 2)
theta_outer = np.random.uniform(0, 2 * np.pi, num_samples // 2)
x_outer = r_outer * np.cos(theta_outer)
y_outer = r_outer * np.sin(theta_outer)

# Combine into dataset
X = np.vstack([
    np.column_stack([x_inner, y_inner]),
    np.column_stack([x_outer, y_outer])
])
y = np.array([0] * (num_samples // 2) + [1] * (num_samples // 2)).reshape(-1, 1)

# Shuffle
shuffle_idx = np.random.permutation(num_samples)
X = X[shuffle_idx]
y = y[shuffle_idx]

print(f"Dataset shape: {X.shape}")
print(f"Labels shape: {y.shape}")
print(f"First data point: {X[0]} -> Label: {y[0][0]}")

# Visualize
plt.figure(figsize=(8, 8))
plt.scatter(X[y.flatten() == 0, 0], X[y.flatten() == 0, 1],
            c='blue', label='Class 0 (inner)', alpha=0.6)
plt.scatter(X[y.flatten() == 1, 0], X[y.flatten() == 1, 1],
            c='red', label='Class 1 (outer)', alpha=0.6)
plt.xlabel('Feature 1 (x)')
plt.ylabel('Feature 2 (y)')
plt.title('Circular Dataset: Can You Draw a Straight Line to Separate These?')
plt.legend()
plt.axis('equal')
plt.grid(True, alpha=0.3)
plt.show()

# ============================================================
# PART 2: Build and Train a Single Neuron
# ============================================================

def sigmoid(z):
    """Squashes any number to range (0, 1)"""
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

# Initialize weights
w = np.random.randn(2, 1) * 0.01
b = 0.0

# Training loop for single neuron
learning_rate = 0.5
losses_single = []

for epoch in range(500):
    # Forward pass: z = X @ w + b, a = sigmoid(z)
    z = np.dot(X, w) + b
    a = sigmoid(z)

    # Compute loss (binary cross-entropy)
    epsilon = 1e-8
    loss = -np.mean(y * np.log(a + epsilon) + (1 - y) * np.log(1 - a + epsilon))
    losses_single.append(loss)

    # Backward pass (compute gradients)
    m = X.shape[0]
    dz = a - y
    dw = (1 / m) * np.dot(X.T, dz)
    db = (1 / m) * np.sum(dz)

    # Update weights (gradient descent!)
    w = w - learning_rate * dw
    b = b - learning_rate * db

# Check accuracy
predictions = (sigmoid(np.dot(X, w) + b) >= 0.5).astype(int)
accuracy_single = np.mean(predictions == y) * 100
print(f"\nSingle Neuron Accuracy: {accuracy_single:.1f}%")
print("A single neuron can only draw a STRAIGHT line. It cannot separate circles!")

# Plot loss
plt.plot(losses_single)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title(f'Single Neuron Training (Accuracy: {accuracy_single:.1f}%)')
plt.show()

# ============================================================
# PART 3: Build a Two-Layer Neural Network
# ============================================================

def initialize_network(input_size, hidden_size, output_size):
    """Initialize weights with small random values"""
    np.random.seed(42)
    params = {
        'W1': np.random.randn(input_size, hidden_size) * 0.5,
        'b1': np.zeros((1, hidden_size)),
        'W2': np.random.randn(hidden_size, output_size) * 0.5,
        'b2': np.zeros((1, output_size))
    }
    return params

def forward(X, params):
    """Forward pass through 2-layer network"""
    # Hidden layer
    z1 = np.dot(X, params['W1']) + params['b1']
    a1 = np.tanh(z1)

    # Output layer
    z2 = np.dot(a1, params['W2']) + params['b2']
    a2 = sigmoid(z2)

    cache = {'z1': z1, 'a1': a1, 'z2': z2, 'a2': a2}
    return a2, cache

def compute_loss(y_true, y_pred):
    """Binary cross-entropy loss"""
    epsilon = 1e-8
    return -np.mean(y_true * np.log(y_pred + epsilon) +
                    (1 - y_true) * np.log(1 - y_pred + epsilon))

def backward(X, y, params, cache):
    """Backward pass: chain rule in action"""
    m = X.shape[0]
    a1 = cache['a1']
    a2 = cache['a2']

    # Output layer gradients
    dz2 = a2 - y
    dW2 = (1 / m) * np.dot(a1.T, dz2)
    db2 = (1 / m) * np.sum(dz2, axis=0, keepdims=True)

    # Hidden layer gradients (this is backpropagation!)
    da1 = np.dot(dz2, params['W2'].T)
    dz1 = da1 * (1 - a1 ** 2)  # tanh derivative
    dW1 = (1 / m) * np.dot(X.T, dz1)
    db1 = (1 / m) * np.sum(dz1, axis=0, keepdims=True)

    return {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}

# ============================================================
# PART 4: Train and Visualize
# ============================================================

# Train the network
params = initialize_network(input_size=2, hidden_size=8, output_size=1)
learning_rate = 1.0
losses_network = []

for epoch in range(2000):
    # Forward
    y_pred, cache = forward(X, params)

    # Loss
    loss = compute_loss(y, y_pred)
    losses_network.append(loss)

    # Backward
    grads = backward(X, y, params, cache)

    # Update
    params['W1'] -= learning_rate * grads['dW1']
    params['b1'] -= learning_rate * grads['db1']
    params['W2'] -= learning_rate * grads['dW2']
    params['b2'] -= learning_rate * grads['db2']

    if epoch % 400 == 0:
        accuracy = np.mean((y_pred >= 0.5).astype(int) == y) * 100
        print(f"Epoch {epoch:4d} | Loss: {loss:.4f} | Accuracy: {accuracy:.1f}%")

# Final accuracy
y_pred, _ = forward(X, params)
accuracy_network = np.mean((y_pred >= 0.5).astype(int) == y) * 100
print(f"\nNeural Network Accuracy: {accuracy_network:.1f}%")
print(f"Single Neuron was: {accuracy_single:.1f}%")

# Plot training loss
plt.plot(losses_network)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title(f'Neural Network Training (Accuracy: {accuracy_network:.1f}%)')
plt.show()

# Visualize decision boundary
def plot_decision_boundary(X, y, params):
    """See what the neural network learned"""
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5

    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                         np.linspace(y_min, y_max, 200))

    grid_points = np.c_[xx.ravel(), yy.ravel()]
    predictions, _ = forward(grid_points, params)
    predictions = predictions.reshape(xx.shape)

    plt.figure(figsize=(8, 8))
    plt.contourf(xx, yy, predictions, levels=50, cmap='RdBu_r', alpha=0.6)
    plt.scatter(X[y.flatten() == 0, 0], X[y.flatten() == 0, 1],
                c='blue', edgecolors='black', linewidth=0.5, label='Class 0')
    plt.scatter(X[y.flatten() == 1, 0], X[y.flatten() == 1, 1],
                c='red', edgecolors='black', linewidth=0.5, label='Class 1')
    plt.xlabel('Feature 1')
    plt.ylabel('Feature 2')
    plt.title('Neural Network Decision Boundary')
    plt.legend()
    plt.colorbar(label='Prediction Confidence')
    plt.axis('equal')
    plt.show()

plot_decision_boundary(X, y, params)

Your neural network is trained and visualized. You've seen the full pipeline: data flows forward through matrix multiplications, loss measures the error, gradients flow backward through the chain rule, and weights update via gradient descent. This is the exact same loop that trains every AI model, from chatbots to image generators.

Secret mission

Classify a Spiral Dataset

Your 2-layer network crushed the circle problem. But can it handle two interleaving spirals? In this challenge, you'll generate a spiral dataset, discover why your current network fails, and build a deeper 3-layer network with ReLU activation to conquer it.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your file and installed packages, or clean them up.

Resources you used:

  • neural_network.py (local Python file).
  • Matplotlib (pip package installed during setup).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your neural network or want to reference the code later.

  • Your neural_network.py file stays where you saved it.
  • Matplotlib remains installed and available for other Python projects.

Pause - I'll come back to this later

Since this project is local-only with no running processes, there is nothing to pause. Your file and packages stay on disk until you choose to delete them.

  • No servers or background processes were started.
  • Come back any time by opening neural_network.py in your editor and running it again.

Delete - I don't want to use this again

Remove all project files and the packages you installed.

  • Delete your project file by running this command in the folder where you saved it:

macOS

rm neural_network.py

Windows

del neural_network.py
  • Uninstall Matplotlib by running:
pip3 uninstall matplotlib
  • Confirm the removal when prompted by typing y.

That's a Wrap!

That's a Wrap!

Nice work. You just built a neural network from absolute scratch. No TensorFlow, no PyTorch. Just you, NumPy, and math.

You've learned how to:

  • Implement gradient descent from scratch and train a neural network to 95%+ accuracy on non-linearly separable data.
  • Build forward propagation and backpropagation using matrix multiplication, derivatives, and the chain rule.
  • Visualize a decision boundary to see exactly how a neural network learns to separate complex data patterns.
  • Classified spiral data with a deeper 3-layer ReLU network using He initialization.

Ready to quiz yourself?