Linear Regression from Scratch

Build a gradient descent optimizer that teaches itself to predict housing prices.

Introduction

30 Second Summary

Every app that recommends songs, predicts weather, or suggests what to buy is running the same core loop under the hood. It starts clueless, measures how wrong it is, and nudges its numbers in the right direction until the predictions get better.

In this project, you will build that loop from scratch. You will implement linear regression with gradient descent using only NumPy, train it on real California housing data, and prove your hand-built model matches the industry-standard scikit-learn library.

What You'll Build

You will watch your model learn to predict housing prices in real-time, with a cost curve dropping on each iteration, and then compare your results against scikit-learn to prove they match exactly.

By the end of this project, you'll have:

  • A cost curve chart that drops iteration by iteration, visually proving your model is learning in real-time.
  • A scatter plot of actual vs. predicted housing prices showing how accurately your model predicts.
  • A printed side-by-side comparison of your from-scratch coefficients and scikit-learn's coefficients, proving they are virtually identical.
  • Secret Mission: Add learning rate experimentation and divergence detection to make your model production-aware.

Are there any prerequisites?

You should be comfortable with basic Python (variables, loops, functions, and NumPy arrays). Familiarity with the equation y = mx + b is all the math you need to get started.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters. Understanding the goal upfront turns every line of code from a copy-paste exercise into a deliberate step toward your ML engineering toolkit.

Set Up Your Notebook and Load the Data

Before writing any machine learning algorithms, you need real data to work with. You need to see the shape of that data and understand the patterns hiding in it.

In this step, you'll create a Jupyter Notebook, load the California Housing dataset, and visualize the relationship between median income and house prices.

In this step, get ready to:
  • Load the California Housing dataset into a pandas DataFrame.
  • Visualize the income-to-house-price relationship with a scatter plot.
  • Split the data into training and test sets.
Create the notebook and load the dataset

You need a project folder to keep your notebook organized, and a notebook file to write your code in.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir linear-regression-scratch
cd linear-regression-scratch
  • Confirm the folder exists by running:
ls

You should see an empty folder with no files listed yet.

  • Open VS Code in this folder by running:
code .

code command not found?

If code . does not work, open VS Code manually through your search bar, then click File then Open Folder and select the linear-regression-scratch folder on your Desktop.

help me set up the VS Code code command in my terminal

  • In VS Code, press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows) to open the Command Palette.
  • Type Create: New Jupyter Notebook and select it from the dropdown.
  • Save the file immediately by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Name the file linear_regression_scratch.ipynb and click Save.

You now have an empty notebook ready for your first cell.

The California Housing dataset comes bundled with scikit-learn. It contains 20,640 records of California census block groups, each with 8 features (like median income and average rooms) and a target value (median house price in $100,000s).

  • In the first cell of your notebook, paste the following code to import your libraries and load the data:
# Cell 1: Imports and Data Loading
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression

# Load dataset
housing = fetch_california_housing()
df = pd.DataFrame(housing.data, columns=housing.feature_names)
df['MedHouseVal'] = housing.target

print(df.shape)
print(df.head())
print(df.describe())

What does this code do?

  • The import block brings in NumPy for math operations, pandas for data handling, and matplotlib for plotting.
  • fetch_california_housing() downloads the dataset from scikit-learn's built-in collection.
  • You wrap the raw data in a DataFrame so you can explore it with .head(), .shape, and .describe().
  • The extra imports (StandardScaler, LinearRegression) are for later steps. Importing everything at the top keeps your notebook organized.
  • Run the cell by pressing Shift+Enter.

You should see (20640, 9) printed as the shape, followed by 5 rows of data and a statistical summary of all columns.

Seeing a ModuleNotFoundError?

This means a library is not installed in your current Python environment. Make sure your notebook kernel matches the Python where you installed scikit-learn, NumPy, and pandas.

Check the kernel selector in the top-right of VS Code and pick the correct Python interpreter.

help me fix a ModuleNotFoundError in my Jupyter Notebook

Visualize the income vs house value relationship

You'll start by modeling just one feature: MedInc (median income). A scatter plot reveals whether income and house value have a linear relationship worth modeling.

  • Create a new cell below your first one and paste this code:
# Cell 2: Visualize single feature relationship
X_single = df[['MedInc']].values
y = df['MedHouseVal'].values

plt.figure(figsize=(8, 5))
plt.scatter(X_single[:, 0], y, alpha=0.1, s=5)
plt.xlabel('Median Income (MedInc)')
plt.ylabel('Median House Value ($100k)')
plt.title('Income vs House Value')
plt.show()

What does this code do?

  • df[['MedInc']].values extracts median income as a 2D NumPy array (shape 20640x1). The double brackets keep it 2D, which is the shape your model will need later.
  • df['MedHouseVal'].values extracts the target (house prices) as a 1D array.
  • alpha=0.1 makes points semi-transparent so overlapping areas appear darker, revealing density patterns.
  • Run the cell by pressing Shift+Enter.

You should see a scatter plot with a clear upward trend. Higher income areas tend to have higher house values, which confirms a linear relationship worth modeling.

Scatter plot not showing?

Make sure you ran Cell 1 first. Cell 2 depends on the df DataFrame created in Cell 1.

If you see a blank output, check that plt.show() is the last line and that your notebook kernel is running.

help me debug why my matplotlib plot is not displaying in Jupyter

Split into training and test sets

Machine learning models need separate data for learning and for testing. The training set teaches the model, and the test set evaluates how well it generalizes to data it has never seen.

  • Create a new cell and paste this code:
# Cell 3: Train/test split (single feature first)
X_train_single, X_test_single, y_train, y_test = train_test_split(
    X_single, y, test_size=0.2, random_state=42
)
print(f"Training samples: {X_train_single.shape[0]}")
print(f"Test samples: {X_test_single.shape[0]}")

What does this code do?

  • train_test_split randomly shuffles the data and splits it into two groups: 80% for training and 20% for testing.
  • random_state=42 locks the shuffle so you get the same split every time you run the cell. This makes your results reproducible.
  • test_size=0.2 means 20% of rows go to the test set and 80% go to training.

Before you run this cell, how many training samples do you expect from 20,640 total rows with an 80/20 split?

  • Run the cell by pressing Shift+Enter.

You should see Training samples: 16512 and Test samples: 4128 printed. That matches the expected 80/20 split of 20,640 rows.

Seeing a NameError for X_single or y?

This means Cell 2 was not run before Cell 3. Jupyter cells must be run in order. Go back and run Cell 2 first, then re-run Cell 3.

help me understand why my Jupyter Notebook cell gives a NameError

✔️ Awesome, I've got everything!

Great. Make sure you've saved your notebook with Cmd+S (macOS) or Ctrl+S (Windows).

ⓧ I'd like to double check the full code

Your notebook should have these three cells in order:

  • Cell 1: Imports and Data Loading (imports all libraries, loads the dataset, prints shape/head/describe).
  • Cell 2: Visualize single feature relationship (extracts MedInc and MedHouseVal, displays scatter plot).
  • Cell 3: Train/test split (splits data 80/20, prints sample counts of 16512 and 4128).

Scroll up and confirm each cell matches the code you entered. If any cell is missing, add it now and run all cells in order from top to bottom.

Your data is loaded, visualized, and split. Next up, you'll write the cost function that measures how wrong your model's predictions are.

Build the Cost Function and Make Your First Prediction

Your data is loaded and split into training and test sets. Now you need to give the model a way to measure how wrong its predictions are.

This measurement is called a cost function. In this step, you will write one from scratch, make your first prediction with random parameters, and see just how terrible an untrained model performs.

In this step, get ready to:
  • Write a cost function that measures prediction error using Mean Squared Error.
  • Make your first prediction with untrained parameters and see how far off it is.
Define the cost function

Before a model can learn, it needs a single number that tells it how wrong it is. The Mean Squared Error (MSE) cost function does this by squaring the difference between each prediction and the actual value, then averaging across all samples.

  • Create a new cell in your notebook. Add the cost function and parameter initialization by typing this code:
# Cell 4: Cost function and initial (terrible) prediction
def compute_cost(X, y, w, b):
    m = len(y)
    predictions = X @ w + b
    cost = (1 / (2 * m)) * np.sum((predictions - y) ** 2)
    return cost

What does this code do?

  • m = len(y) counts the number of training samples so the cost is an average, not a sum that grows with dataset size.
  • X @ w + b computes predictions for all samples at once using matrix multiplication. This is the vectorized form of y = mx + b.
  • The (1 / (2 * m)) divides by 2m instead of just m. The factor of 2 is a convenience that simplifies the derivative when we implement gradient descent in the next step.
  • Below the function, still in the same cell, add the parameter initialization and initial cost computation:
# Initialize parameters
w = np.zeros(X_train_single.shape[1])
b = 0.0

# Compute initial cost
initial_cost = compute_cost(X_train_single, y_train, w, b)
print(f"Initial cost (untrained): {initial_cost:.4f}")

What does this code do?

  • w = np.zeros(X_train_single.shape[1]) sets the weight to zero. With weight 0 and bias 0, the model predicts 0 for every house price.
  • Calling compute_cost on these zeroed-out parameters gives you the "untrained" cost. A high cost means the model is very wrong.

Before you run this cell, what do you think the cost will be? Remember, the model is predicting 0 for every house while actual values range from 0 to 5 (in $100k units).

  • Run the cell by pressing Shift+Enter.

You should see a cost value around 4.5 printed. That high number confirms the model is completely wrong when it starts with zero parameters.

Seeing a NameError?

  • Make sure you ran the previous cells (Cells 1-3) first. The variables X_train_single and y_train come from Cell 3.
  • Check that import numpy as np is in Cell 1 and that Cell 1 has been run.

Why is my cost function returning a NameError?

Visualize how wrong the untrained model is

A cost of ~4.5 is abstract. A chart makes it visceral. You will plot the model's current "prediction" (a flat line at zero) against the actual training data.

  • Add the following plot code below the print statement, still in the same cell:
# Plot the terrible initial prediction
plt.figure(figsize=(8, 5))
plt.scatter(X_train_single[:, 0], y_train, alpha=0.1, s=5, label='Actual')
plt.axhline(y=0, color='red', linewidth=2, label='Prediction (flat line at 0)')
plt.xlabel('Median Income')
plt.ylabel('House Value ($100k)')
plt.title('Untrained Model: Predicts Zero for Everything')
plt.legend()
plt.show()

What does this plot show?

The blue dots are real house prices from the training data. The red horizontal line is the model's current prediction: zero for every single house, regardless of income.

The gap between the red line and the cloud of blue dots is exactly what the cost function measures. Gradient descent (next step) will tilt and shift that line upward to fit the data.

  • Run the cell again with Shift+Enter.

You should see the cost value printed, followed by a scatter plot with blue dots spread across the chart and a flat red line sitting at y=0. That red line is your model's terrible first guess.

Not seeing the plot?

  • Make sure plt.show() is the last line in the plotting section. Without it, some notebook environments suppress the chart.
  • Check that import matplotlib.pyplot as plt is in Cell 1 and that Cell 1 has been run.

Why isn't my matplotlib plot showing in Jupyter?

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here is the complete Cell 4. Make sure everything in your cell matches this exactly:

# Cell 4: Cost function and initial (terrible) prediction
def compute_cost(X, y, w, b):
    m = len(y)
    predictions = X @ w + b
    cost = (1 / (2 * m)) * np.sum((predictions - y) ** 2)
    return cost

# Initialize parameters
w = np.zeros(X_train_single.shape[1])
b = 0.0

# Compute initial cost
initial_cost = compute_cost(X_train_single, y_train, w, b)
print(f"Initial cost (untrained): {initial_cost:.4f}")

# Plot the terrible initial prediction
plt.figure(figsize=(8, 5))
plt.scatter(X_train_single[:, 0], y_train, alpha=0.1, s=5, label='Actual')
plt.axhline(y=0, color='red', linewidth=2, label='Prediction (flat line at 0)')
plt.xlabel('Median Income')
plt.ylabel('House Value ($100k)')
plt.title('Untrained Model: Predicts Zero for Everything')
plt.legend()
plt.show()

Your cost function works, and you can see exactly how bad the starting point is. Next up, you will implement gradient descent to teach the model how to reduce that cost and tilt the red line up toward the data.

Implement Gradient Descent and Watch the Model Learn

In the last step, you measured how wrong your model is. With a cost of ~4.5 and a flat line at zero, the model is completely useless.

Now you need the algorithm that actually improves those weights. Gradient descent is the core optimization engine behind every neural network and ML model. It computes which direction to adjust the weights to reduce the cost, then takes a small step in that direction, over and over.

In this step, get ready to:
  • Write the gradient computation function that tells the model which direction to move.
  • Build the full gradient descent loop that iteratively improves your weights.
  • Run the optimizer and watch a cost curve drop in real-time as your model learns.
Write the gradient computation function

The gradient tells the model which direction to nudge the weights to reduce cost. It is the partial derivative of the cost function with respect to each parameter.

  • Create a new cell in your notebook (Cell 5) and define the compute_gradients function by entering this code:
# Cell 5: Gradient computation
def compute_gradients(X, y, w, b):
    m = len(y)
    predictions = X @ w + b
    error = predictions - y
    dw = (1 / m) * (X.T @ error)
    db = (1 / m) * np.sum(error)
    return dw, db

What does this code do?

This function computes how much to adjust each parameter to reduce the cost.

  • error is the difference between predictions and actual values. Positive error means the model predicted too high.
  • dw tells you how much the weight contributes to the error. The X.T @ operation correlates each feature with the error across all samples.
  • db is the average error. It tells the bias which direction to shift.
  • Run the cell. It executes without error, confirming your function is defined and ready to use.

Seeing a NameError?

  • Make sure you ran Cells 1-4 first. The function uses np from the NumPy import in Cell 1.
  • Check that X.T uses a capital T (transpose), not lowercase.

Why isn't my compute_gradients function working?

Build the gradient descent loop

A single gradient computation tells the model which way to move. The gradient descent loop repeats this hundreds or thousands of times, taking a small step each iteration until the cost stops decreasing.

  • Create a new cell (Cell 6) and define the gradient_descent function by entering this code:
# Cell 6: Gradient descent implementation
def gradient_descent(X, y, w, b, learning_rate, num_iterations):
    cost_history = []

    for i in range(num_iterations):
        dw, db = compute_gradients(X, y, w, b)
        w = w - learning_rate * dw
        b = b - learning_rate * db
        cost = compute_cost(X, y, w, b)
        cost_history.append(cost)

    return w, b, cost_history

What does this code do?

This is the full learning loop. Each iteration, the model gets slightly better.

  • learning_rate controls how big each step is. Too large and the model overshoots. Too small and it takes forever.
  • w = w - learning_rate * dw moves the weight in the opposite direction of the gradient. This is the core update rule.
  • cost_history records the cost at every iteration so you can plot the model's progress.
  • Run the cell. It executes without error, confirming both your gradient descent function and its reference to your earlier functions are valid.

Seeing a NameError for compute_gradients or compute_cost?

  • Both compute_gradients (Cell 5) and compute_cost (Cell 4) must have been run before this cell. Go back and run them if needed.
  • Check that the function names match exactly, including the underscore.

Help me fix my gradient descent function

Run gradient descent and watch the model learn

Your optimizer is ready. Time to let it loose on the training data and watch the cost drop iteration by iteration.

  • Create a new cell (Cell 7) that resets the parameters, runs gradient descent, and plots the result:
# Cell 7: Run gradient descent on single feature
w = np.zeros(X_train_single.shape[1])
b = 0.0

w, b, cost_history = gradient_descent(
    X_train_single, y_train, w, b,
    learning_rate=0.01, num_iterations=1000
)

# Plot cost curve
plt.figure(figsize=(8, 5))
plt.plot(cost_history)
plt.xlabel('Iteration')
plt.ylabel('Cost (MSE / 2)')
plt.title('Cost Curve: Model Learning Over Time')
plt.show()

print(f"Final cost: {cost_history[-1]:.4f}")
print(f"Learned weight: {w}")
print(f"Learned bias: {b:.4f}")

What does this code do?

  • The parameters are reset to zeros so the model starts from scratch.
  • learning_rate=0.01 means each step adjusts the weights by 1% of the gradient. This keeps the learning stable.
  • num_iterations=1000 gives the model 1000 chances to improve. The cost curve shows whether that was enough.
  • The plot visualizes the model's learning progress. A healthy training run shows a curve that drops steeply at first, then flattens as it converges.

Before you run this cell, what do you think will happen to the cost over 1000 iterations? Will it go up, go down, or stay flat?

  • Run the cell.

You should see a cost curve that starts high (around 4.5, where you left off in the last step) and drops rapidly, flattening out as it approaches a value below 0.3. The printed output shows your final cost and the learned weight and bias values.

What should I see?

The cost curve is your proof that the model is learning. The steep initial drop means the model improved rapidly in the first few hundred iterations. The flattening means it has nearly converged.

Your final cost should be below 0.3. Compare that to the ~4.5 you started with. The model went from predicting zero for everything to making reasonable predictions about house prices.

Cost isn't decreasing or the plot looks wrong?

  • If the cost stays flat or increases, check that your update rule uses subtraction: w = w - learning_rate * dw. Using + instead of - makes the model worse every iteration.
  • If you get a shape error, confirm X_train_single is 2D (shape (16512, 1) not (16512,)). Re-run Cell 3 if needed.
  • If the cost is NaN or infinity, your learning rate might be too high. Confirm it is 0.01, not 0.1.

My gradient descent cost curve doesn't look right. Help me debug.

✔️ Awesome, I've got everything!

Great. Make sure your notebook is saved before moving on.

ⓧ I'd like to double check the full code

Scroll through your notebook and confirm you have these cells in order:

  • Cells 1-4 from the previous steps (imports, scatter plot, train/test split, cost function with initial prediction).
  • Cell 5 defines compute_gradients(X, y, w, b) returning dw and db.
  • Cell 6 defines gradient_descent(X, y, w, b, learning_rate, num_iterations) returning w, b, and cost_history.
  • Cell 7 resets parameters, runs gradient descent with learning_rate=0.01 and num_iterations=1000, plots the cost curve, and prints the final cost, weight, and bias.

Your model just taught itself to predict house prices from a single feature. The cost dropped from ~4.5 to below 0.3 in 1000 iterations. Next up, you will unlock all 8 features and discover why raw data breaks gradient descent.

Fix Training with Feature Scaling

Your gradient descent is working. You proved it on a single feature and watched the cost curve drop. But real-world models use many features, not just one.

The California Housing dataset has 8 features with wildly different scales. MedInc ranges from 0 to 15, while Population ranges from 0 to hundreds of thousands. In this step, you will experience firsthand what happens when gradient descent meets unscaled data, then fix it.

In this step, get ready to:
  • Run gradient descent on all 8 raw features and watch it struggle.
  • Apply feature scaling with StandardScaler to fix training.
  • Evaluate your model's accuracy with R-squared.
Run gradient descent on all 8 features

So far you have trained on one feature. The dataset has 8 features that could all contribute to better predictions. Time to use them all.

Before you run this, think: you used a learning rate of 0.01 on the single feature and it worked beautifully. Do you think the same learning rate will work on 8 features with different scales?

  • Create a new cell in your notebook and add this code to switch to all 8 features and run gradient descent on them:
# Cell 8: Switch to all 8 features - experience the scaling problem
X_all = housing.data
X_train_all, X_test_all, y_train, y_test = train_test_split(
    X_all, y, test_size=0.2, random_state=42
)

# Try gradient descent on raw (unscaled) features
w_raw = np.zeros(X_train_all.shape[1])
b_raw = 0.0

w_raw, b_raw, cost_history_raw = gradient_descent(
    X_train_all, y_train, w_raw, b_raw,
    learning_rate=0.00001, num_iterations=100
)

print(f"Cost after 100 iterations (unscaled): {cost_history_raw[-1]:.4f}")
print("The cost barely moved! Features have wildly different scales.")

What does this code do?

You extract all 8 features from housing.data and create a new train/test split. Notice the learning rate is set to 0.00001 (100x smaller than before) and only 100 iterations. Even with these concessions, gradient descent will still struggle.

The problem is that features like Population are thousands of times larger than features like AveRooms. The gradient for the large-scale feature dominates, causing the optimizer to zig-zag instead of converging.

  • Run the cell.

You will see the cost printed, but it has barely decreased from the initial value. The model learned almost nothing in 100 iterations. This is the scaling problem in action.

Why does this happen?

When features have wildly different magnitudes, the cost function forms an elongated bowl shape instead of a nice circular one. Gradient descent bounces off the steep walls (large-scale features) while barely moving along the shallow floor (small-scale features).

This is not a bug in your code. It is a fundamental property of gradient descent that every ML practitioner must handle.

Scale the features with StandardScaler

The fix is feature scaling. You normalize every feature to have zero mean and unit variance, so no single feature dominates the gradient.

  • Create a new cell and add this code to scale your features and re-run gradient descent:
# Cell 9: Fix with feature scaling
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train_all)
X_test_scaled = scaler.transform(X_test_all)

# Re-run gradient descent on scaled data
w = np.zeros(X_train_scaled.shape[1])
b = 0.0

w, b, cost_history = gradient_descent(
    X_train_scaled, y_train, w, b,
    learning_rate=0.01, num_iterations=1000
)

# Plot cost curve
plt.figure(figsize=(8, 5))
plt.plot(cost_history)
plt.xlabel('Iteration')
plt.ylabel('Cost (MSE / 2)')
plt.title('Cost Curve: All 8 Features (Scaled)')
plt.show()

print(f"Final cost: {cost_history[-1]:.4f}")

What does this code do?

  • fit_transform on the training data computes the mean and standard deviation of each feature, then scales the values. on the training data computes the mean and standard deviation of each feature, then scales the values.
  • transform on the test data applies the same mean and std from training. This prevents data leakage from the test set. on the test data applies the same mean and std from training. This prevents data leakage from the test set.
  • Now you can use the same learning rate of 0.01 and 1000 iterations that worked on the single feature. and 1000 iterations that worked on the single feature.
  • Run the cell.

You should see a smooth cost curve that drops from around 2.3 down to around 0.26. The final cost is printed below the chart.

Cost curve not dropping?

  • Make sure you ran Cell 8 first. Cell 9 depends on X_train_all and X_test_all being defined.
  • Check that your learning rate is 0.01 (not the 0.00001 from the unscaled attempt). (not the 0.00001 from the unscaled attempt).
  • Confirm your compute_cost and compute_gradients functions are defined in earlier cells and those cells have been run.

Why isn't my scaled gradient descent converging?

Calculate your R-squared score

The cost curve proves the model is learning, but you need a score that measures how well it predicts on unseen test data. R-squared tells you what fraction of the variation in house prices your model explains. A score of 1.0 means perfect predictions. A score of 0.0 means your model is no better than guessing the average.

  • Create a new cell and add this code to compute predictions and calculate R-squared:
# Cell 10: Evaluate with R-squared
y_pred = X_test_scaled @ w + b
ss_res = np.sum((y_test - y_pred) ** 2)
ss_tot = np.sum((y_test - y_test.mean()) ** 2)
r2_scratch = 1 - ss_res / ss_tot

print(f"R-squared (from scratch): {r2_scratch:.4f}")

What does this code do?

  • y_pred generates predictions on the test set using your learned weights and bias.
  • ss_res is the residual sum of squares. It measures how far your predictions are from actual values.
  • ss_tot is the total sum of squares. It measures how far actual values are from their mean.
  • R-squared is 1 minus the ratio of these two. The closer to 1, the better your model explains the data.

Before you run this, what R-squared do you expect? Remember, 1.0 is perfect and 0.0 means no better than guessing the mean.

  • Run the cell.

You should see an R-squared around 0.60. Your from-scratch model explains about 60% of the variation in California house prices using 8 features.

R-squared much lower than 0.60?

  • Confirm you used the scaled test data (X_test_scaled) and not the raw X_test_all for predictions.) and not the raw X_test_all for predictions.
  • Make sure Cell 9 finished running with 1000 iterations. If you interrupted it early, the weights are not fully trained.

Why is my R-squared score low?

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook with all cells through Cell 10.

ⓧ I'd like to double check the full code

Your notebook should now contain Cells 1 through 10. Scroll through and confirm these cells are present in order:

  • Cell 1: Imports and Data Loading.
  • Cell 2: Visualize single feature relationship.
  • Cell 3: Train/test split (single feature first).
  • Cell 4: Cost function and initial (terrible) prediction.
  • Cell 5: Gradient computation.
  • Cell 6: Gradient descent implementation.
  • Cell 7: Run gradient descent on single feature.
  • Cell 8: Switch to all 8 features - experience the scaling problem.
  • Cell 9: Fix with feature scaling.
  • Cell 10: Evaluate with R-squared.

If any cell is missing, scroll up in this step and copy the code for that cell.

Your model trains on all 8 features and scores 0.60 on unseen data. But how do you know your from-scratch implementation is actually correct? Next up, you will put it head-to-head against scikit-learn's production-quality LinearRegression and see if the numbers match.

Compare Your Model Against scikit-learn

You have a working gradient descent model with an R-squared of about 0.60. But how do you know your implementation actually found the right answer?

The ultimate proof is comparing your results against a trusted industry tool. In this step, you will train scikit-learn's LinearRegression on the exact same data and compare every number side by side.

In this step, get ready to:
  • Train scikit-learn's LinearRegression on the same scaled training data.
  • Compare coefficients, intercepts, and R-squared scores between both models.
  • Create a scatter plot proving both models produce nearly identical predictions.
Train scikit-learn and compare the numbers

If your from-scratch coefficients and R-squared match scikit-learn's output, you have genuinely implemented the same optimization algorithm that powers production ML tools.

Before you run this cell, think: do you expect your learned weights to match scikit-learn exactly, or will there be small differences? Why?

  • Create a new cell in your notebook (Cell 11). Train scikit-learn's LinearRegression on the same scaled data and print the comparison by pasting this code:
# Cell 11: Compare with scikit-learn
sklearn_model = LinearRegression()
sklearn_model.fit(X_train_scaled, y_train)

r2_sklearn = sklearn_model.score(X_test_scaled, y_test)

print("=== Comparison ===")
print(f"R-squared (scratch):  {r2_scratch:.4f}")
print(f"R-squared (sklearn):  {r2_sklearn:.4f}")
print(f"\nCoefficients (scratch): {np.round(w, 4)}")
print(f"Coefficients (sklearn): {np.round(sklearn_model.coef_, 4)}")
print(f"\nIntercept (scratch): {b:.4f}")
print(f"Intercept (sklearn): {sklearn_model.intercept_:.4f}")

What does this code do?

You train scikit-learn's LinearRegression on the exact same X_train_scaled and y_train data your gradient descent used.

  • sklearn_model.score() returns R-squared, the same metric you computed manually in the previous step.
  • sklearn_model.coef_ holds the 8 learned weights (equivalent to your w array).
  • sklearn_model.intercept_ is the bias term (equivalent to your b).
  • Run the cell.

Both R-squared values should be within 0.01 of each other (around 0.60). The coefficient arrays and intercepts should be nearly identical. Your gradient descent found the same answer as scikit-learn's optimized solver.

Why aren't they perfectly identical?

Scikit-learn uses the normal equation (a closed-form linear algebra solution) internally, while your implementation uses iterative gradient descent. With enough iterations, gradient descent converges to the same answer, but tiny floating-point differences remain.

Seeing large differences between the values?

  • Make sure you trained scikit-learn on the scaled data (X_train_scaled), not the raw unscaled data (X_train_all).
  • Confirm your gradient descent ran for 1000 iterations with a learning rate of 0.01. Fewer iterations means the model has not fully converged.
  • Check that you are comparing the w and b from Cell 9 (the scaled run), not from Cell 7 (the single-feature run).

Help me debug why my from-scratch model doesn't match scikit-learn's output.

Create the final comparison plot

Numbers matching is convincing. A visual where both models' predictions overlap perfectly is undeniable proof.

  • Create a new cell (Cell 12). Generate predictions from both models and plot them against actual values by pasting this code:
# Cell 12: Final comparison plot
y_pred_scratch = X_test_scaled @ w + b
y_pred_sklearn = sklearn_model.predict(X_test_scaled)

plt.figure(figsize=(8, 6))
plt.scatter(y_test, y_pred_scratch, alpha=0.3, s=10, color='blue', label='From Scratch')
plt.scatter(y_test, y_pred_sklearn, alpha=0.3, s=10, color='red', marker='x', label='scikit-learn')
plt.plot([0, 5], [0, 5], 'k--', linewidth=1, label='Perfect prediction')
plt.xlabel('Actual House Value ($100k)')
plt.ylabel('Predicted House Value ($100k)')
plt.title('From Scratch vs scikit-learn: Predictions Match')
plt.legend()
plt.show()

What does this plot show?

  • Blue dots are your from-scratch model's predictions plotted against actual house values.
  • Red x markers are scikit-learn's predictions on the same test data.
  • The dashed diagonal line represents perfect prediction (where predicted equals actual).
  • If the blue and red points overlap, both models are producing the same predictions.
  • Run the cell.

You should see the blue dots and red x markers overlapping almost perfectly, clustered around the diagonal line. The two models are making identical predictions.

Plot looks empty or only shows one color?

  • If you only see red x markers, the blue dots are hidden underneath. That actually confirms the models match perfectly. Try swapping the plot order (plot sklearn first, scratch second) to verify.
  • If the plot is empty, check that y_test and y_pred_scratch are not empty arrays. Print their shapes to confirm.

Help me fix my comparison scatter plot.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook with both Cell 11 and Cell 12 completed.

ⓧ I'd like to double check the full code

Your notebook now has 12 cells. Scroll through and confirm each one is present in order:

  • Cell 1: Imports and data loading.
  • Cell 2: Single feature scatter plot.
  • Cell 3: Train/test split.
  • Cell 4: Cost function and initial prediction.
  • Cell 5: Gradient computation function.
  • Cell 6: Gradient descent implementation.
  • Cell 7: Run gradient descent on single feature with cost curve plot.
  • Cell 8: All 8 features on unscaled data (the designed obstacle).
  • Cell 9: Feature scaling fix with cost curve.
  • Cell 10: R-squared evaluation.
  • Cell 11: Compare with scikit-learn (coefficients, intercept, R-squared).
  • Cell 12: Final comparison scatter plot.

Your hand-built gradient descent produces the same predictions as scikit-learn's production-quality solver. You now understand the core optimization loop that powers every neural network and deep learning model. Next up, a secret mission to make your model production-aware with learning rate experimentation.

Secret mission

Add Learning Rate Experimentation and Divergence Detection

In real ML work, picking the wrong learning rate can make your model explode or crawl. Can you upgrade your gradient descent to detect when things go wrong and auto-stop before the math blows up?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • Jupyter Notebook file linear_regression_scratch.ipynb on your Desktop.

Keep everything running

No action needed. Choose this if you plan to revisit the notebook, experiment with different learning rates, or extend the model further.

  • Your linear_regression_scratch.ipynb file stays on your Desktop, ready to reopen any time.
  • All Python packages (NumPy, pandas, matplotlib, scikit-learn) remain installed and available for other projects.

Pause - I'll come back to this later

Shut down any running Jupyter processes to free up memory. Your files stay on disk for when you return.

  • If you have a Jupyter server running in a terminal, press Ctrl+C to stop it.
  • Close any open notebook tabs in VS Code or your browser.
  • Your linear_regression_scratch.ipynb file remains on your Desktop. Reopen it whenever you are ready to continue.

Delete - I don't want to use this again

Remove the project notebook from your machine entirely.

  • If you have a Jupyter server running, press Ctrl+C in the terminal to stop it.
  • Delete the notebook file by running this command in your terminal:

macOS

rm ~/Desktop/linear_regression_scratch.ipynb

Windows

Remove-Item ~\Desktop\linear_regression_scratch.ipynb
  • Your Python packages (NumPy, pandas, matplotlib, scikit-learn) remain installed. They are useful for other projects, so there is no need to uninstall them.

Nice Work!

Nice Work!

You just built a linear regression model from scratch using nothing but NumPy, and proved it produces the same results as scikit-learn's production-quality implementation.

You've learned how to:

  • Implement gradient descent from scratch, the core optimization algorithm that powers every neural network and deep learning model.
  • Diagnose and fix the feature scaling problem using StandardScaler, a real issue that trips up every ML pipeline working with multi-feature data.
  • Validate a from-scratch implementation against scikit-learn's LinearRegression by comparing coefficients, intercepts, and R-squared scores side by side.
  • Secret Mission: Added divergence detection and learning rate experimentation, a critical skill for tuning any ML model in production.

Ready to quiz yourself?