Real Estate Prediction: XGBoost vs LSTM

Train XGBoost and LSTM models to predict house prices and compare them.

Introduction

30 Second Summary

Every home buyer wonders whether a house is actually worth the asking price. Behind the scenes, real estate platforms use machine learning models to estimate property values from dozens of factors.

In this project, you will build two competing models that predict house prices from real sales data. One uses XGBoost to learn from tabular features, while the other uses an LSTM neural network to capture price trends over time.

What You'll Build

You will run a Google Colab notebook that predicts house prices two different ways from 21,613 real King County sales records, then compare the models side by side with explainability charts that show exactly why each prediction was made.

By the end of this project, you'll have:

  • A trained XGBoost model with engineered features like property age and living space ratios that dramatically outperforms a simple linear regression baseline on individual house price predictions.
  • An LSTM neural network that forecasts monthly price trends from sequential sales data, with a line chart showing its predictions tracking the real market.
  • A side-by-side metrics comparison of all three models across RMSE, MAE, and R-squared, plus SHAP waterfall and beeswarm charts that reveal exactly which features drive each XGBoost prediction.
  • Secret Mission: Build a hybrid XGBoost-LSTM ensemble that feeds XGBoost feature predictions as inputs to the LSTM, combining both models' strengths.

Are there any prerequisites?

You should be comfortable with basic Python syntax and understand what a train/test split does. You will also need a free Kaggle account to download the dataset used in this project.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Colab Notebook and Load the Data

Every machine learning project starts with two things: a working environment and real data to work with. Before you can explore housing prices or train any models, you need a notebook with the right libraries and a loaded dataset.

In this step, you will open a Google Colab notebook, verify that all the libraries you need are pre-installed, download the King County House Sales dataset from Kaggle, and confirm the data looks correct.

In this step, get ready to:
  • Open a Colab notebook with a T4 GPU runtime and verify library versions.
  • Install SHAP and kagglehub, then download the King County dataset.
  • Inspect the dataset to confirm its shape, data types, and value ranges.
Open a Colab notebook and verify pre-installed libraries

Google Colab comes with most popular data science libraries pre-installed. You just need to confirm they are available and check their versions.

Training the LSTM model later will benefit from GPU acceleration. Let's switch the runtime to use a T4 GPU.

  • In the top menu, click Runtime.
  • Select Change runtime type.
  • Under Hardware accelerator, choose T4 GPU.
  • Click Save.

Now let's verify the pre-installed library versions. Paste and run the following code in the first cell of your notebook.

  • Import all the libraries this project uses and print their versions by pasting this code into the first cell:
# Cell 1: Setup and Version Check
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import xgboost as xgb
import tensorflow as tf
from tensorflow import keras
from keras import layers
import warnings
warnings.filterwarnings('ignore')
  • Below the imports in the same cell, add the version print statements:
print(f"pandas: {pd.__version__}")
print(f"scikit-learn: {__import__('sklearn').__version__}")
print(f"TensorFlow: {tf.__version__}")
print(f"XGBoost: {xgb.__version__}")
print(f"NumPy: {np.__version__}")
  • Click the play button (or press Shift+Enter) to run the cell.

What does this code do?

The import block loads every library you will use across the entire project in one go. This includes pandas for data handling, scikit-learn for baseline modeling, TensorFlow/Keras for the LSTM, and XGBoost for gradient-boosted trees.

The warnings.filterwarnings('ignore') line suppresses non-critical deprecation warnings so your output stays clean.

You should see five version numbers printed, similar to pandas: 2.2.2, TensorFlow: 2.20.0, and XGBoost: 3.3.0. The exact versions depend on the current Colab runtime.

Not seeing version output?

  • Make sure you clicked the play button on the cell or pressed Shift+Enter. The cell should show a green checkmark when finished.
  • If you see a ModuleNotFoundError, confirm you selected the T4 GPU runtime. Some modules behave differently on CPU-only runtimes.
  • ask me why my Colab cell isn't producing output
Install SHAP and kagglehub, then download the dataset

Most libraries come pre-installed on Colab, but SHAP (for model explainability) and kagglehub (for downloading Kaggle datasets) need to be installed separately.

  • Create a new code cell and install both packages by running:
!pip install shap kagglehub

You should see installation logs scroll by. When they finish, both packages are ready to use.

What is kagglehub?

kagglehub is Kaggle's official Python library for downloading datasets directly into your notebook. It handles authentication and file management so you can load datasets with a single function call.

Before you can download the dataset, you need to connect your Kaggle account to Colab. The kagglehub library reads your credentials from a Colab secret named KAGGLE_API_TOKEN.

  • If you do not have a Kaggle account, sign up for free at kaggle.com.
  • Once signed in, go to kaggle.com/settings/api.
  • Click the Generate New Token button. This copies your API token to your clipboard.

Now store this token as a Colab secret so kagglehub can find it automatically.

  • In your Colab notebook, click the key icon in the left sidebar to open the Secrets panel.
  • Click Add a new secret.
  • Set the Name to KAGGLE_API_TOKEN.
  • Paste your Kaggle API token into the Value field.
  • Enable the Notebook access toggle so your current notebook can read this secret.

Keep your token private

Colab secrets are stored securely by Google and are not included when you share your notebook. Other users will need to create their own KAGGLE_API_TOKEN secret to run the download cell.

With your credentials in place, you can now download the King County House Sales dataset. This dataset contains 21,613 home sales from May 2014 to May 2015 in King County, Washington (which includes Seattle).

  • Create a new code cell and download the dataset by pasting this code:
# Cell 2: Install SHAP and kagglehub, Load Data
# !pip install shap kagglehub
import kagglehub
import os

path = kagglehub.dataset_download('harlfoxem/housesalesprediction')
csv_path = os.path.join(path, 'kc_house_data.csv')
df = pd.read_csv(csv_path)
print(f"Dataset shape: {df.shape}")
df.head()
  • Run the cell by pressing Shift+Enter.

What does this code do?

  • kagglehub.dataset_download('harlfoxem/housesalesprediction') downloads the dataset to a local cache directory and returns the file path.
  • os.path.join(path, 'kc_house_data.csv') builds the full path to the CSV file inside the downloaded folder.
  • pd.read_csv() loads the CSV into a pandas DataFrame called df.
  • df.head() displays the first five rows so you can see the actual data.

You should see Dataset shape: (21613, 21) printed, followed by a table showing columns like id, date, price, bedrooms, sqft_living, and more.

Download failing with an authentication error?

  • Double-check that your Colab secret is named exactly KAGGLE_API_TOKEN (all caps, with underscores). The name must match exactly.
  • Make sure the Notebook access toggle is enabled for this notebook in the Secrets panel.
  • If your token has expired, go back to kaggle.com/settings/api, click Generate New Token, and update the secret value.
  • help me fix my kagglehub authentication error in Colab
Inspect the data to confirm it loaded correctly

Before doing any modeling, you need to confirm the data is clean and contains the columns you expect. A quick inspection reveals data types, null values, and value ranges.

  • Create a new code cell and inspect the DataFrame by running:
# Cell 3: Data Inspection
print(df.info())
print(df.describe())
print(f"Null values:\n{df.isnull().sum().sum()}")
print(f"Price range: ${df['price'].min():,.0f} - ${df['price'].max():,.0f}")

What does this code do?

  • df.info() prints each column's name, data type, and non-null count. This tells you whether any columns have missing data.
  • df.describe() shows summary statistics (mean, min, max, percentiles) for every numeric column.
  • df.isnull().sum().sum() counts the total number of missing values across the entire DataFrame.
  • The price range line formats the minimum and maximum values of the price column as dollar amounts.

Before you run this cell, what do you expect the null value count to be for a well-maintained Kaggle dataset?

  • Run the cell by pressing Shift+Enter.

You should see Null values: 0 confirming no missing data. The price range should show approximately $75,000 - $7,700,000. That wide range (from starter homes to luxury estates) is exactly why a simple linear model will struggle.

Seeing a NameError for df?

  • This means the previous cell (Cell 2) did not run successfully. Go back and re-run it first.
  • Colab cells must be run in order. If you restarted the runtime, re-run all cells from Cell 1.
  • help me fix a NameError in my Colab notebook

✔️ Awesome, I've got everything!

Great. Double check you have three cells in your notebook (Cell 1, Cell 2, Cell 3) and that all have run successfully with output.

ⓧ I'd like to double check the full code

Here is the complete code across all three cells so far. Compare it against your notebook to make sure nothing is missing.

Cell 1:

# Cell 1: Setup and Version Check
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import xgboost as xgb
import tensorflow as tf
from tensorflow import keras
from keras import layers
import warnings
warnings.filterwarnings('ignore')

print(f"pandas: {pd.__version__}")
print(f"scikit-learn: {__import__('sklearn').__version__}")
print(f"TensorFlow: {tf.__version__}")
print(f"XGBoost: {xgb.__version__}")
print(f"NumPy: {np.__version__}")

Cell 2:

# Cell 2: Install SHAP and kagglehub, Load Data
# !pip install shap kagglehub
import kagglehub
import os

path = kagglehub.dataset_download('harlfoxem/housesalesprediction')
csv_path = os.path.join(path, 'kc_house_data.csv')
df = pd.read_csv(csv_path)
print(f"Dataset shape: {df.shape}")
df.head()

Cell 3:

# Cell 3: Data Inspection
print(df.info())
print(df.describe())
print(f"Null values:\n{df.isnull().sum().sum()}")
print(f"Price range: ${df['price'].min():,.0f} - ${df['price'].max():,.0f}")

Your environment is locked in and your dataset is loaded. Next up, you will explore the data visually and build a baseline model to see how well a simple approach performs.

Explore the Data and Build a Baseline Model

You have 21,613 rows of housing data loaded, but raw numbers alone do not reveal patterns. Before jumping to a complex model, you need to see how prices are distributed and which features influence them most.

You also need a simple model to set a benchmark. A linear regression baseline tells you the floor. If a fancier model cannot beat that floor, it is not adding value.

In this step, get ready to:
  • Create visualizations that reveal the price distribution, top correlated features, and the relationship between living space and price.
  • Prepare a train/test split with standardized features.
  • Train a linear regression baseline and measure its accuracy with R-squared, RMSE, and MAE.
Visualize price distribution and feature correlations

Before modeling, you need to understand the shape of your data. A price histogram reveals whether prices are evenly spread or skewed. A correlation bar chart highlights which columns are most predictive of price.

  • Create a new code cell below your data inspection cell and paste the following code to generate three EDA charts side by side:
# Cell 4: Exploratory Data Analysis
fig, axes = plt.subplots(1, 3, figsize=(18, 5))

axes[0].hist(df['price'], bins=50, edgecolor='black', alpha=0.7)
axes[0].set_title('Price Distribution')
axes[0].set_xlabel('Price ($)')
axes[0].set_ylabel('Count')

numeric_cols = df.select_dtypes(include=[np.number]).columns
corr_with_price = df[numeric_cols].corr()['price'].drop('price').sort_values(ascending=False)
corr_with_price.head(10).plot(kind='bar', ax=axes[1], color='steelblue')
axes[1].set_title('Top 10 Features Correlated with Price')
axes[1].set_ylabel('Correlation')

axes[2].scatter(df['sqft_living'], df['price'], alpha=0.3, s=5)
axes[2].set_title('Living Space vs Price')
axes[2].set_xlabel('sqft_living')
axes[2].set_ylabel('Price ($)')

plt.tight_layout()
plt.show()

What does this code do?

  • The first panel (axes[0]) draws a histogram of price values. A right-skewed shape means most homes sell below the median while a few luxury sales pull the average up.
  • The second panel (axes[1]) computes the Pearson correlation of every numeric column against price, then plots the top 10 as a bar chart.
  • The third panel (axes[2]) is a scatter plot of sqft_living vs price. Notice the upward trend is not a straight line. Prices fan out at higher square footages, hinting at non-linear relationships.
  • Run the cell by pressing Shift+Enter.

You should see three charts appear below the cell. The left histogram is right-skewed, the middle bar chart shows sqft_living, grade, and sqft_above as the top correlators, and the right scatter plot fans out at higher square footages.

Charts not rendering?

  • Make sure you ran Cell 1 first. The matplotlib and numpy imports need to be loaded before this cell can use them.
  • Check that df exists by running df.shape in a scratch cell. If it raises a NameError, re-run Cell 2 to reload the dataset.
  • Still stuck? Help me debug why my EDA charts are not rendering in Google Colab.
Prepare features and split the data

The id column is a database identifier with no predictive value. The date column is a string that needs special handling (you will use it later for the LSTM). Dropping both gives you clean numeric features.

  • Create a new code cell and paste the following code to prepare the features, split the data, and standardize it:
# Cell 5: Prepare Features and Train Baseline
X = df.drop(columns=['id', 'date', 'price'])
y = df['price']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

What does this code do?

  • X holds the 18 remaining feature columns. y holds the price column, which is the value you are predicting.
  • train_test_split reserves 20% of the data for testing. Setting random_state=42 ensures you get the same split every time you run the cell.
  • StandardScaler rescales each feature to have zero mean and unit variance. This prevents large-valued columns like sqft_living from dominating smaller-valued ones like bedrooms.
Train the baseline model and evaluate its accuracy

Linear regression draws the single best straight line through your data. It is fast to train and easy to interpret, which makes it a good baseline. The question is whether a straight line can capture the non-linear patterns you saw in the scatter plot.

  • Add the following code below your feature preparation code in the same cell to train, evaluate, and plot the baseline:
lr_model = LinearRegression()
lr_model.fit(X_train_scaled, y_train)
lr_preds = lr_model.predict(X_test_scaled)

lr_r2 = r2_score(y_test, lr_preds)
lr_rmse = np.sqrt(mean_squared_error(y_test, lr_preds))
lr_mae = mean_absolute_error(y_test, lr_preds)

print(f"Baseline Linear Regression:")
print(f"  R-squared: {lr_r2:.4f}")
print(f"  RMSE: ${lr_rmse:,.0f}")
print(f"  MAE: ${lr_mae:,.0f}")

plt.figure(figsize=(8, 6))
plt.scatter(y_test, lr_preds, alpha=0.3, s=5)
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'r--', lw=2)
plt.xlabel('Actual Price ($)')
plt.ylabel('Predicted Price ($)')
plt.title(f'Baseline: Predicted vs Actual (R2={lr_r2:.3f})')
plt.tight_layout()
plt.show()

What does this code do?

  • lr_model.fit() trains the linear regression on the scaled training data. lr_model.predict() generates price predictions for the held-out test set.
  • R-squared measures how much of the price variation the model explains (1.0 = perfect, 0.0 = no better than guessing the average). RMSE and MAE measure error in dollar terms.
  • The scatter plot compares each predicted price against the actual sale price. The red dashed line represents perfect predictions. Points far from that line are errors.

Before you run this cell, think about what you expect. Given the fan-shaped scatter and non-linear patterns in the EDA charts, do you think a straight-line model will predict house prices accurately?

  • Run the cell by pressing Shift+Enter.

You should see an R-squared score around 0.50 to 0.65 printed, followed by a scatter plot where points spread widely away from the red dashed line. That wide scatter confirms the baseline cannot capture the non-linear relationships between features and price.

Why is this R-squared score a problem?

An R-squared of 0.50 to 0.65 means the model explains only about half to two-thirds of the price variation. The rest is noise to the model. For a real estate prediction system, that gap represents tens of thousands of dollars in prediction error per house.

This is exactly the benchmark you need. In the next step, XGBoost will attack this gap with gradient-boosted decision trees that capture the non-linear patterns linear regression misses.

Seeing a NameError or unexpected results?

  • If you see a NameError for LinearRegression or train_test_split, re-run Cell 1 to reload the imports.
  • Make sure both the feature preparation code and the training code are in the same cell. The training code depends on X_train_scaled and X_test_scaled created in the preparation block above it.
  • Still stuck? Help me debug why my baseline linear regression is throwing an error in Google Colab.

✔️ Awesome, I've got everything!

Great. Double check you have both Cell 4 (EDA) and Cell 5 (feature prep + baseline) saved in your notebook.

ⓧ I'd like to double check the full code

Scroll up through your notebook and confirm these cells are present in order:

  • Cell 1: Setup and Version Check (imports and version prints).
  • Cell 2: Install SHAP and kagglehub, Load Data (kagglehub.dataset_download and pd.read_csv).
  • Cell 3: Data Inspection (.info(), .describe(), null check, price range).
  • Cell 4: Exploratory Data Analysis (3-panel figure with histogram, correlation bar chart, scatter plot).
  • Cell 5: Prepare Features and Train Baseline (feature prep, train/test split, scaler, linear regression, metrics, scatter plot).

You have your benchmark. The baseline's mediocre R-squared proves that a straight line cannot capture how features like grade, location, and square footage interact to determine price. Next up, you will build an XGBoost model with engineered features that attacks exactly these non-linear gaps.

Build and Train the XGBoost Model

Your baseline linear regression produced a mediocre R-squared score, and the scatter plot told the story clearly: predictions were scattered far from the actual prices.

The problem is that linear models assume a straight-line relationship between features and price. Real estate prices are driven by complex, non-linear interactions. A house's grade, square footage, and location interact in ways a straight line cannot capture. XGBoost solves this by building hundreds of small decision trees, each one correcting the mistakes of the previous one.

In this step, get ready to:
  • Engineer new features that give the model richer signals about each property.
  • Train an XGBoost model on the enriched dataset.
  • Evaluate XGBoost's accuracy and compare it head-to-head against the baseline.
Engineer new features from the raw data

Raw columns like yr_built contain useful information, but in a form that is hard for models to use directly. A year like 1985 does not tell the model how old the house is relative to the sale date. Feature engineering transforms raw columns into derived features that better represent the underlying signal.

  • Create a new code cell in your Colab notebook and add the feature engineering function by pasting this code:
# Cell 6: Feature Engineering for XGBoost
def engineer_features(dataframe):
    df_eng = dataframe.copy()
    df_eng['house_age'] = 2015 - df_eng['yr_built']
    df_eng['renovated'] = (df_eng['yr_renovated'] > 0).astype(int)
    df_eng['living_ratio'] = df_eng['sqft_living'] / (df_eng['sqft_lot'] + 1)
    df_eng['above_ratio'] = df_eng['sqft_above'] / (df_eng['sqft_living'] + 1)
    return df_eng

X_eng = engineer_features(df.drop(columns=['id', 'date', 'price']))
y_eng = df['price']

X_train_eng, X_test_eng, y_train_eng, y_test_eng = train_test_split(
    X_eng, y_eng, test_size=0.2, random_state=42
)

What does this code do?

The engineer_features function creates four new columns from the raw data:

  • house_age converts yr_built into the property's age at the time of sale. A house built in 1990 becomes 25 years old.
  • renovated turns yr_renovated into a simple 0 or 1 flag. The model only needs to know whether a renovation happened, not the year.
  • living_ratio divides sqft_living by sqft_lot to capture how densely built a property is.
  • above_ratio measures how much of the living space is above ground versus in a basement.

The + 1 in the denominators prevents division-by-zero errors for lots with zero square footage.

  • Run the cell. It should execute without errors.
  • Verify the new features exist by creating a quick cell below it and running:
print(X_eng.shape)

You should see (21613, 22). The original 18 features plus your 4 new engineered columns.

Seeing a NameError or KeyError?

  • If you see NameError: name 'df' is not defined, make sure you ran the earlier cells that load the dataset first. Colab cells must be run in order.
  • If you see a KeyError on a column name, double check that the column names in your function match the dataset exactly. Column names are case-sensitive.

Still stuck? Help me debug my feature engineering cell. I'm getting an error when running engineer_features on the King County dataset.

Create and train the XGBoost model

With enriched features ready, it is time to train the model. XGBoost works by building a sequence of decision trees. Each new tree focuses on the errors the previous trees made, gradually zeroing in on accurate predictions.

  • Create a new code cell and train the XGBoost model by pasting this code:
# Cell 7: Train XGBoost
xgb_model = xgb.XGBRegressor(
    n_estimators=500,
    max_depth=6,
    learning_rate=0.05,
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42,
    early_stopping_rounds=20
)

xgb_model.fit(
    X_train_eng, y_train_eng,
    eval_set=[(X_test_eng, y_test_eng)],
    verbose=50
)

What do these hyperparameters control?

  • n_estimators=500 sets the maximum number of trees to build. More trees can capture more patterns.
  • max_depth=6 limits how deep each tree can grow. Deeper trees learn more complex relationships but risk memorizing the training data.
  • learning_rate=0.05 controls how much each tree contributes to the final prediction. A smaller rate means more cautious learning.
  • early_stopping_rounds=20 stops training if the model has not improved for 20 rounds, preventing wasted computation.
  • Run the cell.

You should see training progress printed every 50 rounds, with the validation RMSE decreasing as XGBoost builds more trees. Training may stop before reaching 500 rounds if early stopping kicks in.

Seeing a NameError on X_train_eng?

  • This means the feature engineering cell (Cell 6) has not been run yet. Go back and run it first.
  • If your Colab runtime disconnected, you need to re-run all previous cells in order. Click Runtime, then Run all to re-execute everything from the top.

Help me troubleshoot my XGBoost training cell in Google Colab.

Evaluate XGBoost and compare with the baseline

The model is trained. Now it is time to find out whether all that gradient boosting actually paid off compared to the linear regression baseline.

Before you run the next cell, make a prediction: do you think XGBoost's R-squared will be closer to 0.70 or 0.90? How much improvement do you expect over the baseline?

  • Generate predictions and compute metrics by creating a new code cell with this code:
xgb_preds = xgb_model.predict(X_test_eng)

xgb_r2 = r2_score(y_test_eng, xgb_preds)
xgb_rmse = np.sqrt(mean_squared_error(y_test_eng, xgb_preds))
xgb_mae = mean_absolute_error(y_test_eng, xgb_preds)

print(f"\nXGBoost Results:")
print(f"  R-squared: {xgb_r2:.4f}")
print(f"  RMSE: ${xgb_rmse:,.0f}")
print(f"  MAE: ${xgb_mae:,.0f}")

What do these metrics mean?

  • R-squared measures how well the model explains the variation in house prices. 1.0 is perfect. Your baseline scored around 0.50-0.65.
  • RMSE (Root Mean Squared Error) is the average prediction error in dollars. Lower is better.
  • MAE (Mean Absolute Error) is similar but less sensitive to large outlier errors. Compare both to see how consistently the model predicts.
  • Run the cell.

You should see the R-squared jump to approximately 0.85-0.90. That is a dramatic improvement over the baseline's 0.50-0.65. The RMSE should also drop substantially, meaning predictions are much closer to actual prices.

Now let's see the improvement visually. The next cell creates a side-by-side comparison scatter plot and a feature importance chart.

  • Create a new code cell and add the evaluation plots by pasting this code:
# Cell 8: XGBoost Evaluation Plots
fig, axes = plt.subplots(1, 2, figsize=(16, 6))

axes[0].scatter(y_test, lr_preds, alpha=0.2, s=5, label=f'Baseline (R2={lr_r2:.3f})')
axes[0].scatter(y_test_eng, xgb_preds, alpha=0.2, s=5, label=f'XGBoost (R2={xgb_r2:.3f})')
axes[0].plot([0, y_test.max()], [0, y_test.max()], 'r--', lw=2)
axes[0].set_xlabel('Actual Price ($)')
axes[0].set_ylabel('Predicted Price ($)')
axes[0].set_title('Predicted vs Actual: Baseline vs XGBoost')
axes[0].legend()

xgb.plot_importance(xgb_model, max_num_features=15, ax=axes[1], height=0.5)
axes[1].set_title('XGBoost Feature Importance')

plt.tight_layout()
plt.show()

What should I see?

Two charts appear side by side. On the left, the predicted vs actual scatter plot shows baseline predictions spread loosely around the diagonal line, while XGBoost predictions cluster tightly along it. The tighter the cluster, the more accurate the model.

On the right, the feature importance bar chart reveals which features XGBoost relied on most. You will likely see sqft_living, grade, and location features near the top.

  • Run the cell and compare both charts against the baseline plot from Step 2.

The scatter plot makes the improvement unmistakable. Baseline predictions were scattered widely, but XGBoost's predictions hug the diagonal line.

Plots not displaying?

  • Make sure you ran both the baseline cell (Cell 5) and the XGBoost prediction cell above this one. The plots reference variables from both cells.
  • If the feature importance chart is missing, confirm that xgb_model was trained successfully by checking the training output from the previous cell.

Help me debug my XGBoost evaluation plots in Google Colab.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook. You should have three new cells: feature engineering (Cell 6), XGBoost training (Cell 7), and evaluation plots (Cell 8).

ⓧ I'd like to double check the full code

Scroll up through your notebook and confirm these cells are present, in order:

  • Cell 1: Setup and Version Check (from Step 1).
  • Cell 2: Install SHAP and kagglehub, Load Data (from Step 1).
  • Cell 3: Data Inspection (from Step 1).
  • Cell 4: Exploratory Data Analysis (from Step 2).
  • Cell 5: Prepare Features and Train Baseline (from Step 2).
  • Cell 6: Feature Engineering for XGBoost (this step). Defines engineer_features, creates X_eng, and splits into train/test.
  • Cell 7: Train XGBoost (this step). Creates xgb_model, trains it, generates predictions, and prints R-squared, RMSE, and MAE.
  • Cell 8: XGBoost Evaluation Plots (this step). Comparison scatter plot and feature importance bar chart.

XGBoost crushed the baseline by capturing non-linear feature interactions that linear regression missed entirely. But XGBoost treats every house sale as an independent data point. It has no concept of time. Next up, you will build an LSTM neural network that reads housing prices as a sequence, capturing the temporal trends XGBoost ignores.

Build and Train the LSTM Model

Your XGBoost model crushed the baseline, jumping from an R-squared of around 0.5 to nearly 0.9. However, it treats every house sale as an isolated data point with no memory of what came before.

Housing prices have momentum. If prices rose for three months straight, the next month is more likely to continue that trend. In this step, you will build an LSTM (Long Short-Term Memory) neural network that learns from sequences of monthly prices and uses that temporal pattern to forecast where prices are heading.

In this step, get ready to:
  • Aggregate individual sales into a monthly median price time series.
  • Create sliding-window sequences that teach the LSTM to predict from recent history.
  • Build, train, and evaluate an LSTM model on the monthly price data.
Aggregate the data into a monthly time series

XGBoost saw 21,613 individual rows. An LSTM needs a chronological sequence instead. You will collapse all those sales into one median price per month, producing a short time series of roughly 13 data points.

Why only 13 data points?

The King County dataset covers May 2014 to May 2015. That is only about 13 calendar months. This makes the time series very small by production standards.

This step is a demonstration of how LSTM sequence modeling works and how it contrasts with tabular approaches. With a longer dataset spanning several years, the LSTM would have far more data to learn from.

  • Create a new code cell in your notebook and paste the following code to aggregate sales by month and normalize the prices:
# Cell 9: Prepare Time Series for LSTM
df['date_parsed'] = pd.to_datetime(df['date'])
df['year_month'] = df['date_parsed'].dt.to_period('M')

# Group all sales by month and take the median price
monthly_prices = df.groupby('year_month')['price'].median().sort_index()
monthly_prices.index = monthly_prices.index.to_timestamp()

print(f"Monthly time series: {len(monthly_prices)} data points")
print(monthly_prices)

# Normalize prices to 0-1 range for LSTM training
price_values = monthly_prices.values.reshape(-1, 1)
lstm_scaler = MinMaxScaler(feature_range=(0, 1))
price_scaled = lstm_scaler.fit_transform(price_values)

What does this code do?

  • pd.to_datetime(df['date']) converts the raw date strings into proper datetime objects so you can group by calendar month.
  • groupby('year_month')['price'].median() collapses thousands of individual sales into one representative price per month. The median is more robust to outlier mansions than the mean.
  • MinMaxScaler rescales prices to the 0-1 range. Neural networks train much faster and more stably when input values are small and uniform.
  • Run the cell.

You should see "Monthly time series: 13 data points" followed by a table of 13 year-month entries with their median prices.

Seeing a KeyError on the date column?

  • Make sure you did not drop the date column from df directly. In earlier steps, you dropped it only from the feature set X, leaving df intact.
  • If df was modified, re-run the data loading cell (Cell 2) to reload the original DataFrame.

Why is my time series aggregation failing?

Create sliding-window sequences for LSTM input

An LSTM does not look at the entire series at once. Instead, it slides a fixed-size window across the data, using the values in each window to predict the next value. You will use a window of 3 months: the model sees months 1-2-3 and predicts month 4, then sees 2-3-4 and predicts month 5, and so on.

  • Create a new code cell and paste this code to build sliding-window sequences and split into train and test sets:
# Create sliding windows: 3 months of history predict the next month
window_size = 3
X_seq, y_seq = [], []
for i in range(len(price_scaled) - window_size):
    X_seq.append(price_scaled[i:i + window_size])
    y_seq.append(price_scaled[i + window_size])

X_seq = np.array(X_seq)
y_seq = np.array(y_seq)

# Split chronologically: first 70% for training, rest for testing
split = int(len(X_seq) * 0.7)
X_train_lstm, X_test_lstm = X_seq[:split], X_seq[split:]
y_train_lstm, y_test_lstm = y_seq[:split], y_seq[split:]

print(f"LSTM training samples: {X_train_lstm.shape[0]}")
print(f"LSTM test samples: {X_test_lstm.shape[0]}")
print(f"Input shape: {X_train_lstm.shape}")

What does this code do?

  • The for loop slides a window of size 3 across the normalized prices. Each iteration captures 3 consecutive months as input (X_seq) and the following month as the target (y_seq).
  • The resulting shape (samples, 3, 1) is the 3D format LSTMs require: number of samples, number of timesteps per sample, and number of features per timestep.
  • The split uses the first 70% of sequences for training and the rest for testing. This is a chronological split, not random, because shuffling time-series data would leak future information into training.
  • Run the cell.

You should see the number of training and test samples printed, along with an input shape like (7, 3, 1). The 3 in the middle is your window size, and the 1 means one feature (price).

Seeing 0 training samples?

  • This means price_scaled is empty or too short. Go back and re-run the previous cell to confirm you see 13 monthly data points.
  • Make sure window_size is set to 3, not a larger number that exceeds the length of the series.

Why is my LSTM sequence array empty?

Build and train the LSTM model

With the sequences ready, you can now define the neural network. The architecture is simple: one LSTM layer to learn temporal patterns, followed by one Dense output layer to produce a single price prediction.

Why LSTM instead of a simpler time-series model?

Traditional time-series approaches like ARIMA require manual tuning of lag and differencing parameters. An LSTM automatically learns which parts of the input sequence matter through its internal gates. It also scales naturally to multivariate inputs, which you will use in the Secret Mission.

  • Create a new code cell and paste this code to define, compile, and train the LSTM:
# Cell 10: Build and Train LSTM
lstm_model = keras.Sequential([
    # 50 LSTM units learn temporal patterns from 3-month windows
    layers.LSTM(50, activation='relu', input_shape=(window_size, 1)),
    # Single output neuron predicts the next month's price
    layers.Dense(1)
])

lstm_model.compile(optimizer='adam', loss='mse')
lstm_model.summary()

# Train for 200 epochs on the small time series
history = lstm_model.fit(
    X_train_lstm, y_train_lstm,
    epochs=200,
    batch_size=1,
    verbose=0
)

# Plot training loss to confirm the model is learning
plt.figure(figsize=(10, 4))
plt.plot(history.history['loss'])
plt.title('LSTM Training Loss')
plt.xlabel('Epoch')
plt.ylabel('MSE Loss')
plt.tight_layout()
plt.show()

What does this code do?

  • keras.Sequential stacks layers in order. The LSTM layer has 50 units, meaning it maintains 50 internal "memory cells" that each track a different aspect of the price sequence.
  • input_shape=(window_size, 1) tells the model to expect 3 timesteps with 1 feature each, matching the sliding-window sequences you created.
  • verbose=0 silences the per-epoch output during training. The loss curve plotted afterward gives you a cleaner view of how the model learned over all 200 epochs.
  • Run the cell. This may take 30-60 seconds as the LSTM trains for 200 epochs.

You should see a model summary table printed first, followed by a training loss curve that drops steeply in the early epochs and flattens as the model converges.

Loss curve stays flat or spikes?

  • Confirm you ran the MinMaxScaler normalization cell first. Training on raw prices (hundreds of thousands) instead of 0-1 values causes the loss to explode.
  • Make sure X_train_lstm is not empty. Print its shape to verify it has samples.

Why is my LSTM training loss not decreasing?

The model has learned from the training data. Now you will see whether those learned patterns hold up on the test months it has never seen.

Before you run the evaluation, think about this: the LSTM was trained on only about 7 sequences of 3 months each, and it will be tested on just a few data points. Do you expect strong or weak performance on such a small test set?

  • Create a new code cell and paste this code to evaluate the LSTM and plot its predictions:
# Cell 11: Evaluate LSTM
# Convert predictions back to actual dollar values
lstm_preds_scaled = lstm_model.predict(X_test_lstm)
lstm_preds_actual = lstm_scaler.inverse_transform(lstm_preds_scaled)
y_test_lstm_actual = lstm_scaler.inverse_transform(y_test_lstm)

# Calculate performance metrics
lstm_rmse = np.sqrt(mean_squared_error(y_test_lstm_actual, lstm_preds_actual))
lstm_mae = mean_absolute_error(y_test_lstm_actual, lstm_preds_actual)
lstm_r2 = r2_score(y_test_lstm_actual, lstm_preds_actual)

print(f"LSTM Results (Monthly Median Price):")
print(f"  R-squared: {lstm_r2:.4f}")
print(f"  RMSE: ${lstm_rmse:,.0f}")
print(f"  MAE: ${lstm_mae:,.0f}")

# Plot actual vs predicted monthly prices
test_months = monthly_prices.index[split + window_size:]
plt.figure(figsize=(12, 5))
plt.plot(monthly_prices.index, monthly_prices.values, 'b-o', label='Actual Monthly Median', markersize=6)
plt.plot(test_months, lstm_preds_actual.flatten(), 'r--s', label='LSTM Predicted', markersize=8)
plt.xlabel('Month')
plt.ylabel('Median Price ($)')
plt.title('LSTM: Monthly Median Price Prediction')
plt.legend()
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

What does this code do?

  • lstm_scaler.inverse_transform converts the model's 0-1 predictions back to actual dollar amounts so the metrics are interpretable.
  • The line chart plots the full actual monthly median series in blue, with the LSTM's predictions overlaid in red dashes. You can visually see how closely the predicted trend follows the real market movement.
  • test_months = monthly_prices.index[split + window_size:] aligns the prediction dates correctly by offsetting for both the train/test split and the 3-month window.
  • Run the cell.

You should see LSTM metrics printed (R-squared, RMSE, MAE) and a line chart where the red dashed prediction line follows the blue actual monthly median trend. The LSTM captures the general direction of price movement across the test months.

How do I compare XGBoost and LSTM fairly?

These two models solve different problems. XGBoost predicts individual house prices from tabular features (21,613 predictions). The LSTM predicts monthly median price trends from a 13-point time series. Their RMSE values are not directly comparable because they operate on different scales and sample sizes.

The next step will put both models side by side in a structured comparison so you can see when each approach shines.

Predicted line looks completely flat?

  • Re-run the training cell (Cell 10). With so few data points, the LSTM can sometimes converge to a mean prediction on the first attempt. Training again with a fresh random initialization often fixes this.
  • Confirm that your training loss curve showed a clear downward trend. If the loss barely moved, the model did not learn enough from the data.

Why is my LSTM predicting a flat line instead of a trend?

✔️ Awesome, I've got everything!

Great. Make sure all three cells you added in this step ran successfully and produced visible output.

ⓧ I'd like to double check the full code

Scroll through your notebook and confirm these cells are present in order, matching the code you pasted above:

  • The time series aggregation cell that parses dates, creates monthly_prices, and normalizes with MinMaxScaler.
  • The sliding-window cell that creates X_seq and y_seq with window_size = 3 and splits into X_train_lstm / X_test_lstm.
  • Cell 10 that builds the keras.Sequential LSTM model, trains it for 200 epochs, and plots the training loss curve.
  • Cell 11 that evaluates the LSTM, prints metrics, and plots actual vs predicted monthly median prices.

You now have two models that attack the same dataset from completely different angles. XGBoost dominates individual house price prediction using tabular features. The LSTM captures temporal momentum from monthly price trends. Next up, you will compare all three models head-to-head and use SHAP to explain exactly why XGBoost makes each prediction.

Compare Models and Explain with SHAP

Your LSTM model is trained and predicting monthly price trends. Now you have three models, each answering a different question about King County house prices.

Trained models mean nothing until you put them on equal footing and compare their performance. In this step, you will build a side-by-side metrics comparison, then use SHAP to crack open XGBoost's decision-making and see exactly which features drive each prediction.

In this step, get ready to:
  • Print a formatted metrics comparison table for all three models.
  • Generate SHAP beeswarm and waterfall plots to explain XGBoost predictions.
  • Create a dual-axis visualization combining both models' strengths.
Print the model comparison table

Each of your three models operates on different data shapes. Linear Regression and XGBoost predict individual house prices from tabular features. LSTM predicts monthly median price trends from time-series sequences.

A formatted table with RMSE, MAE, and R-squared side by side makes the performance gap impossible to miss.

  • Create a new code cell in your notebook and paste the following comparison code:
# Cell 12: Model Comparison
print("=" * 60)
print(f"{'Model':<25} {'RMSE':>12} {'MAE':>12} {'R-squared':>10}")
print("=" * 60)
print(f"{'Linear Regression':<25} ${lr_rmse:>10,.0f} ${lr_mae:>10,.0f} {lr_r2:>9.4f}")
print(f"{'XGBoost':<25} ${xgb_rmse:>10,.0f} ${xgb_mae:>10,.0f} {xgb_r2:>9.4f}")
print(f"{'LSTM (monthly)':<25} ${lstm_rmse:>10,.0f} ${lstm_mae:>10,.0f} {lstm_r2:>9.4f}")
print("=" * 60)
print("\nXGBoost excels at cross-sectional prediction (individual house prices).")
print("LSTM excels at sequential trend forecasting (monthly price momentum).")

What does this code do?

This cell pulls in the metrics you computed in previous steps (lr_rmse, xgb_rmse, lstm_rmse, etc.) and prints them in a single formatted table so you can compare all three models at a glance.

The final two print statements summarize when to use each approach. XGBoost handles cross-sectional tabular prediction (one house at a time). LSTM handles sequential trend forecasting (monthly price momentum over time).

  • Run the cell.

You should see a neatly formatted table with three rows. XGBoost's R-squared should be dramatically higher than Linear Regression's, confirming the improvement you saw in Step 3.

Seeing a NameError?

If you see an error about an undefined variable like lr_rmse or xgb_rmse, your earlier cells have not been run in this session. Go back to Cell 5 (baseline) and Cell 7 (XGBoost) and re-run them.

Help me figure out why my comparison table variables are undefined.

Generate SHAP explanations for XGBoost

The comparison table tells you that XGBoost is more accurate, but it does not tell you why. SHAP (SHapley Additive exPlanations) breaks each XGBoost prediction into individual feature contributions.

For any single house, you can see exactly how much sqft_living pushed the price up, how much grade contributed, and how much every other feature mattered.

  • Create a new code cell and paste the following SHAP code:
# Cell 13: SHAP Explanations
import shap

explainer = shap.TreeExplainer(xgb_model)
X_sample = X_test_eng.sample(200, random_state=42)
shap_values = explainer(X_sample)

shap.plots.beeswarm(shap_values, max_display=15, show=True)

shap.plots.waterfall(shap_values[0], max_display=15, show=True)

What does this code do?

  • shap.TreeExplainer(xgb_model) creates an explainer optimized for tree-based models like XGBoost. It calculates how each feature shifts a prediction away from the average.
  • X_sample takes 200 random test observations so the computation finishes quickly. SHAP on the full test set would take much longer.
  • The beeswarm plot shows all 200 predictions at once. Each dot is one feature for one house, colored by the feature value (red = high, blue = low). Features at the top have the biggest overall impact.
  • The waterfall plot zooms into a single prediction (index 0) and shows how each feature pushed the price up or down from the base value to the final prediction.
  • Run the cell.

You should see two plots. The beeswarm plot shows feature importance across all 200 samples, with features like grade and sqft_living near the top. The waterfall plot tells the story of one specific house's predicted price.

Why SHAP instead of built-in feature importance?

XGBoost's built-in feature importance (which you plotted in Step 3) counts how often a feature is used for splitting. SHAP goes further by measuring the actual impact of each feature on each individual prediction.

This means SHAP can tell you that for one specific house, its high grade added $150,000 to the predicted price while its small lot size pulled it down by $30,000. Built-in importance cannot do that.

SHAP cell taking a long time?

The SHAP computation on 200 samples should finish in under a minute. If it seems stuck, make sure you are using the T4 GPU runtime (check Runtime > Change runtime type). Also confirm you used X_test_eng.sample(200, random_state=42) and not the full test set.

Help me troubleshoot my SHAP computation in Colab.

Create the final combined visualization

XGBoost and LSTM each shine in a different dimension. A dual-axis chart puts both on the same canvas so you can see how they complement each other at a glance.

  • Create a new code cell and paste the following visualization code:
# Cell 14: Final Combined Visualization
fig, ax1 = plt.subplots(figsize=(14, 6))

sample_idx = np.random.choice(len(y_test_eng), 500, replace=False)
ax1.scatter(
    y_test_eng.iloc[sample_idx], xgb_preds[sample_idx],
    alpha=0.3, s=10, color='steelblue', label='XGBoost: Individual Predictions'
)
ax1.plot([0, y_test_eng.max()], [0, y_test_eng.max()], 'k--', lw=1, alpha=0.5)
ax1.set_xlabel('Actual Price ($)')
ax1.set_ylabel('XGBoost Predicted Price ($)', color='steelblue')
ax1.tick_params(axis='y', labelcolor='steelblue')

ax2 = ax1.twinx()
ax2.plot(
    range(len(y_test_lstm_actual)),
    y_test_lstm_actual.flatten(), 'g-o',
    label='LSTM: Actual Monthly Trend', markersize=8
)
ax2.plot(
    range(len(lstm_preds_actual)),
    lstm_preds_actual.flatten(), 'r--s',
    label='LSTM: Predicted Monthly Trend', markersize=8
)
ax2.set_ylabel('Monthly Median Price ($)', color='green')
ax2.tick_params(axis='y', labelcolor='green')

lines1, labels1 = ax1.get_legend_handles_labels()
lines2, labels2 = ax2.get_legend_handles_labels()
ax1.legend(lines1 + lines2, labels1 + labels2, loc='upper left')

plt.title('Two Models, Two Strengths: XGBoost (Tabular) vs LSTM (Temporal)')
plt.tight_layout()
plt.show()

What does this code do?

  • ax1 (left y-axis) plots a scatter of 500 randomly sampled XGBoost predictions vs actual house prices in steelblue. The closer the dots sit to the dashed diagonal line, the more accurate the predictions.
  • ax2 (right y-axis) overlays the LSTM's monthly trend line. Green circles are actual monthly medians. Red squares are LSTM's predicted trend.
  • The combined legend merges both axes' labels into one box so you can read all four data series in one place.

Before you run this cell, think about what you expect to see. Will the XGBoost scatter points cluster tightly around the diagonal, or spread wide? Will the LSTM trend line follow the green actual trend closely?

  • Run the cell.

You should see a dual-axis chart with the XGBoost scatter clustering along the diagonal (strong individual accuracy) and the LSTM trend lines on the right axis tracking monthly momentum. This single image captures exactly why you need both approaches.

Chart looks empty or labels overlap?

If the scatter points are missing, check that y_test_eng and xgb_preds exist by re-running Cells 6-7. If the LSTM trend lines are missing, re-run Cell 11 to restore y_test_lstm_actual and lstm_preds_actual.

Help me fix my dual-axis chart in matplotlib.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook. You should have Cells 12, 13, and 14 all producing output.

ⓧ I'd like to double check the full code

The full notebook is too long to display here. Scroll up through your notebook and confirm these three cells are present in order, each producing the correct output:

  • Cell 12: Model Comparison prints a formatted table with three rows (Linear Regression, XGBoost, LSTM) and three metric columns (RMSE, MAE, R-squared).
  • Cell 13: SHAP Explanations imports shap, creates a TreeExplainer, and produces a beeswarm plot followed by a waterfall plot.
  • Cell 14: Final Combined Visualization produces a dual-axis chart with XGBoost scatter on the left and LSTM trend lines on the right.

Secret mission

Build a Hybrid XGBoost-LSTM Ensemble

Your XGBoost model nails individual house prices from tabular features. Your LSTM captures monthly price momentum. What if you combined both strengths into a single model? In this mission, you will feed XGBoost's per-sale predictions as an additional input feature into a new LSTM, creating a hybrid ensemble that sees both the tabular signal and the temporal trend at once.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on Google Colab's free tier, so there are no ongoing costs. Decide whether to keep your notebook in Google Drive, let the runtime disconnect on its own, or delete the notebook entirely.

Resources you used:

  • Google Colab notebook (stored in your Google Drive, free).

What about the dataset and installed packages?

The King County dataset downloaded via kagglehub is stored in a temporary runtime directory. It disappears automatically when your Colab runtime disconnects.

Packages you installed with pip (like SHAP and kagglehub) are also runtime-only. They are wiped when the runtime resets.

Keep everything running

No action needed. Choose this if you want to revisit or extend your models later.

  • Your notebook stays in Google Drive indefinitely at no cost.
  • The Colab runtime (GPU, memory, installed packages) disconnects automatically after a period of inactivity. Reconnecting later restarts a fresh runtime, so you will need to re-run your cells from the top.
  • Your Kaggle account remains active for downloading other datasets in future projects.

Pause - I'll come back to this later

Colab runtimes disconnect automatically after inactivity. There is nothing you need to do manually.

  • Your notebook file stays in Google Drive, ready to reopen.
  • When you come back, click Runtime > Run all to re-execute every cell from the top. This reinstalls SHAP, re-downloads the dataset, and retrains your models.

Delete - I don't want to use this again

Remove the notebook from Google Drive to clean up completely.

  • Open Google Drive at drive.google.com.
  • Navigate to Colab Notebooks (or wherever you saved your notebook).
  • Right-click your notebook file and select Move to trash.
  • Empty your Google Drive trash to permanently delete the file.

Nice Work!

Nice Work!

Nice work! You just built a complete real estate price prediction system using two fundamentally different machine learning architectures and compared them head-to-head.

You've learned how to:

  • Engineered features and trained an XGBoost model that dramatically outperformed a linear regression baseline on the King County House Sales dataset (21,613 records, 21 features).
  • Aggregated tabular sales data into a monthly time series and trained an LSTM neural network to forecast price trends, learning when sequential models outperform tree-based models.
  • Compared all three models with RMSE, MAE, and R-squared metrics, then used SHAP to explain which features drive XGBoost's predictions, turning a black box into an interpretable model.
  • Secret Mission: Built a hybrid XGBoost-LSTM ensemble that combines tabular feature extraction with temporal sequence learning.

Ready to quiz yourself?