Adaptive Email Spam Detection with AI

Build a fusion spam classifier that combines email text and metadata features.

Introduction

30 Second Summary

Your email inbox filters out dozens of spam messages every day. Most get caught, but some slip through because they rely on tricks like excessive capitalization, suspicious links, and keyword stuffing that a simple text scanner completely misses.

In this project, you will build an adaptive email spam detection system that fuses email text content with engineered metadata features. You will prove that fusion outperforms text-only classification, then build an adaptive pipeline that keeps learning from new emails without retraining from scratch.

What You'll Build

You will run your classifier against over 33,000 real emails from the Enron dataset, test it with handcrafted spam examples, and watch a live accuracy curve climb as your model adapts to streaming data.

By the end of this project, you'll have:

  • A spam classifier you can test with handcrafted email examples, showing where a text-only baseline fails and how your content-metadata fusion model catches what it misses.
  • A grouped bar chart comparing 3 classifiers across 3 feature sets. This is a ready-made ablation study figure for a thesis defense or portfolio.
  • An adaptive learning pipeline that streams time-ordered email batches and plots accuracy climbing in real time as the model learns incrementally.
  • Secret Mission: Add confidence-based drift detection that monitors prediction certainty and auto-retrains when it drops.

Are there any prerequisites?

You should be comfortable with basic Python and familiar with pandas DataFrames. You will install scikit-learn, pandas, and matplotlib in Step 1.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Environment and Dataset

Before building any models, you need the right tools and data in place. This project relies on three Python libraries and a real-world email dataset with over 33,000 messages.

In this step, you will create a Python virtual environment, install your dependencies, download the Enron Spam Dataset from Kaggle, and write a preprocessing script that loads and cleans the data.

In this step, get ready to:
  • Install scikit-learn, pandas, and matplotlib in a virtual environment.
  • Download and organize the Enron Spam Dataset.
  • Create preprocess.py to load and clean the dataset.
Set up the project folder and install dependencies

This project requires Python 3.9 or higher. Let's confirm your version first.

  • Open a terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check your Python version by running:
python3 --version

✔️ I see version 3.9 or higher

You are all set. Move on to the next instruction below.

ⓧ I see an older version

You need Python 3.9 or higher for this project. On macOS, the system Python may be outdated even after installing a newer version.

  • Download and install the latest Python 3.9+ release from python.org/downloads.
  • On macOS, if python3 --version still shows the old version after installing, try running python3.11 --version (using the specific version number you installed). If that works, use python3.11 in place of python3 for all commands in this project.
  • On Windows, run the installer from python.org and make sure to check Add Python to PATH during installation. Then close and reopen your terminal.
  • Run python3 --version again to confirm you see 3.9 or higher.

ⓧ Command not found

Python is not installed on your system yet.

  • Download the latest Python 3.9+ installer from python.org/downloads.
  • Run the installer. On Windows, check Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal after installation.
  • Run python3 --version to confirm it prints 3.9 or higher. On Windows, try python --version if python3 is not recognized.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir adaptive-spam-detection
cd adaptive-spam-detection
  • Open VS Code. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the adaptive-spam-detection folder, then click Open.
  • Open the integrated terminal in VS Code by pressing Ctrl+` (backtick key, same on macOS and Windows).
  • Create a virtual environment by running this command:
python3 -m venv venv
  • Activate the virtual environment:

macOS

  • Run this command:
source venv/bin/activate

Windows

  • Run this command:
venv\Scripts\activate

Your terminal prompt now starts with (venv). This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated Python workspace. Packages you install inside it stay separate from your system Python, so different projects can use different library versions without conflicts.

  • In the VS Code Explorer sidebar, click the new file icon and name the file requirements.txt.
  • Paste the following contents into requirements.txt:
scikit-learn==1.6.1
pandas==2.2.3
matplotlib==3.8.4

What are these packages?

scikit-learn provides the machine learning classifiers and text vectorizers you will use throughout this project. pandas handles loading and manipulating the email dataset. matplotlib generates the charts for your ablation study and learning curves.

  • Save requirements.txt.
  • Install all three packages by running:
pip install -r requirements.txt

You will see download and install progress for each package. This may take a minute.

  • Verify the install succeeded by running:
python -c "import sklearn; print(sklearn.__version__)"

✔️ I see 1.6.1

Your dependencies are installed correctly. Move on to the next section.

ⓧ I see an error

Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt.

  • If the prompt does not show (venv), re-run the activation command from the platform tabs above.
  • Then run pip install -r requirements.txt again.
Download the Enron Spam Dataset

The Enron Spam Dataset contains 33,716 real emails labeled as spam or ham. It includes the raw email body text and dates, which is exactly what you need to engineer metadata features and simulate a time-ordered email stream.

  • Open your browser and navigate to the Enron Spam Dataset on Kaggle: kaggle.com/datasets/jaswanth257/spam-detection-enron.
  • Sign in to your Kaggle account. If you do not have one, create a free account first.
  • Click the Download button near the top of the dataset page. This downloads a zip file to your computer.
  • Extract the zip file. Inside you will find enron_spam_data.csv.

Now create the folders your project needs and move the dataset into place.

  • In your VS Code terminal, create the data and outputs folders by running:
mkdir data outputs
  • Move or copy the extracted enron_spam_data.csv file into the data/ folder inside adaptive-spam-detection. You can drag and drop the file into the data folder in VS Code's Explorer sidebar.
  • Confirm the file is in the right place by running:
ls data/

You should see enron_spam_data.csv listed in the output.

Don't see the CSV file?

  • Make sure you extracted the zip file first. The downloaded file from Kaggle is a .zip archive, not the CSV itself.
  • Check that you moved enron_spam_data.csv into the data/ folder inside your adaptive-spam-detection project folder, not into a different location.
  • help me find where my downloaded Kaggle dataset ended up
Create the preprocessing script

Every model in this project starts by loading and cleaning the email data. A shared preprocessing function keeps this logic in one place so every script can reuse it.

  • In the VS Code Explorer sidebar, click the new file icon and name the file preprocess.py.
  • Paste the following code into preprocess.py:
import pandas as pd
import numpy as np
import re


def load_dataset(filepath):
    """Load the Enron spam dataset and prepare it for modeling."""
    # Read the CSV with latin-1 encoding to handle special characters
    df = pd.read_csv(filepath, encoding="latin-1")
    # Drop rows where the email body is missing
    df = df.dropna(subset=["Message"])
    # Map spam/ham text labels to numeric 1/0
    df["label"] = df["Spam/Ham"].map({"spam": 1, "ham": 0})
    # Combine subject and message into one text field for analysis
    df["text"] = df["Subject"].fillna("") + " " + df["Message"].fillna("")
    # Remove any rows that didn't map to a valid label
    df = df.dropna(subset=["label"])
    return df

What does this code do?

  • pd.read_csv(..., encoding="latin-1") loads the raw CSV. The latin-1 encoding handles special characters that appear in some Enron emails.
  • dropna(subset=["Message"]) removes rows with empty email bodies. You cannot classify an email that has no text.
  • .map({"spam": 1, "ham": 0}) converts the text labels into numbers that classifiers can work with.
  • The text column concatenates the subject and message body into a single string. This gives your models the full email content to analyze.

The numpy and re imports are not used yet. You will use them in a later step when engineering metadata features.

  • Save preprocess.py.

Before you run the verification, what do you expect the dataset shape and spam/ham split to look like? Think about whether you expect a balanced or imbalanced dataset.

  • Test your preprocessing function by running:
python -c "from preprocess import load_dataset; df = load_dataset('data/enron_spam_data.csv'); print('Shape:', df.shape); print(df['label'].value_counts())"

You should see a shape of approximately 33,716 rows. The label distribution shows roughly 17,171 spam emails (label 1) and 16,545 ham emails (label 0). The dataset is nearly balanced.

Seeing a ModuleNotFoundError?

  • If you see an error mentioning pandas, your virtual environment may not be active. Check that your prompt starts with (venv) and re-run the activation command if needed.
  • If the error mentions the file path, make sure you are running the command from inside the adaptive-spam-detection folder. Run ls and confirm you see preprocess.py and the data/ folder.
  • help me debug my preprocess.py import error

✔️ Awesome, I've got everything!

Great. Make sure you have saved preprocess.py and requirements.txt.

ⓧ I'd like to double check the full code

Here is the complete preprocess.py file:

import pandas as pd
import numpy as np
import re


def load_dataset(filepath):
    """Load the Enron spam dataset and prepare it for modeling."""
    df = pd.read_csv(filepath, encoding="latin-1")
    df = df.dropna(subset=["Message"])
    df["label"] = df["Spam/Ham"].map({"spam": 1, "ham": 0})
    df["text"] = df["Subject"].fillna("") + " " + df["Message"].fillna("")
    df = df.dropna(subset=["label"])
    return df

And here is requirements.txt:

scikit-learn==1.6.1
pandas==2.2.3
matplotlib==3.8.4

Your environment is ready and the dataset is loaded. Next up, you will build a text-only baseline model and see exactly where it falls short on metadata-heavy spam.

Build a Text-Only Baseline Model

Your environment is set up, your dataset is loaded, and preprocess.py is ready to feed clean data into a model. Now it is time to build your first spam classifier and see what it can do.

Before jumping to a complex fusion approach, you need a baseline. A TF-IDF vectorizer paired with an SGDClassifier is the standard text classification starting point. You will train it, see it score impressively, then test it on spam that relies on structural giveaways instead of natural language. Watching it stumble is the whole point.

In this step, get ready to:
  • Train a text-only TF-IDF + SGDClassifier baseline on the Enron dataset.
  • Test the baseline on metadata-heavy spam to expose where it fails.
Create the text-only baseline

TF-IDF stands for Term Frequency-Inverse Document Frequency. It converts email text into numbers by measuring how important each word is across the entire dataset. Rare words get higher scores, common words get lower ones.

You will feed those numbers into an SGDClassifier from scikit-learn, which learns to draw a line between spam and ham based purely on the words in each email.

  • Create a new file called baseline.py in your adaptive-spam-detection folder and paste the following code:
"""Text-only baseline model for spam detection."""
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import classification_report, accuracy_score

from preprocess import load_dataset

# Load data
df = load_dataset("data/enron_spam_data.csv")
X_train, X_test, y_train, y_test = train_test_split(
    df["text"], df["label"], test_size=0.2, random_state=42, stratify=df["label"]
)

# Build TF-IDF + SGDClassifier (linear SVM via hinge loss)
tfidf = TfidfVectorizer(max_features=5000, stop_words="english")
X_train_tfidf = tfidf.fit_transform(X_train)
X_test_tfidf = tfidf.transform(X_test)

clf = SGDClassifier(loss="hinge", random_state=42, max_iter=1000)
clf.fit(X_train_tfidf, y_train)

# Evaluate
y_pred = clf.predict(X_test_tfidf)
print("=== Text-Only Baseline Results ===")
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(classification_report(y_test, y_pred, target_names=["Ham", "Spam"]))

What does this code do?

  • load_dataset() reuses the function you built in preprocess.py to load and clean the Enron CSV.
  • train_test_split holds out 20% of emails for testing. The stratify parameter keeps the spam/ham ratio equal in both sets.
  • TfidfVectorizer converts raw email text into a matrix of 5,000 TF-IDF features, filtering out common English stop words like "the" and "is".
  • SGDClassifier(loss="hinge") trains a linear SVM using stochastic gradient descent. It learns a decision boundary between spam and ham based on word frequencies alone.

Before you run this, what accuracy do you expect from a model that only looks at the words in an email? Take a guess.

  • Save baseline.py.
  • Run the baseline by executing this command in your terminal:
python baseline.py

You should see an accuracy between 95% and 98%, along with a full classification report showing precision, recall, and F1 scores for both Ham and Spam. That is impressive for a model that only reads words.

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • Check that baseline.py is in the same folder as preprocess.py. The import will fail if they are in different directories.
  • If sklearn is not found, re-run pip install -r requirements.txt with your virtual environment active.

Why is my baseline.py import failing?

Expose the baseline's blind spot

95%+ accuracy sounds great. But spam is not always about the words. Some spam emails are obvious from their structure: all-caps text, stuffed with links, or containing a single suspicious word. These are metadata patterns, not language patterns.

Your text-only model has never seen these structural signals. It only knows word frequencies. Time to see what happens when you throw metadata-heavy spam at it.

  • In baseline.py, scroll to the bottom of the file and add the following code below the classification_report print statement:
# Demonstrate limitation: metadata-heavy spam examples
tricky_emails = [
    "Hi",
    "CLICK HERE NOW FREE MONEY WIN BIG CASH PRIZES",
    "Dear friend visit http://example.com http://win.com http://free.com",
]
print("\n=== Tricky Spam Examples (text-only model) ===")
X_tricky = tfidf.transform(tricky_emails)
predictions = clf.predict(X_tricky)
for email_text, pred in zip(tricky_emails, predictions):
    result = "SPAM" if pred == 1 else "HAM"
    print(f"  '{email_text[:60]}' -> {result}")

What does this code do?

  • tricky_emails holds three handcrafted spam examples that rely on metadata patterns. "Hi" is suspiciously short. The all-caps email screams keyword stuffing. The third one is packed with links.
  • tfidf.transform() converts these emails into the same TF-IDF feature space the model was trained on. The model then predicts each one as spam or ham.
  • The loop prints each email alongside the model's verdict so you can see exactly where it gets fooled.

Before you run this, do you think the text-only model will correctly flag all three as spam? Think about what the model actually learned. It only knows word frequencies from the training set.

  • Save baseline.py.
  • Run the updated script by executing this command:
python baseline.py

Scroll down past the classification report to the "Tricky Spam Examples" section. You should see at least one of the three labelled as HAM instead of SPAM. That is the gap. The model cannot see uppercase ratios, link counts, or keyword stuffing patterns because it only learned from word frequencies.

Why did the model get tricked?

The "Hi" email has almost no words, so TF-IDF produces a near-empty feature vector. The all-caps email contains spam keywords, but TF-IDF treats them the same as lowercase versions. The link-heavy email's URLs get split into fragments that do not match training vocabulary.

In all three cases, the giveaway is structural: message length, uppercase ratio, link count. These are metadata features, and the text-only model has no way to see them. That is exactly what you will fix in the next step.

All three predicted as SPAM?

Results vary slightly depending on the random split. If all three happen to be correctly classified, the point still stands: the model is guessing on these examples because TF-IDF features are near-zero for them. The fusion model you build next will handle these patterns with consistent confidence.

Why is my text-only model getting different results than expected on the tricky spam examples?

✔️ Awesome, I've got everything!

Great. Double check you have saved baseline.py with both sections: the training and evaluation section at the top, and the tricky spam examples section at the bottom.

ⓧ I'd like to double check the full code

Your baseline.py should contain these two sections in order. Scroll up and confirm each one is present:

  • Imports, data loading, train/test split, TF-IDF vectorizer, SGDClassifier training, and evaluation print statements.
  • Tricky spam examples list, transform, predict, and the print loop.

Your baseline model is strong on normal emails but blind to structural spam patterns. Next up, you will engineer 19 metadata features from the raw email text and fuse them with TF-IDF to close that gap.

Engineer Metadata Features and Build the Fusion Model

Your text-only baseline got solid accuracy on the test set, but it failed on spam that screams "spam" through its structure. All-caps keyword stuffing, multiple links, excessive punctuation. The model saw the words but missed the patterns surrounding them.

In this step, you will engineer 19 metadata features from the raw email text and fuse them with the TF-IDF vectors to build a content-metadata fusion model. You will then run both models side by side and compare the results.

In this step, get ready to:
  • Engineer 19 metadata features from raw email text and add them to your preprocessing pipeline.
  • Build a content-metadata fusion model that fuses TF-IDF text features with scaled metadata features.
  • Compare the fused model against the text-only baseline and prove the improvement.
Add the metadata feature engineering function

Each email carries signals beyond the words it uses. A short, all-caps email stuffed with dollar signs looks nothing like a normal message. Your feature engineering function will extract 19 of these structural signals from every email.

  • Switch back to preprocess.py in VS Code.
  • Add the engineer_metadata_features() function below your load_dataset() function by copying the following code:
def engineer_metadata_features(df):
    """Engineer metadata features from raw email text."""
    df = df.copy()
    df["word_count"] = df["text"].apply(lambda x: len(str(x).split()))
    df["char_count"] = df["text"].apply(lambda x: len(str(x)))
    df["uppercase_ratio"] = df["text"].apply(
        lambda x: sum(1 for c in str(x) if c.isupper()) / max(len(str(x)), 1)
    )
    df["exclamation_count"] = df["text"].apply(lambda x: str(x).count("!"))
    df["question_count"] = df["text"].apply(lambda x: str(x).count("?"))
    df["link_count"] = df["text"].apply(
        lambda x: len(re.findall(r"http[s]?://\S+", str(x)))
    )
    df["dollar_count"] = df["text"].apply(lambda x: str(x).count("$"))
    df["digit_ratio"] = df["text"].apply(
        lambda x: sum(1 for c in str(x) if c.isdigit()) / max(len(str(x)), 1)
    )
    df["avg_word_length"] = df["text"].apply(
        lambda x: np.mean([len(w) for w in str(x).split()]) if str(x).split() else 0
    )
    spam_keywords = [
        "free", "win", "winner", "click", "offer",
        "urgent", "limited", "buy", "money", "cash",
    ]
    for kw in spam_keywords:
        df[f"has_{kw}"] = (
            df["text"].str.lower().str.contains(kw, na=False).astype(int)
        )
    return df

What does this function do?

This function extracts 19 features from each email's raw text, split into two groups.

  • 9 numerical features capture the structure of the email: how many words and characters it has, how much of it is uppercase, how many exclamation marks, question marks, links, and dollar signs it contains, the ratio of digits, and the average word length.
  • 10 spam keyword flags check whether the email contains known spam trigger words like "free", "win", "click", and "money". Each flag is a binary 0 or 1.

These are the exact patterns the text-only model missed. An all-caps email with three links scores high on uppercase_ratio and link_count, giving the model new evidence to work with.

Now add the METADATA_COLUMNS list at the very bottom of preprocess.py, outside both functions. This list tells the fusion model which columns to use as metadata features.

  • Below the engineer_metadata_features() function (after the last line of the function body), add this module-level constant:
METADATA_COLUMNS = [
    "word_count", "char_count", "uppercase_ratio", "exclamation_count",
    "question_count", "link_count", "dollar_count", "digit_ratio",
    "avg_word_length", "has_free", "has_win", "has_winner", "has_click",
    "has_offer", "has_urgent", "has_limited", "has_buy", "has_money",
    "has_cash",
]

Why a separate list?

Defining the column names in one place prevents typos from creeping in. Every script that needs the metadata columns will import METADATA_COLUMNS from preprocess.py instead of retyping all 19 names.

  • Save preprocess.py.
  • Verify the new function and column list import correctly by running this command:
python -c "from preprocess import engineer_metadata_features, METADATA_COLUMNS; print(len(METADATA_COLUMNS), 'metadata features defined')"

You should see 19 metadata features defined printed in your terminal.

Seeing an ImportError?

  • Make sure METADATA_COLUMNS is defined at the module level (no indentation), not inside a function body.
  • Check that engineer_metadata_features is spelled exactly the same in both the function definition and the import.
  • Confirm your terminal is in the adaptive-spam-detection folder with your virtual environment activated.

Help me debug an ImportError when importing engineer_metadata_features from preprocess.py

✔️ Awesome, I've got everything!

Great. Double check you have saved preprocess.py.

ⓧ I'd like to double check the full code

Your preprocess.py file is now longer than a quick glance. Scroll through it and confirm these three sections appear in this order:

  • Imports at the top: pandas, numpy, and re.
  • load_dataset() function (from Step 1, unchanged).
  • engineer_metadata_features() function with 9 numerical features and a spam_keywords loop creating 10 binary flags.
  • METADATA_COLUMNS list at the module level (no indentation) with exactly 19 string entries.
Create the fusion model script

Now you have the features. The next step is to fuse them with the TF-IDF text features into one combined feature matrix and train a classifier on the fused result.

  • Create a new file called fusion_model.py in your adaptive-spam-detection folder.
  • Add the imports, data loading, train/test split, and text feature extraction by copying the following code:
"""Content-metadata fusion model for spam detection."""
from scipy.sparse import hstack
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import classification_report, accuracy_score

from preprocess import load_dataset, engineer_metadata_features, METADATA_COLUMNS

# Load and engineer features
df = load_dataset("data/enron_spam_data.csv")
df = engineer_metadata_features(df)

X_train_df, X_test_df, y_train, y_test = train_test_split(
    df, df["label"], test_size=0.2, random_state=42, stratify=df["label"]
)

# Text features via TF-IDF
tfidf = TfidfVectorizer(max_features=5000, stop_words="english")
X_train_text = tfidf.fit_transform(X_train_df["text"])
X_test_text = tfidf.transform(X_test_df["text"])

What does this code do?

  • hstack from scipy.sparse will horizontally concatenate the text and metadata feature matrices later.
  • engineer_metadata_features(df) adds the 19 metadata columns you just built to the DataFrame before splitting.
  • The split passes the full DataFrame (not just the text column) so both text and metadata columns survive into train and test sets.
  • Now add the metadata scaling, feature fusion, model training, and comparison output below the code you just pasted:
# Metadata features (scaled)
scaler = StandardScaler()
X_train_meta = scaler.fit_transform(X_train_df[METADATA_COLUMNS].values)
X_test_meta = scaler.transform(X_test_df[METADATA_COLUMNS].values)

# Fuse: concatenate text + metadata features
X_train_fused = hstack([X_train_text, X_train_meta])
X_test_fused = hstack([X_test_text, X_test_meta])

# Train fused model
clf_fused = SGDClassifier(loss="hinge", random_state=42, max_iter=1000)
clf_fused.fit(X_train_fused, y_train)

# Train text-only for comparison
clf_text = SGDClassifier(loss="hinge", random_state=42, max_iter=1000)
clf_text.fit(X_train_text, y_train)

# Compare results
y_pred_text = clf_text.predict(X_test_text)
y_pred_fused = clf_fused.predict(X_test_fused)

print("=== Text-Only Model ===")
print(f"Accuracy: {accuracy_score(y_test, y_pred_text):.4f}")
print(classification_report(y_test, y_pred_text, target_names=["Ham", "Spam"]))

print("\n=== Fused Model (Text + Metadata) ===")
print(f"Accuracy: {accuracy_score(y_test, y_pred_fused):.4f}")
print(classification_report(y_test, y_pred_fused, target_names=["Ham", "Spam"]))

What is happening in the fusion pipeline?

  • StandardScaler normalizes each metadata column to zero mean and unit variance. Without scaling, a feature like char_count (values in the thousands) would dominate uppercase_ratio (values between 0 and 1).
  • hstack glues the 5,000 TF-IDF columns and 19 metadata columns side by side into a single feature matrix. This is the "fusion" step.
  • The script trains two identical SGDClassifier models. One sees only text features. The other sees fused features. Same algorithm, same hyperparameters, different inputs.
  • Save fusion_model.py.

Before you run the script, think about this: do you expect the fused model to beat the text-only model on accuracy and F1, or will adding 19 extra features not make a difference?

  • Run the fusion model comparison by running this command:
python fusion_model.py

You should see two full classification reports printed side by side. The fused model's accuracy and F1 score should exceed the text-only model's scores.

What should I look for in the output?

Compare the Accuracy line and the F1 column in the classification report for both models. Even a small improvement (a fraction of a percent) proves that the metadata features capture signal the text features alone could not.

Seeing a KeyError or ValueError?

  • A KeyError on a column name like word_count means engineer_metadata_features() did not create the expected columns. Check that you saved preprocess.py after adding the function.
  • A ValueError about inconsistent sample numbers means the DataFrame and label arrays are out of sync. Make sure your train_test_split passes df (the full DataFrame), not df["text"].
  • Verify that METADATA_COLUMNS in preprocess.py lists exactly 19 column names matching the ones created inside engineer_metadata_features().

Help me debug an error in fusion_model.py when running the content-metadata fusion pipeline

✔️ Awesome, I've got everything!

Great. Double check you have saved fusion_model.py.

ⓧ I'd like to double check the full code

Scroll through fusion_model.py and confirm these two sections appear in order:

  • Section 1 (imports through TF-IDF): starts with the docstring and from scipy.sparse import hstack, loads the data with engineer_metadata_features(df), splits 80/20, and builds TF-IDF on X_train_df["text"].
  • Section 2 (metadata through comparison): creates a StandardScaler for metadata, fuses with hstack, trains both clf_fused and clf_text, and prints both classification reports.

You have proven that fusing text and metadata features improves classification. One comparison is a good start, but a single classifier on a single feature set is not enough for a thesis. Next up, you will run a full ablation study comparing multiple classifiers across multiple feature sets and generate a publication-ready chart.

Run a Comparative Ablation Study

Your fused model beat the text-only baseline. That is a strong result, but a single comparison is not enough for a thesis or a final-year defense.

An ablation study systematically removes or swaps components to prove each one earns its place. In this step, you will train three classifiers across three feature sets and produce a publication-ready comparison chart.

In this step, get ready to:
  • Train SGDClassifier, MultinomialNB, and Random Forest across text-only, metadata-only, and fused feature sets.
  • Print a formatted comparison table of accuracy, precision, recall, and F1.
  • Generate a grouped bar chart and save it to outputs/ablation_study.png.
Create the ablation study and define all feature-classifier combinations

Most of the data preparation code mirrors what you wrote in fusion_model.py. The new part is packaging the feature sets and classifiers into dictionaries so a loop can run every combination automatically.

  • Create a new file called ablation_study.py in your adaptive-spam-detection folder.
  • Paste the following imports, data loading, and train/test split:
"""Ablation study comparing feature sets and classifiers."""
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from scipy.sparse import hstack, csr_matrix
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import SGDClassifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

from preprocess import load_dataset, engineer_metadata_features, METADATA_COLUMNS

# Load and prepare
df = load_dataset("data/enron_spam_data.csv")
df = engineer_metadata_features(df)

X_train_df, X_test_df, y_train, y_test = train_test_split(
    df, df["label"], test_size=0.2, random_state=42, stratify=df["label"]
)

What does this code do?

This is the same data pipeline you built in fusion_model.py. It loads the Enron dataset, engineers the 19 metadata columns, and splits 80/20 with stratification.

The matplotlib.use("Agg") line sets a non-interactive backend so matplotlib saves charts to files without opening a window.

  • Add the feature set preparation and classifier dictionaries below the train/test split:
# Prepare feature sets
tfidf = TfidfVectorizer(max_features=5000, stop_words="english")
X_train_text = tfidf.fit_transform(X_train_df["text"])
X_test_text = tfidf.transform(X_test_df["text"])

scaler = StandardScaler()
X_train_meta = scaler.fit_transform(X_train_df[METADATA_COLUMNS].values)
X_test_meta = scaler.transform(X_test_df[METADATA_COLUMNS].values)

X_train_fused = hstack([X_train_text, X_train_meta])
X_test_fused = hstack([X_test_text, X_test_meta])

feature_sets = {
    "Text Only": (X_train_text, X_test_text),
    "Metadata Only": (csr_matrix(X_train_meta), csr_matrix(X_test_meta)),
    "Fused": (X_train_fused, X_test_fused),
}

classifiers = {
    "SGD (Linear SVM)": lambda: SGDClassifier(loss="hinge", random_state=42, max_iter=1000),
    "MultinomialNB": lambda: MultinomialNB(alpha=1.0),
    "Random Forest": lambda: RandomForestClassifier(n_estimators=100, random_state=42),
}

What's happening here?

  • Three feature sets are prepared: Text Only (TF-IDF vectors), Metadata Only (19 scaled columns wrapped in csr_matrix for sparse compatibility), and Fused (both stacked with hstack).
  • Three classifiers are stored as factory functions (lambda) so a fresh instance is created for each combination.
  • SGDClassifier is your linear SVM baseline. MultinomialNB is a probabilistic classifier popular for text. RandomForestClassifier is an ensemble that handles mixed feature types well.
  • Save ablation_study.py. The next chunk adds the ablation loop that trains each combination and prints results.
Run all combinations and visualize the results

The ablation loop iterates over every feature-set and classifier pair, trains a fresh model, and records four metrics. One important detail: MultinomialNB requires non-negative input features. StandardScaler produces negative values, so the loop skips MultinomialNB for the Metadata Only and Fused sets.

  • Add the ablation loop and comparison table printer below the classifier dictionaries in ablation_study.py:
# Run ablation
results = []
for feat_name, (X_tr, X_te) in feature_sets.items():
    for clf_name, clf_factory in classifiers.items():
        # MultinomialNB requires non-negative features
        if clf_name == "MultinomialNB" and feat_name in ("Metadata Only", "Fused"):
            continue
        clf = clf_factory()
        clf.fit(X_tr, y_train)
        y_pred = clf.predict(X_te)
        results.append({
            "Feature Set": feat_name,
            "Classifier": clf_name,
            "Accuracy": accuracy_score(y_test, y_pred),
            "Precision": precision_score(y_test, y_pred),
            "Recall": recall_score(y_test, y_pred),
            "F1": f1_score(y_test, y_pred),
        })

# Print comparison table
print("=== Ablation Study Results ===")
header = f"{'Feature Set':<16} {'Classifier':<20} {'Accuracy':<10} {'Precision':<10} {'Recall':<10} {'F1':<10}"
print(header)
print("-" * len(header))
for r in results:
    print(
        f"{r['Feature Set']:<16} {r['Classifier']:<20} "
        f"{r['Accuracy']:<10.4f} {r['Precision']:<10.4f} "
        f"{r['Recall']:<10.4f} {r['F1']:<10.4f}"
    )

What does this code do?

  • The nested loop tries every feature-set and classifier combination. The continue statement skips invalid pairings where MultinomialNB would receive negative scaled values.
  • Each combination records accuracy, precision, recall, and F1 into a list of dictionaries.
  • The formatted table uses Python f-string alignment (:<16, :<10.4f) to produce clean columns.
  • Save ablation_study.py.
  • Run the script by entering this command in your terminal:
python ablation_study.py

You should see a neatly formatted table with 7 rows. Each row shows accuracy, precision, recall, and F1 for one feature-set and classifier combination.

Why only 7 rows instead of 9?

Three feature sets times three classifiers would give 9 combinations. But MultinomialNB is skipped for Metadata Only and Fused because StandardScaler can produce negative values, and MultinomialNB only accepts non-negative inputs. That leaves 7 valid combinations.

Seeing an import error?

  • Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • Check that preprocess.py is in the same folder as ablation_study.py. The import expects it at the same level.
  • Confirm scikit-learn is installed by running pip list | grep scikit.

Why is my ablation study failing to run?

A table of numbers is useful, but a grouped bar chart makes the patterns immediately obvious. Time to add the visualization.

  • Add the bar chart code at the bottom of ablation_study.py, below the print loop:
# Grouped bar chart
fig, ax = plt.subplots(figsize=(10, 6))
feature_names = list(feature_sets.keys())
classifier_names = list(classifiers.keys())
x = np.arange(len(feature_names))
width = 0.25

for i, clf_name in enumerate(classifier_names):
    f1_scores = []
    for feat_name in feature_names:
        match = [r for r in results if r["Feature Set"] == feat_name and r["Classifier"] == clf_name]
        f1_scores.append(match[0]["F1"] if match else 0)
    ax.bar(x + i * width, f1_scores, width, label=clf_name)

ax.set_xlabel("Feature Set")
ax.set_ylabel("F1 Score")
ax.set_title("Ablation Study: F1 Score by Feature Set and Classifier")
ax.set_xticks(x + width)
ax.set_xticklabels(feature_names)
ax.legend()
ax.set_ylim(0, 1.05)
plt.tight_layout()
plt.savefig("outputs/ablation_study.png", dpi=150)
print("\nChart saved to outputs/ablation_study.png")

What does this code do?

  • For each classifier, the loop collects F1 scores across all three feature sets. If a combination was skipped (like MultinomialNB on Metadata Only), it falls back to 0.
  • ax.bar() offsets each classifier's bars by width so they appear side by side at each feature set position.
  • The chart is saved at 150 DPI to outputs/ablation_study.png. This is the publication-ready figure for your thesis.
  • Save ablation_study.py.

Before you run the script, think about this: which feature set do you predict will give the highest F1 across all classifiers? Will it be text-only, metadata-only, or fused?

  • Run the full script by entering this command:
python ablation_study.py

You should see the comparison table printed again, followed by the line Chart saved to outputs/ablation_study.png.

  • Open the outputs folder in the VS Code Explorer sidebar.
  • Click on ablation_study.png to preview the chart. You should see grouped bars for each classifier across three feature sets on the x-axis.

Chart file not appearing?

  • Check that the outputs/ folder exists inside your adaptive-spam-detection folder. If it is missing, create it and run the script again.
  • Make sure you are running the script from inside the adaptive-spam-detection folder. Run pwd (macOS/Linux) or cd (Windows) to check.

Why isn't my ablation chart saving?

✔️ Awesome, I've got everything!

Great. Double check you have saved ablation_study.py and that outputs/ablation_study.png exists in your project folder.

ⓧ I'd like to double check the full code

Your ablation_study.py file should contain the following four sections in order. Scroll up and confirm each is present:

  • Imports + data loading + train/test split (the first chunk you pasted).
  • Feature set preparation + classifier dictionaries (the second chunk).
  • Ablation loop + comparison table printer (the third chunk).
  • Grouped bar chart generation + save to outputs/ablation_study.png (the fourth chunk).

Your ablation study proves that fusing text and metadata features consistently improves classification across multiple classifiers. Next up, you will build an adaptive learning pipeline that can learn from new emails without retraining from scratch.

Build the Adaptive Learning Pipeline

Your ablation study proved that content-metadata fusion outperforms text-only classification across multiple classifiers. But every model you have built so far is static. You trained it once on a fixed dataset, and that is where its learning stopped.

Real spam evolves. Spammers change tactics, invent new keyword patterns, and shift their formatting. A model trained in January might misclassify March's spam because the patterns have drifted. In this step, you will build an adaptive pipeline that streams time-ordered email batches and incrementally updates the model using partial_fit. No full retraining required.

In this step, get ready to:
  • Set up a streaming-compatible pipeline with HashingVectorizer and MaxAbsScaler.
  • Stream time-ordered email batches and incrementally train the model with SGDClassifier.partial_fit().
  • Generate a learning curve showing accuracy and F1 improving over batches.
Create the pipeline file and prepare the data stream

The adaptive pipeline needs two things your previous scripts did not: time-ordered data and a vectorizer that works without seeing the full corpus first.

  • In VS Code, create a new file called adaptive_pipeline.py in your adaptive-spam-detection folder.
  • Add the imports, data loading, date sorting, and train/test split by pasting this code:
"""Adaptive learning pipeline with incremental updates."""
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from scipy.sparse import hstack, csr_matrix
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.preprocessing import MaxAbsScaler
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import accuracy_score, f1_score

from preprocess import load_dataset, engineer_metadata_features, METADATA_COLUMNS

# Load and sort by date for temporal simulation
df = load_dataset("data/enron_spam_data.csv")
df = engineer_metadata_features(df)
df = df.dropna(subset=["Date"])
df = df.sort_values("Date").reset_index(drop=True)

# Hold out final 20% as fixed test set
split_idx = int(len(df) * 0.8)
df_stream = df.iloc[:split_idx].copy()
df_test = df.iloc[split_idx:].copy()

What does this code do?

Unlike your previous scripts that split data randomly, this pipeline sorts emails by their Date column first. This simulates a real inbox where emails arrive in chronological order.

Rows with missing dates are dropped via dropna(subset=["Date"]) because the pipeline needs every email to have a position in the time sequence. The final 20% of emails (the newest ones) are held out as a fixed test set so you can measure how the model improves as it sees more historical data.

Now you need a text vectorizer and metadata scaler that work in a streaming context. Your previous scripts used TfidfVectorizer, but that requires fitting on the entire corpus before it can transform anything.

  • Add the vectorizer, scaler, and test feature preparation below your existing code:
# Fixed-space text vectorizer (no fitting required)
hasher = HashingVectorizer(
    n_features=2**14, stop_words="english", alternate_sign=False
)

# Scale metadata (fit on training portion, apply to test)
scaler = MaxAbsScaler()
scaler.fit(df_stream[METADATA_COLUMNS].values)

X_test_text = hasher.transform(df_test["text"])
X_test_meta_scaled = scaler.transform(df_test[METADATA_COLUMNS].values)
X_test_fused = hstack([X_test_text, csr_matrix(X_test_meta_scaled)])
y_test = df_test["label"].values

Why HashingVectorizer instead of TfidfVectorizer?

TfidfVectorizer builds a vocabulary from all training data during fit(). If a future email contains a word the vocabulary has never seen, that word is silently ignored. This makes TfidfVectorizer unsuitable for streaming, where new vocabulary appears constantly.

HashingVectorizer hashes each word into a fixed-size feature space. It never needs to fit, so it handles any word at any time. The trade-off is that you lose the ability to look up which word maps to which feature, but for classification that does not matter.

Similarly, MaxAbsScaler replaces StandardScaler here because it scales by the maximum absolute value. It does not require knowing the full dataset's mean and variance upfront.

  • Save adaptive_pipeline.py.
  • Run the file to confirm the data loads and the vectorizer/scaler set up without errors:
python adaptive_pipeline.py

The script should exit cleanly with no output. That confirms the data loaded, sorted, split, and the test features were built successfully.

Seeing a KeyError for Date?

Make sure you are using the Enron Spam Dataset from Kaggle, which includes a Date column. If your CSV does not have this column, re-download the dataset from the Kaggle page.

help me debug a KeyError for the Date column in my adaptive_pipeline.py

Build the incremental learning loop

This is the core of adaptive learning. Instead of calling fit() once on all training data, you will call partial_fit() on one batch at a time. After each batch, the model evaluates itself on the held-out test set so you can watch its accuracy climb.

  • Add the incremental learning loop below your existing code in adaptive_pipeline.py:
# Incremental learning loop
clf = SGDClassifier(loss="log_loss", random_state=42)
batch_size = 1000
classes = np.array([0, 1])

accuracy_history = []
f1_history = []
n_samples_seen = []
total_seen = 0

for start in range(0, len(df_stream), batch_size):
    batch = df_stream.iloc[start : start + batch_size]
    if len(batch) == 0:
        break

    X_batch_text = hasher.transform(batch["text"])
    X_batch_meta = scaler.transform(batch[METADATA_COLUMNS].values)
    X_batch_fused = hstack([X_batch_text, csr_matrix(X_batch_meta)])
    y_batch = batch["label"].values

    clf.partial_fit(X_batch_fused, y_batch, classes=classes)
    total_seen += len(batch)

    y_pred = clf.predict(X_test_fused)
    acc = accuracy_score(y_test, y_pred)
    f1 = f1_score(y_test, y_pred)
    accuracy_history.append(acc)
    f1_history.append(f1)
    n_samples_seen.append(total_seen)
    print(f"Batch {start // batch_size + 1}: seen={total_seen}, accuracy={acc:.4f}, f1={f1:.4f}")

What does this code do?

  • SGDClassifier(loss="log_loss") creates a logistic regression model trained via stochastic gradient descent. Your earlier scripts used loss="hinge" (SVM), but log_loss is needed here because it supports predict_proba(), which you will need for the Secret Mission.
  • partial_fit() runs one epoch of SGD on just the current batch. The classes=classes parameter tells the model what labels to expect. It is required on the first call and ignored on subsequent ones.
  • After each batch, the model predicts on the full held-out test set. This lets you track how accuracy and F1 evolve as the model sees more data.
  • Save adaptive_pipeline.py.

Before you run this, what do you think the accuracy will look like on the very first batch versus the last batch? Will it start high or low?

  • Run the pipeline to see batch-by-batch learning in action:
python adaptive_pipeline.py

You should see output like this, with accuracy and F1 printed after each batch of 1,000 emails:

Batch 1: seen=1000, accuracy=0.XXXX, f1=0.XXXX
Batch 2: seen=2000, accuracy=0.XXXX, f1=0.XXXX
...

The early batches will have lower scores because the model has seen very little data. As more batches stream in, both accuracy and F1 should climb and stabilize.

Seeing very low accuracy that never improves?

Check that you are passing classes=classes to partial_fit(). Without it, the model does not know how many classes exist and predictions will be unreliable.

Also confirm that loss="log_loss" is set on the SGDClassifier, not loss="hinge".

help me debug low accuracy in my adaptive_pipeline.py incremental learning loop

Plot the learning curve and verify the results

Numbers scrolling past in the terminal prove the model works, but a chart makes the learning trajectory unmistakable. This visualization shows how quickly the model reaches strong performance and how stable it becomes.

  • Add the plotting code at the bottom of adaptive_pipeline.py:
# Plot learning curve
fig, ax = plt.subplots(figsize=(10, 5))
ax.plot(n_samples_seen, accuracy_history, marker="o", label="Accuracy")
ax.plot(n_samples_seen, f1_history, marker="s", label="F1 Score")
ax.set_xlabel("Training Samples Seen")
ax.set_ylabel("Score")
ax.set_title("Adaptive Learning: Model Performance Over Streaming Data")
ax.legend()
ax.set_ylim(0, 1.05)
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("outputs/adaptive_learning_curve.png", dpi=150)
print("\nChart saved to outputs/adaptive_learning_curve.png")
print(f"\nFinal accuracy: {accuracy_history[-1]:.4f}")
print(f"Final F1 score: {f1_history[-1]:.4f}")

What does this code do?

This creates a line chart with training samples seen on the x-axis and two lines: one for accuracy and one for F1 score. The chart is saved to outputs/adaptive_learning_curve.png. The final accuracy and F1 are printed so you can compare them against your static fusion model from Step 3.

  • Save adaptive_pipeline.py.

Before you run the final version, think about this: will the learning curve show a steady climb, a sudden jump, or something else?

  • Run the complete adaptive pipeline:
python adaptive_pipeline.py

You should see batch-by-batch progress printed, followed by the chart saved confirmation and final scores. The last two lines print the final accuracy and F1 score.

  • Open outputs/adaptive_learning_curve.png in your file explorer or VS Code to see the learning curve. You should see both the accuracy and F1 lines rising over the early batches and then levelling off as the model stabilizes.

Chart file not appearing in outputs/?

Make sure the outputs/ folder exists in your project directory. You created it in Step 1. If it is missing, create it by running:

mkdir outputs

Then re-run python adaptive_pipeline.py.

help me debug why my adaptive_pipeline.py is not saving the learning curve chart

✔️ Awesome, I've got everything!

Great. Double check you have saved adaptive_pipeline.py and that outputs/adaptive_learning_curve.png exists.

ⓧ I'd like to double check the full code

Scroll up through adaptive_pipeline.py and confirm these sections appear in order:

  • Imports (numpy, matplotlib, scipy, scikit-learn, and your preprocess module).
  • Data loading, metadata engineering, date sorting, and 80/20 time-ordered split into df_stream and df_test.
  • HashingVectorizer and MaxAbsScaler setup, plus test feature preparation (X_test_fused and y_test).
  • The incremental learning loop: SGDClassifier with log_loss, batch size of 1000, the for loop calling partial_fit() and printing per-batch results.
  • The plotting section saving the chart to outputs/adaptive_learning_curve.png and printing final accuracy and F1.

Your spam detection system can now learn on the fly. It streams emails in chronological batches and updates its model incrementally, without ever retraining from scratch. Next up, you will take this adaptive system even further in the Secret Mission.

Secret mission

Confidence-Based Drift Detection

Your adaptive pipeline learns from new data, but it retrains on every batch regardless of whether the model actually needs updating. What if the model could monitor its own prediction confidence and only retrain when it detects that spam patterns have shifted? In this mission, you will build a confidence-based drift detection system that watches for drops in prediction certainty and triggers targeted retraining.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your project files, pause your environment, or delete everything.

Resources you used:

  • The adaptive-spam-detection project folder on your Desktop (Python scripts, virtual environment, dataset, and generated plots).

Keep everything running

No action needed. Choose this if you want to keep experimenting with your spam detection models, tweak the metadata features, or extend the adaptive pipeline.

  • Your project folder, virtual environment, dataset, and all generated plots remain on your Desktop.
  • To resume working, open VS Code, activate your virtual environment, and run any of your scripts.

Pause - I'll come back to this later

Deactivate your virtual environment to free up your terminal. Your files stay on disk for when you return.

  • If your virtual environment is still active in your terminal, deactivate it by running:
deactivate
  • Your adaptive-spam-detection folder, dataset, scripts, and generated plots all remain on your Desktop.
  • To resume later, open a terminal in your project folder, reactivate the virtual environment, and pick up where you left off.

Delete - I don't want to use this again

Remove all project files, the virtual environment, the dataset, and generated plots from your machine.

  • If your virtual environment is still active, deactivate it first by running:
deactivate
  • Delete the entire project folder. On macOS or Linux, run:
cd ~/Desktop
rm -rf adaptive-spam-detection
  • On Windows, run:
cd ~\Desktop
rmdir /s /q adaptive-spam-detection
  • Confirm the folder is gone by running ls (macOS/Linux) or dir (Windows) on your Desktop. You should no longer see adaptive-spam-detection listed.

Nice Work!

Nice Work!

You just built a complete adaptive email spam detection system from scratch. You started with a simple text-only baseline, proved its limitations, fused it with engineered metadata features, ran a rigorous ablation study, and capped it off with an incremental learning pipeline that adapts to new data without full retraining.

You've learned how to:

  • Build a text-only baseline spam classifier using TF-IDF and SGDClassifier, and identify where it falls short on metadata-heavy spam.
  • Engineer 19 metadata features from raw email text and build a content-metadata fusion model that outperforms the baseline by combining TF-IDF text vectors with structured features via scipy.sparse.hstack.
  • Run a full ablation study comparing 3 classifiers across 3 feature sets, producing a publication-ready grouped bar chart with matplotlib.
  • Build an adaptive learning pipeline using HashingVectorizer and SGDClassifier.partial_fit that incrementally learns from streaming email batches without full retraining.
  • Secret Mission: Add confidence-based drift detection that monitors prediction certainty over a sliding window and auto-retrains when it drops.

Ready to quiz yourself?