Email Spam Detector with ML Fusion

Build a spam classifier that fuses email text and metadata features.

Introduction

30 Second Summary

Your inbox catches most spam automatically, but the cleverest phishing emails slip through because they sound perfectly normal. Their structural patterns are what give them away.

In this project, you will build a spam and phishing email detector that combines what an email says with how it is structured. You will train a text-only baseline first, then fuse in engineered metadata features to build a classifier that outperforms text analysis alone.

What You'll Build

Run your finished script on any email and watch it return a spam-or-legitimate prediction with confidence scores, backed by side-by-side evaluation figures you can drop straight into a university report.

By the end of this project, you'll have:

  • A text-only spam baseline using TF-IDF and Logistic Regression that reveals where analyzing words alone falls short on social engineering emails.
  • Engineered metadata features extracted from raw email data that capture structural spam signals like excessive capitals, suspicious URL counts, and missing sender information.
  • A fused classifier that combines text and metadata via scikit-learn's ColumnTransformer, with side-by-side confusion matrices, ROC curves, and comparison bar charts proving it outperforms the baseline.
  • Secret Mission: Load a pre-trained RoBERTa transformer from Hugging Face and benchmark it against your fusion model for a three-way comparison.

Are there any prerequisites?

No prior machine learning experience is required. This guide walks you through setting up Python and every library you need from scratch.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Python Environment

Before you can analyze emails or train a spam classifier, you need a Python environment with the right data science libraries. This step gets your tools installed and verified.

You will check your Python version, create a dedicated project folder with a virtual environment, and install four libraries: scikit-learn, pandas, matplotlib, and the Hugging Face datasets library.

In this step, get ready to:
  • Check your Python version and confirm it is 3.11 or higher.
  • Create a project folder and a virtual environment.
  • Install the required data science libraries.
Check your Python version

Python 3.11 or higher is required for the specific library versions this project uses. Let's confirm your installation before setting up anything else.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Open the integrated terminal by pressing Ctrl+` (the backtick key, usually above Tab).
  • Check your Python version by running this command:

macOS

python3 --version

Windows

py --version

You should see a version number printed in the terminal. Check which scenario matches what you see:

✔️ I see version 3.11 or higher

Your Python version is ready. Move on to the pip check below.

ⓧ I see an older version

You have Python installed, but the version is too old for the libraries this project uses. You need 3.11 or higher.

  • Download the latest Python installer from python.org/downloads.
  • Run the installer and follow the prompts.
  • On Windows, check the Add Python to PATH checkbox during installation.

macOS users with an old system Python

macOS often ships with an older Python that the system itself uses. Installing a newer version from python.org does not replace it. After installation, try running python3.11 --version or python3.12 --version to target the version you just installed.

Use whichever versioned command prints 3.11 or higher in place of python3 for the rest of this project.

After installing, close and reopen your VS Code terminal, then re-run the version check to confirm you see 3.11 or higher.

ⓧ Command not found

Python is not installed on your machine yet. Let's fix that.

  • Go to python.org/downloads and download the latest Python installer for your operating system.
  • Run the installer and follow the prompts.
  • On Windows, check the Add Python to PATH checkbox during installation.
  • After installing, close and reopen your VS Code terminal, then re-run the version check command above.
  • Verify that pip is available by running this command:

macOS

python3 -m pip --version

Windows

py -m pip --version

You should see a version number for pip printed in the terminal.

pip not found?

If you installed Python from python.org, pip is bundled automatically. Try closing and reopening your VS Code terminal, then run the command again.

help me fix a missing pip installation

Create your project folder and virtual environment

A virtual environment keeps this project's libraries separate from your system Python. This prevents version conflicts with other projects on your machine.

  • Move to your Desktop and create the project folder by running these commands:
cd ~/Desktop
mkdir spam-detector
cd spam-detector
  • Confirm you are inside the folder by running:
ls

The folder is empty, so ls should produce no output. That confirms you are in the right place.

  • Create and activate a virtual environment by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

py -m venv venv
venv\Scripts\activate

Your terminal prompt should now show (venv) at the beginning. This means the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated copy of Python that lives inside your project folder. When you install libraries with pip while the environment is active, they go into the venv folder instead of your global Python installation.

This means you can use specific library versions for this project without affecting anything else on your machine.

Don't see (venv) in your prompt?

On Windows, if you see an error about running scripts being disabled, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned and then try the activate command again.

On macOS, make sure you used source venv/bin/activate (with source at the start). Without it, the command will not activate the environment.

help me activate my Python virtual environment

Install the required libraries

With your virtual environment active, you can now install the four libraries this project depends on. Each one handles a different piece of the spam detection workflow.

  • Install all required libraries in one command by running:
pip install scikit-learn==1.9.0 pandas==3.0.5 matplotlib==3.11.1 datasets==5.0.1

What do these libraries do?

  • scikit-learn provides the machine learning tools you will use to build and evaluate your classifiers.
  • pandas handles loading and manipulating the email data as tables.
  • matplotlib creates the evaluation charts and figures for your report.
  • datasets is the Hugging Face library that downloads the email corpus you will train on.

The install may take a minute or two. Wait until you see the terminal return to the (venv) prompt before continuing.

Before you run the verification command, what do you expect to see if all four libraries installed correctly?

  • Verify that every library loads successfully by running this command:
python -c "import sklearn; import pandas; import matplotlib; import datasets; print('All libraries loaded successfully')"

You should see All libraries loaded successfully printed in the terminal. This confirms every library is installed and importable.

Seeing a ModuleNotFoundError?

Make sure your terminal prompt still shows (venv). If it disappeared, your virtual environment was deactivated. Re-activate it and run the install command again.

If the error names a specific library, re-run the full pip install command. A network interruption may have caused one package to fail while the others succeeded.

help me fix a ModuleNotFoundError in my spam detector project

Your Python environment is locked and loaded. Next up, you will load a real email dataset from Hugging Face and build your first spam classifier.

Load the Dataset and Build a Text-Only Baseline

Your Python environment is ready with all the libraries you need. Now it is time to load real email data and find out how well a text-only classifier can separate spam from legitimate messages.

Before you can improve spam detection with metadata, you need to see how far text alone gets you. Building a baseline first reveals the gap your fusion approach will fill in later steps.

In this step, get ready to:
  • Load a real multi-source email corpus from Hugging Face and explore its structure.
  • Build a text-only spam classifier using TF-IDF and Logistic Regression.
  • Evaluate the baseline and identify where it struggles.
Create the script and load the email dataset

Your spam detector lives in a single Python script. Before training any model, you need to load the email dataset, clean out unusable rows, and understand what you are working with.

  • In the VS Code Explorer sidebar, click the New File icon next to the spam-detector folder name.
  • Name the file spam_detector.py and press Enter.
  • Add the following imports to the top of spam_detector.py:
import warnings
warnings.filterwarnings("ignore")

import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

from datasets import load_dataset

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.metrics import (
    classification_report,
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    roc_auc_score,
    ConfusionMatrixDisplay,
    RocCurveDisplay,
)

What do these imports do?

  • warnings.filterwarnings("ignore") silences convergence warnings from the ML models so your terminal output stays clean.
  • matplotlib.use("Agg") sets a non-interactive backend so plots save to files instead of trying to open GUI windows.
  • The sklearn imports cover everything you will use across the entire project, including tools for later steps like ColumnTransformer and RocCurveDisplay.

The dataset you will use aggregates seven public email corpora into one collection. It contains over 200,000 emails with both text content and metadata fields.

  • Below your imports, add this code to load the dataset, clean it, and print its structure:
# ── Step 2: Load dataset and build text-only baseline ──

print("Loading dataset from Hugging Face...")
ds = load_dataset("mamtakumar/seven-phishing-email-datasets", split="train")
df = ds.to_pandas()

df = df.dropna(subset=["text"])
df = df[df["text"].str.strip().astype(bool)].reset_index(drop=True)

print(f"Total samples: {len(df)}")
print(f"Class distribution:\n{df['label'].value_counts().rename({0: 'Legitimate (0)', 1: 'Phishing/Spam (1)'})}\n")

What does this code do?

  • load_dataset downloads the mamtakumar/seven-phishing-email-datasets corpus from Hugging Face and caches it locally so future runs skip the download.
  • to_pandas() converts the dataset into a pandas DataFrame so you can filter, slice, and engineer features easily.
  • dropna(subset=["text"]) removes emails with missing text, and the second filter removes rows where the text is only whitespace.
  • The label column uses 0 for legitimate emails and 1 for phishing/spam.
  • Save spam_detector.py.
  • Run the script in your VS Code terminal by running:
python spam_detector.py

What should I see?

The first run downloads the dataset, so you will see progress bars while data is fetched from Hugging Face. This can take a minute or two depending on your internet speed.

Once downloaded, the script prints the total sample count and a class distribution showing how many emails are legitimate (0) vs. phishing/spam (1).

Seeing a download error?

  • Check your internet connection. The dataset download requires a stable connection to Hugging Face servers.
  • Make sure the virtual environment is activated. Your terminal prompt should start with (venv).
  • If you see a ModuleNotFoundError, re-run the install command from Step 1 to make sure all libraries are present.

Help me debug a download error when loading the Hugging Face dataset in my spam detector script.

Build and train the text-only pipeline

With the data loaded, you can now build your first classifier. A TF-IDF vectorizer converts each email's raw text into a numerical feature vector, and Logistic Regression learns which word patterns signal spam.

  • Add the following code below your data loading section in spam_detector.py to split the data and train the pipeline:
X_text = df["text"]
y = df["label"]

# Split into 80% training, 20% test (stratified to keep class balance)
X_train_text, X_test_text, y_train, y_test = train_test_split(
    X_text, y, test_size=0.2, stratify=y, random_state=42
)

# Build a text-only pipeline: TF-IDF vectorizer → Logistic Regression
text_pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=10000, stop_words="english")),
    ("clf", LogisticRegression(max_iter=1000, random_state=42)),
])

print("Training text-only model...")
text_pipeline.fit(X_train_text, y_train)

What does this code do?

  • train_test_split holds back 20% of the data for testing. stratify=y ensures both sets keep the same ratio of spam to legitimate emails.
  • TfidfVectorizer(max_features=10000) converts each email into a vector of the 10,000 most important words, weighted by how distinctive each word is across the corpus.
  • scikit-learn's Pipeline chains the vectorizer and classifier into one object. When you call .fit(), it vectorizes the text and trains the model in one step.
  • Save spam_detector.py and re-run the script:
python spam_detector.py

You should see the dataset stats from before, followed by a new line: Training text-only model... and then the script finishes without errors.

Script hanging during training?

  • Training on 200,000+ emails can take 10-20 seconds. Wait for the script to finish before assuming something is wrong.
  • If you see a NameError for df or X_text, make sure the new code is placed below the data loading section, not above it.

Help me debug a training issue with my text-only spam classifier pipeline.

Evaluate the baseline and spot its weakness

The model is trained. Now you need to measure how well it performs on emails it has never seen, and save a confusion matrix that visualizes where it gets things right and wrong.

  • Add the following code below your training section in spam_detector.py to generate predictions, print a classification report, and save a confusion matrix plot:
y_pred_text = text_pipeline.predict(X_test_text)

print("\n=== TEXT-ONLY MODEL ===")
print(classification_report(y_test, y_pred_text, target_names=["Legitimate", "Spam/Phishing"]))

# Save a confusion matrix showing correct vs incorrect predictions
fig, ax = plt.subplots(figsize=(6, 5))
ConfusionMatrixDisplay.from_predictions(
    y_test, y_pred_text,
    display_labels=["Legitimate", "Spam/Phishing"],
    cmap="Blues", ax=ax,
)
ax.set_title("Text-Only Model - Confusion Matrix")
plt.tight_layout()
plt.savefig("text_only_confusion.png", dpi=150)
plt.close()
print("Saved: text_only_confusion.png")

What does this code do?

  • classification_report prints precision, recall, and F1 scores for each class. These metrics tell you not just overall accuracy, but how well the model handles each category.
  • ConfusionMatrixDisplay.from_predictions creates a heatmap showing how many emails were correctly classified vs. misclassified. The diagonal shows correct predictions.
  • The plot saves to text_only_confusion.png in your project folder so you can include it in your report later.
  • Save spam_detector.py.

Before you run the script, take a guess: do you think a text-only model can correctly classify over 95% of spam emails, or will it struggle?

  • Run the full script by running:
python spam_detector.py

You should see the full classification report printed under the === TEXT-ONLY MODEL === header, showing accuracy, precision, recall, and F1 scores for both classes. At the end, the script confirms it saved text_only_confusion.png.

  • Confirm the confusion matrix image was saved by running:
ls

You should see text_only_confusion.png listed alongside spam_detector.py and your venv folder.

Why does this model struggle with social engineering?

The text-only model performs reasonably on obvious spam full of words like "free", "winner", and "click here". But social engineering emails deliberately use normal, professional-sounding language to trick both people and text classifiers.

Look at the recall score for Spam/Phishing in your report. Any misses represent phishing emails that slipped through because their text alone did not look suspicious. In the next step, you will engineer metadata features that catch the structural giveaways these emails leave behind.

Don't see the confusion matrix file?

  • Make sure the matplotlib.use("Agg") line is near the top of your imports, before import matplotlib.pyplot as plt. Without it, matplotlib may try to open a GUI window and fail.
  • Check that plt.savefig("text_only_confusion.png", dpi=150) appears before plt.close(). Closing the figure before saving it produces an empty file.

Help me debug why my confusion matrix plot is not saving to a file.

✔️ Awesome, I've got everything!

Great. Double check you have saved spam_detector.py and that text_only_confusion.png exists in your project folder.

ⓧ I'd like to double check the full code

Your spam_detector.py should contain these sections in order. Scroll up and confirm each one is present:

  • The import block (warnings, numpy, pandas, matplotlib, datasets, and all sklearn imports).
  • The dataset loading section (load_dataset, dropna, empty-text filter, print statements).
  • The train/test split and pipeline section (train_test_split, Pipeline with TfidfVectorizer and LogisticRegression, fit).
  • The evaluation section (predict, classification_report, ConfusionMatrixDisplay, savefig).

You have a working baseline, and its classification report reveals a clear gap. Next up, you will engineer metadata features from the raw email data to capture the structural signals that text analysis misses.

Engineer Metadata Features

Your text-only model from the previous step can catch obvious spam keywords. But look at the confusion matrix you saved. Some phishing emails slip through because they use perfectly normal-sounding language.

These social engineering emails give themselves away in other ways: excessive capital letters, suspicious numbers of URLs, or missing sender information. In this step, you will extract those structural signals into new metadata features that your model can learn from.

In this step, get ready to:
  • Write a feature engineering function that extracts six metadata signals from raw email data.
  • Run the function and inspect the engineered features alongside the original email text.
  • Compare average metadata values between spam and legitimate emails to confirm the structural differences.
Write the feature engineering function

Each metadata feature captures a structural property of the email that TF-IDF text analysis cannot see. For example, a high ratio of capital letters screams "urgency" even if the individual words are common.

  • In spam_detector.py, scroll to the very bottom of your file (below the print("Saved: text_only_confusion.png") line).
  • Add the following function by pasting this code at the bottom:

# ── Step 3: Engineer metadata features ──

def engineer_features(dataframe):
    out = dataframe.copy()
    out["text_length"] = out["text"].str.len()
    out["capital_ratio"] = out["text"].apply(
        lambda t: sum(1 for c in t if c.isupper()) / max(len(t), 1)
    )
    out["punctuation_density"] = out["text"].apply(
        lambda t: sum(1 for c in t if c in "!?.,:;\"'()-") / max(len(t), 1)
    )
    out["subject_length"] = out["subject"].fillna("").str.len()
    out["has_sender"] = out["sender"].notna().astype(int)
    out["url_count"] = out["urls"].fillna(0).astype(int)
    return out

What does this code do?

  • text_length counts how many characters are in the email body. Spam tends to be either very short ("Click here!") or very long (walls of scam text).
  • capital_ratio measures what fraction of the text is uppercase. Spam loves ALL CAPS.
  • punctuation_density checks how much of the text is punctuation. Excessive exclamation marks and special characters are common spam giveaways.
  • subject_length, has_sender, and url_count capture email header metadata. Missing senders and high URL counts are strong phishing indicators.
  • The url_count feature reuses the existing urls column from the dataset, which already contains URL counts. The code fills missing values with 0 and converts to integer.
  • Save spam_detector.py.
Call the function and inspect the results

The function is defined but not called yet. Now you need to run it on your DataFrame and print the results so you can visually confirm the features look correct.

  • Add the following code below the engineer_features function you just wrote:

df = engineer_features(df)

metadata_cols = [
    "text_length", "capital_ratio", "punctuation_density",
    "url_count", "subject_length", "has_sender",
]

print("\nFirst 5 rows of engineered features:")
print(df[["text"] + metadata_cols].head().to_string())

print("\nMean metadata by class:")
print(df.groupby("label")[metadata_cols].mean().round(4).to_string())

What does this code do?

  • engineer_features(df) runs the function on your full DataFrame and reassigns the result back to df, adding all six new columns.
  • metadata_cols stores the list of new column names so you can reference them easily in later steps.
  • The first print shows the raw feature values for five sample emails, so you can eyeball whether the numbers make sense.
  • The second print groups all emails by label (0 = legitimate, 1 = spam/phishing) and shows the average of each metadata feature per class. This reveals the structural patterns your model will learn.

Before you run this, think about what you expect. Do you think spam emails will have a higher or lower capital letter ratio than legitimate ones? What about URL count?

  • Save spam_detector.py.
  • Run the updated script by running this command in your terminal:
python spam_detector.py

After the text-only classification report from Step 2, you should now see two new sections in the output. First, a table showing the first 5 rows of engineered features alongside the email text. Second, a summary showing the mean value of each metadata feature grouped by label.

What do the summary statistics reveal?

Look at the mean metadata by class. Spam/phishing emails (label 1) should show noticeably different averages compared to legitimate emails (label 0).

For example, spam emails typically show a higher capital_ratio and higher url_count. These are exactly the structural signals that your text-only model could not see. In the next step, you will fuse these features with the TF-IDF text features so the model can use both signal streams at once.

Seeing a KeyError or missing column?

  • If you see KeyError: 'text' or a similar column error, make sure the engineer_features function is called AFTER the dataset loading code from Step 2. The function needs the df DataFrame to already exist.
  • If you see a SyntaxError on the lambda lines, check that the parentheses and quotes match exactly. The punctuation_density lambda contains escaped quotes inside the string.
  • Make sure you pasted the function definition ABOVE the lines that call it. Python reads top to bottom, so df = engineer_features(df) must come after the def engineer_features block.

Help me debug my engineer_features function in spam_detector.py

✔️ Awesome, I've got everything!

Great. Double check you have saved spam_detector.py and that both the feature rows table and the mean metadata by class printed in your terminal.

ⓧ I'd like to double check the full code

Your spam_detector.py is now too long for a single code block. Scroll up through your file and confirm these sections appear in order:

  • All imports at the top (warnings, numpy, pandas, matplotlib, datasets, and all sklearn imports).
  • Step 2 code: dataset loading, train/test split, text_pipeline training, classification report, and confusion matrix save.
  • Step 3 comment: # ── Step 3: Engineer metadata features ──.
  • The engineer_features function definition with all six feature columns.
  • df = engineer_features(df) followed by the metadata_cols list and both print statements.

You now have six structural features that capture patterns invisible to text analysis. Next up, you will fuse these metadata features with the TF-IDF text features into a single model and see how much the detection improves.

Build the Fused Model and Compare

In the last step, you discovered that spam and legitimate emails have measurably different structural fingerprints. Features like capital letter ratios, URL counts, and punctuation density all shift between the two classes.

Now comes the payoff. You will fuse these metadata features with the text-based TF-IDF features into a single classifier using scikit-learn's ColumnTransformer. Then you will compare the fused model head-to-head against your text-only baseline to measure whether the structural signals actually move the needle.

In this step, get ready to:
  • Build a fused pipeline that combines text and metadata features with ColumnTransformer.
  • Evaluate the fused model with a classification report and confusion matrix.
  • Create ROC curves and a comparison bar chart showing the improvement over the text-only model.
Prepare training data and build the fused pipeline

Your text-only model used a flat series of email text strings. The fused model needs a DataFrame containing both the text column and all six metadata columns, so ColumnTransformer can route each to its own transformer.

  • In spam_detector.py, scroll to the bottom of the file (below the print(df.groupby("label")...) line) and add this code to prepare the train/test DataFrames:
# ── Step 4: Build fused model and compare ──

# Reuse the same train/test split indices from the text-only model
train_idx = X_train_text.index
test_idx = X_test_text.index

# Build DataFrames with text + metadata columns for the fused model
X_train_df = df.loc[train_idx, ["text"] + metadata_cols].reset_index(drop=True)
X_test_df = df.loc[test_idx, ["text"] + metadata_cols].reset_index(drop=True)
y_train_reset = y_train.reset_index(drop=True)
y_test_reset = y_test.reset_index(drop=True)

What does this code do?

You already split the data in Step 2, so you reuse those exact indices (train_idx, test_idx) to ensure a fair comparison. Both models see the same emails during training and testing.

The reset_index(drop=True) calls realign the row numbers so scikit-learn can match rows between the feature DataFrame and the label series.

  • Now build the fused pipeline and train it by adding this code directly below:
# ColumnTransformer routes each column group to its own transformer
preprocessor = ColumnTransformer(
    transformers=[
        ("tfidf", TfidfVectorizer(max_features=10000, stop_words="english"), "text"),
        ("meta", "passthrough", metadata_cols),
    ]
)

# Wrap the preprocessor and classifier into a single Pipeline
fused_pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("clf", LogisticRegression(max_iter=1000, random_state=42)),
])

print("\nTraining fused model...")
fused_pipeline.fit(X_train_df, y_train_reset)
y_pred_fused = fused_pipeline.predict(X_test_df)

What is ColumnTransformer doing here?

ColumnTransformer is scikit-learn's tool for applying different transformations to different columns. Here it sends the text column through TfidfVectorizer and passes the six metadata columns through unchanged ("passthrough").

Behind the scenes, it concatenates the sparse TF-IDF matrix with the dense metadata array into a single feature set. The LogisticRegression classifier then learns from both signal streams at once.

Why ColumnTransformer instead of manual concatenation?

You could manually vectorize the text, stack the metadata array beside it, and feed the result to a classifier. But ColumnTransformer handles sparse-plus-dense concatenation automatically. It also fits cleanly inside a Pipeline, so your entire model (preprocessing + classifier) is a single object you can fit, predict, and evaluate in one call.

  • Save spam_detector.py.
  • Run the script to confirm the fused model trains successfully:
python spam_detector.py

You should see all the prior output from Steps 2 and 3, followed by a new line: Training fused model.... The script finishes without errors.

Seeing a KeyError or column mismatch?

  • Make sure the metadata_cols list matches the exact column names created by engineer_features in Step 3.
  • Check that your Step 4 code is placed after the Step 3 code, not inside the engineer_features function body.

Help me debug a KeyError in my ColumnTransformer fused pipeline.

Generate predictions and evaluate the fused model

The fused model trained successfully, but you do not know yet whether the metadata features actually improved anything. Time to print the classification report and save a confusion matrix so you can compare.

  • Add this code below the y_pred_fused line to print the report and save the confusion matrix:
print("\n=== FUSED MODEL ===")
print(classification_report(y_test_reset, y_pred_fused, target_names=["Legitimate", "Spam/Phishing"]))

# Save the fused model's confusion matrix with a different colormap
fig, ax = plt.subplots(figsize=(6, 5))
ConfusionMatrixDisplay.from_predictions(
    y_test_reset, y_pred_fused,
    display_labels=["Legitimate", "Spam/Phishing"],
    cmap="Oranges", ax=ax,
)
ax.set_title("Fused Model - Confusion Matrix")
plt.tight_layout()
plt.savefig("fused_confusion.png", dpi=150)
plt.close()
print("Saved: fused_confusion.png")

What does this code do?

  • classification_report prints precision, recall, and F1 for each class, plus overall accuracy. You can compare these numbers directly against the text-only report from Step 2.
  • The confusion matrix uses the "Oranges" colormap so you can visually distinguish it from the text-only model's blue matrix.
  • Save spam_detector.py.

Before you run the script, take a moment to predict: do you think the fused model will outperform the text-only baseline across all metrics, or only some?

  • Run the script to see the fused model's evaluation:
python spam_detector.py

You should see the fused model's classification report printed under === FUSED MODEL ===, followed by Saved: fused_confusion.png. Compare the precision, recall, and F1 numbers to the text-only report above it. The fused model should show improvement, especially in recall for spam/phishing emails.

Fused model scores are identical to text-only?

  • Verify that your ColumnTransformer includes ("meta", "passthrough", metadata_cols) as the second transformer. Without it, only the text features reach the classifier.
  • Make sure you are fitting on X_train_df (the DataFrame with text + metadata), not X_train_text (text only).

Help me figure out why my fused model scores match the text-only model.

Plot ROC curves and create a comparison bar chart

Numbers in a table are useful, but visualizations tell the story at a glance. An ROC curve plots the trade-off between true positive rate and false positive rate. A model whose curve hugs the top-left corner is better at separating the classes.

  • Add this code below the confusion matrix block to plot overlaid ROC curves for both models:
# ROC curves
fig, ax = plt.subplots(figsize=(8, 6))
RocCurveDisplay.from_estimator(
    text_pipeline, X_test_text, y_test,
    name="Text-Only", ax=ax,
)
RocCurveDisplay.from_estimator(
    fused_pipeline, X_test_df, y_test_reset,
    name="Fused (Text + Metadata)", ax=ax,
)
ax.set_title("ROC Curve Comparison")
ax.plot([0, 1], [0, 1], "k--", label="Random")
ax.legend()
plt.tight_layout()
plt.savefig("roc_comparison.png", dpi=150)
plt.close()
print("Saved: roc_comparison.png")

What does this code do?

  • RocCurveDisplay.from_estimator plots each model's ROC curve on the same axes, so you can visually compare their performance.
  • The dashed diagonal line ("k--") represents a random classifier. Any model worth using should curve well above this line.
  • Save spam_detector.py and run the script:
python spam_detector.py

You should see Saved: roc_comparison.png in the output. Open the file from your spam-detector folder to see both ROC curves overlaid on the same figure.

ROC plot is empty or missing a curve?

  • Each from_estimator call must pass the correct test data. The text-only model gets X_test_text and y_test. The fused model gets X_test_df and y_test_reset.
  • Make sure both calls share the same ax=ax argument so the curves land on the same plot.

Help me fix my overlaid ROC curve plot.

Now you will compute accuracy, precision, recall, and F1 for both models and prepare the data for a grouped bar chart.

  • Add this code below the ROC curve block to compute the comparison metrics:
# Comparison bar chart
metrics_text = {
    "Accuracy": accuracy_score(y_test, y_pred_text),
    "Precision": precision_score(y_test, y_pred_text),
    "Recall": recall_score(y_test, y_pred_text),
    "F1": f1_score(y_test, y_pred_text),
}
metrics_fused = {
    "Accuracy": accuracy_score(y_test_reset, y_pred_fused),
    "Precision": precision_score(y_test_reset, y_pred_fused),
    "Recall": recall_score(y_test_reset, y_pred_fused),
    "F1": f1_score(y_test_reset, y_pred_fused),
}

labels = list(metrics_text.keys())
text_vals = [metrics_text[k] for k in labels]
fused_vals = [metrics_fused[k] for k in labels]

What does this code do?

metrics_text and metrics_fused each store four evaluation scores as dictionaries. These dictionaries power the bar chart and will also be reused in Step 5 for the formatted summary table.

  • Now add the code that plots the grouped bar chart directly below:
x = np.arange(len(labels))
width = 0.35

fig, ax = plt.subplots(figsize=(8, 5))
bars1 = ax.bar(x - width / 2, text_vals, width, label="Text-Only", color="#4C72B0")
bars2 = ax.bar(x + width / 2, fused_vals, width, label="Fused", color="#DD8452")

ax.set_ylabel("Score")
ax.set_title("Model Comparison: Text-Only vs Fused")
ax.set_xticks(x)
ax.set_xticklabels(labels)
ax.set_ylim(0, 1.05)
ax.legend()

# Add exact score labels above each bar
for bar in bars1:
    ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.01,
            f"{bar.get_height():.3f}", ha="center", va="bottom", fontsize=8)
for bar in bars2:
    ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.01,
            f"{bar.get_height():.3f}", ha="center", va="bottom", fontsize=8)

plt.tight_layout()
plt.savefig("model_comparison.png", dpi=150)
plt.close()
print("Saved: model_comparison.png")

What does this code do?

  • Two sets of bars are placed side by side for each metric (Accuracy, Precision, Recall, F1). Blue bars represent the text-only model and orange bars represent the fused model.
  • The loops at the end add the exact numeric scores above each bar, so your chart is self-documenting and ready to paste into a report.
  • Save spam_detector.py.

Before you run, predict: will the bar chart show the fused model winning on every metric, or will the text-only model hold its own on some?

  • Run the script one final time for this step:
python spam_detector.py

You should see Saved: model_comparison.png at the end of the output.

  • Open model_comparison.png from your spam-detector folder. You should see a grouped bar chart with blue bars (Text-Only) and orange bars (Fused) for each metric, with numeric scores printed above each bar.
  • Confirm you also have fused_confusion.png and roc_comparison.png saved in the same folder.

Bar chart not saving or looks wrong?

  • If the file is missing, check that plt.savefig("model_comparison.png", dpi=150) appears before plt.close(). Closing the figure first produces an empty file.
  • If the chart appears but bars overlap, make sure width = 0.35 is set and the two ax.bar calls use x - width / 2 and x + width / 2 to offset the bars.

Help me fix my matplotlib grouped bar chart.

✔️ Awesome, I've got everything!

Great. Double check you have saved spam_detector.py and confirmed all three new images (fused_confusion.png, roc_comparison.png, model_comparison.png) exist in your spam-detector folder.

ⓧ I'd like to double check the full code

Your spam_detector.py should now contain all of the code from Steps 2 and 3, followed by the Step 4 code you just added. Scroll through your file and confirm these blocks are present in order:

  • Imports and warnings suppression at the top.
  • Step 2: Dataset loading, train/test split, text-only pipeline training, classification report, and text_only_confusion.png save.
  • Step 3: engineer_features function, df = engineer_features(df) call, metadata_cols list, and summary statistics printout.
  • Step 4: Train/test DataFrames (X_train_df, X_test_df), ColumnTransformer preprocessor, fused pipeline training, classification report, fused_confusion.png save, ROC curve comparison, metrics computation, and bar chart save.

Your fused model is trained, evaluated, and visualized alongside the text-only baseline. Next up, you will extract feature importances and produce a formatted summary table ready for your university report.

Generate Academic Evaluation Outputs

In the last step, you built the fused model and proved it outperforms the text-only baseline with comparison charts, classification reports, and ROC curves.

Your advisor needs more than accuracy numbers though. This step produces the complete set of academic figures and a formatted summary you can paste directly into your report's results section.

In this step, get ready to:
  • Extract and visualize the top 15 most important features from the fused model.
  • Print a formatted summary table comparing both models across all metrics.
  • Add a prediction demo that classifies a sample email through both models.
Extract and visualize feature importances

Logistic Regression stores a coefficient for every feature it learned from. The larger the absolute coefficient, the more that feature influences the spam-or-legitimate decision.

Plotting the top 15 coefficients lets you show your advisor exactly which signals drive the fused model. You can demonstrate that metadata features like capital_ratio rank alongside high-signal words.

  • In spam_detector.py, scroll to the very bottom of the file (after the comparison bar chart code) and add this code:


# ── Step 5: Academic evaluation outputs ──

# Feature importance
clf = fused_pipeline.named_steps["clf"]
feature_names = fused_pipeline.named_steps["preprocessor"].get_feature_names_out()
coefs = clf.coef_[0]

top_n = 15
top_idx = np.argsort(np.abs(coefs))[-top_n:]
top_names = feature_names[top_idx]
top_coefs = coefs[top_idx]

fig, ax = plt.subplots(figsize=(8, 6))
colors = ["#DD8452" if c > 0 else "#4C72B0" for c in top_coefs]
ax.barh(range(top_n), top_coefs, color=colors)
ax.set_yticks(range(top_n))
ax.set_yticklabels(top_names, fontsize=8)
ax.set_xlabel("Coefficient Value")
ax.set_title("Top 15 Feature Importances (Fused Model)")
plt.tight_layout()
plt.savefig("feature_importance.png", dpi=150)
plt.close()
print("Saved: feature_importance.png")

What does this code do?

fused_pipeline.named_steps["clf"].coef_[0] pulls the trained coefficients from the Logistic Regression inside your pipeline. Each coefficient tells you how much a single feature pushes the prediction toward spam (positive) or legitimate (negative).

get_feature_names_out() retrieves human-readable names for all features, including TF-IDF word tokens and your metadata columns. The code sorts by absolute value, takes the top 15, and plots them as a horizontal bar chart color-coded by direction.

  • Save spam_detector.py.
  • Run the script by entering this command in your terminal:
python spam_detector.py

Scroll through the terminal output. At the bottom, you should see Saved: feature_importance.png after the comparison chart output from the previous step.

  • Open feature_importance.png from the file explorer in VS Code to see your chart. You should see a horizontal bar chart with 15 features. Orange bars push the prediction toward spam, blue bars push toward legitimate.

Not seeing the feature importance plot?

  • Make sure you saved spam_detector.py before running. Unsaved changes are not executed.
  • Check that the new code is placed after the comparison bar chart section, not inside it. The section comment # ── Step 5 should appear on its own line below the last plt.close() from Step 4.
  • If you see a KeyError, double check that named_steps["clf"] and named_steps["preprocessor"] match the step names you used in your fused_pipeline from Step 4.
  • Still stuck? Help me debug why my feature importance chart isn't being saved.
Print a formatted summary table

A report-ready summary table needs more than raw scores. It should include dataset metadata, a side-by-side metric comparison, and the delta between models so your reader can see the improvement at a glance.

  • Add this code directly below the feature importance block in spam_detector.py:
# Summary table
text_auc = roc_auc_score(y_test, text_pipeline.predict_proba(X_test_text)[:, 1])
fused_auc = roc_auc_score(y_test_reset, fused_pipeline.predict_proba(X_test_df)[:, 1])

print("\n" + "=" * 60)
print("SUMMARY TABLE FOR REPORT")
print("=" * 60)
print(f"Dataset: mamtakumar/seven-phishing-email-datasets")
print(f"Total samples: {len(df)}")
print(f"Train/Test split: 80/20 (stratified)")
print(f"Class distribution: {dict(df['label'].value_counts())}")
print("-" * 60)
print(f"{'Metric':<15} {'Text-Only':>12} {'Fused':>12} {'Delta':>12}")
print("-" * 60)
for metric in labels:
    t = metrics_text[metric]
    f = metrics_fused[metric]
    print(f"{metric:<15} {t:>12.4f} {f:>12.4f} {f - t:>+12.4f}")
print(f"{'ROC-AUC':<15} {text_auc:>12.4f} {fused_auc:>12.4f} {fused_auc - text_auc:>+12.4f}")
print("=" * 60)

What does this code do?

predict_proba returns the probability each model assigns to each class. Taking column [:, 1] gives the spam probability, which roc_auc_score uses to compute the area under the ROC curve.

The formatted table prints each metric for both models side by side, with a delta column showing the exact improvement. The + sign in the delta format makes positive improvements immediately visible. This entire block is designed to be copy-pasted directly into your report.

  • Save spam_detector.py and run the script again:
python spam_detector.py

Scroll to the bottom of your terminal output. You should see a neatly formatted table with column headers for Metric, Text-Only, Fused, and Delta, showing Accuracy, Precision, Recall, F1, and ROC-AUC for both models.

Not seeing the summary table?

  • If you see a NameError for labels or metrics_text, make sure the comparison bar chart code from Step 4 is still intact above this section. Those variables are defined there.
  • If the table appears but numbers look wrong, confirm that y_test is used with text_pipeline and y_test_reset is used with fused_pipeline. Mixing these up causes index misalignment.
  • Still stuck? Help me debug why my summary table isn't printing correctly.
Add a prediction demo

A report's demonstration section is stronger when you show the detector classifying a real-looking email. This final block feeds a sample spam email through both models so your reader can see the predictions and confidence scores side by side.

  • Add this code at the bottom of spam_detector.py, directly after the summary table block:
# Prediction demo
sample_email = "Congratulations! You have been selected to receive a FREE gift card worth $1000. Click here NOW to claim your prize before it expires!"
text_pred = text_pipeline.predict([sample_email])[0]
text_proba = text_pipeline.predict_proba([sample_email])[0]

sample_df = pd.DataFrame([{
    "text": sample_email,
    "text_length": len(sample_email),
    "capital_ratio": sum(1 for c in sample_email if c.isupper()) / len(sample_email),
    "punctuation_density": sum(1 for c in sample_email if c in "!?.,:;\"'()-") / len(sample_email),
    "url_count": 0,
    "subject_length": 0,
    "has_sender": 0,
}])
fused_pred = fused_pipeline.predict(sample_df)[0]
fused_proba = fused_pipeline.predict_proba(sample_df)[0]

print("\n=== PREDICTION DEMO ===")
print(f"Email: {sample_email[:80]}...")
print(f"Text-Only:  {'SPAM' if text_pred else 'LEGIT'} (confidence: {max(text_proba):.3f})")
print(f"Fused:      {'SPAM' if fused_pred else 'LEGIT'} (confidence: {max(fused_proba):.3f})")
print("\nDone! All figures saved to the current directory.")

What does this code do?

The text-only model can classify the raw string directly. The fused model needs both the text and its metadata features, so you build a one-row DataFrame with the same engineered columns your engineer_features function creates.

predict_proba returns a probability array for each class. Taking max(text_proba) gives the confidence the model has in its chosen label. Higher confidence means the model is more certain about its classification.

  • Save spam_detector.py.

Before you run the script, think about this: the sample email screams "FREE gift card" and "Click here NOW." Do you think both models will catch it as spam, or will one be more confident than the other?

  • Run the final script:
python spam_detector.py

At the very bottom of the output, you should see the prediction demo. Both models should label the email as SPAM, each with a confidence score. The fused model can draw on the high capital ratio and punctuation density of the sample email, giving it additional signal beyond the text alone.

  • Confirm all five evaluation figures are saved by running:
ls *.png

You should see five files listed: feature_importance.png, fused_confusion.png, model_comparison.png, roc_comparison.png, and text_only_confusion.png.

Seeing fewer than five PNG files?

  • Make sure your terminal is inside the spam-detector folder. Run pwd to check.
  • If any individual figure is missing, scroll through the terminal output for an error traceback. The error usually appears right before the missing file's save message.
  • Still stuck? Help me figure out why some of my PNG evaluation figures are missing.

✔️ Awesome, I've got everything!

Great. Double check you've saved spam_detector.py and that all five PNG files appear in your spam-detector folder.

ⓧ I'd like to double check the full code

Your spam_detector.py is too long to show in one block, so scroll through and confirm you have all six sections in order:

  • Imports and warning suppression at the top.
  • Step 2: Dataset loading, text-only pipeline, classification report, and text_only_confusion.png.
  • Step 3: engineer_features function, metadata column creation, and summary statistics.
  • Step 4: ColumnTransformer, fused pipeline, comparison report, fused_confusion.png, roc_comparison.png, and model_comparison.png.
  • Step 5 (feature importance): coefficient extraction, top-15 bar chart, and feature_importance.png.
  • Step 5 (summary table and prediction demo): ROC-AUC computation, formatted summary printout, sample email classification through both models.

Secret mission

Benchmark Against a Pre-Trained Transformer

Your fused model outperformed the text-only baseline. But how does a lightweight scikit-learn pipeline compare to a 125-million-parameter deep learning model trained specifically for spam detection? Load a pre-trained RoBERTa classifier from Hugging Face and run a three-way showdown.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources on disk, slim them down to reclaim space, or delete them entirely.

Resources you used:

  • The spam-detector folder on your Desktop, containing Python scripts, a virtual environment, and saved plot images.
  • Hugging Face cache directory with the downloaded dataset and any cached model weights (stored in ~/.cache/huggingface/ by default).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your spam detector or extending it further.

  • Your virtual environment, scripts, cached dataset, and plot images all stay on disk.
  • The Hugging Face cache (dataset and model weights) also remains in your user cache directory.

Pause - I'll come back to this later

Deactivate your virtual environment and optionally reclaim disk space from the cached dataset. Your project files stay intact for when you return.

  • Deactivate your virtual environment by running this command in your VS Code terminal:
deactivate
  • To reclaim disk space from the cached dataset, find its location by running:
python -c "from datasets import config; print(config.HF_DATASETS_CACHE)"
  • Navigate to the printed path in Finder (macOS) or File Explorer (Windows) and delete the folder contents.
  • Your spam-detector folder and scripts remain on disk, ready for you to reactivate the virtual environment later.

Delete - I don't want to use this again

Remove all project files and cached data to start fresh.

  • Deactivate your virtual environment first by running:
deactivate
  • Delete the entire spam-detector folder. On macOS/Linux, run:
cd ~/Desktop
rm -rf spam-detector
  • On Windows, run:
cd ~\Desktop
Remove-Item -Recurse -Force spam-detector
  • Delete the Hugging Face cache directory. First, find the cache location by running:
python -c "from datasets import config; print(config.HF_DATASETS_CACHE)"
  • The printed path typically ends in huggingface/datasets. Navigate up one level to the huggingface parent folder and delete it to remove both the dataset cache and any downloaded model weights.
  • Confirm the spam-detector folder is gone by running ls on your Desktop. It should no longer appear in the listing.

Nice Work!

Nice Work!

You just built a content-metadata fusion spam detector that demonstrably outperforms a text-only approach, backed by academic evaluation metrics you can present to your advisor.

You've learned how to:

  • Build a text-only spam baseline using TF-IDF and Logistic Regression, then identify its weakness with social engineering emails that use normal-sounding language.
  • Engineer metadata features from raw email data (text length, capital letter ratio, punctuation density, URL count, subject length, sender presence) and fuse them with text content using scikit-learn's ColumnTransformer into a single classification pipeline.
  • Produce academic-quality evaluation outputs including confusion matrices, ROC curves, feature importance plots, and model comparison charts with proper metrics (accuracy, precision, recall, F1, ROC-AUC).
  • Secret Mission: Benchmarked your models against a pre-trained RoBERTa transformer for a three-way comparison.

Ready to quiz yourself?