Explainable Email Spam Classifier

Parse raw emails, compare classifiers, and explain spam predictions in Python.

Introduction

30 Second Summary

A filter's headline score can hide the mistake that matters most to you. A legitimate message might disappear into spam.

In this project, you will build a local Python email spam classifier that predicts whether raw email files are ham or spam. One reproducible run produces a model comparison, a confusion matrix, and a Markdown report.

What You'll Build

Your finished classifier prints a side-by-side model comparison in the Visual Studio Code terminal before saving a visual error summary and a submission-ready report.

By the end of this project, you'll have:

  • A raw-email workflow that turns the ham and spam folders into readable labeled messages with visible class counts.
  • A reproducible model comparison that shows accuracy, spam precision, spam recall, and spam F1 for Naive Bayes and a TF-IDF logistic regression pipeline.
  • Submission-ready evidence in outputs/confusion_matrix.png and outputs/report.md that exposes false positives, false negatives, influential terms, and model limitations.
  • Secret Mission: Challenge the trained classifier with your own obvious ham, obvious spam, and deliberately ambiguous messages.

Are there any prerequisites?

You need basic Python syntax on a Mac with Python 3.11 or newer. You will also use Visual Studio Code and a free manual dataset download from Kaggle.

Before We Start

This moment locks in your goal before any hands-on work begins: build a Python email spam classifier. Its accuracy can hide harmful false positives or false negatives, so you need to evaluate its mistakes.

Prepare the Project and Dataset

Package conflicts or missing folders can make a machine-learning classifier failure look like a coding failure.

You'll create a predictable workspace in Visual Studio Code. A virtual environment will keep the project's dependencies separate from the rest of your Mac.

In this step, get ready to:
  • Open the project workspace with Python 3.11 or newer.
  • Install the pinned machine-learning dependencies in an isolated environment.
  • Extract the ham and spam email folders under data/.
Create the project workspace and check Python

The project folder gives every later command one predictable starting point. Checking Python now prevents incompatible packages from blocking the model later.

  • Press Cmd+Space on macOS or the Windows key on Windows to open system search.
  • Type Visual Studio Code into the search field.
  • Press Enter to start Visual Studio Code.
  • Select File in the menu bar.
  • Select Open Folder....
  • Select Desktop in the folder picker.
  • Select New Folder.
  • Enter email-spam-classifier as the folder name.
  • Select Open to use the folder as your workspace.
  • Select Yes, I trust the authors if the Workspace Trust prompt appears.

Good start. The Explorer view now shows email-spam-classifier as the open folder.

  • Select View in the menu bar.
  • Select Terminal.
  • Check the available Python version by running this command:
python3 --version

What Does This Check Prove?

The command prints the Python version available to your terminal. Both pinned dependencies require Python 3.11 or newer.

✔️ I see version 3.11 or higher

Python is ready for the pinned dependencies. You can continue with the open project folder.

ⓧ I see an older version

Your current Python cannot install the pinned dependencies. Install the current stable macOS release before creating the environment.

  • Open the official Python macOS downloads page.
  • Select Download macOS installer for the current stable release.
  • Complete the installer using its default options.
  • Return to Visual Studio Code.
  • Select Terminal in the menu bar.
  • Select New Terminal.
  • Check the updated Python version by running:
python3 --version

What Should Change?

The new terminal loads the updated Python installation. Continue once the reported version is 3.11 or newer.

Still Seeing the Older Version?

Close the existing terminal before creating a new one. This lets the shell detect the updated Python installation.

Ask for help if the older installation still takes priority. Help me make my new Python installation available in the Visual Studio Code terminal on macOS.

ⓧ Command not found

Python is unavailable from the terminal. Install the current stable macOS release before continuing.

  • Open the official Python macOS downloads page.
  • Select Download macOS installer for the current stable release.
  • Complete the installer using its default options.
  • Return to Visual Studio Code.
  • Select Terminal in the menu bar.
  • Select New Terminal.
  • Confirm Python is now available by running:
python3 --version

What Should Change?

The new terminal should print an installed Python version. Continue when that version is 3.11 or newer.

Python Still Unavailable?

Create a fresh terminal after the installer finishes. Existing terminals can keep an older command path.

Ask for help if the command is still unavailable. Help me make Python available in the Visual Studio Code terminal on macOS.

That's the compatibility check complete. Your project now has a supported Python installation behind it.

Build the isolated environment

The dependency file makes the setup reproducible. Its exact version pins ensure that your machine uses the same scikit-learn and Matplotlib releases as the project.

  • Select the Explorer view from earlier.
  • Select New File....
  • Enter requirements.txt as the file name.
  • Press Enter to create the file.
  • Paste these dependency pins into requirements.txt:
scikit-learn==1.9.1
matplotlib==3.11.2

What Do These Pins Control?

  • The first line installs scikit-learn version 1.9.1 for text processing and classification.
  • The second line installs Matplotlib version 3.11.2 for the confusion-matrix image.
  • Exact pins keep the project behavior reproducible across different computers.
  • Save requirements.txt by pressing Cmd+S on macOS or Ctrl+S on Windows.
  • Confirm the Explorer view lists requirements.txt inside email-spam-classifier.

File Missing or Unsaved?

Check that requirements.txt sits directly inside email-spam-classifier. Its name must not contain an extra file extension.

Ask for help if the file does not appear. Help me create requirements.txt in the open Visual Studio Code project folder.

✔️ Awesome, I've got everything!

Great. Confirm that requirements.txt is saved before you continue.

ⓧ I'd like to double check the full code

Your complete requirements.txt file should match this exactly.

scikit-learn==1.9.1
matplotlib==3.11.2

What Should Match?

Check both package names and both version pins. Even one different character can install a different release.

The environment keeps these packages inside the project. This prevents them from changing Python tools used by your other projects.

  • Return to the integrated terminal from earlier.
  • Create .venv by running this command:
python3 -m venv .venv

What Does This Command Do?

Python creates an isolated environment named .venv inside the open project folder. The environment receives its own Python and package installation locations.

  • Confirm the Explorer view now contains .venv.

Environment Folder Missing?

Check that the terminal is inside the open email-spam-classifier workspace. A Python version below 3.11 also needs to be corrected before continuing.

Ask for help if creation still fails. Help me troubleshoot python3 -m venv .venv in my Visual Studio Code terminal on macOS.

  • Activate .venv in the current terminal by running:
source .venv/bin/activate

What Does Activation Change?

Activation makes this terminal use the Python and packages stored inside .venv. Later commands stay isolated to this project.

Your terminal prompt should now include .venv. This confirms the environment is active.

Environment Not Activating?

Check that the terminal is using a macOS shell. Confirm that .venv exists in the Explorer view.

Ask for help if the prompt does not change. Help me activate the .venv environment in the Visual Studio Code terminal on macOS.

  • Install the pinned dependencies into the active environment by running:
python3 -m pip install -r requirements.txt

What Does This Installation Use?

The command reads each package pin from requirements.txt. The active environment receives those exact releases.

The terminal may pause while it downloads the packages. Allow a minute or two for the installation to finish.

Dependency Installation Failed?

Confirm that the terminal prompt includes .venv. Check that requirements.txt contains the two exact pins shown above.

Ask for help with any remaining package error. Help me troubleshoot installing my pinned Python requirements inside .venv.

  • Verify both installed packages by running these commands:
python -m pip show scikit-learn
python -m pip show matplotlib

What Do These Checks Show?

Each command prints details for one package inside the active environment. This confirms which release the project actually uses.

The scikit-learn details should identify version 1.9.1. The Matplotlib details should identify version 3.11.2.

Package Details Missing?

Reactivate .venv if your terminal prompt no longer includes its name. Run the installation command again if either package is absent.

Ask for help if the displayed version is different. Help me verify the packages installed inside my active .venv environment.

Your isolated environment is working. The classifier now has the exact libraries it needs.

Download and verify the email dataset

The Kaggle dataset stores ham and spam as raw email files. Keeping both class folders under data/ lets the future script discover them recursively.

  • Return to the Explorer view from earlier.
  • Select New Folder....
  • Enter data as the folder name.
  • Press Enter to create the folder.

You should now see data directly inside email-spam-classifier.

Give the Archive Time

The dataset contains more than 9,000 files. Downloading and extracting the archive can take a few minutes.

  • Open the Email Spam Dataset (Extended) page in your browser.
  • Select Download on the dataset page.
  • Wait for the browser to finish downloading the archive.

Keep Both Classes Intact

The downloaded content includes main_ham and main_spam directories. Their folder names become the labels used by the classifier.

  • Press Cmd+Space to open Spotlight on your Mac.
  • Type Finder into Spotlight.
  • Press Enter to open Finder.
  • Select Downloads in the Finder sidebar.
  • Double-click the downloaded ZIP archive to extract it.
  • Open the extracted dataset folder.
  • Press Cmd+A to select its contents.
  • Press Cmd+C to copy the selected contents.

The extracted dataset contents are now ready to move into your project.

  • Select Desktop in the Finder sidebar.
  • Open email-spam-classifier.
  • Open data.
  • Press Cmd+V to paste the dataset contents.
  • Double-click each nested ZIP archive inside data until main_ham and main_spam are visible in the extracted folders.
  • Return to Visual Studio Code.

Before you run the final check, what would prove that both email classes are discoverable below data/?

  • Search recursively for both class directories by running:
find data -type d \( -name "main_ham" -o -name "main_spam" \)

What Does This Check Do?

The find command searches every directory below data. It prints only directories whose names match the two required classes.

You'll see one path ending in main_ham plus one path ending in main_spam. This proves the future script can discover both classes recursively.

  • Expand the printed main_ham path in the Explorer view.
  • Confirm that main_ham contains raw email files.
  • Expand the printed main_spam path in the Explorer view.
  • Confirm that main_spam contains raw email files.

One or Both Folders Missing?

Check for ZIP archives that still need extraction inside data. Confirm that the folder names remain exactly main_ham and main_spam.

Ask for help if the recursive search stays empty. Help me arrange the extracted Kaggle email dataset so main_ham and main_spam are discoverable under data.

Your workspace now has supported Python, isolated dependencies, and both raw email classes. Next, you'll turn those files into labeled text that a classifier can learn from.

Parse Raw Emails into Labeled Text

Your dataset is ready, but each message still lives inside a raw email file. The classifier needs dependable labels plus readable text before it can learn anything useful.

In this step, you will use Python's MIME-aware email parser to combine each subject with its preferred plain-text body. You will also use the folder structure to label every parsed message as ham or spam.

In this step, get ready to:
  • Discover the ham and spam directories recursively under data/.
  • Parse each raw email into normalized subject-plus-body text.
  • Load the labeled messages for a readable terminal check.
Discover the class folders

The downloaded archive can place main_ham and main_spam at different depths. A recursive search lets the script find both directories without relying on one fixed extraction layout.

  • In the Explorer sidebar from earlier, click the new-file icon beside the email-spam-classifier folder.
  • Enter spam_classifier.py as the file name.
  • Add the project paths plus recursive folder discovery by pasting this code into spam_classifier.py:
from email import policy
from email.parser import BytesParser
from pathlib import Path

DATA_DIR = Path("data")
OUTPUT_DIR = Path("outputs")
LABELS = ("ham", "spam")
DATASET_URL = "https://www.kaggle.com/datasets/maharshipandya/email-spam-dataset-extended"


def find_class_directory(folder_name, data_dir=DATA_DIR):
    matches = sorted(
        path for path in data_dir.rglob(folder_name) if path.is_dir()
    )
    if not matches:
        raise FileNotFoundError(
            f"Could not find a folder named {folder_name!r} under {data_dir}. "
            "Download and extract the Kaggle dataset into the data folder."
        )
    return matches[0]

What Does This Code Do?

  • The imports provide MIME parsing plus file-system paths. The parser uses an explicit email policy when it reads each message.
  • The path constants give the script consistent locations for its input data and later output files.
  • The find_class_directory() function searches every level below DATA_DIR for the requested directory.
  • The error explains which class directory is missing when the extracted dataset cannot be found.
  • Save spam_classifier.py.
  • Confirm spam_classifier.py now appears beside requirements.txt in the Explorer sidebar.

File Missing from the Explorer Sidebar?

Check that you created spam_classifier.py directly inside email-spam-classifier. A file created under data/ would mix your script with the downloaded messages.

Use Help me find where spam_classifier.py was created in my VS Code project. if the file still does not appear.

Parse each raw email

Raw emails can contain headers plus several body formats. The parser selects the preferred plain-text body before normalizing repeated whitespace into one clean string.

  • In spam_classifier.py, place your cursor two lines below return matches[0].
  • Add the email parser by pasting this code:
def extract_email_text(path):
    try:
        with path.open("rb") as source:
            message = BytesParser(policy=policy.default).parse(source)

        subject = str(message.get("subject", ""))
        body_part = message.get_body(preferencelist=("plain",))
        body = body_part.get_content() if body_part is not None else ""
        if not isinstance(body, str):
            body = ""
        if not body.strip():
            body = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        subject = ""
        body = path.read_text(encoding="utf-8", errors="ignore")

    return " ".join(f"{subject}\n{body}".split())

How Does the Parser Handle Real Emails?

  • The file opens in binary mode so BytesParser(policy=policy.default) can interpret its email structure.
  • The parser reads the subject separately because it belongs to the message headers.
  • The get_body(preferencelist=("plain",)) call selects a plain-text body when the message contains several formats.
  • The fallback reads undecodable or unusual files as text so one malformed message does not stop the entire dataset load.
  • The final expression combines the subject with the body. It also collapses repeated whitespace into a consistent format.
  • Save spam_classifier.py.

Before you run this check, do you expect the parser definitions to load without a traceback?

  • Check the parser definitions in the activated terminal by running:
python3 spam_classifier.py

What Should You See?

The terminal returns to its prompt without a traceback. That confirms Python can load the imports plus both function definitions.

Seeing a Syntax Error?

Check the indentation inside extract_email_text(). Every line under try: and except Exception: must remain indented.

Compare the quotes and parentheses around preferencelist=("plain",). The comma keeps the value as a one-item tuple.

Use Help me fix the syntax error in my extract_email_text function. if the traceback points somewhere unexpected.

Load the labeled dataset

The directory name supplies each message's label. The loader pairs every readable email with either ham or spam before returning parallel text and label lists.

  • In spam_classifier.py, place your cursor two lines below the final line of extract_email_text().
  • Add the dataset loader by pasting this code:
def load_dataset(data_dir=DATA_DIR):
    texts = []
    labels = []
    folder_map = {"ham": "main_ham", "spam": "main_spam"}

    for label, folder_name in folder_map.items():
        class_directory = find_class_directory(folder_name, data_dir)
        email_files = sorted(
            path
            for path in class_directory.iterdir()
            if path.is_file() and not path.name.startswith(".")
        )

        for email_path in email_files:
            text = extract_email_text(email_path)
            if text:
                texts.append(text)
                labels.append(label)

    if not texts:
        raise ValueError("No readable email files were found in the dataset folders.")

    return texts, labels

How Are the Labels Created?

  • The folder_map dictionary connects each class label to its dataset directory.
  • The loader ignores hidden entries plus directories inside each class folder.
  • Each non-empty parsed message enters texts. Its matching class enters the same position in labels.
  • The final guard stops the experiment with a clear message when no readable emails were collected.
  • Save spam_classifier.py.
  • Press Cmd+F on macOS or Ctrl+F on Windows to open the editor search.
  • Enter return texts, labels into the search field.

You should see one match at the end of load_dataset(). This confirms the loader returns both synchronized lists.

Loader Code in the Wrong Place?

Make sure load_dataset() starts at the far-left indentation level. Placing it inside extract_email_text() prevents the rest of the script from calling it.

Use Help me check whether load_dataset is nested inside another function. if the function does not appear at the top level.

The final flow calls the loader once. It turns the resulting labels into class counts plus a short preview that proves the email text is readable.

  • Place your cursor two lines below return texts, labels.
  • Add the runnable flow by pasting this code:
def main():
    texts, labels = load_dataset()
    counts = {label: labels.count(label) for label in LABELS}

    print(
        f"Loaded {len(texts)} emails: "
        f"{counts['ham']} ham and {counts['spam']} spam."
    )
    print(f"Sample preview: {texts[0][:300]}\n")


if __name__ == "__main__":
    main()

What Does the Main Flow Prove?

  • The call to load_dataset() discovers both class directories. It also parses every readable file.
  • The counts dictionary totals the labels collected for each class.
  • The first print statement reveals the complete loaded dataset balance.
  • The second print statement limits the first parsed message to 300 characters for a compact readability check.
  • Save spam_classifier.py.

Parsing the full collection can keep the terminal busy for a short while because the script opens every raw email file.

Before you run it, which class do you expect to have more loaded messages?

  • Run the complete parser in the activated terminal with:
python3 spam_classifier.py

What Should You See?

You should see one line with the total number of loaded emails plus separate ham and spam counts. Both class counts should be greater than zero.

You should also see a 300-character sample preview containing readable email text. That output proves the raw files now form a labeled text dataset.

Missing Counts or a Readable Preview?

If the script reports a missing folder, confirm the extracted dataset still contains main_ham and main_spam somewhere below data/.

If the script reports No readable email files were found in the dataset folders. check that the raw email files sit directly inside the discovered class directories.

Use Help me debug why spam_classifier.py cannot load my extracted email dataset. for help tracing the failed stage.

✔️ Awesome, I've got everything!

Great work. Your script now turns the raw ham and spam files into labeled text that a classifier can use.

ⓧ I'd like to double check the full code

Compare your saved spam_classifier.py with this complete version:

from email import policy
from email.parser import BytesParser
from pathlib import Path

DATA_DIR = Path("data")
OUTPUT_DIR = Path("outputs")
LABELS = ("ham", "spam")
DATASET_URL = "https://www.kaggle.com/datasets/maharshipandya/email-spam-dataset-extended"


def find_class_directory(folder_name, data_dir=DATA_DIR):
    matches = sorted(
        path for path in data_dir.rglob(folder_name) if path.is_dir()
    )
    if not matches:
        raise FileNotFoundError(
            f"Could not find a folder named {folder_name!r} under {data_dir}. "
            "Download and extract the Kaggle dataset into the data folder."
        )
    return matches[0]


def extract_email_text(path):
    try:
        with path.open("rb") as source:
            message = BytesParser(policy=policy.default).parse(source)

        subject = str(message.get("subject", ""))
        body_part = message.get_body(preferencelist=("plain",))
        body = body_part.get_content() if body_part is not None else ""
        if not isinstance(body, str):
            body = ""
        if not body.strip():
            body = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        subject = ""
        body = path.read_text(encoding="utf-8", errors="ignore")

    return " ".join(f"{subject}\n{body}".split())


def load_dataset(data_dir=DATA_DIR):
    texts = []
    labels = []
    folder_map = {"ham": "main_ham", "spam": "main_spam"}

    for label, folder_name in folder_map.items():
        class_directory = find_class_directory(folder_name, data_dir)
        email_files = sorted(
            path
            for path in class_directory.iterdir()
            if path.is_file() and not path.name.startswith(".")
        )

        for email_path in email_files:
            text = extract_email_text(email_path)
            if text:
                texts.append(text)
                labels.append(label)

    if not texts:
        raise ValueError("No readable email files were found in the dataset folders.")

    return texts, labels


def main():
    texts, labels = load_dataset()
    counts = {label: labels.count(label) for label in LABELS}

    print(
        f"Loaded {len(texts)} emails: "
        f"{counts['ham']} ham and {counts['spam']} spam."
    )
    print(f"Sample preview: {texts[0][:300]}\n")


if __name__ == "__main__":
    main()

How Is the File Organized?

The file moves from shared paths to folder discovery. It then parses individual messages before loading the complete labeled dataset.

The main() function sits last because it uses every helper defined above it.

Your raw emails now have dependable labels plus normalized text. Next, you will use that dataset to train a fast count-based baseline.

Train the Naive Bayes Baseline

Your parser already turns raw MIME email into labeled text. That gives your classifier dependable input from both folders.

A count-based Naive Bayes model gives you a fast benchmark. This step tests whether its single accuracy score gives enough evidence about the model's mistakes.

In this step, get ready to:
  • Import the scikit-learn components for the baseline.
  • Build the count-based model pipeline.
  • Train the pipeline on a reproducible train/test split.
Import the baseline tools

The baseline needs tools for converting email text into counts. It also needs tools for splitting data, training the classifier, and measuring the result.

  • Switch back to spam_classifier.py in Visual Studio Code.
  • Find from pathlib import Path near the top of the file.
  • Paste these imports directly below it:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

What Do These Imports Provide?

  • CountVectorizer converts email text into token counts that a classifier can process.
  • train_test_split reserves part of the dataset for testing.
  • MultinomialNB supplies the classifier used by the baseline.
  • Pipeline keeps text conversion connected to model training.
  • accuracy_score calculates the fraction of test predictions that are correct.
  • Save spam_classifier.py.
  • Confirm the imports load by running this command in the activated terminal:
python3 spam_classifier.py

What Should You See?

You should see the loaded email count first. The parsed sample preview should follow.

This output confirms that the new imports load without interrupting your existing parser.

Imports Failing?

  • Confirm that the .venv environment is still activated in the terminal.
  • Confirm that the dependencies from requirements.txt were installed in that environment.
  • Help me diagnose the failed scikit-learn imports in my activated Python environment.
Build the count-based pipeline

A bag-of-words representation records how often tokens appear in each email. The classifier can use those counts to learn which patterns occur more often in ham or spam.

  • Find def main(): in spam_classifier.py.
  • Paste this function directly above it:
def build_baseline_model():
    return Pipeline([
        ("counts", CountVectorizer(min_df=2)),
        ("model", MultinomialNB()),
    ])

What Does This Model Do?

  • Pipeline connects text conversion to classification in one model object.
  • CountVectorizer(min_df=2) creates token-count features for terms found in at least two documents.
  • MultinomialNB() learns how those token counts differ between ham and spam.
  • Save spam_classifier.py.
  • Confirm the model definition keeps the script executable by running:
python3 spam_classifier.py

What Should You See?

You should still see the loaded ham and spam counts. You should also see the parsed-email preview.

That successful run confirms that Python can create the new function without a syntax failure.

Script Stopped Running?

  • Check that build_baseline_model appears above main with no extra indentation.
  • Check that each opening bracket in the pipeline has a matching closing bracket.
  • Help me fix the build_baseline_model function in spam_classifier.py.
Train the baseline on a held-out split

The test messages need to stay separate while the pipeline learns its vocabulary. Splitting first prevents data leakage from making the result look stronger than it is.

  • Find print(f"Sample preview: {texts[0][:300]}\n") inside main().
  • Paste these lines directly below it:
    training_texts, test_texts, training_labels, test_labels = train_test_split(texts, labels, test_size=0.20, random_state=42, stratify=labels)
    baseline_model = build_baseline_model()
    baseline_model.fit(training_texts, training_labels)
    baseline_predictions = baseline_model.predict(test_texts)
    print(f"Baseline accuracy: {accuracy_score(test_labels, baseline_predictions):.3f}")

How Does the Training Flow Work?

  • test_size=0.20 reserves twenty percent of the messages for evaluation.
  • random_state=42 makes the split reproducible across runs.
  • stratify=labels preserves the class balance in both portions.
  • baseline_model.fit fits the complete pipeline using only the training portion.
  • baseline_model.predict classifies the held-out messages.
  • accuracy_score measures the overall fraction of correct predictions.
  • Save spam_classifier.py.

Before you run it, predict whether one accuracy score can reveal the types of mistakes the model makes.

  • Train the baseline on the held-out split by running:
python3 spam_classifier.py

What Should You See?

The parser output should still appear first.

A final line should start with Baseline accuracy: followed by a decimal score.

You have your first working classifier. Its accuracy gives you a fast benchmark for the next model.

The score is one overall fraction. Ham predicted as spam is a false positive.

Spam predicted as ham is a false negative. The baseline accuracy does not show either error count.

Baseline Run Failing?

  • Check that the split appears after texts and labels are loaded inside main().
  • Check that baseline_model.fit uses the training variables.
  • Help me troubleshoot the baseline training run in spam_classifier.py.

✔️ Awesome, I've got everything!

Great. Your baseline file now matches the checkpoint.

ⓧ I'd like to double check the full code

  • Compare your saved spam_classifier.py with this complete checkpoint:
from email import policy
from email.parser import BytesParser
from pathlib import Path

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

DATA_DIR = Path("data")
OUTPUT_DIR = Path("outputs")
LABELS = ("ham", "spam")
DATASET_URL = "https://www.kaggle.com/datasets/maharshipandya/email-spam-dataset-extended"


def find_class_directory(folder_name, data_dir=DATA_DIR):
    matches = sorted(path for path in data_dir.rglob(folder_name) if path.is_dir())
    if not matches:
        raise FileNotFoundError(f"Could not find a folder named {folder_name!r} under {data_dir}. Download and extract the Kaggle dataset into the data folder.")
    return matches[0]


def extract_email_text(path):
    try:
        with path.open("rb") as source:
            message = BytesParser(policy=policy.default).parse(source)
        subject = str(message.get("subject", ""))
        body_part = message.get_body(preferencelist=("plain",))
        body = body_part.get_content() if body_part is not None else ""
        if not isinstance(body, str):
            body = ""
        if not body.strip():
            body = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        subject = ""
        body = path.read_text(encoding="utf-8", errors="ignore")
    return " ".join(f"{subject}\n{body}".split())


def load_dataset(data_dir=DATA_DIR):
    texts = []
    labels = []
    folder_map = {"ham": "main_ham", "spam": "main_spam"}
    for label, folder_name in folder_map.items():
        class_directory = find_class_directory(folder_name, data_dir)
        email_files = sorted(path for path in class_directory.iterdir() if path.is_file() and not path.name.startswith("."))
        for email_path in email_files:
            text = extract_email_text(email_path)
            if text:
                texts.append(text)
                labels.append(label)
    if not texts:
        raise ValueError("No readable email files were found in the dataset folders.")
    return texts, labels


def build_baseline_model():
    return Pipeline([
        ("counts", CountVectorizer(min_df=2)),
        ("model", MultinomialNB()),
    ])


def main():
    texts, labels = load_dataset()
    counts = {label: labels.count(label) for label in LABELS}
    print(f"Loaded {len(texts)} emails: {counts['ham']} ham and {counts['spam']} spam.")
    print(f"Sample preview: {texts[0][:300]}\n")
    training_texts, test_texts, training_labels, test_labels = train_test_split(texts, labels, test_size=0.20, random_state=42, stratify=labels)
    baseline_model = build_baseline_model()
    baseline_model.fit(training_texts, training_labels)
    baseline_predictions = baseline_model.predict(test_texts)
    print(f"Baseline accuracy: {accuracy_score(test_labels, baseline_predictions):.3f}")


if __name__ == "__main__":
    main()

How Is the File Organized?

  • The import section supplies the parser and baseline modeling tools.
  • The dataset helpers discover each class folder and normalize every readable message.
  • The baseline builder connects token counts to Multinomial Naive Bayes.
  • The main flow creates the held-out split before fitting the pipeline.

Your baseline gives you a number to beat. Next, you will measure spam-specific errors and compare them with a final model.

Evaluate the TF-IDF Final Model

Your count-based baseline now classifies held-out emails. Its accuracy score still hides the types of mistakes it makes.

A TF-IDF pipeline with logistic regression provides a stronger final model. Spam precision, spam recall, and spam F1 expose the errors that accuracy compresses.

In this step, get ready to:
  • Build the final TF-IDF and logistic regression pipeline.
  • Evaluate both models with the same reusable function.
  • Compare accuracy with spam precision, spam recall, and spam F1.
Build the final pipeline

TF-IDF reduces the influence of terms that appear throughout the email collection. Word bigrams let the model learn adjacent pairs of words.

  • In spam_classifier.py, find the scikit-learn imports near the top of the file.
  • Select the current feature-extraction import line.
  • Extend the selection through the current metrics import line.
  • Replace the selection with this import group:
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report

What Do These Imports Provide?

  • TfidfVectorizer converts email text into weighted numerical features.
  • LogisticRegression learns how those features separate ham from spam.
  • classification_report calculates class-specific precision, recall, and F1 values.
  • Save spam_classifier.py.
  • Confirm that the updated imports load by running this command:
python3 spam_classifier.py

What Does This Check Prove?

Python loads every import before running the baseline. A completed run confirms that the new classifier and metric classes are available in your environment.

You should see the dataset summary plus the familiar baseline accuracy. The completed run confirms that the import group works.

Imports Not Loading?

  • Confirm that the terminal prompt still shows the activated .venv environment.
  • Compare the three import lines with the code block above.
  • Select Help me diagnose the import failure..

The final pipeline combines weighted word features with a class-balanced classifier. Its frequency filters remove extremely rare terms plus terms found in almost every document.

  • In spam_classifier.py, find build_baseline_model().
  • Add this final model function below build_baseline_model():
def build_final_model():
    return Pipeline(
        [
            (
                "tfidf",
                TfidfVectorizer(
                    ngram_range=(1, 2),
                    min_df=2,
                    max_df=0.95,
                    sublinear_tf=True,
                ),
            ),
            (
                "model",
                LogisticRegression(class_weight="balanced", max_iter=1000),
            ),
        ]
    )

How Does the Final Model Work?

  • ngram_range includes individual words plus adjacent two-word phrases.
  • min_df excludes terms found in fewer than two documents.
  • max_df excludes terms found in more than 95 percent of documents.
  • sublinear_tf reduces the impact of repeated occurrences within one email.
  • class_weight adjusts the fit according to class frequency.
  • Save spam_classifier.py.
  • Confirm that Python loads the final model definition by running this command:
python3 spam_classifier.py

What Does This Run Confirm?

Python reads build_final_model() before calling main(). A completed baseline run confirms that the new function has valid syntax.

You should see the baseline output finish without a traceback. Your script now contains both model definitions.

Final Pipeline Not Loading?

  • Check that every closing parenthesis in build_final_model() matches the indentation shown above.
  • Confirm that TfidfVectorizer appears in the feature-extraction import.
  • Confirm that LogisticRegression has its own import line.
  • Select Help me debug build_final_model()..
Add shared evaluation helpers

A fair comparison fits each model on the same training messages. It also scores each model against the same held-out labels.

  • In spam_classifier.py, find build_final_model().
  • Add this evaluator below build_final_model():
def evaluate_model(model, training_texts, training_labels, test_texts, test_labels):
    model.fit(training_texts, training_labels)
    predictions = model.predict(test_texts)
    report = classification_report(
        test_labels,
        predictions,
        labels=list(LABELS),
        output_dict=True,
        zero_division=0,
    )
    metrics = {
        "accuracy": accuracy_score(test_labels, predictions),
        "spam_precision": report["spam"]["precision"],
        "spam_recall": report["spam"]["recall"],
        "spam_f1": report["spam"]["f1-score"],
    }
    return predictions, metrics

What Does the Evaluator Return?

  • model.fit() trains the supplied pipeline on the training portion.
  • model.predict() assigns labels to the held-out email text.
  • classification_report() calculates the spam row used by the comparison.
  • metrics stores accuracy plus spam precision, recall, and F1 under consistent keys.
  • predictions remains available for the confusion matrix in the next step.
  • Save spam_classifier.py.
  • Confirm that Python accepts the evaluator by running this command:
python3 spam_classifier.py

What Does This Run Establish?

Python loads evaluate_model() before starting the current baseline flow. A completed run confirms that the evaluator is defined correctly.

You should see the dataset summary plus baseline accuracy again. The reusable evaluator is ready to score both pipelines.

Evaluator Definition Failing?

  • Confirm that each metric key matches the spelling shown above.
  • Check that every nested report lookup has matching quotation marks and brackets.
  • Select Help me debug evaluate_model()..

A compact comparison table puts both models on the same scale. Each row shows accuracy beside the spam-specific evidence that the baseline lacked.

  • In spam_classifier.py, find evaluate_model().
  • Add this display function below evaluate_model():
def print_comparison(baseline_metrics, final_metrics):
    print("\nModel comparison")
    print(f"{'Model':<20} {'Accuracy':>10} {'Precision':>10} {'Recall':>10} {'F1':>10}")
    for name, metrics in (
        ("Count + Naive Bayes", baseline_metrics),
        ("TF-IDF + Logistic", final_metrics),
    ):
        print(
            f"{name:<20} "
            f"{metrics['accuracy']:>10.3f} "
            f"{metrics['spam_precision']:>10.3f} "
            f"{metrics['spam_recall']:>10.3f} "
            f"{metrics['spam_f1']:>10.3f}"
        )

How Is the Table Formatted?

  • The header gives every metric a fixed-width column.
  • The loop applies the same formatting to both metric dictionaries.
  • Each metric uses three decimal places so small differences remain visible.
  • Save spam_classifier.py.
  • Confirm that Python loads the comparison function by running this command:
python3 spam_classifier.py

What Does This Check Establish?

The command loads print_comparison() before running the existing baseline flow. A completed run confirms that the formatted strings contain valid Python.

You should see the baseline output complete without a formatting error. The next edit connects both pipelines to the table.

Comparison Function Failing?

  • Check that each formatted metric name matches a key created inside evaluate_model().
  • Confirm that the braces inside each formatted string match the code block.
  • Select Help me debug print_comparison()..
Compare both models

The current main() function trains the baseline directly. The replacement below sends the same training split through both pipelines.

  • In main(), find the line that creates baseline_model.
  • Select that line through the current baseline accuracy print line.
  • Replace the selected block with this shared evaluation flow:
    baseline_model = build_baseline_model()
    final_model = build_final_model()

    _, baseline_metrics = evaluate_model(
        baseline_model,
        training_texts,
        training_labels,
        test_texts,
        test_labels,
    )
    final_predictions, final_metrics = evaluate_model(
        final_model,
        training_texts,
        training_labels,
        test_texts,
        test_labels,
    )

    print_comparison(baseline_metrics, final_metrics)

How Does the Comparison Stay Fair?

  • Both model builders create fresh pipelines.
  • Both evaluator calls receive the same training messages.
  • Both evaluator calls receive the same held-out messages.
  • baseline_metrics and final_metrics use matching keys for the table.
  • final_predictions preserves the final model output for later error analysis.
  • Save spam_classifier.py.
  • Predict which model will have the stronger spam F1 score before you run the script.
  • Run both models on the shared split by using this command:
python3 spam_classifier.py

What Does the Final Run Test?

The script parses the emails before fitting both pipelines on the same training portion. It evaluates both sets of predictions against the same held-out labels.

You should see a two-row table for Count + Naive Bayes and TF-IDF + Logistic. Each row should show accuracy, precision, recall, and F1 values.

How Should You Read the Results?

Spam precision describes how often an email predicted as spam is actually spam. Lower precision means more legitimate email is blocked.

Spam recall describes how much real spam the model catches. Lower recall means more spam reaches the inbox.

Spam F1 combines precision with recall into one comparison value. The stronger model depends on which mistake carries more weight for your goal.

Comparison Table Missing?

  • Confirm that both evaluate_model() calls remain indented inside main().
  • Check that print_comparison(baseline_metrics, final_metrics) appears after both evaluator calls.
  • Select Help me restore the comparison table..

That closes the accuracy-only blind spot. Your terminal now shows how each model handles spam-specific mistakes.

✔️ Awesome, I've got everything!

Your final pipeline now runs beside the baseline. Both models report the same four evaluation measures.

ⓧ I'd like to double check the full code

  • Compare your cumulative spam_classifier.py file with this reference:
from email import policy
from email.parser import BytesParser
from pathlib import Path

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

DATA_DIR = Path("data")
OUTPUT_DIR = Path("outputs")
LABELS = ("ham", "spam")
DATASET_URL = "https://www.kaggle.com/datasets/maharshipandya/email-spam-dataset-extended"


def find_class_directory(folder_name, data_dir=DATA_DIR):
    matches = sorted(
        path for path in data_dir.rglob(folder_name) if path.is_dir()
    )
    if not matches:
        raise FileNotFoundError(
            f"Could not find a folder named {folder_name!r} under {data_dir}. "
            "Download and extract the Kaggle dataset into the data folder."
        )
    return matches[0]


def extract_email_text(path):
    try:
        with path.open("rb") as source:
            message = BytesParser(policy=policy.default).parse(source)

        subject = str(message.get("subject", ""))
        body_part = message.get_body(preferencelist=("plain",))
        body = body_part.get_content() if body_part is not None else ""
        if not isinstance(body, str):
            body = ""
        if not body.strip():
            body = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        subject = ""
        body = path.read_text(encoding="utf-8", errors="ignore")

    return " ".join(f"{subject}\n{body}".split())


def load_dataset(data_dir=DATA_DIR):
    texts = []
    labels = []
    folder_map = {"ham": "main_ham", "spam": "main_spam"}

    for label, folder_name in folder_map.items():
        class_directory = find_class_directory(folder_name, data_dir)
        email_files = sorted(
            path
            for path in class_directory.iterdir()
            if path.is_file() and not path.name.startswith(".")
        )

        for email_path in email_files:
            text = extract_email_text(email_path)
            if text:
                texts.append(text)
                labels.append(label)

    if not texts:
        raise ValueError("No readable email files were found in the dataset folders.")

    return texts, labels


def build_baseline_model():
    return Pipeline(
        [
            ("counts", CountVectorizer(min_df=2)),
            ("model", MultinomialNB()),
        ]
    )


def build_final_model():
    return Pipeline(
        [
            (
                "tfidf",
                TfidfVectorizer(
                    ngram_range=(1, 2),
                    min_df=2,
                    max_df=0.95,
                    sublinear_tf=True,
                ),
            ),
            (
                "model",
                LogisticRegression(class_weight="balanced", max_iter=1000),
            ),
        ]
    )


def evaluate_model(model, training_texts, training_labels, test_texts, test_labels):
    model.fit(training_texts, training_labels)
    predictions = model.predict(test_texts)
    report = classification_report(
        test_labels,
        predictions,
        labels=list(LABELS),
        output_dict=True,
        zero_division=0,
    )
    metrics = {
        "accuracy": accuracy_score(test_labels, predictions),
        "spam_precision": report["spam"]["precision"],
        "spam_recall": report["spam"]["recall"],
        "spam_f1": report["spam"]["f1-score"],
    }
    return predictions, metrics


def print_comparison(baseline_metrics, final_metrics):
    print("\nModel comparison")
    print(f"{'Model':<20} {'Accuracy':>10} {'Precision':>10} {'Recall':>10} {'F1':>10}")
    for name, metrics in (
        ("Count + Naive Bayes", baseline_metrics),
        ("TF-IDF + Logistic", final_metrics),
    ):
        print(
            f"{name:<20} "
            f"{metrics['accuracy']:>10.3f} "
            f"{metrics['spam_precision']:>10.3f} "
            f"{metrics['spam_recall']:>10.3f} "
            f"{metrics['spam_f1']:>10.3f}"
        )


def main():
    texts, labels = load_dataset()
    counts = {label: labels.count(label) for label in LABELS}

    print(
        f"Loaded {len(texts)} emails: "
        f"{counts['ham']} ham and {counts['spam']} spam."
    )
    print(f"Sample preview: {texts[0][:300]}\n")

    training_texts, test_texts, training_labels, test_labels = train_test_split(
        texts,
        labels,
        test_size=0.20,
        random_state=42,
        stratify=labels,
    )

    baseline_model = build_baseline_model()
    final_model = build_final_model()

    _, baseline_metrics = evaluate_model(
        baseline_model,
        training_texts,
        training_labels,
        test_texts,
        test_labels,
    )
    final_predictions, final_metrics = evaluate_model(
        final_model,
        training_texts,
        training_labels,
        test_texts,
        test_labels,
    )

    print_comparison(baseline_metrics, final_metrics)


if __name__ == "__main__":
    main()

Your classifier now reports the evidence that accuracy left hidden. Next, you will turn those results into a confusion matrix and a submission-ready report.

Generate Submission Evidence

Your classifier now compares two models with accuracy, spam precision, spam recall, and spam F1. Those terminal results still need evidence that you can inspect and submit.

In this step, you will turn the final predictions into a confusion matrix and a Markdown report. You will also inspect the model's feature weights to reveal the terms associated with ham and spam.

In this step, get ready to:
  • Identify the ten strongest ham-associated terms and ten strongest spam-associated terms.
  • Save a visual confusion matrix for the final model's predictions.
  • Generate a report that explains the data, results, errors, influential terms, limitations, and conclusion.
Reveal the model's strongest terms

Logistic regression assigns a coefficient to every TF-IDF feature. Ranking these coefficients reveals which words or phrases pull predictions toward ham or spam.

  • In spam_classifier.py, find the end of print_comparison.
  • Add the following function directly below print_comparison:
def get_top_terms(final_model, count=10):
    vectorizer = final_model.named_steps["tfidf"]
    classifier = final_model.named_steps["model"]
    feature_names = vectorizer.get_feature_names_out()
    coefficients = classifier.coef_[0]

    if classifier.classes_[1] != "spam":
        coefficients = -coefficients

    ranked_terms = sorted(
        zip(feature_names, coefficients), key=lambda item: item[1]
    )
    ham_terms = [str(term) for term, _ in ranked_terms[:count]]
    spam_terms = [str(term) for term, _ in ranked_terms[-count:][::-1]]
    return ham_terms, spam_terms

How are the terms ranked?

  • The pipeline's named_steps dictionary provides the fitted TF-IDF vectorizer and logistic-regression classifier.
  • The get_feature_names_out() method returns the word or phrase represented by each coefficient.
  • The classes_ check makes positive coefficients consistently point toward spam.
  • The lowest ten coefficients become the ham terms. The highest ten coefficients become the spam terms.
Save the confusion matrix

A confusion matrix separates correct predictions from false positives and false negatives. This makes the final model's mistakes visible instead of compressing them into one score.

  • At the top of spam_classifier.py, replace the import and constant section with this complete version:
from email import policy
from email.parser import BytesParser
from pathlib import Path

import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

DATA_DIR = Path("data")
OUTPUT_DIR = Path("outputs")
LABELS = ("ham", "spam")
DATASET_URL = "https://www.kaggle.com/datasets/maharshipandya/email-spam-dataset-extended"

What changed at the top?

  • Matplotlib provides the plotting functions used to lay out, save, and close the matrix figure.
  • ConfusionMatrixDisplay converts the true and predicted labels into a labeled matrix.
  • OUTPUT_DIR keeps both generated artifacts inside outputs.
  • DATASET_URL gives the generated report a reproducible source reference.
  • Add the following function directly below get_top_terms:
def save_confusion_matrix(test_labels, predictions):
    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
    ConfusionMatrixDisplay.from_predictions(
        test_labels,
        predictions,
        labels=list(LABELS),
        display_labels=["Ham", "Spam"],
        colorbar=False,
    )
    plt.tight_layout()
    plt.savefig(OUTPUT_DIR / "confusion_matrix.png", dpi=150)
    plt.close()

What does this function save?

  • The output directory is created automatically if it does not exist.
  • The matrix compares the held-out labels with the final model's predictions.
  • The axes use Ham and Spam so each error type is readable.
  • The figure is saved at outputs/confusion_matrix.png before its plotting resources are closed.
  • In main, find the call to print_comparison.
  • Replace that call with this evidence-building sequence:
    print_comparison(baseline_metrics, final_metrics)
    ham_terms, spam_terms = get_top_terms(final_model)
    save_confusion_matrix(test_labels, final_predictions)

How does this connect the evidence?

The fitted final model supplies the feature weights. Its held-out predictions supply the confusion matrix.

Before you run this version, where do you expect the new image to appear?

  • Save spam_classifier.py.
  • Generate the first visual artifact by running:
python3 spam_classifier.py

What should you see?

The terminal prints the dataset preview and both metric rows as before. The file tree now contains outputs/confusion_matrix.png.

  • Select outputs/confusion_matrix.png in the VS Code file tree.

You will see a two-by-two matrix labeled Ham and Spam. The off-diagonal cells show false positives and false negatives.

Matrix file missing?

  • Check that save_confusion_matrix(test_labels, final_predictions) sits inside main after the final model has been evaluated.
  • Check that OUTPUT_DIR uses Path("outputs").
  • Confirm the terminal still shows the activated .venv.
  • Help me find why my confusion matrix file was not created.
Generate and verify the report

The report combines your method, model comparison, error interpretation, influential terms, and limitations. The function is long because it creates a complete submission document in one run.

  • In spam_classifier.py, place the next three segments directly below save_confusion_matrix.
  • Build one continuous write_report function by keeping the segments in the order shown.
def write_report(labels, training_size, test_size, baseline_metrics, final_metrics, ham_terms, spam_terms):
    counts = {label: labels.count(label) for label in LABELS}
    report = f"""# Email Spam Classifier Report

## Objective

Build and evaluate a machine-learning classifier that predicts whether a raw email is ham or spam.

## Dataset

- Source: {DATASET_URL}
- Parsed records used: {len(labels)}
- Ham records: {counts['ham']}
- Spam records: {counts['spam']}
- Training records: {training_size}
- Test records: {test_size}

What does the first segment capture?

This segment starts the report with the objective and dataset evidence. The counts come from the same labels used for training and evaluation.

  • Continue the open report string immediately below the test-record count with this segment:


## Preprocessing and method

The script parses each raw file with Python's MIME-aware email parser, combines the subject with the preferred plain-text body, and normalizes whitespace. It creates a stratified 80/20 split before either vectorizer is fitted. The baseline uses token counts with Multinomial Naive Bayes. The final pipeline uses TF-IDF unigrams and bigrams with class-balanced logistic regression.

## Results

| Model | Accuracy | Spam precision | Spam recall | Spam F1 |
| --- | ---: | ---: | ---: | ---: |
| Count + Naive Bayes | {baseline_metrics['accuracy']:.3f} | {baseline_metrics['spam_precision']:.3f} | {baseline_metrics['spam_recall']:.3f} | {baseline_metrics['spam_f1']:.3f} |
| TF-IDF + Logistic Regression | {final_metrics['accuracy']:.3f} | {final_metrics['spam_precision']:.3f} | {final_metrics['spam_recall']:.3f} | {final_metrics['spam_f1']:.3f} |

What does the second segment capture?

This segment records the preprocessing method and reproducible split. It also formats both models' metrics as a Markdown table.

  • Finish the report string and file-writing logic by adding this final segment directly below the metric table:


## Error analysis

The final model's confusion matrix is saved as `confusion_matrix.png`. A ham message predicted as spam is a false positive. A spam message predicted as ham is a false negative. These two error counts explain behavior that accuracy alone hides.

## Influential features

Terms most associated with ham: {', '.join(ham_terms)}

Terms most associated with spam: {', '.join(spam_terms)}

These weights show associations learned from this corpus. They do not prove that an individual word causes an email to be spam or ham.

## Limitations

- The dataset was assembled from SpamAssassin's Old Public Corpus, so it may not represent current spam campaigns.
- Text-only features do not inspect sender reputation, links, attachments, or mail-server behavior.
- Results describe this held-out split and should not be treated as production guarantees.

## Conclusion

The experiment produced a reproducible baseline and a leak-safe final pipeline. The final model is easier to inspect because its feature weights reveal influential terms, while precision, recall, F1, and the confusion matrix make its errors visible.
"""
    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
    (OUTPUT_DIR / "report.md").write_text(report, encoding="utf-8")

What does the final segment explain?

  • The error-analysis section defines false positives and false negatives using the matrix.
  • The influential-feature section inserts the ranked terms from the fitted final model.
  • The limitations prevent the held-out result from being presented as a production guarantee.
  • The final lines create outputs/report.md with UTF-8 encoding.
  • In main, add the report call and completion messages directly below save_confusion_matrix(test_labels, final_predictions):
    write_report(labels, len(training_texts), len(test_texts), baseline_metrics, final_metrics, ham_terms, spam_terms)
    print("\nSaved outputs/confusion_matrix.png")
    print("Saved outputs/report.md")

What completes the workflow?

The report receives the full dataset labels, split sizes, both metric dictionaries, and both term lists. The final messages confirm the paths created by the run.

Before you run the complete script, which result should appear in the terminal and which two files should appear under outputs?

  • Save spam_classifier.py.
  • Generate the final submission evidence by running:
python3 spam_classifier.py

What should the final run show?

The terminal shows the dataset counts, sample preview, and two-row model comparison. It ends with Saved outputs/confusion_matrix.png and Saved outputs/report.md.

The outputs folder contains the confusion-matrix image and generated report.

  • Select outputs/report.md in the VS Code file tree.
  • Confirm the report contains the objective, dataset counts, preprocessing method, model comparison, error analysis, influential features, limitations, and conclusion.
  • Return to outputs/confusion_matrix.png.
  • Identify the false-positive cell and false-negative cell from their row and column labels.

That is the evidence layer working. Your classifier now produces results that another person can inspect without rerunning your analysis by hand.

Report missing or incomplete?

  • Check that the closing triple quotes appear immediately before the line that creates OUTPUT_DIR.
  • Check that write_report receives both metric dictionaries and both term lists.
  • Compare the generated report headings with the complete file below.
  • Help me debug my generated spam-classifier report.

✔️ Awesome, I've got everything!

Your completed script now generates both submission artifacts. Make sure spam_classifier.py is saved.

ⓧ I'd like to double check the full code

Compare your complete spam_classifier.py with this reference file.

from email import policy
from email.parser import BytesParser
from pathlib import Path

import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

DATA_DIR = Path("data")
OUTPUT_DIR = Path("outputs")
LABELS = ("ham", "spam")
DATASET_URL = "https://www.kaggle.com/datasets/maharshipandya/email-spam-dataset-extended"


def find_class_directory(folder_name, data_dir=DATA_DIR):
    matches = sorted(
        path for path in data_dir.rglob(folder_name) if path.is_dir()
    )
    if not matches:
        raise FileNotFoundError(
            f"Could not find a folder named {folder_name!r} under {data_dir}. "
            "Download and extract the Kaggle dataset into the data folder."
        )
    return matches[0]


def extract_email_text(path):
    try:
        with path.open("rb") as source:
            message = BytesParser(policy=policy.default).parse(source)

        subject = str(message.get("subject", ""))
        body_part = message.get_body(preferencelist=("plain",))
        body = body_part.get_content() if body_part is not None else ""
        if not isinstance(body, str):
            body = ""
        if not body.strip():
            body = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        subject = ""
        body = path.read_text(encoding="utf-8", errors="ignore")

    return " ".join(f"{subject}\n{body}".split())


def load_dataset(data_dir=DATA_DIR):
    texts = []
    labels = []
    folder_map = {"ham": "main_ham", "spam": "main_spam"}

    for label, folder_name in folder_map.items():
        class_directory = find_class_directory(folder_name, data_dir)
        email_files = sorted(
            path
            for path in class_directory.iterdir()
            if path.is_file() and not path.name.startswith(".")
        )

        for email_path in email_files:
            text = extract_email_text(email_path)
            if text:
                texts.append(text)
                labels.append(label)

    if not texts:
        raise ValueError("No readable email files were found in the dataset folders.")

    return texts, labels


def build_baseline_model():
    return Pipeline(
        [
            ("counts", CountVectorizer(min_df=2)),
            ("model", MultinomialNB()),
        ]
    )


def build_final_model():
    return Pipeline(
        [
            (
                "tfidf",
                TfidfVectorizer(
                    ngram_range=(1, 2),
                    min_df=2,
                    max_df=0.95,
                    sublinear_tf=True,
                ),
            ),
            (
                "model",
                LogisticRegression(class_weight="balanced", max_iter=1000),
            ),
        ]
    )


def evaluate_model(model, training_texts, training_labels, test_texts, test_labels):
    model.fit(training_texts, training_labels)
    predictions = model.predict(test_texts)
    report = classification_report(
        test_labels,
        predictions,
        labels=list(LABELS),
        output_dict=True,
        zero_division=0,
    )
    metrics = {
        "accuracy": accuracy_score(test_labels, predictions),
        "spam_precision": report["spam"]["precision"],
        "spam_recall": report["spam"]["recall"],
        "spam_f1": report["spam"]["f1-score"],
    }
    return predictions, metrics


def print_comparison(baseline_metrics, final_metrics):
    print("\nModel comparison")
    print(f"{'Model':<20} {'Accuracy':>10} {'Precision':>10} {'Recall':>10} {'F1':>10}")
    for name, metrics in (
        ("Count + Naive Bayes", baseline_metrics),
        ("TF-IDF + Logistic", final_metrics),
    ):
        print(
            f"{name:<20} "
            f"{metrics['accuracy']:>10.3f} "
            f"{metrics['spam_precision']:>10.3f} "
            f"{metrics['spam_recall']:>10.3f} "
            f"{metrics['spam_f1']:>10.3f}"
        )


def get_top_terms(final_model, count=10):
    vectorizer = final_model.named_steps["tfidf"]
    classifier = final_model.named_steps["model"]
    feature_names = vectorizer.get_feature_names_out()
    coefficients = classifier.coef_[0]

    if classifier.classes_[1] != "spam":
        coefficients = -coefficients

    ranked_terms = sorted(
        zip(feature_names, coefficients), key=lambda item: item[1]
    )
    ham_terms = [str(term) for term, _ in ranked_terms[:count]]
    spam_terms = [str(term) for term, _ in ranked_terms[-count:][::-1]]
    return ham_terms, spam_terms


def save_confusion_matrix(test_labels, predictions):
    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
    ConfusionMatrixDisplay.from_predictions(
        test_labels,
        predictions,
        labels=list(LABELS),
        display_labels=["Ham", "Spam"],
        colorbar=False,
    )
    plt.tight_layout()
    plt.savefig(OUTPUT_DIR / "confusion_matrix.png", dpi=150)
    plt.close()


def write_report(labels, training_size, test_size, baseline_metrics, final_metrics, ham_terms, spam_terms):
    counts = {label: labels.count(label) for label in LABELS}
    report = f"""# Email Spam Classifier Report

## Objective

Build and evaluate a machine-learning classifier that predicts whether a raw email is ham or spam.

## Dataset

- Source: {DATASET_URL}
- Parsed records used: {len(labels)}
- Ham records: {counts['ham']}
- Spam records: {counts['spam']}
- Training records: {training_size}
- Test records: {test_size}

## Preprocessing and method

The script parses each raw file with Python's MIME-aware email parser, combines the subject with the preferred plain-text body, and normalizes whitespace. It creates a stratified 80/20 split before either vectorizer is fitted. The baseline uses token counts with Multinomial Naive Bayes. The final pipeline uses TF-IDF unigrams and bigrams with class-balanced logistic regression.

## Results

| Model | Accuracy | Spam precision | Spam recall | Spam F1 |
| --- | ---: | ---: | ---: | ---: |
| Count + Naive Bayes | {baseline_metrics['accuracy']:.3f} | {baseline_metrics['spam_precision']:.3f} | {baseline_metrics['spam_recall']:.3f} | {baseline_metrics['spam_f1']:.3f} |
| TF-IDF + Logistic Regression | {final_metrics['accuracy']:.3f} | {final_metrics['spam_precision']:.3f} | {final_metrics['spam_recall']:.3f} | {final_metrics['spam_f1']:.3f} |

## Error analysis

The final model's confusion matrix is saved as `confusion_matrix.png`. A ham message predicted as spam is a false positive. A spam message predicted as ham is a false negative. These two error counts explain behavior that accuracy alone hides.

## Influential features

Terms most associated with ham: {', '.join(ham_terms)}

Terms most associated with spam: {', '.join(spam_terms)}

These weights show associations learned from this corpus. They do not prove that an individual word causes an email to be spam or ham.

## Limitations

- The dataset was assembled from SpamAssassin's Old Public Corpus, so it may not represent current spam campaigns.
- Text-only features do not inspect sender reputation, links, attachments, or mail-server behavior.
- Results describe this held-out split and should not be treated as production guarantees.

## Conclusion

The experiment produced a reproducible baseline and a leak-safe final pipeline. The final model is easier to inspect because its feature weights reveal influential terms, while precision, recall, F1, and the confusion matrix make its errors visible.
"""
    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
    (OUTPUT_DIR / "report.md").write_text(report, encoding="utf-8")


def main():
    texts, labels = load_dataset()
    counts = {label: labels.count(label) for label in LABELS}
    print(f"Loaded {len(texts)} emails: {counts['ham']} ham and {counts['spam']} spam.")
    print(f"Sample preview: {texts[0][:300]}\n")
    training_texts, test_texts, training_labels, test_labels = train_test_split(texts, labels, test_size=0.20, random_state=42, stratify=labels)
    baseline_model = build_baseline_model()
    final_model = build_final_model()
    _, baseline_metrics = evaluate_model(baseline_model, training_texts, training_labels, test_texts, test_labels)
    final_predictions, final_metrics = evaluate_model(final_model, training_texts, training_labels, test_texts, test_labels)
    print_comparison(baseline_metrics, final_metrics)
    ham_terms, spam_terms = get_top_terms(final_model)
    save_confusion_matrix(test_labels, final_predictions)
    write_report(labels, len(training_texts), len(test_texts), baseline_metrics, final_metrics, ham_terms, spam_terms)
    print("\nSaved outputs/confusion_matrix.png")
    print("Saved outputs/report.md")


if __name__ == "__main__":
    main()

How should you use this reference?

Compare each import, constant, function, and main call with your saved file. The generated artifacts depend on these names matching exactly.

Secret mission

Challenge the Classifier with Your Own Emails

A held-out score can look strong while unfamiliar wording exposes different behavior. Create a small behavioral test with original ham, spam, and ambiguous emails to see how the classifier responds.

Clean Up Your Resources

Clean Up Your Resources

Everything in this project stays on your Mac, so there are no cloud resources or ongoing costs. Your options are to keep the workspace available, pause your editing session, or delete the local project entirely.

Resources you used:

  • The complete email-spam-classifier folder on your Mac. It holds all code plus every local artifact created during this project.

Keep everything running

No action is needed. This option preserves your complete classifier for future experiments.

  • Leave the email-spam-classifier folder on your Mac.
  • Keep data/ inside the folder so both scripts can load the raw emails.
  • Keep outputs/report.md plus outputs/confusion_matrix.png as your submission evidence.
  • Leave .venv in place so the pinned packages remain available.

Pause - I'll come back to this later

Pausing closes the active editing session while preserving every project artifact. The classifier has no background service to stop.

  • Return to the Visual Studio Code window from earlier.
  • Close the active terminal panel to end its .venv session.
  • Close Visual Studio Code.
  • Leave email-spam-classifier on your Mac for your next session.

Delete - I don't want to use this again

Remove every local project resource when you no longer need the classifier. This option is permanent, but the command targets only the email-spam-classifier folder.

  • Return to the activated Visual Studio Code terminal from earlier.
  • Delete the complete project folder by running these commands:
deactivate
cd ..
rm -rf email-spam-classifier
ls

What does this cleanup do?

  • The first line exits the active virtual environment.
  • The second line moves the terminal outside the project folder.
  • The third line permanently removes the complete folder.
  • The final line lists the remaining folders so you can verify the deletion.
  • Confirm that email-spam-classifier is absent from the printed folder list.

Still See the Project Folder?

  • Check that the terminal started inside email-spam-classifier before the second line ran.
  • Check that the folder name uses the same spelling shown in the cleanup command.
  • Move the folder to the Trash using your Mac's file browser if the terminal reports a permission problem.

Help me troubleshoot deleting my local project folder.

Nice Work!

Nice Work!

You did it! Your Python email spam classifier now turns raw messages into labeled text for a reproducible experiment.

You've learned how to:

  • Built a raw-email parser that extracts normalized subject-plus-body text from MIME messages. The dataset folder structure supplies dependable ham or spam labels.
  • Compared a count-based Naive Bayes baseline with a leak-safe TF-IDF plus logistic regression pipeline. Spam precision measures unwanted ham blocking. Spam recall measures missed spam. Spam F1 balances both views.
  • Produced submission-ready evidence in outputs/report.md plus outputs/confusion_matrix.png. The confusion matrix separates false positives from false negatives. Influential terms make the final model easier to inspect.
  • Completed the optional Secret Mission by testing original obvious messages plus an ambiguous message. You recorded any mismatch between expected labels and predicted labels in outputs/report.md.

Ready to quiz yourself?