Build an Evidence-First Spam Triage

Build an evaluated SMS spam classifier with human review routing.

Introduction

30 Second Summary

One missed text can hide a delivery update or a message from someone you know. One unwanted promotion can still demand attention when a filter lets it through.

In this project, you will build a local SMS triage tool that compares a keyword baseline with a TF-IDF logistic regression model. You will measure both approaches on held-out messages before routing uncertain cases to human review.

What You'll Build

You paste a message into the terminal to see its predicted label, top model score, and route: ALLOW, BLOCK, or HUMAN REVIEW.

By the end of this project, you'll have:

  • A measured model comparison that generates separate baseline and model reports from one held-out test set.
  • An interactive triage demo that shows a predicted label plus its top model score. Each message receives an ALLOW, BLOCK, or HUMAN REVIEW route.
  • A public GitHub case study with reproducible setup instructions. Reviewers can inspect results.md for measured errors plus stated limitations.
  • Secret Mission: A cost-aware decision log that compares review thresholds. You defend one threshold using observed scores plus the relative costs of false positives and false negatives.

Are there any prerequisites?

You'll need Windows 10 or newer, Visual Studio Code, and a GitHub account. No prior machine learning project is required.

Before We Start

Evidence gives an AI or data hiring manager a reason to trust your judgment. This checkpoint locks in what the SMS triage system will prove before any hands-on work begins.

Set Up a Reproducible Python Workspace

A model result becomes credible when another person can recreate the environment that produced it. A fixed Python workspace gives your later evidence a repeatable foundation.

This step pins Python 3.15.2 for the project. A dedicated virtual environment isolates scikit-learn 1.9.1 from packages used by other projects.

In this step, get ready to:
  • Confirm that your Windows device supports the pinned Python installation.
  • Build an isolated workspace in Visual Studio Code with PowerShell.
  • Install the pinned machine learning dependency.
Confirm Windows and Python compatibility

Python 3.15.2 supports Windows 10 or newer. The operating system check protects you from installing an unsupported interpreter.

  • Select Start on the Windows taskbar.
  • Select Settings.
  • Select System.
  • Select About.
  • Read the edition beside Edition under Windows specifications.

Continue when Edition shows Windows 10 or Windows 11. Your computer meets the operating system requirement.

Using an Older Windows Version?

Python 3.15.2 requires Windows 10 or newer.

  • Move to a compatible Windows device before continuing.

The Python launcher reports which interpreter Windows can use. Checking it now prevents the virtual environment from inheriting an older installation.

  • Press the Windows key to open search.
  • Type Visual Studio Code into Windows search.
  • Press Enter.
  • Select View from the Visual Studio Code menu.
  • Select Terminal.
  • Select the Launch Profile dropdown if the terminal uses another shell.
  • Select PowerShell.
  • Check the available Python version by running this command:
py -3 --version

What Does This Command Check?

The Python launcher finds an installed Python 3 interpreter. The command prints the version that later workspace commands use.

The version check has three common outcomes. Choose the tab that matches your terminal.

✔️ I see the required Python version

Good work. Python 3.15.2 is available for your reproducible workspace.

ⓧ I see an older Python version

The project uses Python 3.15.2 so each learner starts from the same interpreter release. The installer opens a setup wizard before adding the pinned version.

  • Open the official release page for Python 3.15.2.
  • Select Windows installer (64-bit).
  • Run the downloaded installer.
  • Complete the setup wizard with its recommended selections.
  • Close every Visual Studio Code window after installation finishes.

A fresh Visual Studio Code session lets its terminal detect the new interpreter.

  • Press the Windows key to reopen search.
  • Type Visual Studio Code into Windows search.
  • Press Enter.
  • Select View from the menu.
  • Select Terminal.
  • Check the upgraded Python version by running this command:
py -3 --version

What Should the Check Confirm?

The reopened terminal checks the Python launcher again. You should see Python 3.15.2 in the output.

Still Seeing the Older Version?

Confirm that you selected the Python 3.15.2 installer. Close every Visual Studio Code window before reopening the editor.

Ask for help with the version mismatch: Why does the Python launcher still report an older version?

ⓧ Python launcher is unavailable

Windows cannot use the Python launcher because Python is unavailable. The official installer adds the interpreter required by this project.

  • Open the official release page for Python 3.15.2.
  • Select Windows installer (64-bit).
  • Run the downloaded installer.
  • Complete the setup wizard with its recommended selections.
  • Close every Visual Studio Code window after installation finishes.

A fresh Visual Studio Code session lets its terminal find the newly installed launcher.

  • Press the Windows key to reopen search.
  • Type Visual Studio Code into Windows search.
  • Press Enter.
  • Select View from the menu.
  • Select Terminal.
  • Check the new Python installation by running this command:
py -3 --version

What Should the Check Confirm?

The command confirms that Windows can now use the Python launcher. You should see Python 3.15.2 in the output.

Still Cannot Use the Python Launcher?

Confirm that the installer completed successfully. Restart Visual Studio Code so its terminal can detect the installation.

Ask for help with the missing launcher: Why is the Python launcher still unavailable after installation?

Create and activate the isolated workspace

A Visual Studio Code workspace keeps the experiment files under one named folder. Its local .venv directory keeps project packages separate from other Python work.

  • Select File from the Visual Studio Code menu.
  • Select Open Folder....
  • Select Desktop in the folder dialog.
  • Select New Folder.
  • Enter evidence-first-spam-triage as the folder name.
  • Select Select Folder.
  • Select Yes, I trust the authors if the Workspace Trust dialog appears.

The Explorer view now shows evidence-first-spam-triage. That folder is the dedicated home for your experiment.

Visual Studio Code starts its integrated terminal inside the open workspace. Creating the environment there places .venv in the correct folder.

  • Select View from the menu.
  • Select Terminal.
  • Select the Launch Profile dropdown if the terminal uses another shell.
  • Select PowerShell.
  • Create the local environment by running this command:
py -3 -m venv .venv

What Does This Command Create?

The command uses your Python installation to create an environment named .venv. Packages installed through its interpreter remain isolated inside the project.

  • Confirm that .venv appears beneath evidence-first-spam-triage in the Explorer view.

Your project now contains its own Python environment. The next command points the current terminal session at that environment.

Cannot See the Environment Folder?

Confirm that the Explorer shows evidence-first-spam-triage as the open folder. Retry the environment command from its integrated terminal.

Ask for help with environment creation: Why did the virtual environment folder fail to appear?

  • Activate the project environment by running this command:
.\.venv\Scripts\Activate.ps1

What Does Activation Change?

Activation points the current PowerShell session at the Python interpreter inside .venv. The next package installation stays inside this project.

  • Check the left side of the PowerShell prompt.

You should see the environment name before the project path. That visible prefix confirms that the isolated interpreter is active.

PowerShell Blocked the Activation Script?

PowerShell can block local activation scripts through its execution policy. The verified fallback changes the policy only for your current Windows user.

  • Allow local activation scripts for your current user by running this command:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

This policy allows locally created scripts to run for your user account. It leaves the machine-wide policy unchanged.

  • Retry the environment activation by running this command:
.\.venv\Scripts\Activate.ps1

The PowerShell prompt should now show the active environment name.

Ask for help with activation: Why is PowerShell blocking my virtual environment script?

Install and verify scikit-learn

A version pin makes the direct dependency repeatable. Running pip through the active python command keeps the package inside .venv.

  • Install the pinned package by running this command:
python -m pip install scikit-learn==1.9.1

What Does This Command Install?

The active Python interpreter runs the package installer. The exact-version pin installs scikit-learn 1.9.1 inside the project environment.

The installation is complete when the terminal returns to the PowerShell prompt without a package installation error.

Installation Did Not Finish?

Confirm that the PowerShell prompt shows the active environment name. Retry the command if the package download was interrupted.

Ask for help with the installation: Why is the pinned scikit-learn installation failing?

Before you run the final check, which package version do you expect the active environment to report?

  • Verify the installed package by running this command:
python -m pip show scikit-learn

What Does This Check Show?

The command prints metadata for the scikit-learn package used by the active Python interpreter. Its version field is the final reproducibility check.

You should see Version: 1.9.1 in the terminal. That is the workspace locked in: your project now has its pinned interpreter plus its required machine learning dependency.

Seeing a Different Package Version?

Confirm that the PowerShell prompt shows the active environment name. A package reported outside .venv belongs to another Python environment.

Ask for help with the mismatch: Why does the active environment report a different scikit-learn version?

Your reproducible workspace is ready. Next up, you will build a keyword baseline that looks plausible before a fixed message exposes its blind spot.

Build a Keyword Baseline

Your reproducible Python workspace is ready. The SMS triage system now needs a measured starting point called a baseline.

A keyword filter can look convincing when a message contains an obvious trigger. Its hand-written vocabulary misses spam that uses different wording.

In this step, get ready to:
  • Download the public SMS dataset into your project.
  • Evaluate keyword predictions on a held-out test set.
  • Expose a guaranteed blind spot with a fixed voucher offer.
Create the dataset loader

The UCI Machine Learning Repository SMS Spam Collection stores one class label beside each SMS message. Your first runnable version downloads that archive before loading every labeled line into memory.

  • In the Visual Studio Code sidebar, select the file-creation control beside the evidence-first-spam-triage folder.
  • Enter spam_triage.py as the file name.
  • Build the download layer by pasting this code into spam_triage.py:
from pathlib import Path
from urllib.request import urlretrieve
from zipfile import ZipFile

from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split

DATA_URL = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
DATA_DIR = Path("data")
ZIP_PATH = DATA_DIR / "sms-spam-collection.zip"
DATA_PATH = DATA_DIR / "SMSSpamCollection"
RANDOM_STATE = 42


def download_dataset():
    DATA_DIR.mkdir(parents=True, exist_ok=True)

    if not DATA_PATH.exists():
        print("Downloading the UCI SMS Spam Collection...")
        urlretrieve(DATA_URL, str(ZIP_PATH))
        with ZipFile(ZIP_PATH, "r") as archive:
            archive.extractall(DATA_DIR)

    return DATA_PATH

What does this code do?

  • The path constants keep the archive inside the project’s data folder.
  • The existence check prevents another download after the dataset has been extracted.
  • The archive extraction creates data/SMSSpamCollection for the loader.
  • In spam_triage.py, place the loader and runnable workflow below return DATA_PATH by pasting:
def load_dataset(path):
    messages = []
    labels = []

    with path.open(encoding="utf-8", errors="replace") as source:
        for line in source:
            line = line.strip()
            if not line:
                continue
            label, message = line.split(maxsplit=1)
            labels.append(label)
            messages.append(message)

    return messages, labels


def main():
    data_path = download_dataset()
    messages, labels = load_dataset(data_path)
    print(f"Loaded {len(messages):,} labeled messages.")


if __name__ == "__main__":
    main()

How does the loader work?

  • The loader separates each line into its label and message text.
  • The two lists preserve the matching order between every message and its correct label.
  • The main() workflow prints the number of records loaded from the extracted file.
  • Save spam_triage.py.
  • Download and load the dataset by running this command in the activated PowerShell terminal:
python spam_triage.py

What does this run prove?

The first run retrieves the official archive before extracting its dataset file. The message count proves that the loader read every labeled record.

You should see Loaded 5,574 labeled messages. in the terminal.

Good progress: your project can now retrieve the research dataset and turn it into paired messages with labels.

Dataset did not load?

Check that the activated environment marker remains visible in the PowerShell prompt. Confirm that spam_triage.py is inside the open evidence-first-spam-triage folder.

A download failure can come from a blocked network request. Retry after confirming that your browser can reach the UCI Machine Learning Repository.

Help me diagnose why my dataset download or loader failed.

Split the data and evaluate the baseline

A train-test split reserves unseen messages for evaluation. Stratification preserves the label proportions across the training and test groups.

The classification report measures each label’s prediction quality. The confusion matrix shows the counts behind those metrics.

  • In spam_triage.py, place the keyword configuration directly below RANDOM_STATE = 42 by pasting:
KEYWORDS = (
    "free",
    "winner",
    "win",
    "cash",
    "prize",
    "claim",
    "urgent",
    "reward",
    "call now",
    "txt",
)

STRESS_TEST = "Exclusive offer waiting. Reply now to receive a voucher."

What does this configuration hold?

  • The KEYWORDS tuple defines the baseline’s complete vocabulary.
  • The STRESS_TEST value gives every run the same fixed challenge.
  • In spam_triage.py, place the prediction function directly below return messages, labels by pasting:
def keyword_predict(message):
    normalized = message.lower()
    return "spam" if any(keyword in normalized for keyword in KEYWORDS) else "ham"

How does the rule make a prediction?

  • Lowercasing the message makes keyword matching consistent.
  • A matching trigger returns spam.
  • A message without a matching trigger returns ham.
  • Save spam_triage.py.
  • Confirm that the loader still runs after the new rule by running:
python spam_triage.py

What does this check protect?

This run catches a misplaced tuple or function before the evaluation code is added. A clean run keeps the feedback loop small.

You should still see Loaded 5,574 labeled messages. in the terminal.

Seeing a Python syntax error?

Check that every keyword has matching quotation marks. Confirm that the KEYWORDS tuple ends with its closing parenthesis.

Help me fix the keyword configuration or prediction function.

  • In main(), place the split and evaluation code directly below the loaded-message print by pasting:
    train_messages, test_messages, train_labels, test_labels = train_test_split(
        messages,
        labels,
        test_size=0.20,
        random_state=RANDOM_STATE,
        stratify=labels,
    )

    baseline_predictions = [
        keyword_predict(message) for message in test_messages
    ]
    baseline_report = classification_report(
        test_labels,
        baseline_predictions,
        labels=["ham", "spam"],
        digits=3,
        zero_division=0,
    )
    baseline_matrix = confusion_matrix(
        test_labels,
        baseline_predictions,
        labels=["ham", "spam"],
    )

What does this evaluation do?

  • The fixed RANDOM_STATE makes the split reproducible.
  • The test_size=0.20 setting reserves 20 percent of the messages for held-out evaluation.
  • The baseline predicts every message in the test set before its report is calculated.
  • The label order keeps the report aligned with the confusion matrix.
  • In main(), place the baseline output directly below the baseline_matrix calculation by pasting:
    print("\nKEYWORD BASELINE")
    print(baseline_report)
    print("Confusion matrix, rows actual and columns predicted:")
    print(baseline_matrix)

Why print both views?

The report summarizes prediction quality by label. The matrix exposes the underlying correct and incorrect counts.

  • Save spam_triage.py.
  • Generate the keyword-baseline evidence by running:
python spam_triage.py

What does this run measure?

The script applies the keyword rule to the held-out messages. It prints evidence from data that the rule did not use for training.

You should see the KEYWORD BASELINE heading above a classification report. A confusion matrix should follow the report.

Baseline report missing?

Check that the evaluation code is indented inside main(). Confirm that the output lines appear after the baseline_matrix calculation.

Help me find why the baseline evaluation does not print.

Expose the keyword blind spot

A fixed stress test gives the baseline a repeatable challenge outside the held-out report. Running it reveals how the rule handles a plausible voucher offer.

  • In main(), find this report heading:
    print("\nKEYWORD BASELINE")

Why locate this line?

This heading marks where the evaluation output begins. The stress-test result belongs immediately before it.

  • Insert the stress-test print above that heading so the two lines look like this:
    print(f"Keyword baseline stress test: {keyword_predict(STRESS_TEST)}")
    print("\nKEYWORD BASELINE")

What does this addition test?

The new line sends the fixed voucher offer through the same keyword rule. It prints the returned label before the broader evaluation report.

  • Save spam_triage.py.

Before you run it, do you think the fixed voucher offer will match the baseline as spam?

  • Run the completed baseline to test your prediction:
python spam_triage.py

What does the final run check?

This run repeats the same split before evaluating the baseline. It also sends the fixed voucher offer through the keyword rule.

The terminal prints Loaded 5,574 labeled messages. before printing Keyword baseline stress test: ham.

That shortfall is intentional. The offer avoids every configured trigger even though its wording deserves scrutiny.

You now have evidence that a plausible hand-written rule can miss messages outside its vocabulary.

Stress test result looks different?

Compare the STRESS_TEST value with the reference below. Check that KEYWORDS contains only the configured trigger strings.

Help me explain why my keyword stress test does not return ham.

✔️ Awesome, I've got everything!

Your saved baseline downloads the dataset and prints reproducible evaluation evidence. The fixed voucher offer also exposes the intended keyword blind spot.

ⓧ I'd like to double check the full code

from pathlib import Path
from urllib.request import urlretrieve
from zipfile import ZipFile

from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split

DATA_URL = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
DATA_DIR = Path("data")
ZIP_PATH = DATA_DIR / "sms-spam-collection.zip"
DATA_PATH = DATA_DIR / "SMSSpamCollection"
RANDOM_STATE = 42

KEYWORDS = (
    "free",
    "winner",
    "win",
    "cash",
    "prize",
    "claim",
    "urgent",
    "reward",
    "call now",
    "txt",
)

STRESS_TEST = "Exclusive offer waiting. Reply now to receive a voucher."


def download_dataset():
    DATA_DIR.mkdir(parents=True, exist_ok=True)

    if not DATA_PATH.exists():
        print("Downloading the UCI SMS Spam Collection...")
        urlretrieve(DATA_URL, str(ZIP_PATH))
        with ZipFile(ZIP_PATH, "r") as archive:
            archive.extractall(DATA_DIR)

    return DATA_PATH


def load_dataset(path):
    messages = []
    labels = []

    with path.open(encoding="utf-8", errors="replace") as source:
        for line in source:
            line = line.strip()
            if not line:
                continue
            label, message = line.split(maxsplit=1)
            labels.append(label)
            messages.append(message)

    return messages, labels


def keyword_predict(message):
    normalized = message.lower()
    return "spam" if any(keyword in normalized for keyword in KEYWORDS) else "ham"


def main():
    data_path = download_dataset()
    messages, labels = load_dataset(data_path)
    print(f"Loaded {len(messages):,} labeled messages.")

    train_messages, test_messages, train_labels, test_labels = train_test_split(
        messages,
        labels,
        test_size=0.20,
        random_state=RANDOM_STATE,
        stratify=labels,
    )

    baseline_predictions = [
        keyword_predict(message) for message in test_messages
    ]
    baseline_report = classification_report(
        test_labels,
        baseline_predictions,
        labels=["ham", "spam"],
        digits=3,
        zero_division=0,
    )
    baseline_matrix = confusion_matrix(
        test_labels,
        baseline_predictions,
        labels=["ham", "spam"],
    )

    print(f"Keyword baseline stress test: {keyword_predict(STRESS_TEST)}")
    print("\nKEYWORD BASELINE")
    print(baseline_report)
    print("Confusion matrix, rows actual and columns predicted:")
    print(baseline_matrix)


if __name__ == "__main__":
    main()

How to use this reference

Compare this reference with your saved spam_triage.py from top to bottom. The order matters because the constants and functions must exist before main() uses them.

Your baseline now has a measurable report and a visible failure case. Next, you will test whether a TF-IDF and logistic regression pipeline handles broader wording more effectively.

Evaluate a TF-IDF Model

Your keyword baseline now exposes a guaranteed blind spot. That gives you an honest starting point for measuring whether a learned model adds value.

A TF-IDF and logistic regression pipeline can learn broader text patterns than a fixed keyword list. You will train it only on the training split before comparing both approaches on the same held-out messages.

In this step, get ready to:
  • Build a text-classification pipeline from TF-IDF features and logistic regression.
  • Evaluate the keyword baseline and model on the same held-out split.
  • Generate a reproducible evidence report with metrics and observed errors.
Build the text-classification pipeline

A Pipeline keeps feature extraction and classification in one repeatable workflow. Scikit-learn applies each pipeline step in sequence whenever you fit or predict.

  • In spam_triage.py, locate these existing metric and split imports:
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split

What are you locating?

These imports already produce the baseline evaluation. The model imports belong beside them because they extend the same evaluation workflow.

  • Replace that import section with the expanded scikit-learn imports below:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

What do these imports add?

  • TfidfVectorizer converts message text into weighted numeric features.
  • LogisticRegression learns which feature patterns distinguish ham from spam.
  • Pipeline binds those operations into one estimator.
  • Save spam_triage.py.
  • Confirm the new imports load successfully by running:
python spam_triage.py

What should you see?

You should still see the keyword stress test and baseline evaluation. Reaching that output confirms Python loaded the new model classes successfully.

Import failed?

Confirm the activated environment marker is still visible in the terminal prompt. The scikit-learn installation belongs inside that environment.

If the imports still fail, help me diagnose the scikit-learn import problem.

The model configuration controls which text patterns it can learn. Bigrams capture short phrases while balanced class weights give the smaller spam class more influence during training.

  • In spam_triage.py, find the completed keyword_predict() function.
  • Add build_model() immediately below that function by pasting:
def build_model():
    return Pipeline(
        [
            (
                "tfidf",
                TfidfVectorizer(
                    ngram_range=(1, 2),
                    min_df=2,
                    sublinear_tf=True,
                ),
            ),
            (
                "classifier",
                LogisticRegression(
                    class_weight="balanced",
                    max_iter=1000,
                ),
            ),
        ]
    )

What does this model do?

  • The tfidf step learns weighted single-word and two-word features from the training messages.
  • The minimum document frequency removes features that appear in fewer than two messages.
  • The classifier step learns a decision boundary from those sparse features.
  • The balanced weighting adjusts each class according to its frequency in the training labels.
  • Save spam_triage.py.
  • Confirm Python can load the new function by running:
python spam_triage.py

What does this check prove?

You should reach the existing baseline report without a syntax error. That confirms the model factory is ready to connect to the training split.

  • In main(), find the line that prints baseline_matrix.
  • Add the model training and evaluation block directly below that line by pasting:
    model = build_model()
    model.fit(train_messages, train_labels)
    model_predictions = model.predict(test_messages)
    model_report = classification_report(
        test_labels,
        model_predictions,
        labels=["ham", "spam"],
        digits=3,
        zero_division=0,
    )
    model_matrix = confusion_matrix(
        test_labels,
        model_predictions,
        labels=["ham", "spam"],
    )

    print("\nTF-IDF LOGISTIC REGRESSION")
    print(model_report)
    print("Confusion matrix, rows actual and columns predicted:")
    print(model_matrix)

How is the comparison kept fair?

The pipeline fits only on train_messages and train_labels. The keyword baseline and trained model both make predictions for the same test_messages.

The classification report shows precision, recall, and F1 score by class. The confusion matrix exposes the exact counts behind those summary metrics.

  • Save spam_triage.py.
  • Predict whether the model report will appear below the baseline report when you run the updated script.
  • Train and evaluate the pipeline by running:
python spam_triage.py

What should you see?

You should see separate KEYWORD BASELINE and TF-IDF LOGISTIC REGRESSION sections. Each section includes a classification report followed by a confusion matrix.

That comparison is your first measured answer to whether the learned pipeline improves on the brittle keyword rule.

Model report missing?

Check that the new model block remains indented inside main(). Confirm that model.fit() receives the training variables from the existing split.

If the run stops before the model report, help me debug the TF-IDF pipeline.

Capture errors in a reusable report

Summary metrics reveal overall behavior. Error analysis reveals the real messages behind false positives and false negatives.

The report will preserve both forms of evidence in results.md. It will also record the dataset source and held-out split policy.

  • Near the top of spam_triage.py, locate this dataset path constant:
DATA_PATH = DATA_DIR / "SMSSpamCollection"

Why locate this constant?

The generated evidence needs its own stable destination. Keeping that path beside the dataset paths makes the file locations easy to audit.

  • Add the results path directly below the dataset path so the constants look like this:
DATA_PATH = DATA_DIR / "SMSSpamCollection"
RESULTS_PATH = Path("results.md")

What does this path represent?

RESULTS_PATH points to the Markdown evidence file in the evidence-first-spam-triage folder. Each complete run replaces that file with current results.

  • Add format_confusion_matrix() below build_model() by pasting:
def format_confusion_matrix(matrix):
    return (
        "| Actual / Predicted | ham | spam |\n"
        "|---|---:|---:|\n"
        f"| ham | {matrix[0][0]} | {matrix[0][1]} |\n"
        f"| spam | {matrix[1][0]} | {matrix[1][1]} |"
    )

What does this helper produce?

The helper converts a two-by-two confusion matrix into a Markdown table. Its labels make the actual and predicted directions explicit.

  • Add format_error_examples() directly below the confusion-matrix helper by pasting:
def format_error_examples(messages, actual, predicted, actual_label, predicted_label):
    rows = []

    for message, true_label, predicted_label_value in zip(
        messages, actual, predicted
    ):
        if true_label == actual_label and predicted_label_value == predicted_label:
            safe_message = message.replace("|", "\\|").replace("\n", " ")
            rows.append(
                f"| {actual_label} | {predicted_label} | {safe_message[:160]} |"
            )
        if len(rows) == 3:
            break

    if not rows:
        return "No examples appeared in this held-out split."

    return "\n".join(
        [
            "| Actual | Predicted | Message excerpt |",
            "|---|---|---|",
            *rows,
        ]
    )

How are errors selected?

The helper scans the held-out messages for a requested actual-label and predicted-label combination. It keeps up to three representative examples.

Message excerpts are shortened before entering the report. Markdown-sensitive characters are escaped so the table remains readable.

  • Save spam_triage.py.
  • Confirm both helpers parse successfully by running:
python spam_triage.py

What should still work?

You should still reach both evaluation reports. That confirms the formatting helpers load without disrupting the working model evaluation.

The report writer is one function split into four copyable segments. Keep every segment at the same indentation before saving the complete function.

  • Add write_results() directly below format_error_examples() by pasting this first segment:
def write_results(
    message_count,
    test_messages,
    test_labels,
    baseline_predictions,
    baseline_report,
    baseline_matrix,
    model_predictions,
    model_report,
    model_matrix,
):
    baseline_false_negatives = format_error_examples(
        test_messages,
        test_labels,
        baseline_predictions,
        actual_label="spam",
        predicted_label="ham",
    )

What does the first segment collect?

The function receives the evidence already produced in main(). Its first error table captures spam messages that the keyword baseline allowed as ham.

  • Continue inside write_results() by pasting this second segment immediately below the first:
    model_false_positives = format_error_examples(
        test_messages,
        test_labels,
        model_predictions,
        actual_label="ham",
        predicted_label="spam",
    )
    model_false_negatives = format_error_examples(
        test_messages,
        test_labels,
        model_predictions,
        actual_label="spam",
        predicted_label="ham",
    )

Which model errors are preserved?

False positives are legitimate messages predicted as spam. False negatives are spam messages predicted as ham.

Preserving both directions stops a strong summary score from hiding costly individual mistakes.

  • Continue the same function by pasting the first half of the report template:
    report = f"""# Evaluation Results

Dataset: UCI SMS Spam Collection

Source: {DATA_URL}

Messages loaded: {message_count:,}

Evaluation policy: 80 percent training data and 20 percent held-out test data, stratified by label, with `random_state={RANDOM_STATE}`.

## Keyword Baseline

```text
{baseline_report}
```

{format_confusion_matrix(baseline_matrix)}

### Baseline False Negatives

{baseline_false_negatives}

What evidence appears first?

The report begins with dataset provenance and the reproducible split policy. It then records the complete keyword-baseline report and its false negatives.

  • Complete the open report template by pasting this final segment immediately below the previous segment:
## TF-IDF Logistic Regression

```text
{model_report}
```

{format_confusion_matrix(model_matrix)}

### Model False Positives

{model_false_positives}

### Model False Negatives

{model_false_negatives}

## Interpretation Boundary

These are held-out results from a public historical research dataset. They demonstrate a reproducible experiment, not production readiness. The top model score is used as a review signal, but this project does not claim that the probabilities are calibrated for real-world blocking decisions.
"""

    RESULTS_PATH.write_text(report, encoding="utf-8")

Why include an interpretation boundary?

The final section records the model report and both model error directions. It also limits the claim to a reproducible experiment on historical research data.

That boundary keeps measured evidence separate from untested production claims.

  • Save spam_triage.py.
  • Confirm the completed report function parses by running:
python spam_triage.py

What should this run confirm?

You should reach both terminal reports again. That proves Python can parse the completed multiline report before you connect it to main().

Seeing a syntax or indentation problem?

Confirm every report segment sits inside write_results() with four spaces of indentation where shown. Check that the opening and closing triple quotes both remain present.

If the traceback points into the report template, help me repair the write_results function.

Run and inspect the evaluation

The final connection passes both evaluation results into the report writer. One run will now refresh the terminal comparison and the saved evidence artifact.

  • In main(), find the line that prints model_matrix.
  • Add the report-writing call directly below that line by pasting:
    write_results(
        message_count=len(messages),
        test_messages=test_messages,
        test_labels=test_labels,
        baseline_predictions=baseline_predictions,
        baseline_report=baseline_report,
        baseline_matrix=baseline_matrix,
        model_predictions=model_predictions,
        model_report=model_report,
        model_matrix=model_matrix,
    )
    print("Saved evaluation evidence to results.md")

What completes the evidence flow?

The call passes the shared test messages and labels into write_results(). It also passes each approach's predictions and metrics.

The final print statement gives you a visible confirmation after results.md is written successfully.

  • Save spam_triage.py.
  • Predict whether the run will produce one shared evidence file containing both evaluations.
  • Generate the final evaluation evidence by running:
python spam_triage.py

What should you see?

You should see separate KEYWORD BASELINE and TF-IDF LOGISTIC REGRESSION reports. The final terminal line should read Saved evaluation evidence to results.md.

That is the evidence layer working: one run now reproduces the comparison and preserves the observed errors.

Results file missing?

Confirm write_results() is called inside main() after the model evaluation. Check that RESULTS_PATH is defined near the other path constants.

If the terminal reaches the reports without creating the file, help me trace why results.md is not being written.

  • In Visual Studio Code's left sidebar, select results.md.
  • Confirm the file starts with # Evaluation Results.
  • Confirm the file records 5,574 loaded messages.
  • Confirm the split policy records 80 percent training data and 20 percent held-out test data.
  • Confirm the keyword section includes baseline false negatives.
  • Confirm the model section includes false positives and false negatives.
  • Confirm the final section states the prototype interpretation boundary.

✔️ Awesome, I've got everything!

Great work. Save spam_triage.py and keep the generated results.md beside it.

ⓧ I'd like to double check the full code

from pathlib import Path
from urllib.request import urlretrieve
from zipfile import ZipFile

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

DATA_URL = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
DATA_DIR = Path("data")
ZIP_PATH = DATA_DIR / "sms-spam-collection.zip"
DATA_PATH = DATA_DIR / "SMSSpamCollection"
RESULTS_PATH = Path("results.md")
RANDOM_STATE = 42

KEYWORDS = (
    "free",
    "winner",
    "win",
    "cash",
    "prize",
    "claim",
    "urgent",
    "reward",
    "call now",
    "txt",
)

STRESS_TEST = "Exclusive offer waiting. Reply now to receive a voucher."


def download_dataset():
    DATA_DIR.mkdir(parents=True, exist_ok=True)

    if not DATA_PATH.exists():
        print("Downloading the UCI SMS Spam Collection...")
        urlretrieve(DATA_URL, str(ZIP_PATH))
        with ZipFile(ZIP_PATH, "r") as archive:
            archive.extractall(DATA_DIR)

    return DATA_PATH


def load_dataset(path):
    messages = []
    labels = []

    with path.open(encoding="utf-8", errors="replace") as source:
        for line in source:
            line = line.strip()
            if not line:
                continue
            label, message = line.split(maxsplit=1)
            labels.append(label)
            messages.append(message)

    return messages, labels


def keyword_predict(message):
    normalized = message.lower()
    return "spam" if any(keyword in normalized for keyword in KEYWORDS) else "ham"


def build_model():
    return Pipeline(
        [
            (
                "tfidf",
                TfidfVectorizer(
                    ngram_range=(1, 2),
                    min_df=2,
                    sublinear_tf=True,
                ),
            ),
            (
                "classifier",
                LogisticRegression(
                    class_weight="balanced",
                    max_iter=1000,
                ),
            ),
        ]
    )


def format_confusion_matrix(matrix):
    return (
        "| Actual / Predicted | ham | spam |\n"
        "|---|---:|---:|\n"
        f"| ham | {matrix[0][0]} | {matrix[0][1]} |\n"
        f"| spam | {matrix[1][0]} | {matrix[1][1]} |"
    )


def format_error_examples(messages, actual, predicted, actual_label, predicted_label):
    rows = []

    for message, true_label, predicted_label_value in zip(
        messages, actual, predicted
    ):
        if true_label == actual_label and predicted_label_value == predicted_label:
            safe_message = message.replace("|", "\\|").replace("\n", " ")
            rows.append(
                f"| {actual_label} | {predicted_label} | {safe_message[:160]} |"
            )
        if len(rows) == 3:
            break

    if not rows:
        return "No examples appeared in this held-out split."

    return "\n".join(
        [
            "| Actual | Predicted | Message excerpt |",
            "|---|---|---|",
            *rows,
        ]
    )


def write_results(
    message_count,
    test_messages,
    test_labels,
    baseline_predictions,
    baseline_report,
    baseline_matrix,
    model_predictions,
    model_report,
    model_matrix,
):
    baseline_false_negatives = format_error_examples(
        test_messages,
        test_labels,
        baseline_predictions,
        actual_label="spam",
        predicted_label="ham",
    )
    model_false_positives = format_error_examples(
        test_messages,
        test_labels,
        model_predictions,
        actual_label="ham",
        predicted_label="spam",
    )
    model_false_negatives = format_error_examples(
        test_messages,
        test_labels,
        model_predictions,
        actual_label="spam",
        predicted_label="ham",
    )

    report = f"""# Evaluation Results

Dataset: UCI SMS Spam Collection

Source: {DATA_URL}

Messages loaded: {message_count:,}

Evaluation policy: 80 percent training data and 20 percent held-out test data, stratified by label, with `random_state={RANDOM_STATE}`.

## Keyword Baseline

```text
{baseline_report}
```

{format_confusion_matrix(baseline_matrix)}

### Baseline False Negatives

{baseline_false_negatives}

## TF-IDF Logistic Regression

```text
{model_report}
```

{format_confusion_matrix(model_matrix)}

### Model False Positives

{model_false_positives}

### Model False Negatives

{model_false_negatives}

## Interpretation Boundary

These are held-out results from a public historical research dataset. They demonstrate a reproducible experiment, not production readiness. The top model score is used as a review signal, but this project does not claim that the probabilities are calibrated for real-world blocking decisions.
"""

    RESULTS_PATH.write_text(report, encoding="utf-8")


def main():
    data_path = download_dataset()
    messages, labels = load_dataset(data_path)
    print(f"Loaded {len(messages):,} labeled messages.")

    train_messages, test_messages, train_labels, test_labels = train_test_split(
        messages,
        labels,
        test_size=0.20,
        random_state=RANDOM_STATE,
        stratify=labels,
    )

    baseline_predictions = [
        keyword_predict(message) for message in test_messages
    ]
    baseline_report = classification_report(
        test_labels,
        baseline_predictions,
        labels=["ham", "spam"],
        digits=3,
        zero_division=0,
    )
    baseline_matrix = confusion_matrix(
        test_labels,
        baseline_predictions,
        labels=["ham", "spam"],
    )

    print(f"Keyword baseline stress test: {keyword_predict(STRESS_TEST)}")
    print("\nKEYWORD BASELINE")
    print(baseline_report)
    print("Confusion matrix, rows actual and columns predicted:")
    print(baseline_matrix)

    model = build_model()
    model.fit(train_messages, train_labels)
    model_predictions = model.predict(test_messages)
    model_report = classification_report(
        test_labels,
        model_predictions,
        labels=["ham", "spam"],
        digits=3,
        zero_division=0,
    )
    model_matrix = confusion_matrix(
        test_labels,
        model_predictions,
        labels=["ham", "spam"],
    )

    print("\nTF-IDF LOGISTIC REGRESSION")
    print(model_report)
    print("Confusion matrix, rows actual and columns predicted:")
    print(model_matrix)

    write_results(
        message_count=len(messages),
        test_messages=test_messages,
        test_labels=test_labels,
        baseline_predictions=baseline_predictions,
        baseline_report=baseline_report,
        baseline_matrix=baseline_matrix,
        model_predictions=model_predictions,
        model_report=model_report,
        model_matrix=model_matrix,
    )
    print("Saved evaluation evidence to results.md")


if __name__ == "__main__":
    main()

How should you compare the file?

Check the imports and constants first. Then compare each helper before reviewing the evaluation order inside main().

The finished evaluation version ends after writing results.md and calling main().

Your model comparison is now reproducible and backed by inspectable evidence. Next, you will turn the model's score into an action policy with a human review route.

Add Human Review and Publish Evidence

Your held-out evaluation now compares the keyword baseline with the trained model. This final step turns that evidence into a triage tool that someone else can inspect.

A useful machine learning system needs an action policy with honest limits. You will route uncertain predictions to a person before publishing the tested prototype.

In this step, get ready to:
  • Add score-based routes to the trained model.
  • Document the prototype’s evidence and limitations.
  • Publish the reproducible case study on GitHub.
Add the human review policy

The top model score becomes a review signal. Scores below 0.80 go to HUMAN REVIEW.

Higher-scoring ham predictions go to ALLOW. Higher-scoring spam predictions go to BLOCK.

  • In spam_triage.py, find RESULTS_PATH = Path("results.md") near the top of the file.
  • Add the review threshold immediately below that line by entering this code:
REVIEW_THRESHOLD = 0.80

What does this value control?

The threshold separates automatic routes from human review. The code treats it as a teaching policy instead of proof that the score is calibrated for production decisions.

  • Scroll to the blank line below the completed write_results function.
  • Add the triage decision function by pasting this code:
def triage_message(model, message):
    probabilities = model.predict_proba([message])[0]
    classes = model.named_steps["classifier"].classes_
    best_index = probabilities.argmax()
    predicted_label = classes[best_index]
    top_score = float(probabilities[best_index])

    if top_score < REVIEW_THRESHOLD:
        route = "HUMAN REVIEW"
    elif predicted_label == "spam":
        route = "BLOCK"
    else:
        route = "ALLOW"

    return predicted_label, top_score, route

How does the decision function work?

  • The model’s predict_proba method produces a score for each possible label.
  • The classes_ values connect each score to its matching label.
  • The highest score selects the predicted label.
  • The threshold converts that prediction into an operational route.
  • Save spam_triage.py.
  • Confirm the new function has valid syntax by running this command:
python spam_triage.py

What does this check prove?

Python reads the new function before running the existing evaluation workflow. A complete evaluation run confirms that the policy code contains no syntax errors.

You should see both evaluation reports. You should also see Saved evaluation evidence to results.md at the end.

Seeing a syntax error?

Check that triage_message begins at the left edge of the file. Confirm that each route branch uses consistent four-space indentation.

Compare the closing return line with the code above. Help me find the syntax error in my triage_message function.

  • Add the interactive prompt directly below triage_message by pasting this code:
def run_interactive_demo(model):
    print("\nINTERACTIVE TRIAGE")
    print("Enter a blank message to exit.")

    while True:
        message = input("\nMessage: ").strip()
        if not message:
            break

        label, score, route = triage_message(model, message)
        print(f"Predicted label: {label}")
        print(f"Top model score: {score:.3f}")
        print(f"Route: {route}")

What does the interactive demo do?

  • The loop accepts one message at a time.
  • A blank message ends the demo.
  • Each completed prediction displays the label.
  • The same prediction also displays its top score and route.
  • Save spam_triage.py.
  • In the main function, find print("Saved evaluation evidence to results.md").
  • Add the interactive demo call immediately below that line by entering this code:
    run_interactive_demo(model)

Why call the demo here?

The model has already been trained when this line runs. The same fitted model can evaluate the held-out test set before handling messages entered by a person.

Before you run the finished program, which route do you think the voucher message receives?

  • Save spam_triage.py.
  • Run the completed program by entering this command:
python spam_triage.py

What happens during this run?

The program evaluates both approaches before opening the interactive prompt. Your terminal then waits for a message under INTERACTIVE TRIAGE.

  • At the Message: prompt, test the model with this message:
Exclusive offer waiting. Reply now to receive a voucher.

What are you testing?

This message exposed the keyword baseline’s blind spot earlier. The interactive run now shows how the trained model scores the same wording before assigning a route.

  • Press Enter on a blank Message: prompt to exit the demo.

You should see Predicted label: followed by Top model score:. The final line shows ALLOW, BLOCK, or HUMAN REVIEW.

Interactive prompt not appearing?

Confirm that run_interactive_demo(model) is inside main. It should align with the nearby print statement.

If the program exits after writing results.md, save the file before running it again. Help me find why my interactive prompt does not run.

✔️ Awesome, I've got everything!

Your completed spam_triage.py now evaluates the model before opening the interactive triage prompt.

ⓧ I'd like to double check the full code

from pathlib import Path
from urllib.request import urlretrieve
from zipfile import ZipFile

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

DATA_URL = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
DATA_DIR = Path("data")
ZIP_PATH = DATA_DIR / "sms-spam-collection.zip"
DATA_PATH = DATA_DIR / "SMSSpamCollection"
RESULTS_PATH = Path("results.md")
REVIEW_THRESHOLD = 0.80
RANDOM_STATE = 42

KEYWORDS = (
    "free",
    "winner",
    "win",
    "cash",
    "prize",
    "claim",
    "urgent",
    "reward",
    "call now",
    "txt",
)

STRESS_TEST = "Exclusive offer waiting. Reply now to receive a voucher."


def download_dataset():
    DATA_DIR.mkdir(parents=True, exist_ok=True)

    if not DATA_PATH.exists():
        print("Downloading the UCI SMS Spam Collection...")
        urlretrieve(DATA_URL, str(ZIP_PATH))
        with ZipFile(ZIP_PATH, "r") as archive:
            archive.extractall(DATA_DIR)

    return DATA_PATH


def load_dataset(path):
    messages = []
    labels = []

    with path.open(encoding="utf-8", errors="replace") as source:
        for line in source:
            line = line.strip()
            if not line:
                continue
            label, message = line.split(maxsplit=1)
            labels.append(label)
            messages.append(message)

    return messages, labels


def keyword_predict(message):
    normalized = message.lower()
    return "spam" if any(keyword in normalized for keyword in KEYWORDS) else "ham"


def build_model():
    return Pipeline(
        [
            (
                "tfidf",
                TfidfVectorizer(
                    ngram_range=(1, 2),
                    min_df=2,
                    sublinear_tf=True,
                ),
            ),
            (
                "classifier",
                LogisticRegression(
                    class_weight="balanced",
                    max_iter=1000,
                ),
            ),
        ]
    )


def format_confusion_matrix(matrix):
    return (
        "| Actual / Predicted | ham | spam |\n"
        "|---|---:|---:|\n"
        f"| ham | {matrix[0][0]} | {matrix[0][1]} |\n"
        f"| spam | {matrix[1][0]} | {matrix[1][1]} |"
    )


def format_error_examples(messages, actual, predicted, actual_label, predicted_label):
    rows = []

    for message, true_label, predicted_label_value in zip(
        messages, actual, predicted
    ):
        if true_label == actual_label and predicted_label_value == predicted_label:
            safe_message = message.replace("|", "\\|").replace("\n", " ")
            rows.append(
                f"| {actual_label} | {predicted_label} | {safe_message[:160]} |"
            )
        if len(rows) == 3:
            break

    if not rows:
        return "No examples appeared in this held-out split."

    return "\n".join(
        [
            "| Actual | Predicted | Message excerpt |",
            "|---|---|---|",
            *rows,
        ]
    )


def write_results(
    message_count,
    test_messages,
    test_labels,
    baseline_predictions,
    baseline_report,
    baseline_matrix,
    model_predictions,
    model_report,
    model_matrix,
):
    baseline_false_negatives = format_error_examples(
        test_messages,
        test_labels,
        baseline_predictions,
        actual_label="spam",
        predicted_label="ham",
    )
    model_false_positives = format_error_examples(
        test_messages,
        test_labels,
        model_predictions,
        actual_label="ham",
        predicted_label="spam",
    )
    model_false_negatives = format_error_examples(
        test_messages,
        test_labels,
        model_predictions,
        actual_label="spam",
        predicted_label="ham",
    )

    report = f"""# Evaluation Results

Dataset: UCI SMS Spam Collection

Source: {DATA_URL}

Messages loaded: {message_count:,}

Evaluation policy: 80 percent training data and 20 percent held-out test data, stratified by label, with `random_state={RANDOM_STATE}`.

## Keyword Baseline

```text
{baseline_report}
```

{format_confusion_matrix(baseline_matrix)}

### Baseline False Negatives

{baseline_false_negatives}

## TF-IDF Logistic Regression

```text
{model_report}
```

{format_confusion_matrix(model_matrix)}

### Model False Positives

{model_false_positives}

### Model False Negatives

{model_false_negatives}

## Interpretation Boundary

These are held-out results from a public historical research dataset. They demonstrate a reproducible experiment, not production readiness. The top model score is used as a review signal, but this project does not claim that the probabilities are calibrated for real-world blocking decisions.
"""

    RESULTS_PATH.write_text(report, encoding="utf-8")


def triage_message(model, message):
    probabilities = model.predict_proba([message])[0]
    classes = model.named_steps["classifier"].classes_
    best_index = probabilities.argmax()
    predicted_label = classes[best_index]
    top_score = float(probabilities[best_index])

    if top_score < REVIEW_THRESHOLD:
        route = "HUMAN REVIEW"
    elif predicted_label == "spam":
        route = "BLOCK"
    else:
        route = "ALLOW"

    return predicted_label, top_score, route


def run_interactive_demo(model):
    print("\nINTERACTIVE TRIAGE")
    print("Enter a blank message to exit.")

    while True:
        message = input("\nMessage: ").strip()
        if not message:
            break

        label, score, route = triage_message(model, message)
        print(f"Predicted label: {label}")
        print(f"Top model score: {score:.3f}")
        print(f"Route: {route}")


def main():
    data_path = download_dataset()
    messages, labels = load_dataset(data_path)
    print(f"Loaded {len(messages):,} labeled messages.")

    train_messages, test_messages, train_labels, test_labels = train_test_split(
        messages,
        labels,
        test_size=0.20,
        random_state=RANDOM_STATE,
        stratify=labels,
    )

    baseline_predictions = [
        keyword_predict(message) for message in test_messages
    ]
    baseline_report = classification_report(
        test_labels,
        baseline_predictions,
        labels=["ham", "spam"],
        digits=3,
        zero_division=0,
    )
    baseline_matrix = confusion_matrix(
        test_labels,
        baseline_predictions,
        labels=["ham", "spam"],
    )

    print(f"Keyword baseline stress test: {keyword_predict(STRESS_TEST)}")
    print("\nKEYWORD BASELINE")
    print(baseline_report)
    print("Confusion matrix, rows actual and columns predicted:")
    print(baseline_matrix)

    model = build_model()
    model.fit(train_messages, train_labels)
    model_predictions = model.predict(test_messages)
    model_report = classification_report(
        test_labels,
        model_predictions,
        labels=["ham", "spam"],
        digits=3,
        zero_division=0,
    )
    model_matrix = confusion_matrix(
        test_labels,
        model_predictions,
        labels=["ham", "spam"],
    )

    print("\nTF-IDF LOGISTIC REGRESSION")
    print(model_report)
    print("Confusion matrix, rows actual and columns predicted:")
    print(model_matrix)

    write_results(
        message_count=len(messages),
        test_messages=test_messages,
        test_labels=test_labels,
        baseline_predictions=baseline_predictions,
        baseline_report=baseline_report,
        baseline_matrix=baseline_matrix,
        model_predictions=model_predictions,
        model_report=model_report,
        model_matrix=model_matrix,
    )
    print("Saved evaluation evidence to results.md")

    run_interactive_demo(model)


if __name__ == "__main__":
    main()
Document the tested prototype

A case study needs enough context for another person to reproduce the experiment. It also needs boundaries that stop historical test results from being presented as production proof.

  • At the top of Visual Studio Code’s file sidebar, select the new-file control.
  • Name the file README.md.
  • Add the project purpose and reproduction instructions by pasting this first section:
# Evidence-First SMS Spam Triage

A tested machine learning prototype that compares a deliberately limited keyword baseline with a TF-IDF logistic regression model, analyzes held-out errors, and sends uncertain predictions to human review.

## What This Project Proves

- A baseline can expose whether a more complex model adds value.
- Precision, recall, F1 score, and confusion matrices reveal different failure modes.
- False positives and false negatives should be inspected, not hidden behind one accuracy value.
- An ML prediction still needs an action policy and an escalation path.

## Dataset

The project downloads the UCI Machine Learning Repository SMS Spam Collection from:

https://archive.ics.uci.edu/dataset/228/sms%2Bspam%2Bcollection

The repository describes 5,574 labeled SMS messages. The downloaded dataset is excluded from Git because the script can retrieve it reproducibly from the official source.

## Reproduce the Experiment on Windows

```powershell
py -3 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python spam_triage.py
```

What does this section establish?

  • The opening defines the project as a tested prototype.
  • The evidence claims focus on observable evaluation practices.
  • The dataset section names the public source.
  • The Windows commands give another person a reproducible path.
  • Save README.md.
  • Confirm that the editor shows the project title and the Windows reproduction commands.

The first half of your case study now explains what the experiment proves. It also shows how another Windows user can run it.

  • Place the cursor below the closing code fence in README.md.
  • Complete the case study by pasting the remaining sections:
The run prints both evaluation reports, writes `results.md`, and opens an interactive prompt.

## Decision Policy

The model returns a predicted label and a top model score. Scores below `0.80` route to `HUMAN REVIEW`. Higher-scoring ham predictions route to `ALLOW`, while higher-scoring spam predictions route to `BLOCK`.

This threshold is a demonstration policy, not a claim that the model is calibrated for production decisions.

## Evidence

See [`results.md`](results.md) for the reproducible split policy, baseline comparison, confusion matrices, and observed error examples.

## Non-Goals and Limitations

- This is a tested prototype, not a production spam filter.
- The public dataset is historical and may not represent current messaging patterns.
- No private messages or personal data should be added to the repository.
- A production system would need current representative data, privacy review, probability calibration, drift monitoring, abuse testing, and a validated cost model.

## Decision Log

- Baseline: A keyword rule establishes a transparent lower bound and creates visible failure cases.
- Model: TF-IDF plus logistic regression provides a reproducible local text-classification pipeline and probability estimates.
- Escalation: Low top scores go to a person instead of forcing every message into an automated action.
- Scope: Deployment, live message ingestion, and automatic enforcement are intentionally excluded so the evidence remains inspectable.

## Citation

Almeida, T. and Hidalgo, J. (2011). SMS Spam Collection. UCI Machine Learning Repository. https://doi.org/10.24432/C5CC84

Why document these limits?

The decision policy connects model output to an action. The limitations section keeps the evidence aligned with what the historical held-out test can support.

The citation credits the dataset source. The decision log records why each project choice exists.

  • Save README.md.
  • Confirm that the file now ends with the UCI citation.

README formatting looks broken?

Check that each heading begins with the same number of hash characters shown above. Confirm that both PowerShell code fences use three backticks.

Make sure the second chunk begins below the first chunk’s closing code fence. Help me repair the Markdown structure in my README.

✔️ Awesome, I've got everything!

Your README.md documents the problem, evidence, decision policy, limitations, and reproduction steps.

ⓧ I'd like to double check the full code

# Evidence-First SMS Spam Triage

A tested machine learning prototype that compares a deliberately limited keyword baseline with a TF-IDF logistic regression model, analyzes held-out errors, and sends uncertain predictions to human review.

## What This Project Proves

- A baseline can expose whether a more complex model adds value.
- Precision, recall, F1 score, and confusion matrices reveal different failure modes.
- False positives and false negatives should be inspected, not hidden behind one accuracy value.
- An ML prediction still needs an action policy and an escalation path.

## Dataset

The project downloads the UCI Machine Learning Repository SMS Spam Collection from:

https://archive.ics.uci.edu/dataset/228/sms%2Bspam%2Bcollection

The repository describes 5,574 labeled SMS messages. The downloaded dataset is excluded from Git because the script can retrieve it reproducibly from the official source.

## Reproduce the Experiment on Windows

```powershell
py -3 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python spam_triage.py
```

The run prints both evaluation reports, writes `results.md`, and opens an interactive prompt.

## Decision Policy

The model returns a predicted label and a top model score. Scores below `0.80` route to `HUMAN REVIEW`. Higher-scoring ham predictions route to `ALLOW`, while higher-scoring spam predictions route to `BLOCK`.

This threshold is a demonstration policy, not a claim that the model is calibrated for production decisions.

## Evidence

See [`results.md`](results.md) for the reproducible split policy, baseline comparison, confusion matrices, and observed error examples.

## Non-Goals and Limitations

- This is a tested prototype, not a production spam filter.
- The public dataset is historical and may not represent current messaging patterns.
- No private messages or personal data should be added to the repository.
- A production system would need current representative data, privacy review, probability calibration, drift monitoring, abuse testing, and a validated cost model.

## Decision Log

- Baseline: A keyword rule establishes a transparent lower bound and creates visible failure cases.
- Model: TF-IDF plus logistic regression provides a reproducible local text-classification pipeline and probability estimates.
- Escalation: Low top scores go to a person instead of forcing every message into an automated action.
- Scope: Deployment, live message ingestion, and automatic enforcement are intentionally excluded so the evidence remains inspectable.

## Citation

Almeida, T. and Hidalgo, J. (2011). SMS Spam Collection. UCI Machine Learning Repository. https://doi.org/10.24432/C5CC84

Two small project files make the case study reproducible without publishing the virtual environment or downloaded dataset.

  • At the top of the file sidebar, select the new-file control.
  • Name the file requirements.txt.
  • Pin the project dependency by entering this line:
scikit-learn==1.9.1

Why pin the dependency?

The exact package version records the environment used for the evidence. Another person can install the same direct dependency before rerunning the experiment.

  • Save requirements.txt.
  • Confirm that requirements.txt appears beside spam_triage.py in the file sidebar.

Dependency file missing?

Check that the filename ends with .txt. Confirm that you created it inside evidence-first-spam-triage.

Help me check whether requirements.txt is in the correct project folder.

  • At the top of the file sidebar, select the new-file control.
  • Name the file .gitignore.
  • Exclude generated and local-only files by pasting these patterns:
.venv/
__pycache__/
data/
*.zip

What stays out of the repository?

  • The local virtual environment stays on your computer.
  • Python cache files stay out of the case study.
  • The downloaded dataset stays out because the script retrieves it reproducibly.
  • Downloaded ZIP archives stay out of the published files.
  • Save .gitignore.
  • Confirm that .gitignore appears in the file sidebar with four exclusion patterns.

Hidden file not visible?

Confirm that the filename starts with a period. Make sure Visual Studio Code did not add a .txt extension.

Help me check why my .gitignore file is not visible or is not excluding the project data.

✔️ Awesome, I've got everything!

Your local folder now contains the five files needed for publication. The generated dataset remains local.

ⓧ I'd like to double check the full code

Compare these two project files with yours.

scikit-learn==1.9.1

.venv/
__pycache__/
data/
*.zip
Publish the evidence on GitHub

A public GitHub repository gives reviewers one place to inspect the code and measured results. The browser upload path also avoids adding an unconfirmed command-line dependency.

Why use the browser upload?

Your GitHub account is already available for this project. Uploading through the browser publishes the evidence without requiring a separate local Git installation.

  • Open GitHub in your browser.
  • Sign in with your existing GitHub account.
  • Select the upper-right menu.
  • Choose New repository.
  • Choose your GitHub account as the repository owner.
  • Enter evidence-first-spam-triage in the repository name field.
  • Select Public as the visibility.
  • Leave the optional initialization choices unselected.
  • Click Create repository.

Your empty public repository is ready to receive the local evidence files.

  • Click Add file.
  • Select Upload files.
  • Choose spam_triage.py, README.md, requirements.txt, .gitignore, and results.md from the local project folder.

You should see all five filenames in the upload list. The data folder and .venv stay on your computer.

Before you publish the files, what evidence do you expect the repository home page to show?

  • Click Propose changes to complete the upload.

You should see README.md rendered on the repository home page. The file list should include results.md with the other four published files.

Repository files incomplete?

Return to the repository home page and compare the file list with the five filenames above. Upload any missing local file through Add file.

If the README does not render, confirm that its filename is exactly README.md. Help me diagnose why my GitHub repository does not show the complete case study.

That completes the evidence-first case study. Your repository now shows a reproducible model comparison with an explicit human review route and documented limits.

Secret mission

Defend a Cost-Aware Review Threshold

Test six messages against three candidate review thresholds. Publish a cost-aware decision log that uses observed model scores to defend when the system should automate a route or involve a person.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the project, archive the repository, or delete both copies. Your local files and this GitHub Free repository cost $0 to keep.

Resources you used:

  • Local resource: the evidence-first-spam-triage folder. It contains .venv, data/, spam_triage.py, README.md, requirements.txt, .gitignore, results.md, and decision_log.md.
  • Remote resource: the public GitHub repository named evidence-first-spam-triage.

Keep everything running

No action needed. Choose this if you want to rerun the experiment or keep presenting its evidence.

  • Keep the evidence-first-spam-triage folder so the reproducible local experiment remains available.
  • Keep the public GitHub repository so your published evidence remains visible.

Pause - I'll come back to this later

Pausing ends the active local session. An archived repository preserves your evidence in a read-only state.

  • Close Visual Studio Code to end the active PowerShell session.
  • Return to the public evidence-first-spam-triage repository from earlier.
  • Click Settings.
  • Scroll to Danger Zone.
  • Choose Archive this repository.
  • Complete the confirmation shown by GitHub.

You have a clean stopping point. Your published evidence remains available in a read-only repository until you unarchive it.

Delete - I don't want to use this again

Deletion is the clean-slate option. Both the local folder and remote repository leave your normal workflow.

Deletion is destructive

Deleting both copies is a big step. Any evidence without another copy can be lost.

GitHub states that some deleted repositories can be restored within 90 days.

Delete the local copy:

  • Close Visual Studio Code to release the active .venv session.
  • Press the Windows key to open search.
  • Type File Explorer and press Enter.
  • Find the evidence-first-spam-triage folder in the location where you created it.
  • Select the evidence-first-spam-triage folder.
  • Press the Delete key.

You should no longer see the project folder in its original location. Its virtual environment, downloaded dataset, source files, and evidence files are removed with it.

Delete the remote repository:

  • Return to the public evidence-first-spam-triage repository from earlier.
  • Click Settings.
  • Select General.
  • Scroll to Danger Zone.
  • Choose Delete this repository.
  • Complete the repository-name confirmation with evidence-first-spam-triage.

The repository no longer appears among your active GitHub repositories. Your published project files are removed from their former repository page.

Nice Work!

Nice Work!

You did it! You built a reproducible machine learning prototype that tests a keyword baseline against an evaluated text classifier. You published its evidence in a public GitHub case study.

You've learned how to:

  • Built a reproducible experiment from 5,574 labeled messages in the UCI SMS Spam Collection. Protected the evaluation with a stratified held-out split.
  • Compared a keyword baseline with an evaluated TF-IDF logistic regression pipeline. Produced classification reports for both approaches. Inspected each confusion matrix. Reviewed concrete prediction errors.
  • Created a human review policy that routes uncertain messages to HUMAN REVIEW. Published a GitHub case study with reproducible setup instructions. Added measured evidence. Documented transparent limitations.
  • Secret Mission: Produced a cost-aware decision log from six or more observed model scores. Compared three review thresholds. Defended one threshold against explicit error costs. Documented the validation required before production use.

Ready to quiz yourself?