Fraud Detection Accuracy Trap

Build a fraud classifier that exposes accuracy traps and tunes safer decisions.

Introduction

30 Second Summary

Fraudulent payments are rare enough to disappear inside thousands of normal transactions. A reassuring accuracy score can therefore leave the costly cases completely uncaught.

In this project, you will build a local Jupyter notebook that exposes misleading accuracy in fraud detection. The completed analysis trains a logistic regression model before validating a tuned decision threshold on untouched test data.

What You'll Build

When you run the finished notebook, a visual report shows how an impressive accuracy score can conceal missed fraud.

By the end of this project, you'll have:

  • A class imbalance report that makes rare fraud visible across 90,000 training rows.
  • An honest model scorecard that exposes the all-legitimate baseline through fraud recall plus a confusion matrix.
  • A portfolio-ready HTML report that compares default versus tuned thresholds on 30,000 untouched test rows.
  • Secret Mission: Compare the original classifier with a class-weighted logistic regression model at separately validated thresholds.

Are there any prerequisites?

You'll need a Kaggle account plus a Windows computer with Visual Studio Code. Basic Python knowledge is enough because the guide checks your local setup.

Before We Start

Before the hands-on work begins, commit to an honest fraud-risk analysis that serves a real decision. Your chosen user plus the relative cost of missed fraud and false alarms will guide how you judge the model.

Prepare the Local Notebook Workspace

An honest fraud analysis depends on a repeatable workspace. Package conflicts or a mismatched notebook kernel can make the same cells behave differently.

You will connect Visual Studio Code to one project-specific Python environment. Your Jupyter notebook will use the same package versions throughout the analysis.

In this step, get ready to:
  • Verify the required VS Code extensions plus a supported Python runtime.
  • Build an isolated Python kernel for the fraud analysis.
  • Prepare a dataset-backed notebook whose import check succeeds.
Confirm your Windows tools

The Python extension for Visual Studio Code lets VS Code discover local Python interpreters. The Jupyter Extension for Visual Studio Code provides notebook editing plus kernel selection.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to open Visual Studio Code.
  • Select the Extensions icon in the Activity Bar.
  • Enter Python extension for Visual Studio Code in the extension search field.

You will see the Python extension published by Microsoft in the results.

  • Select the Microsoft Python extension.
  • Select Install if that option is available.

The extension page confirms when the Python extension is installed.

  • Return to the extension search field.
  • Enter Jupyter Extension for Visual Studio Code.

You will see the Jupyter extension published by Microsoft in the results.

  • Select the Microsoft Jupyter extension.
  • Select Install if that option is available.

Both extensions are now available for the local notebook workflow.

Are the Extensions Unavailable?

Check that Visual Studio Code can reach the internet. Search for each extension by its complete name.

Use help me install the Microsoft Python and Jupyter extensions in Visual Studio Code on Windows if the extension results remain unavailable.

The project folder keeps every notebook file under one known location. You will place it on your Desktop so it stays easy to find.

  • Select File from the top menu.
  • Select Open Folder.
  • Select Desktop in the Windows folder picker.
  • Select New folder.
  • Enter fraud-detection as the folder name.
  • Press Enter to create the folder.

You will see fraud-detection inside your Desktop folder.

A workspace trust prompt may appear when the folder opens. You created this folder yourself, so approving it enables VS Code features inside this workspace.

  • Select the new fraud-detection folder.
  • Select Select Folder.
  • Confirm that you trust the folder if VS Code asks.

The Explorer sidebar now shows fraud-detection as the open folder.

  • Select Terminal from the top menu.
  • Select New Terminal.
  • Select PowerShell from the terminal profile menu if PowerShell is not already active.
  • Check the installed Python version by running this command:
python --version

What Does This Command Check?

The command asks the active Python installation to report its version. This project needs Python 3.11 or newer because the pinned data packages require that runtime.

✔️ I see version 3.11 or higher

Your Python runtime supports the pinned project packages. Keep the PowerShell terminal open for the environment commands that come next.

ⓧ I see an older version

The reported runtime is below the project minimum. Install the current stable Windows release before creating the environment.

Windows may request permission when the installer starts. Approving that request allows the installer to add Python to your computer.

  • Run the downloaded Python installer.
  • Approve the Windows permission request if it appears.
  • Complete the guided installation.
  • Close Visual Studio Code.

The newer Python runtime is now available to applications opened after the installation.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to reopen Visual Studio Code.
  • Return to fraud-detection from the recent folders list.
  • Select Terminal from the top menu.
  • Select New Terminal.
  • Confirm the upgraded runtime by running this command:
python --version

What Confirms the Upgrade?

PowerShell now reports Python 3.11 or newer. The upgraded interpreter can create the project environment.

ⓧ Command not found

PowerShell cannot currently find a Python installation. Install the current stable Windows release before creating the environment.

Windows may request permission when the installer starts. Approving that request allows the installer to add Python to your computer.

  • Run the downloaded Python installer.
  • Approve the Windows permission request if it appears.
  • Complete the guided installation.
  • Close Visual Studio Code.

Python is now installed for applications opened after the installation.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to reopen Visual Studio Code.
  • Return to fraud-detection from the recent folders list.
  • Select Terminal from the top menu.
  • Select New Terminal.
  • Confirm the installation by running this command:
python --version

What Confirms the Installation?

PowerShell now reports Python 3.11 or newer. The installed interpreter can create the project environment.

Still Seeing the Previous Python Result?

  • Close every open PowerShell terminal in VS Code.
  • Reopen Visual Studio Code after the Python installation finishes.
  • Run the version check from a new PowerShell terminal.

Use help me fix the Python version reported by PowerShell in Visual Studio Code if the result remains older or missing.

Build the project environment

A virtual environment stores this project's packages separately from other Python projects. Version pins make the notebook reproducible when you return to it later.

  • Select the New File control in the Explorer sidebar.
  • Enter requirements.txt as the file name.

The new requirements.txt file opens inside fraud-detection.

  • Pin the project dependencies by pasting this content into requirements.txt:
pandas==3.0.6
scikit-learn==1.9.1
matplotlib==3.11.2
ipykernel==7.3.0

What Do These Pins Control?

  • Pandas version 3.0.6 loads the CSV datasets into tables.
  • The scikit-learn package at version 1.9.1 provides preprocessing plus model evaluation tools.
  • Matplotlib version 3.11.2 draws the class distribution plus later evaluation plots.
  • ipykernel version 7.3.0 connects the virtual environment to the notebook kernel picker.
  • Save requirements.txt.
  • Confirm that the editor tab no longer shows an unsaved-change indicator.

✔️ Awesome, I've got everything!

Your saved requirements file now pins all four project packages.

ⓧ I'd like to double check the full code

pandas==3.0.6
scikit-learn==1.9.1
matplotlib==3.11.2
ipykernel==7.3.0

What Should Match?

Compare all four package names plus their version pins. The order should match the reference file.

  • Create the virtual environment inside fraud-detection by running this command in PowerShell:
python -m venv .venv

What Does This Command Create?

Python creates a .venv directory inside fraud-detection. That directory contains the isolated interpreter that runs this notebook.

  • Refresh the Explorer sidebar after the command finishes.

You will see a new .venv directory under fraud-detection.

Is the Virtual Environment Missing?

  • Confirm that the PowerShell prompt points to the fraud-detection folder.
  • Confirm that the Python version check reports 3.11 or newer.
  • Refresh the VS Code Explorer sidebar after the command completes.

Use help me diagnose why .venv was not created if the directory remains missing.

The pinned packages must be installed through the interpreter inside .venv. This keeps the installation attached to the correct project environment.

  • Install the pinned packages into .venv by running this command:
.\.venv\Scripts\python.exe -m pip install -r requirements.txt

What Does This Installation Use?

The command invokes the Python interpreter stored inside .venv. The requirements file supplies the exact package versions to install.

PowerShell returns to its prompt after the packages finish installing. The project environment now contains every library imported by the first notebook cell.

Does the Package Installation Fail?

  • Confirm that requirements.txt is saved inside fraud-detection.
  • Confirm that .venv appears in the VS Code Explorer sidebar.
  • Confirm that your internet connection can reach the package registry.

Use help me diagnose my requirements installation if the installation still fails.

Jupyter identifies kernels by an internal name plus a display name. Registering the environment makes it selectable from the notebook interface.

  • Register the project kernel by running this command:
.\.venv\Scripts\python.exe -m ipykernel install --user --name fraud-detection --display-name "Python (fraud-detection)"

What Does Kernel Registration Store?

The internal kernel name is fraud-detection. The notebook menu presents the display name Python (fraud-detection).

PowerShell reports that the kernel installation completed. VS Code can now connect a notebook to the project environment.

Is the Kernel Registration Failing?

  • Confirm that the package installation completed before registering the kernel.
  • Confirm that the command starts with the interpreter inside .venv.
  • Run the registration command from the fraud-detection PowerShell terminal.

Use help me register my fraud-detection ipykernel if registration remains unsuccessful.

Complete the runnable notebook

The notebook needs three local CSV files before later cells can analyze fraud. Keeping them under data gives every notebook cell stable relative paths.

  • Select the New Folder control in the VS Code Explorer sidebar.
  • Enter data as the folder name.

The Explorer sidebar now shows data inside fraud-detection.

The Kaggle dataset provides separate training rows, test rows, plus answer labels. You will keep those original file names so the notebook can load them directly.

  • Open the Imbalanced Fraud Sandbox page in your browser.
  • Sign in with your Kaggle account if prompted.
  • Use the dataset page's download control.
  • Wait for the browser to finish downloading the archive.

Your browser shows the completed dataset download when the archive is ready.

  • Use the completed download entry to reveal the archive in Windows File Explorer.
  • Extract the downloaded archive.

The extracted files include train.csv, test.csv, plus solution.csv.

  • Select train.csv, test.csv, plus solution.csv in File Explorer.
  • Copy the selected files.
  • Select Desktop in the File Explorer sidebar.
  • Open the fraud-detection folder.
  • Open the data folder.
  • Paste the selected files into data.

The project now contains data/train.csv, data/test.csv, plus data/solution.csv.

  • Return to Visual Studio Code.
  • Select View from the top menu.
  • Select Command Palette.
  • Enter Jupyter: Create New Jupyter Notebook.
  • Select the matching command.

VS Code opens a new unsaved notebook with an empty code cell.

  • Select File from the top menu.
  • Select Save As.
  • Choose the fraud-detection folder as the destination.
  • Enter fraud_detection.ipynb as the file name.
  • Select Save.

The Explorer sidebar now lists fraud_detection.ipynb inside fraud-detection.

  • Select the kernel picker at the top right of the notebook.
  • Select Python (fraud-detection) from the available kernels.

The notebook header now shows Python (fraud-detection) as the active kernel.

  • Select the first code cell.
  • Add the environment check plus project imports by pasting this code:
# Cell 1: Environment check and imports
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    precision_recall_curve,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

print("Environment ready")

What Does This Cell Check?

  • The first import group loads the table tools plus plotting interface used by the analysis.
  • The scikit-learn imports load preprocessing, model training, dataset splitting, plus evaluation tools.
  • The final print statement provides one visible signal after every import succeeds.
  • Save fraud_detection.ipynb.
  • Confirm that fraud_detection.ipynb remains visible in the Explorer sidebar.
  • Confirm that data/train.csv appears under the data folder.
  • Confirm that data/test.csv appears under the data folder.
  • Confirm that data/solution.csv appears under the data folder.

Before you run the first cell, do you expect every import to resolve inside the selected kernel?

  • Run the first cell using its run control.

You will see Environment ready beneath the cell without an import error.

Your local notebook environment now works from kernel selection through package imports. Every later fraud result can build on this reproducible starting point.

Does the First Cell Fail?

  • Confirm that the notebook header shows Python (fraud-detection).
  • Confirm that the package installation completed inside .venv.
  • Restart the selected notebook kernel after changing the environment.

Use help me diagnose my notebook import error if the cell still fails.

✔️ Awesome, I've got everything!

Your notebook is saved with the project kernel selected. The first cell now confirms that the environment is ready.

ⓧ I'd like to double check the full code

# Cell 1: Environment check and imports
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    precision_recall_curve,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

print("Environment ready")

What Should Match?

Compare the imports plus the final print statement in order. This is the complete notebook content at the end of this step.

Your environment is ready for real transaction data. Next, you will load the three CSV files and reveal how strongly legitimate transactions outnumber fraud.

Reveal the Fraud Imbalance

Your Jupyter notebook in Visual Studio Code already runs with the project kernel. Its first cell confirms the environment with Environment ready.

Fraud is rare in this dataset. This class imbalance can let the majority class dominate a headline accuracy score.

In this step, get ready to:
  • Load the three local fraud datasets.
  • Validate the dataset structure.
  • Visualize the minority fraud class with a bar chart.
Load and validate the datasets

Pandas loads each local CSV file into a table called a DataFrame. A schema check confirms that the expected features exist before the analysis continues.

  • Use the notebook toolbar to add a Python cell below the environment-check cell.
  • Load the datasets and define their expected columns by pasting this code into the new cell:
# Cell 2: Load and validate the Kaggle files
train = pd.read_csv("data/train.csv")
test = pd.read_csv("data/test.csv")
solution = pd.read_csv("data/solution.csv")

target = "is_fraud"
numeric_features = [
    "amount",
    "hour",
    "account_age_days",
    "num_prior_txns",
    "distance_from_home_km",
    "is_foreign",
    "device_change",
]
categorical_features = ["merchant_category"]
feature_columns = numeric_features + categorical_features

What Does This Code Do?

  • The three pd.read_csv() calls load the training rows, test rows, and hidden test labels from the data folder.
  • The target variable identifies is_fraud as the label the model eventually predicts.
  • The numeric_features list identifies the seven numeric inputs.
  • The categorical_features list keeps merchant_category separate for later encoding.
  • The feature_columns list combines all eight model inputs in a consistent order.
  • Save fraud_detection.ipynb.
  • Run the new data-loading cell.

You will see the cell finish without a traceback. Its completed execution marker confirms that all three files loaded from the expected paths.

Cannot Load a CSV File?

  • Confirm that train.csv, test.csv, and solution.csv appear inside the data folder.
  • Check that the notebook remains inside the fraud-detection folder.
  • Confirm that each filename uses lowercase letters.

Help me fix the dataset-loading cell.

The files now exist as DataFrames in memory. The next part of the same cell checks their columns before showing you a preview.

  • Place the cursor at the end of the data-loading cell.
  • Append the validation checks and visible previews by pasting this code:
missing_train = set(feature_columns + [target]) - set(train.columns)
missing_test = set(feature_columns) - set(test.columns)
missing_solution = {target} - set(solution.columns)

if missing_train or missing_test or missing_solution:
    raise ValueError(
        f"Missing columns. train={missing_train}, test={missing_test}, "
        f"solution={missing_solution}"
    )

if len(test) != len(solution):
    raise ValueError("test.csv and solution.csv must have the same number of rows")

print("Train shape:", train.shape)
print("Test shape:", test.shape)
print("Solution shape:", solution.shape)
print(train.head())

How Does the Validation Work?

  • The three missing sets compare the expected columns with the columns found in each file.
  • The first check stops the notebook when a required feature or target is absent.
  • The row-alignment check confirms that every test transaction has one solution label.
  • The shape outputs reveal the row count plus the column count for each DataFrame.
  • The train.head() output shows the first five training rows.
  • Save fraud_detection.ipynb.
  • Run the completed data-loading cell.

You will see 90,000 training rows. You will also see 30,000 rows in each test file.

Good progress. Your notebook has now proved that the three datasets align with the expected fraud-analysis structure.

Does a Validation Check Stop the Cell?

  • Check that you downloaded all three files from the Imbalanced Fraud Sandbox.
  • Confirm that the files were extracted before you placed them in the data folder.
  • Replace any renamed copy with the original Kaggle file.

Help me diagnose the dataset validation failure.

Measure the minority class

A class count shows how many training rows belong to each target value. The fraud rate turns the minority count into a percentage that is easier to compare.

  • Use the notebook toolbar to add another Python cell below the data-loading cell.
  • Calculate the class counts and fraud rate by pasting this code into the new cell:
# Cell 3: Explore the class imbalance
class_counts = train[target].value_counts().sort_index()
legitimate_count = int((train[target] == 0).sum())
fraud_count = int((train[target] == 1).sum())
fraud_rate = train[target].mean()

print("Class counts:")
print(class_counts)
print(f"Fraud rate: {fraud_rate:.2%}")

What Do These Values Measure?

  • The class_counts Series shows how often each target value occurs.
  • The legitimate_count variable counts rows where is_fraud equals 0.
  • The fraud_count variable counts rows where is_fraud equals 1.
  • The fraud_rate variable stores the average of the binary target values.
  • The percentage format makes the minority share visible in the output.
  • Save fraud_detection.ipynb.
  • Run the class-count cell.

You will see counts for legitimate transactions plus fraudulent transactions. The reported fraud rate is about 1.8%.

Why Accuracy Is Already Suspicious

A model could label every transaction as legitimate and still agree with roughly 98.2% of the rows. That score would hide the fact that the model catches no fraud.

Do the Counts Look Wrong?

  • Run the data-loading cell before running the class-count cell.
  • Confirm that target contains is_fraud.
  • Check that you are using the downloaded data/train.csv file.

Help me investigate unexpected fraud counts.

Plot the imbalance and record the warning

Matplotlib turns the two class counts into a visual comparison. The height gap makes the minority class harder to overlook than a percentage alone.

  • Place the cursor below the fraud-rate output code in the same cell.
  • Append the class-imbalance chart by pasting this code:
plt.figure(figsize=(7, 4))
plt.bar(
    ["Legitimate", "Fraud"],
    [legitimate_count, fraud_count],
    color=["#4C78A8", "#E45756"],
)
plt.title("Training Transactions by Class")
plt.ylabel("Transaction count")
plt.tight_layout()
plt.show()

What Does the Chart Show?

  • The plt.bar() call compares legitimate transactions with fraudulent transactions.
  • The count variables keep the chart tied to the values calculated directly from train.csv.
  • The title identifies the plotted population as the training dataset.
  • The final plt.show() call renders the chart inside the notebook.

Before you run the cell, which bar do you expect to dominate the chart?

  • Save fraud_detection.ipynb.
  • Run the completed class-imbalance cell.

You will see a bar chart titled Training Transactions by Class. The fraud bar is a small minority beside the legitimate bar.

That is your first visual proof of the accuracy trap. A headline score can look strong while the minority class receives almost no useful attention.

Missing the Bar Chart?

  • Run the complete class-imbalance cell from its first line.
  • Confirm that legitimate_count plus fraud_count were calculated above the chart code.
  • Check that the final line contains plt.show().

Help me restore the class-imbalance chart.

  • Use the notebook toolbar to add a Markdown cell below the chart cell.
  • Write one or two sentences explaining why the small fraud minority makes accuracy alone suspicious.
  • Save fraud_detection.ipynb.

✔️ Awesome, I've got everything!

Your three Python cells now load the data, validate its structure, calculate the fraud rate, and display the class-imbalance chart.

ⓧ I'd like to double check the full code

Compare the three Python cells in fraud_detection.ipynb with this cumulative code. Keep your Markdown observation directly below these cells.

# Cell 1: Environment check and imports
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    precision_recall_curve,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

print("Environment ready")

# Cell 2: Load and validate the Kaggle files
train = pd.read_csv("data/train.csv")
test = pd.read_csv("data/test.csv")
solution = pd.read_csv("data/solution.csv")

target = "is_fraud"
numeric_features = [
    "amount",
    "hour",
    "account_age_days",
    "num_prior_txns",
    "distance_from_home_km",
    "is_foreign",
    "device_change",
]
categorical_features = ["merchant_category"]
feature_columns = numeric_features + categorical_features

missing_train = set(feature_columns + [target]) - set(train.columns)
missing_test = set(feature_columns) - set(test.columns)
missing_solution = {target} - set(solution.columns)

if missing_train or missing_test or missing_solution:
    raise ValueError(
        f"Missing columns. train={missing_train}, test={missing_test}, "
        f"solution={missing_solution}"
    )

if len(test) != len(solution):
    raise ValueError("test.csv and solution.csv must have the same number of rows")

print("Train shape:", train.shape)
print("Test shape:", test.shape)
print("Solution shape:", solution.shape)
print(train.head())

# Cell 3: Explore the class imbalance
class_counts = train[target].value_counts().sort_index()
legitimate_count = int((train[target] == 0).sum())
fraud_count = int((train[target] == 1).sum())
fraud_rate = train[target].mean()

print("Class counts:")
print(class_counts)
print(f"Fraud rate: {fraud_rate:.2%}")

plt.figure(figsize=(7, 4))
plt.bar(
    ["Legitimate", "Fraud"],
    [legitimate_count, fraud_count],
    color=["#4C78A8", "#E45756"],
)
plt.title("Training Transactions by Class")
plt.ylabel("Transaction count")
plt.tight_layout()
plt.show()

You have exposed the imbalance that makes accuracy dangerous. Next, you will build the all-legitimate baseline that turns this warning into measurable evidence.

Build the Misleading Accuracy Baseline

Your class imbalance chart exposed how rare fraud is in the training data. Now you will test how easily that imbalance can flatter a model.

A baseline gives later models a minimum standard to beat. The danger is an all-legitimate prediction that earns high accuracy while missing every fraud case.

In this step, get ready to:
  • Create a reproducible stratified training-validation split.
  • Define reusable evaluation helpers.
  • Expose the all-legitimate baseline with fraud-focused evidence.
Create a stratified validation split

A stratified split keeps the fraud proportion similar across the training and validation sets. A fixed random state makes the same rows land in each set whenever you rerun the notebook.

  • Return to fraud_detection.ipynb in Visual Studio Code.
  • Add a code cell below the class-imbalance chart cell.
  • Create the model inputs plus the reproducible split by entering this code:
# Cell 4: Create a validation split and evaluation helpers
X = train[feature_columns]
y = train[target].astype(int)

X_train, X_valid, y_train, y_valid = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y,
)

What Does This Code Do?

  • The X table contains the eight model features selected by feature_columns.
  • The y series contains integer fraud labels from is_fraud.
  • The test_size=0.2 setting reserves 20 percent of the labeled rows for validation.
  • The stratify=y setting preserves the class proportions in both sets.
  • The random_state=42 setting makes the split reproducible.
  • Save fraud_detection.ipynb.
  • Run the new code cell with the run icon beside it.

The cell finishes without an exception. Your notebook now holds separate training and validation variables.

Does the Split Cell Fail?

  • Run Cells 1 through 3 again if the notebook does not know train or feature_columns.
  • Compare feature_columns with its earlier definition if the notebook reports a missing feature.

Help me fix the training-validation split in my fraud notebook.

Define consistent evaluation helpers

Every model needs the same scoring process for a fair comparison. The helpers use scikit-learn metrics to evaluate predictions now and select a decision threshold later.

  • Add the evaluation function directly below the split code in the same cell:
def evaluate_scores(name, y_true, scores, threshold):
    predictions = (scores >= threshold).astype(int)
    precision = precision_score(y_true, predictions, zero_division=0)
    recall = recall_score(y_true, predictions, zero_division=0)
    denominator = precision + recall
    f1 = 0.0 if denominator == 0 else 2 * precision * recall / denominator

    metrics = {
        "model": name,
        "threshold": float(threshold),
        "accuracy": accuracy_score(y_true, predictions),
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "average_precision": average_precision_score(y_true, scores),
    }
    return metrics, predictions

How Does the Evaluation Work?

  • The threshold converts each risk score into a binary prediction.
  • Precision measures how many flagged transactions are truly fraudulent.
  • Recall measures how many fraud cases the model catches.
  • F1 combines precision with recall into one comparison score.
  • Average precision summarizes how well the scores rank fraud across the precision-recall trade-off.
  • The function returns the metric values.
  • It also returns the binary predictions for later inspection.
  • Save fraud_detection.ipynb.
  • Run Cell 4 with the run icon beside it.

The cell completes without an exception. The evaluate_scores() helper is now available to later cells.

Is the Evaluation Function Rejected?

  • Match the indentation inside evaluate_scores() with the reference code.
  • Check each closing brace in the metrics dictionary.

Help me debug the evaluate_scores function.

  • Add the threshold-selection function directly below evaluate_scores() in the same cell:
def choose_f1_threshold(y_true, scores):
    precision, recall, thresholds = precision_recall_curve(y_true, scores)
    f1_values = []

    for current_precision, current_recall in zip(precision[:-1], recall[:-1]):
        denominator = current_precision + current_recall
        score = (
            0.0
            if denominator == 0
            else 2 * current_precision * current_recall / denominator
        )
        f1_values.append(score)

    best_index = max(range(len(f1_values)), key=f1_values.__getitem__)
    return float(thresholds[best_index]), precision, recall

How Does Threshold Selection Work?

  • The precision-recall curve supplies precision plus recall values across possible thresholds.
  • The loop calculates an F1 score for each available threshold.
  • The best_index value identifies the highest F1 score.
  • The function returns the selected threshold.
  • It also returns the curve values for later plotting.
  • Save fraud_detection.ipynb.
  • Run Cell 4 with the run icon beside it.

The cell completes without an exception. Both evaluation helpers are now ready for the baseline and later classifiers.

Does the Threshold Helper Fail?

  • Check the indentation of the calculation inside the for loop.
  • Confirm the final return line remains inside choose_f1_threshold().

Help me debug the threshold-selection helper.

Expose the accuracy trap

The all-legitimate baseline gives every validation row a fraud score of zero. Its output tests whether one impressive metric can conceal failure on the minority class.

  • Add a code cell below Cell 4.
  • Create the baseline plus its labeled output by entering this code:
# Cell 5: Experience the accuracy trap
baseline_scores = pd.Series(0.0, index=y_valid.index)
baseline_metrics, baseline_predictions = evaluate_scores(
    "All-legitimate baseline",
    y_valid,
    baseline_scores,
    threshold=0.5,
)

print(pd.DataFrame([baseline_metrics]).round(4))
print(
    pd.DataFrame(
        confusion_matrix(y_valid, baseline_predictions, labels=[0, 1]),
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

What Does the Baseline Cell Do?

  • The pandas Series assigns a score of 0.0 to every validation row.
  • The 0.5 threshold converts those scores into binary predictions.
  • The first output displays the shared metric table.
  • The second output labels the actual classes by row.
  • It labels the predicted classes by column.
  • Save fraud_detection.ipynb.

Before you run this cell, do you expect the headline accuracy to agree with the fraud-focused metrics?

  • Run Cell 5 with the run icon beside it.

The table shows high accuracy. Its recall value is 0.0.

The Actual fraud row contains zero under Predicted fraud. The baseline catches no fraud cases.

You caught the trap: the baseline looks successful in one column while failing the class this project needs to find.

How Do You Read This Failure?

A confusion matrix separates correct predictions from mistakes for each class. The bottom-left cell counts fraud cases predicted as legitimate.

The bottom-right cell counts correctly detected fraud cases. A zero there proves the high accuracy is operationally useless for catching fraud.

Missing the Baseline Output?

  • Run Cell 4 again if y_valid or evaluate_scores() is unavailable.
  • Confirm the baseline code sits in a separate code cell below Cell 4.

Help me restore the baseline metric table and confusion matrix.

✔️ Awesome, I've got everything!

Your notebook now contains the data exploration, reproducible split, evaluation helpers, and all-legitimate baseline.

ⓧ I'd like to double check the full code

  • Compare Cells 1 through 5 in fraud_detection.ipynb with this cumulative reference:
# Cell 1: Environment check and imports
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    precision_recall_curve,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

print("Environment ready")

# Cell 2: Load and validate the Kaggle files
train = pd.read_csv("data/train.csv")
test = pd.read_csv("data/test.csv")
solution = pd.read_csv("data/solution.csv")

target = "is_fraud"
numeric_features = [
    "amount",
    "hour",
    "account_age_days",
    "num_prior_txns",
    "distance_from_home_km",
    "is_foreign",
    "device_change",
]
categorical_features = ["merchant_category"]
feature_columns = numeric_features + categorical_features

missing_train = set(feature_columns + [target]) - set(train.columns)
missing_test = set(feature_columns) - set(test.columns)
missing_solution = {target} - set(solution.columns)

if missing_train or missing_test or missing_solution:
    raise ValueError(
        f"Missing columns. train={missing_train}, test={missing_test}, "
        f"solution={missing_solution}"
    )

if len(test) != len(solution):
    raise ValueError("test.csv and solution.csv must have the same number of rows")

print("Train shape:", train.shape)
print("Test shape:", test.shape)
print("Solution shape:", solution.shape)
print(train.head())

# Cell 3: Explore the class imbalance
class_counts = train[target].value_counts().sort_index()
legitimate_count = int((train[target] == 0).sum())
fraud_count = int((train[target] == 1).sum())
fraud_rate = train[target].mean()

print("Class counts:")
print(class_counts)
print(f"Fraud rate: {fraud_rate:.2%}")

plt.figure(figsize=(7, 4))
plt.bar(
    ["Legitimate", "Fraud"],
    [legitimate_count, fraud_count],
    color=["#4C78A8", "#E45756"],
)
plt.title("Training Transactions by Class")
plt.ylabel("Transaction count")
plt.tight_layout()
plt.show()

# Cell 4: Create a validation split and evaluation helpers
X = train[feature_columns]
y = train[target].astype(int)

X_train, X_valid, y_train, y_valid = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y,
)


def evaluate_scores(name, y_true, scores, threshold):
    predictions = (scores >= threshold).astype(int)
    precision = precision_score(y_true, predictions, zero_division=0)
    recall = recall_score(y_true, predictions, zero_division=0)
    denominator = precision + recall
    f1 = 0.0 if denominator == 0 else 2 * precision * recall / denominator

    metrics = {
        "model": name,
        "threshold": float(threshold),
        "accuracy": accuracy_score(y_true, predictions),
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "average_precision": average_precision_score(y_true, scores),
    }
    return metrics, predictions


def choose_f1_threshold(y_true, scores):
    precision, recall, thresholds = precision_recall_curve(y_true, scores)
    f1_values = []

    for current_precision, current_recall in zip(precision[:-1], recall[:-1]):
        denominator = current_precision + current_recall
        score = (
            0.0
            if denominator == 0
            else 2 * current_precision * current_recall / denominator
        )
        f1_values.append(score)

    best_index = max(range(len(f1_values)), key=f1_values.__getitem__)
    return float(thresholds[best_index]), precision, recall

# Cell 5: Experience the accuracy trap
baseline_scores = pd.Series(0.0, index=y_valid.index)
baseline_metrics, baseline_predictions = evaluate_scores(
    "All-legitimate baseline",
    y_valid,
    baseline_scores,
    threshold=0.5,
)

print(pd.DataFrame([baseline_metrics]).round(4))
print(
    pd.DataFrame(
        confusion_matrix(y_valid, baseline_predictions, labels=[0, 1]),
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

How to Use This Reference

The # Cell comments mark the notebook cell boundaries. The function spacing keeps each helper inside Cell 4.

Your baseline has exposed the accuracy trap with evidence. Next, you will train a classifier that ranks fraud risk while keeping preprocessing isolated from the validation data.

Train an Interpretable Fraud Model

Your all-legitimate baseline produced high accuracy. Its fraud recall stayed at zero.

A logistic regression model can rank each transaction by fraud risk. A preprocessing pipeline keeps every learned transformation inside the training process.

In this step, get ready to:
  • Preprocess the numeric and categorical transaction features.
  • Fit logistic regression using only the training split.
  • Evaluate validation predictions at the default threshold.
Preprocess mixed transaction features

The numeric columns need feature scaling so their different units do not distort the model. The categorical column needs one-hot encoding so each merchant category becomes a model-readable indicator.

  • Create Cell 6 directly below the all-legitimate baseline by entering this code:
# Cell 6: Build and fit the preprocessing and model pipeline
preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", StandardScaler(), numeric_features),
        (
            "categorical",
            OneHotEncoder(handle_unknown="ignore"),
            categorical_features,
        ),
    ]
)

What Does This Code Do?

  • The numeric transformer applies StandardScaler to the seven columns in numeric_features.
  • The categorical transformer applies OneHotEncoder to merchant_category.
  • The handle_unknown="ignore" setting lets the transformer process a category that did not appear during fitting.
  • The ColumnTransformer keeps both transformations together for the same transaction rows.
  • Save fraud_detection.ipynb.
  • Run Cell 6.

The execution indicator completes without an exception. Your notebook now holds the mixed-feature preprocessor.

Does the Preprocessor Cell Fail?

Confirm that Cell 1 ran successfully in the current kernel. That cell imports ColumnTransformer, StandardScaler plus OneHotEncoder.

Confirm that the earlier loading cell defines numeric_features plus categorical_features before Cell 6 runs.

Help me fix the preprocessing cell.

Fit the model on training data

The pipeline fits preprocessing from X_train before logistic regression learns from y_train. Validation rows pass through the fitted pipeline only when you request fraud probabilities.

  • Place the pipeline definition directly below preprocessor in Cell 6 by entering this code:
model = Pipeline(
    steps=[
        ("preprocess", preprocessor),
        ("classifier", LogisticRegression(max_iter=1000)),
    ]
)

model.fit(X_train, y_train)
validation_scores = model.predict_proba(X_valid)[:, 1]

How Does the Pipeline Protect Validation Data?

  • The preprocess step learns scaling values plus category columns from X_train during fitting.
  • The classifier step fits LogisticRegression on those transformed training rows.
  • The predict_proba call transforms X_valid with the fitted preprocessing steps.
  • The [:, 1] selection keeps the probability assigned to the fraud class.
  • Save fraud_detection.ipynb.
  • Run Cell 6.

The execution indicator completes without a fitting exception. The fitted model now produces one fraud-risk score for each validation row.

Does Model Fitting Fail?

Run the earlier cells in order if Python reports that a training variable is undefined. Cell 4 must create X_train, X_valid, y_train plus y_valid.

Check that model.fit receives the training split. Passing the validation split there would compromise the final check.

Help me diagnose the pipeline fit.

Evaluate the default threshold

A decision threshold converts fraud probabilities into binary predictions. The default 0.5 cutoff labels a transaction as fraud when its score reaches at least half.

  • Add the default-threshold evaluation directly below validation_scores in Cell 6 by entering this code:
default_validation_metrics, default_validation_predictions = evaluate_scores(
    "Logistic regression at 0.5",
    y_valid,
    validation_scores,
    threshold=0.5,
)

print(pd.DataFrame([default_validation_metrics]).round(4))
print(
    pd.DataFrame(
        confusion_matrix(y_valid, default_validation_predictions, labels=[0, 1]),
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

What Does This Evaluation Reveal?

  • The helper applies the 0.5 cutoff to every value in validation_scores.
  • The metrics table reports accuracy, precision, recall, F1 plus average precision.
  • The labeled confusion matrix separates correct legitimate predictions, false alarms, missed fraud plus caught fraud.
  • The validation split measures the fitted model without exposing the untouched test labels.

Before you run the completed cell, do you expect the classifier to catch fraud that the all-legitimate baseline missed?

  • Save fraud_detection.ipynb.
  • Run Cell 6.

You will see a metrics table with non-zero fraud recall plus average precision. You will also see a two-by-two validation confusion matrix at threshold 0.5.

The matrix contains caught fraud cases. It also exposes fraud cases that the default cutoff still misses.

Still Seeing Zero Fraud Recall?

Confirm that validation_scores comes from model.predict_proba(X_valid)[:, 1]. Using the first probability column evaluates the legitimate class.

Confirm that default_validation_predictions comes from evaluate_scores with validation_scores plus threshold 0.5.

Help me inspect zero recall at the default threshold.

✔️ Awesome, I've got everything!

Great work. Your saved notebook now fits a leakage-safe logistic regression pipeline plus evaluates its validation predictions at threshold 0.5.

ⓧ I'd like to double check the full code

Compare Cells 1 through 6 in fraud_detection.ipynb with this cumulative reference.

# Cell 1: Environment check and imports
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    precision_recall_curve,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

print("Environment ready")

# Cell 2: Load and validate the Kaggle files
train = pd.read_csv("data/train.csv")
test = pd.read_csv("data/test.csv")
solution = pd.read_csv("data/solution.csv")

target = "is_fraud"
numeric_features = [
    "amount",
    "hour",
    "account_age_days",
    "num_prior_txns",
    "distance_from_home_km",
    "is_foreign",
    "device_change",
]
categorical_features = ["merchant_category"]
feature_columns = numeric_features + categorical_features

missing_train = set(feature_columns + [target]) - set(train.columns)
missing_test = set(feature_columns) - set(test.columns)
missing_solution = {target} - set(solution.columns)

if missing_train or missing_test or missing_solution:
    raise ValueError(
        f"Missing columns. train={missing_train}, test={missing_test}, "
        f"solution={missing_solution}"
    )

if len(test) != len(solution):
    raise ValueError("test.csv and solution.csv must have the same number of rows")

print("Train shape:", train.shape)
print("Test shape:", test.shape)
print("Solution shape:", solution.shape)
print(train.head())

# Cell 3: Explore the class imbalance
class_counts = train[target].value_counts().sort_index()
legitimate_count = int((train[target] == 0).sum())
fraud_count = int((train[target] == 1).sum())
fraud_rate = train[target].mean()

print("Class counts:")
print(class_counts)
print(f"Fraud rate: {fraud_rate:.2%}")

plt.figure(figsize=(7, 4))
plt.bar(
    ["Legitimate", "Fraud"],
    [legitimate_count, fraud_count],
    color=["#4C78A8", "#E45756"],
)
plt.title("Training Transactions by Class")
plt.ylabel("Transaction count")
plt.tight_layout()
plt.show()

# Cell 4: Create a validation split and evaluation helpers
X = train[feature_columns]
y = train[target].astype(int)

X_train, X_valid, y_train, y_valid = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y,
)


def evaluate_scores(name, y_true, scores, threshold):
    predictions = (scores >= threshold).astype(int)
    precision = precision_score(y_true, predictions, zero_division=0)
    recall = recall_score(y_true, predictions, zero_division=0)
    denominator = precision + recall
    f1 = 0.0 if denominator == 0 else 2 * precision * recall / denominator

    metrics = {
        "model": name,
        "threshold": float(threshold),
        "accuracy": accuracy_score(y_true, predictions),
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "average_precision": average_precision_score(y_true, scores),
    }
    return metrics, predictions


def choose_f1_threshold(y_true, scores):
    precision, recall, thresholds = precision_recall_curve(y_true, scores)
    f1_values = []

    for current_precision, current_recall in zip(precision[:-1], recall[:-1]):
        denominator = current_precision + current_recall
        score = (
            0.0
            if denominator == 0
            else 2 * current_precision * current_recall / denominator
        )
        f1_values.append(score)

    best_index = max(range(len(f1_values)), key=f1_values.__getitem__)
    return float(thresholds[best_index]), precision, recall

# Cell 5: Experience the accuracy trap
baseline_scores = pd.Series(0.0, index=y_valid.index)
baseline_metrics, baseline_predictions = evaluate_scores(
    "All-legitimate baseline",
    y_valid,
    baseline_scores,
    threshold=0.5,
)

print(pd.DataFrame([baseline_metrics]).round(4))
print(
    pd.DataFrame(
        confusion_matrix(y_valid, baseline_predictions, labels=[0, 1]),
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

# Cell 6: Build and fit the preprocessing and model pipeline
preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", StandardScaler(), numeric_features),
        (
            "categorical",
            OneHotEncoder(handle_unknown="ignore"),
            categorical_features,
        ),
    ]
)

model = Pipeline(
    steps=[
        ("preprocess", preprocessor),
        ("classifier", LogisticRegression(max_iter=1000)),
    ]
)

model.fit(X_train, y_train)
validation_scores = model.predict_proba(X_valid)[:, 1]

default_validation_metrics, default_validation_predictions = evaluate_scores(
    "Logistic regression at 0.5",
    y_valid,
    validation_scores,
    threshold=0.5,
)

print(pd.DataFrame([default_validation_metrics]).round(4))
print(
    pd.DataFrame(
        confusion_matrix(y_valid, default_validation_predictions, labels=[0, 1]),
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

You have replaced the empty baseline with a classifier that catches fraud plus exposes the remaining misses. Next, you will choose a stronger operating threshold on validation data before testing it on untouched rows.

Tune the Threshold and Evaluate Test Data

Your fitted logistic regression pipeline already ranks each validation transaction by fraud risk. Its default 0.5 decision threshold still leaves fraud uncaught.

The next choice must come from validation data before the solution labels enter the workflow. This sequence prevents data leakage from the untouched test rows.

In this step, get ready to:
  • Select the highest-F1 threshold using validation results.
  • Compare the default threshold with the tuned threshold on test data.
  • Export the completed analysis as an HTML report.
Choose a threshold on validation data

The precision-recall curve provides candidate thresholds for different balances between caught fraud and false alarms. Your helper selects the candidate with the highest validation F1 score.

  • Add a new code cell below the existing model cell in fraud_detection.ipynb.
  • Paste the validation-threshold code into the new cell:
# Cell 7: Tune on validation data and evaluate untouched test data
best_threshold, curve_precision, curve_recall = choose_f1_threshold(
    y_valid,
    validation_scores,
)

tuned_validation_metrics, tuned_validation_predictions = evaluate_scores(
    "Logistic regression tuned",
    y_valid,
    validation_scores,
    threshold=best_threshold,
)

How Is the Threshold Chosen?

  • The choose_f1_threshold() helper searches the validation curve for the threshold with the highest F1 score.
  • The returned curve values stay available for the trade-off plot later in this step.
  • The evaluate_scores() helper measures the selected threshold against the same validation labels.
  • Run the new notebook cell.
  • Confirm the cell finishes without an error.

The completed cell creates best_threshold without reading solution.csv. Your operating point now comes entirely from validation evidence.

Does the Threshold Cell Fail?

Run the earlier helper cell if choose_f1_threshold() is unavailable. Run the fitted-model cell if validation_scores is unavailable.

Confirm that the notebook still uses the Python (fraud-detection) kernel.

Help me diagnose the validation-threshold cell.

The test labels can enter only after best_threshold is fixed. This keeps the final comparison independent from the threshold decision.

  • Place the cursor after the tuned_validation_metrics evaluation in the same cell.
  • Append the test-evaluation code below it:
X_test = test[feature_columns]
y_test = solution[target].astype(int)
test_scores = model.predict_proba(X_test)[:, 1]

default_test_metrics, default_test_predictions = evaluate_scores(
    "Test at 0.5",
    y_test,
    test_scores,
    threshold=0.5,
)
tuned_test_metrics, tuned_test_predictions = evaluate_scores(
    "Test at tuned threshold",
    y_test,
    test_scores,
    threshold=best_threshold,
)

comparison = pd.DataFrame(
    [baseline_metrics, default_validation_metrics, tuned_validation_metrics,
     default_test_metrics, tuned_test_metrics]
).set_index("model")

print(f"Validation-selected threshold: {best_threshold:.4f}")
print(comparison.round(4))

How Does This Protect the Test Set?

  • The test features come from test.csv after the validation threshold has been selected.
  • The true test labels come from solution.csv for the final evaluation.
  • Both test evaluations use the same model probabilities. Only the threshold changes.
  • The comparison table keeps the baseline plus the validation and test results together.

Before you run this cell, consider whether the tuned threshold will catch more fraud at the cost of more false alarms.

  • Save fraud_detection.ipynb.
  • Run the completed threshold-and-test cell.

You will see the validation-selected threshold followed by a five-row comparison table. The final two rows compare the default and tuned thresholds across the 30,000 test transactions.

Is the Test Comparison Missing?

Confirm that data/test.csv and data/solution.csv remain inside the fraud-detection folder. The earlier loading cell must also have completed successfully.

Run the notebook cells from the data-loading cell through the model cell if a required variable is unavailable.

Help me restore the test comparison.

Visualize the operating trade-off

A metric table summarizes each operating point. A curve makes the relationship between precision and recall visible across every candidate threshold.

  • Add a new code cell below the threshold-and-test cell.
  • Paste the plotting code into the new cell:
# Cell 8: Visualize the trade-off and report final confusion matrices
plt.figure(figsize=(7, 5))
plt.plot(curve_recall, curve_precision, label="Validation precision-recall curve")
plt.scatter(
    [default_validation_metrics["recall"]],
    [default_validation_metrics["precision"]],
    label="Threshold 0.5",
)
plt.scatter(
    [tuned_validation_metrics["recall"]],
    [tuned_validation_metrics["precision"]],
    label="Tuned threshold",
)
plt.xlabel("Recall")
plt.ylabel("Precision")
plt.title("Fraud Precision-Recall Trade-off")
plt.legend()
plt.tight_layout()
plt.show()

What Does the Plot Show?

  • The line shows the validation trade-off across the available thresholds.
  • The first marker identifies the default 0.5 operating point.
  • The second marker identifies the threshold selected by validation F1.
  • Run the new plotting cell.

You will see a chart titled Fraud Precision-Recall Trade-off. Two labeled points show where the default and tuned thresholds sit on the validation curve.

Is the Curve Missing?

Run the threshold cell first if curve_precision or curve_recall is unavailable. Confirm that the plotting cell ends with plt.show().

Help me display the precision-recall plot.

A confusion matrix turns the threshold trade-off into counts. Comparing both matrices reveals how many fraud cases are recovered and how many legitimate transactions become false alarms.

  • Place the cursor after plt.show() in the plotting cell.
  • Append the confusion-matrix code below it:
print("Default threshold test confusion matrix")
default_matrix = confusion_matrix(y_test, default_test_predictions, labels=[0, 1])
print(
    pd.DataFrame(
        default_matrix,
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

print("Tuned threshold test confusion matrix")
tuned_matrix = confusion_matrix(y_test, tuned_test_predictions, labels=[0, 1])
print(
    pd.DataFrame(
        tuned_matrix,
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

How Do the Matrices Help?

  • The default matrix counts test outcomes produced by the 0.5 threshold.
  • The tuned matrix counts outcomes produced by best_threshold.
  • The labeled rows and columns expose missed fraud plus false alarms directly.
  • Run the expanded plotting cell.

You will see the precision-recall plot followed by two labeled test confusion matrices. The Actual fraud row shows how each threshold divides fraud between missed and caught cases.

Are the Matrices Unavailable?

Run the test-evaluation cell if default_test_predictions or tuned_test_predictions is unavailable. Keep labels=[0, 1] in both matrix calls so their positions remain comparable.

Help me diagnose the test confusion matrices.

The final recommendation translates the matrices into an operational decision. It reports the recall change plus the added review workload.

  • Place the cursor after the tuned confusion matrix in the same cell.
  • Append the recommendation code below it:
false_positive_change = int(tuned_matrix[0, 1]) - int(default_matrix[0, 1])
print(
    f"Recommendation: the validation-selected threshold changes test fraud recall "
    f"from {default_test_metrics['recall']:.1%} to "
    f"{tuned_test_metrics['recall']:.1%}, with "
    f"{false_positive_change:+d} additional false alarms. "
    "Choose this operating point only if that review workload is acceptable."
)

Why Include a Recommendation?

  • The recall values quantify how the tuned threshold changes the share of fraud caught.
  • The false-positive difference quantifies the extra legitimate transactions sent for review.
  • The final sentence makes acceptance depend on the available review capacity.

Before you run the completed cell, consider whether the recall gain will justify the additional false alarms.

  • Save fraud_detection.ipynb.
  • Run the completed visualization-and-recommendation cell.

You will see the plot followed by both test matrices and a recommendation. The recommendation states the test recall change plus the exact number of additional false alarms.

Is the Recommendation Missing?

Confirm that the matrix code appears above false_positive_change in the same cell. Run the test-evaluation cell first if either test metric dictionary is unavailable.

Help me restore the business recommendation.

✔️ Awesome, I've got everything!

Great. Your notebook now selects its threshold from validation data and evaluates that choice once on the untouched test labels.

ⓧ I'd like to double check the full code

Compare the cells in fraud_detection.ipynb with this complete notebook code. Keep each # Cell section in its own notebook cell.

# Cell 1: Environment check and imports
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    precision_recall_curve,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

print("Environment ready")

# Cell 2: Load and validate the Kaggle files
train = pd.read_csv("data/train.csv")
test = pd.read_csv("data/test.csv")
solution = pd.read_csv("data/solution.csv")

target = "is_fraud"
numeric_features = [
    "amount",
    "hour",
    "account_age_days",
    "num_prior_txns",
    "distance_from_home_km",
    "is_foreign",
    "device_change",
]
categorical_features = ["merchant_category"]
feature_columns = numeric_features + categorical_features

missing_train = set(feature_columns + [target]) - set(train.columns)
missing_test = set(feature_columns) - set(test.columns)
missing_solution = {target} - set(solution.columns)

if missing_train or missing_test or missing_solution:
    raise ValueError(
        f"Missing columns. train={missing_train}, test={missing_test}, "
        f"solution={missing_solution}"
    )

if len(test) != len(solution):
    raise ValueError("test.csv and solution.csv must have the same number of rows")

print("Train shape:", train.shape)
print("Test shape:", test.shape)
print("Solution shape:", solution.shape)
print(train.head())

# Cell 3: Explore the class imbalance
class_counts = train[target].value_counts().sort_index()
legitimate_count = int((train[target] == 0).sum())
fraud_count = int((train[target] == 1).sum())
fraud_rate = train[target].mean()

print("Class counts:")
print(class_counts)
print(f"Fraud rate: {fraud_rate:.2%}")

plt.figure(figsize=(7, 4))
plt.bar(
    ["Legitimate", "Fraud"],
    [legitimate_count, fraud_count],
    color=["#4C78A8", "#E45756"],
)
plt.title("Training Transactions by Class")
plt.ylabel("Transaction count")
plt.tight_layout()
plt.show()

# Cell 4: Create a validation split and evaluation helpers
X = train[feature_columns]
y = train[target].astype(int)

X_train, X_valid, y_train, y_valid = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y,
)


def evaluate_scores(name, y_true, scores, threshold):
    predictions = (scores >= threshold).astype(int)
    precision = precision_score(y_true, predictions, zero_division=0)
    recall = recall_score(y_true, predictions, zero_division=0)
    denominator = precision + recall
    f1 = 0.0 if denominator == 0 else 2 * precision * recall / denominator

    metrics = {
        "model": name,
        "threshold": float(threshold),
        "accuracy": accuracy_score(y_true, predictions),
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "average_precision": average_precision_score(y_true, scores),
    }
    return metrics, predictions


def choose_f1_threshold(y_true, scores):
    precision, recall, thresholds = precision_recall_curve(y_true, scores)
    f1_values = []

    for current_precision, current_recall in zip(precision[:-1], recall[:-1]):
        denominator = current_precision + current_recall
        score = (
            0.0
            if denominator == 0
            else 2 * current_precision * current_recall / denominator
        )
        f1_values.append(score)

    best_index = max(range(len(f1_values)), key=f1_values.__getitem__)
    return float(thresholds[best_index]), precision, recall

# Cell 5: Experience the accuracy trap
baseline_scores = pd.Series(0.0, index=y_valid.index)
baseline_metrics, baseline_predictions = evaluate_scores(
    "All-legitimate baseline",
    y_valid,
    baseline_scores,
    threshold=0.5,
)

print(pd.DataFrame([baseline_metrics]).round(4))
print(
    pd.DataFrame(
        confusion_matrix(y_valid, baseline_predictions, labels=[0, 1]),
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

# Cell 6: Build and fit the preprocessing and model pipeline
preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", StandardScaler(), numeric_features),
        (
            "categorical",
            OneHotEncoder(handle_unknown="ignore"),
            categorical_features,
        ),
    ]
)

model = Pipeline(
    steps=[
        ("preprocess", preprocessor),
        ("classifier", LogisticRegression(max_iter=1000)),
    ]
)

model.fit(X_train, y_train)
validation_scores = model.predict_proba(X_valid)[:, 1]

default_validation_metrics, default_validation_predictions = evaluate_scores(
    "Logistic regression at 0.5",
    y_valid,
    validation_scores,
    threshold=0.5,
)

print(pd.DataFrame([default_validation_metrics]).round(4))
print(
    pd.DataFrame(
        confusion_matrix(y_valid, default_validation_predictions, labels=[0, 1]),
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

# Cell 7: Tune on validation data and evaluate untouched test data
best_threshold, curve_precision, curve_recall = choose_f1_threshold(
    y_valid,
    validation_scores,
)

tuned_validation_metrics, tuned_validation_predictions = evaluate_scores(
    "Logistic regression tuned",
    y_valid,
    validation_scores,
    threshold=best_threshold,
)

X_test = test[feature_columns]
y_test = solution[target].astype(int)
test_scores = model.predict_proba(X_test)[:, 1]

default_test_metrics, default_test_predictions = evaluate_scores(
    "Test at 0.5",
    y_test,
    test_scores,
    threshold=0.5,
)
tuned_test_metrics, tuned_test_predictions = evaluate_scores(
    "Test at tuned threshold",
    y_test,
    test_scores,
    threshold=best_threshold,
)

comparison = pd.DataFrame(
    [baseline_metrics, default_validation_metrics, tuned_validation_metrics,
     default_test_metrics, tuned_test_metrics]
).set_index("model")

print(f"Validation-selected threshold: {best_threshold:.4f}")
print(comparison.round(4))

# Cell 8: Visualize the trade-off and report final confusion matrices
plt.figure(figsize=(7, 5))
plt.plot(curve_recall, curve_precision, label="Validation precision-recall curve")
plt.scatter(
    [default_validation_metrics["recall"]],
    [default_validation_metrics["precision"]],
    label="Threshold 0.5",
)
plt.scatter(
    [tuned_validation_metrics["recall"]],
    [tuned_validation_metrics["precision"]],
    label="Tuned threshold",
)
plt.xlabel("Recall")
plt.ylabel("Precision")
plt.title("Fraud Precision-Recall Trade-off")
plt.legend()
plt.tight_layout()
plt.show()

print("Default threshold test confusion matrix")
default_matrix = confusion_matrix(y_test, default_test_predictions, labels=[0, 1])
print(
    pd.DataFrame(
        default_matrix,
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

print("Tuned threshold test confusion matrix")
tuned_matrix = confusion_matrix(y_test, tuned_test_predictions, labels=[0, 1])
print(
    pd.DataFrame(
        tuned_matrix,
        index=["Actual legitimate", "Actual fraud"],
        columns=["Predicted legitimate", "Predicted fraud"],
    )
)

false_positive_change = int(tuned_matrix[0, 1]) - int(default_matrix[0, 1])
print(
    f"Recommendation: the validation-selected threshold changes test fraud recall "
    f"from {default_test_metrics['recall']:.1%} to "
    f"{tuned_test_metrics['recall']:.1%}, with "
    f"{false_positive_change:+d} additional false alarms. "
    "Choose this operating point only if that review workload is acceptable."
)
Export the completed analysis

An HTML export preserves your notebook narrative plus its executed outputs in a shareable report. The final run proves that the whole analysis remains reproducible from top to bottom.

Before rerunning the notebook, consider whether every cell will reproduce the same threshold comparison and recommendation.

  • Run every cell in fraud_detection.ipynb from top to bottom.

You will see Environment ready near the start of the notebook. The final outputs show the 30,000-row test comparison plus the precision-recall plot and both confusion matrices.

  • Save fraud_detection.ipynb.
  • Open the Command Palette in Visual Studio Code.
  • Enter Jupyter: Export to HTML in the Command Palette.
  • Select Jupyter: Export to HTML.

The exported fraud_detection.html report opens with the completed narrative and outputs. Its final section includes the validation-selected threshold plus the business recommendation.

Did the HTML Export Fail?

Confirm that every notebook cell has finished running before exporting. Save fraud_detection.ipynb after the final output appears.

Confirm that the Jupyter extension remains enabled in Visual Studio Code if the export command is unavailable.

Help me diagnose the HTML export.

That completes the analysis. Your HTML report now explains why accuracy was misleading and supports its threshold recommendation with untouched test results.

Secret mission

Test a Class-Weighted Model

Give rare fraud cases more influence during training without changing the evaluation rules. Compare the original and class-weighted models at separately validated thresholds on the same untouched test labels.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the local fraud-detection workspace, close it for now, or remove it entirely. The analysis runs locally in Visual Studio Code, so keeping it creates no ongoing costs.

Resources you used:

  • The fraud-detection folder that contains the complete local workspace.
  • The .venv directory that stores the project-local Python packages.
  • The data directory that holds all three Kaggle CSV files.
  • The fraud_detection.ipynb notebook containing the original model evaluation plus the class-weighted comparison.
  • The fraud_detection.html report containing the completed analysis.
  • The requirements.txt file containing the pinned dependencies.

Keep everything running

No action is required. Choose this option if you are still comparing thresholds or refining the recommendation.

  • Keep the fraud-detection folder in its current location.
  • Leave fraud_detection.ipynb open with the Python (fraud-detection) kernel selected if you plan to continue experimenting.
  • Retain fraud_detection.html as the shareable report of your completed analysis.

Pause - I'll come back to this later

Close the active notebook session to free local memory. Your complete workspace remains ready for another session.

  • Save fraud_detection.ipynb.
  • Close Visual Studio Code.
  • Leave the fraud-detection folder in its current location.

The Visual Studio Code window closes. Your fraud-detection folder remains unchanged.

Delete - I don't want to use this again

Remove the local workspace if you want to start fresh. Deletion is permanent, which makes any copy outside the folder your only backup.

  • Move fraud_detection.html outside the fraud-detection folder if you want to keep the report.
  • Close Visual Studio Code.
  • Use the Windows file browser to return to the parent location where you created fraud-detection.
  • Select the fraud-detection folder.

Permanent Deletion

The next shortcut permanently removes the selected folder. Windows may ask you to confirm the deletion.

  • Press Shift+Delete.
  • Confirm permanent deletion if Windows asks.

You no longer see the fraud-detection folder in its parent location. This confirms that every project-local resource listed above was removed.

Nice Work!

Nice Work!

You did it! Your local fraud analysis now exposes the accuracy trap. Its HTML report explains a defensible fraud-review threshold.

You've learned how to:

  • Explore 90,000 labeled transactions with pandas. Visualize the rare fraud class with Matplotlib.
  • Expose the accuracy trap with an all-legitimate baseline. Confirm that high accuracy can hide zero fraud recall.
  • Build a leakage-safe pipeline with scikit-learn. Select its decision threshold using validation data. Verify the trade-off once on untouched test labels in fraud_detection.html.
  • Secret Mission: Compare ordinary and class-weighted logistic regression at separately validated thresholds. Choose the better operational trade-off from the same untouched test labels.

Ready to quiz yourself?