Leakage-Safe Fraud Detection Benchmark

Build a fraud benchmark with leakage-safe evaluation and threshold selection.

Introduction

30 Second Summary

Most transactions are legitimate, so a fraud system can appear highly accurate by approving every payment. The few fraudulent payments are exactly the cases that matter.

In this project, you will build a reproducible Google Colab notebook that turns an anonymized credit-card dataset into a defensible fraud benchmark. The benchmark uses validation evidence to choose a model plus an operating threshold before opening the test set once.

What You'll Build

You will run the finished notebook to see the winning fraud model beside the evidence supporting its frozen threshold.

By the end of this project, you'll have:

A reproducible data audit that validates the expected 31-column schema. Three saved figures make the class imbalance visible.
A leakage-safe model comparison that scores four Logistic Regression strategies against XGBoost plus Isolation Forest using Average Precision.
A frozen benchmark package containing the selected pipeline plus its operating threshold. The exported joblib bundle is ready for the Streamlit follow-on project. Report-ready files capture the same evidence.
Secret Mission: Stress-test the operating threshold across four false-negative cost assumptions to reveal whether the decision stays stable.

Are there any prerequisites?

You need a Google account plus a Kaggle account. The walkthrough includes browser-based setup in Google Colab plus Google Drive, so local Python or VS Code are unnecessary.

Before We Start

Before any modeling begins, let's commit to building a defensible fraud-screening evaluation artifact. Its score supports a fraud-review decision with two distinct consequences: missed fraud from false negatives versus unnecessary review or customer friction from false positives.

Set Up a Reproducible Colab Workspace

A defensible benchmark needs a reproducible environment. Google Colab runtimes are temporary. Package versions can also change between sessions.

This step anchors your notebook in Google Drive. It also brings the unrenamed dataset from Kaggle into persistent storage. A pinned Python environment makes later model comparisons repeatable.

In this step, get ready to:

Configure fraud_benchmark.ipynb with Colab runtime 2026.10.
Stage creditcard.csv inside the persistent Drive workspace.
Reproduce the pinned package environment.

Create the pinned Colab notebook

A Colab runtime version fixes the notebook's base operating system and Python release. Selecting runtime 2026.10 gives the package pins a known foundation.

Open Google Colab in your browser.
Sign in with your Google account if prompted.
Create a blank notebook from the Colab home page.
Select the notebook title in the upper-left corner.
Enter fraud_benchmark.ipynb as the notebook name.

You should now see fraud_benchmark.ipynb in the notebook header.

Click Runtime in the top menu.
Select Change runtime type.
Choose 2026.10 from Runtime Version.
Apply the runtime selection.

The notebook now targets runtime 2026.10. The final version check will prove that it supplies Python 3.13.18.

Why pin the runtime?

Colab updates its default runtime over time. Pinning 2026.10 prevents a future runtime change from silently replacing the Python foundation beneath your benchmark.

Build the persistent Drive workspace

The Colab virtual machine disappears after the session ends. Drive keeps the dataset and future benchmark outputs beyond that temporary machine.

The authorization request can feel sensitive. It gives this notebook permission to create the project folders inside your Drive.

Select the first code cell in fraud_benchmark.ipynb.
Start the persistent Drive workspace by running this cell:

from pathlib import Path

from google.colab import drive

drive.mount('/content/drive')
ROOT = Path('/content/drive/My Drive/fraud-benchmark')
DATA_DIR = ROOT / 'data'
ARTIFACT_DIR = ROOT / 'artifacts'
FIGURE_DIR = ROOT / 'figures'
for folder in (DATA_DIR, ARTIFACT_DIR, FIGURE_DIR):
    folder.mkdir(parents=True, exist_ok=True)

What does this setup cell do?

The drive.mount() call connects Drive at /content/drive.
The ROOT path keeps every project file inside one fraud-benchmark folder.
The loop creates dedicated folders for input data, exported artifacts, and saved figures.

Drive mounting can pause briefly while the authorization window completes. A short wait here is normal.

Choose your signed-in Google account when the authorization window opens.
Approve the requested Drive access.
Wait for the code cell to finish.
Save fraud_benchmark.ipynb using the notebook's save control.
Return to Google Drive in your browser.
Confirm that data, artifacts, and figures appear inside My Drive/fraud-benchmark.

Drive folders missing?

Check that the mount cell finished without an authorization interruption.
Confirm that your browser allows the Google authorization window to open.
Rerun the mount cell after approving Drive access.

Help me fix my Colab Drive mount.

The benchmark expects the original Kaggle filename. Keeping creditcard.csv unchanged gives every later notebook cell one stable input path.

Open the Credit Card Fraud Detection dataset page.
Complete Kaggle's account flow until the dataset page recognizes your signed-in session.
Download the dataset using the page's download control.
Keep the downloaded file named creditcard.csv.

The CSV is now on your computer. Its persistent home is the data folder you just created in Drive.

Return to Google Drive in your browser.
Open My Drive.
Open the fraud-benchmark folder.
Open the data folder.
Upload creditcard.csv using Drive's file upload control.

Return to fraud_benchmark.ipynb.
Add a code cell beneath the Drive setup cell.
Confirm the persistent dataset path by running this cell:

DATA_PATH = DATA_DIR / 'creditcard.csv'
if not DATA_PATH.exists():
    raise FileNotFoundError(
        f"Place the Kaggle file at {DATA_PATH} and rerun this cell."
    )

print('Artifacts created:')
for path in sorted(ROOT.rglob('*')):
    if path.is_file():
        print(path)

What does this check prove?

The DATA_PATH value points to the exact location required by the benchmark.
The existence check stops the notebook when the CSV is missing.
The final loop prints every file currently stored inside the persistent project folder.

Save fraud_benchmark.ipynb using the notebook's save control.
Check the output for the full /content/drive/My Drive/fraud-benchmark/data/creditcard.csv path.

The printed path proves that the notebook can reach the unrenamed CSV from persistent storage.

CSV path not found?

Confirm that the filename is exactly creditcard.csv.
Move the file into /content/drive/My Drive/fraud-benchmark/data if it was uploaded beside the folder.
Rerun the Drive setup cell if the runtime restarted after the upload.

Help me find my CSV in the mounted Drive folder.

Pin the notebook environment

A reproducible notebook records the environment choices that shape its results. The setup note makes those choices visible before any modeling begins.

What should the setup note record?

Runtime 2026.10 supplies Python 3.13.18.
Drive preserves notebook outputs after the temporary Colab virtual machine disappears.
pandas is pinned to 2.3.3 because imbalanced-learn 0.14.2 declares optional DataFrame support below pandas 3.

Document the three notes above in a Markdown cell beneath the Drive setup cell.

The installation can take a few minutes while Colab resolves the pinned packages. A quiet progress period does not mean the notebook is stuck.

Install the pinned dependencies by running this command in a new code cell:

!pip install pandas==2.3.3 numpy==2.5.4 scikit-learn==1.9.1 imbalanced-learn==0.14.2 xgboost==3.4.1 matplotlib==3.11.2 joblib==1.6.0

What does this installation pin?

The command fixes every dependency at a verified release. This prevents a later package update from changing the benchmark underneath the same notebook code.

NumPy 2.5.4 supplies numerical array operations.
scikit-learn 1.9.1 supplies splitting, preprocessing, models, and evaluation metrics.
XGBoost 3.4.1 supplies the supervised tree model used later.
Matplotlib 3.11.2 supplies the saved benchmark figures.
joblib 1.6.0 supplies model persistence.

Wait for the installation output to finish.
Accept the runtime restart if Colab requests one.
Return to the Drive setup cell after any restart.
Rerun the Drive setup cell to restore /content/drive.
Save fraud_benchmark.ipynb using the notebook's save control.

Package installation failed?

Confirm that the notebook still uses runtime 2026.10.
Rerun the pinned installation cell after the runtime reconnects.
Restart the runtime if Colab reports that newly installed packages require it.

Help me debug the pinned Colab package installation.

Before you run the final check, do you expect every installed version to match the notebook's asserted environment?

Print and validate the dependency versions by running this cell:

from importlib.metadata import version
import sys

import pandas as pd

PACKAGE_NAMES = {
    "pandas": "pandas",
    "numpy": "numpy",
    "scikit-learn": "scikit-learn",
    "imbalanced-learn": "imbalanced-learn",
    "xgboost": "xgboost",
    "matplotlib": "matplotlib",
    "joblib": "joblib",
}
VERSIONS = {"python": sys.version.split()[0]}
VERSIONS.update({label: version(package) for label, package in PACKAGE_NAMES.items()})
print(pd.Series(VERSIONS, name="version"))

EXPECTED_VERSIONS = {
    "python": "3.13.18",
    "pandas": "2.3.3",
    "numpy": "2.5.4",
    "scikit-learn": "1.9.1",
    "imbalanced-learn": "0.14.2",
    "xgboost": "3.4.1",
    "matplotlib": "3.11.2",
    "joblib": "1.6.0",
}
if VERSIONS != EXPECTED_VERSIONS:
    raise RuntimeError(f"Version mismatch. Expected {EXPECTED_VERSIONS}, observed {VERSIONS}")

How does the version check work?

The PACKAGE_NAMES mapping lists every dependency the notebook must reproduce.
The VERSIONS dictionary records the active Python and package releases.
The printed Series gives you a readable environment table.
The final comparison stops the notebook if any observed version differs from EXPECTED_VERSIONS.

The table now reveals whether the runtime and every dependency match the pinned environment.

✔️ I see Python 3.13.18 and every pinned version

That is the reproducibility foundation in place. Your notebook now uses the expected Python runtime and the complete pinned dependency set.

Confirm that the version cell finishes without raising an exception.
Confirm that Drive remains mounted at /content/drive.

ⓧ I see another Python version

The notebook is connected to a different Colab runtime. Switch back to the required runtime before reinstalling the packages.

Click Runtime in the top menu.
Select Change runtime type.
Choose 2026.10 from Runtime Version.
Apply the runtime selection.
Rerun the pinned installation cell.
Accept the restart if Colab requests one.
Rerun the Drive setup cell.
Rerun the version-check cell.

ⓧ I see a package error or mismatch

The runtime is correct, but at least one dependency did not settle on its pinned release. Reinstall the complete set so compatible versions are resolved together.

Rerun the pinned installation cell above.
Accept the restart if Colab requests one.
Rerun the Drive setup cell after the restart.
Rerun the version-check cell.
Confirm that the final cell completes without an exception.

✔️ Awesome, I've got everything!

Your notebook now contains the persistent workspace setup, dataset check, pinned installation command, and strict version verification.

ⓧ I'd like to double check the full code

from pathlib import Path

from google.colab import drive

drive.mount('/content/drive')
ROOT = Path('/content/drive/My Drive/fraud-benchmark')
DATA_DIR = ROOT / 'data'
ARTIFACT_DIR = ROOT / 'artifacts'
FIGURE_DIR = ROOT / 'figures'
for folder in (DATA_DIR, ARTIFACT_DIR, FIGURE_DIR):
    folder.mkdir(parents=True, exist_ok=True)

DATA_PATH = DATA_DIR / 'creditcard.csv'
if not DATA_PATH.exists():
    raise FileNotFoundError(
        f"Place the Kaggle file at {DATA_PATH} and rerun this cell."
    )

print('Artifacts created:')
for path in sorted(ROOT.rglob('*')):
    if path.is_file():
        print(path)

!pip install pandas==2.3.3 numpy==2.5.4 scikit-learn==1.9.1 imbalanced-learn==0.14.2 xgboost==3.4.1 matplotlib==3.11.2 joblib==1.6.0

from importlib.metadata import version
import sys

import pandas as pd

PACKAGE_NAMES = {
    "pandas": "pandas",
    "numpy": "numpy",
    "scikit-learn": "scikit-learn",
    "imbalanced-learn": "imbalanced-learn",
    "xgboost": "xgboost",
    "matplotlib": "matplotlib",
    "joblib": "joblib",
}
VERSIONS = {"python": sys.version.split()[0]}
VERSIONS.update({label: version(package) for label, package in PACKAGE_NAMES.items()})
print(pd.Series(VERSIONS, name="version"))

EXPECTED_VERSIONS = {
    "python": "3.13.18",
    "pandas": "2.3.3",
    "numpy": "2.5.4",
    "scikit-learn": "1.9.1",
    "imbalanced-learn": "0.14.2",
    "xgboost": "3.4.1",
    "matplotlib": "3.11.2",
    "joblib": "1.6.0",
}
if VERSIONS != EXPECTED_VERSIONS:
    raise RuntimeError(f"Version mismatch. Expected {EXPECTED_VERSIONS}, observed {VERSIONS}")

Your dataset and environment now survive beyond the temporary Colab session. Next, you will audit the CSV before allowing any model to learn from it.

Audit the Dataset and Visualize Imbalance

Your Google Colab workspace can now reproduce the environment. Your persistent Google Drive folders also protect the project from a temporary runtime disappearing.

A successful CSV load cannot prove that the data is safe for modeling. This step validates the ordered schema before making the class imbalance visible.

In this step, get ready to:

Load the raw dataset while validating its columns and missing values.
Preserve the raw data while creating an audited modeling copy.
Generate three figures that expose class frequency, amount skew, and elapsed-time coverage.

Validate the raw dataset

The raw CSV becomes a pandas DataFrame called df_raw. An exact column check stops the benchmark if the file has an unexpected structure.

Add a code cell below the workspace setup cells in fraud_benchmark.ipynb.
Load the CSV and run its first safety checks by using this code:

EXPECTED_COLUMNS = [
    "Time",
    *[f"V{i}" for i in range(1, 29)],
    "Amount",
    "Class",
]

DATA_PATH = DATA_DIR / 'creditcard.csv'
if not DATA_PATH.exists():
    raise FileNotFoundError(
        f"Place the Kaggle file at {DATA_PATH} and rerun this cell."
    )

df_raw = pd.read_csv(DATA_PATH)
if list(df_raw.columns) != EXPECTED_COLUMNS:
    raise ValueError(
        "Unexpected schema.\n"
        f"Expected: {EXPECTED_COLUMNS}\n"
        f"Observed: {list(df_raw.columns)}"
    )

missing_by_column = df_raw.isna().sum()
duplicate_later_rows = int(df_raw.duplicated(keep='first').sum())
duplicate_group_rows = int(df_raw.duplicated(keep=False).sum())
if int(missing_by_column.sum()) != 0:
    raise ValueError("Unexpected missing values found. Inspect before modeling.")

What does this code check?

The EXPECTED_COLUMNS list defines the required order from Time through Class.
The df_raw variable preserves the dataset exactly as pandas loads it.
The missing_by_column result counts empty cells in every column.
The two duplicate counts distinguish later repeated rows from every row that belongs to a repeated group.

Run the cell in Colab.
Confirm the cell reaches a completed state without a file, schema, or missing-value exception.

Seeing a dataset validation error?

Confirm creditcard.csv still uses its original filename.
Confirm the file is inside /content/drive/My Drive/fraud-benchmark/data.
Check whether the CSV header starts with Time and ends with Class.

Help me diagnose why my fraud dataset failed its file, schema, or missing-value validation.

Create the modeling audit

Exact duplicate rows can cross random data splits. The modeling copy keeps the first occurrence while preserving df_raw for an honest record of the source file.

The missing transaction identifier creates an important limitation. Two identical rows could represent separate real transactions.

Add a new code cell below the validation cell.
Create the duplicate-free modeling copy and print its audit by using this code:

raw_class_counts = df_raw['Class'].value_counts().sort_index()
df = df_raw.loc[~df_raw.duplicated(keep='first')].copy()
model_class_counts = df['Class'].value_counts().sort_index()

audit = {
    "raw_rows": int(len(df_raw)),
    "raw_columns": int(df_raw.shape[1]),
    "raw_legitimate": int(raw_class_counts.get(0, 0)),
    "raw_fraud": int(raw_class_counts.get(1, 0)),
    "missing_cells": int(missing_by_column.sum()),
    "later_duplicate_rows_removed": duplicate_later_rows,
    "all_rows_in_duplicate_groups": duplicate_group_rows,
    "modeling_rows": int(len(df)),
    "modeling_legitimate": int(model_class_counts.get(0, 0)),
    "modeling_fraud": int(model_class_counts.get(1, 0)),
    "duplicate_decision": (
        "Preserve the raw DataFrame but remove later exact duplicates from the "
        "modeling copy to reduce identical rows crossing data splits."
    ),
    "duplicate_limitation": (
        "No transaction identifier exists, so some exact repeats could represent "
        "separate real transactions."
    ),
}
print(json.dumps(audit, indent=2))

How does the audit protect the benchmark?

The raw counts document what arrived from the source file.
The modeling counts document the dataset after later exact duplicates are removed.
The decision text records why the cleaning rule exists.
The limitation text prevents the duplicate rule from being presented as certain business truth.

Run the audit cell.
Confirm the printed JSON includes raw counts, modeling counts, duplicate counts, and both duplicate notes.

Your observed modeling row count should be no greater than the raw row count. The notebook uses its own observed values instead of assuming the dataset matches a published summary.

Audit counts missing or incomplete?

Run the previous validation cell before running the audit cell.
Check that the target column is spelled Class with an uppercase first letter.
Confirm the modeling variable is named df while the preserved raw variable remains df_raw.

Help me fix an incomplete audit dictionary without changing the duplicate-handling decision.

A reusable audit needs files that survive the current notebook session. The next cell exports the audit plus descriptive statistics for the cleaned modeling data.

Add another code cell below the printed audit.
Export the audit files and inspect the modeling data types by using this code:

(ARTIFACT_DIR / 'data_audit.json').write_text(
    json.dumps(audit, indent=2), encoding='utf-8'
)
df.describe().T.to_csv(ARTIFACT_DIR / 'descriptive_statistics.csv')
print(json.dumps(audit, indent=2))
print(df.dtypes)

What gets preserved?

The data_audit.json file preserves the counts plus the duplicate decision.
The descriptive_statistics.csv file preserves summary statistics for df.
The data-type output confirms how pandas interpreted every column before modeling.

Run the export cell.
Confirm the output prints the audit followed by the data type for every expected column.

Artifact files not saving?

Confirm the mounted Drive session is still connected.
Run the workspace cell from earlier if the runtime restarted.
Confirm ARTIFACT_DIR still points to the existing artifacts folder.

Help me diagnose why my audit files are not being written to the mounted Drive folder.

Visualize the imbalance

Tables record exact values. Matplotlib turns their scale into evidence that is easier to inspect.

The amount chart uses NumPy to calculate a display-only logarithm. The anonymized V1 through V28 columns remain PCA components without guessed business meanings.

Add a code cell below the audit output.
Plot the duplicate-free class counts by using this code:

fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(['Legitimate', 'Fraud'], [audit['modeling_legitimate'], audit['modeling_fraud']])
ax.set_yscale('log')
ax.set_title('Class distribution after exact-duplicate handling')
ax.set_ylabel('Transactions, logarithmic scale')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'class_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

Why use a logarithmic axis?

The logarithmic y-axis keeps both classes visible despite their large count difference.
The saved figure uses the post-duplicate counts from audit.
The chart makes the majority-class accuracy trap visible before a model is trained.

Run the class-distribution cell.
Confirm the notebook displays separate bars for legitimate and fraud transactions on a logarithmic scale.
Add a Markdown cell below the chart containing INSERT RESULT: Describe the observed class imbalance without relying on accuracy..

Class chart missing a bar?

Run the audit cell again to restore the audit dictionary.
Check that modeling_legitimate and modeling_fraud appear in the printed audit.
Confirm FIGURE_DIR still points to the existing figures folder.

Help me fix a class-distribution chart with a missing bar or an empty saved image.

Add a new code cell below the class interpretation.
Visualize the amount distribution by using this code:

fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(np.log1p(df['Amount']), bins=60)
ax.set_title('Transaction amount distribution')
ax.set_xlabel('log1p(Amount), visualization only')
ax.set_ylabel('Transactions')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'amount_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

Why transform Amount for this chart?

The np.log1p calculation compresses large values so the distribution is easier to inspect.
The transformation affects this visualization only.
The modeling DataFrame keeps its original Amount values.

Run the amount-distribution cell.
Confirm the notebook displays a histogram with log1p(Amount), visualization only on the horizontal axis.
Add a Markdown cell below the chart containing INSERT RESULT: Describe the observed amount skew after the display-only transformation..

Amount histogram not rendering?

Run the modeling-audit cell again to restore df.
Confirm the validated schema contains Amount with the same capitalization.
Check that the earlier setup imported NumPy as np.

Help me diagnose why the transformed Amount histogram does not render or save correctly.

Before you run the next cell, predict whether transaction frequency stays uniform across the recorded elapsed hours.

Add a new code cell below the amount interpretation.
Convert seconds to elapsed hours for the time-distribution chart by using this code:

fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(df['Time'] / 3600, bins=48)
ax.set_title('Transactions across elapsed hours')
ax.set_xlabel('Hours since the first transaction')
ax.set_ylabel('Transactions')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'time_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

What does the time chart show?

Dividing Time by 3600 converts the recorded seconds into elapsed hours.
The histogram reveals how transaction frequency changes across the recorded period.
The chart supports a statement about time coverage without claiming that the values represent calendar dates.

Run the time-distribution cell.
Compare the displayed transaction pattern with your prediction.
Add a Markdown cell below the chart containing INSERT RESULT: Describe the elapsed-time coverage. State why V1 through V28 should not receive guessed business meanings..

Time chart empty or compressed?

Confirm the validated schema contains Time with an uppercase first letter.
Check that the histogram receives df['Time'] / 3600.
Run the cell that creates the duplicate-free df copy before rerunning the plot.

Help me fix an empty or compressed elapsed-time histogram in my fraud audit notebook.

✔️ Awesome, I've got everything!

Your audit cells now preserve the raw data, create the modeling copy, export two audit files, and save three EDA figures.

ⓧ I'd like to double check the full code

Compare the audit and EDA cells in your notebook with this combined reference.

EXPECTED_COLUMNS = [
    "Time",
    *[f"V{i}" for i in range(1, 29)],
    "Amount",
    "Class",
]

DATA_PATH = DATA_DIR / 'creditcard.csv'
if not DATA_PATH.exists():
    raise FileNotFoundError(
        f"Place the Kaggle file at {DATA_PATH} and rerun this cell."
    )

df_raw = pd.read_csv(DATA_PATH)
if list(df_raw.columns) != EXPECTED_COLUMNS:
    raise ValueError(
        "Unexpected schema.\n"
        f"Expected: {EXPECTED_COLUMNS}\n"
        f"Observed: {list(df_raw.columns)}"
    )

missing_by_column = df_raw.isna().sum()
duplicate_later_rows = int(df_raw.duplicated(keep='first').sum())
duplicate_group_rows = int(df_raw.duplicated(keep=False).sum())
if int(missing_by_column.sum()) != 0:
    raise ValueError("Unexpected missing values found. Inspect before modeling.")

raw_class_counts = df_raw['Class'].value_counts().sort_index()
df = df_raw.loc[~df_raw.duplicated(keep='first')].copy()
model_class_counts = df['Class'].value_counts().sort_index()

audit = {
    "raw_rows": int(len(df_raw)),
    "raw_columns": int(df_raw.shape[1]),
    "raw_legitimate": int(raw_class_counts.get(0, 0)),
    "raw_fraud": int(raw_class_counts.get(1, 0)),
    "missing_cells": int(missing_by_column.sum()),
    "later_duplicate_rows_removed": duplicate_later_rows,
    "all_rows_in_duplicate_groups": duplicate_group_rows,
    "modeling_rows": int(len(df)),
    "modeling_legitimate": int(model_class_counts.get(0, 0)),
    "modeling_fraud": int(model_class_counts.get(1, 0)),
    "duplicate_decision": (
        "Preserve the raw DataFrame but remove later exact duplicates from the "
        "modeling copy to reduce identical rows crossing data splits."
    ),
    "duplicate_limitation": (
        "No transaction identifier exists, so some exact repeats could represent "
        "separate real transactions."
    ),
}
(ARTIFACT_DIR / 'data_audit.json').write_text(
    json.dumps(audit, indent=2), encoding='utf-8'
)
df.describe().T.to_csv(ARTIFACT_DIR / 'descriptive_statistics.csv')
print(json.dumps(audit, indent=2))
print(df.dtypes)

fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(['Legitimate', 'Fraud'], [audit['modeling_legitimate'], audit['modeling_fraud']])
ax.set_yscale('log')
ax.set_title('Class distribution after exact-duplicate handling')
ax.set_ylabel('Transactions, logarithmic scale')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'class_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(np.log1p(df['Amount']), bins=60)
ax.set_title('Transaction amount distribution')
ax.set_xlabel('log1p(Amount), visualization only')
ax.set_ylabel('Transactions')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'amount_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(df['Time'] / 3600, bins=48)
ax.set_title('Transactions across elapsed hours')
ax.set_xlabel('Hours since the first transaction')
ax.set_ylabel('Transactions')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'time_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

Before you inspect the saved outputs, predict which two audit files and three figure files should now be present.

Switch back to Google Drive from earlier.
Open the fraud-benchmark folder inside My Drive.
Open the artifacts folder.
Confirm it contains data_audit.json plus descriptive_statistics.csv.
Return to the fraud-benchmark folder.
Open the figures folder.
Confirm it contains class_distribution.png, amount_distribution.png, and time_distribution.png.

Your notebook now proves the dataset's condition before any model can influence the story.

Your audited modeling data is ready. Next, you will use it to expose how a majority prediction can look accurate while missing every fraud case.

Expose the Accuracy Trap

Your audited Google Colab notebook now proves that the dataset has the expected structure. The saved figures make the severe class imbalance visible.

An imbalanced dataset allows a simple prediction rule to score highly on accuracy. This step tests whether that score represents useful fraud detection before any real models compete.

In this step, get ready to:

Freeze reproducible stratified train, validation, and test partitions.
Evaluate a majority-class baseline using validation data only.
Establish the metric hierarchy for every later model comparison.

Create and seal the data splits

A stratified split preserves the rare fraud proportion in each partition. The test partition stays sealed until every model choice and threshold decision is final.

Add the split logic beneath the final EDA cell in fraud_benchmark.ipynb by running the code below:

X = df[FEATURES].copy()
y = df['Class'].astype(int).copy()

X_train_val, X_test, y_train_val, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=SEED,
    stratify=y,
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.25,
    random_state=SEED,
    stratify=y_train_val,
)

split_counts = pd.DataFrame(
    {
        "train": y_train.value_counts().sort_index(),
        "validation": y_val.value_counts().sort_index(),
        "test_sealed": y_test.value_counts().sort_index(),
    }
).fillna(0).astype(int)
print(split_counts)
split_counts.to_csv(ARTIFACT_DIR / 'split_class_counts.csv')

What does this code do?

The first split reserves 20 percent of the modeling data for X_test and y_test.
The second split creates a 60 percent training partition and a 20 percent validation partition.
The shared SEED value makes both split operations reproducible.
Stratification preserves the legitimate-to-fraud ratio in every partition.
The test_sealed column records the test counts without using those rows for selection.
The final line saves the class-count table as split_class_counts.csv in the existing artifacts directory.

Confirm the printed table contains train, validation, and test_sealed columns.
Confirm each partition contains legitimate and fraud rows.
Confirm split_class_counts.csv now exists in /content/drive/My Drive/fraud-benchmark/artifacts.

That partition is now locked in. Your training and validation work can continue without allowing the final test rows to influence a decision.

Split cell not completing?

Check that the earlier audit cells created df before this cell runs.
Check that FEATURES contains the 30 predictors from the validated schema.
Rerun the setup cell if the runtime restart cleared the imported split function.

Help me diagnose the stratified split.

Run the majority-class baseline

Every later model needs a minimum standard to beat. The reusable metric helper puts threshold metrics and ranking metrics into one consistent row.

Add the metric helper in a new code cell by running the code below:

def metric_row(name, y_true, scores, threshold=0.5):
    predictions = (scores >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(
        y_true, predictions, labels=[0, 1]
    ).ravel()
    return {
        "model": name,
        "threshold": float(threshold),
        "accuracy": float((predictions == np.asarray(y_true)).mean()),
        "precision": float(precision_score(y_true, predictions, zero_division=0)),
        "recall": float(recall_score(y_true, predictions, zero_division=0)),
        "f1": float(f1_score(y_true, predictions, zero_division=0)),
        "roc_auc": float(roc_auc_score(y_true, scores)),
        "average_precision": float(average_precision_score(y_true, scores)),
        "tn": int(tn),
        "fp": int(fp),
        "fn": int(fn),
        "tp": int(tp),
    }

What does this helper measure?

The threshold converts continuous scores into legitimate or fraud predictions.
The confusion matrix separates true negatives, false positives, false negatives, and true positives.
Accuracy, precision, recall, and F1 describe predictions at one threshold.
ROC-AUC and Average Precision describe ranking performance across possible thresholds.

Confirm the helper cell finishes running without an exception.
Confirm the completed helper cell remains saved in fraud_benchmark.ipynb.

Metric helper not loading?

Check that the setup cell imported NumPy and every metric used inside metric_row.
Check that the function body remains indented beneath the def line.

Help me debug the metric helper.

Before you run the baseline, do you think its accuracy will reveal whether it catches fraud?

Fit the majority-class baseline on the training partition by running the code below:

dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
dummy_scores = dummy.predict_proba(X_val)[:, 1]
dummy_metrics = metric_row('Dummy majority baseline', y_val, dummy_scores)
print(pd.DataFrame([dummy_metrics]).T)

What does this baseline test?

The baseline fits only on X_train and y_train.
The strategy='most_frequent' setting predicts the dominant training class for every validation row.
The probability scores come from X_val.
The metric row compares those validation scores with y_val.
The sealed variables X_test and y_test remain untouched.

Confirm the metric table includes accuracy, precision, recall, F1, ROC-AUC, Average Precision, and confusion-matrix counts.
Confirm the recall row shows 0.0.

You caught the accuracy trap. The baseline earns high accuracy while missing every fraud case in the validation partition.

Baseline table not appearing?

Check that the split cell created X_train, y_train, X_val, and y_val.
Check that the helper cell completed before the baseline calls metric_row.
Rerun the setup imports if a runtime restart cleared DummyClassifier from memory.

Help me debug the majority baseline.

Set the evaluation rules

The failed baseline gives every later comparison a clear hierarchy. Average Precision becomes the primary validation metric because it summarizes precision as recall changes across score thresholds.

What does each metric answer?

Accuracy asks how often the thresholded predictions match the labels.
Precision asks how many transactions flagged as fraud are actually fraud.
Recall asks how many fraud cases the chosen threshold catches.
F1 summarizes precision and recall at one threshold.
Average Precision summarizes precision-recall ranking across thresholds.
ROC-AUC provides a secondary view of ranking across both classes.

Add a Markdown cell beneath the baseline output.
Document that the high accuracy comes from predicting the dominant legitimate class.
Define precision as the share of fraud flags that are correct.
Define recall as the share of fraud cases caught at the chosen threshold.
Describe F1 as a threshold-specific balance between precision and recall.

Why keep the test set sealed?

Every model choice made from test results would tune the benchmark to its final exam. Training and validation data must settle the intervention, model, and threshold before the test set provides one unbiased estimate.

Record Average Precision as the primary validation selection metric.
Record ROC-AUC as the secondary ranking metric.
State that model selection cannot use X_test or y_test.
Label future plots as Precision-Recall curves with AP values.

Before the final check, which metric do you expect to expose the majority baseline most clearly?

Display the validation baseline again by running the code below:

print(pd.DataFrame([dummy_metrics]).T)

What does this check prove?

This prints the metrics already calculated from validation data. It reproduces the accuracy trap without fitting another model or opening the sealed test partition.

You should again see high accuracy beside fraud recall of 0.0. That contrast proves why accuracy cannot lead this benchmark.

Metrics variable missing?

Rerun the split cell to restore the partition variables.
Rerun the metric helper cell to restore metric_row.
Rerun the baseline cell to restore dummy_metrics.

Help me restore the validation metric table.

✔️ Awesome, I've got everything!

Your notebook now contains the stratified 60/20/20 split, the sealed test-set convention, the reusable metric helper, and the validation-only majority baseline.

ⓧ I'd like to double check the full code

from pathlib import Path
from importlib.metadata import version
import json
import sys

from google.colab import drive
import joblib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline
from imblearn.under_sampling import RandomUnderSampler
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import IsolationForest
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold, train_test_split
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier

SEED = 42
MIN_PRECISION = 0.20
FALSE_POSITIVE_COST = 1
FALSE_NEGATIVE_COST = 25

EXPECTED_COLUMNS = [
    "Time",
    *[f"V{i}" for i in range(1, 29)],
    "Amount",
    "Class",
]
FEATURES = EXPECTED_COLUMNS[:-1]
MODEL_FEATURE_ORDER = ["Time", "Amount", *[f"V{i}" for i in range(1, 29)]]

PACKAGE_NAMES = {
    "pandas": "pandas",
    "numpy": "numpy",
    "scikit-learn": "scikit-learn",
    "imbalanced-learn": "imbalanced-learn",
    "xgboost": "xgboost",
    "matplotlib": "matplotlib",
    "joblib": "joblib",
}

VERSIONS = {"python": sys.version.split()[0]}
VERSIONS.update({label: version(package) for label, package in PACKAGE_NAMES.items()})
print(pd.Series(VERSIONS, name="version"))

EXPECTED_VERSIONS = {
    "python": "3.13.18",
    "pandas": "2.3.3",
    "numpy": "2.5.4",
    "scikit-learn": "1.9.1",
    "imbalanced-learn": "0.14.2",
    "xgboost": "3.4.1",
    "matplotlib": "3.11.2",
    "joblib": "1.6.0",
}
if VERSIONS != EXPECTED_VERSIONS:
    raise RuntimeError(f"Version mismatch. Expected {EXPECTED_VERSIONS}, observed {VERSIONS}")

drive.mount('/content/drive')
ROOT = Path('/content/drive/My Drive/fraud-benchmark')
DATA_DIR = ROOT / 'data'
ARTIFACT_DIR = ROOT / 'artifacts'
FIGURE_DIR = ROOT / 'figures'
for folder in (DATA_DIR, ARTIFACT_DIR, FIGURE_DIR):
    folder.mkdir(parents=True, exist_ok=True)

DATA_PATH = DATA_DIR / 'creditcard.csv'
if not DATA_PATH.exists():
    raise FileNotFoundError(
        f"Place the Kaggle file at {DATA_PATH} and rerun this cell."
    )

df_raw = pd.read_csv(DATA_PATH)
if list(df_raw.columns) != EXPECTED_COLUMNS:
    raise ValueError(
        "Unexpected schema.\n"
        f"Expected: {EXPECTED_COLUMNS}\n"
        f"Observed: {list(df_raw.columns)}"
    )

missing_by_column = df_raw.isna().sum()
duplicate_later_rows = int(df_raw.duplicated(keep='first').sum())
duplicate_group_rows = int(df_raw.duplicated(keep=False).sum())
if int(missing_by_column.sum()) != 0:
    raise ValueError("Unexpected missing values found. Inspect before modeling.")

raw_class_counts = df_raw['Class'].value_counts().sort_index()
df = df_raw.loc[~df_raw.duplicated(keep='first')].copy()
model_class_counts = df['Class'].value_counts().sort_index()

audit = {
    "raw_rows": int(len(df_raw)),
    "raw_columns": int(df_raw.shape[1]),
    "raw_legitimate": int(raw_class_counts.get(0, 0)),
    "raw_fraud": int(raw_class_counts.get(1, 0)),
    "missing_cells": int(missing_by_column.sum()),
    "later_duplicate_rows_removed": duplicate_later_rows,
    "all_rows_in_duplicate_groups": duplicate_group_rows,
    "modeling_rows": int(len(df)),
    "modeling_legitimate": int(model_class_counts.get(0, 0)),
    "modeling_fraud": int(model_class_counts.get(1, 0)),
    "duplicate_decision": (
        "Preserve the raw DataFrame but remove later exact duplicates from the "
        "modeling copy to reduce identical rows crossing data splits."
    ),
    "duplicate_limitation": (
        "No transaction identifier exists, so some exact repeats could represent "
        "separate real transactions."
    ),
}
(ARTIFACT_DIR / 'data_audit.json').write_text(
    json.dumps(audit, indent=2), encoding='utf-8'
)
df.describe().T.to_csv(ARTIFACT_DIR / 'descriptive_statistics.csv')
print(json.dumps(audit, indent=2))
print(df.dtypes)

fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(['Legitimate', 'Fraud'], [audit['modeling_legitimate'], audit['modeling_fraud']])
ax.set_yscale('log')
ax.set_title('Class distribution after exact-duplicate handling')
ax.set_ylabel('Transactions, logarithmic scale')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'class_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(np.log1p(df['Amount']), bins=60)
ax.set_title('Transaction amount distribution')
ax.set_xlabel('log1p(Amount), visualization only')
ax.set_ylabel('Transactions')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'amount_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(df['Time'] / 3600, bins=48)
ax.set_title('Transactions across elapsed hours')
ax.set_xlabel('Hours since the first transaction')
ax.set_ylabel('Transactions')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'time_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

X = df[FEATURES].copy()
y = df['Class'].astype(int).copy()

X_train_val, X_test, y_train_val, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=SEED,
    stratify=y,
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.25,
    random_state=SEED,
    stratify=y_train_val,
)

split_counts = pd.DataFrame(
    {
        "train": y_train.value_counts().sort_index(),
        "validation": y_val.value_counts().sort_index(),
        "test_sealed": y_test.value_counts().sort_index(),
    }
).fillna(0).astype(int)
print(split_counts)
split_counts.to_csv(ARTIFACT_DIR / 'split_class_counts.csv')


def metric_row(name, y_true, scores, threshold=0.5):
    predictions = (scores >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(
        y_true, predictions, labels=[0, 1]
    ).ravel()
    return {
        "model": name,
        "threshold": float(threshold),
        "accuracy": float((predictions == np.asarray(y_true)).mean()),
        "precision": float(precision_score(y_true, predictions, zero_division=0)),
        "recall": float(recall_score(y_true, predictions, zero_division=0)),
        "f1": float(f1_score(y_true, predictions, zero_division=0)),
        "roc_auc": float(roc_auc_score(y_true, scores)),
        "average_precision": float(average_precision_score(y_true, scores)),
        "tn": int(tn),
        "fp": int(fp),
        "fn": int(fn),
        "tp": int(tp),
    }


dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
dummy_scores = dummy.predict_proba(X_val)[:, 1]
dummy_metrics = metric_row('Dummy majority baseline', y_val, dummy_scores)
print(pd.DataFrame([dummy_metrics]).T)

Your benchmark now has a sealed test set and a baseline that exposes the accuracy trap. Next, you will compare leakage-safe imbalance strategies using validation Average Precision.

Compare Leakage-Safe Fraud Models

Your majority baseline exposed the accuracy trap. Average Precision now gives you a stronger way to compare fraud-ranking performance without touching the sealed test set.

You can now test whether imbalance strategies change that ranking quality. Every sampling intervention stays inside an imbalanced-learn Pipeline, so validation rows remain untouched.

In this step, get ready to:

Train four Logistic Regression strategies under the same validation conditions.
Tune a CPU XGBoost model using training-only cross-validation.
Compare the supervised models with a legitimate-only Isolation Forest.

Train four Logistic Regression strategies

A fair ablation keeps the classifier and preprocessing fixed. Only the imbalance intervention changes between experiments.

Define the shared preprocessor and four experiment specifications by running this code in a new Colab cell:

def make_preprocessor():
    return ColumnTransformer(
        transformers=[('scale', StandardScaler(), ['Time', 'Amount'])],
        remainder='passthrough',
        verbose_feature_names_out=False,
    )


logistic_specs = {
    "Logistic - no correction": {"sampler": None, "class_weight": None},
    "Logistic - class weight": {"sampler": None, "class_weight": 'balanced'},
    "Logistic - SMOTE": {"sampler": SMOTE(random_state=SEED), "class_weight": None},
    "Logistic - undersampling": {
        "sampler": RandomUnderSampler(random_state=SEED),
        "class_weight": None,
    },
}

logistic_models = {}
logistic_scores = {}
validation_rows = [dummy_metrics]

What does this setup do?

The make_preprocessor() function scales Time and Amount when each pipeline fits on training data.
The remainder='passthrough' setting preserves V1 through V28 without extra scaling.
The logistic_specs dictionary isolates the four imbalance choices while keeping the classifier consistent.
The result dictionaries retain each fitted model and its validation scores for later comparison.

Confirm the cell finishes without an exception.

Seeing an import or name error?

Return to the setup cells from earlier. Rerun the imports for ColumnTransformer, StandardScaler, SMOTE, RandomUnderSampler, and Pipeline.

Rerun the cell that defines SEED if Colab restarted the temporary runtime.

Help me resolve a missing import or variable in my Logistic Regression setup.

Fit all four pipelines on the training partition by running this code in the next cell:

for name, spec in logistic_specs.items():
    steps = [('preprocess', make_preprocessor())]
    if spec['sampler'] is not None:
        steps.append(('sampler', spec['sampler']))
    steps.append(
        (
            'model',
            LogisticRegression(
                class_weight=spec['class_weight'],
                solver='lbfgs',
                max_iter=1000,
            ),
        )
    )
    model = Pipeline(steps)
    model.fit(X_train, y_train)
    scores = model.predict_proba(X_val)[:, 1]
    logistic_models[name] = model
    logistic_scores[name] = scores
    validation_rows.append(metric_row(name, y_val, scores))

best_logistic_name = max(
    logistic_scores,
    key=lambda name: average_precision_score(y_val, logistic_scores[name]),
)
best_logistic_model = logistic_models[best_logistic_name]
best_logistic_scores = logistic_scores[best_logistic_name]
print(f"Best Logistic Regression strategy: {best_logistic_name}")

How does this stay leakage-safe?

Each pipeline learns preprocessing from X_train only.
SMOTE or undersampling operates during fit inside its own pipeline.
Every model scores the original X_val rows without resampling them.
The max selection keeps the Logistic strategy with the strongest validation Average Precision.

Confirm the output begins with Best Logistic Regression strategy:.
Add a Markdown cell below the result.
Record which Logistic strategy achieved the strongest validation Average Precision.
Explain why synthetic samples or removed majority rows could change fraud ranking.
State the trade-off introduced by the winning imbalance strategy.

Logistic fitting failed?

Confirm that X_train, X_val, y_train, and y_val still come from the stratified split.

Rerun the cell that defines metric_row if the runtime lost earlier notebook state.

Help me debug my leakage-safe Logistic Regression pipelines.

Tune CPU XGBoost on training data

Logistic Regression gives you a controlled imbalance ablation. XGBoost adds nonlinear tree interactions while preserving the same validation metric.

Why XGBoost for this benchmark?

XGBoost provides a positive-class weight and histogram-based CPU training. Its scikit-learn-compatible estimator also fits inside the same preprocessing pattern.

A small randomized search keeps the experiment focused on imbalance and threshold decisions. Adding another ensemble such as Random Forest would dilute that comparison.

Define the class ratio and CPU XGBoost pipeline by running this code in a new cell:

negative_count = int((y_train == 0).sum())
positive_count = int((y_train == 1).sum())
training_ratio = negative_count / positive_count

xgb_pipeline = Pipeline(
    [
        ('preprocess', make_preprocessor()),
        (
            'model',
            XGBClassifier(
                objective='binary:logistic',
                tree_method='hist',
                device='cpu',
                random_state=SEED,
                n_jobs=1,
            ),
        ),
    ]
)

What does this pipeline control?

The training_ratio value captures the number of legitimate training rows per fraud row.
The tree_method='hist' setting uses histogram-based tree training.
The device='cpu' setting keeps the experiment compatible with the selected Colab runtime.
The preprocessor remains inside the pipeline so every cross-validation fit learns its transformations from the corresponding training fold.

Confirm the pipeline-definition cell finishes without an exception.

XGBoost pipeline not defined?

Rerun the package import cell if the Colab runtime no longer recognizes XGBClassifier.

Confirm that both classes remain present in y_train before calculating the ratio.

Help me diagnose my CPU XGBoost pipeline setup.

Configure the training-only randomized search by running this code in the next cell:

parameter_space = {
    'model__n_estimators': [150, 250],
    'model__max_depth': [3, 5],
    'model__learning_rate': [0.05, 0.10],
    'model__subsample': [0.80, 1.00],
    'model__colsample_bytree': [0.80, 1.00],
    'model__scale_pos_weight': [1.0, training_ratio],
}
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=SEED)
xgb_search = RandomizedSearchCV(
    estimator=xgb_pipeline,
    param_distributions=parameter_space,
    n_iter=4,
    scoring='average_precision',
    n_jobs=-1,
    cv=cv,
    refit=True,
    random_state=SEED,
    verbose=1,
)

How is the search constrained?

The parameter space varies tree count and depth.
It also varies learning rate and row sampling.
Column sampling and positive-class weighting complete each candidate configuration.
Three-fold stratification preserves the rare fraud proportion across training folds.
The search evaluates four sampled configurations using Average Precision.

Confirm the search object is created without an exception.

This search is the slowest cell in the step. Expect it to run for several minutes while four parameter settings move through three stratified folds.

Fit the search on training data by running this code in the next cell:

xgb_search.fit(X_train, y_train)
xgb_model = xgb_search.best_estimator_
xgb_scores = xgb_model.predict_proba(X_val)[:, 1]
validation_rows.append(metric_row('XGBoost', y_val, xgb_scores))
print('Best XGBoost parameters:', xgb_search.best_params_)
print('Best cross-validated AP:', float(xgb_search.best_score_))

What does the search return?

The search fits every candidate using X_train and y_train only.
The refitted best estimator becomes xgb_model.
The validation probabilities become xgb_scores for a single untouched comparison.
The printed cross-validated AP describes training-fold performance.

Confirm the output prints the best XGBoost parameters.
Confirm the output prints the best cross-validated AP.
Add a Markdown cell below the tuning output.
Record the selected parameters and cross-validated AP.
Explain why parameter tuning could change fraud ranking.
State the CPU search-time trade-off.

XGBoost search interrupted?

Confirm that the notebook still has an active runtime. A disconnected runtime loses fitted models held only in memory.

Rerun the pipeline-definition and search-configuration cells before retrying the fit.

Help me debug an interrupted XGBoost randomized search in Colab.

Fit Isolation Forest and compare validation results

An Isolation Forest learns the shape of legitimate training activity. Fraud-like validation rows should receive stronger anomaly scores after the native score direction is reversed.

Fit the legitimate-only anomaly detector by running this code in a new cell:

isolation_model = Pipeline(
    [
        ('preprocess', make_preprocessor()),
        (
            'model',
            IsolationForest(
                n_estimators=200,
                max_samples='auto',
                contamination='auto',
                n_jobs=-1,
                random_state=SEED,
            ),
        ),
    ]
)
isolation_model.fit(X_train.loc[y_train == 0])
isolation_scores = -isolation_model.decision_function(X_val)
validation_rows.append(metric_row('Isolation Forest', y_val, isolation_scores, threshold=0.0))

Why reverse the anomaly scores?

The fit uses only rows where y_train == 0.
The native decision function gives lower scores to more abnormal rows.
The leading minus sign reverses that direction so larger isolation_scores values mean more fraud-like.
The reversed values can now share the same ranking-metric direction as supervised fraud probabilities.

Confirm the Isolation Forest cell finishes without an exception.
Add a Markdown cell below the anomaly-detector result.
Record its validation Average Precision.
Explain why the anomaly-score sign was reversed.
State the trade-off of learning only from legitimate transactions.
Document that PCA components limit semantic feature interpretation.

Isolation Forest scoring failed?

Confirm that y_train uses integer labels so the legitimate-row filter selects class 0.

Confirm that X_val still contains the same 30 feature columns used for training.

Help me debug legitimate-only Isolation Forest fitting or score reversal.

Before you run the comparison, predict which model you expect to lead the Average Precision column.

Create the sorted validation table and save it to Drive by running this code:

comparison_df = pd.DataFrame(validation_rows).sort_values(
    'average_precision', ascending=False
)
comparison_df.to_csv(ARTIFACT_DIR / 'model_comparison.csv', index=False)
print(comparison_df)

What does the final table prove?

The table includes the Dummy majority baseline.
It includes all four Logistic Regression strategies.
It includes XGBoost and Isolation Forest.
Sorting by average_precision places the primary validation metric first.
The CSV preserves the complete validation comparison for threshold selection in the next step.

You should see seven rows sorted by validation Average Precision. The file /content/drive/My Drive/fraud-benchmark/artifacts/model_comparison.csv now contains the same comparison.

Comparison table missing models?

Confirm that validation_rows began with dummy_metrics before the four Logistic rows were appended.

Rerun the XGBoost and Isolation Forest cells if either result is absent from the printed table.

Help me find a missing row in my validation comparison table.

✔️ Awesome, I've got everything

Your comparison is complete. Confirm that Colab has saved the latest notebook changes to Drive.

ⓧ I'd like to double check the full code

from pathlib import Path
from importlib.metadata import version
import json
import sys

from google.colab import drive
import joblib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline
from imblearn.under_sampling import RandomUnderSampler
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import IsolationForest
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold, train_test_split
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier

SEED = 42
MIN_PRECISION = 0.20
FALSE_POSITIVE_COST = 1
FALSE_NEGATIVE_COST = 25

EXPECTED_COLUMNS = [
    "Time",
    *[f"V{i}" for i in range(1, 29)],
    "Amount",
    "Class",
]
FEATURES = EXPECTED_COLUMNS[:-1]
MODEL_FEATURE_ORDER = ["Time", "Amount", *[f"V{i}" for i in range(1, 29)]]

PACKAGE_NAMES = {
    "pandas": "pandas",
    "numpy": "numpy",
    "scikit-learn": "scikit-learn",
    "imbalanced-learn": "imbalanced-learn",
    "xgboost": "xgboost",
    "matplotlib": "matplotlib",
    "joblib": "joblib",
}

VERSIONS = {"python": sys.version.split()[0]}
VERSIONS.update({label: version(package) for label, package in PACKAGE_NAMES.items()})
print(pd.Series(VERSIONS, name="version"))

EXPECTED_VERSIONS = {
    "python": "3.13.18",
    "pandas": "2.3.3",
    "numpy": "2.5.4",
    "scikit-learn": "1.9.1",
    "imbalanced-learn": "0.14.2",
    "xgboost": "3.4.1",
    "matplotlib": "3.11.2",
    "joblib": "1.6.0",
}
if VERSIONS != EXPECTED_VERSIONS:
    raise RuntimeError(f"Version mismatch. Expected {EXPECTED_VERSIONS}, observed {VERSIONS}")

drive.mount('/content/drive')
ROOT = Path('/content/drive/My Drive/fraud-benchmark')
DATA_DIR = ROOT / 'data'
ARTIFACT_DIR = ROOT / 'artifacts'
FIGURE_DIR = ROOT / 'figures'
for folder in (DATA_DIR, ARTIFACT_DIR, FIGURE_DIR):
    folder.mkdir(parents=True, exist_ok=True)

DATA_PATH = DATA_DIR / 'creditcard.csv'
if not DATA_PATH.exists():
    raise FileNotFoundError(
        f"Place the Kaggle file at {DATA_PATH} and rerun this cell."
    )

df_raw = pd.read_csv(DATA_PATH)
if list(df_raw.columns) != EXPECTED_COLUMNS:
    raise ValueError(
        "Unexpected schema.\n"
        f"Expected: {EXPECTED_COLUMNS}\n"
        f"Observed: {list(df_raw.columns)}"
    )

missing_by_column = df_raw.isna().sum()
duplicate_later_rows = int(df_raw.duplicated(keep='first').sum())
duplicate_group_rows = int(df_raw.duplicated(keep=False).sum())
if int(missing_by_column.sum()) != 0:
    raise ValueError("Unexpected missing values found. Inspect before modeling.")

raw_class_counts = df_raw['Class'].value_counts().sort_index()
df = df_raw.loc[~df_raw.duplicated(keep='first')].copy()
model_class_counts = df['Class'].value_counts().sort_index()

audit = {
    "raw_rows": int(len(df_raw)),
    "raw_columns": int(df_raw.shape[1]),
    "raw_legitimate": int(raw_class_counts.get(0, 0)),
    "raw_fraud": int(raw_class_counts.get(1, 0)),
    "missing_cells": int(missing_by_column.sum()),
    "later_duplicate_rows_removed": duplicate_later_rows,
    "all_rows_in_duplicate_groups": duplicate_group_rows,
    "modeling_rows": int(len(df)),
    "modeling_legitimate": int(model_class_counts.get(0, 0)),
    "modeling_fraud": int(model_class_counts.get(1, 0)),
    "duplicate_decision": (
        "Preserve the raw DataFrame but remove later exact duplicates from the "
        "modeling copy to reduce identical rows crossing data splits."
    ),
    "duplicate_limitation": (
        "No transaction identifier exists, so some exact repeats could represent "
        "separate real transactions."
    ),
}
(ARTIFACT_DIR / 'data_audit.json').write_text(
    json.dumps(audit, indent=2), encoding='utf-8'
)
df.describe().T.to_csv(ARTIFACT_DIR / 'descriptive_statistics.csv')
print(json.dumps(audit, indent=2))
print(df.dtypes)

fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(['Legitimate', 'Fraud'], [audit['modeling_legitimate'], audit['modeling_fraud']])
ax.set_yscale('log')
ax.set_title('Class distribution after exact-duplicate handling')
ax.set_ylabel('Transactions, logarithmic scale')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'class_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(np.log1p(df['Amount']), bins=60)
ax.set_title('Transaction amount distribution')
ax.set_xlabel('log1p(Amount), visualization only')
ax.set_ylabel('Transactions')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'amount_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(df['Time'] / 3600, bins=48)
ax.set_title('Transactions across elapsed hours')
ax.set_xlabel('Hours since the first transaction')
ax.set_ylabel('Transactions')
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'time_distribution.png', dpi=160, bbox_inches='tight')
plt.show()

X = df[FEATURES].copy()
y = df['Class'].astype(int).copy()

X_train_val, X_test, y_train_val, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=SEED,
    stratify=y,
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.25,
    random_state=SEED,
    stratify=y_train_val,
)

split_counts = pd.DataFrame(
    {
        "train": y_train.value_counts().sort_index(),
        "validation": y_val.value_counts().sort_index(),
        "test_sealed": y_test.value_counts().sort_index(),
    }
).fillna(0).astype(int)
print(split_counts)
split_counts.to_csv(ARTIFACT_DIR / 'split_class_counts.csv')


def metric_row(name, y_true, scores, threshold=0.5):
    predictions = (scores >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(
        y_true, predictions, labels=[0, 1]
    ).ravel()
    return {
        "model": name,
        "threshold": float(threshold),
        "accuracy": float((predictions == np.asarray(y_true)).mean()),
        "precision": float(precision_score(y_true, predictions, zero_division=0)),
        "recall": float(recall_score(y_true, predictions, zero_division=0)),
        "f1": float(f1_score(y_true, predictions, zero_division=0)),
        "roc_auc": float(roc_auc_score(y_true, scores)),
        "average_precision": float(average_precision_score(y_true, scores)),
        "tn": int(tn),
        "fp": int(fp),
        "fn": int(fn),
        "tp": int(tp),
    }


dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
dummy_scores = dummy.predict_proba(X_val)[:, 1]
dummy_metrics = metric_row('Dummy majority baseline', y_val, dummy_scores)
print(pd.DataFrame([dummy_metrics]).T)


def make_preprocessor():
    return ColumnTransformer(
        transformers=[('scale', StandardScaler(), ['Time', 'Amount'])],
        remainder='passthrough',
        verbose_feature_names_out=False,
    )


logistic_specs = {
    "Logistic - no correction": {"sampler": None, "class_weight": None},
    "Logistic - class weight": {"sampler": None, "class_weight": 'balanced'},
    "Logistic - SMOTE": {"sampler": SMOTE(random_state=SEED), "class_weight": None},
    "Logistic - undersampling": {
        "sampler": RandomUnderSampler(random_state=SEED),
        "class_weight": None,
    },
}

logistic_models = {}
logistic_scores = {}
validation_rows = [dummy_metrics]

for name, spec in logistic_specs.items():
    steps = [('preprocess', make_preprocessor())]
    if spec['sampler'] is not None:
        steps.append(('sampler', spec['sampler']))
    steps.append(
        (
            'model',
            LogisticRegression(
                class_weight=spec['class_weight'],
                solver='lbfgs',
                max_iter=1000,
            ),
        )
    )
    model = Pipeline(steps)
    model.fit(X_train, y_train)
    scores = model.predict_proba(X_val)[:, 1]
    logistic_models[name] = model
    logistic_scores[name] = scores
    validation_rows.append(metric_row(name, y_val, scores))

best_logistic_name = max(
    logistic_scores,
    key=lambda name: average_precision_score(y_val, logistic_scores[name]),
)
best_logistic_model = logistic_models[best_logistic_name]
best_logistic_scores = logistic_scores[best_logistic_name]
print(f"Best Logistic Regression strategy: {best_logistic_name}")

negative_count = int((y_train == 0).sum())
positive_count = int((y_train == 1).sum())
training_ratio = negative_count / positive_count

xgb_pipeline = Pipeline(
    [
        ('preprocess', make_preprocessor()),
        (
            'model',
            XGBClassifier(
                objective='binary:logistic',
                tree_method='hist',
                device='cpu',
                random_state=SEED,
                n_jobs=1,
            ),
        ),
    ]
)

parameter_space = {
    'model__n_estimators': [150, 250],
    'model__max_depth': [3, 5],
    'model__learning_rate': [0.05, 0.10],
    'model__subsample': [0.80, 1.00],
    'model__colsample_bytree': [0.80, 1.00],
    'model__scale_pos_weight': [1.0, training_ratio],
}
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=SEED)
xgb_search = RandomizedSearchCV(
    estimator=xgb_pipeline,
    param_distributions=parameter_space,
    n_iter=4,
    scoring='average_precision',
    n_jobs=-1,
    cv=cv,
    refit=True,
    random_state=SEED,
    verbose=1,
)
xgb_search.fit(X_train, y_train)
xgb_model = xgb_search.best_estimator_
xgb_scores = xgb_model.predict_proba(X_val)[:, 1]
validation_rows.append(metric_row('XGBoost', y_val, xgb_scores))
print('Best XGBoost parameters:', xgb_search.best_params_)
print('Best cross-validated AP:', float(xgb_search.best_score_))

isolation_model = Pipeline(
    [
        ('preprocess', make_preprocessor()),
        (
            'model',
            IsolationForest(
                n_estimators=200,
                max_samples='auto',
                contamination='auto',
                n_jobs=-1,
                random_state=SEED,
            ),
        ),
    ]
)
isolation_model.fit(X_train.loc[y_train == 0])
isolation_scores = -isolation_model.decision_function(X_val)
validation_rows.append(metric_row('Isolation Forest', y_val, isolation_scores, threshold=0.0))

comparison_df = pd.DataFrame(validation_rows).sort_values(
    'average_precision', ascending=False
)
comparison_df.to_csv(ARTIFACT_DIR / 'model_comparison.csv', index=False)
print(comparison_df)

How to use this reference

Compare the cells in your notebook with this cumulative reference through the validation comparison. Keep X_test and y_test sealed for the final evaluation step.

You now have a leakage-safe validation benchmark covering four imbalance strategies and two additional fraud-modeling approaches. Next, you will select the stronger supervised pipeline and freeze its operating threshold before opening the test set once.

Freeze, Test, and Export the Benchmark

Your Google Colab notebook now compares four Logistic Regression strategies with XGBoost and Isolation Forest on validation data. The untouched test partition remains sealed.

A fraud score becomes an operational decision only after you freeze a threshold. In this step, you will make every remaining choice from validation data before opening the test set once.

In this step, get ready to:

Select the strongest supervised pipeline using validation Average Precision.
Freeze a validation-selected threshold before evaluating the sealed test set once.
Export the benchmark bundle, metrics, figures, examples, and documentation to Drive.

Select the model and freeze the threshold

Average Precision measures ranking quality across thresholds. Your selected pipeline must have the stronger validation Average Precision among the best Logistic Regression strategy and XGBoost.

The operating threshold answers a separate question about which scores trigger a fraud decision. Both threshold choices use validation predictions so the test set remains independent.

Add a code cell below the validation comparison output.
Select the stronger supervised pipeline by running the code below:

supervised_candidates = {
    best_logistic_name: (best_logistic_model, best_logistic_scores),
    'XGBoost': (xgb_model, xgb_scores),
}
final_name = max(
    supervised_candidates,
    key=lambda name: average_precision_score(
        y_val, supervised_candidates[name][1]
    ),
)
final_model, validation_scores = supervised_candidates[final_name]
print(f"Selected supervised model by validation AP: {final_name}")

What does this selection do?

The candidate dictionary keeps the best Logistic Regression pipeline beside XGBoost.
The selection function compares validation Average Precision only.
The chosen pipeline becomes final_model. Its existing validation probabilities become validation_scores.

Run the cell.

You will see the selected supervised model printed beneath the cell. The test partition has not influenced that choice.

Selection cell not running?

Confirm the earlier model-comparison cells still define best_logistic_model and xgb_model.
Rerun the validation comparison cell if the Colab runtime has lost its in-memory variables.

Help me restore the supervised model selection variables in my Colab notebook.

The first threshold maximizes recall while meeting the illustrative minimum precision of 0.20 when possible. This creates a transparent operating point for discussing the precision and recall trade-off.

Add the validation threshold calculations in a new code cell by running:

precision_values, recall_values, thresholds = precision_recall_curve(
    y_val, validation_scores
)
valid_precision_indices = [
    index
    for index, value in enumerate(precision_values[:-1])
    if value >= MIN_PRECISION
]
if valid_precision_indices:
    operating_index = max(
        valid_precision_indices,
        key=lambda index: recall_values[index],
    )
else:
    operating_index = int(np.argmax(precision_values[:-1]))
operating_threshold = float(thresholds[operating_index])

costs = []
for threshold in thresholds:
    predictions = (validation_scores >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(
        y_val, predictions, labels=[0, 1]
    ).ravel()
    costs.append(FALSE_POSITIVE_COST * fp + FALSE_NEGATIVE_COST * fn)
cost_index = int(np.argmin(costs))
frozen_threshold = float(thresholds[cost_index])

How are the thresholds chosen?

The precision-recall curve supplies candidate thresholds from validation scores.
The operating point favors the largest available recall among candidates meeting the minimum precision assumption.
The cost calculation assigns 1 unit to each false positive.
The same calculation assigns 25 units to each false negative.
The least expensive validation candidate becomes frozen_threshold.

Run the cell.
Confirm the cell completes without an exception.

Why are these costs illustrative?

The false-positive cost of 1 and false-negative cost of 25 are classroom assumptions. They provide a repeatable threshold exercise without claiming to represent bank policy.

A real deployment needs costs grounded in investigation effort, customer impact, fraud losses, and regulatory obligations. The threshold is defensible only within its stated assumptions.

Add the threshold summary export in a new code cell by running:

threshold_summary = {
    "selected_model": final_name,
    "minimum_precision_assumption": MIN_PRECISION,
    "max_recall_at_minimum_precision_threshold": operating_threshold,
    "max_recall_at_minimum_precision": float(recall_values[operating_index]),
    "precision_at_operating_threshold": float(precision_values[operating_index]),
    "false_positive_cost_assumption": FALSE_POSITIVE_COST,
    "false_negative_cost_assumption": FALSE_NEGATIVE_COST,
    "frozen_cost_minimizing_threshold": frozen_threshold,
    "minimum_validation_cost": int(costs[cost_index]),
}
(ARTIFACT_DIR / 'threshold.json').write_text(
    json.dumps(threshold_summary, indent=2), encoding='utf-8'
)
print(json.dumps(threshold_summary, indent=2))

What does the threshold summary preserve?

The summary records the selected model and both validation operating points.
The summary stores the precision requirement and cost assumptions beside the threshold.
The JSON file gives future users enough context to interpret the saved cutoff.

Run the cell.

You will see the threshold summary printed beneath the cell. You will also have threshold.json in the artifacts folder.

Threshold summary missing?

Confirm the Drive mount is still active before writing the JSON file.
Confirm the threshold calculation cell completed before this export cell.

Help me diagnose why threshold.json was not written to my fraud-benchmark artifacts folder.

Saving the validation predictions makes the threshold decision reproducible. The same scores also support a direct visual comparison of supervised and anomaly-detection rankings.

Save the selected validation scores and draw the model comparison curves by running:

validation_predictions = pd.DataFrame(
    {"true_class": y_val, "fraud_score": validation_scores}
)
validation_predictions.to_csv(
    ARTIFACT_DIR / 'validation_predictions.csv', index=False
)

fig, ax = plt.subplots(figsize=(7, 5))
curve_scores = {
    best_logistic_name: best_logistic_scores,
    'XGBoost': xgb_scores,
    'Isolation Forest': isolation_scores,
}
for name, scores in curve_scores.items():
    curve_precision, curve_recall, _ = precision_recall_curve(y_val, scores)
    ap = average_precision_score(y_val, scores)
    ax.plot(curve_recall, curve_precision, label=f"{name}, AP={ap:.3f}")
ax.set_title('Validation Precision-Recall curves')
ax.set_xlabel('Recall')
ax.set_ylabel('Precision')
ax.legend()
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'validation_pr_curves.png', dpi=160, bbox_inches='tight')
plt.show()

What does this comparison show?

The CSV stores each validation label beside the selected supervised fraud score.
The chart compares the strongest Logistic Regression strategy with XGBoost and Isolation Forest.
The Isolation Forest curve uses the reversed anomaly scores from the previous step.
Each legend entry reports AP without describing it as trapezoidal PR area.

Run the cell.

You will see three validation Precision-Recall curves with an AP value in each label. The figure is saved as validation_pr_curves.png.

Curve missing a model?

Confirm best_logistic_scores and xgb_scores still contain validation probabilities.
Confirm isolation_scores contains reversed decision scores for the same validation rows.

Help me find why one validation Precision-Recall curve is missing from my fraud benchmark.

The next figure places both threshold decisions on the selected model's validation curve. This separates the minimum-precision operating point from the frozen cost-minimizing threshold.

Plot both validation-selected operating points by running:

fig, ax = plt.subplots(figsize=(7, 5))
ax.plot(recall_values, precision_values, label=final_name)
ax.scatter(
    [recall_values[operating_index]],
    [precision_values[operating_index]],
    label='Max recall at minimum precision',
)
ax.scatter(
    [recall_values[cost_index]],
    [precision_values[cost_index]],
    label='Frozen cost-minimizing threshold',
)
ax.set_title('Validation threshold choices')
ax.set_xlabel('Recall')
ax.set_ylabel('Precision')
ax.legend()
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'threshold_selection.png', dpi=160, bbox_inches='tight')
plt.show()

How should you read this figure?

One marker shows the maximum-recall validation point meeting the minimum precision assumption. The second marker shows the validation point minimizing the illustrative error cost.

Their positions can differ because a precision constraint and a cost function answer different operational questions.

Run the cell.

You will see two labeled points on the validation curve. The saved threshold_selection.png now documents why the benchmark uses its frozen threshold.

Open the test set once

The selected model and threshold are now frozen. The sealed test partition can provide one final estimate without shaping any modeling decision.

This is the one deliberate reveal. Every intervention, model choice, and operating threshold is already fixed.

Before you run the evaluation, do you expect the test metrics to match the validation metrics exactly?

Generate the test probabilities for the first and only time by running:

print('ONE-TIME TEST EVALUATION')
test_scores = final_model.predict_proba(X_test)[:, 1]
test_metrics = metric_row(
    final_name,
    y_test,
    test_scores,
    threshold=frozen_threshold,
)
print(pd.DataFrame([test_metrics]).T)
(ARTIFACT_DIR / 'test_metrics.json').write_text(
    json.dumps(test_metrics, indent=2), encoding='utf-8'
)

What does the one-time evaluation report?

Average Precision and ROC-AUC measure ranking quality on held-out rows.
Accuracy, precision, recall, and F1 use the frozen validation-selected threshold.
TN, FP, FN, and TP preserve the confusion-matrix counts behind those threshold metrics.
The JSON export records the selected model and threshold beside the final results.

Run the cell once.

You will see ONE-TIME TEST EVALUATION followed by the held-out metrics. The values may differ from validation because the partitions contain different transactions.

Test evaluation interrupted?

Confirm final_model and frozen_threshold still exist in memory.
Rerun the model-selection and threshold cells if the runtime restarted.
Avoid changing the model or threshold after viewing the test results.

Help me recover an interrupted one-time test evaluation without tuning against the test results.

A confusion matrix turns the threshold metrics into visible error counts. It shows how many legitimate transactions were flagged and how many fraud cases were missed.

Create the held-out confusion-matrix figure by running:

test_predictions = (test_scores >= frozen_threshold).astype(int)
test_confusion = confusion_matrix(y_test, test_predictions, labels=[0, 1])
fig, ax = plt.subplots(figsize=(5, 4))
image = ax.imshow(test_confusion, cmap='Blues')
for row in range(2):
    for column in range(2):
        ax.text(column, row, int(test_confusion[row, column]), ha='center', va='center')
ax.set_xticks([0, 1], labels=['Legitimate', 'Fraud'])
ax.set_yticks([0, 1], labels=['Legitimate', 'Fraud'])
ax.set_xlabel('Predicted class')
ax.set_ylabel('True class')
ax.set_title('One-time test confusion matrix')
fig.colorbar(image, ax=ax)
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'test_confusion_matrix.png', dpi=160, bbox_inches='tight')
plt.show()

What does the matrix preserve?

The rows represent the true class. The columns represent the class predicted at the frozen threshold.

The off-diagonal cells are the false positives and false negatives that deserve error review.

Run the cell.

You will see the four held-out confusion-matrix counts. The same image is saved as test_confusion_matrix.png.

The aggregate counts reveal how often the benchmark fails. The error table preserves the corresponding held-out rows for later review without assigning business meaning to anonymous PCA components.

Export the false-positive and false-negative rows by running:

error_table = X_test.copy()
error_table['true_class'] = y_test
error_table['fraud_score'] = test_scores
error_table['predicted_class'] = test_predictions
error_table['error_type'] = 'correct'
error_table.loc[
    (error_table['true_class'] == 0) & (error_table['predicted_class'] == 1),
    'error_type',
] = 'false_positive'
error_table.loc[
    (error_table['true_class'] == 1) & (error_table['predicted_class'] == 0),
    'error_type',
] = 'false_negative'
error_table.loc[error_table['error_type'] != 'correct'].sort_values(
    'fraud_score', ascending=False
).to_csv(ARTIFACT_DIR / 'test_errors.csv', index=False)

Why save only the errors?

The table adds the true class, fraud score, predicted class, and error type to each held-out mistake. Sorting by score places the most fraud-like flagged rows first.

The export supports review while keeping the one-time evaluation unchanged.

Run the cell.
Return to the existing artifacts folder in Drive.
Confirm test_errors.csv is listed.

Error table empty?

An empty table means the frozen threshold produced no false positives or false negatives on the held-out rows. Compare that result with the printed confusion matrix before assuming the export failed.

Help me verify whether my empty test error table is consistent with the confusion matrix.

Feature importance describes which anonymized inputs most influenced the selected model. Its interpretation depends on whether XGBoost or Logistic Regression won.

Calculate the selected model's feature signals by running:

if final_name == 'XGBoost':
    importance_values = final_model.named_steps['model'].feature_importances_.tolist()
    importance_label = 'gain-based model importance'
else:
    importance_values = [
        abs(float(value))
        for value in final_model.named_steps['model'].coef_[0].tolist()
    ]
    importance_label = 'absolute logistic coefficient'

importance_df = pd.DataFrame(
    {
        'feature': MODEL_FEATURE_ORDER,
        'importance': importance_values,
        'importance_type': importance_label,
    }
).sort_values('importance', ascending=False)
importance_df.to_csv(ARTIFACT_DIR / 'feature_importance.csv', index=False)

Why does the importance type change?

XGBoost exposes tree-based feature importance. Logistic Regression exposes coefficients whose absolute values indicate signal strength after preprocessing.

The PCA components remain anonymous. Importance can identify influential inputs without supplying a business interpretation for them.

Run the cell.
Confirm feature_importance.csv appears in the Drive artifacts folder.

Plot the 15 strongest feature signals by running:

fig, ax = plt.subplots(figsize=(8, 6))
top_importance = importance_df.head(15).sort_values('importance')
ax.barh(top_importance['feature'], top_importance['importance'])
ax.set_title(f'Top feature signals: {final_name}')
ax.set_xlabel(importance_label)
fig.tight_layout()
fig.savefig(FIGURE_DIR / 'feature_importance.png', dpi=160, bbox_inches='tight')
plt.show()

What does the feature chart show?

The chart displays the 15 largest selected-model signals using the applicable importance definition. It supports model discussion without assigning invented meanings to V1 through V28.

Run the cell.

You will see a horizontal chart titled with the selected model. The figure is saved as feature_importance.png.

Export up to three legitimate examples and three fraud examples by running:

demo_rows = pd.concat(
    [
        error_table.loc[error_table['true_class'] == 0].head(3),
        error_table.loc[error_table['true_class'] == 1].head(3),
    ]
)
demo_rows.to_csv(ARTIFACT_DIR / 'demo_examples.csv', index=False)

Why export held-out demo rows?

The examples let a follow-on application demonstrate predictions without selecting rows from training data. The export includes scores and predicted classes produced during the one-time evaluation.

Run the cell.
Confirm demo_examples.csv appears in the Drive artifacts folder.

Package and verify the benchmark

Model persistence packages the fitted preprocessing pipeline with its selected threshold and input contract. The bundle also records the environment needed to reproduce its predictions.

The bundle uses joblib persistence. Load only the artifact created in this trusted Drive folder because pickle-based files can execute code when loaded.

Create the trusted model bundle and verify its first five validation probabilities by running:

bundle = {
    "model": final_model,
    "threshold": frozen_threshold,
    "selected_model": final_name,
    "input_features": FEATURES,
    "model_feature_order": MODEL_FEATURE_ORDER,
    "class_labels": {0: "legitimate", 1: "fraud"},
    "cost_assumptions": {
        "false_positive": FALSE_POSITIVE_COST,
        "false_negative": FALSE_NEGATIVE_COST,
    },
    "versions": VERSIONS,
    "dataset_source": "https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud/data",
    "warning": "Demo only. Do not use as a banking control.",
}
bundle_path = ARTIFACT_DIR / 'fraud_model_bundle.joblib'
joblib.dump(bundle, bundle_path, compress=3)
restored_bundle = joblib.load(bundle_path)
restored_scores = restored_bundle['model'].predict_proba(X_val.head(5))[:, 1]
if not np.allclose(restored_scores, validation_scores[:5]):
    raise RuntimeError('Reloaded bundle predictions do not match.')

What does the bundle contain?

The fitted pipeline keeps preprocessing and prediction behavior together.
The threshold converts supervised probabilities into the benchmark's operating decision.
The feature order and class labels define the input and output contracts.
The assumptions and dependency versions preserve the conditions behind the result.
The reload check proves the restored pipeline reproduces the first five saved validation probabilities.

Run the cell.

The cell will finish without a RuntimeError when the restored probabilities match. Your trusted fraud_model_bundle.joblib is now ready for the follow-on application.

Reloaded predictions do not match?

Confirm the bundle stores final_model after supervised selection.
Confirm validation_scores still belongs to the same selected model.
Confirm the current environment still uses the pinned dependency versions recorded in VERSIONS.

Help me diagnose why my trusted joblib bundle does not reproduce the saved validation probabilities.

The README turns the binary bundle into a handoff artifact. It defines the required columns and explains how a future application should apply the saved threshold.

Write the reproduction checklist and limitations by running:

readme = f"""# Fraud benchmark artifact

Selected model: {final_name}
Frozen threshold: {frozen_threshold:.8f}

## Input contract
- Required columns in this order: {FEATURES}
- Every input value must be numeric.
- `Class` is not an input feature.

## Reproduction checklist
- Use the dependency versions in `fraud_model_bundle.joblib`.
- Load only a bundle created by your team and stored in trusted Drive space.
- Keep preprocessing inside the saved pipeline.
- Apply the saved threshold after `predict_proba(... )[:, 1]`.
- Do not retrain inside the future Streamlit application.

## Limitations
- Random stratified splits do not simulate future transaction drift.
- PCA-anonymized features limit business interpretation.
- Duplicate removal may discard separate transactions with identical recorded values.
- False-positive and false-negative costs are classroom assumptions.
- This is a demonstration artifact, not a banking control.
"""
(ROOT / 'README.md').write_text(readme, encoding='utf-8')

Why does the README matter?

The input contract prevents a future application from silently changing column order. The checklist keeps preprocessing inside the fitted pipeline and applies the saved threshold after probability prediction.

The limitations state that random splitting does not simulate future drift. They also preserve the duplicate, anonymity, cost, and demo-only caveats.

Run the cell.
Confirm README.md appears directly inside the existing fraud-benchmark Drive folder.

The report template creates a boundary between computed evidence and written interpretation. Every result prompt must be completed from notebook output after the final run.

Create the report-notes template by running:

report_notes = """# Report Notes

## Objective
[INSERT RESULT: one sentence describing the fraud-screening decision]

## Data audit
[INSERT RESULT: observed rows, fraud count, missing cells, and duplicates removed]

## Exploratory findings
[INSERT RESULT: class imbalance, amount skew, and elapsed-time pattern]

## Imbalance ablation
[INSERT RESULT: which Logistic Regression strategy had the strongest validation AP and why]

## Model comparison
[INSERT RESULT: validation AP and ROC-AUC for Logistic Regression, XGBoost, and Isolation Forest]

## Threshold decision
[INSERT RESULT: frozen threshold, precision, recall, and illustrative cost assumptions]

## One-time test result
[INSERT RESULT: AP, ROC-AUC, precision, recall, F1, FP, and FN]

## Error analysis
[INSERT RESULT: patterns observed in false positives and false negatives without assigning meaning to PCA components]

## Limitations and ethics
[INSERT RESULT: drift, anonymization, random split, duplicate decision, cost assumptions, privacy, and demo-only warning]
"""
(ROOT / 'report_notes.md').write_text(report_notes, encoding='utf-8')

What belongs in the report notes?

Each section points to evidence generated by the notebook. The prompts prevent invented model metrics or unsupported meanings for the PCA components.

Complete the template only with values from the final run. Keep the drift, duplicate-handling, cost, privacy, and demo-only limitations visible.

Run the cell.
Confirm report_notes.md appears directly inside the existing fraud-benchmark Drive folder.

Before the final check, predict which files the recursive artifact listing should include. Think about the bundle, metrics, figures, examples, README, and report template.

Print every exported file path by running:

print('Artifacts created:')
for path in sorted(ROOT.rglob('*')):
    if path.is_file():
        print(path)

What does this final check prove?

The recursive listing checks the persistent Drive folder instead of relying on in-memory variables. It confirms the handoff package survived every export step.

Run the cell.

You will see Artifacts created: followed by paths for the validation predictions, threshold summary, test metrics, error rows, feature importance, demo examples, model bundle, figures, README, and report notes.

An exported path is missing?

Locate the cell responsible for the missing artifact.
Rerun only that export cell.
Run the artifact-listing cell again to confirm the path now appears.

Help me identify which export cell creates a missing fraud benchmark artifact.

That is the benchmark frozen. Your selected pipeline, operating threshold, one-time test evidence, and deployment handoff files now persist together in Drive.

Secret mission

Stress-Test the Operating Threshold

A frozen threshold can look decisive until the assumed cost of missed fraud changes. Stress-test four false-negative costs to reveal whether your operating decision is stable enough to defend.

Clean Up Your Resources

Choose whether to keep the Google Drive files, pause the temporary runtime, or delete the folder entirely. This project uses only standard Google Colab at no charge under dynamic resource and usage limits.

Resources you used:

The Google Drive folder at /content/drive/My Drive/fraud-benchmark, including data, artifacts, figures, README.md, and report_notes.md.

Keep everything running

No action needed. Choose this if you plan to use the frozen bundle in the follow-on project.

Keep /content/drive/My Drive/fraud-benchmark unchanged in Google Drive.
Load fraud_model_bundle.joblib from artifacts when you begin the follow-on project.

Pause - I'll come back to this later

Disconnect the Colab runtime to release temporary compute while preserving every Drive artifact.

In fraud_benchmark.ipynb, disconnect the current Colab runtime using the runtime connection control.
Confirm that the notebook no longer has an active runtime connection.

Your Drive artifacts remain unchanged. The installed packages and in-memory variables disappear with the temporary virtual machine.

Delete - I don't want to use this again

Remove the saved benchmark folder if you no longer need its data or exports. Emptying Drive trash makes this cleanup permanent, so copy anything you want to keep first.

Return to Google Drive.
In My Drive, locate the fraud-benchmark folder.
Use the folder context menu to move fraud-benchmark to Drive trash.
Open Drive trash.
Empty Drive trash after confirming that no other files there need to be kept.
Search Drive for fraud-benchmark.

You should see no matching folder. The raw CSV, frozen model, figures, reports, and threshold-sensitivity outputs are now removed from Drive.

Nice Work!

Brilliant work! You built a reproducible fraud-detection benchmark in Google Colab. You can now defend its evaluation choices in a report or viva.

You've learned how to:

Complete a data audit that validates the expected schema. Use saved figures to expose class imbalance. Demonstrate the accuracy trap with a majority baseline that returns 0.0 fraud recall.
Build leakage-safe pipelines for four Logistic Regression strategies. Compare their validation Average Precision with XGBoost and Isolation Forest.
Freeze a validation-selected operating threshold. Complete a one-time test evaluation. Export the trusted fraud_model_bundle.joblib handoff with metrics, figures, report notes, and safe demo examples.
Secret Mission: Run a cost sensitivity analysis for false-negative costs 5, 10, 25, and 50. Show whether the operating threshold remains stable across the four validation scenarios.

Ready to quiz yourself?