Credit Card Fraud Detection

Build an XGBoost fraud detector with SHAP explainability and threshold tuning.

Introduction

30 Second Summary

Every time you tap your card at a coffee shop, a fraud detection system decides in milliseconds whether that transaction is really you. These systems need to catch criminals without accidentally blocking your morning latte.

In this project, you will build a fraud detection model that exposes why "99% accuracy" is a trap, then fix it with cost-sensitive learning, business-oriented threshold tuning, and explainability visualizations that make your model decisions auditable.

What You'll Build

A polished Jupyter notebook where you run a single cell and watch your model flag fraudulent transactions in real time, then explain exactly why each one was flagged using interactive SHAP plots.

By the end of this project, you'll have:

  • A baseline model that proves high accuracy can be meaningless. You run it, see 99.9% accuracy, then discover it misses most fraud.
  • A production-ready XGBoost fraud detector that catches 80%+ of fraud using a single parameter instead of complex synthetic data pipelines.
  • A cost-benefit analysis where you pick a detection threshold and instantly see how many dollars it saves or costs the business.
  • A SHAP explainability dashboard that shows non-technical stakeholders exactly which features drove each fraud prediction.
  • Secret Mission: Build a threshold comparison dashboard that visualizes the business impact of different detection thresholds side by side.

Are there any prerequisites?

You should be comfortable with basic Python syntax and have used pandas DataFrames before. Prior exposure to ML model training helps, even if you didn't fully understand every step.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for a data science role.

Set Up and Load the Data

Fraud detection starts with understanding the data you are working with. The Kaggle Credit Card Fraud dataset is the industry benchmark for imbalanced classification. It contains 284,807 real transactions from European cardholders, with only 492 labeled as fraud.

In this step, you will download the dataset, load it into a Jupyter Notebook, and visualize just how extreme the class imbalance is. You will also scale the features and split the data so it is ready for model training in the next step.

In this step, get ready to:
  • Download the Kaggle Credit Card Fraud dataset and create a Jupyter notebook.
  • Load and explore the dataset to reveal the class distribution.
  • Visualize the imbalance, scale features, and split into train/test sets.
Download the dataset and create your notebook

The dataset lives on Kaggle and requires a free account to download. The CSV file is about 150 MB containing 31 columns of transaction data.

  • Go to the Kaggle Credit Card Fraud Detection dataset page:
  • https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud
  • Click Download to download the ZIP file.
  • Extract the ZIP file to get creditcard.csv.
  • Move creditcard.csv to your Desktop (or any folder you will use as your working directory).

What is in this dataset?

The dataset contains 284,807 credit card transactions from September 2013. Features V1 through V28 are PCA-transformed (anonymized for privacy). The only non-PCA features are Time (seconds since the first transaction) and Amount (the transaction amount).

The Class column is the target: 0 for legitimate and 1 for fraud.

  • Open Jupyter Notebook by pressing Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Jupyter Notebook and press Enter to launch it in your browser.
  • Navigate to the folder where you placed creditcard.csv.
  • Click New in the top right, then select Python 3 to create a new notebook.
  • Rename the notebook to fraud_detection.ipynb by clicking on the title at the top of the page.

Now you have a blank notebook sitting in the same folder as your dataset. Time to import the libraries.

  • Add the following imports to the first cell of your notebook:
# Cell 1: Imports and Setup
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    accuracy_score,
    ConfusionMatrixDisplay
)
import xgboost as xgb
import shap

What does this code do?

  • pandas and numpy handle data loading and numerical operations.
  • matplotlib produces charts and visualizations.
  • sklearn provides model training, evaluation metrics, and preprocessing tools.
  • xgboost is the gradient boosting library you will use for the production model.
  • shap generates explainability visualizations for model predictions.
  • Run the cell by pressing Shift+Enter.

The cell should complete with no output and no errors. If you see an import error, install the missing package with pip install <package-name> in a new cell, then re-run the imports.

Seeing an ImportError?

If a library is missing, create a new cell above and run the install command. For example: !pip install xgboost shap. After installing, restart the kernel (Kernel > Restart) and re-run the imports cell.

help me fix this ImportError in my Jupyter notebook

Load and explore the dataset

With your imports ready, load the CSV and inspect its structure. This is where the class imbalance becomes visible in the numbers.

  • Create a new cell below and add this code to load the dataset and print its key statistics:
# Cell 2: Load Dataset
df = pd.read_csv('creditcard.csv')
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
print(f"\nNull values: {df.isnull().sum().sum()}")
print(f"\nClass distribution:")
print(df['Class'].value_counts())
print(f"\nFraud percentage: {df['Class'].mean() * 100:.3f}%")

What does this code do?

  • pd.read_csv('creditcard.csv') loads the entire dataset into a DataFrame called df.
  • df.shape tells you the total rows and columns (284,807 x 31).
  • df.isnull().sum().sum() checks for missing data across the entire dataset.
  • df['Class'].value_counts() shows the count of legitimate vs. fraud transactions.
  • Run the cell with Shift+Enter.

You should see the dataset has 284,807 rows and 31 columns. There are zero null values. The class distribution shows 284,315 legitimate transactions and only 492 frauds (0.172%).

Seeing a FileNotFoundError?

This means creditcard.csv is not in the same folder as your notebook. Check that you extracted the ZIP file and that the CSV lives in the same directory where Jupyter launched your notebook.

help me fix a FileNotFoundError when loading creditcard.csv in Jupyter

Visualize the imbalance and prepare features

Numbers alone do not convey how extreme 492 out of 284,807 truly is. A bar chart makes the imbalance impossible to ignore.

  • Create a new cell and add this code to visualize the class distribution:
# Cell 3: Visualize Class Imbalance
fig, ax = plt.subplots(figsize=(6, 4))
df['Class'].value_counts().plot(kind='bar', ax=ax, color=['steelblue', 'coral'])
ax.set_xticklabels(['Legitimate (0)', 'Fraud (1)'], rotation=0)
ax.set_ylabel('Count')
ax.set_title('Transaction Class Distribution')
for i, v in enumerate(df['Class'].value_counts().values):
    ax.text(i, v + 1000, str(v), ha='center', fontweight='bold')
plt.tight_layout()
plt.show()

What does this code do?

  • value_counts().plot(kind='bar') creates a bar chart of the two classes.
  • The for loop adds the exact count above each bar so the numbers are visible at a glance.
  • The legitimate bar towers over the fraud bar, making the 578:1 ratio viscerally obvious.
  • Run the cell with Shift+Enter.

You should see a bar chart where the legitimate bar dominates with 284,315 transactions, and the fraud bar is barely visible at 492. This is the core challenge of fraud detection.

Now you need to scale the raw features and split the data for training. The Amount and Time columns are on a different scale than the PCA features, so you will standardize them.

  • Create a new cell and add this code to scale features and split the data:
# Cell 4: Feature Scaling and Train/Test Split
scaler = StandardScaler()
df['Amount_scaled'] = scaler.fit_transform(df[['Amount']])
df['Time_scaled'] = scaler.fit_transform(df[['Time']])

X = df.drop(columns=['Class', 'Amount', 'Time'])
y = df['Class']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"Fraud in training: {y_train.sum()} ({y_train.mean()*100:.3f}%)")
print(f"Fraud in test: {y_test.sum()} ({y_test.mean()*100:.3f}%)")

What does this code do?

  • StandardScaler() normalizes Amount and Time to have mean 0 and standard deviation 1, matching the PCA features.
  • X drops the raw Amount, Time, and Class columns, keeping V1-V28 plus the new scaled columns.
  • stratify=y ensures the 80/20 train/test split preserves the 0.172% fraud ratio in both sets. Without this, the test set could end up with zero fraud cases by random chance.

Before you run this cell, think about what the fraud percentage should be in both the training and test sets. Should they match the original 0.172%, or will they differ?

  • Run the cell with Shift+Enter.

You should see that both the training set and test set maintain approximately 0.172% fraud. That is stratify=y doing its job, preserving the class ratio across both sets.

Seeing different fraud percentages in train and test?

If the percentages differ significantly, make sure you included stratify=y in the train_test_split call. Without it, the random split may not preserve the class balance.

help me fix my train_test_split so both sets have the same fraud ratio

✔️ Awesome, I've got everything!

Great. Double check that you have saved your notebook with all four cells.

ⓧ I'd like to double check the full code

Scroll up and confirm you have these four cells in order, each running without errors:

  • Cell 1: Imports and Setup (all imports, no output).
  • Cell 2: Load Dataset (prints shape, columns, null count, class distribution, and fraud percentage).
  • Cell 3: Visualize Class Imbalance (renders a bar chart with counts above each bar).
  • Cell 4: Feature Scaling and Train/Test Split (prints training/test set sizes and fraud percentages in both).

If any cell shows an error, fix it before moving on. The most common issues are a missing creditcard.csv file or a typo in column names.

Your data is loaded, visualized, and split. You have seen the extreme imbalance with your own eyes: just 492 fraud cases hiding in 284,807 transactions. Next up, you will train a baseline model and discover firsthand why this imbalance is a trap for naive classifiers.

Expose the Accuracy Trap with a Baseline Model

Your data is split and ready for training. The natural first step is to throw a simple model at the problem and check accuracy. You are about to see why that instinct leads data scientists astray.

In this step, you will train a Logistic Regression model and discover that 99% accuracy can be completely meaningless. This is the most common mistake in fraud detection, and you need to experience it to understand why better metrics exist.

In this step, get ready to:
  • Train a Logistic Regression baseline and print its accuracy score.
  • Reveal the fraud recall gap hiding beneath that impressive accuracy.
  • Plot a confusion matrix and calculate the dollar cost of missed fraud.
Train the baseline Logistic Regression

A Logistic Regression model with default parameters is the standard first pass for any binary classification problem. It trains fast and produces a clear accuracy number.

  • Create a new cell in your notebook (Cell 5) and add the following code to train the baseline model and evaluate it:
# Cell 5: Baseline Logistic Regression
lr_model = LogisticRegression(max_iter=1000, random_state=42)
lr_model.fit(X_train, y_train)
lr_preds = lr_model.predict(X_test)

print(f"Baseline Accuracy: {accuracy_score(y_test, lr_preds):.4f}")
print("\n--- Classification Report (Baseline) ---")
print(classification_report(y_test, lr_preds, target_names=['Legitimate', 'Fraud']))

What does this code do?

This cell trains and evaluates your first fraud detection model in one shot.

  • LogisticRegression(max_iter=1000, random_state=42) creates the model. The default max_iter is 100, but this dataset needs more iterations to converge without warnings.
  • .fit(X_train, y_train) trains the model on your training data.
  • .predict(X_test) generates predictions for every transaction in the test set.
  • classification_report prints precision, recall, and F1-score for each class (Legitimate and Fraud).

Before you run this cell, what do you think the accuracy will be? Do you think a model trained on 99.8% legitimate transactions will be good at catching the 0.2% that are fraud?

  • Run the cell.

The accuracy reads around 99.9%. Looks incredible. Now look at the Fraud row in the classification report. The recall is below 65%. The model is letting most fraud slip through undetected.

Seeing a convergence warning?

If you see a warning about the solver not converging, double-check that you set max_iter=1000 in the LogisticRegression() call. The default of 100 is not enough for this dataset.

help me fix the convergence warning in my Logistic Regression model

Diagnose the fraud recall gap

Look at the classification report output again. The accuracy is 99.9%, but the fraud recall tells the real story. A recall below 65% means the model misses more than a third of all fraud.

In business terms, every missed fraud case is money out the door. A model that "looks good" on accuracy but fails at its actual job is worse than useless. It gives false confidence.

Why is accuracy misleading here?

When 99.8% of your data is one class, a model can predict "legitimate" for every single transaction and still score 99.8% accuracy. Accuracy rewards the majority class and ignores the minority class entirely.

This is the accuracy trap. The metric that feels most natural is actively hiding your model's biggest failure. For fraud detection, recall on the fraud class is what matters.

  • Add a new markdown cell below Cell 5 in your notebook.
  • In that markdown cell, write a brief note explaining: the model achieves 99%+ accuracy but misses over a third of fraud cases, and missed fraud means real dollar losses for the business.

Portfolio documentation matters

Hiring managers look for candidates who can explain why they chose specific metrics. Writing markdown cells that narrate your analytical reasoning turns a notebook into a portfolio piece, not just a code exercise.

Plot the confusion matrix and calculate dollar losses

A confusion matrix shows exactly where the model gets it right and where it fails. You will see the number of fraud transactions the model missed, then translate that into a dollar figure.

  • Create a new cell (Cell 6) and add this code to visualize the confusion matrix and compute the cost of missed fraud:
# Cell 6: Baseline Confusion Matrix
fig, ax = plt.subplots(figsize=(6, 5))
ConfusionMatrixDisplay.from_predictions(
    y_test, lr_preds,
    display_labels=['Legitimate', 'Fraud'],
    cmap='Blues', ax=ax
)
ax.set_title('Baseline Logistic Regression - Confusion Matrix')
plt.tight_layout()
plt.show()

fraud_missed = confusion_matrix(y_test, lr_preds)[1][0]
avg_fraud_amount = df[df['Class'] == 1]['Amount'].mean()
print(f"\nFraud transactions missed: {fraud_missed}")
print(f"Average fraud transaction amount: ${avg_fraud_amount:.2f}")
print(f"Potential losses from missed fraud: ${fraud_missed * avg_fraud_amount:,.2f}")

What does this code do?

  • ConfusionMatrixDisplay.from_predictions creates a heatmap showing true positives, true negatives, false positives, and false negatives.
  • confusion_matrix(y_test, lr_preds)[1][0] extracts the false negative count (fraud transactions the model labelled as legitimate).
  • avg_fraud_amount calculates the mean transaction amount for fraud cases. Multiplying by fraud_missed gives you the estimated dollar exposure.

What do you think the dollar figure will be? Consider that the model missed over a third of fraud cases.

  • Run the cell.

The confusion matrix heatmap renders showing a large number in the false negative quadrant. The printed output shows the exact count of missed frauds and the potential dollar losses from those missed transactions.

  • Add a markdown cell below Cell 6 with a summary stating how many fraud transactions the model missed, the average fraud amount, and the total potential losses. Frame it as: this is the business cost of trusting accuracy alone.

Confusion matrix not rendering?

Make sure you have import matplotlib.pyplot as plt in Cell 1. Also check that lr_preds exists by running Cell 5 first. Cells must be run in order.

help me fix my confusion matrix plot that is not showing up

✔️ Awesome, I've got everything!

Great. Make sure you have saved your notebook with Cells 5 and 6 added, plus your markdown cells documenting the business cost.

ⓧ I'd like to double check the full code

Scroll through your notebook and confirm you have these cells in order:

  • Cell 1: Imports and Setup (from Step 1).
  • Cell 2: Load Dataset (from Step 1).
  • Cell 3: Visualize Class Imbalance (from Step 1).
  • Cell 4: Feature Scaling and Train/Test Split (from Step 1).
  • Cell 5: Baseline Logistic Regression (trains lr_model, prints accuracy and classification report).
  • Cell 6: Baseline Confusion Matrix (plots heatmap, prints fraud_missed and dollar losses).

Each cell should run without errors when executed in order from top to bottom.

Your baseline model has exposed the accuracy trap. It scores 99.9% accuracy while letting real fraud slip through at enormous cost. Next up, you will fix this with a model that actually prioritizes catching fraud.

Train XGBoost with Cost-Sensitive Learning

Your baseline model exposed the core problem: accuracy is meaningless when 99.8% of transactions are legitimate. The model learned to say "not fraud" almost every time and still scored 99%+ accuracy.

The baseline fails because it has no incentive to focus on the rare fraud class. In this step, you fix that using XGBoost's scale_pos_weight parameter. This single parameter tells the model "fraud cases matter ~578 times more than legitimate ones." No synthetic oversampling. No complex pipelines. One parameter.

In this step, get ready to:
  • Calculate the class imbalance ratio and train an XGBoost model with cost-sensitive learning.
  • Compare fraud detection rates before and after to quantify the improvement in real dollars.
  • Document why this approach works and how it compares to SMOTE.
Calculate the imbalance ratio and train XGBoost

The imbalance ratio tells the model exactly how much more attention to pay to fraud samples. You compute it from the training data: count of legitimate transactions divided by count of fraud transactions.

  • Create a new cell in your notebook (Cell 7) and add the following code to calculate the ratio and train the XGBClassifier:
# Cell 7: XGBoost with scale_pos_weight
n_legitimate = (y_train == 0).sum()
n_fraud = (y_train == 1).sum()
scale_ratio = n_legitimate / n_fraud
print(f"Class imbalance ratio: {scale_ratio:.1f}:1")

xgb_model = xgb.XGBClassifier(
    scale_pos_weight=scale_ratio,
    eval_metric='aucpr',
    n_estimators=100,
    random_state=42,
    use_label_encoder=False
)
xgb_model.fit(X_train, y_train)
xgb_preds = xgb_model.predict(X_test)

print("\n--- Classification Report (XGBoost) ---")
print(classification_report(y_test, xgb_preds, target_names=['Legitimate', 'Fraud']))

What does this code do?

  • scale_pos_weight=scale_ratio tells XGBoost to treat each fraud sample as ~578x more important than a legitimate one during training. The model pays a much heavier penalty for missing fraud.
  • eval_metric='aucpr' uses Area Under the Precision-Recall Curve as the evaluation metric. This is what the Kaggle dataset documentation recommends for imbalanced classification.
  • use_label_encoder=False suppresses a deprecation warning in XGBoost, keeping your output clean.

Before you run this cell, think: do you expect the fraud recall to improve significantly with just this one parameter change? What about precision?

  • Run Cell 7.

You should see the imbalance ratio printed (around 578:1), followed by the new classification report. Fraud recall jumps above 80% while precision stays above 80%. The model now catches the vast majority of fraud.

Seeing low recall or an error?

  • If you see a NameError, make sure you ran all previous cells (Cells 1-6) so that X_train, y_train, X_test, and y_test exist.
  • If recall is still low, double check that scale_pos_weight is set to scale_ratio (the variable), not a hardcoded number.

Help me debug my XGBoost model training. My fraud recall is still low even with scale_pos_weight.

Compare baseline vs XGBoost detection rates

Raw metrics are harder to grasp than a direct comparison. This cell puts the improvement in concrete terms: how many more frauds caught, and how much money that represents.

  • Create a new cell (Cell 8) and add this comparison code:
# Cell 8: Compare Results
lr_fraud_caught = confusion_matrix(y_test, lr_preds)[1][1]
xgb_fraud_caught = confusion_matrix(y_test, xgb_preds)[1][1]
total_fraud = y_test.sum()

print(f"Baseline caught: {lr_fraud_caught}/{total_fraud} frauds ({lr_fraud_caught/total_fraud*100:.1f}% recall)")
print(f"XGBoost caught: {xgb_fraud_caught}/{total_fraud} frauds ({xgb_fraud_caught/total_fraud*100:.1f}% recall)")
print(f"Additional frauds detected: {xgb_fraud_caught - lr_fraud_caught}")
print(f"Additional fraud dollars detected: ${(xgb_fraud_caught - lr_fraud_caught) * avg_fraud_amount:,.2f}")

What does this code do?

  • confusion_matrix(y_test, preds)[1][1] extracts the true positive count (correctly caught frauds) from each model's confusion matrix.
  • The final line multiplies the additional frauds caught by the average fraud amount to translate the improvement into dollar savings.
  • Run Cell 8.

You should see a clear comparison showing how many additional fraud transactions XGBoost catches, and the dollar value those represent.

Seeing a NameError for xgb_preds?

  • Make sure Cell 7 ran successfully before running Cell 8. Cell 8 depends on xgb_preds from Cell 7.
  • Check that avg_fraud_amount still exists from Cell 6. If you restarted your kernel, re-run all cells from the top.

Help me fix my comparison cell.

Document why cost-sensitive learning works

A portfolio notebook that shows results without explaining the reasoning behind them misses the point. This is where you demonstrate that you understand the mechanism, not just the syntax.

  • Add a new Markdown cell below Cell 8 in your notebook.
  • In this markdown cell, explain why scale_pos_weight works and how it compares to SMOTE. Use the explanation in the callout below as a guide for what to write.

Why does scale_pos_weight work?

During training, XGBoost computes a gradient (a direction to improve) for each sample. scale_pos_weight multiplies the gradient contribution of positive (fraud) samples by the ratio you set. This forces the model to pay 578x more attention to getting fraud predictions right.

Compare this to SMOTE, which generates synthetic fraud samples to balance the dataset. SMOTE requires a separate preprocessing pipeline, introduces artificial data points that can distort decision boundaries, and makes your training set much larger. Cost-sensitive learning achieves the same goal with one parameter. Simpler is better in production.

  • Save your notebook after adding the markdown cell.

✔️ Awesome, I've got everything!

Great. Double check that you have saved your notebook with all cells through Cell 8.

ⓧ I'd like to double check the full code

Scroll through your notebook and confirm these cells are present in order:

  • Cell 1: Imports and Setup (pandas, numpy, matplotlib, sklearn, xgboost, shap).
  • Cell 2: Load Dataset (reads creditcard.csv, prints shape and class distribution).
  • Cell 3: Visualize Class Imbalance (bar chart).
  • Cell 4: Feature Scaling and Train/Test Split (StandardScaler on Amount and Time, stratified split).
  • Cell 5: Baseline Logistic Regression (train, predict, print accuracy and classification report).
  • Cell 6: Baseline Confusion Matrix (heatmap, fraud_missed and avg_fraud_amount computed).
  • Cell 7: XGBoost with scale_pos_weight (ratio calculation, XGBClassifier trained, predictions generated, classification report printed).
  • Cell 8: Compare Results (lr_fraud_caught vs xgb_fraud_caught, additional frauds and dollars detected).

Your fraud detector now catches the vast majority of fraudulent transactions. One parameter change turned a model that let most fraud slip through into one that actively hunts it down. Next up, you will fine-tune the decision threshold to balance precision and recall based on real business costs.

Tune the Decision Threshold with Precision-Recall Curves

Your XGBoost model now catches 80%+ of fraud. But the default 0.5 probability threshold it uses to decide "fraud or not" is completely arbitrary.

In fraud detection, the business decides the threshold by weighing the cost of missed fraud against the cost of false alarms (blocking legitimate customers). This step shows you think like an analyst, not just a coder.

In this step, get ready to:
  • Plot a Precision-Recall curve with the optimal F1 threshold annotated.
  • Build a cost-benefit framework that translates threshold choice into dollar savings.
Plot the Precision-Recall curve with optimal threshold

Instead of a hard 0/1 prediction, your model can output a probability score for each transaction. A score of 0.9 means the model is very confident it's fraud. A score of 0.3 means it's less sure.

The Precision-Recall curve shows how precision and recall change at every possible threshold. It lets you pick exactly where you want to sit on that tradeoff.

  • Create a new cell in your notebook (Cell 9) and add the following code to compute probability scores, find the optimal threshold, and plot the Precision-Recall curve:
# Cell 9: Precision-Recall Curve
xgb_probs = xgb_model.predict_proba(X_test)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_test, xgb_probs)

# Calculate F1 at each threshold
f1_scores = 2 * (precision[:-1] * recall[:-1]) / (precision[:-1] + recall[:-1] + 1e-8)
optimal_idx = np.argmax(f1_scores)
optimal_threshold = thresholds[optimal_idx]
optimal_precision = precision[optimal_idx]
optimal_recall = recall[optimal_idx]

What does this code do?

predict_proba(X_test)[:, 1] gets the model's fraud probability for each transaction (the second column is the positive class).

precision_recall_curve from scikit-learn computes precision and recall at every possible threshold. The F1 formula 2 * (precision * recall) / (precision + recall) finds the threshold that balances both metrics optimally.

Now add the plotting code directly below in the same cell to visualize the curve with the optimal point annotated:

fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(recall, precision, color='steelblue', linewidth=2, label='XGBoost PR Curve')
ax.scatter(optimal_recall, optimal_precision, color='red', s=100, zorder=5,
           label=f'Optimal (threshold={optimal_threshold:.3f})')
ax.annotate(f'F1={f1_scores[optimal_idx]:.3f}\nPrecision={optimal_precision:.3f}\nRecall={optimal_recall:.3f}',
            xy=(optimal_recall, optimal_precision),
            xytext=(optimal_recall - 0.15, optimal_precision - 0.1),
            fontsize=9, arrowprops=dict(arrowstyle='->', color='red'))
ax.set_xlabel('Recall (Fraud Detection Rate)')
ax.set_ylabel('Precision (Alarm Accuracy)')
ax.set_title('Precision-Recall Curve: XGBoost Fraud Detector')
ax.legend(loc='lower left')
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print(f"Optimal threshold: {optimal_threshold:.4f}")
print(f"At this threshold: Precision={optimal_precision:.3f}, Recall={optimal_recall:.3f}")

What does this plot show?

The blue curve traces how precision ("how many flagged transactions are actually fraud") trades off against recall ("how many total frauds did we catch"). Higher recall means catching more fraud, but at the cost of more false alarms.

The red dot marks the threshold where F1 is maximized. F1 is the harmonic mean of precision and recall. It penalizes extreme values in either direction.

Before you run this cell, what do you expect the shape of the curve to look like? Do you think precision will drop smoothly or sharply as recall increases?

  • Run Cell 9.

You should see a Precision-Recall curve with a red dot marking the optimal threshold. The printed output shows the exact threshold value and corresponding precision and recall percentages.

Not seeing the plot?

  • Make sure the computation code (the first part) and the plotting code are in the same cell. If they're in separate cells, the plotting cell won't have access to the variables.
  • Check that you ran Cell 7 and Cell 8 earlier so xgb_model and y_test exist in memory.

Help me debug my Precision-Recall curve plot in Jupyter

Build the cost-benefit framework

The optimal F1 threshold is a statistical answer. But the business needs a dollar answer. What does each threshold choice actually cost?

  • Create a new cell (Cell 10) and add the following cost-benefit analysis:
# Cell 10: Cost-Benefit Analysis
cost_per_missed_fraud = avg_fraud_amount
cost_per_false_positive = 10  # Customer service + inconvenience

def calculate_costs(threshold, y_true, probabilities):
    preds = (probabilities >= threshold).astype(int)
    cm = confusion_matrix(y_true, preds)
    tn, fp, fn, tp = cm.ravel()
    total_cost = (fn * cost_per_missed_fraud) + (fp * cost_per_false_positive)
    return {'threshold': threshold, 'frauds_caught': tp, 'frauds_missed': fn,
            'false_alarms': fp, 'total_cost': total_cost}

thresholds_to_test = [0.3, optimal_threshold, 0.7]
results = [calculate_costs(t, y_test, xgb_probs) for t in thresholds_to_test]
cost_df = pd.DataFrame(results)
cost_df['total_cost_formatted'] = cost_df['total_cost'].apply(lambda x: f"${x:,.2f}")
print("\n--- Cost-Benefit Analysis at Different Thresholds ---")
print(cost_df[['threshold', 'frauds_caught', 'frauds_missed', 'false_alarms', 'total_cost_formatted']].to_string(index=False))

What does this code do?

cost_per_missed_fraud is set to the average fraud transaction amount from the data. Each fraud the model misses costs the business that amount. cost_per_false_positive is set to $10 to represent customer service time and inconvenience when a legitimate transaction is blocked.

The calculate_costs function takes a threshold, converts probabilities into binary predictions, then counts frauds caught, frauds missed, and false alarms. It multiplies each by their dollar cost to get a total.

The table compares three strategies: aggressive (0.3 threshold catches more fraud but blocks more customers), balanced (optimal F1), and conservative (0.7 threshold blocks fewer customers but misses more fraud).

  • Run Cell 10.

You should see a table with three rows showing how threshold choice translates directly into dollar cost. Notice how a lower threshold catches more fraud but racks up more false alarms, while a higher threshold reduces false alarms but lets more fraud through.

Seeing a NameError for optimal_threshold?

  • Cell 10 depends on variables from Cell 9. Make sure Cell 9 ran successfully before running Cell 10.
  • If you restarted your kernel, re-run all cells from Cell 1 through Cell 9 first.

Help me fix a NameError in my cost-benefit analysis cell

Why does this matter for interviews?

Most junior data scientists stop at "my model has 80% recall." A cost-benefit table shows you understand that model performance is a business decision. Different stakeholders will prefer different thresholds depending on their risk tolerance.

✔️ Awesome, I've got everything!

Great. Make sure you have saved your notebook with both Cell 9 and Cell 10 complete.

ⓧ I'd like to double check the full code

Your notebook should now contain Cells 1 through 10. Scroll through and confirm the following cells are present, in order:

  • Cell 1: Imports and Setup
  • Cell 2: Load Dataset
  • Cell 3: Visualize Class Imbalance
  • Cell 4: Feature Scaling and Train/Test Split
  • Cell 5: Baseline Logistic Regression
  • Cell 6: Baseline Confusion Matrix
  • Cell 7: XGBoost with scale_pos_weight
  • Cell 8: Compare Results
  • Cell 9: Precision-Recall Curve (computes probabilities, F1 scores, optimal threshold, and plots the PR curve with annotation).
  • Cell 10: Cost-Benefit Analysis (defines calculate_costs function and prints the cost comparison table for thresholds 0.3, optimal, and 0.7).

You now have a framework that translates abstract model metrics into dollar values. Next up, you'll make your model explainable with SHAP so fraud analysts can trust its decisions.

Explain Predictions with SHAP

Your model catches fraud and your cost-benefit analysis proves its value in dollar terms. But a model that works without being explainable will never be trusted by compliance teams or fraud analysts.

SHAP values decompose each prediction into per-feature contributions, showing exactly which inputs pushed the model toward "fraud" or "legitimate." This is the section that separates a portfolio piece from a tutorial copy.

In this step, get ready to:
  • Compute SHAP values for every prediction in the test set.
  • Generate a beeswarm plot showing global feature importance with directional impact.
  • Create waterfall plots that explain individual fraud and legitimate predictions.
Create the SHAP explainer and compute values

SHAP uses Shapley values from cooperative game theory to measure each feature's contribution to each individual prediction. For tree-based models like XGBoost, SHAP uses the TreeExplainer algorithm for fast exact computation.

  • Create a new cell in your notebook (Cell 11) and add the following code to build the SHAP explainer and compute values:
# Cell 11: SHAP Explainability
explainer = shap.Explainer(xgb_model, X_test)
shap_values = explainer(X_test)

What does this code do?

shap.Explainer(xgb_model, X_test) creates an explainer object that automatically detects your model type (XGBoost) and selects the fastest exact algorithm (TreeExplainer).

explainer(X_test) computes SHAP values for every sample in the test set, returning an Explanation object that stores per-feature contributions for each of the 56,962 test transactions.

  • Run the cell. The computation may take 10-30 seconds depending on your machine.
  • You should see the cell complete without errors. The shap_values object is now ready to power your visualizations.

Seeing a memory or timeout error?

SHAP computation on 56,000+ samples can be memory-intensive. If your machine struggles, try restarting the kernel (Kernel > Restart & Run All) to free up memory before running this cell again.

Help me debug my SHAP explainer failing on XGBoost

Generate the beeswarm plot

A beeswarm plot shows global feature importance AND the direction of each feature's impact. Each dot is one transaction, colored by the feature's value (red = high, blue = low), positioned by how much it pushed the prediction toward fraud.

  • Create a new cell (Cell 12) and generate the beeswarm plot:
# Cell 12: SHAP Beeswarm Plot
shap.plots.beeswarm(shap_values, max_display=10, show=True)

What does this code do?

  • max_display=10 limits the plot to the top 10 most impactful features, keeping the visualization readable.
  • Each row is one feature. Dots spread horizontally based on how strongly that feature affected each prediction.
  • Color encodes the actual feature value: red dots on the right mean high feature values push toward fraud.

Before you run this cell, what do you expect to see at the top of the feature importance ranking? Will it be the Amount, Time, or one of the V-features?

  • Run the cell.

You should see a beeswarm plot with 10 features ranked by importance. Notice that V14 likely appears near the top. Large negative V14 values (blue dots on the right) strongly push the model toward predicting fraud.

  • Add a markdown cell below Cell 12 interpreting the top 3 features you see. For each feature, describe what direction (high or low values) pushes predictions toward fraud. For example: "V14: Strongly negative values push toward fraud. V4: High positive values push toward fraud."

Why interpret the features in markdown?

Compliance teams and fraud analysts reading your notebook cannot interpret a SHAP plot on their own. A short written summary below the plot turns your notebook into a communication tool, not just a code file. This is what makes it a portfolio piece.

Plot not rendering?

Make sure you ran Cell 11 first and it completed without error. The shap_values object must exist before plotting.

If you see a blank output, try adding %matplotlib inline at the top of the cell.

Help me get my SHAP beeswarm plot to render in Jupyter

Generate waterfall plots for individual predictions

The beeswarm shows the big picture. Waterfall plots show how the model made ONE specific decision. This is the explanation you would hand to a fraud analyst investigating a flagged transaction.

  • Create a new cell (Cell 13) to find correctly-classified examples and generate their waterfall plots:
# Cell 13: SHAP Waterfall Plots
fraud_indices = y_test[y_test == 1].index
legit_indices = y_test[y_test == 0].index

# Find a correctly predicted fraud transaction
fraud_test_positions = [i for i, idx in enumerate(y_test.index) if y_test.iloc[i] == 1 and xgb_preds[i] == 1]
legit_test_positions = [i for i, idx in enumerate(y_test.index) if y_test.iloc[i] == 0 and xgb_preds[i] == 0]

if fraud_test_positions:
    print("--- Waterfall Plot: Correctly Caught Fraud Transaction ---")
    shap.plots.waterfall(shap_values[fraud_test_positions[0]], show=True)

if legit_test_positions:
    print("\n--- Waterfall Plot: Legitimate Transaction ---")
    shap.plots.waterfall(shap_values[legit_test_positions[0]], show=True)

What does this code do?

  • fraud_test_positions finds indices of transactions that are truly fraud AND were correctly predicted as fraud by the model.
  • legit_test_positions finds indices of transactions that are truly legitimate AND were correctly predicted as legitimate.
  • shap.plots.waterfall(shap_values[index]) shows how each feature contributed to that single prediction, stacking from the base value (average prediction) to the final output.
  • Run the cell.
  • You should see two waterfall plots. The fraud transaction's waterfall shows features pushing the prediction strongly toward positive (fraud). The legitimate transaction's waterfall shows features holding the prediction near zero.
  • Add a markdown cell below Cell 13 explaining the fraud waterfall in plain language. Write something like: "For this fraud transaction, the model was most influenced by features V14, V4, and V12. This is the explanation you would give to a fraud analyst reviewing a flagged transaction."

Why does this matter for stakeholders?

A fraud analyst does not need to understand XGBoost internals. They need to see "this transaction was flagged because of unusual values in features X, Y, and Z." Waterfall plots make your model auditable and build the trust required for production deployment.

Seeing an index error?

Make sure Cell 7 ran successfully so that xgb_preds exists. The list comprehensions compare y_test labels against xgb_preds values, so both must be available.

Help me fix an index error in my SHAP waterfall plot code

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook with all three new cells (11, 12, and 13).

ⓧ I'd like to double check the full code

Scroll through your notebook and confirm these cells are present in order:

  • Cells 1-10 from previous steps (imports, data loading, class imbalance visualization, feature scaling, baseline model, confusion matrix, XGBoost training, comparison, precision-recall curve, cost-benefit analysis).
  • Cell 11: SHAP Explainability. Creates explainer and computes shap_values.
  • Cell 12: SHAP Beeswarm Plot. Calls shap.plots.beeswarm(shap_values, max_display=10, show=True).
  • Cell 13: SHAP Waterfall Plots. Computes fraud_test_positions and legit_test_positions, then renders waterfall plots for one fraud and one legitimate transaction.

Secret mission

Threshold Comparison Dashboard

Real fraud detection systems let business stakeholders choose their own risk tolerance. Can you build a visual dashboard that shows the cost tradeoff across multiple thresholds, so a non-technical manager can pick the right one for their team?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally on your machine, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your project files for future reference or delete them to free up disk space.

Resources you used:

  • fraud_detection.ipynb notebook (your complete fraud detection analysis).
  • creditcard.csv dataset file (downloaded from Kaggle).

Keep everything running

No action needed. Choose this if you plan to extend the notebook, experiment with different thresholds, or use it as a portfolio piece.

  • Your notebook and dataset remain wherever you saved them.
  • You can reopen fraud_detection.ipynb in Jupyter Notebook at any time to re-run cells or add new analysis.

Pause - I'll come back to this later

Shut down any running Jupyter Notebook server to free up system memory. Your files stay on disk for when you return.

  • In the terminal where Jupyter is running, press Ctrl+C to stop the server.
  • Your fraud_detection.ipynb and creditcard.csv files remain saved on your machine.

Delete - I don't want to use this again

Remove all project files from your machine. This frees up disk space (the dataset alone is about 150 MB).

  • In the terminal where Jupyter is running, press Ctrl+C to stop the server.
  • Navigate to the folder containing your project files and delete them by running:
rm fraud_detection.ipynb creditcard.csv

On Windows, use this command instead:

del fraud_detection.ipynb, creditcard.csv
  • Confirm the files are gone by running ls (macOS/Linux) or dir (Windows). You should no longer see fraud_detection.ipynb or creditcard.csv in the output.

Nice Work!

Nice Work!

You just built a production-ready fraud detection notebook that goes far beyond a typical tutorial. You diagnosed why accuracy is a trap, fixed the problem with cost-sensitive learning, tuned your model for real business outcomes, and made it explainable to non-technical stakeholders.

You've learned how to:

  • Expose the accuracy trap in imbalanced classification by training a baseline model, then fix it with XGBoost's cost-sensitive learning (scale_pos_weight) to catch 80%+ of fraud.
  • Evaluate models using Precision-Recall curves and apply threshold tuning with a cost-benefit framework that translates ML metrics into dollar values for business stakeholders.
  • Generate SHAP explainability visualizations (beeswarm and waterfall plots) that make model decisions auditable and interpretable for compliance teams.
  • Secret Mission: Built a threshold comparison dashboard as a decision-support tool for business stakeholders to visualize the tradeoff between catching more fraud and blocking legitimate customers.

Ready to quiz yourself?