Detect Fraud in Business Loan Data

Train a Random Forest model to catch fraudulent SBA loan applications.

Introduction

30 Second Summary

Every year, billions of dollars are lost to fraudulent business loans. Banks and government agencies need automated systems that can spot suspicious applications before the money goes out the door.

In this project, you will build a machine learning model that analyzes nearly 900,000 real business loan records to predict which applications are likely to default or be fraudulent.

What You'll Build

A fraud detection pipeline that loads historical loan data, engineers predictive features, handles severe class imbalance, and trains a Random Forest classifier to flag high-risk loan applications.

By the end of this project, you'll have:

  • A trained Random Forest model that identifies fraudulent or defaulting business loan applications with high recall on the fraud class.
  • A complete data analysis notebook proving that class imbalance distorts accuracy metrics, and showing how SMOTE corrects it.
  • Visualizations (confusion matrix, ROC curve, feature importance chart) that explain which loan attributes signal fraud risk.
  • A reusable scoring function that rates new loan applications as "high risk" or "low risk."
  • Secret Mission: Optimize your model's decision threshold using a custom business cost function that balances catching fraud against false alarms.

Do I need machine learning experience?

Nope. You only need basic Python knowledge (variables, functions, loops). This project teaches machine learning concepts as you go, starting from scratch.

Set Up Your Python Environment

Before you can train a fraud detection model, you need a Python environment with the right tools. Python is the industry standard for machine learning, and the libraries you install in this step handle everything from data loading to model evaluation.

In this step, you will install Python, set up VS Code as your notebook editor, and pull in the data science packages that power the pipeline.

In this step, get ready to:
  • Install and verify Python 3.11+.
  • Set up VS Code with the Jupyter extension.
  • Install the data science packages for modeling and visualization.
Install Python 3.11+

The fraud detection pipeline requires Python 3.11 or higher. The scikit-learn package you install shortly requires this minimum version.

  • Check your Python version by running this command in a terminal:

macOS

python3 --version

Windows

python --version

Check the output and pick the tab that matches what you see.

✔️ I see version 3.11 or higher

Your Python version is compatible. Move on to the next substep.

ⓧ I see an older version

You have Python installed, but you need version 3.11 or higher. Your system may ship with an older Python that shadows newer installs.

  • Visit python.org/downloads and download the latest Python 3 installer for your operating system.
  • Run the installer. On Windows, make sure to check the box that says Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal, then run the version check command again.

macOS users: why you might still see an old version

macOS often ships with an older Python that python3 resolves to by default. After installing a newer version from python.org, try running python3.11 --version or python3.12 --version to check if the new version is accessible under a versioned name.

If you use Homebrew, you can run brew install python@3.12 and then use python3.12 as your command going forward.

ⓧ Command not found

Python is not installed yet. Follow these steps to install it.

  • Visit python.org/downloads and download the latest Python 3 installer for your operating system.
  • Run the installer. On Windows, make sure to check the box that says Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal after the installation finishes.
  • Run the version check command again to confirm it shows 3.11 or higher.
Set up VS Code and create your project
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.

VS Code not installed?

Download it for free from code.visualstudio.com. Run the installer for your platform and try the search again.

Help me install VS Code on my machine.

Now install the Jupyter extension so VS Code can run notebook files.

  • Click the Extensions icon in the left sidebar, or press Cmd+Shift+X (macOS) or Ctrl+Shift+X (Windows).
  • Type Jupyter in the search bar at the top of the Extensions panel.
  • Find the result called Jupyter by Microsoft.
  • Click Install.

What is the Jupyter extension?

The Jupyter extension lets VS Code open and run .ipynb notebook files directly in the editor. You write code in cells and run them one at a time, which is ideal for data exploration and machine learning workflows.

Next, create your project folder on the Desktop.

  • Open the integrated terminal by pressing Ctrl+` (backtick key, same on both macOS and Windows).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder by running this command:
mkdir loan-fraud-detection
  • Open this folder in VS Code by clicking File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the loan-fraud-detection folder, and click Open.

Now create the notebook file inside your project.

  • In the Explorer sidebar on the left, hover over the folder name and click the New File... button (the file icon with a plus sign).
  • Type fraud_detection.ipynb and press Enter.
  • Confirm the file exists by running this in the terminal:
ls

You should see fraud_detection.ipynb in the output. VS Code should also show the file open as a notebook with an empty cell.

Don't see the notebook interface?

If the file opens as raw text instead of a notebook, the Jupyter extension may not have finished installing. Close the file tab, wait a few seconds, then reopen fraud_detection.ipynb from the Explorer.

Help me get the Jupyter notebook interface working in VS Code.

Install the required packages

The pipeline depends on five packages. You will install them all in one command with pinned versions to ensure compatibility.

  • Install the project dependencies by running this command in the VS Code terminal:
pip install scikit-learn==1.8.0 pandas==2.3.3 imbalanced-learn==0.14.1 matplotlib seaborn

What does each package do?

  • scikit-learn provides the machine learning algorithms (Random Forest) and evaluation metrics.
  • pandas loads and manipulates the loan dataset as DataFrames.
  • imbalanced-learn provides SMOTE for handling the class imbalance problem.
  • matplotlib draws charts like ROC curves and confusion matrices.
  • seaborn adds statistical plotting on top of matplotlib for cleaner visuals.

After the install finishes, you should see output ending with a line that says Successfully installed followed by the package names and versions.

Installation failed or pip not found?

  • If you see a permissions error, try adding --user to the end of the command.
  • If pip is not recognized, try pip3 instead. On some systems, pip is called pip3 for Python 3.
  • If you see a version conflict error, make sure you verified Python 3.11+ in the first substep. The pinned package versions require it.

Help me fix my pip installation error.

Your environment is ready. Next up, you will load nearly 900,000 real business loan records and explore what the data looks like.

Load and Explore the SBA Loan Dataset

Your environment is set up and ready to go. Now it's time to get some real data into your notebook.

You will work with almost 900,000 historical SBA (Small Business Administration) loan records from Kaggle. The goal is to understand the shape of the data before building any model.

In this step, get ready to:
  • Download the SBA National loan dataset and load it into a DataFrame.
  • Identify the target variable and visualize the class imbalance.
  • Explore key financial features for patterns between legitimate and defaulting loans.
Download the dataset and set up your notebook

The dataset you need is hosted on Kaggle, a free platform for sharing datasets. You will need a free Kaggle account to download it.

  • Open kaggle.com/datasets/mirbektoktogaraev/should-this-loan-be-approved-or-denied in your browser.
  • If you don't have a Kaggle account, click Register and create a free account (you can sign up with Google).
  • Click the Download button near the top of the dataset page.
  • Extract the downloaded zip file. You should find a file called SBAnational.csv inside it.
  • Move SBAnational.csv into your loan-fraud-detection folder (the same folder that contains your notebook).

What is the SBA National dataset?

This dataset contains historical loan records from the U.S. Small Business Administration, covering loans issued between 1987 and 2014. Each row represents one business loan application with its outcome: paid in full, or charged off (defaulted).

Now add all the imports your notebook needs. You will import everything upfront so later cells can use any library without re-importing.

  • In your fraud_detection.ipynb notebook, click into the first empty cell and paste the following code:
# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score,
    roc_curve,
    ConfusionMatrixDisplay,
)
from imblearn.over_sampling import SMOTE

What does this code do?

  • pandas and numpy handle data loading and numeric operations.
  • matplotlib and seaborn create charts and visualizations.
  • scikit-learn provides the machine learning model, data splitting, and evaluation metrics.
  • SMOTE from imbalanced-learn will balance the training data in a later step.
  • Run the cell by pressing Shift+Enter.

The cell should complete with no output and no errors. If you see an ImportError, double check that you installed all packages in Step 1.

Seeing an ImportError?

  • Make sure you ran the pip install command from Step 1 in the same Python environment your notebook is using.
  • Check the kernel in the top-right corner of VS Code. It should point to the Python 3.11+ interpreter where you installed the packages.
  • help me fix an ImportError in my Jupyter notebook

Next, load the CSV file into a DataFrame so you can start exploring.

  • Create a new cell below the imports cell and paste the following code:
# Cell 2: Load the dataset
df = pd.read_csv("SBAnational.csv", low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head()

What does this code do?

  • pd.read_csv("SBAnational.csv", low_memory=False) reads the entire CSV file into memory. The low_memory=False flag prevents mixed-type warnings on large files.
  • df.shape prints the number of rows and columns so you know how large the dataset is.
  • df.head() displays the first five rows so you can see what the data looks like.
  • Run the cell with Shift+Enter.
  • You should see output showing approximately 899,000 rows and 27 columns, followed by a table of the first five records.

Seeing a FileNotFoundError?

  • Make sure SBAnational.csv is in the same folder as your notebook file (loan-fraud-detection).
  • Check that the file name matches exactly, including capitalization.
  • help me fix FileNotFoundError when loading SBAnational.csv
Identify the target variable and visualize the class imbalance

Every supervised learning project needs a target variable. In this dataset, MIS_Status tells you whether each loan was paid in full or charged off (defaulted).

  • Create a new cell and paste the following code:
# Cell 3: Explore the target variable
print(f"\nTarget variable (MIS_Status) distribution:")
print(df["MIS_Status"].value_counts())
print(f"\nMissing values in target: {df['MIS_Status'].isna().sum()}")

fig, ax = plt.subplots(figsize=(6, 4))
df["MIS_Status"].value_counts().plot(kind="bar", ax=ax, color=["steelblue", "coral"])
ax.set_title("Loan Status Distribution")
ax.set_xlabel("Status (PIF=Paid in Full, CHGOFF=Charged Off)")
ax.set_ylabel("Count")
plt.tight_layout()
plt.show()

What does this code do?

  • value_counts() shows how many loans are PIF (Paid in Full) versus CHGOFF (Charged Off/Default).
  • The isna().sum() check counts missing values in the target column so you know how many rows need cleaning later.
  • The bar chart gives you a visual sense of how imbalanced the classes are. You will see that legitimate loans heavily outnumber defaults.
  • Run the cell with Shift+Enter.
  • You should see the class counts printed, followed by a bar chart where the PIF bar is much taller than the CHGOFF bar.

Why does class imbalance matter?

When one class dominates the dataset, a model can achieve high accuracy by simply predicting the majority class every time. A model that always says "legitimate" would be over 80% accurate here. It would also catch zero fraud.

This is the core problem you will solve in this project. You need metrics that reward catching the minority class, not just overall correctness.

Bar chart not appearing?

  • Make sure you ran the imports cell (Cell 1) first. The matplotlib import must execute before any plotting code.
  • If the chart appears in a separate window instead of inline, add %matplotlib inline at the top of Cell 1 and rerun both cells.
  • help me display matplotlib charts inline in my Jupyter notebook
Explore key financial features

Before building a model, you need to understand which features might help distinguish legitimate loans from defaults. Comparing summary statistics between the two groups reveals early patterns.

  • Create a new cell and paste the following code:
# Cell 4: Explore key features
features_to_explore = ["DisbursementGross", "GrAppv", "Term", "NoEmp", "CreateJob", "RetainedJob"]
print("\nSummary statistics by loan status:")
for col in features_to_explore:
    if col in df.columns:
        print(f"\n--- {col} ---")
        print(df.groupby("MIS_Status")[col].describe().loc[["PIF", "CHGOFF"]])

What does this code do?

  • It groups the data by loan outcome (PIF vs CHGOFF) and prints descriptive statistics (mean, median, min, max) for each of six financial features.
  • DisbursementGross is the actual amount disbursed. GrAppv is the approved amount. Term is the loan duration in months.
  • NoEmp is the number of employees. CreateJob and RetainedJob show job creation and retention numbers.
  • Look for differences in mean or median values between the two groups. Larger differences hint at features that will be useful for the model.
  • Run the cell with Shift+Enter.
  • You should see a table of statistics for each feature, broken down by PIF and CHGOFF. Notice whether defaulting loans tend to have different term lengths or disbursement amounts.

Seeing a KeyError for one of the columns?

  • If a column is missing, it means your version of the CSV might use slightly different column names. Run df.columns.tolist() in a new cell to see the exact column names.
  • The code already includes if col in df.columns as a safety check, so it should skip missing columns without crashing.
  • help me debug a KeyError in my pandas groupby code

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here is the complete notebook code up to this point. Compare it against your cells to make sure nothing was missed.

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score,
    roc_curve,
    ConfusionMatrixDisplay,
)
from imblearn.over_sampling import SMOTE

# Cell 2: Load the dataset
df = pd.read_csv("SBAnational.csv", low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head()

# Cell 3: Explore the target variable
print(f"\nTarget variable (MIS_Status) distribution:")
print(df["MIS_Status"].value_counts())
print(f"\nMissing values in target: {df['MIS_Status'].isna().sum()}")

fig, ax = plt.subplots(figsize=(6, 4))
df["MIS_Status"].value_counts().plot(kind="bar", ax=ax, color=["steelblue", "coral"])
ax.set_title("Loan Status Distribution")
ax.set_xlabel("Status (PIF=Paid in Full, CHGOFF=Charged Off)")
ax.set_ylabel("Count")
plt.tight_layout()
plt.show()

# Cell 4: Explore key features
features_to_explore = ["DisbursementGross", "GrAppv", "Term", "NoEmp", "CreateJob", "RetainedJob"]
print("\nSummary statistics by loan status:")
for col in features_to_explore:
    if col in df.columns:
        print(f"\n--- {col} ---")
        print(df.groupby("MIS_Status")[col].describe().loc[["PIF", "CHGOFF"]])

You now have a clear picture of the data you are working with. Next up, you will clean this data and engineer new features that help your model spot risky loans.

Clean Data and Engineer Features

In the previous step, you explored the SBA loan dataset and saw the class imbalance between legitimate and defaulting loans. Now it is time to prepare the data for machine learning.

Raw datasets are never model-ready. This dataset has missing values in the target column, currency fields stored as text strings with $ signs and commas, and columns that need to be transformed into features a model can learn from.

In this step, get ready to:
  • Clean the raw data by removing missing targets and converting currency strings to numbers.
  • Engineer new features that help the model detect suspicious loan patterns.
  • Split the data into training and test sets for fair model evaluation.
Handle missing values and clean currency columns

Machine learning models cannot work with missing target labels or text-formatted numbers. You need to drop incomplete rows and convert the currency columns into numeric values the model can process.

  • Create a new cell in your notebook (Cell 5) and add the following data cleaning code:
# Cell 5: Data cleaning
# Drop rows with missing target
df_clean = df.dropna(subset=["MIS_Status"]).copy()
print(f"Shape after dropping missing target: {df_clean.shape}")

# Clean currency columns (remove $ and commas, convert to numeric)
currency_cols = ["DisbursementGross", "GrAppv", "SBA_Appv"]
for col in currency_cols:
    if col in df_clean.columns:
        df_clean[col] = (
            df_clean[col]
            .astype(str)
            .str.replace("$", "", regex=False)
            .str.replace(",", "", regex=False)
            .astype(float)
        )

# Encode target: PIF=0, CHGOFF=1
df_clean["Target"] = (df_clean["MIS_Status"] == "CHGOFF").astype(int)
print(f"\nTarget distribution after encoding:")
print(df_clean["Target"].value_counts())

What does this code do?

  • dropna(subset=["MIS_Status"]) removes any row where the loan outcome is unknown. You cannot train a model on rows with no answer.
  • The for loop strips $ signs and commas from the three currency columns, then converts them to floats. The regex=False parameter tells pandas to treat $ as a literal character, not a regular expression pattern.
  • The last section encodes the target column as a binary number: 0 for Paid in Full (legitimate) and 1 for Charged Off (default/fraud).
  • Run Cell 5.

You should see the shape after dropping missing rows (slightly smaller than 899,164) and a count of 0s and 1s in the target column.

Seeing a KeyError or NameError?

  • Make sure Cells 1-4 have been executed first. The df variable must already exist in memory.
  • Check that the column name is exactly MIS_Status with that exact capitalization.

Help me debug my data cleaning cell.

Engineer new features from raw columns

Raw columns like loan amounts and business type codes do not tell the full story. A ratio between how much money was disbursed versus approved, for example, can reveal suspicious patterns that individual dollar amounts cannot.

  • Create a new cell (Cell 6) and add the following feature engineering code:
# Cell 6: Feature engineering
# Loan to Approval Ratio
df_clean["LoanToApproval_Ratio"] = df_clean["DisbursementGross"] / df_clean["GrAppv"].replace(0, np.nan)

# Is New Business flag
df_clean["IsNewBusiness"] = (df_clean["NewExist"] == 2).astype(int)

# Is Urban flag
df_clean["IsUrban"] = (df_clean["UrbanRural"] == 1).astype(int)

# Select features
feature_cols = [
    "Term",
    "NoEmp",
    "CreateJob",
    "RetainedJob",
    "DisbursementGross",
    "GrAppv",
    "SBA_Appv",
    "LoanToApproval_Ratio",
    "IsNewBusiness",
    "IsUrban",
]

# Drop rows with NaN in feature columns
df_model = df_clean[feature_cols + ["Target"]].dropna()
print(f"\nFinal dataset shape for modeling: {df_model.shape}")

X = df_model[feature_cols]
y = df_model["Target"]

What does this code do?

  • LoanToApproval_Ratio divides the disbursed amount by the approved amount. A ratio significantly above 1.0 could signal a red flag. The .replace(0, np.nan) prevents division-by-zero errors.
  • IsNewBusiness converts the NewExist column (where 2 means new business) into a simple 0/1 flag. New businesses are statistically more likely to default.
  • IsUrban converts the UrbanRural column (where 1 means urban) into a 0/1 flag.
  • feature_cols defines the final list of 10 features the model will use. The last dropna() removes any rows that still have missing values in these columns.
  • X holds the feature matrix and y holds the target labels. This is the standard naming convention in machine learning.
  • Run Cell 6.

You should see the final dataset shape printed, showing how many rows survived all the cleaning and feature engineering.

Seeing fewer rows than expected?

Some rows may have missing values in the NewExist or UrbanRural columns. The final dropna() removes these. This is expected behavior.

Help me understand why my dataset is smaller after feature engineering.

Split the data into training and test sets

Before training a model, you need to hold out a portion of data it has never seen. This test set gives you an honest measure of how well the model performs on new loan applications.

  • Create a new cell (Cell 7) and add the following train/test split code:
# Cell 7: Train/test split with stratification
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"Training class distribution:\n{y_train.value_counts()}")

Why use stratify=y?

Your dataset is imbalanced. Without stratify=y, a random split could accidentally put more fraud cases in the test set than the training set (or vice versa).

Stratified sampling forces both sets to have the same proportion of fraud vs. legitimate loans as the original data. This ensures a fair evaluation later.

  • Run Cell 7.

You should see the training set size (80% of the data), the test set size (20%), and a class distribution confirming that both classes are represented in training.

Getting a NameError on X or y?

  • Make sure Cell 6 ran successfully before Cell 7. The variables X and y are defined at the end of Cell 6.
  • If you restarted the kernel, you need to re-run all cells from Cell 1 onwards.

Help me fix my train/test split error.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook.

ⓧ I'd like to double check the full code

Here is the full notebook up to this point. Compare your cells against this reference:

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score,
    roc_curve,
    ConfusionMatrixDisplay,
)
from imblearn.over_sampling import SMOTE

# Cell 2: Load the dataset
df = pd.read_csv("SBAnational.csv", low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head()

# Cell 3: Explore the target variable
print(f"\nTarget variable (MIS_Status) distribution:")
print(df["MIS_Status"].value_counts())
print(f"\nMissing values in target: {df['MIS_Status'].isna().sum()}")

fig, ax = plt.subplots(figsize=(6, 4))
df["MIS_Status"].value_counts().plot(kind="bar", ax=ax, color=["steelblue", "coral"])
ax.set_title("Loan Status Distribution")
ax.set_xlabel("Status (PIF=Paid in Full, CHGOFF=Charged Off)")
ax.set_ylabel("Count")
plt.tight_layout()
plt.show()

# Cell 4: Explore key features
features_to_explore = ["DisbursementGross", "GrAppv", "Term", "NoEmp", "CreateJob", "RetainedJob"]
print("\nSummary statistics by loan status:")
for col in features_to_explore:
    if col in df.columns:
        print(f"\n--- {col} ---")
        print(df.groupby("MIS_Status")[col].describe().loc[["PIF", "CHGOFF"]])

# Cell 5: Data cleaning
df_clean = df.dropna(subset=["MIS_Status"]).copy()
print(f"Shape after dropping missing target: {df_clean.shape}")

currency_cols = ["DisbursementGross", "GrAppv", "SBA_Appv"]
for col in currency_cols:
    if col in df_clean.columns:
        df_clean[col] = (
            df_clean[col]
            .astype(str)
            .str.replace("$", "", regex=False)
            .str.replace(",", "", regex=False)
            .astype(float)
        )

df_clean["Target"] = (df_clean["MIS_Status"] == "CHGOFF").astype(int)
print(f"\nTarget distribution after encoding:")
print(df_clean["Target"].value_counts())

# Cell 6: Feature engineering
df_clean["LoanToApproval_Ratio"] = df_clean["DisbursementGross"] / df_clean["GrAppv"].replace(0, np.nan)
df_clean["IsNewBusiness"] = (df_clean["NewExist"] == 2).astype(int)
df_clean["IsUrban"] = (df_clean["UrbanRural"] == 1).astype(int)

feature_cols = [
    "Term",
    "NoEmp",
    "CreateJob",
    "RetainedJob",
    "DisbursementGross",
    "GrAppv",
    "SBA_Appv",
    "LoanToApproval_Ratio",
    "IsNewBusiness",
    "IsUrban",
]

df_model = df_clean[feature_cols + ["Target"]].dropna()
print(f"\nFinal dataset shape for modeling: {df_model.shape}")

X = df_model[feature_cols]
y = df_model["Target"]

# Cell 7: Train/test split with stratification
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"Training class distribution:\n{y_train.value_counts()}")

Your data is cleaned, engineered, and split into training and test sets. Next up, you will train a baseline model and discover why accuracy is a misleading metric for fraud detection.

Train a Baseline Model and See Why Accuracy Lies

Your data is cleaned, features are engineered, and your train/test split is ready. Now comes the moment of truth: training your first model.

In this step, you will train a Random Forest classifier on the raw imbalanced data and discover a critical lesson. A model can look 80%+ accurate while being completely useless at catching fraud.

In this step, get ready to:
  • Train a baseline Random Forest model on imbalanced data.
  • Expose the accuracy trap using a confusion matrix.
  • Learn why ROC-AUC is the right metric for fraud detection.
Train the baseline Random Forest model

A Random Forest is an ensemble of decision trees that each vote on whether a loan application is legitimate or fraudulent. By training it on the raw imbalanced data first, you will see exactly how class imbalance corrupts model performance.

  • Create a new cell in your notebook (Cell 8) and add the following code to train the baseline model:
# Cell 8: Baseline model (imbalanced)
rf_baseline = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
rf_baseline.fit(X_train, y_train)

y_pred_baseline = rf_baseline.predict(X_test)
accuracy_baseline = rf_baseline.score(X_test, y_test)
print(f"Baseline Accuracy: {accuracy_baseline:.4f}")
print(f"\nBaseline Classification Report:")
print(classification_report(y_test, y_pred_baseline, target_names=["Legitimate", "Fraud/Default"]))

What does this code do?

RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1) creates 100 decision trees. n_jobs=-1 uses all your CPU cores for faster training.

The classification_report prints precision, recall, and F1 for each class. This is where the accuracy trap becomes visible. The model scores high overall accuracy but has poor recall for the Fraud/Default class.

  • Run the cell by pressing Shift+Enter.
  • You should see a Baseline Accuracy score of approximately 0.80 or higher. Look at the classification report and notice that recall for the Fraud/Default class is significantly lower than recall for Legitimate.

Why is high accuracy misleading here?

Your dataset has roughly 80% legitimate loans and 20% defaults. A model that blindly predicts "legitimate" for every single application would score 80% accuracy while catching zero fraud.

This is the accuracy trap. The model learns it is "safer" to predict the majority class because it gets rewarded with a high accuracy number. In fraud detection, the minority class is the entire point.

Cell taking too long to run?

Training a Random Forest on hundreds of thousands of rows can take 30-60 seconds depending on your machine. The n_jobs=-1 parameter already parallelizes across all CPU cores. If it takes longer than two minutes, make sure no other heavy processes are running.

Why is my Random Forest model training slowly?

Visualize the confusion matrix

A confusion matrix shows exactly where your model gets confused. It breaks predictions into four categories: true positives, true negatives, false positives, and false negatives.

  • Create a new cell (Cell 9) and add this code to visualize the confusion matrix:
# Cell 9: Baseline confusion matrix
fig, ax = plt.subplots(figsize=(6, 5))
cm_baseline = confusion_matrix(y_test, y_pred_baseline)
ConfusionMatrixDisplay(cm_baseline, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=ax, cmap="Blues")
ax.set_title("Baseline Model - Confusion Matrix")
plt.tight_layout()
plt.show()

How do I read this matrix?

  • Top-left: legitimate loans correctly identified as legitimate (true negatives).
  • Bottom-right: fraudulent loans correctly caught (true positives). This number is what matters most.
  • Bottom-left: fraudulent loans the model missed and labeled "Legitimate" (false negatives). These are the dangerous ones.
  • Top-right: legitimate loans wrongly flagged as fraud (false positives). Annoying, but not catastrophic.
  • Run the cell.
  • You should see a heatmap with "Baseline Model - Confusion Matrix" as the title. Notice the bottom-left cell (false negatives) is large relative to the bottom-right (true positives). The model is missing a lot of fraud.

Seeing a blank or oddly shaped chart?

Make sure you ran Cell 8 first. The variable y_pred_baseline must exist before Cell 9 can compute the confusion matrix. If you restarted the kernel, re-run all cells from Cell 1.

Help me fix my confusion matrix display

Plot the ROC curve and AUC score

The ROC curve plots the trade-off between catching fraud (true positive rate) and triggering false alarms (false positive rate) at every possible decision threshold. The area under this curve (AUC) gives you a single number that summarizes model quality regardless of class imbalance.

  • Create a new cell (Cell 10) and add this code to compute and plot the ROC curve:
# Cell 10: Baseline ROC curve
y_prob_baseline = rf_baseline.predict_proba(X_test)[:, 1]
auc_baseline = roc_auc_score(y_test, y_prob_baseline)
fpr_baseline, tpr_baseline, _ = roc_curve(y_test, y_prob_baseline)

fig, ax = plt.subplots(figsize=(7, 5))
ax.plot(fpr_baseline, tpr_baseline, label=f"Baseline (AUC = {auc_baseline:.3f})", color="steelblue")
ax.plot([0, 1], [0, 1], "k--", label="Random (AUC = 0.5)")
ax.set_xlabel("False Positive Rate")
ax.set_ylabel("True Positive Rate (Recall)")
ax.set_title("ROC Curve - Baseline Model")
ax.legend()
plt.tight_layout()
plt.show()

What does this code do?

predict_proba(X_test)[:, 1] gets the predicted probability of being fraudulent (class 1) for each loan. Instead of a hard yes/no prediction, this gives you a confidence score between 0 and 1.

The dashed diagonal line represents a model that guesses randomly (AUC = 0.5). The further your model's curve bows toward the top-left corner, the better it separates legitimate loans from fraudulent ones.

  • Run the cell.
  • You should see an ROC curve that bows above the diagonal line, with the AUC score displayed in the legend. An AUC above 0.5 means the model has learned something, but the poor recall from the classification report shows it still fails at the task that matters.

Why is AUC-ROC better than accuracy for fraud detection?

Accuracy treats all mistakes equally. In fraud detection, missing a fraudulent loan (false negative) is far more costly than flagging a legitimate one for review (false positive). AUC-ROC evaluates the model across all possible thresholds, giving you a fuller picture of its discrimination ability.

Getting a ValueError on roc_auc_score?

This can happen if y_test only contains one class. Make sure your Cell 7 train/test split used stratify=y to preserve both classes in the test set. If you modified the split, re-run Cell 7 exactly as written above.

Help me fix my ROC-AUC calculation

✔️ Awesome, I've got everything!

Great. Double check you have Cells 8, 9, and 10 all producing output before moving on.

ⓧ I'd like to double check the full code

Here is the full notebook up to this point. Compare your cells against this:

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score,
    roc_curve,
    ConfusionMatrixDisplay,
)
from imblearn.over_sampling import SMOTE

# Cell 2: Load the dataset
df = pd.read_csv("SBAnational.csv", low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head()

# Cell 3: Explore the target variable
print(f"\nTarget variable (MIS_Status) distribution:")
print(df["MIS_Status"].value_counts())
print(f"\nMissing values in target: {df['MIS_Status'].isna().sum()}")

fig, ax = plt.subplots(figsize=(6, 4))
df["MIS_Status"].value_counts().plot(kind="bar", ax=ax, color=["steelblue", "coral"])
ax.set_title("Loan Status Distribution")
ax.set_xlabel("Status (PIF=Paid in Full, CHGOFF=Charged Off)")
ax.set_ylabel("Count")
plt.tight_layout()
plt.show()

# Cell 4: Explore key features
features_to_explore = ["DisbursementGross", "GrAppv", "Term", "NoEmp", "CreateJob", "RetainedJob"]
print("\nSummary statistics by loan status:")
for col in features_to_explore:
    if col in df.columns:
        print(f"\n--- {col} ---")
        print(df.groupby("MIS_Status")[col].describe().loc[["PIF", "CHGOFF"]])

# Cell 5: Data cleaning
df_clean = df.dropna(subset=["MIS_Status"]).copy()
print(f"Shape after dropping missing target: {df_clean.shape}")

currency_cols = ["DisbursementGross", "GrAppv", "SBA_Appv"]
for col in currency_cols:
    if col in df_clean.columns:
        df_clean[col] = (
            df_clean[col]
            .astype(str)
            .str.replace("$", "", regex=False)
            .str.replace(",", "", regex=False)
            .astype(float)
        )

df_clean["Target"] = (df_clean["MIS_Status"] == "CHGOFF").astype(int)
print(f"\nTarget distribution after encoding:")
print(df_clean["Target"].value_counts())

# Cell 6: Feature engineering
df_clean["LoanToApproval_Ratio"] = df_clean["DisbursementGross"] / df_clean["GrAppv"].replace(0, np.nan)
df_clean["IsNewBusiness"] = (df_clean["NewExist"] == 2).astype(int)
df_clean["IsUrban"] = (df_clean["UrbanRural"] == 1).astype(int)

feature_cols = [
    "Term",
    "NoEmp",
    "CreateJob",
    "RetainedJob",
    "DisbursementGross",
    "GrAppv",
    "SBA_Appv",
    "LoanToApproval_Ratio",
    "IsNewBusiness",
    "IsUrban",
]

df_model = df_clean[feature_cols + ["Target"]].dropna()
print(f"\nFinal dataset shape for modeling: {df_model.shape}")

X = df_model[feature_cols]
y = df_model["Target"]

# Cell 7: Train/test split with stratification
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"Training class distribution:\n{y_train.value_counts()}")

# Cell 8: Baseline model (imbalanced)
rf_baseline = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
rf_baseline.fit(X_train, y_train)

y_pred_baseline = rf_baseline.predict(X_test)
accuracy_baseline = rf_baseline.score(X_test, y_test)
print(f"Baseline Accuracy: {accuracy_baseline:.4f}")
print(f"\nBaseline Classification Report:")
print(classification_report(y_test, y_pred_baseline, target_names=["Legitimate", "Fraud/Default"]))

# Cell 9: Baseline confusion matrix
fig, ax = plt.subplots(figsize=(6, 5))
cm_baseline = confusion_matrix(y_test, y_pred_baseline)
ConfusionMatrixDisplay(cm_baseline, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=ax, cmap="Blues")
ax.set_title("Baseline Model - Confusion Matrix")
plt.tight_layout()
plt.show()

# Cell 10: Baseline ROC curve
y_prob_baseline = rf_baseline.predict_proba(X_test)[:, 1]
auc_baseline = roc_auc_score(y_test, y_prob_baseline)
fpr_baseline, tpr_baseline, _ = roc_curve(y_test, y_prob_baseline)

fig, ax = plt.subplots(figsize=(7, 5))
ax.plot(fpr_baseline, tpr_baseline, label=f"Baseline (AUC = {auc_baseline:.3f})", color="steelblue")
ax.plot([0, 1], [0, 1], "k--", label="Random (AUC = 0.5)")
ax.set_xlabel("False Positive Rate")
ax.set_ylabel("True Positive Rate (Recall)")
ax.set_title("ROC Curve - Baseline Model")
ax.legend()
plt.tight_layout()
plt.show()

You have proven the problem: accuracy is a trap for imbalanced datasets. Your model looks great on paper but misses a huge portion of the fraud it should be catching. In the next step, you will fix this imbalance using SMOTE and retrain the model to actually prioritize catching fraud.

Fix the Imbalance with SMOTE and Retrain

In the previous step, you exposed a critical flaw in your baseline model. It looks accurate on paper, but it completely fails at the one job it was built to do: catching fraud.

The root cause is class imbalance. Your training data has far more legitimate loans than defaults, so the model learns to play it safe and predict "legitimate" almost every time. In this step, you will fix the imbalance using SMOTE (Synthetic Minority Oversampling Technique), retrain the model, and watch the fraud recall jump dramatically.

In this step, get ready to:
  • Apply SMOTE to generate synthetic fraud examples and balance the training data.
  • Train an improved Random Forest with balanced class weights.
  • Compare before/after confusion matrices to see the recall improvement.
Apply SMOTE to balance the training data

SMOTE works by creating synthetic examples of the minority class (fraud/default). It picks a fraud sample, finds its nearest neighbors in feature space, and generates new points along the line between them. This gives the model more fraud examples to learn from without simply duplicating existing rows.

  • Create a new cell in your notebook (Cell 11) and apply SMOTE to the training data by running this code:
# Cell 11: Apply SMOTE to balance training data
smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)
print(f"Before SMOTE: {y_train.value_counts().to_dict()}")
print(f"After SMOTE: {pd.Series(y_train_resampled).value_counts().to_dict()}")

What does this code do?

SMOTE only touches the training data. The test set stays untouched so your evaluation reflects real-world conditions where fraud is still rare.

The fit_resample() method analyzes the class distribution, then creates enough synthetic fraud examples to match the count of legitimate loans. The print statements let you see the before/after counts.

  • Run the cell by pressing Shift+Enter.
  • You should see two lines of output. The "Before" line shows a large gap between class 0 and class 1. The "After" line shows both classes with the same count.

Why only resample the training set?

If you applied SMOTE to the entire dataset before splitting, synthetic fraud examples based on test-set records would leak into training. Your model would memorize patterns it should never have seen, producing inflated metrics that collapse on real data.

Seeing a MemoryError?

SMOTE creates many new rows. If your machine runs low on memory, restart the kernel (Kernel > Restart in the notebook toolbar), then re-run all cells from the top. Close other heavy applications to free up RAM.

Help me debug a MemoryError when running SMOTE on my training data.

Train the improved model on balanced data

Now that your training data has equal representation of both classes, the model can learn what fraud actually looks like. You will also add class_weight="balanced" as a second layer of protection, which tells the algorithm to pay extra attention to whichever class is rarer during each tree's construction.

  • Create Cell 12 and train the improved model by running this code:
# Cell 12: Train improved model on balanced data
rf_improved = RandomForestClassifier(
    n_estimators=200,
    class_weight="balanced",
    random_state=42,
    n_jobs=-1,
)
rf_improved.fit(X_train_resampled, y_train_resampled)

y_pred_improved = rf_improved.predict(X_test)
print(f"\nImproved Model Classification Report:")
print(classification_report(y_test, y_pred_improved, target_names=["Legitimate", "Fraud/Default"]))

What does this code do?

  • n_estimators=200 doubles the number of decision trees from the baseline's 100. More trees means more "votes" on each prediction, reducing noise.
  • class_weight="balanced" adjusts sample weights inversely proportional to class frequencies. Combined with SMOTE, this gives the model a strong signal to prioritize the fraud class.
  • The model is evaluated on the original imbalanced test set (X_test, y_test). This means the results reflect how the model would perform on real incoming loans.
  • Run the cell. This may take 10-30 seconds depending on your machine because training 200 trees on the larger resampled dataset takes more time.
  • In the classification report output, look at the "Fraud/Default" row. The recall value should be significantly higher than the baseline model's recall (which was poor).

Model training seems frozen?

Training 200 trees on hundreds of thousands of resampled rows can take up to a minute on older hardware. The n_jobs=-1 parameter uses all CPU cores to speed things up. If it takes longer than 2 minutes, check that no other CPU-intensive programs are running.

Help me figure out why my Random Forest training is taking too long.

Compare the before and after confusion matrices

Numbers in a classification report can be hard to internalize. A side-by-side confusion matrix comparison makes the improvement visually obvious. You will see the model's fraud detection jump from the baseline to the SMOTE-balanced version.

  • Create Cell 13 and plot the comparison by running this code:
# Cell 13: Compare confusion matrices side by side
fig, axes = plt.subplots(1, 2, figsize=(12, 5))

cm_improved = confusion_matrix(y_test, y_pred_improved)
ConfusionMatrixDisplay(cm_baseline, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=axes[0], cmap="Blues")
axes[0].set_title("Baseline (Imbalanced)")

ConfusionMatrixDisplay(cm_improved, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=axes[1], cmap="Greens")
axes[1].set_title("After SMOTE + Balanced Weights")

plt.tight_layout()
plt.show()

What does this code do?

This plots two confusion matrices next to each other. The left matrix (blue) is your baseline from the previous step. The right matrix (green) is the new SMOTE-balanced model.

Focus on the bottom-right cell of each matrix. That is the count of correctly detected fraud cases (true positives). The green matrix should show a much larger number there, meaning the model now catches far more fraudulent loans.

  • Run the cell.
  • You should see two heatmaps side by side. The right matrix (green) should have a dramatically higher number in the bottom-right cell (Fraud/Default predicted as Fraud/Default) compared to the left matrix (blue).

What is the trade-off?

You may notice the top-right cell in the green matrix is larger than in the blue matrix. This means some legitimate loans are now flagged as suspicious (false positives). That is the precision-recall trade-off in action.

In fraud detection, this trade-off is acceptable. Investigating a few extra legitimate applications is far cheaper than letting fraudulent loans slip through unchallenged.

Both matrices look the same?

Make sure Cell 12 trained on X_train_resampled (not X_train). If you accidentally passed the original training data, the model will produce similar results to the baseline.

Help me figure out why my improved model's confusion matrix looks the same as the baseline.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook file.

ⓧ I'd like to double check the full code

Here is the complete notebook code up to this point for comparison.

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score,
    roc_curve,
    ConfusionMatrixDisplay,
)
from imblearn.over_sampling import SMOTE

# Cell 2: Load the dataset
df = pd.read_csv("SBAnational.csv", low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head()

# Cell 3: Explore the target variable
print(f"\nTarget variable (MIS_Status) distribution:")
print(df["MIS_Status"].value_counts())
print(f"\nMissing values in target: {df['MIS_Status'].isna().sum()}")

fig, ax = plt.subplots(figsize=(6, 4))
df["MIS_Status"].value_counts().plot(kind="bar", ax=ax, color=["steelblue", "coral"])
ax.set_title("Loan Status Distribution")
ax.set_xlabel("Status (PIF=Paid in Full, CHGOFF=Charged Off)")
ax.set_ylabel("Count")
plt.tight_layout()
plt.show()

# Cell 4: Explore key features
features_to_explore = ["DisbursementGross", "GrAppv", "Term", "NoEmp", "CreateJob", "RetainedJob"]
print("\nSummary statistics by loan status:")
for col in features_to_explore:
    if col in df.columns:
        print(f"\n--- {col} ---")
        print(df.groupby("MIS_Status")[col].describe().loc[["PIF", "CHGOFF"]])

# Cell 5: Data cleaning
df_clean = df.dropna(subset=["MIS_Status"]).copy()
print(f"Shape after dropping missing target: {df_clean.shape}")

currency_cols = ["DisbursementGross", "GrAppv", "SBA_Appv"]
for col in currency_cols:
    if col in df_clean.columns:
        df_clean[col] = (
            df_clean[col]
            .astype(str)
            .str.replace("$", "", regex=False)
            .str.replace(",", "", regex=False)
            .astype(float)
        )

df_clean["Target"] = (df_clean["MIS_Status"] == "CHGOFF").astype(int)
print(f"\nTarget distribution after encoding:")
print(df_clean["Target"].value_counts())

# Cell 6: Feature engineering
df_clean["LoanToApproval_Ratio"] = df_clean["DisbursementGross"] / df_clean["GrAppv"].replace(0, np.nan)
df_clean["IsNewBusiness"] = (df_clean["NewExist"] == 2).astype(int)
df_clean["IsUrban"] = (df_clean["UrbanRural"] == 1).astype(int)

feature_cols = [
    "Term",
    "NoEmp",
    "CreateJob",
    "RetainedJob",
    "DisbursementGross",
    "GrAppv",
    "SBA_Appv",
    "LoanToApproval_Ratio",
    "IsNewBusiness",
    "IsUrban",
]

df_model = df_clean[feature_cols + ["Target"]].dropna()
print(f"\nFinal dataset shape for modeling: {df_model.shape}")

X = df_model[feature_cols]
y = df_model["Target"]

# Cell 7: Train/test split with stratification
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"Training class distribution:\n{y_train.value_counts()}")

# Cell 8: Baseline model (imbalanced)
rf_baseline = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
rf_baseline.fit(X_train, y_train)

y_pred_baseline = rf_baseline.predict(X_test)
accuracy_baseline = rf_baseline.score(X_test, y_test)
print(f"Baseline Accuracy: {accuracy_baseline:.4f}")
print(f"\nBaseline Classification Report:")
print(classification_report(y_test, y_pred_baseline, target_names=["Legitimate", "Fraud/Default"]))

# Cell 9: Baseline confusion matrix
fig, ax = plt.subplots(figsize=(6, 5))
cm_baseline = confusion_matrix(y_test, y_pred_baseline)
ConfusionMatrixDisplay(cm_baseline, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=ax, cmap="Blues")
ax.set_title("Baseline Model - Confusion Matrix")
plt.tight_layout()
plt.show()

# Cell 10: Baseline ROC curve
y_prob_baseline = rf_baseline.predict_proba(X_test)[:, 1]
auc_baseline = roc_auc_score(y_test, y_prob_baseline)
fpr_baseline, tpr_baseline, _ = roc_curve(y_test, y_prob_baseline)

fig, ax = plt.subplots(figsize=(7, 5))
ax.plot(fpr_baseline, tpr_baseline, label=f"Baseline (AUC = {auc_baseline:.3f})", color="steelblue")
ax.plot([0, 1], [0, 1], "k--", label="Random (AUC = 0.5)")
ax.set_xlabel("False Positive Rate")
ax.set_ylabel("True Positive Rate (Recall)")
ax.set_title("ROC Curve - Baseline Model")
ax.legend()
plt.tight_layout()
plt.show()

# Cell 11: Apply SMOTE to balance training data
smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)
print(f"Before SMOTE: {y_train.value_counts().to_dict()}")
print(f"After SMOTE: {pd.Series(y_train_resampled).value_counts().to_dict()}")

# Cell 12: Train improved model on balanced data
rf_improved = RandomForestClassifier(
    n_estimators=200,
    class_weight="balanced",
    random_state=42,
    n_jobs=-1,
)
rf_improved.fit(X_train_resampled, y_train_resampled)

y_pred_improved = rf_improved.predict(X_test)
print(f"\nImproved Model Classification Report:")
print(classification_report(y_test, y_pred_improved, target_names=["Legitimate", "Fraud/Default"]))

# Cell 13: Compare confusion matrices side by side
fig, axes = plt.subplots(1, 2, figsize=(12, 5))

cm_improved = confusion_matrix(y_test, y_pred_improved)
ConfusionMatrixDisplay(cm_baseline, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=axes[0], cmap="Blues")
axes[0].set_title("Baseline (Imbalanced)")

ConfusionMatrixDisplay(cm_improved, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=axes[1], cmap="Greens")
axes[1].set_title("After SMOTE + Balanced Weights")

plt.tight_layout()
plt.show()

Your model can now actually detect fraud. Next up, you will dig into which loan features the model considers the most suspicious and build a reusable scoring function that can flag new applications in real time.

Interpret Results and Build a Scoring Function

Your improved model catches far more fraudulent loans than the baseline. But a model is only useful if you can explain why it flags certain applications and put it to work on new data.

In this step, you will extract the features your Random Forest relies on most, visualize them, and wrap everything into a reusable function that scores new loan applications as "High Risk" or "Low Risk."

In this step, get ready to:
  • Extract and visualize the top 10 most important features from your trained model.
  • Build a reusable scoring function that classifies new loan applications.
  • Test the function with example loan applications to verify it works.
Extract and plot feature importances

Random Forest models track how much each feature contributes to splitting decisions across all their trees. This is stored as a numeric importance score for every input column.

  • Create a new cell in your notebook (Cell 14) and add the following code to extract and visualize feature importances:
# Cell 14: Feature importance
importances = rf_improved.feature_importances_
feature_importance_df = pd.DataFrame({
    "Feature": feature_cols,
    "Importance": importances
}).sort_values("Importance", ascending=False)

fig, ax = plt.subplots(figsize=(8, 5))
sns.barplot(data=feature_importance_df.head(10), x="Importance", y="Feature", ax=ax, palette="viridis")
ax.set_title("Top 10 Feature Importances for Fraud Detection")
plt.tight_layout()
plt.show()

print("\nTop fraud indicators:")
for _, row in feature_importance_df.head(5).iterrows():
    print(f"  {row['Feature']}: {row['Importance']:.4f}")

What does this code do?

The .feature_importances_ attribute gives you a score for each feature showing how much it contributed to the model's decisions. Higher values mean the model relies more heavily on that feature when separating fraudulent from legitimate loans.

The seaborn bar chart makes it easy to see at a glance which loan attributes are the strongest fraud signals. The printed output lists the top 5 numerically.

  • Run the cell.
  • You should see a horizontal bar chart titled "Top 10 Feature Importances for Fraud Detection" followed by a printed list of the top 5 features with their scores.

What makes a loan look suspicious?

Look at which features rank highest. Typical strong fraud indicators include:

  • A high disbursement-to-approval ratio (the borrower received more than expected).
  • A short loan term (less time to repay suggests higher risk).
  • A new business with few employees and no jobs created or retained.

Not seeing the chart?

Make sure you ran all previous cells first (Cells 1-13). The cell depends on rf_improved and feature_cols from earlier cells.

help me debug my feature importance chart

Build a scoring function for new loan applications

Your model currently works on test data you already have. To make it useful in the real world, you need a function that takes raw loan attributes and returns a clear risk verdict.

  • Create a new cell (Cell 15) and add the scoring function:
# Cell 15: Scoring function
def score_loan_application(term, no_emp, create_job, retained_job,
                           disbursement_gross, gr_appv, sba_appv,
                           is_new_business, is_urban):
    """Score a loan application for fraud/default risk."""
    loan_to_approval = disbursement_gross / gr_appv if gr_appv > 0 else 1.0

    features = np.array([[
        term, no_emp, create_job, retained_job,
        disbursement_gross, gr_appv, sba_appv,
        loan_to_approval, is_new_business, is_urban
    ]])

    probability = rf_improved.predict_proba(features)[0][1]
    risk_label = "High Risk" if probability >= 0.5 else "Low Risk"

    return {"risk_label": risk_label, "default_probability": round(probability, 4)}

What does this function do?

The function accepts raw loan attributes as parameters. It computes loan_to_approval internally (matching the feature engineering you did earlier), then assembles all features into the exact order the model expects.

It calls predict_proba to get the probability of default (class 1). If that probability is 0.5 or higher, the loan is labeled "High Risk." The function returns a dictionary with both the label and the exact probability.

  • Run the cell to define the function.
  • You should see no output yet. The function is now loaded into memory and ready to use.

Seeing a NameError?

If you see an error mentioning rf_improved or np, your kernel may have restarted. Re-run all cells from Cell 1 through Cell 12 to restore the model and imports.

help me fix my scoring function

Test the scoring function with example loans

A good test uses two contrasting inputs: one that looks clearly legitimate and one that has multiple red flags. This confirms the model is distinguishing between the two.

  • Create a new cell (Cell 16) and test your scoring function with two example applications:
# Cell 16: Test the scoring function
# Example 1: A seemingly legitimate loan
result_legitimate = score_loan_application(
    term=84, no_emp=10, create_job=5, retained_job=10,
    disbursement_gross=150000, gr_appv=150000, sba_appv=120000,
    is_new_business=0, is_urban=1
)
print(f"Legitimate-looking loan: {result_legitimate}")

# Example 2: A suspicious loan
result_suspicious = score_loan_application(
    term=12, no_emp=1, create_job=0, retained_job=0,
    disbursement_gross=500000, gr_appv=300000, sba_appv=250000,
    is_new_business=1, is_urban=0
)
print(f"Suspicious loan: {result_suspicious}")

What do these test cases represent?

The first loan is an established urban business with 10 employees, requesting exactly what it was approved for, over an 84-month term. This profile has many markers of a stable, legitimate application.

The second loan is a brand-new rural business with 1 employee, creating no jobs, requesting $500K when only $300K was approved, on a 12-month term. Multiple red flags appear at once: the high disbursement-to-approval ratio, short term, and new business with no jobs.

  • Run the cell.
  • You should see two printed lines. The legitimate loan should show Low Risk with a low probability. The suspicious loan should show High Risk with a high probability.

Both loans showing the same risk label?

Double-check that Cell 12 trained rf_improved on the SMOTE-resampled data. If you accidentally trained on X_train instead of X_train_resampled, the model may be biased toward always predicting "Low Risk."

help me debug my loan scoring results

✔️ Awesome, I've got everything!

Great. Double-check you have saved your notebook file.

ⓧ I'd like to double check the full code

Here is the complete notebook up through Cell 16. Compare it against your file:

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score,
    roc_curve,
    ConfusionMatrixDisplay,
)
from imblearn.over_sampling import SMOTE

# Cell 2: Load the dataset
df = pd.read_csv("SBAnational.csv", low_memory=False)
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head()

# Cell 3: Explore the target variable
print(f"\nTarget variable (MIS_Status) distribution:")
print(df["MIS_Status"].value_counts())
print(f"\nMissing values in target: {df['MIS_Status'].isna().sum()}")

fig, ax = plt.subplots(figsize=(6, 4))
df["MIS_Status"].value_counts().plot(kind="bar", ax=ax, color=["steelblue", "coral"])
ax.set_title("Loan Status Distribution")
ax.set_xlabel("Status (PIF=Paid in Full, CHGOFF=Charged Off)")
ax.set_ylabel("Count")
plt.tight_layout()
plt.show()

# Cell 4: Explore key features
features_to_explore = ["DisbursementGross", "GrAppv", "Term", "NoEmp", "CreateJob", "RetainedJob"]
print("\nSummary statistics by loan status:")
for col in features_to_explore:
    if col in df.columns:
        print(f"\n--- {col} ---")
        print(df.groupby("MIS_Status")[col].describe().loc[["PIF", "CHGOFF"]])

# Cell 5: Data cleaning
df_clean = df.dropna(subset=["MIS_Status"]).copy()
print(f"Shape after dropping missing target: {df_clean.shape}")

currency_cols = ["DisbursementGross", "GrAppv", "SBA_Appv"]
for col in currency_cols:
    if col in df_clean.columns:
        df_clean[col] = (
            df_clean[col]
            .astype(str)
            .str.replace("$", "", regex=False)
            .str.replace(",", "", regex=False)
            .astype(float)
        )

df_clean["Target"] = (df_clean["MIS_Status"] == "CHGOFF").astype(int)
print(f"\nTarget distribution after encoding:")
print(df_clean["Target"].value_counts())

# Cell 6: Feature engineering
df_clean["LoanToApproval_Ratio"] = df_clean["DisbursementGross"] / df_clean["GrAppv"].replace(0, np.nan)
df_clean["IsNewBusiness"] = (df_clean["NewExist"] == 2).astype(int)
df_clean["IsUrban"] = (df_clean["UrbanRural"] == 1).astype(int)

feature_cols = [
    "Term",
    "NoEmp",
    "CreateJob",
    "RetainedJob",
    "DisbursementGross",
    "GrAppv",
    "SBA_Appv",
    "LoanToApproval_Ratio",
    "IsNewBusiness",
    "IsUrban",
]

df_model = df_clean[feature_cols + ["Target"]].dropna()
print(f"\nFinal dataset shape for modeling: {df_model.shape}")

X = df_model[feature_cols]
y = df_model["Target"]

# Cell 7: Train/test split with stratification
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"Training class distribution:\n{y_train.value_counts()}")

# Cell 8: Baseline model (imbalanced)
rf_baseline = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
rf_baseline.fit(X_train, y_train)

y_pred_baseline = rf_baseline.predict(X_test)
accuracy_baseline = rf_baseline.score(X_test, y_test)
print(f"Baseline Accuracy: {accuracy_baseline:.4f}")
print(f"\nBaseline Classification Report:")
print(classification_report(y_test, y_pred_baseline, target_names=["Legitimate", "Fraud/Default"]))

# Cell 9: Baseline confusion matrix
fig, ax = plt.subplots(figsize=(6, 5))
cm_baseline = confusion_matrix(y_test, y_pred_baseline)
ConfusionMatrixDisplay(cm_baseline, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=ax, cmap="Blues")
ax.set_title("Baseline Model - Confusion Matrix")
plt.tight_layout()
plt.show()

# Cell 10: Baseline ROC curve
y_prob_baseline = rf_baseline.predict_proba(X_test)[:, 1]
auc_baseline = roc_auc_score(y_test, y_prob_baseline)
fpr_baseline, tpr_baseline, _ = roc_curve(y_test, y_prob_baseline)

fig, ax = plt.subplots(figsize=(7, 5))
ax.plot(fpr_baseline, tpr_baseline, label=f"Baseline (AUC = {auc_baseline:.3f})", color="steelblue")
ax.plot([0, 1], [0, 1], "k--", label="Random (AUC = 0.5)")
ax.set_xlabel("False Positive Rate")
ax.set_ylabel("True Positive Rate (Recall)")
ax.set_title("ROC Curve - Baseline Model")
ax.legend()
plt.tight_layout()
plt.show()

# Cell 11: Apply SMOTE to balance training data
smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)
print(f"Before SMOTE: {y_train.value_counts().to_dict()}")
print(f"After SMOTE: {pd.Series(y_train_resampled).value_counts().to_dict()}")

# Cell 12: Train improved model on balanced data
rf_improved = RandomForestClassifier(
    n_estimators=200,
    class_weight="balanced",
    random_state=42,
    n_jobs=-1,
)
rf_improved.fit(X_train_resampled, y_train_resampled)

y_pred_improved = rf_improved.predict(X_test)
print(f"\nImproved Model Classification Report:")
print(classification_report(y_test, y_pred_improved, target_names=["Legitimate", "Fraud/Default"]))

# Cell 13: Compare confusion matrices side by side
fig, axes = plt.subplots(1, 2, figsize=(12, 5))

cm_improved = confusion_matrix(y_test, y_pred_improved)
ConfusionMatrixDisplay(cm_baseline, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=axes[0], cmap="Blues")
axes[0].set_title("Baseline (Imbalanced)")

ConfusionMatrixDisplay(cm_improved, display_labels=["Legitimate", "Fraud/Default"]).plot(ax=axes[1], cmap="Greens")
axes[1].set_title("After SMOTE + Balanced Weights")

plt.tight_layout()
plt.show()

# Cell 14: Feature importance
importances = rf_improved.feature_importances_
feature_importance_df = pd.DataFrame({
    "Feature": feature_cols,
    "Importance": importances
}).sort_values("Importance", ascending=False)

fig, ax = plt.subplots(figsize=(8, 5))
sns.barplot(data=feature_importance_df.head(10), x="Importance", y="Feature", ax=ax, palette="viridis")
ax.set_title("Top 10 Feature Importances for Fraud Detection")
plt.tight_layout()
plt.show()

print("\nTop fraud indicators:")
for _, row in feature_importance_df.head(5).iterrows():
    print(f"  {row['Feature']}: {row['Importance']:.4f}")

# Cell 15: Scoring function
def score_loan_application(term, no_emp, create_job, retained_job,
                           disbursement_gross, gr_appv, sba_appv,
                           is_new_business, is_urban):
    """Score a loan application for fraud/default risk."""
    loan_to_approval = disbursement_gross / gr_appv if gr_appv > 0 else 1.0

    features = np.array([[
        term, no_emp, create_job, retained_job,
        disbursement_gross, gr_appv, sba_appv,
        loan_to_approval, is_new_business, is_urban
    ]])

    probability = rf_improved.predict_proba(features)[0][1]
    risk_label = "High Risk" if probability >= 0.5 else "Low Risk"

    return {"risk_label": risk_label, "default_probability": round(probability, 4)}

# Cell 16: Test the scoring function
result_legitimate = score_loan_application(
    term=84, no_emp=10, create_job=5, retained_job=10,
    disbursement_gross=150000, gr_appv=150000, sba_appv=120000,
    is_new_business=0, is_urban=1
)
print(f"Legitimate-looking loan: {result_legitimate}")

result_suspicious = score_loan_application(
    term=12, no_emp=1, create_job=0, retained_job=0,
    disbursement_gross=500000, gr_appv=300000, sba_appv=250000,
    is_new_business=1, is_urban=0
)
print(f"Suspicious loan: {result_suspicious}")

You now have a complete fraud detection pipeline that trains, evaluates, explains, and scores new applications. Next up is the Secret Mission, where you will tune the decision threshold to maximize real business value instead of relying on the default 0.5 cutoff.

Secret mission

Optimize the Decision Threshold for Business Value

Your model currently uses 0.5 as the decision threshold. But in real fraud detection, that default is almost never optimal. In this mission, you will define a custom business cost function and tune the threshold to maximize net savings.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources for future experimentation, pause them to free up memory, or delete them entirely.

Resources you used:

  • loan-fraud-detection project folder (includes fraud_detection.ipynb and SBAnational.csv).
  • Python packages: scikit-learn, pandas, imbalanced-learn, matplotlib, seaborn.
  • VS Code Jupyter extension.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with the notebook, try different algorithms, or revisit the Secret Mission.

  • Your project folder stays on your Desktop.
  • The installed Python packages remain available for other projects.
  • The Jupyter extension stays active in VS Code.

Pause - I'll come back to this later

Shut down running notebook kernels to free up memory. Your files and packages stay intact for when you return.

  • In VS Code, open the Command Palette with Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows).
  • Type Jupyter: Restart Kernel and select it to stop the running kernel.
  • Close the notebook tab in VS Code.

Delete - I don't want to use this again

Remove all project files and packages completely. This cannot be undone.

  • Close the notebook and any open files from the project in VS Code.
  • Delete the project folder by running the command for your operating system below.

macOS

rm -rf ~/Desktop/loan-fraud-detection

Windows

rmdir /s /q %USERPROFILE%\Desktop\loan-fraud-detection
  • Uninstall the Python packages by running:
pip uninstall scikit-learn pandas imbalanced-learn matplotlib seaborn -y
  • To remove the Jupyter extension, open the Extensions panel in VS Code (Cmd+Shift+X on macOS, Ctrl+Shift+X on Windows).
  • Search for Jupyter, click the extension, then click Uninstall.

Nice Work!

Nice Work!

You just built a fraud detection model that actually catches fraud. Starting from 899K+ real SBA loan records, you proved why accuracy lies, fixed the problem with SMOTE, and delivered a scoring function ready to flag risky applications.

Here's what you learned:

  • Proved that accuracy is a trap for imbalanced datasets by training a baseline model that looked 80%+ accurate but caught almost no fraud, then fixed it using SMOTE and class_weight="balanced" to dramatically improve recall.
  • Evaluated your model using metrics that matter for fraud detection: precision, recall, F1, ROC-AUC, and confusion matrices instead of relying on misleading accuracy scores.
  • Built a reusable score_loan_application() function that takes loan attributes and returns a risk label with predicted default probability for any new application.
  • Secret Mission: Tuned the decision threshold using a custom business cost function to maximize real-world net savings instead of relying on the default 0.5 cutoff.

Ready to quiz yourself?