Benchmark Classification Models

Compare classifiers with leakage-safe validation and a held-out test set.

Introduction

30 Second Summary

A single score can make one option look like an obvious winner. That confidence fades when the result changes with the sample or overlooks the cases that matter most.

In this project, you will build a notebook-style Python benchmark in Visual Studio Code that compares four binary classification model families plus a class-balanced variant. You will rank them with stratified cross-validation before evaluating the winner once on an untouched holdout set.

What You'll Build

Your final run displays a ranked scorecard beside its uncertainty chart before revealing the selected model's untouched holdout confusion matrix.

By the end of this project, you'll have:

  • A runnable accuracy-only leaderboard that compares a dummy baseline, logistic regression, a support vector classifier, and a random forest on one development split.
  • A visible data diagnosis that surfaces missing values, duplicate rows, class imbalance, and multicollinearity before you trust the rankings.
  • A defensible model recommendation supported by balanced accuracy, malignant-class recall, ROC AUC, score variation, fit time, and one final holdout evaluation.
  • Secret Mission: Tune a classifier's decision threshold with internal cross-validation to compare the resulting malignant-class recall trade-off.

Are there any prerequisites?

Basic Python knowledge and prior exposure to supervised machine learning will help. The project guides you through all project-specific setup for your Windows workspace.

Before We Start

This benchmark supports one decision: compare classifiers for the Breast Cancer Wisconsin (Diagnostic) dataset with malignant tumors as positive label 1. Your priority metrics are balanced accuracy for unequal class frequencies plus malignant-class recall for the cost of missed malignant cases.

Set Up the Benchmark Workspace

The benchmark decision is set. Trustworthy model comparisons now depend on a workspace that reproduces the same package versions.

You will use Visual Studio Code to run notebook-style Python cells. A dedicated virtual environment keeps this benchmark isolated from other projects on your computer.

In this step, get ready to:
  • Prepare Visual Studio Code with the Microsoft Python extension plus the Microsoft Jupyter extension.
  • Create the benchmark files inside the ml-model-benchmark folder.
  • Build a .venv environment that runs the pinned benchmark dependencies.
Verify Python and prepare Visual Studio Code

The pinned NumPy release requires Python 3.12 or newer. Checking the runtime first prevents package compatibility problems later.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to start Visual Studio Code.
  • Use the account icon in the lower-left corner to sign in to Visual Studio Code.
  • Complete the browser verification if your sign-in opens a browser window.

Your signed-in account should now appear in the account menu. This keeps your editor session connected while you build the benchmark.

  • Select Terminal from the top menu.
  • Select New Terminal to create an integrated terminal session.
  • Check your installed Python version by running this command:
python --version

What does this command check?

The command asks the active Python installation to report its version. The result determines whether your runtime can install every pinned package in this project.

✔️ I see the required version

Python reports version 3.12 or newer. Your runtime meets the benchmark requirement.

ⓧ I see an older version

Your current Python installation is below the project requirement. The Windows installer opens a guided setup flow.

  • Visit the official Python downloads page for Windows.
  • Download a current stable Windows release.
  • Run the downloaded installer.
  • Follow the installer prompts until installation completes.
  • Return to Visual Studio Code after the installation.
  • Rerun the version check shown above.
  • Confirm Python now reports version 3.12 or newer.

ⓧ Command not found

Windows cannot find a Python installation through this command. Installing a current stable release gives Visual Studio Code the runtime it needs.

  • Visit the official Python downloads page for Windows.
  • Download a current stable Windows release.
  • Run the downloaded installer.
  • Follow the installer prompts until installation completes.
  • Return to Visual Studio Code after the installation.
  • Rerun the version check shown above.
  • Confirm Python now reports version 3.12 or newer.

The two Microsoft extensions give Visual Studio Code its Python environment tools plus notebook-style cell support.

  • Press Ctrl+Shift+X to open the Extensions view.
  • Search for Python.
  • Select the Python extension published by Microsoft.
  • Click Install.
  • Search for Jupyter.
  • Select the Jupyter extension published by Microsoft.
  • Click Install.
  • Confirm both extension pages show an installed state.

That is the editor side ready. Visual Studio Code can now create the isolated environment plus run your notebook-style cells.

Extensions not installing?

Check that each extension is published by Microsoft. Confirm Visual Studio Code has an internet connection before retrying the installation.

Help me diagnose why the Microsoft Python or Jupyter extension will not install in Visual Studio Code. Share the visible extension status with the NextWork community if the issue is specific to your computer.

Create the benchmark files

Both project files belong inside one named folder so Visual Studio Code can associate them with the same interpreter. You created this folder yourself, so you can trust it if Visual Studio Code asks.

  • Select File from the top menu.
  • Select Open Folder.
  • Select your Desktop in the folder picker.
  • Use the New Folder control.
  • Enter ml-model-benchmark as the folder name.
  • Select the new ml-model-benchmark folder.
  • Click Select Folder.

You should see ml-model-benchmark at the top of the Explorer sidebar. This confirms Visual Studio Code is working inside the correct folder.

  • Select the New File icon in the Explorer sidebar.
  • Enter requirements.txt as the file name.
  • Add the pinned package list by copying this code:
numpy==2.5.3
matplotlib==3.11.2
scikit-learn==1.9.1
ipykernel==7.3.0

Why pin every package?

Each == pin tells the environment to install one exact release. NumPy handles arrays plus numerical summaries.

Matplotlib creates the benchmark charts. scikit-learn supplies the dataset plus model evaluation tools.

ipykernel connects the selected Python environment to notebook-style execution. Exact pins make the workspace reproducible.

  • Save requirements.txt by pressing Ctrl+S.
  • Confirm the Explorer sidebar lists requirements.txt inside ml-model-benchmark.

Requirements file missing?

Check that the file name ends with .txt. Confirm the Explorer sidebar is showing the ml-model-benchmark folder.

Help me find or correct my requirements.txt file in Visual Studio Code.

✔️ Awesome, I've got everything!

Your package manifest is saved inside the benchmark folder.

ⓧ I'd like to double check the full code

numpy==2.5.3
matplotlib==3.11.2
scikit-learn==1.9.1
ipykernel==7.3.0

What should match?

Compare all four package names plus their version pins with your saved file. Every character should match this reference.

The Python file begins with notebook cell markers. Its opening cell also prepares every import plus shared setting needed by later benchmark stages.

  • Select the New File icon in the Explorer sidebar.
  • Enter model_benchmark.py as the file name.
  • Build the notebook-style workspace by copying this code:
# %% [markdown]
# # Reproducible Classification Model Benchmark
# Malignant tumors are label 1, so recall measures the share of malignant
# cases identified by a model. This dataset is educational and this project
# does not create or validate a clinical diagnostic system.

# %%
print("Model benchmark workspace ready")

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, classification_report
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

RANDOM_STATE = 42
HOLDOUT_SIZE = 0.20
CORRELATION_THRESHOLD = 0.90
CLASS_NAMES = ["Benign", "Malignant"]

What does this code prepare?

  • The markdown cell states the benchmark objective plus the educational-use boundary.
  • The first executable cell prints a visible workspace check before loading the project tools.
  • The imports prepare data loading, preprocessing, model fitting, validation, metrics, plus plotting.
  • The constants keep the random seed, holdout size, correlation threshold, plus class names consistent.
  • Save model_benchmark.py by pressing Ctrl+S.
  • Confirm the Explorer sidebar lists model_benchmark.py beside requirements.txt.

Python file showing problems?

Check that the file name ends with .py. Make sure both cell markers begin with the exact characters # %%.

Help me compare my model_benchmark.py setup code with the required workspace file.

✔️ Awesome, I've got everything!

Your notebook-style Python file now contains the complete workspace scaffold.

ⓧ I'd like to double check the full code

# %% [markdown]
# # Reproducible Classification Model Benchmark
# Malignant tumors are label 1, so recall measures the share of malignant
# cases identified by a model. This dataset is educational and this project
# does not create or validate a clinical diagnostic system.

# %%
print("Model benchmark workspace ready")

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, classification_report
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

RANDOM_STATE = 42
HOLDOUT_SIZE = 0.20
CORRELATION_THRESHOLD = 0.90
CLASS_NAMES = ["Benign", "Malignant"]

What should match?

Compare the cell markers, imports, constants, plus class names with your saved file. This reference is the complete file for this step.

Build and verify the virtual environment

Visual Studio Code can build .venv directly from requirements.txt. The selected interpreter then controls both terminal commands plus Interactive window cells.

Creating the environment can take a few minutes while the pinned packages download. A quiet progress indicator during this stage is expected.

  • Press Ctrl+Shift+P to open the Command Palette.
  • Enter Python: Create Environment.
  • Select Python: Create Environment from the results.
  • Select Venv as the environment type.
  • Select a Python interpreter at version 3.12 or newer.
  • Select requirements.txt when Visual Studio Code asks for dependencies.
  • Wait until environment creation finishes.

Visual Studio Code now has a project-specific .venv containing the benchmark packages. Selecting it explicitly keeps terminal plus cell execution aligned.

  • Press Ctrl+Shift+P to return to the Command Palette.
  • Enter Python: Select Interpreter.
  • Select Python: Select Interpreter from the results.
  • Select the interpreter inside .venv.
  • Return to the terminal panel from earlier.
  • Close its current session using the trash icon.
  • Start a fresh terminal session using the plus icon.

Before you check the packages, do you expect every installed version to match the four pins in requirements.txt?

  • Inspect the installed benchmark packages by running this command:
python -m pip show scikit-learn numpy matplotlib ipykernel

What does this command inspect?

The command asks the selected Python interpreter to display installation details for all four packages. Each package section includes its installed version plus environment location.

You should see NumPy at 2.5.3. Matplotlib should report 3.11.2.

scikit-learn should report 1.9.1. ipykernel should report 7.3.0.

Package missing or wrong version?

Confirm the terminal prompt shows the selected .venv environment. Recreate the environment from requirements.txt if a package section is absent or its version differs.

Help me diagnose a missing or incorrect package version inside my .venv environment. Share the package name plus visible version with the NextWork community if installation still fails.

The final check proves that Visual Studio Code can execute the file through the selected environment. The Interactive window may ask you to choose a kernel on its first run.

  • Select model_benchmark.py in the Explorer sidebar.
  • Locate the executable cell marked with # %% below the markdown introduction.

Before you run the cell, which workspace message do you expect the Interactive window to print?

  • Select Run Cell above the executable cell.
  • Select the .venv environment if Visual Studio Code asks you to choose a kernel.

You should see Model benchmark workspace ready in the Python Interactive window. That message proves the notebook-style file is running through your prepared workspace.

Interactive window not ready?

Confirm the selected kernel belongs to .venv. Check that the executable marker is exactly # %%.

Help me connect the Python Interactive window to my selected .venv environment.

Your reproducible benchmark workspace is live. Next up, you will load the dataset plus expose the limits of an accuracy-only leaderboard.

Build and Diagnose the Baseline

Your reproducible workspace is running in Visual Studio Code. You can now turn that setup into a working model comparison.

A quick accuracy-only leaderboard is the fastest way to prove the data works with several classifiers. You will test how much confidence that first result deserves before using it for a recommendation.

In this step, get ready to:
  • Load the dataset with malignant cases defined as label 1.
  • Run four classifiers while preserving the final holdout.
  • Diagnose the build data before interpreting the leaderboard.
Load the data and protect the holdout

Holdout evaluation reserves data that the model-selection process cannot inspect. Stratification keeps the class proportions similar across each split.

  • In model_benchmark.py, place your cursor below the CLASS_NAMES line.
  • Add the dataset-loading cells by copying this code:
# %% [markdown]
# ## Load the data and reserve the final holdout

# %%
dataset = load_breast_cancer()
X = dataset.data
y = 1 - dataset.target
feature_names = dataset.feature_names

class_values, class_counts = np.unique(y, return_counts=True)
print(f"Samples: {X.shape[0]}")
print(f"Features: {X.shape[1]}")
for class_value, class_count in zip(class_values, class_counts):
    print(f"{CLASS_NAMES[class_value]}: {class_count}")

X_build, X_holdout, y_build, y_holdout = train_test_split(
    X,
    y,
    test_size=HOLDOUT_SIZE,
    stratify=y,
    random_state=RANDOM_STATE,
)

X_dev, X_validation, y_dev, y_validation = train_test_split(
    X_build,
    y_build,
    test_size=0.25,
    stratify=y_build,
    random_state=RANDOM_STATE,
)

What does this code do?

  • The dataset object holds the bundled measurements and labels.
  • The X array contains the 30 numeric features.
  • The expression y = 1 - dataset.target makes malignant cases the positive class with label 1.
  • The first split reserves 20 percent of the data in X_holdout and y_holdout.
  • The second split creates development and validation partitions inside the build data.
  • Save model_benchmark.py.
  • Run the new Python cell from the editor's cell controls.

You will see Samples: 569 and Features: 30. You will also see separate counts for benign and malignant cases.

Dataset cell not completing?

Check that this code sits below the existing imports and constants. Confirm that the Python Interactive window still uses the project .venv.

Help me debug the dataset-loading cell.

Run the accuracy-only leaderboard

A baseline gives every later result a reference point. A pipeline also lets the scale-sensitive classifiers apply scaling before fitting.

  • Place your cursor below the second train_test_split block.
  • Define the four baseline candidates by copying this code:
# %% [markdown]
# ## The intentionally incomplete accuracy-only benchmark

# %%
naive_models = {
    "Dummy baseline": DummyClassifier(strategy="prior"),
    "Logistic regression": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", LogisticRegression(max_iter=2000)),
        ]
    ),
    "SVC": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", SVC()),
        ]
    ),
    "Random forest": RandomForestClassifier(random_state=RANDOM_STATE),
}

How are the candidates different?

  • The dummy classifier predicts from the class distribution and establishes a basic reference score.
  • Logistic regression receives scaled features through StandardScaler.
  • The support vector classifier uses the same scaling pattern.
  • The random forest works with the original feature values and uses the shared random seed.
  • Save model_benchmark.py.
  • Run the model-definition cell from the editor's cell controls.
  • Inspect naive_models in the Interactive window's variable viewer.

You will see a dictionary containing four named model candidates. This confirms that every estimator is ready for the same development split.

Missing one of the models?

Compare every model name and closing bracket with the reference code. A misplaced bracket can keep naive_models from being created.

Help me fix the model dictionary.

Before you run the comparison, which candidate do you expect to lead on this single split?

  • Add the fitting loop directly below naive_models by copying this code:
print("\nOne-split accuracy only")
for model_name, model in naive_models.items():
    model.fit(X_dev, y_dev)
    accuracy = model.score(X_validation, y_validation)
    print(f"{model_name:32s} {accuracy:.3f}")

print(
    "\nLimitation: this ranking uses one split and accuracy alone, so it hides "
    "fold-to-fold variation and the cost of missing malignant cases."
)

What does this comparison prove?

  • The model.fit call trains each candidate on the same development data.
  • The model.score call calculates accuracy on the same validation data.
  • The final message records the two weaknesses exposed by this first leaderboard.
  • Save model_benchmark.py.
  • Run the completed accuracy-only cell from the editor's cell controls.

You will see four accuracy values under One-split accuracy only. The holdout remains untouched because every model uses only the development and validation partitions.

You now have a working baseline. More importantly, its printed limitation shows why one split cannot support the final recommendation.

Leaderboard not printing?

Run the earlier dataset cell before this cell so that the split arrays exist. Check that the fitting loop remains indented beneath the for line.

Help me debug the baseline output.

Diagnose the build data

Data defects and class imbalance can change how a leaderboard should be interpreted. Multicollinearity can reveal groups of predictors carrying similar information.

  • Place your cursor below the printed leaderboard limitation.
  • Add the data-quality checks by copying this code:
# %% [markdown]
# ## Data quality, imbalance, and multicollinearity diagnostics

# %%
missing_values = len(np.argwhere(np.isnan(X_build)))
duplicate_rows = len(X_build) - len(np.unique(X_build, axis=0))
print(f"\nMissing values in build data: {missing_values}")
print(f"Duplicate feature rows in build data: {duplicate_rows}")

build_classes, build_counts = np.unique(y_build, return_counts=True)
for class_value, class_count in zip(build_classes, build_counts):
    print(f"Build data {CLASS_NAMES[class_value]}: {class_count}")

What do these checks measure?

  • The missing-value check counts every numeric entry identified as missing in X_build.
  • The duplicate-row check compares the build-data row count with its unique-row count.
  • The class-count loop shows how many benign and malignant examples remain available for model development.
  • Every check uses build data so the holdout stays outside the diagnostic process.
  • Save model_benchmark.py.
  • Run the diagnostics cell from the editor's cell controls.

You will see zero missing values and zero duplicate feature rows. You will also see separate build-data counts for benign and malignant cases.

Counts not appearing?

Confirm that the dataset cell ran in the current Interactive session. The diagnostics depend on X_build and y_build.

Help me fix the data-quality checks.

Pearson correlation measures the linear relationship between pairs of features. This project flags absolute correlations at or above 0.90 as a screen for redundancy.

  • Add the correlation scan directly below the build-class loop by copying this code:
correlation_matrix = np.corrcoef(X_build, rowvar=False)
high_correlation_pairs = []

for first_index in range(len(feature_names)):
    for second_index in range(first_index + 1, len(feature_names)):
        correlation = abs(correlation_matrix[first_index, second_index])
        if correlation >= CORRELATION_THRESHOLD:
            high_correlation_pairs.append(
                (
                    feature_names[first_index],
                    feature_names[second_index],
                    correlation,
                )
            )

high_correlation_pairs.sort(key=lambda pair: pair[2], reverse=True)
print(
    f"\nFeature pairs with absolute Pearson correlation "
    f">= {CORRELATION_THRESHOLD:.2f}: {len(high_correlation_pairs)}"
)
for first_feature, second_feature, correlation in high_correlation_pairs[:10]:
    print(f"{first_feature:24s} | {second_feature:24s} | {correlation:.3f}")

How does the correlation screen work?

  • The correlation matrix uses only X_build.
  • The nested loops inspect each feature pair once.
  • The threshold keeps only pairs with a strong absolute linear relationship under this project's convention.
  • The final loop prints the ten strongest flagged pairs.

Treat each flagged pair as evidence of redundant predictors. Feature removal needs additional evidence.

  • Save model_benchmark.py.
  • Run the updated diagnostics cell from the editor's cell controls.

You will see a total number of flagged pairs followed by the ten strongest feature pairs. Each printed row includes two feature names plus their absolute correlation.

Correlation pairs missing?

Check that correlation_matrix uses X_build with rowvar=False. Confirm that the comparison uses CORRELATION_THRESHOLD.

Help me debug the correlation scan.

  • Add the heatmap directly below the correlation output by copying this code:
plt.figure(figsize=(8, 6))
image = plt.imshow(
    abs(correlation_matrix),
    cmap="coolwarm",
    vmin=0,
    vmax=1,
    interpolation="nearest",
)
plt.colorbar(image, label="Absolute Pearson correlation")
plt.xticks([])
plt.yticks([])
plt.title("Feature Correlation Screen on Build Data")
plt.tight_layout()
plt.show()

What does the heatmap add?

  • The heatmap turns the absolute correlation matrix into a visual screen.
  • The fixed scale from 0 to 1 makes weak and strong relationships comparable.
  • The color bar explains how each color maps to correlation strength.
  • The title records that the screen uses build data.

Before you run the completed diagnostics cell, where do you expect the strongest feature relationships to appear on the heatmap?

  • Save model_benchmark.py.
  • Run the completed diagnostics cell from the editor's cell controls.

You will see a square heatmap titled Feature Correlation Screen on Build Data. Stronger absolute relationships sit near the high end of the color scale.

Strong work. Your baseline now exposes its scoring limitation while the diagnostics show which data conditions must inform the final comparison.

Heatmap not displaying?

Run the full diagnostics cell so that correlation_matrix exists before the plotting lines execute. Confirm that the Interactive window remains connected to the selected .venv.

Help me restore the correlation heatmap.

✔️ Awesome, I've got everything!

Your saved model_benchmark.py contains the complete baseline and build-data diagnostics.

ⓧ I'd like to double check the full code

# %% [markdown]
# # Reproducible Classification Model Benchmark
# Malignant tumors are label 1, so recall measures the share of malignant
# cases identified by a model. This dataset is educational and this project
# does not create or validate a clinical diagnostic system.

# %%
print("Model benchmark workspace ready")

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, classification_report
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

RANDOM_STATE = 42
HOLDOUT_SIZE = 0.20
CORRELATION_THRESHOLD = 0.90
CLASS_NAMES = ["Benign", "Malignant"]

# %% [markdown]
# ## Load the data and reserve the final holdout

# %%
dataset = load_breast_cancer()
X = dataset.data
y = 1 - dataset.target
feature_names = dataset.feature_names

class_values, class_counts = np.unique(y, return_counts=True)
print(f"Samples: {X.shape[0]}")
print(f"Features: {X.shape[1]}")
for class_value, class_count in zip(class_values, class_counts):
    print(f"{CLASS_NAMES[class_value]}: {class_count}")

X_build, X_holdout, y_build, y_holdout = train_test_split(
    X,
    y,
    test_size=HOLDOUT_SIZE,
    stratify=y,
    random_state=RANDOM_STATE,
)

X_dev, X_validation, y_dev, y_validation = train_test_split(
    X_build,
    y_build,
    test_size=0.25,
    stratify=y_build,
    random_state=RANDOM_STATE,
)

# %% [markdown]
# ## The intentionally incomplete accuracy-only benchmark

# %%
naive_models = {
    "Dummy baseline": DummyClassifier(strategy="prior"),
    "Logistic regression": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", LogisticRegression(max_iter=2000)),
        ]
    ),
    "SVC": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", SVC()),
        ]
    ),
    "Random forest": RandomForestClassifier(random_state=RANDOM_STATE),
}

print("\nOne-split accuracy only")
for model_name, model in naive_models.items():
    model.fit(X_dev, y_dev)
    accuracy = model.score(X_validation, y_validation)
    print(f"{model_name:32s} {accuracy:.3f}")

print(
    "\nLimitation: this ranking uses one split and accuracy alone, so it hides "
    "fold-to-fold variation and the cost of missing malignant cases."
)

# %% [markdown]
# ## Data quality, imbalance, and multicollinearity diagnostics

# %%
missing_values = len(np.argwhere(np.isnan(X_build)))
duplicate_rows = len(X_build) - len(np.unique(X_build, axis=0))
print(f"\nMissing values in build data: {missing_values}")
print(f"Duplicate feature rows in build data: {duplicate_rows}")

build_classes, build_counts = np.unique(y_build, return_counts=True)
for class_value, class_count in zip(build_classes, build_counts):
    print(f"Build data {CLASS_NAMES[class_value]}: {class_count}")

correlation_matrix = np.corrcoef(X_build, rowvar=False)
high_correlation_pairs = []

for first_index in range(len(feature_names)):
    for second_index in range(first_index + 1, len(feature_names)):
        correlation = abs(correlation_matrix[first_index, second_index])
        if correlation >= CORRELATION_THRESHOLD:
            high_correlation_pairs.append(
                (
                    feature_names[first_index],
                    feature_names[second_index],
                    correlation,
                )
            )

high_correlation_pairs.sort(key=lambda pair: pair[2], reverse=True)
print(
    f"\nFeature pairs with absolute Pearson correlation "
    f">= {CORRELATION_THRESHOLD:.2f}: {len(high_correlation_pairs)}"
)
for first_feature, second_feature, correlation in high_correlation_pairs[:10]:
    print(f"{first_feature:24s} | {second_feature:24s} | {correlation:.3f}")

plt.figure(figsize=(8, 6))
image = plt.imshow(
    abs(correlation_matrix),
    cmap="coolwarm",
    vmin=0,
    vmax=1,
    interpolation="nearest",
)
plt.colorbar(image, label="Absolute Pearson correlation")
plt.xticks([])
plt.yticks([])
plt.title("Feature Correlation Screen on Build Data")
plt.tight_layout()
plt.show()

Your benchmark now has an honest starting point plus evidence about the data behind it. Next, you will make the comparison resistant to data leakage and class imbalance.

Cross-Validate and Select a Model

Your diagnostic checks exposed class imbalance and correlated predictors. The early leaderboard still depends on one development split, while your final holdout evaluation remains untouched.

This step resolves that uncertainty with shared stratified cross-validation. You will compare score variation before evaluating the selected model once on the holdout set.

In this step, get ready to:
  • Build five leakage-safe candidate estimators with class weighting.
  • Rank every candidate with shared stratified folds and multiple metrics.
  • Evaluate the selected model once on the untouched holdout set.
Define leakage-safe candidates and shared folds

A Pipeline learns feature scaling from each training fold. This keeps validation-fold statistics out of model fitting and prevents data leakage.

Estimator-native class weighting increases the influence of the less frequent class during fitting. Comparing ordinary logistic regression with its balanced variant makes that trade-off visible.

  • Create a notebook section at the bottom of model_benchmark.py by pasting these cell markers:
# %% [markdown]
# ## Leakage-safe pipelines and class weighting

# %%

What Does This Section Mark?

The markdown cell labels the purpose of the next code cell. The following cell contains every candidate plus the shared evaluation configuration.

  • Define the five candidate estimators below the new code-cell marker by pasting this code:
models = {
    "Dummy baseline": DummyClassifier(strategy="prior"),
    "Logistic regression": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", LogisticRegression(max_iter=2000)),
        ]
    ),
    "Logistic regression balanced": Pipeline(
        [
            ("scale", StandardScaler()),
            (
                "model",
                LogisticRegression(class_weight="balanced", max_iter=2000),
            ),
        ]
    ),
    "SVC balanced": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", SVC(class_weight="balanced")),
        ]
    ),
    "Random forest balanced": RandomForestClassifier(
        class_weight="balanced",
        random_state=RANDOM_STATE,
    ),
}

How Are the Candidates Different?

  • The dummy model provides a baseline that ignores the feature values.
  • Both logistic variants place StandardScaler inside their pipelines, so scaling is fitted separately within each fold.
  • The balanced estimators use class_weight="balanced" to adjust the influence of each class from the training data.
  • The random forest stays unscaled because its tree splits operate directly on feature values.
  • Save model_benchmark.py.
  • Run the current code cell using the cell control from earlier.

You will see the cell execution count advance without a traceback. The five estimators are now available in models.

Seeing a Traceback in the Model Cell?

Check that every opening parenthesis has a matching closing parenthesis. Confirm that the final model entry ends before the dictionary closes.

Help me fix the candidate-model cell.

  • Add the shared scoring dictionary and fold configuration directly below models by pasting this code:
scoring = {
    "balanced_accuracy": "balanced_accuracy",
    "malignant_recall": "recall",
    "roc_auc": "roc_auc",
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)
print("\nModels ready:")
for model_name in models:
    print(f"- {model_name}")
print(cv)

What Does This Configuration Measure?

  • Balanced accuracy gives equal importance to performance on benign and malignant cases.
  • The standard recall scorer measures malignant-class recall because malignant is label 1.
  • ROC AUC measures how well each model separates the two classes across possible score thresholds.
  • StratifiedKFold creates five shuffled folds with similar class proportions and a repeatable seed.
  • Save model_benchmark.py.
  • Run the updated model-definition cell using the cell control.

You will see five model names followed by the shared StratifiedKFold configuration. Your candidates are now ready for the same evaluation process.

Missing a Model Name?

Compare the keys in models with the five expected labels. Check the indentation of the loop that prints each name.

Help me find the missing candidate.

Cross-validate and compare the candidates

Cross-validation fits every candidate on the same folds. Recording each metric's mean plus its standard deviation shows both typical performance and model-selection uncertainty.

  • Create the cross-validation section below the model-definition cell by pasting these cell markers:
# %% [markdown]
# ## Cross-validated comparison

# %%

Why Use a Separate Cell?

This boundary keeps the reusable model definitions separate from the benchmark run. You can rerun the comparison after changing a candidate without rerunning the earlier diagnostics.

  • Collect the fold-level results into benchmark_rows by pasting this code below the marker:
benchmark_rows = []

for model_name, model in models.items():
    scores = cross_validate(
        model,
        X_build,
        y_build,
        scoring=scoring,
        cv=cv,
        return_train_score=True,
    )
    benchmark_rows.append(
        {
            "model": model_name,
            "balanced_accuracy_mean": float(
                np.mean(scores["test_balanced_accuracy"])
            ),
            "balanced_accuracy_std": float(
                np.std(scores["test_balanced_accuracy"])
            ),
            "malignant_recall_mean": float(
                np.mean(scores["test_malignant_recall"])
            ),
            "roc_auc_mean": float(np.mean(scores["test_roc_auc"])),
            "fit_time_mean": float(np.mean(scores["fit_time"])),
        }
    )

How Are the Fold Results Summarized?

  • cross_validate fits each estimator across the same five folds.
  • np.mean summarizes the typical test score for each metric.
  • np.std captures fold-to-fold variation in balanced accuracy.
  • fit_time records the average training time for practical comparison.
  • Save model_benchmark.py.
  • Run the current cross-validation cell using the cell control.

The execution count will advance after every candidate completes without a traceback. The resulting list now holds one summary row per model.

Does Cross-Validation Stop Early?

Check the indentation inside the model loop. Confirm that each scorer key matches the names defined in scoring.

Help me debug the cross-validation loop.

Before you rank the results, do you expect the one-split accuracy winner to remain first when every model is judged across five folds?

  • Rank the summaries by mean balanced accuracy and print the scorecard by pasting this code below the loop:
ranked_rows = sorted(
    benchmark_rows,
    key=lambda row: row["balanced_accuracy_mean"],
    reverse=True,
)

print("\nCross-validated benchmark ranked by balanced accuracy")
print(
    f"{'Model':32s} {'Bal Acc':>8s} {'Std':>8s} "
    f"{'Recall':>8s} {'ROC AUC':>8s} {'Fit s':>8s}"
)
for row in ranked_rows:
    print(
        f"{row['model']:32s} "
        f"{row['balanced_accuracy_mean']:8.3f} "
        f"{row['balanced_accuracy_std']:8.3f} "
        f"{row['malignant_recall_mean']:8.3f} "
        f"{row['roc_auc_mean']:8.3f} "
        f"{row['fit_time_mean']:8.4f}"
    )

How Should You Read the Scorecard?

The first row has the highest mean balanced accuracy across the shared folds. Recall and ROC AUC reveal trade-offs that the ranking metric alone cannot show.

A smaller standard deviation means performance changed less between folds. Fit time adds a practical cost to the comparison.

  • Save model_benchmark.py.
  • Run the updated cross-validation cell using the cell control.

You will see five rows ranked by balanced accuracy. Each row also reports score variation, malignant recall, ROC AUC, and mean fit time.

Seeing an Empty Scorecard?

Confirm that the earlier loop ran and populated benchmark_rows. Run the full cross-validation cell from its first line if the list is empty.

Help me restore the ranked scorecard.

  • Visualize balanced accuracy and its standard deviation by pasting this chart code below the printed scorecard:
chart_rows = list(reversed(ranked_rows))
plt.figure(figsize=(9, 5))
plt.barh(
    [row["model"] for row in chart_rows],
    [row["balanced_accuracy_mean"] for row in chart_rows],
    xerr=[row["balanced_accuracy_std"] for row in chart_rows],
    color="steelblue",
    capsize=4,
)
plt.xlabel("Mean cross-validated balanced accuracy")
plt.title("Classifier Benchmark with One Standard Deviation")
plt.tight_layout()
plt.show()

What Do the Error Bars Show?

Each bar shows mean cross-validated balanced accuracy. Its error bar spans one standard deviation, which makes fold variation visible beside performance.

Compare the two logistic regression bars closely. Their recall and balanced-accuracy differences show the effect of weighting malignant cases more heavily during fitting.

  • Save model_benchmark.py.
  • Run the updated cross-validation cell using the cell control.

You will see a horizontal score chart with one bar per candidate. Each bar includes an error bar for one standard deviation.

Chart Missing a Candidate?

Compare the number of printed scorecard rows with the number of chart bars. Rerun the cell from benchmark_rows = [] if the chart uses an older result list.

Help me fix the benchmark chart.

That is the single-split limitation resolved. Your ranking now reflects repeated, shared folds plus visible score variation.

Evaluate the selected model on the holdout

Model selection is complete before the holdout is touched. The first ranked pipeline now earns one final evaluation on data that played no role in fitting or ranking.

Before you run the final cell, do you expect the selected model's malignant recall to match its cross-validation mean exactly?

  • Create the final evaluation cell and print its classification report by pasting this code below the chart:
# %% [markdown]
# ## Final evaluation on the untouched holdout

# %%
best_model_name = ranked_rows[0]["model"]
best_model = models[best_model_name]
best_model.fit(X_build, y_build)
holdout_predictions = best_model.predict(X_holdout)

print(f"\nSelected model: {best_model_name}")
print(
    classification_report(
        y_holdout,
        holdout_predictions,
        target_names=CLASS_NAMES,
        digits=3,
    )
)

What Does the Final Fit Do?

  • ranked_rows[0] selects the candidate with the strongest mean balanced accuracy.
  • best_model.fit trains that complete pipeline on all build data.
  • best_model.predict produces the only predictions made against the untouched holdout.
  • classification_report displays precision, recall, and F1-score for benign and malignant cases.
  • Save model_benchmark.py.
  • Run the final evaluation cell using the cell control.

You will see the selected model name followed by a classification report. Read the malignant row to find the final holdout recall.

Report Missing Class Names?

Confirm that target_names=CLASS_NAMES remains inside classification_report. Check that holdout_predictions comes from best_model.

Help me repair the holdout report.

  • Display the holdout confusion matrix and recommendation prompt by pasting this code at the end of the final cell:
ConfusionMatrixDisplay.from_predictions(
    y_holdout,
    holdout_predictions,
    display_labels=CLASS_NAMES,
    cmap="Blues",
    values_format="d",
)
plt.title(f"Untouched Holdout: {best_model_name}")
plt.tight_layout()
plt.show()

print(
    "Recommendation prompt: defend the selected model using balanced accuracy, "
    "malignant recall, ROC AUC, fold variation, fit time, and the correlation screen."
)

How Does the Matrix Support Your Decision?

The matrix separates correct predictions from the two error types. The malignant row shows how many malignant cases the selected model identified or missed.

A defensible recommendation combines this holdout result with cross-validated balanced accuracy, malignant recall, ROC AUC, fold variation, fit time, and the earlier multicollinearity screen.

  • Save model_benchmark.py.
  • Run the completed final evaluation cell using the cell control.

You will see a blue holdout confusion matrix labelled with benign and malignant classes. The Interactive window will also print the recommendation prompt beneath the final report.

Confusion Matrix Not Displaying?

Confirm that plt.show() remains after the display configuration. Run the entire final cell so holdout_predictions exists before the chart is created.

Help me display the holdout matrix.

✔️ Awesome, I've got everything!

Great. Save model_benchmark.py so your reproducible benchmark is ready to rerun.

ⓧ I'd like to double check the full code

# %% [markdown]
# # Reproducible Classification Model Benchmark
# Malignant tumors are label 1, so recall measures the share of malignant
# cases identified by a model. This dataset is educational and this project
# does not create or validate a clinical diagnostic system.

# %%
print("Model benchmark workspace ready")

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, classification_report
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

RANDOM_STATE = 42
HOLDOUT_SIZE = 0.20
CORRELATION_THRESHOLD = 0.90
CLASS_NAMES = ["Benign", "Malignant"]

# %% [markdown]
# ## Load the data and reserve the final holdout

# %%
dataset = load_breast_cancer()
X = dataset.data
y = 1 - dataset.target
feature_names = dataset.feature_names

class_values, class_counts = np.unique(y, return_counts=True)
print(f"Samples: {X.shape[0]}")
print(f"Features: {X.shape[1]}")
for class_value, class_count in zip(class_values, class_counts):
    print(f"{CLASS_NAMES[class_value]}: {class_count}")

X_build, X_holdout, y_build, y_holdout = train_test_split(
    X,
    y,
    test_size=HOLDOUT_SIZE,
    stratify=y,
    random_state=RANDOM_STATE,
)

X_dev, X_validation, y_dev, y_validation = train_test_split(
    X_build,
    y_build,
    test_size=0.25,
    stratify=y_build,
    random_state=RANDOM_STATE,
)

# %% [markdown]
# ## The intentionally incomplete accuracy-only benchmark

# %%
naive_models = {
    "Dummy baseline": DummyClassifier(strategy="prior"),
    "Logistic regression": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", LogisticRegression(max_iter=2000)),
        ]
    ),
    "SVC": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", SVC()),
        ]
    ),
    "Random forest": RandomForestClassifier(random_state=RANDOM_STATE),
}

print("\nOne-split accuracy only")
for model_name, model in naive_models.items():
    model.fit(X_dev, y_dev)
    accuracy = model.score(X_validation, y_validation)
    print(f"{model_name:32s} {accuracy:.3f}")

print(
    "\nLimitation: this ranking uses one split and accuracy alone, so it hides "
    "fold-to-fold variation and the cost of missing malignant cases."
)

# %% [markdown]
# ## Data quality, imbalance, and multicollinearity diagnostics

# %%
missing_values = len(np.argwhere(np.isnan(X_build)))
duplicate_rows = len(X_build) - len(np.unique(X_build, axis=0))
print(f"\nMissing values in build data: {missing_values}")
print(f"Duplicate feature rows in build data: {duplicate_rows}")

build_classes, build_counts = np.unique(y_build, return_counts=True)
for class_value, class_count in zip(build_classes, build_counts):
    print(f"Build data {CLASS_NAMES[class_value]}: {class_count}")

correlation_matrix = np.corrcoef(X_build, rowvar=False)
high_correlation_pairs = []

for first_index in range(len(feature_names)):
    for second_index in range(first_index + 1, len(feature_names)):
        correlation = abs(correlation_matrix[first_index, second_index])
        if correlation >= CORRELATION_THRESHOLD:
            high_correlation_pairs.append(
                (
                    feature_names[first_index],
                    feature_names[second_index],
                    correlation,
                )
            )

high_correlation_pairs.sort(key=lambda pair: pair[2], reverse=True)
print(
    f"\nFeature pairs with absolute Pearson correlation "
    f">= {CORRELATION_THRESHOLD:.2f}: {len(high_correlation_pairs)}"
)
for first_feature, second_feature, correlation in high_correlation_pairs[:10]:
    print(f"{first_feature:24s} | {second_feature:24s} | {correlation:.3f}")

plt.figure(figsize=(8, 6))
image = plt.imshow(
    abs(correlation_matrix),
    cmap="coolwarm",
    vmin=0,
    vmax=1,
    interpolation="nearest",
)
plt.colorbar(image, label="Absolute Pearson correlation")
plt.xticks([])
plt.yticks([])
plt.title("Feature Correlation Screen on Build Data")
plt.tight_layout()
plt.show()

# %% [markdown]
# ## Leakage-safe pipelines and class weighting

# %%
models = {
    "Dummy baseline": DummyClassifier(strategy="prior"),
    "Logistic regression": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", LogisticRegression(max_iter=2000)),
        ]
    ),
    "Logistic regression balanced": Pipeline(
        [
            ("scale", StandardScaler()),
            (
                "model",
                LogisticRegression(class_weight="balanced", max_iter=2000),
            ),
        ]
    ),
    "SVC balanced": Pipeline(
        [
            ("scale", StandardScaler()),
            ("model", SVC(class_weight="balanced")),
        ]
    ),
    "Random forest balanced": RandomForestClassifier(
        class_weight="balanced",
        random_state=RANDOM_STATE,
    ),
}

scoring = {
    "balanced_accuracy": "balanced_accuracy",
    "malignant_recall": "recall",
    "roc_auc": "roc_auc",
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)
print("\nModels ready:")
for model_name in models:
    print(f"- {model_name}")
print(cv)

# %% [markdown]
# ## Cross-validated comparison

# %%
benchmark_rows = []

for model_name, model in models.items():
    scores = cross_validate(
        model,
        X_build,
        y_build,
        scoring=scoring,
        cv=cv,
        return_train_score=True,
    )
    benchmark_rows.append(
        {
            "model": model_name,
            "balanced_accuracy_mean": float(
                np.mean(scores["test_balanced_accuracy"])
            ),
            "balanced_accuracy_std": float(
                np.std(scores["test_balanced_accuracy"])
            ),
            "malignant_recall_mean": float(
                np.mean(scores["test_malignant_recall"])
            ),
            "roc_auc_mean": float(np.mean(scores["test_roc_auc"])),
            "fit_time_mean": float(np.mean(scores["fit_time"])),
        }
    )

ranked_rows = sorted(
    benchmark_rows,
    key=lambda row: row["balanced_accuracy_mean"],
    reverse=True,
)

print("\nCross-validated benchmark ranked by balanced accuracy")
print(
    f"{'Model':32s} {'Bal Acc':>8s} {'Std':>8s} "
    f"{'Recall':>8s} {'ROC AUC':>8s} {'Fit s':>8s}"
)
for row in ranked_rows:
    print(
        f"{row['model']:32s} "
        f"{row['balanced_accuracy_mean']:8.3f} "
        f"{row['balanced_accuracy_std']:8.3f} "
        f"{row['malignant_recall_mean']:8.3f} "
        f"{row['roc_auc_mean']:8.3f} "
        f"{row['fit_time_mean']:8.4f}"
    )

chart_rows = list(reversed(ranked_rows))
plt.figure(figsize=(9, 5))
plt.barh(
    [row["model"] for row in chart_rows],
    [row["balanced_accuracy_mean"] for row in chart_rows],
    xerr=[row["balanced_accuracy_std"] for row in chart_rows],
    color="steelblue",
    capsize=4,
)
plt.xlabel("Mean cross-validated balanced accuracy")
plt.title("Classifier Benchmark with One Standard Deviation")
plt.tight_layout()
plt.show()

# %% [markdown]
# ## Final evaluation on the untouched holdout

# %%
best_model_name = ranked_rows[0]["model"]
best_model = models[best_model_name]
best_model.fit(X_build, y_build)
holdout_predictions = best_model.predict(X_holdout)

print(f"\nSelected model: {best_model_name}")
print(
    classification_report(
        y_holdout,
        holdout_predictions,
        target_names=CLASS_NAMES,
        digits=3,
    )
)

ConfusionMatrixDisplay.from_predictions(
    y_holdout,
    holdout_predictions,
    display_labels=CLASS_NAMES,
    cmap="Blues",
    values_format="d",
)
plt.title(f"Untouched Holdout: {best_model_name}")
plt.tight_layout()
plt.show()

print(
    "Recommendation prompt: defend the selected model using balanced accuracy, "
    "malignant recall, ROC AUC, fold variation, fit time, and the correlation screen."
)

Secret mission

Tune the Decision Threshold

Tune a class-weighted logistic classifier with internal cross-validation. Compare its tuned malignant-class recall with the untuned result while keeping the holdout outside the threshold decision.

Clean Up Your Resources

Clean Up Your Resources

Your benchmark runs locally in Visual Studio Code with no ongoing service costs. Choose whether to keep the workspace active, pause its Python session, or remove the local project entirely.

Resources you used:

  • The local ml-model-benchmark folder.
  • The Python Interactive session connected to the project .venv.

Keep everything running

No action is required. Choose this option while you are actively refining the benchmark or comparing model results.

  • Keep the ml-model-benchmark folder in its current location.
  • Leave the Python Interactive window connected while you continue running cells.
  • Rerun cells in model_benchmark.py or threshold_mission.py whenever you want to revisit the model evidence.

Pause - I'll come back to this later

Shut down the active Python session to free memory. Your scripts and .venv remain ready for your return.

  • Close the Python Interactive window in Visual Studio Code.
  • Keep the ml-model-benchmark folder in its current location.
  • Reopen the ml-model-benchmark folder in Visual Studio Code when you return.
  • Reconnect the project .venv before running either Python file.

Delete - I don't want to use this again

Remove the local project if you no longer need the benchmark. This clears the scripts and isolated environment from your computer.

  • Close the Python Interactive window in Visual Studio Code.
  • Close Visual Studio Code.
  • Press the Windows key to open the search field.
  • Type File Explorer into the search field.
  • Press Enter.
  • Navigate to the location where you created the ml-model-benchmark folder.

Check Before Permanent Removal

Permanent removal can feel drastic. It only affects the ml-model-benchmark folder you select.

A separate copy preserves any result you may need later.

  • Move the ml-model-benchmark folder to the Recycle Bin.
  • Permanently delete only the ml-model-benchmark folder from the Recycle Bin.

You should no longer see ml-model-benchmark in its original location or in the Recycle Bin.

Nice Work!

Nice Work!

You did it! Your reproducible classification benchmark now compares five candidates fairly before evaluating one selected model against untouched holdout data.

You've learned how to:

  • Build an accuracy-only benchmark with a dummy baseline, logistic regression, a support vector classifier, and a random forest. Identify why one development split cannot support a defensible ranking.
  • Run data diagnostics for missing values plus duplicate rows. Measure class imbalance and multicollinearity on build data. Use a Pearson correlation screen to surface redundant predictors without treating the threshold as automatic proof for feature removal.
  • Prevent data leakage with scaling pipelines fitted inside each validation fold. Rank candidates through stratified cross-validation using balanced accuracy, malignant-class recall, ROC AUC, score variation, and fit time. Evaluate the selected model once before interpreting its confusion matrix.
  • Secret Mission: Tune a class-weighted logistic regression decision threshold with internal cross-validation on the build data. Compare the tuned and untuned malignant-recall trade-off without using holdout results to choose the cutoff.

Ready to quiz yourself?