Diagnose a Misleading Classifier

Diagnose an imbalanced classifier with baselines, metrics, and a safe pipeline.

Introduction

30 Second Summary

An impressive score can make a weak result sound convincing in an interview. The real test is whether you can uncover what that number leaves out.

In this project, you will build a reproducible binary classification lab that exposes misleading accuracy caused by class imbalance. You will use a majority baseline plus a leakage-resistant logistic regression pipeline to produce an interview-ready model review.

What You'll Build

When you run the notebook, its output turns an impressive baseline score into an evidence-backed comparison you can explain aloud in 60 seconds.

By the end of this project, you'll have:

  • A reproducible churn experiment that generates the same imbalanced dataset each run. Printed class counts make the imbalance visible before scoring.
  • A majority-class diagnosis that pairs accuracy with precision, recall, a confusion matrix, and a classification report. You can point to the missed churn cases instead of trusting one score.
  • A trained-model comparison that shows the trade-off between false alarms and missed churn cases. An interview-ready explanation defends the held-out split plus the pipeline's protection against data leakage.
  • Secret Mission: Compare precision and recall at decision thresholds 0.30, 0.50, and 0.70 when missed churn cases are costly.

Are there any prerequisites?

You need a Google account to use Google Colab. Basic Python syntax is enough for the coding.

Before We Start

Before any code enters the notebook, this project diagnoses a churn-style binary classifier on imbalanced data. The interview goal is to explain why its mistakes matter and evaluate its performance beyond accuracy.

Open and Verify the Interview Lab

Your classifier experiment needs a consistent place to run. Local Python setups can vary across Windows machines.

Google Colab keeps your practice session in the browser. Pinning scikit-learn to 1.9.1 keeps every later experiment on the expected API.

In this step, get ready to:
  • Create a Google Colab notebook named aiml_interview_lab.
  • Install scikit-learn 1.9.1 in the notebook environment.
  • Confirm that the notebook reports scikit-learn 1.9.1.
Create the notebook

A Colab notebook holds the cells you run during the interview lab. Your Google account keeps the notebook available in Google Drive.

The sign-in flow stays on Google's account page. The notebook never needs your Google password.

  • Open a new Google Colab notebook.
  • Complete the Google account sign-in flow if Colab asks.
  • Select the notebook title in the top-left corner.
  • Replace the current title with aiml_interview_lab.
  • Press Enter to save the new title.

You should see aiml_interview_lab at the top of the open notebook. This name identifies the lab you will build throughout the project.

Pin the ML library

A version pin gives every learner the same library release. This keeps your interview practice focused on model evaluation instead of environment differences.

  • Install the pinned scikit-learn release by running this code in the first notebook cell:
%pip install scikit-learn==1.9.1

What Does This Code Do?

The %pip command installs a package into the current notebook kernel. The exact pin selects scikit-learn 1.9.1 for the rest of the lab.

The cell should finish with package-installation output and no reported installation failure.

The Installation Did Not Finish?

  • Confirm that your browser still has internet access.
  • Run the installation cell again.

Ask for targeted help with diagnosing the Colab installation.

Verify the environment

The installed version provides direct evidence that your notebook matches the project environment. This check catches an incomplete installation before any classifier code depends on it.

  • Add a second code cell beneath the installation cell.

Before you run the check, which version do you expect the notebook to report?

  • Verify the installed scikit-learn version by running this code in the second cell:
import sklearn
print(sklearn.__version__)

What Does This Code Do?

  • The import loads scikit-learn into the notebook session.
  • The sklearn.__version__ value identifies the loaded release.

The final output line should read 1.9.1. That result proves the pinned library is active in your notebook.

✔️ I see the pinned version

That confirms your browser-based ML environment is ready. The notebook now uses scikit-learn 1.9.1 for the classifier experiment.

ⓧ I see an older version

An older result means the pinned release is not active in the current notebook session.

  • Return to the installation cell.
  • Run the installation cell again.
  • Return to the version-check cell.
  • Run the version-check cell again.

Still Seeing the Older Version?

Ask for help with activating the pinned scikit-learn release.

ⓧ The import fails

A failed import means the notebook cannot load scikit-learn from its current environment.

  • Return to the installation cell.
  • Check that the installation cell finished without a reported failure.
  • Run the installation cell again if it failed.
  • Run the version-check cell again.

The Import Still Fails?

Ask for help with diagnosing the failed scikit-learn import.

Your interview lab now runs with the expected scikit-learn release. Next, you will create an imbalanced classification problem and watch accuracy set its first trap.

Create the Accuracy Trap

Your Google Colab notebook now runs scikit-learn 1.9.1. You can use that stable runtime to create a reproducible model review.

Accuracy can look impressive when one class dominates. This step creates that misleading result so you can experience the trap directly.

In this step, get ready to:
  • Generate a reproducible binary classification problem with a 90/10 class imbalance.
  • Preserve the class balance across training data and test data.
  • Measure the majority baseline's accuracy.
Generate an imbalanced problem

Class imbalance means one label appears much more often than another. The make_classification function lets you create that imbalance deliberately.

  • Add a code cell beneath the version check cell.
  • Create interview_lab.py by pasting this code into the new cell:
%%writefile interview_lab.py
import sklearn
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split


def print_distribution(name, labels):
    churns = sum(int(label) for label in labels)
    stays = len(labels) - churns
    print(f"{name}: stays={stays}, churns={churns}")


print(f"scikit-learn version: {sklearn.__version__}")

X, y = make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=5,
    n_redundant=2,
    weights=[0.90, 0.10],
    flip_y=0.02,
    random_state=42,
)

What Does This Code Do?

  • The imports provide the dataset generator, data-splitting tool, baseline classifier, and accuracy metric.
  • The print_distribution helper counts each class. It labels class 0 as stays. It labels class 1 as churns.
  • The weights=[0.90, 0.10] setting assigns about 90 percent of samples to the majority class before label noise.
  • The random_state=42 setting makes the generated data reproducible across runs.
  • Run the file-writing cell using the play icon on its left.

The cell output should confirm that interview_lab.py was written to the Colab runtime.

  • Add another code cell beneath the file-writing cell.
  • Execute the first version of the script by pasting this command into the new cell:
%run interview_lab.py

What Should I See?

The script should print the scikit-learn version as 1.9.1. This confirms the generated file runs inside the verified notebook environment.

Script Not Running?

Run the file-writing cell before the script-run cell. Confirm that the filename is exactly interview_lab.py.

Ask for targeted help with running the first version of the interview lab.

Preserve class balance during splitting

Stratification uses the labels to preserve similar class proportions across the training and test subsets. This keeps the evaluation set representative of the generated imbalance.

  • Add a code cell beneath the script-run cell.
  • Append the split logic and class-count output by pasting this code into the new cell:
%%writefile -a interview_lab.py

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

print_distribution("Full data", y)
print_distribution("Training data", y_train)
print_distribution("Test data", y_test)

How Does the Split Work?

  • The train_test_split function creates separate training and test subsets.
  • The test_size=0.20 setting reserves 20 percent of the samples for testing.
  • The stratify=y setting preserves the label proportions in both subsets.
  • The three print_distribution calls make the imbalance visible at each stage.
  • Run the append cell using the play icon on its left.

The cell output should confirm that new content was appended to interview_lab.py.

  • Return to the script-run cell from the previous substep.
  • Run the script-run cell again.

You should see Full data, Training data, and Test data distributions. Each line should show a much larger stays count than churns count.

Class Distributions Missing?

Confirm that you ran the append cell before rerunning the script. Check that stratify=y appears inside the train_test_split call.

Ask for help with diagnosing the missing class distributions.

Run a majority baseline

A majority baseline provides a simple reference point for judging future models. DummyClassifier ignores the feature values and predicts the most frequent training label.

  • Add a code cell beneath the class-distribution cell.
  • Append the majority baseline by pasting this code into the new cell:
%%writefile -a interview_lab.py

dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
dummy_predictions = dummy.predict(X_test)
print(f"Majority baseline accuracy: {accuracy_score(y_test, dummy_predictions):.3f}")

What Does the Baseline Do?

  • The most_frequent strategy selects the most common training label.
  • The fit call learns which label occurs most often.
  • The predict call applies that single-label strategy to every test sample.
  • The accuracy_score call measures the fraction of matching test labels.
  • Run the append cell using the play icon on its left.

The cell output should confirm that the baseline code was appended to interview_lab.py.

Before you rerun the script, do you expect the majority baseline's accuracy to look weak or strong?

  • Return to the script-run cell from earlier.
  • Run the script-run cell again.

What Should I See?

You should see a much larger stays count than churns. The final line should report a majority-baseline accuracy close to the dominant stays share.

You've built the intended accuracy trap. The baseline ignored every feature while still producing a reassuring score.

Baseline Accuracy Missing?

Confirm that the baseline append cell ran successfully. Check that dummy_predictions comes from dummy.predict(X_test).

Ask for help with diagnosing the missing majority-baseline result.

✔️ Awesome, I've got everything!

Your runtime now contains the complete accuracy-trap version of interview_lab.py.

ⓧ I'd like to double check the full code

import sklearn
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split


def print_distribution(name, labels):
    churns = sum(int(label) for label in labels)
    stays = len(labels) - churns
    print(f"{name}: stays={stays}, churns={churns}")


print(f"scikit-learn version: {sklearn.__version__}")

X, y = make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=5,
    n_redundant=2,
    weights=[0.90, 0.10],
    flip_y=0.02,
    random_state=42,
)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

print_distribution("Full data", y)
print_distribution("Training data", y_train)
print_distribution("Test data", y_test)

dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
dummy_predictions = dummy.predict(X_test)
print(f"Majority baseline accuracy: {accuracy_score(y_test, dummy_predictions):.3f}")

How to Use This Reference

Compare this reference with the file-writing cells in your notebook. The combined file should match every line shown here.

Your misleading baseline is now visible. Next, you'll inspect its mistakes and uncover what the accuracy score hid.

Expose Hidden Baseline Errors

Your Google Colab lab now shows a majority baseline with strong accuracy. That result looks reassuring because most customers stay.

Class imbalance can hide a model that misses every churn case. Richer diagnostics expose that failure before you build a better classifier.

In this step, get ready to:
  • Import metrics that reveal positive-class performance.
  • Build one evaluator for consistent model comparisons.
  • Connect the baseline's false negatives to its misleading accuracy.
Expand the metric imports

Positive-class precision measures how often a churn prediction is correct. Positive-class recall measures how many real churn cases the model finds.

A confusion matrix counts each prediction outcome. A classification report gathers the per-class metrics into one view.

  • Select the folder icon in Colab's left sidebar.
  • Double-click interview_lab.py to return to its editor.
  • Locate from sklearn.metrics import accuracy_score near the top of the file.
  • Replace that import with the expanded metric group below:
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    precision_score,
    recall_score,
)

What Do These Imports Add?

  • accuracy_score keeps the familiar overall score available.
  • precision_score measures the correctness of positive predictions.
  • recall_score measures how many positive cases the model finds.
  • confusion_matrix and classification_report expose the errors hidden by one aggregate score.
  • Save interview_lab.py using the editor's save control.
  • Switch back to the aiml_interview_lab notebook.
  • Confirm the expanded imports by running the script in a code cell:
%run interview_lab.py

What Does This Command Do?

The %run magic executes the saved Python script inside the current notebook runtime. A clean run confirms that every new metric import is available.

You should see the class distributions followed by the majority-baseline accuracy. The script should finish without an import error.

Seeing an Import Error?

Check that the metric names sit inside the same pair of parentheses. Confirm that each name matches the code block exactly.

Ask for help with diagnosing the metric imports.

Build the reusable evaluator

A reusable evaluator applies the same measurements to every model. Consistent evaluation makes later comparisons defensible.

  • Switch back to the interview_lab.py editor.
  • Locate the end of print_distribution().
  • Insert the evaluator below print_distribution() by pasting this code:
def evaluate_model(name, y_true, y_pred):
    metrics = {
        "accuracy": accuracy_score(y_true, y_pred),
        "precision": precision_score(y_true, y_pred, zero_division=0),
        "recall": recall_score(y_true, y_pred, zero_division=0),
    }
    print(f"\n{name}")
    print(f"Accuracy:  {metrics['accuracy']:.3f}")
    print(f"Precision: {metrics['precision']:.3f}")
    print(f"Recall:    {metrics['recall']:.3f}")
    print("Confusion matrix:")
    print(confusion_matrix(y_true, y_pred))
    print("Classification report:")
    print(classification_report(y_true, y_pred, labels=[0, 1], target_names=["stays", "churns"], zero_division=0))
    return metrics

What Does This Function Do?

  • The metrics dictionary stores accuracy plus positive-class precision and recall.
  • zero_division=0 gives undefined positive-class scores a numeric value of zero.
  • confusion_matrix() prints the four prediction counts.
  • classification_report() prints precision plus recall plus F1 score plus support for stays and churns.
  • return metrics preserves the scores for a later side-by-side comparison.
  • Save interview_lab.py using the editor's save control.
  • Switch back to the aiml_interview_lab notebook.
  • Check the new function for syntax errors by running the script again:
%run interview_lab.py

What Does This Check Prove?

The command loads the new function before continuing through the experiment. A complete output confirms that the evaluator has valid Python syntax.

You should see the existing distributions and baseline accuracy again. The script should reach the final accuracy line without a syntax error.

Seeing a Syntax Error?

Check the indentation inside evaluate_model(). Confirm that the closing braces and parentheses match the code block.

Ask for help with debugging the evaluator.

Diagnose the majority baseline

The evaluator becomes useful when the baseline predictions pass through it. The resulting error counts reveal what the high accuracy omitted.

  • Switch back to the interview_lab.py editor.
  • Locate the final line that begins with print(f"Majority baseline accuracy:.
  • Replace that final line with this evaluator call:
dummy_metrics = evaluate_model("Majority baseline", y_test, dummy_predictions)

What Does This Line Change?

The evaluator compares dummy_predictions with the held-out y_test labels. The returned scores remain available through dummy_metrics.

  • Save interview_lab.py using the editor's save control.
  • Switch back to the aiml_interview_lab notebook.

Before you run the script, will the baseline's strong accuracy agree with its positive-class recall?

  • Reveal the baseline's complete diagnostic output by running:
%run interview_lab.py

What Does This Run Reveal?

The command reruns the same reproducible experiment. The final section now reports accuracy plus positive-class precision plus positive-class recall.

It also prints the confusion matrix and the per-class classification report. These outputs show where the baseline fails.

You should see a Majority baseline section with high accuracy. You should also see positive-class precision and recall reported as 0.000.

How Do the Errors Expose the Trap?

  • The first matrix row represents actual stays.
  • The second matrix row represents actual churns.
  • The first matrix column represents predictions of stay.
  • The bottom-left cell counts churn cases incorrectly predicted as stays.
  • The zero churn recall confirms that the baseline misses every positive case.

You found the trap. The impressive accuracy now sits beside direct evidence that the baseline cannot identify churn.

Your completed script should now match the organized reference below.

✔️ Awesome, I've got everything!

  • Keep your saved interview_lab.py ready for the next model comparison.

ⓧ I'd like to double check the full code

import sklearn
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split


def print_distribution(name, labels):
    churns = sum(int(label) for label in labels)
    stays = len(labels) - churns
    print(f"{name}: stays={stays}, churns={churns}")


def evaluate_model(name, y_true, y_pred):
    metrics = {
        "accuracy": accuracy_score(y_true, y_pred),
        "precision": precision_score(y_true, y_pred, zero_division=0),
        "recall": recall_score(y_true, y_pred, zero_division=0),
    }
    print(f"\n{name}")
    print(f"Accuracy:  {metrics['accuracy']:.3f}")
    print(f"Precision: {metrics['precision']:.3f}")
    print(f"Recall:    {metrics['recall']:.3f}")
    print("Confusion matrix:")
    print(confusion_matrix(y_true, y_pred))
    print("Classification report:")
    print(classification_report(y_true, y_pred, labels=[0, 1], target_names=["stays", "churns"], zero_division=0))
    return metrics


print(f"scikit-learn version: {sklearn.__version__}")

X, y = make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=5,
    n_redundant=2,
    weights=[0.90, 0.10],
    flip_y=0.02,
    random_state=42,
)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

print_distribution("Full data", y)
print_distribution("Training data", y_train)
print_distribution("Test data", y_test)

dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
dummy_predictions = dummy.predict(X_test)
dummy_metrics = evaluate_model("Majority baseline", y_test, dummy_predictions)

This reference includes the expanded imports plus the reusable evaluator plus the updated baseline call. Its final output exposes the baseline's minority-class failure.

The baseline is now an honest benchmark instead of a reassuring score. Next, you will train a pipeline that learns from the features while protecting the evaluation workflow from leakage.

Build a Leakage-Resistant Pipeline

Your majority baseline exposed the accuracy trap. Its strong score came from predicting only the dominant class.

A scikit-learn pipeline joins feature scaling with logistic regression. This gives the experiment a model that can learn from the features.

Fitting the pipeline only on the training data reduces data leakage. The held-out test data stays separate until evaluation.

In this step, get ready to:
  • Combine feature scaling with logistic regression in one pipeline.
  • Fit the pipeline only on the training data.
  • Compare the pipeline with the majority baseline on the same test labels.
Organize the imports and evaluator

The script needs three components for its trained model. The organized evaluator keeps the same diagnostic behavior while making the classification report easier to inspect.

  • Select Files in Colab's left sidebar.
  • Double-click interview_lab.py in the file list.
  • Replace the import section at the top of interview_lab.py with this organized import block:
import sklearn
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

Why These Imports?

  • The LogisticRegression estimator learns a relationship between the features and churn labels.
  • The make_pipeline factory combines preprocessing with the estimator.
  • The StandardScaler transformer learns scaling values during pipeline fitting.
  • Save interview_lab.py.
  • Confirm that the new imports load by running the script from a notebook code cell:
%run interview_lab.py

What Does This Check Do?

The run command executes the saved script inside the notebook. A completed baseline evaluation proves that every new import loaded successfully.

You should see the data distributions followed by the majority baseline diagnostics. The script is still runnable after the import update.

  • Return to interview_lab.py in the Colab file editor.
  • Replace the existing evaluate_model function with this organized version:
def evaluate_model(name, y_true, y_pred):
    metrics = {
        "accuracy": accuracy_score(y_true, y_pred),
        "precision": precision_score(y_true, y_pred, zero_division=0),
        "recall": recall_score(y_true, y_pred, zero_division=0),
    }

    print(f"\n{name}")
    print(f"Accuracy:  {metrics['accuracy']:.3f}")
    print(f"Precision: {metrics['precision']:.3f}")
    print(f"Recall:    {metrics['recall']:.3f}")
    print("Confusion matrix:")
    print(confusion_matrix(y_true, y_pred))
    print("Classification report:")
    print(
        classification_report(
            y_true,
            y_pred,
            labels=[0, 1],
            target_names=["stays", "churns"],
            zero_division=0,
        )
    )
    return metrics

What Does the Evaluator Preserve?

  • The metrics dictionary stores accuracy plus positive-class precision and recall.
  • The confusion matrix exposes the counts behind the summary metrics.
  • The classification report shows precision plus recall plus F1 score plus support for both classes.
  • The returned dictionary gives both models the same comparison format.
  • Save interview_lab.py.
  • Verify the organized evaluator by running the script again:
%run interview_lab.py

What Does This Run Confirm?

The run exercises the rewritten evaluator with the existing baseline predictions. You should still see accuracy plus precision plus recall plus the confusion matrix plus the full classification report.

The baseline section should finish with the same minority-class failure you diagnosed earlier. This confirms that the organization changed without losing the evidence.

Script Stopped After the Rewrite?

Check that the closing parenthesis for classification_report sits inside the closing parenthesis for print.

Confirm that the entire old evaluate_model function was replaced. A leftover line can break the indentation.

Ask for targeted help with diagnosing the evaluator rewrite.

Build and fit the pipeline

The scaler and classifier belong in one fitted object. Calling fit on that object lets the scaler learn from X_train before the classifier learns from the transformed training features.

Why Use One Pipeline?

A separate scaling step can accidentally learn from the test data when preprocessing happens before the split. The pipeline keeps scaling and model fitting inside the same training workflow.

The test features pass through the already-fitted scaler when the pipeline predicts. This preserves the held-out role of X_test.

  • Find the dummy_metrics assignment at the bottom of interview_lab.py.
  • Add this trained-model flow directly below that assignment:
model = make_pipeline(
    StandardScaler(),
    LogisticRegression(class_weight="balanced", max_iter=1000),
)
model.fit(X_train, y_train)
model_predictions = model.predict(X_test)
model_metrics = evaluate_model(
    "Logistic regression pipeline",
    y_test,
    model_predictions,
)

What Does This Code Do?

  • The model pipeline scales features before passing them to logistic regression.
  • The balanced class weights give the minority class more influence during model fitting.
  • The model.fit call exposes only X_train and y_train to the fitting process.
  • The model.predict call creates labels for the untouched X_test features.
  • The model_metrics dictionary uses the same evaluator and y_test labels as the baseline.
  • Save interview_lab.py.
Compare the diagnostic output

A fair comparison uses the same held-out labels for both models. Accuracy plus precision plus recall plus the error counts reveal the trade-offs created by the trained pipeline.

Before you run the finished script, decide whether you expect the pipeline to recover more churn cases than the majority baseline.

  • Run the finished experiment from a notebook code cell:
%run interview_lab.py

What Does the Final Run Test?

The script fits both classifiers from the same training split. It evaluates both prediction sets against the same y_test labels.

This keeps the comparison focused on model behavior. Each section reports the same diagnostic evidence.

You should see a Majority baseline section followed by a Logistic regression pipeline section. Both sections should include metrics plus a confusion matrix plus a classification report.

  • Compare the accuracy reported for both models.
  • Compare the positive-class precision reported for both models.
  • Compare the positive-class recall reported for both models.
  • Compare the false negatives in both confusion matrices.

The baseline reports zero positive-class recall because it misses every churn case. The pipeline should recover churn cases while changing precision plus accuracy.

That is the key comparison working. You now have evidence that a model can improve minority-class detection even when one summary score moves in an uncomfortable direction.

Pipeline Section Missing?

Confirm that the model block sits below the dummy_metrics assignment. Check that you saved the file before rerunning it.

Verify that LogisticRegression plus make_pipeline plus StandardScaler appear in the import section.

Ask for help with diagnosing the missing pipeline output.

✔️ Awesome, I've got everything!

Your saved script now evaluates the baseline and trained pipeline against the same held-out test labels.

ⓧ I'd like to double check the full code

import sklearn
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def print_distribution(name, labels):
    churns = sum(int(label) for label in labels)
    stays = len(labels) - churns
    print(f"{name}: stays={stays}, churns={churns}")


def evaluate_model(name, y_true, y_pred):
    metrics = {
        "accuracy": accuracy_score(y_true, y_pred),
        "precision": precision_score(y_true, y_pred, zero_division=0),
        "recall": recall_score(y_true, y_pred, zero_division=0),
    }

    print(f"\n{name}")
    print(f"Accuracy:  {metrics['accuracy']:.3f}")
    print(f"Precision: {metrics['precision']:.3f}")
    print(f"Recall:    {metrics['recall']:.3f}")
    print("Confusion matrix:")
    print(confusion_matrix(y_true, y_pred))
    print("Classification report:")
    print(
        classification_report(
            y_true,
            y_pred,
            labels=[0, 1],
            target_names=["stays", "churns"],
            zero_division=0,
        )
    )
    return metrics


print(f"scikit-learn version: {sklearn.__version__}")

X, y = make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=5,
    n_redundant=2,
    weights=[0.90, 0.10],
    flip_y=0.02,
    random_state=42,
)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

print_distribution("Full data", y)
print_distribution("Training data", y_train)
print_distribution("Test data", y_test)

dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
dummy_predictions = dummy.predict(X_test)
dummy_metrics = evaluate_model("Majority baseline", y_test, dummy_predictions)

model = make_pipeline(
    StandardScaler(),
    LogisticRegression(class_weight="balanced", max_iter=1000),
)
model.fit(X_train, y_train)
model_predictions = model.predict(X_test)
model_metrics = evaluate_model(
    "Logistic regression pipeline",
    y_test,
    model_predictions,
)

How to Use the Full-Code Reference

The reference shows the exact cumulative state of interview_lab.py at this checkpoint. Compare the imports plus evaluator plus final model block if your output differs.

Your pipeline now learns from the features without fitting its scaler on the test data. Next, you will turn these diagnostics into a concise interview explanation.

Defend the Model

Your leakage-resistant pipeline now finds churn cases that the majority baseline misses. Its results are useful evidence for a model review.

Interviewers care about the reasoning behind those results. This step turns the raw scores into a comparison plus a concise explanation grounded in precision, recall, leakage control, and limitations.

In this step, get ready to:
  • Print the baseline metrics beside the pipeline metrics in a fixed-width table.
  • Add interview talking points that explain the evaluation choices.
  • Rehearse a 60-second answer using the numbers from your experiment.
Add a side-by-side comparison

Both classifiers were evaluated against the same held-out labels. A single table makes their trade-offs easier to compare without relying on memory.

  • In Google Colab, use the file browser in the left sidebar to open interview_lab.py.
  • Find the line print(f"scikit-learn version: {sklearn.__version__}").
  • Insert the comparison function directly above that line by pasting this code:
def print_metric_comparison(dummy_metrics, model_metrics):
    print("\nMetric comparison")
    print(f"{'Model':<30}{'Accuracy':>10}{'Precision':>12}{'Recall':>10}")
    print(
        f"{'Majority baseline':<30}"
        f"{dummy_metrics['accuracy']:>10.3f}"
        f"{dummy_metrics['precision']:>12.3f}"
        f"{dummy_metrics['recall']:>10.3f}"
    )
    print(
        f"{'Logistic regression pipeline':<30}"
        f"{model_metrics['accuracy']:>10.3f}"
        f"{model_metrics['precision']:>12.3f}"
        f"{model_metrics['recall']:>10.3f}"
    )

What Does This Function Do?

  • The function accepts the metric dictionaries returned by evaluate_model().
  • The width specifiers reserve consistent space for every column.
  • The decimal specifiers display each score to three decimal places.
  • The shared layout puts the baseline trade-offs beside the trained-model trade-offs.
  • Press Cmd+S on macOS or Ctrl+S on Windows to save interview_lab.py.

Before you test the helper, which model do you expect to have the stronger recall?

  • Load the updated script plus the new comparison helper by running this code in a notebook cell:
%run interview_lab.py
print_metric_comparison(dummy_metrics, model_metrics)

What Does This Check Prove?

The first line executes the saved script. The second line passes both metric dictionaries into the new helper.

A formatted table proves that the helper can compare results from the same held-out test labels.

You should see a Metric comparison table with rows for Majority baseline and Logistic regression pipeline.

Comparison Table Missing?

  • Confirm that print_metric_comparison() sits above the line that prints the scikit-learn version.
  • Check each metric dictionary key against accuracy, precision, and recall.
  • Get targeted help with debugging the metric comparison.

The manual helper call proves that the table formatting works. The finished script needs to print the table automatically after both models have been evaluated.

  • Switch back to interview_lab.py in the file editor from earlier.
  • Find the final evaluation block shown below:
model_metrics = evaluate_model(
    "Logistic regression pipeline",
    y_test,
    model_predictions,
)

Where Does the Comparison Belong?

This block creates model_metrics after the pipeline has produced its predictions. At this point, both metric dictionaries are ready for comparison.

  • Add the comparison call directly below that block so the ending looks like this:
model_metrics = evaluate_model(
    "Logistic regression pipeline",
    y_test,
    model_predictions,
)

print_metric_comparison(dummy_metrics, model_metrics)

Why Call It Here?

The function now receives metrics from both completed evaluations. Its position keeps the comparison after the detailed reports that produced those numbers.

  • Press Cmd+S on macOS or Ctrl+S on Windows to save interview_lab.py.
  • Confirm that the comparison runs automatically by running this command in a notebook cell:
%run interview_lab.py

What Does This Run Confirm?

The command executes the saved file from top to bottom. The final function call now produces the comparison without a separate notebook-cell call.

The output should finish with the fixed-width Metric comparison table. You can now compare accuracy against minority-class performance in one place.

Table Only Appears Manually?

  • Confirm that print_metric_comparison(dummy_metrics, model_metrics) is saved at the bottom of interview_lab.py.
  • Rerun the notebook cell after saving the file.
  • Get targeted help with checking why the saved script skips the comparison.
Add the interview explanation

A metric table gives you evidence. Short talking points connect that evidence to the baseline, metric choice, leakage control, and limits of synthetic data.

  • Switch back to interview_lab.py in the file editor from earlier.
  • Find the line print(f"scikit-learn version: {sklearn.__version__}").
  • Insert the interview-summary function directly above that line by pasting this code:
def print_interview_summary():
    print("\nInterview-ready explanation")
    print("1. I used a majority baseline to prove that accuracy alone was misleading.")
    print("2. Precision asks how often a churn alert is correct.")
    print("3. Recall asks how many actual churn cases the model finds.")
    print("4. I kept scaling and logistic regression in one pipeline to reduce leakage risk.")
    print("5. Synthetic data makes the experiment reproducible, not production-ready.")

What Does This Summary Cover?

  • The baseline point explains why a strong-looking accuracy score needs context.
  • The precision point focuses on the reliability of churn alerts.
  • The recall point focuses on the churn cases the model finds.
  • The pipeline point explains how the workflow reduces preprocessing leakage risk.
  • The synthetic-data point limits the conclusion to a reproducible experiment.
  • Press Cmd+S on macOS or Ctrl+S on Windows to save interview_lab.py.

Before you test the helper, which limitation should stop you from presenting this experiment as production evidence?

  • Load the updated script plus the new summary helper by running this code in a notebook cell:
%run interview_lab.py
print_interview_summary()

What Does This Check Prove?

The saved script runs first. The manual helper call then prints the five interview talking points.

This isolates the new function before you make it part of the automatic script output.

You should see an Interview-ready explanation section ending with the limitation that synthetic data is reproducible without being production-ready.

Summary Missing a Talking Point?

  • Confirm that all five print() lines remain inside print_interview_summary().
  • Check the indentation of every numbered statement.
  • Get targeted help with debugging the interview summary.

The manual call proves that the talking points print correctly. The finished script needs to place them after the metric comparison.

  • Switch back to interview_lab.py in the file editor from earlier.
  • Find the comparison call at the bottom of the file:
print_metric_comparison(dummy_metrics, model_metrics)

What Does This Line Mark?

This call prints the compact evidence table. The interview explanation belongs directly after that evidence.

  • Add the summary call directly below the comparison call so the ending looks like this:
print_metric_comparison(dummy_metrics, model_metrics)
print_interview_summary()

Why Use This Order?

The table presents the measured evidence first. The summary then turns those measurements into an interview narrative.

  • Press Cmd+S on macOS or Ctrl+S on Windows to save interview_lab.py.
Rehearse from the evidence

A defensible 60-second answer starts with the baseline. It uses the printed metric changes before explaining leakage protection plus limitations.

Before the final run, which section do you expect to appear last?

  • Run the finished script from a notebook cell with this command:
%run interview_lab.py

What Does the Final Run Prove?

The command rebuilds the same reproducible experiment from the saved script. Both classifiers use the same held-out test labels.

The final sections connect detailed diagnostics to a compact comparison plus a spoken explanation.

The output should end with Metric comparison followed by Interview-ready explanation.

You have turned the model results into an argument you can defend. Every claim now points back to evidence from your own run.

  • Give a 60-second explanation aloud using the accuracy, precision, and recall values printed by your run.

Structure Your Answer

  • Start with the class imbalance plus the majority baseline.
  • Use your printed precision plus recall values to explain the model trade-off.
  • Explain how fitting the pipeline on training data reduces leakage risk.
  • Close with the limits of synthetic evidence.

✔️ Awesome, I've got everything!

Your saved script now prints the full experiment, comparison table, and interview-ready explanation.

ⓧ I'd like to double check the full code

import sklearn
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def print_distribution(name, labels):
    churns = sum(int(label) for label in labels)
    stays = len(labels) - churns
    print(f"{name}: stays={stays}, churns={churns}")


def evaluate_model(name, y_true, y_pred):
    metrics = {
        "accuracy": accuracy_score(y_true, y_pred),
        "precision": precision_score(y_true, y_pred, zero_division=0),
        "recall": recall_score(y_true, y_pred, zero_division=0),
    }

    print(f"\n{name}")
    print(f"Accuracy:  {metrics['accuracy']:.3f}")
    print(f"Precision: {metrics['precision']:.3f}")
    print(f"Recall:    {metrics['recall']:.3f}")
    print("Confusion matrix:")
    print(confusion_matrix(y_true, y_pred))
    print("Classification report:")
    print(
        classification_report(
            y_true,
            y_pred,
            labels=[0, 1],
            target_names=["stays", "churns"],
            zero_division=0,
        )
    )
    return metrics


def print_metric_comparison(dummy_metrics, model_metrics):
    print("\nMetric comparison")
    print(f"{'Model':<30}{'Accuracy':>10}{'Precision':>12}{'Recall':>10}")
    print(
        f"{'Majority baseline':<30}"
        f"{dummy_metrics['accuracy']:>10.3f}"
        f"{dummy_metrics['precision']:>12.3f}"
        f"{dummy_metrics['recall']:>10.3f}"
    )
    print(
        f"{'Logistic regression pipeline':<30}"
        f"{model_metrics['accuracy']:>10.3f}"
        f"{model_metrics['precision']:>12.3f}"
        f"{model_metrics['recall']:>10.3f}"
    )


def print_interview_summary():
    print("\nInterview-ready explanation")
    print("1. I used a majority baseline to prove that accuracy alone was misleading.")
    print("2. Precision asks how often a churn alert is correct.")
    print("3. Recall asks how many actual churn cases the model finds.")
    print("4. I kept scaling and logistic regression in one pipeline to reduce leakage risk.")
    print("5. Synthetic data makes the experiment reproducible, not production-ready.")


print(f"scikit-learn version: {sklearn.__version__}")

X, y = make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=5,
    n_redundant=2,
    weights=[0.90, 0.10],
    flip_y=0.02,
    random_state=42,
)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

print_distribution("Full data", y)
print_distribution("Training data", y_train)
print_distribution("Test data", y_test)

dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
dummy_predictions = dummy.predict(X_test)
dummy_metrics = evaluate_model("Majority baseline", y_test, dummy_predictions)

model = make_pipeline(
    StandardScaler(),
    LogisticRegression(class_weight="balanced", max_iter=1000),
)
model.fit(X_train, y_train)
model_predictions = model.predict(X_test)
model_metrics = evaluate_model(
    "Logistic regression pipeline",
    y_test,
    model_predictions,
)

print_metric_comparison(dummy_metrics, model_metrics)
print_interview_summary()

Secret mission

Test the Decision Threshold

Create a second Python experiment that tests decision thresholds of 0.30, 0.50, and 0.70. Compare their precision and recall before defending the threshold you would investigate when missed churn is costly.

Clean Up Your Resources

Clean Up Your Resources

Your work is saved in the aiml_interview_lab notebook in Google Drive. This project costs $0 on Google Colab's free-of-charge tier.

Resources you used:

  • The aiml_interview_lab notebook stored in Google Drive. Its saved cells recreate interview_lab.py plus threshold_experiment.py inside the temporary Colab runtime.

Keep everything running

Keeping the notebook gives you a reusable interview rehearsal. The temporary runtime can be rebuilt from the saved cells.

  • Leave the aiml_interview_lab notebook in Google Drive.
  • Rerun the saved notebook cells when you want another practice session.
  • Use the printed metrics to rehearse your model-evaluation explanation.

You'll recover the same reproducible experiment whenever Colab gives you a new runtime.

Pause - I'll come back to this later

Pausing means closing the browser tab while keeping the notebook in Google Drive. Your saved work remains ready for another session.

  • Close the browser tab containing aiml_interview_lab.
  • Leave the saved notebook unchanged in Google Drive.

The temporary runtime may be deleted after the session becomes idle. Your notebook cells remain available to recreate both Python scripts.

Delete - I don't want to use this again

Deleting removes the saved notebook from your Google Drive. The temporary copies of interview_lab.py plus threshold_experiment.py disappear with the Colab runtime.

  • Return to Google Drive in the same Google account.
  • Locate the aiml_interview_lab notebook.
  • Remove the aiml_interview_lab notebook from Google Drive.

The notebook no longer appears in its previous Google Drive location. No persistent project resource remains running.

Nice Work!

Nice Work!

You did it! Your interview lab now exposes misleading classifier performance with evidence you can defend aloud.

You've learned how to:

  • Build a reproducible binary classification experiment that preserves class imbalance across a held-out train and test split.
  • Expose why impressive accuracy can hide a majority baseline's failure. Use precision, recall, and a confusion matrix to identify missed churn cases.
  • Train a leakage-resistant pipeline that keeps scaling with logistic regression. Defend its trade-offs through a side-by-side metric comparison.
  • Secret Mission: Test decision thresholds of 0.30, 0.50, and 0.70. Use the resulting precision-recall trade-off to discuss a lower threshold when missed churn is costly.

Ready to quiz yourself?