Build a Readmission Risk Model

Introduction

30 Second Summary

When hospital teams review thousands of past encounters, a single headline score can hide the patients a screening process misses. A useful recommendation needs evidence that reveals those gaps.

In this project, you will build a 30-day hospital readmission-risk notebook in Google Colab. It turns de-identified encounter data into an evidence-based model recommendation for educational screening.

What You'll Build

Run the finished notebook to watch a misleading accuracy result give way to a patient-safe model recommendation backed by a complete final audit.

By the end of this project, you'll have:

• A data audit that turns 101,766 de-identified encounters from the UCI Machine Learning Repository into a clear chart of the 30-day readmission imbalance.
• A patient-safe model comparison that proves why majority-class accuracy fails before ranking Logistic Regression, Random Forest, plus Extra Trees with validation Average Precision.
• An untouched test report with a tuned threshold, confusion matrix, precision-recall curve, calibration evidence, feature signals, plus subgroup checks that support a defensible screening recommendation.
• Secret Mission: A controlled what-if scorer that changes approved fields in one de-identified encounter before returning its risk score plus review flag.

Are there any prerequisites?

Basic to intermediate Python plus basic machine learning concepts are helpful. You only need a Google account because the notebook runs in Google Colab with free compute.

Before We Start

This first checkpoint anchors the notebook as an educational 30-day readmission screening prototype. Its outputs support human review without replacing clinical judgment.

Set Up the Colab Runtime

A reproducible readmission result starts with a known Python runtime. Package drift can change preprocessing behavior before you compare any models.

Google Colab provides the hosted notebook environment for this project. In this step, you will pin its runtime before verifying every project package.

In this step, get ready to:

• Create Diabetes_Readmission_Risk.ipynb on the Colab runtime 2026.07.
• Pin the six project packages inside the notebook.
• Verify the Python runtime plus every package version after a restart.

Create the pinned Colab notebook

A Colab notebook keeps executable cells beside their results. A descriptive filename makes this project easy to find in Google Drive.

• Visit the Google Colab notebook page.
• Sign in with your Google account if prompted.
• Select File from the top menu.
• Select New notebook.
• Rename the notebook to Diabetes_Readmission_Risk.ipynb by replacing its current title at the top.

Your notebook now has a permanent identity in Google Drive. That name follows it through every model experiment in this project.

Colab runtime versions bundle a specific Python release with a tested environment. Selecting runtime 2026.07 gives this notebook Python 3.12.13.

• Select Runtime from the top menu.
• Select Change runtime type.
• Choose 2026.07 from the Runtime Version dropdown.
• Apply the runtime selection from the dialog.

The notebook is now attached to the required Colab runtime. You have removed the first source of environment drift.

Install the pinned package set

The runtime supplies Python while the first notebook cell supplies exact library releases. Each version pin keeps later metrics tied to the environment used to test this project.

• Paste the following code into the first code cell to create Cell 1:

# %%
# CELL 1: Install a reproducible package set.
# Run this cell once, then choose Runtime > Restart runtime before Cell 2.
%pip install -q numpy==2.5.3 pandas==3.0.5 scipy==1.18.1 scikit-learn==1.9.0 matplotlib==3.11.1 ucimlrepo==0.0.7

What Does This Cell Do?

• The %pip command installs packages into the active notebook runtime.
• Each == pin requests one exact package release.
• The -q flag keeps routine installation output brief.

• Run Cell 1 with the play button beside the cell.

The cell's running indicator should stop without a package installation failure. The quiet installation may still show a short status message.

Package Installation Did Not Finish?

• Check that Cell 1 begins with %pip install -q.
• Retry Cell 1 after confirming that your browser has a stable internet connection.
• Get targeted help with diagnosing the Colab package installation.

Installed core packages need a fresh runtime process before the import check. Restarting preserves the notebook cells while clearing previously loaded package state.

• Select Runtime from the top menu.
• Select Restart runtime.
• Confirm the restart if Colab asks for confirmation.

Colab reconnects to a fresh Python process. That reset gives your pinned packages a clean place to load.

Import and verify the environment

Cell 2 gathers every import used by the later analysis. Building the import section in two short passes makes a missing dependency easier to locate.

• Add a code cell beneath Cell 1 using the code-cell control.
• Paste the following opening section into Cell 2:

# %%
# CELL 2: Import tools and verify the runtime before doing any analysis.
# The assertions stop the notebook early if its environment differs from the tested setup.
import platform
from importlib.metadata import version
from time import perf_counter

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
import sklearn
from ucimlrepo import fetch_ucirepo

from sklearn.calibration import CalibrationDisplay
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import ExtraTreesClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

What Do These Imports Prepare?

• The standard-library imports provide runtime details plus package metadata plus training timers.
• The data imports prepare tables plus numerical arrays plus charts plus dataset retrieval.
• The first model imports prepare preprocessing plus the baseline plus all three candidate estimators.

• Run Cell 2 with the play button beside the cell.

Cell 2 should finish without an import traceback. This confirms that the first import group can load from the restarted runtime.

The First Imports Did Not Load?

• Confirm that you restarted the runtime after Cell 1 finished.
• Rerun Cell 1 if the notebook connected to a different runtime session.
• Get targeted help with diagnosing the failed Colab imports.

The remaining imports give every model one shared evaluation vocabulary. This keeps the later comparison consistent across the baseline plus all three candidates.

• Append the following import group directly below LogisticRegression in Cell 2:

from sklearn.metrics import (
    PrecisionRecallDisplay,
    accuracy_score,
    average_precision_score,
    balanced_accuracy_score,
    brier_score_loss,
    confusion_matrix,
    fbeta_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import StratifiedGroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

What Does This Import Group Add?

• The scikit-learn metrics support model comparison plus threshold tuning plus final plots.
• StratifiedGroupKFold supports patient-grouped partitioning in the next step.
• Pipeline plus the preprocessing imports keep transformations attached to each model.

• Rerun Cell 2 with the play button beside the cell.

The expanded cell should finish without an import traceback. Every dependency needed by the notebook is now available.

A Model Import Is Failing?

• Check the indentation inside the parenthesized metrics import.
• Confirm that the closing parenthesis appears above the model-selection import.
• Get targeted help with checking the Cell 2 import group.

A version table makes the environment visible before any data is loaded. Assertions stop the notebook immediately when Python or a package differs from the tested setup.

• Append the following version check directly below the final preprocessing import in Cell 2:

versions = {
    "Python": platform.python_version(),
    "NumPy": np.__version__,
    "pandas": pd.__version__,
    "SciPy": scipy.__version__,
    "scikit-learn": sklearn.__version__,
    "Matplotlib": matplotlib.__version__,
    "ucimlrepo": version("ucimlrepo"),
}

expected_versions = {
    "NumPy": "2.5.3",
    "pandas": "3.0.5",
    "SciPy": "1.18.1",
    "scikit-learn": "1.9.0",
    "Matplotlib": "3.11.1",
    "ucimlrepo": "0.0.7",
}

assert platform.python_version().startswith("3.12"), versions
for package_name, expected_version in expected_versions.items():
    assert versions[package_name] == expected_version, versions

display(pd.DataFrame.from_dict(versions, orient="index", columns=["version"]))

How Does the Version Check Work?

• versions reads the runtime and package releases currently loaded by Cell 2.
• expected_versions stores the six tested package pins.
• The assertions stop execution when the current environment differs from those expectations.
• The final display turns the verified values into a readable table.

Before you run Cell 2, pause to predict whether every version assertion will pass.

• Run the completed Cell 2 with the play button beside the cell.

What Should You See?

Cell 2 should finish without an assertion error. The displayed table should contain these verified rows:

• Python should show 3.12.13.
• NumPy should show 2.5.3.
• pandas should show 3.0.5.
• SciPy should show 1.18.1.
• scikit-learn should show 1.9.0.
• Matplotlib should show 3.11.1.
• ucimlrepo should show 0.0.7.

A Version Assertion Stopped the Cell?

• Compare the reported version table in the assertion details with the pinned values above.
• Rerun Cell 1 when any package version differs.
• Restart the runtime again before rerunning Cell 2.
• Get targeted help with resolving the version mismatch.

The reference below shows the complete notebook state for this step. Both cell sources preserve the tested order.

✔️ Awesome, I've got everything!

Your notebook now contains both reproducibility cells. The passing version table confirms that the runtime is ready.

ⓧ I'd like to double check the full code

# %%
# CELL 1: Install a reproducible package set.
# Run this cell once, then choose Runtime > Restart runtime before Cell 2.
%pip install -q numpy==2.5.3 pandas==3.0.5 scipy==1.18.1 scikit-learn==1.9.0 matplotlib==3.11.1 ucimlrepo==0.0.7

# %%
# CELL 2: Import tools and verify the runtime before doing any analysis.
# The assertions stop the notebook early if its environment differs from the tested setup.
import platform
from importlib.metadata import version
from time import perf_counter

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
import sklearn
from ucimlrepo import fetch_ucirepo

from sklearn.calibration import CalibrationDisplay
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import ExtraTreesClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    PrecisionRecallDisplay,
    accuracy_score,
    average_precision_score,
    balanced_accuracy_score,
    brier_score_loss,
    confusion_matrix,
    fbeta_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import StratifiedGroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

versions = {
    "Python": platform.python_version(),
    "NumPy": np.__version__,
    "pandas": pd.__version__,
    "SciPy": scipy.__version__,
    "scikit-learn": sklearn.__version__,
    "Matplotlib": matplotlib.__version__,
    "ucimlrepo": version("ucimlrepo"),
}

expected_versions = {
    "NumPy": "2.5.3",
    "pandas": "3.0.5",
    "SciPy": "1.18.1",
    "scikit-learn": "1.9.0",
    "Matplotlib": "3.11.1",
    "ucimlrepo": "0.0.7",
}

assert platform.python_version().startswith("3.12"), versions
for package_name, expected_version in expected_versions.items():
    assert versions[package_name] == expected_version, versions

display(pd.DataFrame.from_dict(versions, orient="index", columns=["version"]))

How Is This Reference Organized?

Each # %% marker begins one Colab cell. The reference preserves the exact cumulative code for Cells 1 and 2.

Your reproducible Colab environment is ready. Next, you will load the hospital encounters and expose why raw accuracy can hide every missed readmission.

Load Data and Reveal the Accuracy Trap

Your verified Google Colab runtime is ready for real hospital encounter data. This step turns that data into a target the models can learn from.

A majority-only predictor can look accurate while missing every 30-day readmission. You will run that shortcut to expose why useful evaluation needs more than accuracy.

In this step, get ready to:

• Load the de-identified hospital encounters from UCI dataset 296.
• Create a binary target for readmissions within 30 days.
• Expose the all-negative predictor with recall and balanced accuracy.

Fetch the UCI encounter data

The UCI Machine Learning Repository provides the encounter features and source outcomes. The separate patient_nbr identifier supports patient-grouped splitting without becoming a model feature.

• Add the first part of Cell 3 to the next empty Colab code cell by pasting this code:

# %%
# CELL 3: Fetch the public UCI hospital dataset and create the binary target.
# Only encounters with the original <30 outcome become positive examples.
dataset = fetch_ucirepo(id=296)

X = dataset.data.features.copy()
ids = dataset.data.ids.copy()
target_text = dataset.data.targets.iloc[:, 0].astype("string")

assert "patient_nbr" in ids.columns, "Expected patient_nbr in the UCI ID columns."
groups = ids["patient_nbr"].copy()

# Remove identifiers if a future loader also exposes them among the features.
X = X.drop(columns=["encounter_id", "patient_nbr"], errors="ignore")

What Does This Code Do?

• The fetch_ucirepo(id=296) call downloads the public hospital dataset.
• The X variable holds the encounter features.
• The target_text variable preserves the original readmission labels.
• The groups variable keeps each patient identifier available for safe splitting.
• The final line removes encounter and patient identifiers from the model features.

• Run Cell 3.
• Confirm that the cell finishes without an exception.

Colab should mark the cell as completed. This proves the dataset downloaded successfully.

Good progress. The encounter table is now in memory with its patient key kept outside the model features.

The Dataset Did Not Load?

Confirm that the earlier imports cell created fetch_ucirepo. Check your internet connection before rerunning Cell 3.

Ask for targeted help with diagnosing the UCI dataset fetch.

Binary classification reduces the source outcome to two classes. Only the original <30 label becomes a positive example.

The remaining cleanup converts the source missing marker to a consistent value. It also treats coded admission identifiers as categories.

• Leave one blank line at the bottom of Cell 3.
• Complete Cell 3 by pasting this code:

# Convert the source missing marker and treat coded admission IDs as categories.
X = X.replace("?", np.nan)
coded_categories = [
    "admission_type_id",
    "discharge_disposition_id",
    "admission_source_id",
]
for column in coded_categories:
    if column in X.columns:
        X[column] = X[column].astype("string").astype("object")
X = X.where(pd.notna(X), np.nan)

y = target_text.eq("<30").astype(np.int8)
assert y.nunique() == 2, "The binary target should contain both classes."

print(f"Encounters: {len(X):,}")
print(f"Features supplied by UCI: {X.shape[1]}")
print(f"Unique patients: {groups.nunique():,}")
display(target_text.value_counts(dropna=False).rename_axis("original_label").to_frame("encounters"))

How Is the Dataset Prepared?

• The missing-value conversion replaces every source ? marker with np.nan.
• The coded_categories list prevents admission identifiers from being treated as continuous measurements.
• The y target assigns 1 only to encounters with the original <30 outcome.
• The printed audit confirms the encounter count and original label frequencies.

• Rerun Cell 3.
• Confirm that the output reports 101,766 encounters.

You should see the feature count and unique-patient count below the cell. You should also see a table containing the original source-label counts.

Cell 3 Shows an Exception?

Confirm that the new code sits inside the same cell as the dataset-fetching code. Check that each coded category remains inside the coded_categories list.

Ask for help with debugging the dataset preparation cell.

Audit the imbalance and test the baseline

Class imbalance makes the common outcome dominate raw accuracy. An all-negative prediction provides a deliberate baseline that reveals whether the metric rewards a model with no readmission-detection ability.

• Create Cell 4 directly below Cell 3 with Colab's code-cell control.
• Add the missingness audit and naive baseline by pasting this code:

# %%
# CELL 4: Audit missingness and expose why accuracy alone is misleading.
# The all-negative prediction is a designed failure that should have zero positive recall.
missing_pct = X.isna().mean().mul(100).sort_values(ascending=False)
display(missing_pct.head(12).rename("missing_percent").to_frame())

class_counts = y.value_counts().sort_index()
fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(
    ["Not within 30 days", "Readmitted within 30 days"],
    class_counts.reindex([0, 1]).values,
    color=["#4C78A8", "#E45756"],
)
ax.set_title("30-day readmission target distribution")
ax.set_ylabel("Encounter count")
plt.xticks(rotation=10)
plt.tight_layout()
plt.show()

naive_prediction = np.zeros(len(y), dtype=np.int8)
print(f"Naive accuracy: {accuracy_score(y, naive_prediction):.3f}")
print(f"Naive readmission recall: {recall_score(y, naive_prediction, zero_division=0):.3f}")
print(f"Naive balanced accuracy: {balanced_accuracy_score(y, naive_prediction):.3f}")

What Does This Audit Measure?

• The missing_pct series ranks features by the percentage of missing values.
• The bar chart compares encounters from the two binary target classes.
• The naive_prediction array predicts the negative class for every encounter.
• Accuracy measures the total share of correct predictions.
• Recall measures the share of actual readmissions detected by the predictor.
• Balanced accuracy gives both classes equal influence.

Before you run this cell, which metric do you think will expose the majority predictor's failure most clearly?

• Run Cell 4.

You should see a table containing the 12 largest missingness percentages. You should also see a two-bar target distribution chart.

The naive predictor reports high accuracy. Its readmission recall is 0.

Its balanced accuracy is much weaker because the predictor only serves the majority class. Accuracy alone hides that failure.

That's the trap exposed. Your notebook now proves that a reassuring accuracy score can coexist with no detected 30-day readmissions.

The Chart or Metrics Are Missing?

Confirm that Cell 3 completed before Cell 4. Cell 4 depends on X and y from the previous cell.

Check that the plotting section ends with plt.show(). Rerun Cell 4 after correcting any incomplete line.

Ask for help with debugging the accuracy-trap audit.

✔️ Awesome, I've got everything!

Great. Double-check that Cells 1 through 4 are saved in Diabetes_Readmission_Risk.ipynb.

ⓧ I'd like to double check the full code

# %%
# CELL 1: Install a reproducible package set.
# Run this cell once, then choose Runtime > Restart runtime before Cell 2.
%pip install -q numpy==2.5.3 pandas==3.0.5 scipy==1.18.1 scikit-learn==1.9.0 matplotlib==3.11.1 ucimlrepo==0.0.7

# %%
# CELL 2: Import tools and verify the runtime before doing any analysis.
# The assertions stop the notebook early if its environment differs from the tested setup.
import platform
from importlib.metadata import version
from time import perf_counter

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
import sklearn
from ucimlrepo import fetch_ucirepo

from sklearn.calibration import CalibrationDisplay
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import ExtraTreesClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    PrecisionRecallDisplay,
    accuracy_score,
    average_precision_score,
    balanced_accuracy_score,
    brier_score_loss,
    confusion_matrix,
    fbeta_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import StratifiedGroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

versions = {
    "Python": platform.python_version(),
    "NumPy": np.__version__,
    "pandas": pd.__version__,
    "SciPy": scipy.__version__,
    "scikit-learn": sklearn.__version__,
    "Matplotlib": matplotlib.__version__,
    "ucimlrepo": version("ucimlrepo"),
}

expected_versions = {
    "NumPy": "2.5.3",
    "pandas": "3.0.5",
    "SciPy": "1.18.1",
    "scikit-learn": "1.9.0",
    "Matplotlib": "3.11.1",
    "ucimlrepo": "0.0.7",
}

assert platform.python_version().startswith("3.12"), versions
for package_name, expected_version in expected_versions.items():
    assert versions[package_name] == expected_version, versions

display(pd.DataFrame.from_dict(versions, orient="index", columns=["version"]))

# %%
# CELL 3: Fetch the public UCI hospital dataset and create the binary target.
# Only encounters with the original <30 outcome become positive examples.
dataset = fetch_ucirepo(id=296)

X = dataset.data.features.copy()
ids = dataset.data.ids.copy()
target_text = dataset.data.targets.iloc[:, 0].astype("string")

assert "patient_nbr" in ids.columns, "Expected patient_nbr in the UCI ID columns."
groups = ids["patient_nbr"].copy()

# Remove identifiers if a future loader also exposes them among the features.
X = X.drop(columns=["encounter_id", "patient_nbr"], errors="ignore")

# Convert the source missing marker and treat coded admission IDs as categories.
X = X.replace("?", np.nan)
coded_categories = [
    "admission_type_id",
    "discharge_disposition_id",
    "admission_source_id",
]
for column in coded_categories:
    if column in X.columns:
        X[column] = X[column].astype("string").astype("object")
X = X.where(pd.notna(X), np.nan)

y = target_text.eq("<30").astype(np.int8)
assert y.nunique() == 2, "The binary target should contain both classes."

print(f"Encounters: {len(X):,}")
print(f"Features supplied by UCI: {X.shape[1]}")
print(f"Unique patients: {groups.nunique():,}")
display(target_text.value_counts(dropna=False).rename_axis("original_label").to_frame("encounters"))

# %%
# CELL 4: Audit missingness and expose why accuracy alone is misleading.
# The all-negative prediction is a designed failure that should have zero positive recall.
missing_pct = X.isna().mean().mul(100).sort_values(ascending=False)
display(missing_pct.head(12).rename("missing_percent").to_frame())

class_counts = y.value_counts().sort_index()
fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(
    ["Not within 30 days", "Readmitted within 30 days"],
    class_counts.reindex([0, 1]).values,
    color=["#4C78A8", "#E45756"],
)
ax.set_title("30-day readmission target distribution")
ax.set_ylabel("Encounter count")
plt.xticks(rotation=10)
plt.tight_layout()
plt.show()

naive_prediction = np.zeros(len(y), dtype=np.int8)
print(f"Naive accuracy: {accuracy_score(y, naive_prediction):.3f}")
print(f"Naive readmission recall: {recall_score(y, naive_prediction, zero_division=0):.3f}")
print(f"Naive balanced accuracy: {balanced_accuracy_score(y, naive_prediction):.3f}")

Your target audit now shows why the next model comparison needs imbalance-aware metrics. Next, you will keep repeat encounters from the same patient inside a single data partition.

Create Patient-Safe Splits

Your baseline exposed how class imbalance can hide missed readmissions. The next risk comes from repeat encounters belonging to the same patient.

A patient-grouped split keeps each patient inside one partition. A training-fitted preprocessing pipeline also prevents validation or test data from influencing cleanup decisions.

In this step, get ready to:

• Create patient-disjoint training, validation, and test partitions.
• Learn missingness rules from training data.
• Build a fresh preprocessor for every candidate model.

Split encounters by patient

Repeat encounters from one patient can share patterns that make validation results look stronger than they are. StratifiedGroupKFold approximately preserves the target balance while keeping every patient_nbr group intact.

• Insert a code cell directly below the all-negative baseline cell.
• Create the patient-level partitions by pasting this first part into the new cell:

# %%
# CELL 5: Split by patient while approximately preserving the class distribution.
# No patient is allowed to appear in more than one partition.
outer_splitter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
train_val_idx, test_idx = next(outer_splitter.split(X, y, groups=groups))

X_train_val = X.iloc[train_val_idx].copy()
y_train_val = y.iloc[train_val_idx].copy()
groups_train_val = groups.iloc[train_val_idx].copy()
X_test = X.iloc[test_idx].copy()
y_test = y.iloc[test_idx].copy()
groups_test = groups.iloc[test_idx].copy()

inner_splitter = StratifiedGroupKFold(n_splits=4, shuffle=True, random_state=43)
train_idx, val_idx = next(
    inner_splitter.split(X_train_val, y_train_val, groups=groups_train_val)
)

X_train = X_train_val.iloc[train_idx].copy()
y_train = y_train_val.iloc[train_idx].copy()
groups_train = groups_train_val.iloc[train_idx].copy()
X_val = X_train_val.iloc[val_idx].copy()
y_val = y_train_val.iloc[val_idx].copy()
groups_val = groups_train_val.iloc[val_idx].copy()

assert set(groups_train).isdisjoint(set(groups_val))
assert set(groups_train).isdisjoint(set(groups_test))
assert set(groups_val).isdisjoint(set(groups_test))

What Does This Code Do?

• The outer split holds out approximately one fifth of the encounters for the final test.
• The inner split divides the remaining encounters into training and validation partitions.
• The three assertions verify that no patient identifier is shared across partitions.

• Run the cell to test the patient-overlap assertions.

The cell should finish without an assertion error. This confirms that all three patient groups are disjoint.

The Split Cell Did Not Finish?

Confirm that the runtime still contains X, y, and groups from the earlier data-loading cell.

Check that each split call receives groups=groups or groups=groups_train_val as shown.

Ask for help with diagnosing the patient-grouped split.

A compact summary makes the partition sizes visible. It also lets you compare the positive readmission rates before model training begins.

• Add the partition summary directly below the overlap assertions by pasting this code:

split_summary = pd.DataFrame(
    {
        "partition": ["train", "validation", "test"],
        "encounters": [len(X_train), len(X_val), len(X_test)],
        "patients": [groups_train.nunique(), groups_val.nunique(), groups_test.nunique()],
        "positive_rate": [y_train.mean(), y_val.mean(), y_test.mean()],
    }
)
display(split_summary)
print("Patient overlap checks passed: train, validation, and test are disjoint.")

What Does the Summary Show?

The table reports encounter counts, unique patient counts, and positive rates for all three partitions. Similar positive rates show that stratification kept the rare outcome distributed across the split.

Before you run the completed cell, predict whether any patient will appear in more than one partition.

• Run the completed patient-splitting cell.

You should see three rows for train, validation, and test. Below the table, you should see Patient overlap checks passed: train, validation, and test are disjoint.

The Partition Rates Look Wrong?

Confirm that both splitters receive the matching target and patient-group series. Misaligned inputs can produce incorrect partitions.

Check that train_idx and val_idx index X_train_val instead of the full X table.

Ask for help with checking the split summary.

Learn cleanup rules from training data

Missingness decisions can leak information when they use the full dataset. This audit learns the drop list from X_train before applying that fixed list to validation and test data.

• Insert a code cell directly below the patient-splitting cell.
• Create the training-only audit by pasting this code into the new cell:

# %%
# CELL 6: Learn the data-cleaning decisions from training data only.
# Each candidate model receives a fresh copy of the same leakage-safe preprocessor.
training_audit = pd.DataFrame(
    {
        "dtype": X_train.dtypes.astype(str),
        "missing_percent": X_train.isna().mean().mul(100),
        "unique_values": X_train.nunique(dropna=True),
    }
).sort_values("missing_percent", ascending=False)
display(training_audit.head(12))

drop_columns = training_audit.index[training_audit["missing_percent"] > 60].tolist()
X_train = X_train.drop(columns=drop_columns)
X_val = X_val.drop(columns=drop_columns)
X_test = X_test.drop(columns=drop_columns)

numeric_features = X_train.select_dtypes(include="number").columns.tolist()
categorical_features = X_train.select_dtypes(exclude="number").columns.tolist()

print(f"Dropped for training missingness > 60%: {drop_columns}")
print(f"Numeric features: {len(numeric_features)}")
print(f"Categorical features: {len(categorical_features)}")

What Does This Code Do?

• The audit measures data types, missing percentages, and unique values using training encounters only.
• The drop rule removes columns whose training missingness exceeds 60 percent.
• The same drop_columns list is applied to all three partitions.
• The remaining columns are separated into numeric and categorical feature lists.

• Run the training-audit cell.

You should see the 12 columns with the highest training missingness. Below the table, you should see the fixed drop list plus numeric and categorical feature counts.

The Feature Counts Are Missing?

Confirm that the patient-splitting cell ran successfully before this cell. The audit depends on X_train, X_val, and X_test.

Check that the missingness comparison uses > 60 against training_audit.

Ask for help with debugging the training-only audit.

Build the leakage-safe preprocessor

A fresh ColumnTransformer lets every candidate model fit its own transformations on training data. Numeric fields receive median imputation plus scaling. Categorical fields receive most-frequent imputation plus one-hot encoding.

• Add the preprocessing pipelines below the feature-count lines by pasting this first part:

def make_preprocessor():
    """Return a new preprocessing graph so models cannot share fitted state."""
    numeric_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]
    )
    categorical_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            (
                "onehot",
                OneHotEncoder(
                    handle_unknown="ignore",
                    min_frequency=50,
                    sparse_output=True,
                ),
            ),
        ]
    )

How Are the Features Prepared?

The numeric pipeline fills missing values with each training column's median. StandardScaler then learns training-based scaling statistics.

The categorical pipeline fills missing values with the most frequent training category. OneHotEncoder safely handles unseen categories while grouping categories that appear fewer than 50 times.

• Complete make_preprocessor() by adding this return block directly below the categorical pipeline:

    return ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features),
        ],
        sparse_threshold=1.0,
    )

Why Return a New Transformer?

Each call creates a separate preprocessing graph from the same feature lists. Candidate models can therefore share the same preparation rules without sharing fitted state.

Before you run the completed cell, predict which two feature counts the notebook will print.

• Run the completed training-audit and preprocessing cell.

You should see the training-only audit followed by the dropped-column list. You should also see separate numeric and categorical feature counts without an error.

The Preprocessor Cell Fails?

Check that the return block remains indented inside make_preprocessor().

Confirm that numeric_features and categorical_features were created before the function definition.

Ask for help with debugging the preprocessing function.

✔️ Awesome, I've got everything!

Great. Make sure both new cells have run successfully before you continue.

ⓧ I'd like to double check the full code

# %%
# CELL 1: Install a reproducible package set.
# Run this cell once, then choose Runtime > Restart runtime before Cell 2.
%pip install -q numpy==2.5.3 pandas==3.0.5 scipy==1.18.1 scikit-learn==1.9.0 matplotlib==3.11.1 ucimlrepo==0.0.7

# %%
# CELL 2: Import tools and verify the runtime before doing any analysis.
# The assertions stop the notebook early if its environment differs from the tested setup.
import platform
from importlib.metadata import version
from time import perf_counter

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
import sklearn
from ucimlrepo import fetch_ucirepo

from sklearn.calibration import CalibrationDisplay
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import ExtraTreesClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    PrecisionRecallDisplay,
    accuracy_score,
    average_precision_score,
    balanced_accuracy_score,
    brier_score_loss,
    confusion_matrix,
    fbeta_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import StratifiedGroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

versions = {
    "Python": platform.python_version(),
    "NumPy": np.__version__,
    "pandas": pd.__version__,
    "SciPy": scipy.__version__,
    "scikit-learn": sklearn.__version__,
    "Matplotlib": matplotlib.__version__,
    "ucimlrepo": version("ucimlrepo"),
}

expected_versions = {
    "NumPy": "2.5.3",
    "pandas": "3.0.5",
    "SciPy": "1.18.1",
    "scikit-learn": "1.9.0",
    "Matplotlib": "3.11.1",
    "ucimlrepo": "0.0.7",
}

assert platform.python_version().startswith("3.12"), versions
for package_name, expected_version in expected_versions.items():
    assert versions[package_name] == expected_version, versions

display(pd.DataFrame.from_dict(versions, orient="index", columns=["version"]))

# %%
# CELL 3: Fetch the public UCI hospital dataset and create the binary target.
# Only encounters with the original <30 outcome become positive examples.
dataset = fetch_ucirepo(id=296)

X = dataset.data.features.copy()
ids = dataset.data.ids.copy()
target_text = dataset.data.targets.iloc[:, 0].astype("string")

assert "patient_nbr" in ids.columns, "Expected patient_nbr in the UCI ID columns."
groups = ids["patient_nbr"].copy()

# Remove identifiers if a future loader also exposes them among the features.
X = X.drop(columns=["encounter_id", "patient_nbr"], errors="ignore")

# Convert the source missing marker and treat coded admission IDs as categories.
X = X.replace("?", np.nan)
coded_categories = [
    "admission_type_id",
    "discharge_disposition_id",
    "admission_source_id",
]
for column in coded_categories:
    if column in X.columns:
        X[column] = X[column].astype("string").astype("object")
X = X.where(pd.notna(X), np.nan)

y = target_text.eq("<30").astype(np.int8)
assert y.nunique() == 2, "The binary target should contain both classes."

print(f"Encounters: {len(X):,}")
print(f"Features supplied by UCI: {X.shape[1]}")
print(f"Unique patients: {groups.nunique():,}")
display(target_text.value_counts(dropna=False).rename_axis("original_label").to_frame("encounters"))

# %%
# CELL 4: Audit missingness and expose why accuracy alone is misleading.
# The all-negative prediction is a designed failure that should have zero positive recall.
missing_pct = X.isna().mean().mul(100).sort_values(ascending=False)
display(missing_pct.head(12).rename("missing_percent").to_frame())

class_counts = y.value_counts().sort_index()
fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(
    ["Not within 30 days", "Readmitted within 30 days"],
    class_counts.reindex([0, 1]).values,
    color=["#4C78A8", "#E45756"],
)
ax.set_title("30-day readmission target distribution")
ax.set_ylabel("Encounter count")
plt.xticks(rotation=10)
plt.tight_layout()
plt.show()

naive_prediction = np.zeros(len(y), dtype=np.int8)
print(f"Naive accuracy: {accuracy_score(y, naive_prediction):.3f}")
print(f"Naive readmission recall: {recall_score(y, naive_prediction, zero_division=0):.3f}")
print(f"Naive balanced accuracy: {balanced_accuracy_score(y, naive_prediction):.3f}")

# %%
# CELL 5: Split by patient while approximately preserving the class distribution.
# No patient is allowed to appear in more than one partition.
outer_splitter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
train_val_idx, test_idx = next(outer_splitter.split(X, y, groups=groups))

X_train_val = X.iloc[train_val_idx].copy()
y_train_val = y.iloc[train_val_idx].copy()
groups_train_val = groups.iloc[train_val_idx].copy()
X_test = X.iloc[test_idx].copy()
y_test = y.iloc[test_idx].copy()
groups_test = groups.iloc[test_idx].copy()

inner_splitter = StratifiedGroupKFold(n_splits=4, shuffle=True, random_state=43)
train_idx, val_idx = next(
    inner_splitter.split(X_train_val, y_train_val, groups=groups_train_val)
)

X_train = X_train_val.iloc[train_idx].copy()
y_train = y_train_val.iloc[train_idx].copy()
groups_train = groups_train_val.iloc[train_idx].copy()
X_val = X_train_val.iloc[val_idx].copy()
y_val = y_train_val.iloc[val_idx].copy()
groups_val = groups_train_val.iloc[val_idx].copy()

assert set(groups_train).isdisjoint(set(groups_val))
assert set(groups_train).isdisjoint(set(groups_test))
assert set(groups_val).isdisjoint(set(groups_test))

split_summary = pd.DataFrame(
    {
        "partition": ["train", "validation", "test"],
        "encounters": [len(X_train), len(X_val), len(X_test)],
        "patients": [groups_train.nunique(), groups_val.nunique(), groups_test.nunique()],
        "positive_rate": [y_train.mean(), y_val.mean(), y_test.mean()],
    }
)
display(split_summary)
print("Patient overlap checks passed: train, validation, and test are disjoint.")

# %%
# CELL 6: Learn the data-cleaning decisions from training data only.
# Each candidate model receives a fresh copy of the same leakage-safe preprocessor.
training_audit = pd.DataFrame(
    {
        "dtype": X_train.dtypes.astype(str),
        "missing_percent": X_train.isna().mean().mul(100),
        "unique_values": X_train.nunique(dropna=True),
    }
).sort_values("missing_percent", ascending=False)
display(training_audit.head(12))

drop_columns = training_audit.index[training_audit["missing_percent"] > 60].tolist()
X_train = X_train.drop(columns=drop_columns)
X_val = X_val.drop(columns=drop_columns)
X_test = X_test.drop(columns=drop_columns)

numeric_features = X_train.select_dtypes(include="number").columns.tolist()
categorical_features = X_train.select_dtypes(exclude="number").columns.tolist()

print(f"Dropped for training missingness > 60%: {drop_columns}")
print(f"Numeric features: {len(numeric_features)}")
print(f"Categorical features: {len(categorical_features)}")


def make_preprocessor():
    """Return a new preprocessing graph so models cannot share fitted state."""
    numeric_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]
    )
    categorical_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            (
                "onehot",
                OneHotEncoder(
                    handle_unknown="ignore",
                    min_frequency=50,
                    sparse_output=True,
                ),
            ),
        ]
    )
    return ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features),
        ],
        sparse_threshold=1.0,
    )

Your partitions now protect patient boundaries. Next, you will compare three candidate models using the same training-fitted preparation rules.

Compare Models and Select a Winner

Your patient-safe partitions give every candidate the same leakage-protected starting point. The remaining risk is choosing a model because it sounds sophisticated.

The notebook will compare three scikit-learn candidates against a prior-only baseline. Validation Average Precision will select the winner from evidence.

In this step, get ready to:

• Define shared probability metrics plus a DummyClassifier prior baseline.
• Fit class-weighted Logistic Regression, Random Forest, and Extra Trees pipelines.
• Rank the real candidates by validation Average Precision.

Define the evaluation contract and baseline

A shared evaluation function applies the same metrics to every model. The prior baseline shows how much value each real candidate adds beyond predicting from the training prevalence.

• Add a new code cell directly below Cell 6.
• Build the metric helpers in that cell by pasting this first section:

# %%
# CELL 7: Define one metric function and fit the majority-class sanity baseline.
# Average Precision is the selection metric, while threshold metrics explain operations.
def evaluate_probabilities(model_name, y_true, probabilities, threshold=0.5, train_seconds=np.nan):
    """Return a single comparable metric row from positive-class probabilities."""
    predictions = (probabilities >= threshold).astype(np.int8)
    tn, fp, fn, tp = confusion_matrix(y_true, predictions, labels=[0, 1]).ravel()
    specificity = tn / (tn + fp) if (tn + fp) else np.nan

    return {
        "model": model_name,
        "threshold": threshold,
        "accuracy": accuracy_score(y_true, predictions),
        "average_precision": average_precision_score(y_true, probabilities),
        "roc_auc": roc_auc_score(y_true, probabilities),
        "recall": recall_score(y_true, predictions, zero_division=0),
        "precision": precision_score(y_true, predictions, zero_division=0),
        "f2": fbeta_score(y_true, predictions, beta=2, zero_division=0),
        "specificity": specificity,
        "balanced_accuracy": balanced_accuracy_score(y_true, predictions),
        "brier_score": brier_score_loss(y_true, probabilities),
        "train_seconds": train_seconds,
    }


def positive_probabilities(fitted_pipeline, features):
    """Read the probability column that corresponds to binary class 1."""
    classes = fitted_pipeline.named_steps["model"].classes_
    positive_index = int(np.where(classes == 1)[0][0])
    return fitted_pipeline.predict_proba(features)[:, positive_index]

What Do These Helpers Do?

• The evaluate_probabilities() function converts probabilities into predictions at one threshold.
• Its returned row combines ranking metrics with operational metrics.
• The positive_probabilities() helper finds the probability column associated with class 1.

• Complete the same cell by pasting this baseline section beneath positive_probabilities():

dummy = DummyClassifier(strategy="prior")
dummy_train_X = np.zeros((len(y_train), 1))
dummy_val_X = np.zeros((len(y_val), 1))
dummy.fit(dummy_train_X, y_train)
dummy_positive_index = int(np.where(dummy.classes_ == 1)[0][0])
dummy_val_prob = dummy.predict_proba(dummy_val_X)[:, dummy_positive_index]

validation_rows = [
    evaluate_probabilities("Dummy prior", y_val, dummy_val_prob)
]
validation_probabilities = {"Dummy prior": dummy_val_prob}

How Does the Baseline Work?

The baseline learns only the training class prior. Its validation probabilities establish the minimum evidence that a real model should beat.

The two storage objects will collect one metric row plus one probability array for every candidate.

• Run the completed Cell 7.

The cell should finish without an exception. The baseline probabilities and first validation row are now available for the comparison.

Cell 7 Did Not Finish?

Confirm that Cell 2 imported the metric functions. Check that the training and validation targets from earlier still exist.

Ask for targeted help with debugging the evaluation and baseline cell.

Train the three candidate pipelines

Each candidate receives a fresh preprocessor from make_preprocessor(). Class weighting makes the minority outcome matter during fitting without adding synthetic patient rows.

• Add a new code cell directly below Cell 7.
• Start Cell 8 by pasting this first half of the candidate dictionary:

# %%
# CELL 8: Train the three real candidate models on the same training partition.
# Class weighting makes the minority outcome matter during fitting without synthetic rows.
candidate_pipelines = {
    "Logistic Regression": Pipeline(
        steps=[
            ("preprocess", make_preprocessor()),
            (
                "model",
                LogisticRegression(
                    solver="saga",
                    max_iter=300,
                    tol=1e-3,
                    class_weight="balanced",
                    random_state=42,
                ),
            ),
        ]
    ),
    "Random Forest": Pipeline(
        steps=[
            ("preprocess", make_preprocessor()),
            (
                "model",

What Does the First Half Define?

This first half opens the candidate dictionary. It fully configures Logistic Regression before beginning the Random Forest entry.

Keep both halves in the same code cell. The next block closes every open structure.

• Complete candidate_pipelines by pasting this second half immediately below the first:

                RandomForestClassifier(
                    n_estimators=75,
                    max_depth=12,
                    min_samples_leaf=10,
                    class_weight="balanced",
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
    "Extra Trees": Pipeline(
        steps=[
            ("preprocess", make_preprocessor()),
            (
                "model",
                ExtraTreesClassifier(
                    n_estimators=75,
                    max_depth=12,
                    min_samples_leaf=10,
                    class_weight="balanced",
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
}

Why Use the Same Shape?

Random Forest and Extra Trees use the same tree limits. Every candidate wraps its own preprocessing graph before its estimator.

This shared structure keeps the comparison focused on model behavior.

• Run the candidate dictionary cell.

The execution count should complete without a traceback. The candidate_pipelines dictionary now contains three unfitted pipelines.

Seeing an Incomplete Cell Error?

Confirm that both halves are in the same code cell. Check that the final closing brace appears after the Extra Trees pipeline.

Ask for help with checking the candidate dictionary structure.

• Extend Cell 8 by pasting this training loop beneath the candidate dictionary:

fitted_models = {}
for model_name, pipeline in candidate_pipelines.items():
    started = perf_counter()
    pipeline.fit(X_train, y_train)
    elapsed = perf_counter() - started

    val_prob = positive_probabilities(pipeline, X_val)
    fitted_models[model_name] = pipeline
    validation_probabilities[model_name] = val_prob
    validation_rows.append(
        evaluate_probabilities(
            model_name,
            y_val,
            val_prob,
            threshold=0.5,
            train_seconds=elapsed,
        )
    )
    print(f"Finished {model_name} in {elapsed:.1f} seconds")

What Does the Training Loop Record?

• The loop fits each pipeline on the same training partition.
• It stores each fitted pipeline in fitted_models.
• It generates validation probabilities before adding a comparable metric row.
• It records elapsed fitting time for the final comparison table.

This is the heaviest cell so far. A quiet interval between completion lines is expected while each model fits.

• Run the completed Cell 8.

You should see one Finished line for each of the three candidates. Each line includes that model's elapsed training time.

A Candidate Did Not Finish?

Confirm that Cell 6 created make_preprocessor(). Check that every candidate name has a matching closing parenthesis.

If the runtime disconnected, rerun the earlier cells in order before retrying Cell 8.

Ask for targeted help with diagnosing the failed candidate fit.

Rank the candidates and select the winner

Model selection belongs on validation data because the test partition remains untouched for the final audit. The notebook excludes the dummy baseline from winning even if its row appears in the table.

• Add a new code cell directly below Cell 8.
• Build the evidence-based selection cell by pasting this code:

# %%
# CELL 9: Compare validation results and select the winner from evidence.
# The winner is not predetermined and the Dummy baseline cannot be selected.
validation_results = pd.DataFrame(validation_rows).sort_values(
    "average_precision", ascending=False
)
display(validation_results.round(4))

candidate_results = validation_results[
    validation_results["model"] != "Dummy prior"
]
winner_name = candidate_results.iloc[0]["model"]
winner = fitted_models[winner_name]
winner_val_prob = validation_probabilities[winner_name]

print("Selection rule: highest validation Average Precision among the three candidates.")
print(f"Selected model: {winner_name}")

How Is the Winner Selected?

The table sorts every validation row by average_precision from highest to lowest. The baseline stays visible as a sanity check.

The first non-dummy row supplies winner_name. Its fitted pipeline and validation probabilities are retained for threshold tuning.

Make a quick mental prediction about which candidate will lead on validation Average Precision.

• Run Cell 9 to reveal the validation ranking.

You should see four sorted rows containing the baseline plus all three candidates. The final line names the real candidate with the highest validation Average Precision.

The Comparison Table Is Missing a Model?

Confirm that Cell 8 printed three completion lines. Rerun Cell 7 before Cell 8 if validation_rows was reset.

Ask for help with rebuilding the validation comparison.

That is the model choice grounded in evidence. Your notebook now retains the winning fitted pipeline without hard-coding its name.

✔️ Awesome, I've got everything!

Excellent work. Confirm that Cells 7 through 9 are saved in your notebook.

ⓧ I'd like to double check the full code

# %%
# CELL 1: Install a reproducible package set.
# Run this cell once, then choose Runtime > Restart runtime before Cell 2.
%pip install -q numpy==2.5.3 pandas==3.0.5 scipy==1.18.1 scikit-learn==1.9.0 matplotlib==3.11.1 ucimlrepo==0.0.7

# %%
# CELL 2: Import tools and verify the runtime before doing any analysis.
# The assertions stop the notebook early if its environment differs from the tested setup.
import platform
from importlib.metadata import version
from time import perf_counter

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
import sklearn
from ucimlrepo import fetch_ucirepo

from sklearn.calibration import CalibrationDisplay
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import ExtraTreesClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    PrecisionRecallDisplay,
    accuracy_score,
    average_precision_score,
    balanced_accuracy_score,
    brier_score_loss,
    confusion_matrix,
    fbeta_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import StratifiedGroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

versions = {
    "Python": platform.python_version(),
    "NumPy": np.__version__,
    "pandas": pd.__version__,
    "SciPy": scipy.__version__,
    "scikit-learn": sklearn.__version__,
    "Matplotlib": matplotlib.__version__,
    "ucimlrepo": version("ucimlrepo"),
}

expected_versions = {
    "NumPy": "2.5.3",
    "pandas": "3.0.5",
    "SciPy": "1.18.1",
    "scikit-learn": "1.9.0",
    "Matplotlib": "3.11.1",
    "ucimlrepo": "0.0.7",
}

assert platform.python_version().startswith("3.12"), versions
for package_name, expected_version in expected_versions.items():
    assert versions[package_name] == expected_version, versions

display(pd.DataFrame.from_dict(versions, orient="index", columns=["version"]))

# %%
# CELL 3: Fetch the public UCI hospital dataset and create the binary target.
# Only encounters with the original <30 outcome become positive examples.
dataset = fetch_ucirepo(id=296)

X = dataset.data.features.copy()
ids = dataset.data.ids.copy()
target_text = dataset.data.targets.iloc[:, 0].astype("string")

assert "patient_nbr" in ids.columns, "Expected patient_nbr in the UCI ID columns."
groups = ids["patient_nbr"].copy()

# Remove identifiers if a future loader also exposes them among the features.
X = X.drop(columns=["encounter_id", "patient_nbr"], errors="ignore")

# Convert the source missing marker and treat coded admission IDs as categories.
X = X.replace("?", np.nan)
coded_categories = [
    "admission_type_id",
    "discharge_disposition_id",
    "admission_source_id",
]
for column in coded_categories:
    if column in X.columns:
        X[column] = X[column].astype("string").astype("object")
X = X.where(pd.notna(X), np.nan)

y = target_text.eq("<30").astype(np.int8)
assert y.nunique() == 2, "The binary target should contain both classes."

print(f"Encounters: {len(X):,}")
print(f"Features supplied by UCI: {X.shape[1]}")
print(f"Unique patients: {groups.nunique():,}")
display(target_text.value_counts(dropna=False).rename_axis("original_label").to_frame("encounters"))

# %%
# CELL 4: Audit missingness and expose why accuracy alone is misleading.
# The all-negative prediction is a designed failure that should have zero positive recall.
missing_pct = X.isna().mean().mul(100).sort_values(ascending=False)
display(missing_pct.head(12).rename("missing_percent").to_frame())

class_counts = y.value_counts().sort_index()
fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(
    ["Not within 30 days", "Readmitted within 30 days"],
    class_counts.reindex([0, 1]).values,
    color=["#4C78A8", "#E45756"],
)
ax.set_title("30-day readmission target distribution")
ax.set_ylabel("Encounter count")
plt.xticks(rotation=10)
plt.tight_layout()
plt.show()

naive_prediction = np.zeros(len(y), dtype=np.int8)
print(f"Naive accuracy: {accuracy_score(y, naive_prediction):.3f}")
print(f"Naive readmission recall: {recall_score(y, naive_prediction, zero_division=0):.3f}")
print(f"Naive balanced accuracy: {balanced_accuracy_score(y, naive_prediction):.3f}")

# %%
# CELL 5: Split by patient while approximately preserving the class distribution.
# No patient is allowed to appear in more than one partition.
outer_splitter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
train_val_idx, test_idx = next(outer_splitter.split(X, y, groups=groups))

X_train_val = X.iloc[train_val_idx].copy()
y_train_val = y.iloc[train_val_idx].copy()
groups_train_val = groups.iloc[train_val_idx].copy()
X_test = X.iloc[test_idx].copy()
y_test = y.iloc[test_idx].copy()
groups_test = groups.iloc[test_idx].copy()

inner_splitter = StratifiedGroupKFold(n_splits=4, shuffle=True, random_state=43)
train_idx, val_idx = next(
    inner_splitter.split(X_train_val, y_train_val, groups=groups_train_val)
)

X_train = X_train_val.iloc[train_idx].copy()
y_train = y_train_val.iloc[train_idx].copy()
groups_train = groups_train_val.iloc[train_idx].copy()
X_val = X_train_val.iloc[val_idx].copy()
y_val = y_train_val.iloc[val_idx].copy()
groups_val = groups_train_val.iloc[val_idx].copy()

assert set(groups_train).isdisjoint(set(groups_val))
assert set(groups_train).isdisjoint(set(groups_test))
assert set(groups_val).isdisjoint(set(groups_test))

split_summary = pd.DataFrame(
    {
        "partition": ["train", "validation", "test"],
        "encounters": [len(X_train), len(X_val), len(X_test)],
        "patients": [groups_train.nunique(), groups_val.nunique(), groups_test.nunique()],
        "positive_rate": [y_train.mean(), y_val.mean(), y_test.mean()],
    }
)
display(split_summary)
print("Patient overlap checks passed: train, validation, and test are disjoint.")

# %%
# CELL 6: Learn the data-cleaning decisions from training data only.
# Each candidate model receives a fresh copy of the same leakage-safe preprocessor.
training_audit = pd.DataFrame(
    {
        "dtype": X_train.dtypes.astype(str),
        "missing_percent": X_train.isna().mean().mul(100),
        "unique_values": X_train.nunique(dropna=True),
    }
).sort_values("missing_percent", ascending=False)
display(training_audit.head(12))

drop_columns = training_audit.index[training_audit["missing_percent"] > 60].tolist()
X_train = X_train.drop(columns=drop_columns)
X_val = X_val.drop(columns=drop_columns)
X_test = X_test.drop(columns=drop_columns)

numeric_features = X_train.select_dtypes(include="number").columns.tolist()
categorical_features = X_train.select_dtypes(exclude="number").columns.tolist()

print(f"Dropped for training missingness > 60%: {drop_columns}")
print(f"Numeric features: {len(numeric_features)}")
print(f"Categorical features: {len(categorical_features)}")


def make_preprocessor():
    """Return a new preprocessing graph so models cannot share fitted state."""
    numeric_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]
    )
    categorical_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            (
                "onehot",
                OneHotEncoder(
                    handle_unknown="ignore",
                    min_frequency=50,
                    sparse_output=True,
                ),
            ),
        ]
    )
    return ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features),
        ],
        sparse_threshold=1.0,
    )

# %%
# CELL 7: Define one metric function and fit the majority-class sanity baseline.
# Average Precision is the selection metric, while threshold metrics explain operations.
def evaluate_probabilities(model_name, y_true, probabilities, threshold=0.5, train_seconds=np.nan):
    """Return a single comparable metric row from positive-class probabilities."""
    predictions = (probabilities >= threshold).astype(np.int8)
    tn, fp, fn, tp = confusion_matrix(y_true, predictions, labels=[0, 1]).ravel()
    specificity = tn / (tn + fp) if (tn + fp) else np.nan

    return {
        "model": model_name,
        "threshold": threshold,
        "accuracy": accuracy_score(y_true, predictions),
        "average_precision": average_precision_score(y_true, probabilities),
        "roc_auc": roc_auc_score(y_true, probabilities),
        "recall": recall_score(y_true, predictions, zero_division=0),
        "precision": precision_score(y_true, predictions, zero_division=0),
        "f2": fbeta_score(y_true, predictions, beta=2, zero_division=0),
        "specificity": specificity,
        "balanced_accuracy": balanced_accuracy_score(y_true, predictions),
        "brier_score": brier_score_loss(y_true, probabilities),
        "train_seconds": train_seconds,
    }


def positive_probabilities(fitted_pipeline, features):
    """Read the probability column that corresponds to binary class 1."""
    classes = fitted_pipeline.named_steps["model"].classes_
    positive_index = int(np.where(classes == 1)[0][0])
    return fitted_pipeline.predict_proba(features)[:, positive_index]


dummy = DummyClassifier(strategy="prior")
dummy_train_X = np.zeros((len(y_train), 1))
dummy_val_X = np.zeros((len(y_val), 1))
dummy.fit(dummy_train_X, y_train)
dummy_positive_index = int(np.where(dummy.classes_ == 1)[0][0])
dummy_val_prob = dummy.predict_proba(dummy_val_X)[:, dummy_positive_index]

validation_rows = [
    evaluate_probabilities("Dummy prior", y_val, dummy_val_prob)
]
validation_probabilities = {"Dummy prior": dummy_val_prob}

# %%
# CELL 8: Train the three real candidate models on the same training partition.
# Class weighting makes the minority outcome matter during fitting without synthetic rows.
candidate_pipelines = {
    "Logistic Regression": Pipeline(
        steps=[
            ("preprocess", make_preprocessor()),
            (
                "model",
                LogisticRegression(
                    solver="saga",
                    max_iter=300,
                    tol=1e-3,
                    class_weight="balanced",
                    random_state=42,
                ),
            ),
        ]
    ),
    "Random Forest": Pipeline(
        steps=[
            ("preprocess", make_preprocessor()),
            (
                "model",
                RandomForestClassifier(
                    n_estimators=75,
                    max_depth=12,
                    min_samples_leaf=10,
                    class_weight="balanced",
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
    "Extra Trees": Pipeline(
        steps=[
            ("preprocess", make_preprocessor()),
            (
                "model",
                ExtraTreesClassifier(
                    n_estimators=75,
                    max_depth=12,
                    min_samples_leaf=10,
                    class_weight="balanced",
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
}

fitted_models = {}
for model_name, pipeline in candidate_pipelines.items():
    started = perf_counter()
    pipeline.fit(X_train, y_train)
    elapsed = perf_counter() - started

    val_prob = positive_probabilities(pipeline, X_val)
    fitted_models[model_name] = pipeline
    validation_probabilities[model_name] = val_prob
    validation_rows.append(
        evaluate_probabilities(
            model_name,
            y_val,
            val_prob,
            threshold=0.5,
            train_seconds=elapsed,
        )
    )
    print(f"Finished {model_name} in {elapsed:.1f} seconds")

# %%
# CELL 9: Compare validation results and select the winner from evidence.
# The winner is not predetermined and the Dummy baseline cannot be selected.
validation_results = pd.DataFrame(validation_rows).sort_values(
    "average_precision", ascending=False
)
display(validation_results.round(4))

candidate_results = validation_results[
    validation_results["model"] != "Dummy prior"
]
winner_name = candidate_results.iloc[0]["model"]
winner = fitted_models[winner_name]
winner_val_prob = validation_probabilities[winner_name]

print("Selection rule: highest validation Average Precision among the three candidates.")
print(f"Selected model: {winner_name}")

Your validation evidence has chosen the model. Next, you will tune its decision threshold before opening the untouched test partition.

Tune and Audit the Final Model

Your Google Colab notebook has selected the real candidate with the strongest validation Average Precision. The untouched test partition still holds the evidence needed for a fair final audit.

A default 0.5 cutoff may miss too many readmissions. Validation F2 provides a recall-sensitive operating point before the test data is opened once.

In this step, get ready to:

• Select a recall-sensitive threshold from validation data.
• Evaluate the winner once on the untouched test partition.
• Audit model signals plus subgroup behavior.

Tune the validation threshold

A decision threshold turns each probability into a review flag. The precision-recall curve lets the notebook compare precision with recall across every available validation threshold.

Before you run this cell, where do you expect the recall-sensitive threshold to sit relative to 0.5?

• Add a code cell below Cell 9 in the Colab notebook from earlier.
• Add the threshold-selection logic by copying this code into the new cell:

# %%
# CELL 10: Choose a recall-sensitive operating threshold on validation data.
# F2 gives recall more influence than precision and keeps test data untouched.
val_precision, val_recall, thresholds = precision_recall_curve(
    y_val, winner_val_prob
)
f2_values = (
    5 * val_precision[:-1] * val_recall[:-1]
    / (4 * val_precision[:-1] + val_recall[:-1] + 1e-12)
)
best_index = int(np.nanargmax(f2_values))
best_threshold = float(thresholds[best_index])

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(thresholds, val_precision[:-1], label="Precision")
ax.plot(thresholds, val_recall[:-1], label="Recall")
ax.plot(thresholds, f2_values, label="F2")
ax.axvline(best_threshold, color="black", linestyle="--", label="Selected threshold")
ax.set_xlabel("Decision threshold")
ax.set_ylabel("Validation score")
ax.set_title(f"Threshold selection for {winner_name}")
ax.legend()
plt.tight_layout()
plt.show()

print(f"Validation-selected threshold: {best_threshold:.3f}")
print(f"Validation F2 at selected threshold: {f2_values[best_index]:.3f}")

How Is the Threshold Chosen?

• The curve calculates precision plus recall across the winner's validation probabilities.
• The f2_values array gives recall more influence than precision.
• The best_threshold value comes from the highest validation F2 score.
• The test partition stays untouched throughout this choice.

• Run the new cell.

You should see precision, recall, plus F2 plotted across the available thresholds. A dashed line marks the selected operating point.

The two printed lines should show best_threshold plus its validation F2 value. The model now has an operating rule chosen without test outcomes.

Threshold Plot Missing?

Confirm that Cell 9 ran successfully. The threshold cell depends on winner_name plus winner_val_prob from that cell.

Rerun the earlier model cells if the Colab runtime was disconnected.

Ask for targeted help with diagnosing the threshold-selection cell.

Open the untouched test set

The selected threshold is now fixed. One test evaluation can measure discrimination, threshold behavior, plus calibration without feeding new information back into model selection.

• Add a code cell below the threshold-selection cell.
• Add the one-time test evaluation by copying this first section into the new cell:

# %%
# CELL 11: Open the untouched test set once and assess discrimination and calibration.
# These outputs are the final performance evidence, not another tuning opportunity.
test_prob = positive_probabilities(winner, X_test)
test_metrics = pd.DataFrame(
    [
        evaluate_probabilities(
            winner_name,
            y_test,
            test_prob,
            threshold=best_threshold,
        )
    ]
)
display(test_metrics.round(4))

test_prediction = (test_prob >= best_threshold).astype(np.int8)
confusion = pd.DataFrame(
    confusion_matrix(y_test, test_prediction, labels=[0, 1]),
    index=["Actual not within 30 days", "Actual within 30 days"],
    columns=["Predicted not flagged", "Predicted for review"],
)
display(confusion)

What Does This Evaluation Show?

• The test_metrics table applies the fixed validation threshold to the untouched test probabilities.
• The metric row includes Average Precision plus ROC-AUC.
• The same row includes recall, precision, F2, specificity, balanced accuracy, plus the Brier score.
• The confusion matrix turns the threshold results into encounter counts.

• Run the cell.

You should see one final test metric row. A labeled confusion matrix should appear below it.

Test Evaluation Did Not Run?

Confirm that the threshold cell produced best_threshold. Check that winner still contains the fitted pipeline.

Ask for help with debugging the untouched test evaluation.

Curves now add context that a metric row cannot show. The precision-recall plot reveals ranking behavior under class imbalance.

• Return to the test evaluation cell.
• Add the final test plots directly below display(confusion) by copying this continuation:

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
PrecisionRecallDisplay.from_predictions(
    y_test,
    test_prob,
    name=winner_name,
    plot_chance_level=True,
    ax=axes[0],
)
axes[0].set_title("Test precision-recall curve")

CalibrationDisplay.from_predictions(
    y_test,
    test_prob,
    n_bins=10,
    strategy="quantile",
    name=winner_name,
    ax=axes[1],
)
axes[1].set_title("Test reliability diagram")
plt.tight_layout()
plt.show()

How Do the Plots Complete the Audit?

The precision-recall curve shows how precision changes as recall changes. This view is useful when positive encounters are uncommon.

The reliability diagram compares predicted probabilities with observed outcomes across ten quantile-based bins. The Brier score in test_metrics provides a numeric probability-quality check.

• Run the completed test evaluation cell again.

You should see a test precision-recall curve beside a reliability diagram. These plots complete the one-time test evaluation.

Audit model behavior

Aggregate metrics can hide which transformed features drive a model. Feature signals expose the associations used by the selected estimator.

How Should You Read Feature Signals?

Logistic Regression coefficients have direction. Tree importances show relative influence without direction.

Both describe model associations within this historical dataset. They do not establish causal clinical effects.

• Add a code cell below the test evaluation cell.
• Add the feature-signal inspection by copying this code into the new cell:

# %%
# CELL 12: Inspect the strongest transformed feature signals.
# These are model associations and must not be presented as causal clinical effects.
feature_names = winner.named_steps["preprocess"].get_feature_names_out()
final_estimator = winner.named_steps["model"]

if hasattr(final_estimator, "coef_"):
    raw_signal = final_estimator.coef_.ravel()
    feature_signal = pd.DataFrame(
        {
            "feature": feature_names,
            "signal": raw_signal,
            "importance": np.abs(raw_signal),
            "direction": np.where(raw_signal >= 0, "higher model score", "lower model score"),
        }
    )
else:
    raw_signal = final_estimator.feature_importances_
    feature_signal = pd.DataFrame(
        {
            "feature": feature_names,
            "signal": raw_signal,
            "importance": raw_signal,
            "direction": "non-directional tree importance",
        }
    )

top_signals = feature_signal.nlargest(15, "importance")
display(top_signals)

How Are Signals Normalized?

• The preprocessor supplies the transformed feature names used by the winning estimator.
• A linear winner contributes signed coefficients.
• A tree winner contributes non-directional feature importances.
• The table keeps the fifteen strongest signals under one common importance column.

• Run the feature-signal cell.

You should see a fifteen-row table ordered by signal strength. Its direction column reflects the estimator type selected by validation evidence.

• Return to the feature-signal cell.
• Add the signal chart directly below display(top_signals) by copying this continuation:

plot_data = top_signals.sort_values("importance")
fig, ax = plt.subplots(figsize=(9, 6))
ax.barh(plot_data["feature"], plot_data["importance"], color="#59A14F")
ax.set_title(f"Strongest feature signals for {winner_name}")
ax.set_xlabel("Absolute coefficient or tree importance")
plt.tight_layout()
plt.show()

What Does This Chart Add?

Sorting the selected signals places the strongest bar at the top of the horizontal chart. The axis label remains valid for a linear winner or a tree winner.

• Run the completed feature-signal cell again.

You should see a horizontal chart containing the selected model's fifteen strongest transformed feature signals.

The final audit asks whether threshold behavior differs across sufficiently large age, race, or gender groups. These descriptive checks reveal gaps that aggregate results can conceal.

• Create Cell 13 by copying both code chunks below into one new code cell in the order shown:

def subgroup_report(feature, X_eval, y_eval, probabilities, threshold, min_size=100):
    """Calculate descriptive subgroup metrics for one original feature."""
    group_values = X_eval[feature].fillna("Missing").astype(str)
    probability_series = pd.Series(probabilities, index=X_eval.index)
    prediction_series = (probability_series >= threshold).astype(np.int8)
    rows = []

    for group_value in sorted(group_values.unique()):
        mask = group_values.eq(group_value)
        if int(mask.sum()) < min_size:
            continue

        group_y = y_eval.loc[mask]
        group_prediction = prediction_series.loc[mask]
        tn, fp, fn, tp = confusion_matrix(
            group_y, group_prediction, labels=[0, 1]
        ).ravel()
        rows.append(
            {
                "feature": feature,
                "group": group_value,
                "n": int(mask.sum()),
                "prevalence": group_y.mean(),
                "recall": tp / (tp + fn) if (tp + fn) else np.nan,
                "specificity": tn / (tn + fp) if (tn + fp) else np.nan,
                "positive_prediction_rate": group_prediction.mean(),
            }
        )

    return pd.DataFrame(rows)

What Does the Subgroup Function Measure?

• The function evaluates one original feature at a time.
• Groups with fewer than 100 test encounters are omitted.
• Each remaining row reports sample size, prevalence, recall, specificity, plus positive prediction rate.
• Every subgroup uses the same validation-selected threshold.

subgroup_tables = []
for subgroup_feature in ["age", "race", "gender"]:
    if subgroup_feature in X_test.columns:
        subgroup_tables.append(
            subgroup_report(
                subgroup_feature,
                X_test,
                y_test,
                test_prob,
                best_threshold,
            )
        )

subgroup_results = pd.concat(subgroup_tables, ignore_index=True)
display(subgroup_results.round(4))

print(f"Final recommendation: use {winner_name} at threshold {best_threshold:.3f} for this demo.")
print("Intended use: prioritize de-identified encounters for additional human review.")
print("Do not use this historical educational model for diagnosis, treatment, or real patient decisions.")
print("Before deployment, validate on recent local data, review subgroup gaps, and involve clinical and governance teams.")

Why End With Use Limitations?

The recommendation names the evidence-selected model plus its validation-selected threshold. It limits the prototype to prioritizing de-identified encounters for human review.

Operational use would require recent local validation. It would also require subgroup review plus clinical governance.

Before you run the final cell, which subgroup metric do you expect to vary most across the displayed groups?

• Run the completed Cell 13.

You should see subgroup rows for sufficiently large age, race, plus gender groups. Each row should include sample count, prevalence, recall, specificity, plus positive prediction rate.

Four final lines should name the recommended model. They should also state the selected threshold plus the educational human-review boundary.

Subgroup Table Did Not Appear?

Confirm that the test evaluation created test_prob. Check that X_test still contains the original subgroup columns.

Rerun Cell 12 if the Colab session lost the final estimator state.

Ask for help with debugging the subgroup audit.

✔️ Awesome, I've got everything!

Your notebook now contains the validation threshold, untouched test audit, feature signals, subgroup checks, plus the educational-use conclusion.

ⓧ I'd like to double check the full code

# %%
# CELL 1: Install a reproducible package set.
# Run this cell once, then choose Runtime > Restart runtime before Cell 2.
%pip install -q numpy==2.5.3 pandas==3.0.5 scipy==1.18.1 scikit-learn==1.9.0 matplotlib==3.11.1 ucimlrepo==0.0.7

# %%
# CELL 2: Import tools and verify the runtime before doing any analysis.
# The assertions stop the notebook early if its environment differs from the tested setup.
import platform
from importlib.metadata import version
from time import perf_counter

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
import sklearn
from ucimlrepo import fetch_ucirepo

from sklearn.calibration import CalibrationDisplay
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import ExtraTreesClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    PrecisionRecallDisplay,
    accuracy_score,
    average_precision_score,
    balanced_accuracy_score,
    brier_score_loss,
    confusion_matrix,
    fbeta_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import StratifiedGroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

versions = {
    "Python": platform.python_version(),
    "NumPy": np.__version__,
    "pandas": pd.__version__,
    "SciPy": scipy.__version__,
    "scikit-learn": sklearn.__version__,
    "Matplotlib": matplotlib.__version__,
    "ucimlrepo": version("ucimlrepo"),
}

expected_versions = {
    "NumPy": "2.5.3",
    "pandas": "3.0.5",
    "SciPy": "1.18.1",
    "scikit-learn": "1.9.0",
    "Matplotlib": "3.11.1",
    "ucimlrepo": "0.0.7",
}

assert platform.python_version().startswith("3.12"), versions
for package_name, expected_version in expected_versions.items():
    assert versions[package_name] == expected_version, versions

display(pd.DataFrame.from_dict(versions, orient="index", columns=["version"]))

# %%
# CELL 3: Fetch the public UCI hospital dataset and create the binary target.
# Only encounters with the original <30 outcome become positive examples.
dataset = fetch_ucirepo(id=296)

X = dataset.data.features.copy()
ids = dataset.data.ids.copy()
target_text = dataset.data.targets.iloc[:, 0].astype("string")

assert "patient_nbr" in ids.columns, "Expected patient_nbr in the UCI ID columns."
groups = ids["patient_nbr"].copy()

# Remove identifiers if a future loader also exposes them among the features.
X = X.drop(columns=["encounter_id", "patient_nbr"], errors="ignore")

# Convert the source missing marker and treat coded admission IDs as categories.
X = X.replace("?", np.nan)
coded_categories = [
    "admission_type_id",
    "discharge_disposition_id",
    "admission_source_id",
]
for column in coded_categories:
    if column in X.columns:
        X[column] = X[column].astype("string").astype("object")
X = X.where(pd.notna(X), np.nan)

y = target_text.eq("<30").astype(np.int8)
assert y.nunique() == 2, "The binary target should contain both classes."

print(f"Encounters: {len(X):,}")
print(f"Features supplied by UCI: {X.shape[1]}")
print(f"Unique patients: {groups.nunique():,}")
display(target_text.value_counts(dropna=False).rename_axis("original_label").to_frame("encounters"))

# %%
# CELL 4: Audit missingness and expose why accuracy alone is misleading.
# The all-negative prediction is a designed failure that should have zero positive recall.
missing_pct = X.isna().mean().mul(100).sort_values(ascending=False)
display(missing_pct.head(12).rename("missing_percent").to_frame())

class_counts = y.value_counts().sort_index()
fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(
    ["Not within 30 days", "Readmitted within 30 days"],
    class_counts.reindex([0, 1]).values,
    color=["#4C78A8", "#E45756"],
)
ax.set_title("30-day readmission target distribution")
ax.set_ylabel("Encounter count")
plt.xticks(rotation=10)
plt.tight_layout()
plt.show()

naive_prediction = np.zeros(len(y), dtype=np.int8)
print(f"Naive accuracy: {accuracy_score(y, naive_prediction):.3f}")
print(f"Naive readmission recall: {recall_score(y, naive_prediction, zero_division=0):.3f}")
print(f"Naive balanced accuracy: {balanced_accuracy_score(y, naive_prediction):.3f}")

# %%
# CELL 5: Split by patient while approximately preserving the class distribution.
# No patient is allowed to appear in more than one partition.
outer_splitter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
train_val_idx, test_idx = next(outer_splitter.split(X, y, groups=groups))

X_train_val = X.iloc[train_val_idx].copy()
y_train_val = y.iloc[train_val_idx].copy()
groups_train_val = groups.iloc[train_val_idx].copy()
X_test = X.iloc[test_idx].copy()
y_test = y.iloc[test_idx].copy()
groups_test = groups.iloc[test_idx].copy()

inner_splitter = StratifiedGroupKFold(n_splits=4, shuffle=True, random_state=43)
train_idx, val_idx = next(
    inner_splitter.split(X_train_val, y_train_val, groups=groups_train_val)
)

X_train = X_train_val.iloc[train_idx].copy()
y_train = y_train_val.iloc[train_idx].copy()
groups_train = groups_train_val.iloc[train_idx].copy()
X_val = X_train_val.iloc[val_idx].copy()
y_val = y_train_val.iloc[val_idx].copy()
groups_val = groups_train_val.iloc[val_idx].copy()

assert set(groups_train).isdisjoint(set(groups_val))
assert set(groups_train).isdisjoint(set(groups_test))
assert set(groups_val).isdisjoint(set(groups_test))

split_summary = pd.DataFrame(
    {
        "partition": ["train", "validation", "test"],
        "encounters": [len(X_train), len(X_val), len(X_test)],
        "patients": [groups_train.nunique(), groups_val.nunique(), groups_test.nunique()],
        "positive_rate": [y_train.mean(), y_val.mean(), y_test.mean()],
    }
)
display(split_summary)
print("Patient overlap checks passed: train, validation, and test are disjoint.")

# %%
# CELL 6: Learn the data-cleaning decisions from training data only.
# Each candidate model receives a fresh copy of the same leakage-safe preprocessor.
training_audit = pd.DataFrame(
    {
        "dtype": X_train.dtypes.astype(str),
        "missing_percent": X_train.isna().mean().mul(100),
        "unique_values": X_train.nunique(dropna=True),
    }
).sort_values("missing_percent", ascending=False)
display(training_audit.head(12))

drop_columns = training_audit.index[training_audit["missing_percent"] > 60].tolist()
X_train = X_train.drop(columns=drop_columns)
X_val = X_val.drop(columns=drop_columns)
X_test = X_test.drop(columns=drop_columns)

numeric_features = X_train.select_dtypes(include="number").columns.tolist()
categorical_features = X_train.select_dtypes(exclude="number").columns.tolist()

print(f"Dropped for training missingness > 60%: {drop_columns}")
print(f"Numeric features: {len(numeric_features)}")
print(f"Categorical features: {len(categorical_features)}")


def make_preprocessor():
    """Return a new preprocessing graph so models cannot share fitted state."""
    numeric_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]
    )
    categorical_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            (
                "onehot",
                OneHotEncoder(
                    handle_unknown="ignore",
                    min_frequency=50,
                    sparse_output=True,
                ),
            ),
        ]
    )
    return ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features),
        ],
        sparse_threshold=1.0,
    )

# %%
# CELL 7: Define one metric function and fit the majority-class sanity baseline.
# Average Precision is the selection metric, while threshold metrics explain operations.
def evaluate_probabilities(model_name, y_true, probabilities, threshold=0.5, train_seconds=np.nan):
    """Return a single comparable metric row from positive-class probabilities."""
    predictions = (probabilities >= threshold).astype(np.int8)
    tn, fp, fn, tp = confusion_matrix(y_true, predictions, labels=[0, 1]).ravel()
    specificity = tn / (tn + fp) if (tn + fp) else np.nan

    return {
        "model": model_name,
        "threshold": threshold,
        "accuracy": accuracy_score(y_true, predictions),
        "average_precision": average_precision_score(y_true, probabilities),
        "roc_auc": roc_auc_score(y_true, probabilities),
        "recall": recall_score(y_true, predictions, zero_division=0),
        "precision": precision_score(y_true, predictions, zero_division=0),
        "f2": fbeta_score(y_true, predictions, beta=2, zero_division=0),
        "specificity": specificity,
        "balanced_accuracy": balanced_accuracy_score(y_true, predictions),
        "brier_score": brier_score_loss(y_true, probabilities),
        "train_seconds": train_seconds,
    }


def positive_probabilities(fitted_pipeline, features):
    """Read the probability column that corresponds to binary class 1."""
    classes = fitted_pipeline.named_steps["model"].classes_
    positive_index = int(np.where(classes == 1)[0][0])
    return fitted_pipeline.predict_proba(features)[:, positive_index]


dummy = DummyClassifier(strategy="prior")
dummy_train_X = np.zeros((len(y_train), 1))
dummy_val_X = np.zeros((len(y_val), 1))
dummy.fit(dummy_train_X, y_train)
dummy_positive_index = int(np.where(dummy.classes_ == 1)[0][0])
dummy_val_prob = dummy.predict_proba(dummy_val_X)[:, dummy_positive_index]

validation_rows = [
    evaluate_probabilities("Dummy prior", y_val, dummy_val_prob)
]
validation_probabilities = {"Dummy prior": dummy_val_prob}

# %%
# CELL 8: Train the three real candidate models on the same training partition.
# Class weighting makes the minority outcome matter during fitting without synthetic rows.
candidate_pipelines = {
    "Logistic Regression": Pipeline(
        steps=[
            ("preprocess", make_preprocessor()),
            (
                "model",
                LogisticRegression(
                    solver="saga",
                    max_iter=300,
                    tol=1e-3,
                    class_weight="balanced",
                    random_state=42,
                ),
            ),
        ]
    ),
    "Random Forest": Pipeline(
        steps=[
            ("preprocess", make_preprocessor()),
            (
                "model",
                RandomForestClassifier(
                    n_estimators=75,
                    max_depth=12,
                    min_samples_leaf=10,
                    class_weight="balanced",
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
    "Extra Trees": Pipeline(
        steps=[
            ("preprocess", make_preprocessor()),
            (
                "model",
                ExtraTreesClassifier(
                    n_estimators=75,
                    max_depth=12,
                    min_samples_leaf=10,
                    class_weight="balanced",
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
}

fitted_models = {}
for model_name, pipeline in candidate_pipelines.items():
    started = perf_counter()
    pipeline.fit(X_train, y_train)
    elapsed = perf_counter() - started

    val_prob = positive_probabilities(pipeline, X_val)
    fitted_models[model_name] = pipeline
    validation_probabilities[model_name] = val_prob
    validation_rows.append(
        evaluate_probabilities(
            model_name,
            y_val,
            val_prob,
            threshold=0.5,
            train_seconds=elapsed,
        )
    )
    print(f"Finished {model_name} in {elapsed:.1f} seconds")

# %%
# CELL 9: Compare validation results and select the winner from evidence.
# The winner is not predetermined and the Dummy baseline cannot be selected.
validation_results = pd.DataFrame(validation_rows).sort_values(
    "average_precision", ascending=False
)
display(validation_results.round(4))

candidate_results = validation_results[
    validation_results["model"] != "Dummy prior"
]
winner_name = candidate_results.iloc[0]["model"]
winner = fitted_models[winner_name]
winner_val_prob = validation_probabilities[winner_name]

print("Selection rule: highest validation Average Precision among the three candidates.")
print(f"Selected model: {winner_name}")

# %%
# CELL 10: Choose a recall-sensitive operating threshold on validation data.
# F2 gives recall more influence than precision and keeps test data untouched.
val_precision, val_recall, thresholds = precision_recall_curve(
    y_val, winner_val_prob
)
f2_values = (
    5 * val_precision[:-1] * val_recall[:-1]
    / (4 * val_precision[:-1] + val_recall[:-1] + 1e-12)
)
best_index = int(np.nanargmax(f2_values))
best_threshold = float(thresholds[best_index])

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(thresholds, val_precision[:-1], label="Precision")
ax.plot(thresholds, val_recall[:-1], label="Recall")
ax.plot(thresholds, f2_values, label="F2")
ax.axvline(best_threshold, color="black", linestyle="--", label="Selected threshold")
ax.set_xlabel("Decision threshold")
ax.set_ylabel("Validation score")
ax.set_title(f"Threshold selection for {winner_name}")
ax.legend()
plt.tight_layout()
plt.show()

print(f"Validation-selected threshold: {best_threshold:.3f}")
print(f"Validation F2 at selected threshold: {f2_values[best_index]:.3f}")

# %%
# CELL 11: Open the untouched test set once and assess discrimination and calibration.
# These outputs are the final performance evidence, not another tuning opportunity.
test_prob = positive_probabilities(winner, X_test)
test_metrics = pd.DataFrame(
    [
        evaluate_probabilities(
            winner_name,
            y_test,
            test_prob,
            threshold=best_threshold,
        )
    ]
)
display(test_metrics.round(4))

test_prediction = (test_prob >= best_threshold).astype(np.int8)
confusion = pd.DataFrame(
    confusion_matrix(y_test, test_prediction, labels=[0, 1]),
    index=["Actual not within 30 days", "Actual within 30 days"],
    columns=["Predicted not flagged", "Predicted for review"],
)
display(confusion)

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
PrecisionRecallDisplay.from_predictions(
    y_test,
    test_prob,
    name=winner_name,
    plot_chance_level=True,
    ax=axes[0],
)
axes[0].set_title("Test precision-recall curve")

CalibrationDisplay.from_predictions(
    y_test,
    test_prob,
    n_bins=10,
    strategy="quantile",
    name=winner_name,
    ax=axes[1],
)
axes[1].set_title("Test reliability diagram")
plt.tight_layout()
plt.show()

# %%
# CELL 12: Inspect the strongest transformed feature signals.
# These are model associations and must not be presented as causal clinical effects.
feature_names = winner.named_steps["preprocess"].get_feature_names_out()
final_estimator = winner.named_steps["model"]

if hasattr(final_estimator, "coef_"):
    raw_signal = final_estimator.coef_.ravel()
    feature_signal = pd.DataFrame(
        {
            "feature": feature_names,
            "signal": raw_signal,
            "importance": np.abs(raw_signal),
            "direction": np.where(raw_signal >= 0, "higher model score", "lower model score"),
        }
    )
else:
    raw_signal = final_estimator.feature_importances_
    feature_signal = pd.DataFrame(
        {
            "feature": feature_names,
            "signal": raw_signal,
            "importance": raw_signal,
            "direction": "non-directional tree importance",
        }
    )

top_signals = feature_signal.nlargest(15, "importance")
display(top_signals)

plot_data = top_signals.sort_values("importance")
fig, ax = plt.subplots(figsize=(9, 6))
ax.barh(plot_data["feature"], plot_data["importance"], color="#59A14F")
ax.set_title(f"Strongest feature signals for {winner_name}")
ax.set_xlabel("Absolute coefficient or tree importance")
plt.tight_layout()
plt.show()

# %%
# CELL 13: Check performance across sensitive subgroups and print the conclusion.
# Small groups are omitted because their metrics are too unstable for this short audit.
def subgroup_report(feature, X_eval, y_eval, probabilities, threshold, min_size=100):
    """Calculate descriptive subgroup metrics for one original feature."""
    group_values = X_eval[feature].fillna("Missing").astype(str)
    probability_series = pd.Series(probabilities, index=X_eval.index)
    prediction_series = (probability_series >= threshold).astype(np.int8)
    rows = []

    for group_value in sorted(group_values.unique()):
        mask = group_values.eq(group_value)
        if int(mask.sum()) < min_size:
            continue

        group_y = y_eval.loc[mask]
        group_prediction = prediction_series.loc[mask]
        tn, fp, fn, tp = confusion_matrix(
            group_y, group_prediction, labels=[0, 1]
        ).ravel()
        rows.append(
            {
                "feature": feature,
                "group": group_value,
                "n": int(mask.sum()),
                "prevalence": group_y.mean(),
                "recall": tp / (tp + fn) if (tp + fn) else np.nan,
                "specificity": tn / (tn + fp) if (tn + fp) else np.nan,
                "positive_prediction_rate": group_prediction.mean(),
            }
        )

    return pd.DataFrame(rows)

subgroup_tables = []
for subgroup_feature in ["age", "race", "gender"]:
    if subgroup_feature in X_test.columns:
        subgroup_tables.append(
            subgroup_report(
                subgroup_feature,
                X_test,
                y_test,
                test_prob,
                best_threshold,
            )
        )

subgroup_results = pd.concat(subgroup_tables, ignore_index=True)
display(subgroup_results.round(4))

print(f"Final recommendation: use {winner_name} at threshold {best_threshold:.3f} for this demo.")
print("Intended use: prioritize de-identified encounters for additional human review.")
print("Do not use this historical educational model for diagnosis, treatment, or real patient decisions.")
print("Before deployment, validate on recent local data, review subgroup gaps, and involve clinical and governance teams.")

You have completed the full model audit. The notebook now supports a validation-based recommendation while keeping every real-patient decision with qualified humans.

Secret mission

Score a What-If Encounter

Turn one real de-identified test encounter into a controlled what-if score. Change only approved fields. Return the risk score with its review flag.

Clean Up Your Resources

Clean Up Your Resources

This project costs $0 because it uses free Google Colab compute. Choose whether to keep the notebook, disconnect its temporary runtime, or delete the notebook entirely.

Resources you used:

• One Diabetes_Readmission_Risk.ipynb notebook in Google Drive if you saved it.
• One temporary Colab runtime containing the six pinned packages.
• A temporary copy of the public hospital dataset from the UCI Machine Learning Repository.
• The fitted in-memory models plus the selected threshold.

Keep everything running

No action needed. Choose this if you want to continue working with the readmission-risk notebook.

• Save Diabetes_Readmission_Risk.ipynb to Google Drive if it is not already saved.
• Keep the executed cell outputs inside the saved notebook.
• Continue using the connected Colab runtime while you are actively working.

Your notebook remains available with its charts plus model evidence. The free runtime may still end after a period of inactivity.

Pause - I'll come back to this later

Disconnect the runtime to release its temporary compute. Your saved notebook remains available in Google Drive.

• Switch back to the configured Google Colab notebook from earlier.
• Use Colab's runtime controls to disconnect the active runtime.
• Keep Diabetes_Readmission_Risk.ipynb in Google Drive.

The notebook remains available in Google Drive. Rerunning it recreates the pinned packages plus the temporary analysis state.

Delete - I don't want to use this again

Deleting the notebook removes your saved project artifact. The public dataset remains available if you decide to rebuild the project.

• Return to Google Drive from earlier if you saved the notebook.
• Locate Diabetes_Readmission_Risk.ipynb in Google Drive.
• Select Diabetes_Readmission_Risk.ipynb.
• Use Google Drive's deletion control to remove the selected notebook.
• Permanently remove the notebook from Google Drive's deleted-items area.
• Switch back to the configured Google Colab notebook from earlier.
• Use Colab's runtime controls to disconnect the active runtime.
• Close the notebook tab.

The notebook no longer appears in Google Drive. The runtime's temporary state disappears when the session ends.

Nice Work!

Nice Work!

You did it! Your Google Colab notebook now turns de-identified hospital encounter data into a leakage-aware 30-day readmission screening prototype that supports human review.

You've learned how to:

• Expose the accuracy trap by showing how a majority-only prediction produces zero recall for 30-day readmissions.
• Create patient-grouped partitions that prevent repeat-patient leakage. Apply training-only preprocessing pipelines to protect validation data plus test data.
• Compare Logistic Regression, Random Forest, plus Extra Trees against a majority baseline. Select the winner using validation Average Precision. Tune its operating threshold with F2. Audit its test performance through calibration evidence, feature signals, plus subgroup checks.
• Secret Mission: Build a controlled what-if scorer from a real test encounter. Restrict overrides to five allow-listed fields. Return the risk score with its threshold-based review flag.

Ready to quiz yourself? Test what you remember.