ML Pipeline for Readmission Prediction

Build and deploy an end-to-end ML pipeline on real hospital data.

Introduction

30 Second Summary

When a patient leaves the hospital, their care team hopes they won't need to come back. But for millions of patients each year, an unplanned readmission happens within 30 days, and predicting which patients are most at risk lets hospitals intervene early.

In this project, you will build a production-style machine learning pipeline that ingests real hospital data, validates data quality, engineers clinical features, trains and compares three classification models, and deploys an interactive dashboard to the cloud.

What You'll Build

A deployed ML application that validates 101,766 real clinical records, trains Logistic Regression, Random Forest, and XGBoost models, tracks experiments with full reproducibility metadata, and presents everything through a multi-page Streamlit dashboard deployed to Streamlit Community Cloud.

By the end of this project, you'll have:

  • A data ingestion pipeline with automated quality checks validating 101,766 real hospital records from the UCI Diabetes 130-US Hospitals dataset.
  • A model comparison dashboard deployed live on Streamlit Community Cloud showing performance metrics across three classification models.
  • A reproducible experiment tracking system logging model versions, hyperparameters, evaluation metrics, and training timestamps.
  • Secret Mission: Add SHAP-based feature importance analysis with interactive visualizations comparing explainability across all three models.

Are there any prerequisites?

You should have basic Python knowledge and familiarity with ML concepts like classification, train-test split, and evaluation metrics. No prior experience with Streamlit or XGBoost is needed.

Set Up the Project

Before you can build an ML pipeline, you need a clean, isolated workspace where your Python packages won't clash with anything else on your system.

In this step, you'll create a project folder, install all the libraries your pipeline needs, scaffold the directory structure, and download the real hospital dataset you'll work with throughout this project.

In this step, get ready to:
  • Create a virtual environment and install all project dependencies.
  • Set up the project folder structure and initialize Git.
  • Download and verify the UCI Diabetes 130-US Hospitals dataset.
Create a virtual environment and install dependencies

A virtual environment keeps your project's packages separate from your system Python. This prevents version conflicts when different projects need different library versions.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code (or Cursor if you prefer) and press Enter to open it.
  • Open a terminal inside your editor by pressing Ctrl+` (backtick key, same shortcut on macOS and Windows).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir readmission-pipeline
cd readmission-pipeline
  • Create a virtual environment by running this command:
python -m venv venv

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. When activated, any packages you install go into the venv/ directory instead of your global Python.

This means you can pin exact library versions for this project without affecting other Python work on your machine.

  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

Make sure you ran the correct activation command for your operating system. On macOS, it starts with source. On Windows, you use the backslash path.

If you see a permissions error on Windows, try running PowerShell as Administrator and executing Set-ExecutionPolicy RemoteSigned first.

help me activate my Python virtual environment

  • Create the requirements.txt file that lists all project dependencies. In the Explorer sidebar, right-click on the readmission-pipeline folder and select New File.
  • Name the file requirements.txt and paste the following contents:
streamlit==1.58.0
scikit-learn==1.9.0
xgboost==3.2.0
pandas==2.2.3
numpy==1.26.4
matplotlib==3.9.4
seaborn==0.13.2
shap==0.52.0
ucimlrepo==0.0.7
joblib==1.4.2

Why pin exact versions?

Each line uses == to lock a specific version. This guarantees that anyone who runs your project gets the exact same library versions, eliminating "works on my machine" issues.

This is especially important for ML projects where a minor version change in scikit-learn or xgboost can change model outputs.

  • Install all the dependencies by running this command:
pip install -r requirements.txt

This will download and install all ten packages plus their dependencies. It may take a minute or two depending on your internet speed.

  • Verify that all key packages installed correctly by running:
python -c "import streamlit; import sklearn; import xgboost; print('All packages installed successfully')"

You should see All packages installed successfully printed to your terminal.

Seeing an import error?

Make sure your virtual environment is active. You should see (venv) in your terminal prompt. If not, rerun the activation command.

If a specific package failed to install, try running pip install -r requirements.txt again. Network timeouts sometimes interrupt downloads.

help me fix package installation errors in my Python virtual environment

Set up the project structure

A well-organized folder structure makes your ML project maintainable. Each folder has a clear responsibility: source code, raw data, trained models, experiment logs, and the dashboard app.

  • Create all project folders by running these commands:
mkdir src data data/raw data/processed data/plots models experiments app

What does each folder do?

  • src/ holds your Python modules (data quality, feature engineering, training).
  • data/ stores raw, processed, and plot outputs.
  • models/ stores trained model files.
  • experiments/ stores experiment logs and reproducibility manifests.
  • app/ holds the Streamlit dashboard code.
  • Confirm the folders exist by running:
ls

You should see app, data, experiments, models, requirements.txt, src, and venv listed in the output.

  • Initialize a Git repository by running:
git init
  • Create a .gitignore file to keep large and generated files out of version control. In the Explorer sidebar, right-click on the readmission-pipeline folder and select New File.
  • Name the file .gitignore and paste these contents:
venv/
data/
models/*.pkl
__pycache__/

Why exclude these from Git?

The venv/ folder contains hundreds of megabytes of installed packages that anyone can regenerate from requirements.txt. The data/ folder will hold a large CSV that should be downloaded fresh. The models/*.pkl pattern excludes trained model binaries.

Keeping these out of Git makes your repository small, fast to clone, and safe to share publicly.

  • Save .gitignore.
Download and verify the dataset

Your pipeline will use the Diabetes 130-US Hospitals dataset from the UCI Machine Learning Repository. This is a real clinical dataset with over 100,000 hospital encounters recording patient demographics, diagnoses, medications, and readmission outcomes.

  • Download the dataset by running this Python script in your terminal:
python -c "
from ucimlrepo import fetch_ucirepo
import pandas as pd
from pathlib import Path

# Fetch the Diabetes 130-US Hospitals dataset (ID 296)
print('Downloading dataset from UCI ML Repository...')
dataset = fetch_ucirepo(id=296)

# Access the full dataset (IDs + features + targets)
df = dataset.data.original

# Save to data/raw/
Path('data/raw').mkdir(parents=True, exist_ok=True)
df.to_csv('data/raw/diabetic_data.csv', index=False)

# Verify the download
print(f'Dataset shape: {df.shape}')
print(f'Rows: {df.shape[0]:,}')
print(f'Columns: {df.shape[1]}')
print('Dataset saved to data/raw/diabetic_data.csv')
"

You should see output confirming 101,766 rows and 50 columns.

What is this dataset?

The UCI Diabetes 130-US Hospitals dataset contains 10 years of clinical care records from 130 US hospitals. Each row represents a single hospital encounter for a diabetic patient.

The target variable is readmitted, which records whether the patient was readmitted within 30 days, after 30 days, or not at all. Your ML pipeline will predict 30-day readmission risk.

Download taking too long or failing?

The dataset is about 18 MB. On slower connections, it can take a few minutes to download. Make sure your internet connection is stable.

If you see an SSL error, try upgrading your certificates by running pip install --upgrade certifi and then rerun the download script.

help me download the UCI diabetes dataset

✔️ Awesome, I've got everything!

Great. Double check you've saved both requirements.txt and .gitignore.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt:

streamlit==1.58.0
scikit-learn==1.9.0
xgboost==3.2.0
pandas==2.2.3
numpy==1.26.4
matplotlib==3.9.4
seaborn==0.13.2
shap==0.52.0
ucimlrepo==0.0.7
joblib==1.4.2

Here is the complete .gitignore:

venv/
data/
models/*.pkl
__pycache__/

Your project foundation is solid. Next up, you'll build a data quality validation module that checks your raw dataset for issues before any modeling begins.

Ingest Data and Run Quality Checks

You have 101,766 rows of raw clinical data sitting in your project folder. Before you can train any model on it, you need to know whether the data is trustworthy.

In this step, you will build a data quality module that validates the raw dataset, then clean the data by removing irrelevant rows and converting the target variable into a format suitable for binary classification.

In this step, get ready to:
  • Build a data quality validation module with schema, missing value, duplicate, and class distribution checks.
  • Clean the raw data by removing invalid rows and creating a binary target variable.
  • Save the cleaned dataset and verify the class distribution.
Build a data quality validation module

Real-world datasets are messy. Columns go missing, values get encoded as placeholders, and class distributions can be wildly skewed. A dedicated quality module catches these problems before they silently corrupt your model.

  • Create a new file called data_quality.py inside your src/ folder.
  • Add the imports and first two validation functions by pasting this code:
import pandas as pd
import numpy as np


def validate_schema(df, expected_columns):
    missing_cols = set(expected_columns) - set(df.columns)
    extra_cols = set(df.columns) - set(expected_columns)
    return {
        "schema_valid": len(missing_cols) == 0,
        "missing_columns": list(missing_cols),
        "extra_columns": list(extra_cols),
        "total_columns": len(df.columns),
    }


def check_missing_values(df):
    missing = df.isin(["?"]).sum() + df.isna().sum()
    missing_pct = (missing / len(df) * 100).round(2)
    report = pd.DataFrame({"missing_count": missing, "missing_pct": missing_pct})
    return report[report["missing_count"] > 0].sort_values("missing_pct", ascending=False)

What does this code do?

These two functions form the core of your quality checks.

  • validate_schema() compares the DataFrame's columns against an expected list. It reports any missing or extra columns so you catch structural drift immediately.
  • check_missing_values() counts both standard NaN values and ? placeholders (which this dataset uses instead of blank cells). It returns a sorted report showing which columns have the most gaps.
  • Add the remaining functions below the code you just pasted:
def check_duplicates(df, id_column):
    duplicates = df[id_column].duplicated().sum()
    return {"duplicate_count": duplicates, "is_unique": duplicates == 0}


def check_class_distribution(df, target_column):
    distribution = df[target_column].value_counts(normalize=True).round(4).to_dict()
    return distribution


def run_quality_checks(df, expected_columns, id_column, target_column):
    report = {
        "schema": validate_schema(df, expected_columns),
        "missing_values": check_missing_values(df).to_dict(),
        "duplicates": check_duplicates(df, id_column),
        "class_distribution": check_class_distribution(df, target_column),
        "total_rows": len(df),
        "total_columns": len(df.columns),
    }
    return report

What does this code do?

These three functions complete your quality toolkit.

  • check_duplicates() flags repeated encounter IDs so you don't accidentally double-count patients.
  • check_class_distribution() shows the proportion of each target class. This is critical for detecting class imbalance before training.
  • run_quality_checks() is the orchestrator. It calls all four checks and returns a single report dictionary summarizing the dataset's health.
  • Save src/data_quality.py.

Seeing an import error?

Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt. If not, activate it again before running any Python code.

ask

✔️ Awesome, I've got everything!

Great. Double check you have saved src/data_quality.py before continuing.

ⓧ I'd like to double check the full code

import pandas as pd
import numpy as np


def validate_schema(df, expected_columns):
    missing_cols = set(expected_columns) - set(df.columns)
    extra_cols = set(df.columns) - set(expected_columns)
    return {
        "schema_valid": len(missing_cols) == 0,
        "missing_columns": list(missing_cols),
        "extra_columns": list(extra_cols),
        "total_columns": len(df.columns),
    }


def check_missing_values(df):
    missing = df.isin(["?"]).sum() + df.isna().sum()
    missing_pct = (missing / len(df) * 100).round(2)
    report = pd.DataFrame({"missing_count": missing, "missing_pct": missing_pct})
    return report[report["missing_count"] > 0].sort_values("missing_pct", ascending=False)


def check_duplicates(df, id_column):
    duplicates = df[id_column].duplicated().sum()
    return {"duplicate_count": duplicates, "is_unique": duplicates == 0}


def check_class_distribution(df, target_column):
    distribution = df[target_column].value_counts(normalize=True).round(4).to_dict()
    return distribution


def run_quality_checks(df, expected_columns, id_column, target_column):
    report = {
        "schema": validate_schema(df, expected_columns),
        "missing_values": check_missing_values(df).to_dict(),
        "duplicates": check_duplicates(df, id_column),
        "class_distribution": check_class_distribution(df, target_column),
        "total_rows": len(df),
        "total_columns": len(df.columns),
    }
    return report
Clean and prepare the raw data

The raw dataset contains rows and columns that would mislead a readmission model. Patients who died or transferred to hospice cannot be readmitted, so including them adds noise. Columns like weight are 97% empty and provide no signal.

  • Run the quality checks and clean the data by running this script in your terminal:
python -c "
import pandas as pd
import numpy as np
import sys
sys.path.insert(0, '.')
from src.data_quality import run_quality_checks

# Load the raw dataset
df = pd.read_csv('data/raw/diabetic_data.csv')
print(f'Raw data shape: {df.shape}')

# Run quality checks
expected_cols = df.columns.tolist()
report = run_quality_checks(df, expected_cols, 'encounter_id', 'readmitted')
print(f'\nSchema valid: {report[\"schema\"][\"schema_valid\"]}')
print(f'Duplicate encounters: {report[\"duplicates\"][\"duplicate_count\"]}')
print(f'Class distribution: {report[\"class_distribution\"]}')

# Clean the data
df.replace('?', np.nan, inplace=True)

# Drop columns with excessive missing values
df.drop(columns=['weight', 'payer_code', 'medical_specialty'], inplace=True)

# Remove rows where patient died or went to hospice
exclude_ids = [11, 13, 14, 19, 20, 21]
df = df[~df['discharge_disposition_id'].isin(exclude_ids)]

# Convert readmitted to binary: 1 if <30 days, 0 otherwise
df['readmitted_binary'] = (df['readmitted'] == '<30').astype(int)
df.drop(columns=['readmitted'], inplace=True)

print(f'\nCleaned data shape: {df.shape}')
print(f'Readmission rate (positive class): {df[\"readmitted_binary\"].mean():.2%}')

# Save cleaned data
df.to_csv('data/processed/cleaned_data.csv', index=False)
print('\nSaved cleaned data to data/processed/cleaned_data.csv')
"

What does this script do?

This script runs your quality module against the raw data, then applies four cleaning transformations.

  • It replaces all ? placeholders with proper NaN values so pandas can handle them correctly.
  • It drops weight, payer_code, and medical_specialty because they have too many missing values to be useful.
  • It removes rows where discharge_disposition_id indicates death or hospice transfer. These patients cannot be readmitted, so they do not belong in a readmission prediction dataset.
  • It converts the multiclass readmitted column into a binary readmitted_binary column (1 if readmitted within 30 days, 0 otherwise).
  • Confirm the script completed successfully. You should see output showing the raw data shape, the quality check results, and the cleaned data stats.

Seeing a FileNotFoundError?

Make sure your terminal is in the project's root folder (the one containing src/ and data/). The script looks for data/raw/diabetic_data.csv relative to where you run it.

Help me fix a FileNotFoundError when loading diabetic_data.csv

Verify the cleaned dataset

The cleaning step removed patients who cannot be readmitted and dropped columns that are mostly empty. Your dataset should now be smaller and focused on valid readmission candidates.

  • Verify the cleaned file exists by running:
ls data/processed/
  • You should see cleaned_data.csv in the output.
  • Confirm the class distribution by running:
python -c "import pandas as pd; df = pd.read_csv('data/processed/cleaned_data.csv'); print(f'Shape: {df.shape}'); print(f'Positive class rate: {df[\"readmitted_binary\"].mean():.2%}')"
  • You should see a positive class rate of approximately 11%. This confirms the binary target conversion worked correctly and that the class imbalance is real.

Why does 11% matter?

A naive model that always predicts "not readmitted" would be right 89% of the time. Accuracy alone is misleading here. This is why you will use metrics like AUC-ROC and PR-AUC in later steps to properly evaluate your models against this imbalanced target.

The class_weight='balanced' parameter you will set during training helps models pay more attention to the rare positive class.

Positive class rate seems wrong?

If you see 0% or a very different number, check that the cleaning script ran without errors. The readmitted_binary column should contain 1s and 0s. A common issue is running the verification command from a different directory than where the file was saved.

Help me debug my cleaned dataset's class distribution

Your data pipeline foundation is solid. You have a validated, cleaned dataset ready for feature engineering. Next up, you will transform these raw clinical columns into model-ready features and explore the data visually.

Engineer Features and Explore the Data

Your cleaned dataset is ready, but raw clinical columns are not yet suitable for machine learning models. Models need numeric inputs, and many of the raw fields (like age brackets or diagnosis codes) are either categorical strings or high-cardinality identifiers that add noise.

In this step, you will transform the cleaned data into model-ready features, explore the dataset visually, and split it into training and testing sets. This is the bridge between data cleaning and model training.

In this step, get ready to:
  • Engineer clinical features and encode categorical variables.
  • Generate exploratory data analysis visualizations.
  • Split the data into training and testing sets with stratification.
Engineer clinical features

Machine learning models work best when raw data is transformed into meaningful signals. For hospital readmission prediction, combining related visit counts and flagging medication changes captures clinical patterns better than raw columns alone.

  • Create a new file called src/feature_engineering.py in your src/ folder.
  • Add the clean_data() function by copying the following code into the file:
import pandas as pd
import numpy as np


def clean_data(df):
    df = df.copy()
    df.replace("?", np.nan, inplace=True)

    cols_to_drop = ["weight", "payer_code", "medical_specialty"]
    df.drop(columns=[c for c in cols_to_drop if c in df.columns], inplace=True)

    exclude_ids = [11, 13, 14, 19, 20, 21]
    df = df[~df["discharge_disposition_id"].isin(exclude_ids)]

    df["readmitted_binary"] = (df["readmitted"] == "<30").astype(int)
    df.drop(columns=["readmitted"], inplace=True)

    return df

What does this function do?

The clean_data() function encapsulates all the cleaning logic from the previous step into a reusable function. This makes the pipeline reproducible.

  • Replaces ? placeholders with NaN for proper missing value handling.
  • Drops columns with excessive missing values (weight, payer_code, medical_specialty).
  • Removes rows where patients died or went to hospice (readmission is not applicable).
  • Converts the multiclass readmitted column into a binary target: 1 if readmitted within 30 days, 0 otherwise.
  • Now add the engineer_features() function below clean_data() in the same file:
def engineer_features(df):
    df = df.copy()

    df["total_visits"] = (
        df["number_outpatient"] + df["number_emergency"] + df["number_inpatient"]
    )
    df["medication_change"] = (df["change"] == "Ch").astype(int)
    df["num_medications_bucket"] = pd.cut(
        df["num_medications"], bins=[0, 5, 10, 20, 50, 100], labels=[1, 2, 3, 4, 5]
    ).astype(float)

    drop_cols = [
        "encounter_id",
        "patient_nbr",
        "diag_1",
        "diag_2",
        "diag_3",
        "change",
    ]
    df.drop(columns=[c for c in drop_cols if c in df.columns], inplace=True)

    categorical_cols = df.select_dtypes(include=["object"]).columns.tolist()
    df = pd.get_dummies(df, columns=categorical_cols, drop_first=True)

    return df

What does this function do?

The engineer_features() function transforms raw clinical data into model-ready numeric features.

  • Creates total_visits by summing outpatient, emergency, and inpatient visits. This captures overall healthcare utilization.
  • Creates medication_change as a binary flag indicating whether any medication was changed during the encounter.
  • Bins num_medications into 5 buckets to reduce noise from the raw count.
  • Drops ID columns and high-cardinality columns that would add noise without predictive value.
  • One-hot encodes all remaining categorical columns using pd.get_dummies() with drop_first=True to avoid multicollinearity.
  • Save src/feature_engineering.py.

Seeing a NameError or ImportError?

  • Make sure both import pandas as pd and import numpy as np are at the very top of the file.
  • Confirm your virtual environment is activated. You should see (venv) in your terminal prompt.

Why isn't my feature engineering module importing correctly?

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before moving on.

ⓧ I'd like to double check the full code

import pandas as pd
import numpy as np


def clean_data(df):
    df = df.copy()
    df.replace("?", np.nan, inplace=True)

    cols_to_drop = ["weight", "payer_code", "medical_specialty"]
    df.drop(columns=[c for c in cols_to_drop if c in df.columns], inplace=True)

    exclude_ids = [11, 13, 14, 19, 20, 21]
    df = df[~df["discharge_disposition_id"].isin(exclude_ids)]

    df["readmitted_binary"] = (df["readmitted"] == "<30").astype(int)
    df.drop(columns=["readmitted"], inplace=True)

    return df


def engineer_features(df):
    df = df.copy()

    df["total_visits"] = (
        df["number_outpatient"] + df["number_emergency"] + df["number_inpatient"]
    )
    df["medication_change"] = (df["change"] == "Ch").astype(int)
    df["num_medications_bucket"] = pd.cut(
        df["num_medications"], bins=[0, 5, 10, 20, 50, 100], labels=[1, 2, 3, 4, 5]
    ).astype(float)

    drop_cols = [
        "encounter_id",
        "patient_nbr",
        "diag_1",
        "diag_2",
        "diag_3",
        "change",
    ]
    df.drop(columns=[c for c in drop_cols if c in df.columns], inplace=True)

    categorical_cols = df.select_dtypes(include=["object"]).columns.tolist()
    df = pd.get_dummies(df, columns=categorical_cols, drop_first=True)

    return df
Generate EDA visualizations

Before training models, you need to understand what the data looks like. Exploratory data analysis reveals class imbalance, feature correlations, and patterns that inform modelling decisions.

  • Create a new file called src/eda.py in your src/ folder.
  • Add the following code to generate four EDA plots and save them to data/plots/:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from pathlib import Path

# Load the cleaned data
data = pd.read_csv("data/processed/cleaned_data.csv")

# Create output directory for plots
Path("data/plots").mkdir(parents=True, exist_ok=True)

# Plot 1: Class distribution (readmitted vs not)
fig, ax = plt.subplots(figsize=(8, 5))
data["readmitted_binary"].value_counts().plot(kind="bar", ax=ax, color=["steelblue", "coral"])
ax.set_xticklabels(["Not Readmitted (0)", "Readmitted <30d (1)"], rotation=0)
ax.set_ylabel("Count")
ax.set_title("Target Variable Distribution")
plt.tight_layout()
plt.savefig("data/plots/class_distribution.png")
plt.close()

# Plot 2: Correlation heatmap of top 15 numeric features
numeric_cols = data.select_dtypes(include=[np.number]).columns.tolist()
corr_matrix = data[numeric_cols].corr()
top_corr_cols = corr_matrix["readmitted_binary"].abs().nlargest(15).index.tolist()
fig, ax = plt.subplots(figsize=(12, 10))
sns.heatmap(data[top_corr_cols].corr(), annot=True, fmt=".2f", cmap="coolwarm", ax=ax)
ax.set_title("Correlation Heatmap - Top 15 Features")
plt.tight_layout()
plt.savefig("data/plots/correlation_heatmap.png")
plt.close()

# Plot 3: Readmission rate by age group
age_readmit = data.groupby("age")["readmitted_binary"].mean().sort_index()
fig, ax = plt.subplots(figsize=(10, 5))
age_readmit.plot(kind="bar", ax=ax, color="steelblue")
ax.set_ylabel("Readmission Rate")
ax.set_title("Readmission Rate by Age Group")
ax.set_xlabel("Age Group")
plt.xticks(rotation=45)
plt.tight_layout()
plt.savefig("data/plots/readmission_by_age.png")
plt.close()

# Plot 4: Readmission rate by admission type
admission_readmit = data.groupby("admission_type_id")["readmitted_binary"].mean()
fig, ax = plt.subplots(figsize=(8, 5))
admission_readmit.plot(kind="bar", ax=ax, color="coral")
ax.set_ylabel("Readmission Rate")
ax.set_title("Readmission Rate by Admission Type")
ax.set_xlabel("Admission Type ID")
plt.tight_layout()
plt.savefig("data/plots/readmission_by_admission_type.png")
plt.close()

print("All EDA plots saved to data/plots/")

What do these plots reveal?

  • The class distribution chart shows the severe imbalance (approximately 11% positive class). This tells you that accuracy alone is a poor metric.
  • The correlation heatmap highlights which numeric features have the strongest linear relationship with readmission.
  • The age and admission type plots reveal whether certain patient groups face higher readmission risk.
  • Save src/eda.py.
  • Run the EDA script from your project root by running this command:
python src/eda.py
  • You should see All EDA plots saved to data/plots/ printed in your terminal.
  • Confirm the plots exist by running:
ls data/plots/
  • You should see class_distribution.png, correlation_heatmap.png, readmission_by_age.png, and readmission_by_admission_type.png listed.

Seeing a FileNotFoundError?

  • Make sure you are running the command from your project root directory (the folder containing src/ and data/).
  • Verify that data/processed/cleaned_data.csv exists from the previous step.

Why is my EDA script failing to find the cleaned data file?

Prepare the train-test split

Models need separate training and testing datasets to evaluate how well they generalize to unseen data. Stratification ensures both sets maintain the same class distribution (approximately 11% positive) so your evaluation is fair.

  • Create a new file called src/prepare_split.py in your src/ folder and add the following code:
import pandas as pd
from pathlib import Path
from sklearn.model_selection import train_test_split
import sys

# Add project root to path for imports
sys.path.insert(0, ".")
from src.feature_engineering import engineer_features

# Load the cleaned dataset
data = pd.read_csv("data/processed/cleaned_data.csv")
print(f"Loaded cleaned data: {data.shape}")

# Apply feature engineering
data_engineered = engineer_features(data)
print(f"After feature engineering: {data_engineered.shape}")

# Separate features and target
X = data_engineered.drop(columns=["readmitted_binary"])
y = data_engineered["readmitted_binary"]

# Split 80/20 with stratification on the target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Save splits to CSV
Path("data/processed").mkdir(parents=True, exist_ok=True)
X_train.to_csv("data/processed/X_train.csv", index=False)
X_test.to_csv("data/processed/X_test.csv", index=False)
y_train.to_csv("data/processed/y_train.csv", index=False)
y_test.to_csv("data/processed/y_test.csv", index=False)

# Print shapes and class distributions to confirm stratification
print(f"\nX_train shape: {X_train.shape}")
print(f"X_test shape: {X_test.shape}")
print(f"\ny_train class distribution:")
print(y_train.value_counts(normalize=True).round(4))
print(f"\ny_test class distribution:")
print(y_test.value_counts(normalize=True).round(4))

Why stratify on the target variable?

With only about 11% positive cases, a random split might put 8% in training and 14% in testing (or vice versa). Stratification forces both sets to have the same 11% positive rate.

The random_state=42 parameter ensures the split is reproducible. Anyone running this code gets the exact same training and testing sets.

  • Save src/prepare_split.py.
  • Run the split script from your project root by running this command:
python src/prepare_split.py
  • You should see the shapes printed for both training and testing sets, along with class distributions confirming that both sets have approximately 11% positive class (value of 1).
  • Confirm the split files exist by running:
ls data/processed/
  • You should see X_train.csv, X_test.csv, y_train.csv, and y_test.csv alongside cleaned_data.csv.

Class distributions don't match?

  • If both distributions show approximately 0.89 for class 0 and 0.11 for class 1, stratification worked correctly.
  • If you see a ModuleNotFoundError for src.feature_engineering, make sure you are running the command from the project root (not from inside the src/ folder).

Why is my train-test split script failing with an import error?

Your data pipeline is now complete. Next up, you will train three classification models on these features and compare their performance on the imbalanced readmission prediction task.

Train and Compare Models

Your feature-engineered dataset is ready, split into training and test sets with stratification to preserve the class balance. Now it's time to train models that can predict which patients are likely to be readmitted within 30 days.

With only ~11% of patients in the positive class, a naive model could score 89% accuracy by always predicting "not readmitted." You need models that handle this imbalance and metrics that reveal true performance on the minority class.

In this step, get ready to:
  • Train three classification models with imbalance-handling strategies.
  • Evaluate each model with AUC-ROC, PR-AUC, F1, precision, and recall.
  • Build a comparison summary and identify the best performer.
Train three classification models

Each model uses a different strategy for handling the ~11% positive class rate. You will train a Logistic Regression, a Random Forest, and an XGBoost classifier, each configured to account for class imbalance.

  • Create a new file called src/train.py and add the following imports:
import time
import json
import hashlib
from pathlib import Path

import pandas as pd
import numpy as np
import joblib
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    roc_auc_score,
    average_precision_score,
    f1_score,
    precision_score,
    recall_score,
    classification_report,
)
from xgboost import XGBClassifier

What do these imports do?

  • time tracks how long each model takes to train.
  • joblib serializes trained model objects to disk as .pkl files.
  • LogisticRegression, RandomForestClassifier, and XGBClassifier are the three model classes you will compare.
  • The metrics imports (roc_auc_score, average_precision_score, etc.) calculate evaluation scores for each model's predictions.
  • Below the imports, add the train_and_evaluate function that trains all three models and evaluates them:
def train_and_evaluate(X_train, X_test, y_train, y_test, models_dir="models"):
    Path(models_dir).mkdir(exist_ok=True)

    pos_count = y_train.sum()
    neg_count = len(y_train) - pos_count
    scale_pos_weight = neg_count / pos_count

    models = {
        "logistic_regression": LogisticRegression(
            class_weight="balanced", max_iter=1000, random_state=42
        ),
        "random_forest": RandomForestClassifier(
            n_estimators=100, class_weight="balanced", random_state=42, n_jobs=-1
        ),
        "xgboost": XGBClassifier(
            n_estimators=100,
            scale_pos_weight=scale_pos_weight,
            random_state=42,
            eval_metric="logloss",
            verbosity=0,
        ),
    }

    results = []

    for name, model in models.items():
        start_time = time.time()
        model.fit(X_train, y_train)
        duration = time.time() - start_time

        y_pred = model.predict(X_test)
        y_prob = model.predict_proba(X_test)[:, 1]

        metrics = {
            "auc_roc": round(roc_auc_score(y_test, y_prob), 4),
            "pr_auc": round(average_precision_score(y_test, y_prob), 4),
            "f1": round(f1_score(y_test, y_pred), 4),
            "precision": round(precision_score(y_test, y_pred), 4),
            "recall": round(recall_score(y_test, y_pred), 4),
        }

        model_path = f"{models_dir}/{name}.pkl"
        joblib.dump(model, model_path)

        results.append(
            {
                "model_name": name,
                "metrics": metrics,
                "training_duration_seconds": round(duration, 2),
                "model_path": model_path,
            }
        )

        print(f"\n{name}:")
        print(f"  AUC-ROC: {metrics['auc_roc']}")
        print(f"  PR-AUC: {metrics['pr_auc']}")
        print(f"  F1: {metrics['f1']}")
        print(f"  Training time: {duration:.2f}s")

    return results, models

What does this function do?

The function handles three strategies for class imbalance:

  • class_weight="balanced" (Logistic Regression and Random Forest) automatically adjusts weights inversely proportional to class frequencies.
  • scale_pos_weight (XGBoost) sets the ratio of negative to positive samples, telling the model to weigh the minority class more heavily.
  • For each model, it fits on training data, generates predictions and probability scores on the test set, computes five evaluation metrics, and saves the trained model to disk with joblib.

The function returns both the results list (metrics per model) and the trained model objects for further use.

  • Save src/train.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import time
import json
import hashlib
from pathlib import Path

import pandas as pd
import numpy as np
import joblib
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    roc_auc_score,
    average_precision_score,
    f1_score,
    precision_score,
    recall_score,
    classification_report,
)
from xgboost import XGBClassifier


def train_and_evaluate(X_train, X_test, y_train, y_test, models_dir="models"):
    Path(models_dir).mkdir(exist_ok=True)

    pos_count = y_train.sum()
    neg_count = len(y_train) - pos_count
    scale_pos_weight = neg_count / pos_count

    models = {
        "logistic_regression": LogisticRegression(
            class_weight="balanced", max_iter=1000, random_state=42
        ),
        "random_forest": RandomForestClassifier(
            n_estimators=100, class_weight="balanced", random_state=42, n_jobs=-1
        ),
        "xgboost": XGBClassifier(
            n_estimators=100,
            scale_pos_weight=scale_pos_weight,
            random_state=42,
            eval_metric="logloss",
            verbosity=0,
        ),
    }

    results = []

    for name, model in models.items():
        start_time = time.time()
        model.fit(X_train, y_train)
        duration = time.time() - start_time

        y_pred = model.predict(X_test)
        y_prob = model.predict_proba(X_test)[:, 1]

        metrics = {
            "auc_roc": round(roc_auc_score(y_test, y_prob), 4),
            "pr_auc": round(average_precision_score(y_test, y_prob), 4),
            "f1": round(f1_score(y_test, y_pred), 4),
            "precision": round(precision_score(y_test, y_pred), 4),
            "recall": round(recall_score(y_test, y_pred), 4),
        }

        model_path = f"{models_dir}/{name}.pkl"
        joblib.dump(model, model_path)

        results.append(
            {
                "model_name": name,
                "metrics": metrics,
                "training_duration_seconds": round(duration, 2),
                "model_path": model_path,
            }
        )

        print(f"\n{name}:")
        print(f"  AUC-ROC: {metrics['auc_roc']}")
        print(f"  PR-AUC: {metrics['pr_auc']}")
        print(f"  F1: {metrics['f1']}")
        print(f"  Training time: {duration:.2f}s")

    return results, models

Seeing an import error?

  • Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • If xgboost is not found, run pip install -r requirements.txt again from your project root.

Why can't Python find my xgboost import?

Evaluate all models with proper metrics

Now you will run the training and generate visual comparisons. ROC curves show how well each model separates classes at all thresholds. Precision-Recall curves are more informative for imbalanced datasets because they focus on performance for the minority class.

  • Create a file called run_training.py in your project root and add the following code to load data, train models, and generate evaluation plots:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from pathlib import Path
import joblib
from sklearn.metrics import roc_curve, precision_recall_curve, roc_auc_score, average_precision_score
from src.train import train_and_evaluate

# Load the train-test split data
X_train = pd.read_csv("data/processed/X_train.csv")
X_test = pd.read_csv("data/processed/X_test.csv")
y_train = pd.read_csv("data/processed/y_train.csv").squeeze()
y_test = pd.read_csv("data/processed/y_test.csv").squeeze()

print(f"Training set: {X_train.shape[0]} samples, {X_train.shape[1]} features")
print(f"Test set: {X_test.shape[0]} samples")
print(f"Positive class rate (train): {y_train.mean():.2%}")

# Train all three models
results, models = train_and_evaluate(X_train, X_test, y_train, y_test)

# Plot ROC curves and Precision-Recall curves
Path("data/plots").mkdir(parents=True, exist_ok=True)

fig, axes = plt.subplots(1, 2, figsize=(14, 5))
model_names = ["logistic_regression", "random_forest", "xgboost"]
colors = ["blue", "green", "red"]

for name, color in zip(model_names, colors):
    model = models[name]
    y_prob = model.predict_proba(X_test)[:, 1]

    # ROC curve
    fpr, tpr, _ = roc_curve(y_test, y_prob)
    auc = roc_auc_score(y_test, y_prob)
    axes[0].plot(fpr, tpr, color=color, label=f"{name} (AUC={auc:.3f})")

    # Precision-Recall curve
    precision, recall, _ = precision_recall_curve(y_test, y_prob)
    ap = average_precision_score(y_test, y_prob)
    axes[1].plot(recall, precision, color=color, label=f"{name} (AP={ap:.3f})")

axes[0].plot([0, 1], [0, 1], "k--", alpha=0.5)
axes[0].set_xlabel("False Positive Rate")
axes[0].set_ylabel("True Positive Rate")
axes[0].set_title("ROC Curves")
axes[0].legend()

axes[1].set_xlabel("Recall")
axes[1].set_ylabel("Precision")
axes[1].set_title("Precision-Recall Curves")
axes[1].legend()

plt.tight_layout()
plt.savefig("data/plots/model_comparison_curves.png", dpi=150, bbox_inches="tight")
plt.close()
print("\nROC and PR curves saved to data/plots/model_comparison_curves.png")

Why Precision-Recall curves over just ROC?

ROC curves can look overly optimistic on imbalanced datasets because the False Positive Rate denominator (true negatives) is large. Precision-Recall curves focus on the minority class and reveal how well the model performs where it matters most.

Average Precision (AP) summarizes the PR curve as a single number. A higher AP means the model finds readmitted patients without flagging too many false alarms.

  • Run the training script by executing this command in your terminal:
python run_training.py
  • You should see training output for each model showing AUC-ROC, PR-AUC, F1, and training time.
  • Confirm the models were saved by running:
ls models/

You should see logistic_regression.pkl, random_forest.pkl, and xgboost.pkl listed.

Seeing a ModuleNotFoundError for src.train?

  • Make sure you are running the command from your project root folder (the one containing the src/ directory).
  • If Python cannot find the src package, create an empty src/__init__.py file so Python treats src/ as a package.

Help me fix this ModuleNotFoundError when importing from src.train.

Create a model comparison summary

With all three models trained and evaluated, you need a structured comparison to identify the best performer. This comparison table will also feed into your Streamlit dashboard later.

  • Add the following code to the bottom of run_training.py to build a comparison DataFrame and save it:
# Build model comparison DataFrame
Path("experiments").mkdir(exist_ok=True)

comparison_rows = []
for r in results:
    row = {"model_name": r["model_name"]}
    row.update(r["metrics"])
    comparison_rows.append(row)

comparison_df = pd.DataFrame(comparison_rows)
print("\nModel Comparison:")
print(comparison_df.to_string(index=False))

# Identify the best model by AUC-ROC
best_idx = comparison_df["auc_roc"].idxmax()
best_model = comparison_df.loc[best_idx, "model_name"]
print(f"\nBest model by AUC-ROC: {best_model}")

# Save comparison table
comparison_df.to_csv("experiments/model_comparison.csv", index=False)
print("Comparison saved to experiments/model_comparison.csv")

What does this code do?

  • It extracts each model's metrics from the results list and builds a single DataFrame with all models side by side.
  • idxmax() finds the row index with the highest AUC-ROC score, identifying the best-performing model.
  • The CSV output in experiments/model_comparison.csv is loaded by the Streamlit dashboard in a later step.
  • Save run_training.py and run it again to generate the comparison:
python run_training.py
  • You should see the model comparison table printed in your terminal, with columns for each metric and a line identifying the best model.
  • Confirm the experiment output was saved by running:
ls experiments/

You should see model_comparison.csv in the output.

Training taking too long?

  • Random Forest and XGBoost may take 30-60 seconds each on the full dataset. This is normal for ~80,000+ training samples with many features.
  • If your machine is very slow, reduce n_estimators to 50 temporarily for testing, then set it back to 100 for the final run.

Why is my model training so slow?

You now have three trained models saved to disk, evaluation curves saved as plots, and a comparison CSV ready for your dashboard. Next up, you will build an experiment tracking system that logs every training run for reproducibility.

Track Experiments and Ensure Reproducibility

You have three trained models with metrics saved in a comparison CSV. But what if you want to retrain with different hyperparameters next week and compare against today's results?

Without a structured log of each experiment, you lose context about what was tried and when. In this step, you will build an experiment tracking system that logs every training run with its hyperparameters, metrics, and timestamps. You will also create a reproducibility manifest so anyone can recreate your exact results.

In this step, get ready to:
  • Build an experiment logger that records model runs to JSON.
  • Log all three model experiments with full metadata.
  • Create a reproducibility manifest capturing your environment and data fingerprint.
Build the experiment logger

The experiment tracker is a standalone module that appends structured records to a JSON file each time you train a model. It also generates a manifest that pins your exact Python version, package versions, random seed, and a hash of the dataset.

  • Create a new file called src/experiment_tracker.py in your src/ folder.
  • Add the imports and the log_experiment() function by pasting this code:
import json
import hashlib
from datetime import datetime, timezone
from pathlib import Path


def log_experiment(model_name, hyperparams, metrics, duration, dataset_info, experiments_dir="experiments"):
    Path(experiments_dir).mkdir(exist_ok=True)
    log_path = Path(experiments_dir) / "experiment_log.json"

    experiment = {
        "model_name": model_name,
        "model_version": "1.0.0",
        "hyperparameters": hyperparams,
        "metrics": metrics,
        "training_duration_seconds": duration,
        "training_timestamp": datetime.now(timezone.utc).isoformat(),
        "dataset_info": dataset_info,
    }

    if log_path.exists():
        with open(log_path, "r") as f:
            logs = json.load(f)
    else:
        logs = []

    logs.append(experiment)

    with open(log_path, "w") as f:
        json.dump(logs, f, indent=2)

    return experiment

What does this code do?

The log_experiment() function builds a dictionary containing the model name, a semantic version string, hyperparameters, metrics, training duration, a UTC timestamp, and dataset metadata.

It reads the existing log file (or starts with an empty list), appends the new entry, and writes the updated list back. This append-based design means you never lose previous entries when logging a new experiment.

  • Below the log_experiment() function, add the create_reproducibility_manifest() function:
def create_reproducibility_manifest(packages_str, random_seed, dataset_url, data_path, experiments_dir="experiments"):
    Path(experiments_dir).mkdir(exist_ok=True)

    data_hash = ""
    if Path(data_path).exists():
        with open(data_path, "rb") as f:
            data_hash = hashlib.md5(f.read()).hexdigest()

    manifest = {
        "python_version": "3.12.1",
        "packages": packages_str,
        "random_seed": random_seed,
        "dataset_source": dataset_url,
        "dataset_md5": data_hash,
        "created_at": datetime.now(timezone.utc).isoformat(),
    }

    manifest_path = Path(experiments_dir) / "reproducibility.json"
    with open(manifest_path, "w") as f:
        json.dump(manifest, f, indent=2)

    return manifest

What does this code do?

The create_reproducibility_manifest() function captures everything needed to reproduce your training environment. It records your Python version, all installed packages, the random seed you used, the dataset source URL, and an MD5 hash of the cleaned data file.

The MD5 hash acts as a fingerprint. If anyone changes even a single value in the dataset, the hash will differ, instantly flagging that the data is not identical to your training set.

  • Save src/experiment_tracker.py.

Seeing a syntax error?

Make sure the create_reproducibility_manifest() function is at the same indentation level as log_experiment() (no extra indent). Both functions should be top-level in the file.

help me fix a syntax error in my experiment_tracker.py file

✔️ Awesome, I've got everything!

Great. Double check you have saved src/experiment_tracker.py before moving on.

ⓧ I'd like to double check the full code

import json
import hashlib
from datetime import datetime, timezone
from pathlib import Path


def log_experiment(model_name, hyperparams, metrics, duration, dataset_info, experiments_dir="experiments"):
    Path(experiments_dir).mkdir(exist_ok=True)
    log_path = Path(experiments_dir) / "experiment_log.json"

    experiment = {
        "model_name": model_name,
        "model_version": "1.0.0",
        "hyperparameters": hyperparams,
        "metrics": metrics,
        "training_duration_seconds": duration,
        "training_timestamp": datetime.now(timezone.utc).isoformat(),
        "dataset_info": dataset_info,
    }

    if log_path.exists():
        with open(log_path, "r") as f:
            logs = json.load(f)
    else:
        logs = []

    logs.append(experiment)

    with open(log_path, "w") as f:
        json.dump(logs, f, indent=2)

    return experiment


def create_reproducibility_manifest(packages_str, random_seed, dataset_url, data_path, experiments_dir="experiments"):
    Path(experiments_dir).mkdir(exist_ok=True)

    data_hash = ""
    if Path(data_path).exists():
        with open(data_path, "rb") as f:
            data_hash = hashlib.md5(f.read()).hexdigest()

    manifest = {
        "python_version": "3.12.1",
        "packages": packages_str,
        "random_seed": random_seed,
        "dataset_source": dataset_url,
        "dataset_md5": data_hash,
        "created_at": datetime.now(timezone.utc).isoformat(),
    }

    manifest_path = Path(experiments_dir) / "reproducibility.json"
    with open(manifest_path, "w") as f:
        json.dump(manifest, f, indent=2)

    return manifest
Log all three model experiments

Now you will use your new tracker to log each of the three models you trained in the previous step. This re-runs training so you can capture accurate timing data alongside the metrics.

  • Run the following Python script in your terminal to train the models and log each experiment:
python -c "
import pandas as pd
from src.train import train_and_evaluate
from src.experiment_tracker import log_experiment

# Load the train/test splits
X_train = pd.read_csv('data/processed/X_train.csv')
X_test = pd.read_csv('data/processed/X_test.csv')
y_train = pd.read_csv('data/processed/y_train.csv').squeeze()
y_test = pd.read_csv('data/processed/y_test.csv').squeeze()

# Train models and get results
results, models = train_and_evaluate(X_train, X_test, y_train, y_test)

# Build dataset info dict
dataset_info = {
    'rows': len(X_train) + len(X_test),
    'features': X_train.shape[1],
    'class_balance': round(float(y_train.mean()), 4)
}

# Hyperparameters for each model
hyperparams_map = {
    'logistic_regression': {'class_weight': 'balanced', 'max_iter': 1000, 'random_state': 42},
    'random_forest': {'n_estimators': 100, 'class_weight': 'balanced', 'random_state': 42, 'n_jobs': -1},
    'xgboost': {'n_estimators': 100, 'scale_pos_weight': 'auto', 'random_state': 42, 'eval_metric': 'logloss'}
}

# Log each experiment
for result in results:
    name = result['model_name']
    exp = log_experiment(
        model_name=name,
        hyperparams=hyperparams_map[name],
        metrics=result['metrics'],
        duration=result['training_duration_seconds'],
        dataset_info=dataset_info
    )
    print(f'Logged: {name} (AUC-ROC: {exp[\"metrics\"][\"auc_roc\"]})')

print('\nAll experiments logged to experiments/experiment_log.json')
"

What does this script do?

This script re-runs train_and_evaluate() from your existing src/train.py to get fresh metrics and timing for each model. It then calls log_experiment() three times, once per model, writing structured entries into experiments/experiment_log.json.

Each entry includes the model name, version 1.0.0, hyperparameters, all five evaluation metrics, training duration in seconds, a UTC timestamp, and dataset metadata.

  • You should see output showing each model logged with its AUC-ROC score, followed by the confirmation message.
  • Verify the log file was created by running:
python -c "import json; logs = json.load(open('experiments/experiment_log.json')); print(f'{len(logs)} experiments logged'); [print(f'  - {e[\"model_name\"]}: AUC-ROC {e[\"metrics\"][\"auc_roc\"]}') for e in logs]"

You should see 3 experiments listed with their model names and AUC-ROC values.

Seeing a ModuleNotFoundError?

If Python cannot find src.train or src.experiment_tracker, make sure you are running the command from your project root folder (the one containing the src/ directory). Also confirm your virtual environment is activated.

help me fix a ModuleNotFoundError when running my experiment logging script

Create the reproducibility manifest

The final piece is a manifest that records your exact environment. This lets anyone clone your repo, install the same packages, and reproduce your results down to the decimal.

  • Generate the reproducibility manifest by running:
python -c "
import subprocess
from src.experiment_tracker import create_reproducibility_manifest

# Capture installed packages
packages = subprocess.check_output(['pip', 'freeze']).decode()

# Create the manifest
manifest = create_reproducibility_manifest(
    packages_str=packages,
    random_seed=42,
    dataset_url='https://archive.ics.uci.edu/dataset/296/diabetes+130-us+hospitals+for+years+1999-2008',
    data_path='data/processed/cleaned_data.csv'
)

print('Reproducibility manifest saved to experiments/reproducibility.json')
print(f'  Python version: {manifest[\"python_version\"]}')
print(f'  Random seed: {manifest[\"random_seed\"]}')
print(f'  Dataset MD5: {manifest[\"dataset_md5\"][:16]}...')
"

Why does reproducibility matter?

Machine learning results are notoriously hard to reproduce. Tiny differences in package versions, random seeds, or data preprocessing can shift metrics by several percentage points.

The manifest pins every variable that could affect your output. The MD5 hash of your cleaned data file means you can instantly detect if the dataset has been modified since training.

  • Confirm the manifest was created by running:
python -c "import json; m = json.load(open('experiments/reproducibility.json')); print(json.dumps(m, indent=2)[:500])"

You should see a JSON object with your Python version, packages list, random seed, dataset source URL, and the MD5 hash.

Dataset MD5 shows an empty string?

This means the file at data/processed/cleaned_data.csv was not found. Double check that you saved the cleaned dataset in Step 2. The path must match exactly.

help me troubleshoot why my reproducibility manifest has an empty dataset_md5 value

Your experiment tracking system is in place. Every future training run can be logged with full context, making it easy to compare new approaches against today's baseline. Next up, you will build an interactive Streamlit dashboard and deploy it to the cloud.

Build and Deploy the Streamlit Dashboard

Your ML pipeline is complete. You have trained models, tracked experiments, and saved comparison metrics. The missing piece is a way for anyone to explore these results without running Python scripts.

In this step, you will build a multi-page Streamlit dashboard that pulls together your data quality checks, model comparison curves, and experiment logs into a single interactive interface. Then you will deploy it to the cloud so it is accessible from a public URL.

In this step, get ready to:
  • Build a three-page Streamlit dashboard with data overview, model comparison, and experiment log pages.
  • Test the app locally to verify all pages render correctly.
  • Deploy the app to Streamlit Community Cloud with a live public URL.
Build the Streamlit dashboard

Streamlit turns Python scripts into interactive web apps. Each page of your dashboard will display a different slice of your pipeline results.

  • Create the app/ directory and the main app file by running these commands:
mkdir app
touch app/streamlit_app.py
  • Confirm the file exists by running:
ls app/

You should see streamlit_app.py in the output.

  • Open app/streamlit_app.py in your editor and paste the following imports and configuration:
import streamlit as st
import pandas as pd
import numpy as np
import json
from pathlib import Path
import matplotlib.pyplot as plt
import joblib
from sklearn.metrics import roc_curve, precision_recall_curve, roc_auc_score, average_precision_score

# Configure the page layout and title
st.set_page_config(page_title="ML Readmission Pipeline", layout="wide")

# Define paths to data, experiments, and models
DATA_DIR = Path("data/processed")
EXPERIMENTS_DIR = Path("experiments")
MODELS_DIR = Path("models")


def load_data():
    # Load test set for generating predictions on the fly
    X_test = pd.read_csv(DATA_DIR / "X_test.csv")
    y_test = pd.read_csv(DATA_DIR / "y_test.csv").squeeze()
    return X_test, y_test


# Sidebar navigation between the three pages
page = st.sidebar.selectbox("Navigate", ["Data Overview", "Model Comparison", "Experiment Log"])

What does this code do?

  • st.set_page_config sets the browser tab title and enables wide layout for charts.
  • The three Path constants point to your processed data, experiment logs, and saved models.
  • load_data() reads the test set so the app can compute live predictions from saved models.
  • st.sidebar.selectbox creates a dropdown in the sidebar that controls which page content appears.
  • Below the selectbox line, add the Data Overview page:
if page == "Data Overview":
    st.title("Data Overview")
    st.header("Dataset Statistics")

    # Load the cleaned dataset for summary stats
    cleaned_data = pd.read_csv(DATA_DIR / "cleaned_data.csv")
    col1, col2, col3 = st.columns(3)
    col1.metric("Total Records", f"{len(cleaned_data):,}")
    col2.metric("Features", cleaned_data.shape[1])
    col3.metric("Readmission Rate", f"{cleaned_data['readmitted_binary'].mean():.1%}")

    st.header("Class Distribution")
    fig, ax = plt.subplots(figsize=(6, 4))
    cleaned_data["readmitted_binary"].value_counts().plot(kind="bar", ax=ax, color=["steelblue", "coral"])
    ax.set_xticklabels(["Not Readmitted (0)", "Readmitted <30d (1)"], rotation=0)
    ax.set_ylabel("Count")
    ax.set_title("Target Variable Distribution")
    st.pyplot(fig)

    st.header("Data Quality Summary")
    missing = cleaned_data.isna().sum()
    missing_report = missing[missing > 0].sort_values(ascending=False)
    if len(missing_report) > 0:
        st.dataframe(missing_report.to_frame("Missing Count"))
    else:
        st.success("No missing values in cleaned dataset.")

What does the Data Overview page show?

  • Three metric cards at the top display total records, feature count, and readmission rate at a glance.
  • A bar chart visualizes the class imbalance between readmitted and non-readmitted patients.
  • The quality summary section checks for any remaining missing values in the cleaned data.
  • Below the Data Overview block, add the Model Comparison page:
elif page == "Model Comparison":
    st.title("Model Comparison")

    # Load the comparison CSV generated during training
    comparison_path = EXPERIMENTS_DIR / "model_comparison.csv"
    if comparison_path.exists():
        comparison = pd.read_csv(comparison_path)
        st.header("Performance Metrics")
        st.dataframe(comparison.style.highlight_max(axis=0, subset=comparison.columns[1:]))

        best_model = comparison.loc[comparison["auc_roc"].idxmax(), "model_name"]
        st.success(f"Best model by AUC-ROC: {best_model}")

    st.header("ROC Curves")
    X_test, y_test = load_data()

    fig, axes = plt.subplots(1, 2, figsize=(14, 5))

    model_names = ["logistic_regression", "random_forest", "xgboost"]
    colors = ["blue", "green", "red"]

    for name, color in zip(model_names, colors):
        model_path = MODELS_DIR / f"{name}.pkl"
        if model_path.exists():
            model = joblib.load(model_path)
            y_prob = model.predict_proba(X_test)[:, 1]

            # ROC curve on the left subplot
            fpr, tpr, _ = roc_curve(y_test, y_prob)
            auc = roc_auc_score(y_test, y_prob)
            axes[0].plot(fpr, tpr, color=color, label=f"{name} (AUC={auc:.3f})")

            # Precision-Recall curve on the right subplot
            precision, recall, _ = precision_recall_curve(y_test, y_prob)
            ap = average_precision_score(y_test, y_prob)
            axes[1].plot(recall, precision, color=color, label=f"{name} (AP={ap:.3f})")

    axes[0].plot([0, 1], [0, 1], "k--", alpha=0.5)
    axes[0].set_xlabel("False Positive Rate")
    axes[0].set_ylabel("True Positive Rate")
    axes[0].set_title("ROC Curves")
    axes[0].legend()

    axes[1].set_xlabel("Recall")
    axes[1].set_ylabel("Precision")
    axes[1].set_title("Precision-Recall Curves")
    axes[1].legend()

    plt.tight_layout()
    st.pyplot(fig)

What does the Model Comparison page show?

  • A styled DataFrame highlights the best metric value in each column so the top performer stands out immediately.
  • ROC curves plot all three models together so you can visually compare true positive rates at each threshold.
  • Precision-Recall curves are especially useful here because the dataset is imbalanced (only ~11% positive class).
  • Finally, add the Experiment Log page below the Model Comparison block:
elif page == "Experiment Log":
    st.title("Experiment Log")

    # Read the experiment log JSON created by the tracker
    log_path = EXPERIMENTS_DIR / "experiment_log.json"
    if log_path.exists():
        with open(log_path, "r") as f:
            logs = json.load(f)

        for i, exp in enumerate(logs):
            with st.expander(f"{exp['model_name']} - {exp['training_timestamp'][:19]}"):
                col1, col2 = st.columns(2)
                with col1:
                    st.subheader("Metrics")
                    for metric, value in exp["metrics"].items():
                        st.write(f"- {metric}: {value}")
                with col2:
                    st.subheader("Hyperparameters")
                    for param, value in exp["hyperparameters"].items():
                        st.write(f"- {param}: {value}")
                st.write(f"Training duration: {exp['training_duration_seconds']}s")
    else:
        st.warning("No experiment log found. Run training first.")

    # Display the reproducibility manifest
    reproducibility_path = EXPERIMENTS_DIR / "reproducibility.json"
    if reproducibility_path.exists():
        st.header("Reproducibility Manifest")
        with open(reproducibility_path, "r") as f:
            manifest = json.load(f)
        st.json(manifest)

What does the Experiment Log page show?

  • Each experiment appears as an expandable section showing the model name and timestamp.
  • Inside each expander, metrics and hyperparameters display side by side for quick comparison.
  • The reproducibility manifest at the bottom displays your Python version, package versions, random seed, and dataset hash so anyone can reproduce your exact results.
  • Save app/streamlit_app.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete app/streamlit_app.py file for comparison:

import streamlit as st
import pandas as pd
import numpy as np
import json
from pathlib import Path
import matplotlib.pyplot as plt
import joblib
from sklearn.metrics import roc_curve, precision_recall_curve, roc_auc_score, average_precision_score

st.set_page_config(page_title="ML Readmission Pipeline", layout="wide")

DATA_DIR = Path("data/processed")
EXPERIMENTS_DIR = Path("experiments")
MODELS_DIR = Path("models")


def load_data():
    X_test = pd.read_csv(DATA_DIR / "X_test.csv")
    y_test = pd.read_csv(DATA_DIR / "y_test.csv").squeeze()
    return X_test, y_test


page = st.sidebar.selectbox("Navigate", ["Data Overview", "Model Comparison", "Experiment Log"])

if page == "Data Overview":
    st.title("Data Overview")
    st.header("Dataset Statistics")

    cleaned_data = pd.read_csv(DATA_DIR / "cleaned_data.csv")
    col1, col2, col3 = st.columns(3)
    col1.metric("Total Records", f"{len(cleaned_data):,}")
    col2.metric("Features", cleaned_data.shape[1])
    col3.metric("Readmission Rate", f"{cleaned_data['readmitted_binary'].mean():.1%}")

    st.header("Class Distribution")
    fig, ax = plt.subplots(figsize=(6, 4))
    cleaned_data["readmitted_binary"].value_counts().plot(kind="bar", ax=ax, color=["steelblue", "coral"])
    ax.set_xticklabels(["Not Readmitted (0)", "Readmitted <30d (1)"], rotation=0)
    ax.set_ylabel("Count")
    ax.set_title("Target Variable Distribution")
    st.pyplot(fig)

    st.header("Data Quality Summary")
    missing = cleaned_data.isna().sum()
    missing_report = missing[missing > 0].sort_values(ascending=False)
    if len(missing_report) > 0:
        st.dataframe(missing_report.to_frame("Missing Count"))
    else:
        st.success("No missing values in cleaned dataset.")

elif page == "Model Comparison":
    st.title("Model Comparison")

    comparison_path = EXPERIMENTS_DIR / "model_comparison.csv"
    if comparison_path.exists():
        comparison = pd.read_csv(comparison_path)
        st.header("Performance Metrics")
        st.dataframe(comparison.style.highlight_max(axis=0, subset=comparison.columns[1:]))

        best_model = comparison.loc[comparison["auc_roc"].idxmax(), "model_name"]
        st.success(f"Best model by AUC-ROC: {best_model}")

    st.header("ROC Curves")
    X_test, y_test = load_data()

    fig, axes = plt.subplots(1, 2, figsize=(14, 5))

    model_names = ["logistic_regression", "random_forest", "xgboost"]
    colors = ["blue", "green", "red"]

    for name, color in zip(model_names, colors):
        model_path = MODELS_DIR / f"{name}.pkl"
        if model_path.exists():
            model = joblib.load(model_path)
            y_prob = model.predict_proba(X_test)[:, 1]

            fpr, tpr, _ = roc_curve(y_test, y_prob)
            auc = roc_auc_score(y_test, y_prob)
            axes[0].plot(fpr, tpr, color=color, label=f"{name} (AUC={auc:.3f})")

            precision, recall, _ = precision_recall_curve(y_test, y_prob)
            ap = average_precision_score(y_test, y_prob)
            axes[1].plot(recall, precision, color=color, label=f"{name} (AP={ap:.3f})")

    axes[0].plot([0, 1], [0, 1], "k--", alpha=0.5)
    axes[0].set_xlabel("False Positive Rate")
    axes[0].set_ylabel("True Positive Rate")
    axes[0].set_title("ROC Curves")
    axes[0].legend()

    axes[1].set_xlabel("Recall")
    axes[1].set_ylabel("Precision")
    axes[1].set_title("Precision-Recall Curves")
    axes[1].legend()

    plt.tight_layout()
    st.pyplot(fig)

elif page == "Experiment Log":
    st.title("Experiment Log")

    log_path = EXPERIMENTS_DIR / "experiment_log.json"
    if log_path.exists():
        with open(log_path, "r") as f:
            logs = json.load(f)

        for i, exp in enumerate(logs):
            with st.expander(f"{exp['model_name']} - {exp['training_timestamp'][:19]}"):
                col1, col2 = st.columns(2)
                with col1:
                    st.subheader("Metrics")
                    for metric, value in exp["metrics"].items():
                        st.write(f"- {metric}: {value}")
                with col2:
                    st.subheader("Hyperparameters")
                    for param, value in exp["hyperparameters"].items():
                        st.write(f"- {param}: {value}")
                st.write(f"Training duration: {exp['training_duration_seconds']}s")
    else:
        st.warning("No experiment log found. Run training first.")

    reproducibility_path = EXPERIMENTS_DIR / "reproducibility.json"
    if reproducibility_path.exists():
        st.header("Reproducibility Manifest")
        with open(reproducibility_path, "r") as f:
            manifest = json.load(f)
        st.json(manifest)
Test the app locally

Before deploying, you need to confirm all three pages render correctly with your local data.

  • Launch the Streamlit app by running:
streamlit run app/streamlit_app.py

What should I see?

Your terminal will display a local URL (typically http://localhost:8501). Your browser should open automatically showing the Data Overview page with three metric cards and a bar chart.

  • In the sidebar, select Model Comparison from the dropdown.
  • Verify you see the metrics table with highlighted values, plus the ROC and Precision-Recall curve plots.
  • Select Experiment Log and verify you see expandable sections for each model with metrics and hyperparameters.

App not loading?

  • Make sure your virtual environment is activated. The streamlit command must be available in your current shell.
  • Check that data/processed/cleaned_data.csv and experiments/experiment_log.json exist. The app reads from these files at runtime.
  • If you see an import error, confirm all packages from requirements.txt are installed in your environment.

Help me debug my Streamlit app that won't load locally.

  • Once all three pages display correctly, press Ctrl+C in your terminal to stop the local server.
Deploy to Streamlit Community Cloud

Your app works locally. Now you will push the code to GitHub and deploy it to Streamlit Community Cloud so anyone with the URL can access your dashboard.

Why Streamlit Community Cloud?

Streamlit Community Cloud is free for unlimited public apps and deploys directly from a GitHub repository. Any time you push a code change, the live app updates automatically.

  • Your .gitignore excludes data/ and models/*.pkl, but the deployed app needs these files. Stage everything including the excluded files by running:
git add .
git add -f data/processed/
git add -f models/
git commit -m "ML readmission pipeline with Streamlit dashboard"

What does git add -f do?

The -f flag force-adds files that would normally be ignored by .gitignore. This keeps your .gitignore intact (still protecting raw data from accidental commits) while explicitly tracking the processed data and models that your deployed app needs.

  • Go to github.com and create a new public repository. Name it something descriptive like ml-readmission-pipeline.
  • Do not initialize it with a README or .gitignore (your local repo already has these).
  • Copy the remote URL GitHub gives you, then connect and push by running:
git remote add origin [[YOUR_REPO_URL="https://github.com/your-username/ml-readmission-pipeline.git"]]
git branch -M main
git push -u origin main
  • Verify on GitHub that your repository contains the app/, src/, data/processed/, models/, experiments/, and requirements.txt files.

Now deploy the app to Streamlit Community Cloud.

  • Go to share.streamlit.io and sign in with your GitHub account.
  • Click Create app in the upper-right corner.
  • When asked if you already have an app, click Yup, I have an app.
  • Select your repository, set the branch to main, and set the main file path to app/streamlit_app.py.
  • Click Advanced settings.
  • In the Python version dropdown, verify that 3.12 is selected. This is the default, but confirm it matches your project.
  • Click Save, then click Deploy.

What should I see?

Streamlit will show deployment logs as it installs your dependencies. This typically takes 2-5 minutes. Once complete, your app loads at a public URL ending in .streamlit.app.

  • Once the deployment finishes, verify the live app by navigating through all three pages in the sidebar.

Deployment failing with dependency errors?

  • Make sure your requirements.txt is in the root of your repository (not inside a subfolder). Streamlit Community Cloud looks for it at the project root.
  • Confirm the Python version in Advanced Settings is 3.12. Some packages in your requirements.txt may fail to build on other Python versions.
  • Check that data/processed/ and models/ are actually tracked in your repo. Run git ls-files data/processed/ locally to verify files are committed.

Help me fix my Streamlit Community Cloud deployment error.

Your ML pipeline is now live on the internet. Anyone with your URL can explore the data quality checks, compare model performance, and review experiment history. Next up, you will push this project further with SHAP-based model explainability in the Secret Mission.

Secret mission

Add SHAP Feature Importance Comparison

Your dashboard shows which model performs best, but it doesn't explain why each model makes its predictions. In healthcare, clinicians need to trust a model before acting on it. Add SHAP-based explainability to your dashboard so stakeholders can see exactly which clinical features drive readmission predictions across all three models.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on free services, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Streamlit Community Cloud app (your live dashboard at your-app-name.streamlit.app).
  • GitHub repository (public, stores your project code).
  • Local project folder on your Desktop (includes venv/, source code, data, and trained models).

Keep everything running

No action needed. Choose this if you're still actively building or want your dashboard to remain publicly accessible.

  • Your Streamlit Community Cloud app stays live indefinitely on the free tier.
  • Your GitHub repository remains public and accessible.
  • Your local project files stay on your Desktop for future development.

Pause - I'll come back to this later

Put your app to sleep so it stops running but can be reactivated later. Your local files stay on your machine.

  • Go to share.streamlit.io and sign in with your GitHub account.
  • Find your app in the dashboard and click the kebab menu (three dots).
  • Select Put app to sleep. The app will wake automatically when someone visits its URL.
  • Your GitHub repository and local files remain untouched.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your deployed app, repository, and local files.

  • Delete your Streamlit Community Cloud app: Go to share.streamlit.io, find your app in the dashboard, click the kebab menu (three dots), and select Delete.
  • Delete your GitHub repository: Go to your repository on GitHub, click Settings, scroll to the bottom to the Danger Zone section, and click Delete this repository.
  • Delete your local project files by running this in your terminal (macOS/Linux):
cd ~/Desktop
rm -rf <your-project-folder-name>

On Windows, run this in PowerShell:

cd ~\Desktop
Remove-Item -Recurse -Force <your-project-folder-name>

Replace <your-project-folder-name> with the actual name of your project folder.

Nice Work!

Nice Work!

Nice work! You just built and deployed a production-style machine learning pipeline that ingests real hospital data, trains and compares classification models, and presents everything through a live interactive dashboard.

You've learned how to:

  • Build an end-to-end ML pipeline that ingests 101,766 real clinical records, validates data quality, engineers features, and handles class imbalance in a healthcare dataset.
  • Train and compare three classification models (Logistic Regression, Random Forest, XGBoost) using proper evaluation metrics like AUC-ROC and PR-AUC for imbalanced data.
  • Deploy a multi-page Streamlit dashboard to Streamlit Community Cloud with experiment tracking, model comparison, and a live public URL.
  • Secret Mission: Added SHAP-based feature importance analysis comparing explainability across all three models with interactive visualizations and clinical interpretation.

Ready to quiz yourself?