Clinical Cohort Analytics in Python

Introduction

30 Second Summary

Every hospital visit generates a trail of data. Diagnoses, lab results, medications, encounter timestamps. Healthcare organizations need people who can transform that raw information into actionable clinical insights.

In this project, you will build a Clinical Cohort Analytics pipeline in Python that processes synthetic patient data, constructs diagnosis-based patient cohorts, and computes clinically meaningful metrics like condition prevalence and 30-day readmission rates.

What You'll Build

A multi-stage Python pipeline that ingests synthetic clinical CSV data generated by Synthea, cleans and joins tables with pandas, segments patients into cohorts, and produces publication-quality visualizations with matplotlib.

By the end of this project, you'll have:

• A working Python pipeline that loads, cleans, and joins 5 clinical data tables into a unified analysis DataFrame.
• Clinically meaningful computed metrics including condition prevalence, average length of stay, and a 30-day readmission proxy.
• Publication-quality visualizations (bar charts, box plots, histograms) saved as PNG files and ready for a portfolio.
• A professional GitHub repository with modular Python files, a Jupyter notebook, and a compelling README.
• Secret Mission: A logistic regression model predicting 30-day readmission with an honest limitations disclosure.

Are there any prerequisites?

You should be comfortable with SQL concepts like joins, grouping, and filtering. Familiarity with clinical data concepts (encounters, conditions, observations) is also helpful.

No Python experience is required. This project teaches you Python fundamentals from scratch as you build.

Set Up Your Python Environment

Before you can analyze clinical data, you need a project folder and the right libraries installed in your Python environment.

In this step, you will create your project directory, activate your existing Miniconda environment, install the VS Code extensions for Jupyter notebooks, and add the data science libraries this project needs.

In this step, get ready to:

• Create a project directory and activate your conda environment.
• Install the Python and Jupyter extensions in VS Code.
• Install pandas, matplotlib, and Jupyter via pip.

Create your project directory and activate conda

Your healthcare-analytics conda environment already has Python 3.11.5 installed. You just need to create a project folder and activate the environment.

• Press Cmd+Space to open Spotlight Search.
• Type Terminal and press Enter to open it.
• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir clinical-cohort-analytics
cd clinical-cohort-analytics

• Activate your conda environment by running:

conda activate healthcare-analytics

What does this command do?

conda activate healthcare-analytics switches your terminal session to use the Python interpreter and packages installed inside your healthcare-analytics conda environment.

Any packages you install with pip while this environment is active only exist inside it and will not affect other projects on your machine.

You should see (healthcare-analytics) appear at the beginning of your Terminal prompt. This confirms the environment is active.

• Confirm your Python version by running:

python --version

✔️ I see Python 3.11.5

Your conda environment is active and using the correct Python version.

ⓧ Command not found or wrong version

Make sure you ran conda activate healthcare-analytics first. If your terminal does not recognize conda, you may need to initialize it for your shell by running:

~/miniconda3/bin/conda init zsh

Close and reopen Terminal after running this, then try conda activate healthcare-analytics again.

Don't see (healthcare-analytics) in your prompt?

If your terminal shows (base) instead of (healthcare-analytics), you need to activate the correct environment. Run conda activate healthcare-analytics again.

If you see an error saying the environment does not exist, create it first with conda create -n healthcare-analytics python=3.11.5 and then activate it.

help me activate my conda environment

Set up VS Code for Jupyter notebooks

VS Code needs two extensions to run Jupyter notebooks: the Python extension (for language support) and the Jupyter extension (for notebook rendering and cell execution).

• Press Cmd+Space (macOS) to open your search bar.
• Type Visual Studio Code and press Enter to open it.
• Open the Extensions sidebar by pressing Cmd+Shift+X.
• Type Python in the search bar and click Install on the one published by Microsoft.
• Type Jupyter in the search bar and click Install on the one published by Microsoft.

Already have these extensions?

If you already see a green checkmark or Disable button instead of Install, the extension is already active. Skip to the next bullet.

• Open your project folder in VS Code by clicking File in the top menu bar.
• Click Open Folder.
• Navigate to Desktop, select the clinical-cohort-analytics folder, then click Open.

• Select your conda environment as the Python interpreter by pressing Cmd+Shift+P to open the Command Palette.
• Type Python: Select Interpreter and press Enter.
• Select the interpreter that shows healthcare-analytics and Python 3.11.5 in its path.

Why select an interpreter?

VS Code needs to know which Python environment to use when running your notebook cells. Selecting the healthcare-analytics interpreter ensures your notebook uses the same packages you install with pip inside that environment.

Install core dependencies

With your conda environment active, you can now install the Python libraries this project needs.

• Open the integrated terminal in VS Code by pressing Ctrl+` (backtick key, top-left of keyboard).
• Confirm you see (healthcare-analytics) in the terminal prompt. If not, run conda activate healthcare-analytics first.
• Install pandas, matplotlib, and Jupyter by running:

pip install pandas==3.0.3 matplotlib==3.10.9 jupyter

What are these libraries?

• pandas is a data analysis library. Think of it as SQL for Python. DataFrames are tables, columns are columns, rows are rows.
• matplotlib is a visualization library for creating charts, plots, and graphs.
• jupyter provides the notebook kernel that VS Code's Jupyter extension connects to for running code in cells.

The install may take a minute as pip downloads each package and its dependencies.

• Once the install finishes, verify the libraries are importable by running a quick test:

python -c "import pandas; import matplotlib; print('All imports successful!')"

You should see All imports successful! printed in your terminal.

Seeing a ModuleNotFoundError?

Make sure your conda environment is still active. Check that (healthcare-analytics) appears at the start of your terminal prompt.

If it disappeared, reactivate it by running conda activate healthcare-analytics, then re-run the pip install command.

help me fix a ModuleNotFoundError in my conda environment

Your Python environment is ready to go. Next up, you will learn some Python fundamentals before diving into clinical data.

Learn Python Fundamentals

Your environment is set up and ready to go. Now it is time to learn the language you will use to build your entire analytics pipeline.

If you already know SQL, you already understand the logic of working with data. Python uses the same patterns (filtering, grouping, transforming) but expresses them differently. In this step, you will learn the core building blocks of Python by creating small examples that mirror clinical data concepts.

In this step, get ready to:

• Create variables and understand Python's core data types.
• Store and retrieve data with lists and dictionaries.
• Write a reusable function that calculates a patient's age.

Create variables and explore data types

In Python, a variable stores a single value. Think of it like a column value in one row of a SQL table. Python has four core data types you will use constantly: strings (text), integers (whole numbers), floats (decimal numbers), and booleans (true/false).

• In VS Code, create a new file called fundamentals_scratch.py inside your clinical-cohort-analytics folder by clicking File in the top menu, then New File.
• Save the file immediately with Cmd+S (macOS) or Ctrl+S (Windows) and name it fundamentals_scratch.py.
• Build a mini patient record using variables by copying this code into your file:

# Variables and data types - building a mini patient record
patient_name = "Sarah Chen"
patient_age = 67
temperature = 98.6
is_admitted = True

# Print each variable to see its value and type
print(patient_name)
print(type(patient_name))

print(patient_age)
print(type(patient_age))

print(temperature)
print(type(temperature))

print(is_admitted)
print(type(is_admitted))

What does this code do?

Each line starting with a name and = creates a variable. Python figures out the type automatically based on what you assign.

• "Sarah Chen" is a string (text wrapped in quotes).
• 67 is an integer (a whole number with no decimal).
• 98.6 is a float (a number with a decimal point).
• True is a boolean (either True or False). type() prints the data type so you can confirm what Python sees.

• Save fundamentals_scratch.py.
• Open your terminal in VS Code by pressing Ctrl+` (backtick key, top-left of keyboard).
• Run the file by entering this command:

python fundamentals_scratch.py

You should see four pairs of output. Each pair shows the value followed by its type (like <class 'str'>, <class 'int'>, <class 'float'>, <class 'bool'>).

How does this compare to SQL?

In SQL, you declare a column's type when you create a table (VARCHAR, INT, FLOAT, BOOLEAN). In Python, you just assign a value and Python infers the type. A variable is like one cell in a table. A row with multiple columns becomes a dictionary (which you will build next).

🙋‍♀️ Seeing a SyntaxError or NameError?

• Make sure the quotes around "Sarah Chen" are straight double quotes, not curly/smart quotes.
• Check that True has a capital T. Python booleans are case-sensitive.
• Confirm your conda environment is active. You should see (healthcare-analytics) at the start of your terminal prompt.

Why is my Python script not running?

Store data in lists and dictionaries

Variables hold single values, but real patient data has collections. A list stores an ordered sequence of items (like a column of values). A dictionary stores key-value pairs (like a row with named columns).

• Add the following list code below your existing code in fundamentals_scratch.py:

# A list of SNOMED-CT condition codes for one patient
condition_codes = ["44054006", "59621000", "185086009"]

# Access items by index (0-based, like SQL row numbering)
print(condition_codes[0])
print(condition_codes[1])
print(len(condition_codes))

What does this code do?

A list is created with square brackets [] and holds items in order. You access items by their position number (index), starting at 0. len() tells you how many items are in the list.

These are real SNOMED-CT codes you will use later to build patient cohorts. 44054006 is diabetes, 59621000 is hypertension, and 185086009 is chronic obstructive bronchitis.

• Save and run the file again:

python fundamentals_scratch.py

You should see your earlier output plus three new lines: the first condition code (44054006), the second (59621000), and the count (3).

• Now add this dictionary code below your list code in fundamentals_scratch.py:

# A dictionary representing a single patient record
patient_record = {
    "id": "abc-123-def",
    "name": "Sarah Chen",
    "age": 67,
    "gender": "F",
    "conditions": condition_codes,
    "is_admitted": True
}

# Access values by key (like selecting a column from a row)
print(patient_record["name"])
print(patient_record["age"])
print(patient_record["conditions"])

What does this code do?

A dictionary uses curly braces {} and stores data as key-value pairs. Each key is a string (like a column name), and each value is the data for that column. You access values by their key name in square brackets.

In SQL terms, this dictionary is one row of a table. The keys are column names. Notice that the "conditions" key holds the entire list you created earlier. Python lets you nest data structures inside each other.

• Save and run the file:

python fundamentals_scratch.py

At the bottom of your output you should see Sarah Chen, 67, and the full list of condition codes printed.

Seeing a KeyError?

• Dictionary keys are case-sensitive. Make sure you typed "name" not "Name".
• Check that your dictionary has commas between each key-value pair.

Help me fix my dictionary code.

Write a reusable function

Functions let you package a block of logic so you can reuse it without copying and pasting. In your analytics pipeline, you will need to calculate patient ages from birth dates over and over. A function solves this by defining the logic once.

• Add this function code at the bottom of fundamentals_scratch.py:

from datetime import date

# A function that calculates a patient's age from their birth date
def calculate_age(birth_date):
    today = date.today()
    age = today.year - birth_date.year
    # Adjust if birthday hasn't happened yet this year
    if (today.month, today.day) < (birth_date.month, birth_date.day):
        age -= 1
    return age

# Test the function with a sample birth date
sample_birth_date = date(1959, 3, 15)
result = calculate_age(sample_birth_date)
print(f"Patient born on {sample_birth_date} is {result} years old")

What does this code do?

• from datetime import date brings in Python's built-in date tools.
• def calculate_age(birth_date): defines a function named calculate_age that accepts one parameter (birth_date).
• The function subtracts birth year from today's year, then adjusts down by 1 if the birthday has not happened yet this year.
• return age sends the calculated value back to whoever called the function.
• The f"..." syntax is an f-string. It lets you embed variables directly inside a string using curly braces.

Functions avoid repeating code. Instead of writing the age math every time you encounter a patient's birth date, you call calculate_age() and get the answer back.

• Save and run the file:

python fundamentals_scratch.py

At the bottom of your output you should see a line like Patient born on 1959-03-15 is 67 years old (the age depends on today's date).

Seeing an ImportError or incorrect age?

• If you see ImportError, make sure the import line reads from datetime import date (lowercase datetime, lowercase date).
• If the age seems wrong, check that you used the less-than sign < in the comparison if (today.month, today.day) < (birth_date.month, birth_date.day).

Help me debug my calculate_age function.

✔️ Awesome, I've got everything!

Great. Double check that you have saved fundamentals_scratch.py and that all four sections (variables, list, dictionary, function) run without errors.

ⓧ I'd like to double check the full code

Here is the complete fundamentals_scratch.py file for comparison:

# Variables and data types - building a mini patient record
patient_name = "Sarah Chen"
patient_age = 67
temperature = 98.6
is_admitted = True

# Print each variable to see its value and type
print(patient_name)
print(type(patient_name))

print(patient_age)
print(type(patient_age))

print(temperature)
print(type(temperature))

print(is_admitted)
print(type(is_admitted))

# A list of SNOMED-CT condition codes for one patient
condition_codes = ["44054006", "59621000", "185086009"]

# Access items by index (0-based, like SQL row numbering)
print(condition_codes[0])
print(condition_codes[1])
print(len(condition_codes))

# A dictionary representing a single patient record
patient_record = {
    "id": "abc-123-def",
    "name": "Sarah Chen",
    "age": 67,
    "gender": "F",
    "conditions": condition_codes,
    "is_admitted": True
}

# Access values by key (like selecting a column from a row)
print(patient_record["name"])
print(patient_record["age"])
print(patient_record["conditions"])

from datetime import date

# A function that calculates a patient's age from their birth date
def calculate_age(birth_date):
    today = date.today()
    age = today.year - birth_date.year
    # Adjust if birthday hasn't happened yet this year
    if (today.month, today.day) < (birth_date.month, birth_date.day):
        age -= 1
    return age

# Test the function with a sample birth date
sample_birth_date = date(1959, 3, 15)
result = calculate_age(sample_birth_date)
print(f"Patient born on {sample_birth_date} is {result} years old")

You now have the core Python building blocks under your belt. Next up, you will generate realistic synthetic patient data using Synthea so you have something real to analyze with these skills.

Generate Synthetic Patient Data with Synthea

Your Python environment is ready and you have the fundamentals down. Now it's time to get some data to analyze.

In this step, you'll use Synthea, an open-source synthetic patient generator, to create realistic (but fake) clinical records for 1,000 patients. This gives you a rich dataset of encounters, conditions, medications, and lab observations to work with throughout the rest of the project.

In this step, get ready to:

• Install Java JDK 17 and configure it for Apple Silicon.
• Clone and build the Synthea patient generator.
• Generate 1,000 synthetic patients and copy the CSV files into your project.

Install Java JDK 17 via Homebrew

Synthea is a Java application, so you need a Java Development Kit (JDK) installed before you can build or run it. You'll install OpenJDK 17 using Homebrew and configure your Mac to find it.

• Install OpenJDK 17 by running this command:

brew install openjdk@17

What does this command do?

Homebrew downloads and installs the Java 17 Development Kit into its own managed directory. Because it's "keg-only" (not linked to system paths automatically), you'll need to tell macOS where to find it in the next step.

Homebrew will print a message about the JDK being keg-only. This means macOS cannot find it yet.

• Create a symlink so macOS can locate the JDK by running this command:

sudo ln -sfn /opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-17.jdk

Why do I need sudo here?

The /Library/Java/JavaVirtualMachines/ directory is a protected system folder. The sudo command gives you temporary administrator privileges to write into it. Enter your Mac login password when prompted.

• Verify Java is installed correctly by running:

java --version

✔️ I see version 17 or higher

You should see output like:

openjdk 17.0.19 2025-01-21
OpenJDK Runtime Environment Homebrew (build 17.0.19+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.19+0, mixed mode, sharing)

Java is ready. Move on to cloning Synthea.

ⓧ I see a different version

If you see an older Java version (like 11 or 8), your system has a different JDK taking priority. Add the OpenJDK 17 bin directory to your PATH by running:

echo 'export PATH="/opt/homebrew/opt/openjdk@17/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

• Run java --version again to confirm it now shows version 17.

ⓧ Command not found

If you see a "command not found" error, the symlink may not have worked. Add OpenJDK 17 to your PATH directly by running:

echo 'export PATH="/opt/homebrew/opt/openjdk@17/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

• Run java --version again to confirm it now shows version 17.

Clone and build Synthea

Synthea is a patient population simulator maintained by MITRE. You'll clone the source code from GitHub and build it locally using Gradle (a Java build tool included in the repository).

• Navigate to your Desktop by running:

cd ~/Desktop

• Clone the Synthea repository and move into it by running these commands:

git clone https://github.com/synthetichealth/synthea.git
cd synthea

• Build Synthea and run its test suite by running:

./gradlew build check test

What does this build command do?

The ./gradlew script is a Gradle "wrapper" bundled inside the Synthea repo. It downloads the correct version of the Gradle build tool, compiles all of Synthea's Java source code, and runs the test suite to confirm everything works.

This step takes 5-10 minutes the first time because it downloads dependencies. You'll see a lot of text scroll by. Wait until you see BUILD SUCCESSFUL at the end.

🙋‍♀️ Build failed?

If you see a "Could not determine java version" error, Java is not on your PATH. Re-run the symlink command from the previous substep, then try the build again.

If the build fails with a network error, check your internet connection and retry. Gradle needs to download dependencies from the internet.

Help me troubleshoot my Synthea build failure

Generate patients and copy the CSV files into your project

Now you'll run Synthea to generate a population of 1,000 synthetic patients, exported as CSV files. Then you'll copy the five tables you need into your project's data/ directory.

• Generate 1,000 synthetic patients with CSV export enabled by running:

./run_synthea -p 1000 --exporter.csv.export=true

What do these flags mean?

The -p 1000 flag sets the population size to 1,000 patients. The --exporter.csv.export=true flag tells Synthea to output data as CSV files (instead of its default FHIR JSON format).

This takes 2-3 minutes. You'll see patient names scrolling by as they're generated. Wait until you see the final summary line showing the number of living and deceased patients.

• Verify the CSV files were generated by running:

ls output/csv/

You should see a list of CSV files including patients.csv, encounters.csv, conditions.csv, medications.csv, and observations.csv (plus several others).

• Create a data/ directory inside your project and copy the five key CSV files into it by running:

mkdir -p ~/Desktop/clinical-cohort-analytics/data
cp output/csv/patients.csv ~/Desktop/clinical-cohort-analytics/data/
cp output/csv/encounters.csv ~/Desktop/clinical-cohort-analytics/data/
cp output/csv/conditions.csv ~/Desktop/clinical-cohort-analytics/data/
cp output/csv/medications.csv ~/Desktop/clinical-cohort-analytics/data/
cp output/csv/observations.csv ~/Desktop/clinical-cohort-analytics/data/

Why only five files?

Synthea generates over a dozen CSV tables, but your analysis pipeline only needs these five core tables: patients, encounters, conditions, medications, and observations. Keeping the dataset focused makes it easier to learn pandas without drowning in tables you won't use.

• Confirm the files arrived in your project by running:

ls ~/Desktop/clinical-cohort-analytics/data/

You should see exactly five files: patients.csv, encounters.csv, conditions.csv, medications.csv, and observations.csv.

Don't see the CSV files?

If output/csv/ is empty, the CSV exporter was not enabled. Make sure you typed --exporter.csv.export=true exactly (no spaces around the =). Re-run the ./run_synthea command with the correct flag.

If you see an output/csv/ directory but only FHIR JSON files in output/fhir/, the CSV flag was not picked up. Double-check the exact spelling of the flag.

Help me fix my Synthea CSV export

Your synthetic patient data is ready. Next up, you'll load these CSV files into pandas DataFrames and start exploring what's inside them.

Load and Explore the Raw Data with pandas

You now have 1,000 synthetic patients and all their associated clinical records sitting in your data/ folder. Raw CSV files on disk are useful, but they're not something you can query, filter, or compute on yet.

In this step, you'll load those files into pandas, Python's most popular data analysis library. Think of pandas as SQL for Python. A DataFrame is a table, columns are columns, and rows are rows. Once your data lives in DataFrames, you can run the same kinds of queries you'd write in SQL.

In this step, get ready to:

• Launch a Jupyter notebook and import pandas.
• Load all 5 clinical CSV files into DataFrames and explore their structure.
• Parse date columns into proper datetime objects for calculations.

Create a Jupyter notebook and import pandas

Jupyter Notebook is an interactive coding environment where you write and run code in small cells. Each cell shows its output immediately below, which makes it perfect for data exploration.

• Create a notebooks/ directory inside your project and launch the notebook by running these commands in your terminal:

mkdir notebooks
jupyter notebook notebooks/01_explore.ipynb

What does this command do?

The first command creates a notebooks/ folder to keep your notebooks organized and separate from your source code.

The second command launches Jupyter and opens (or creates) a notebook file called 01_explore.ipynb. Your browser will open automatically with the notebook editor.

🙋‍♀️ Browser didn't open?

Check your terminal output. Jupyter prints a URL starting with http://localhost:8888. Copy and paste that full URL (including the token) into your browser manually.

If you see a "command not found" error, make sure your conda environment is still activated. You should see (healthcare-analytics) at the start of your terminal prompt.

Help me fix my Jupyter Notebook launch issue

Your notebook is now open in the browser. You'll see an empty cell where you can type Python code.

• In the first cell of your notebook, type the following import statement and press Shift+Enter to run the cell:

# Import pandas - the core library for data analysis in Python
import pandas as pd

What does import pandas as pd mean?

This loads the pandas library into your notebook and gives it the short nickname pd. From now on, every time you call a pandas function, you'll write pd.something() instead of pandas.something().

This pd shorthand is a universal convention in the data community. You'll see it in every pandas tutorial and codebase.

• After pressing Shift+Enter, the cell should run without any output (no error messages). A new empty cell appears below.

Load all 5 CSV files into DataFrames

Now you'll bring your clinical data into Python. The function pd.read_csv() reads a CSV file and returns a DataFrame. This is equivalent to loading a table you can query.

• In the next empty cell, load all 5 CSV files by typing this code and pressing Shift+Enter:

# Load the 5 core clinical tables from Synthea CSV output
patients_df = pd.read_csv('../data/patients.csv')
encounters_df = pd.read_csv('../data/encounters.csv')
conditions_df = pd.read_csv('../data/conditions.csv')
medications_df = pd.read_csv('../data/medications.csv')
observations_df = pd.read_csv('../data/observations.csv')

What does this code do?

Each line reads one CSV file into a pandas DataFrame. The ../data/ path goes up one directory (out of notebooks/) then into the data/ folder where your CSVs live.

The _df suffix is a naming convention that reminds you the variable holds a DataFrame. It's like naming your SQL tables clearly.

• The cell should run without errors. No output appears because assignment statements don't produce visible output in Jupyter.

Seeing a FileNotFoundError?

This means the path to your CSV files is wrong. Double-check that your notebook is inside the notebooks/ folder and that your CSV files are in a sibling data/ folder.

Your project should look like this: clinical-cohort-analytics/data/patients.csv and clinical-cohort-analytics/notebooks/01_explore.ipynb.

Help me fix my FileNotFoundError when loading CSV files

Now let's explore what's inside these DataFrames. You'll use five key methods that give you a quick overview of any dataset.

• In the next cell, check the size of each DataFrame by running:

# Check row and column counts for each table
# .shape returns (number_of_rows, number_of_columns)
print('patients:', patients_df.shape)
print('encounters:', encounters_df.shape)
print('conditions:', conditions_df.shape)
print('medications:', medications_df.shape)
print('observations:', observations_df.shape)

What does .shape tell you?

The .shape attribute returns a tuple of (rows, columns). If you see (1000, 28) for patients, that means 1,000 rows (one per patient) and 28 columns.

This is your equivalent of SELECT COUNT(*) FROM patients in SQL. It tells you instantly how much data you're working with.

• You should see output showing each table name followed by its dimensions. The patients table should show approximately 1,000 rows.

• In the next cell, inspect the column types of the patients table:

# Check data types for each column in the patients table
# This is like checking the schema of a SQL table
patients_df.dtypes

What does .dtypes show?

The .dtypes attribute lists every column and its data type. You'll notice that date columns (like BirthDate) show as object instead of a proper date type. You'll fix that in the next substep.

Common types you'll see: object (text/strings), float64 (decimal numbers), int64 (whole numbers).

• In the next cell, preview the first few rows of the patients table:

# Preview the first 5 rows of the patients table
# This gives you a quick look at actual data values
patients_df.head()

What does .head() show?

The .head() method returns the first 5 rows of the DataFrame, displayed as a nicely formatted table in Jupyter. This is like SELECT * FROM patients LIMIT 5 in SQL.

Scrolling through the output, you'll see patient UUIDs, names, dates, demographics, and financial columns.

• In the next cell, get summary statistics for the numeric columns:

# Summary statistics for numeric columns in the patients table
# Shows count, mean, std, min, max, and quartiles
patients_df.describe()

What does .describe() reveal?

The .describe() method computes count, mean, standard deviation, min, max, and percentile values for every numeric column. For the patients table, you'll see stats for Healthcare_Expenses, Healthcare_Coverage, and Income.

• In the next cell, check for missing values across all 5 DataFrames:

# Count missing values in each column for every table
# .isna() marks True for missing values, .sum() counts them per column
print('=== Patients Missing Values ===')
print(patients_df.isna().sum())
print('\n=== Encounters Missing Values ===')
print(encounters_df.isna().sum())
print('\n=== Conditions Missing Values ===')
print(conditions_df.isna().sum())
print('\n=== Medications Missing Values ===')
print(medications_df.isna().sum())
print('\n=== Observations Missing Values ===')
print(observations_df.isna().sum())

Why check for missing values?

Missing data can break calculations or skew your results. The .isna().sum() pattern counts how many null values exist in each column.

You'll likely see some missing values in columns like DeathDate (most patients are alive), Stop dates in conditions (ongoing conditions have no end date), and ReasonCode/ReasonDescription in encounters.

• Review the output. Columns showing 0 have no missing data. Columns with non-zero counts will need attention during the cleaning step.

Parse date columns into datetime objects

When pandas loaded your CSV files, it treated date columns like BirthDate and Start as plain text strings. You need to convert them to proper datetime objects so you can perform calculations like "how many days between admission and discharge."

• In the next cell, convert the key date columns across your DataFrames:

# Convert date columns from strings to proper datetime objects
# This enables date arithmetic (e.g., calculating length of stay)
patients_df['BirthDate'] = pd.to_datetime(patients_df['BirthDate'])
patients_df['DeathDate'] = pd.to_datetime(patients_df['DeathDate'])

encounters_df['Start'] = pd.to_datetime(encounters_df['Start'])
encounters_df['Stop'] = pd.to_datetime(encounters_df['Stop'])

conditions_df['Start'] = pd.to_datetime(conditions_df['Start'])
conditions_df['Stop'] = pd.to_datetime(conditions_df['Stop'])

medications_df['Start'] = pd.to_datetime(medications_df['Start'])
medications_df['Stop'] = pd.to_datetime(medications_df['Stop'])

observations_df['Date'] = pd.to_datetime(observations_df['Date'])

Why do date types matter?

With plain text dates, Python treats "2024-03-15" as just a string of characters. You can't subtract one string from another to get a duration.

Once converted to datetime, you can do math: encounters_df['Stop'] - encounters_df['Start'] gives you the exact length of each encounter. You'll use this to calculate length of stay in a later step.

• Verify the conversion worked by running this in the next cell:

# Confirm date columns are now datetime64 type instead of object
print(patients_df[['BirthDate', 'DeathDate']].dtypes)
print(encounters_df[['Start', 'Stop']].dtypes)

• You should see datetime64[ns] (or similar) as the type for each column instead of object.

Seeing a ParserError or mixed format warning?

Some date columns may contain mixed formats. If you see a warning, add format='mixed' to your pd.to_datetime() call. For example: pd.to_datetime(encounters_df['Start'], format='mixed').

Help me fix a date parsing error in my notebook

• Save your notebook by pressing Cmd+S (macOS) or Ctrl+S (Windows) in the Jupyter browser tab.

✔️ Awesome, I've got everything!

Great. Make sure you've saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here is the complete notebook code across all cells:

# Cell 1: Import pandas
import pandas as pd

# Cell 2: Load all 5 CSV files
patients_df = pd.read_csv('../data/patients.csv')
encounters_df = pd.read_csv('../data/encounters.csv')
conditions_df = pd.read_csv('../data/conditions.csv')
medications_df = pd.read_csv('../data/medications.csv')
observations_df = pd.read_csv('../data/observations.csv')

# Cell 3: Check shapes
print('patients:', patients_df.shape)
print('encounters:', encounters_df.shape)
print('conditions:', conditions_df.shape)
print('medications:', medications_df.shape)
print('observations:', observations_df.shape)

# Cell 4: Check data types
patients_df.dtypes

# Cell 5: Preview first rows
patients_df.head()

# Cell 6: Summary statistics
patients_df.describe()

# Cell 7: Missing values
print('=== Patients Missing Values ===')
print(patients_df.isna().sum())
print('\n=== Encounters Missing Values ===')
print(encounters_df.isna().sum())
print('\n=== Conditions Missing Values ===')
print(conditions_df.isna().sum())
print('\n=== Medications Missing Values ===')
print(medications_df.isna().sum())
print('\n=== Observations Missing Values ===')
print(observations_df.isna().sum())

# Cell 8: Parse date columns
patients_df['BirthDate'] = pd.to_datetime(patients_df['BirthDate'])
patients_df['DeathDate'] = pd.to_datetime(patients_df['DeathDate'])

encounters_df['Start'] = pd.to_datetime(encounters_df['Start'])
encounters_df['Stop'] = pd.to_datetime(encounters_df['Stop'])

conditions_df['Start'] = pd.to_datetime(conditions_df['Start'])
conditions_df['Stop'] = pd.to_datetime(conditions_df['Stop'])

medications_df['Start'] = pd.to_datetime(medications_df['Start'])
medications_df['Stop'] = pd.to_datetime(medications_df['Stop'])

observations_df['Date'] = pd.to_datetime(observations_df['Date'])

# Cell 9: Verify conversion
print(patients_df[['BirthDate', 'DeathDate']].dtypes)
print(encounters_df[['Start', 'Stop']].dtypes)

Your clinical data is now loaded, explored, and date-ready. Next up, you'll clean these DataFrames, handle missing values, and join the tables together into a unified analysis dataset.

Clean, Transform, and Join Clinical Tables

In the previous step, you loaded raw Synthea CSV files into pandas DataFrames and explored their shapes, data types, and missing values. Now you know what you're working with, but the data isn't ready for analysis yet.

Raw clinical data almost always needs cleaning before you can compute meaningful metrics. In this step, you'll handle missing values, standardize column names, compute new columns like patient age and length of stay, and join tables together into one unified DataFrame.

In this step, get ready to:

• Clean each DataFrame by handling missing values and standardizing column names.
• Compute derived columns for patient age and encounter length of stay.
• Join tables on the Patient UUID to create a unified analysis DataFrame.

Clean each DataFrame

Before computing any metrics, you need to remove rows with missing critical identifiers and standardize column names across tables. This is your ETL step, just like preparing staging tables before loading a data warehouse.

• Open the Command Palette with Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows).
• Type Create: New Jupyter Notebook and press Enter.
• Save the notebook as 02_clean_transform.ipynb inside your notebooks/ folder using File > Save As.
• Select your .venv kernel by clicking Select Kernel in the top right corner of the notebook.

Why a new notebook?

Separating exploration from transformation keeps each notebook focused on one task. Your 01_explore.ipynb is a scratchpad for understanding the data. This new notebook is where you build the clean, repeatable pipeline.

• In the first cell of your new notebook, load the data and parse date columns by pasting this code:

import pandas as pd
from datetime import date

# Load raw CSV files into DataFrames
patients_df = pd.read_csv('../data/patients.csv')
encounters_df = pd.read_csv('../data/encounters.csv')
conditions_df = pd.read_csv('../data/conditions.csv')
medications_df = pd.read_csv('../data/medications.csv')
observations_df = pd.read_csv('../data/observations.csv')

# Parse date columns into proper datetime objects
patients_df['BirthDate'] = pd.to_datetime(patients_df['BirthDate'])
patients_df['DeathDate'] = pd.to_datetime(patients_df['DeathDate'])
encounters_df['Start'] = pd.to_datetime(encounters_df['Start'])
encounters_df['Stop'] = pd.to_datetime(encounters_df['Stop'])
conditions_df['Start'] = pd.to_datetime(conditions_df['Start'])
conditions_df['Stop'] = pd.to_datetime(conditions_df['Stop'])
medications_df['Start'] = pd.to_datetime(medications_df['Start'])
medications_df['Stop'] = pd.to_datetime(medications_df['Stop'])
observations_df['Date'] = pd.to_datetime(observations_df['Date'])

What does this code do?

This cell re-loads all five CSVs and converts their date columns from raw strings into datetime objects. You need proper datetime types so that subtraction and date arithmetic work correctly in later cells.

• Run the cell by pressing Shift+Enter.
• In a new cell, clean each DataFrame by handling missing values and renaming columns. Paste this code:

# Drop rows where critical identifiers are missing
patients_df = patients_df.dropna(subset=['Id', 'BirthDate', 'Gender', 'Race'])
encounters_df = encounters_df.dropna(subset=['Id', 'Start', 'Stop', 'Patient', 'EncounterClass'])
conditions_df = conditions_df.dropna(subset=['Patient', 'Encounter', 'Code'])
medications_df = medications_df.dropna(subset=['Patient', 'Encounter', 'Code'])
observations_df = observations_df.dropna(subset=['Patient', 'Encounter', 'Code'])

# Fill optional missing values with NaT (Not a Time) for dates
patients_df['DeathDate'] = patients_df['DeathDate'].fillna(pd.NaT)
conditions_df['Stop'] = conditions_df['Stop'].fillna(pd.NaT)
medications_df['Stop'] = medications_df['Stop'].fillna(pd.NaT)

# Rename primary key columns for clarity when joining later
patients_df = patients_df.rename(columns={'Id': 'PatientId'})
encounters_df = encounters_df.rename(columns={'Id': 'EncounterId'})

# Confirm row counts after cleaning
print(f"Patients: {patients_df.shape[0]} rows")
print(f"Encounters: {encounters_df.shape[0]} rows")
print(f"Conditions: {conditions_df.shape[0]} rows")
print(f"Medications: {medications_df.shape[0]} rows")
print(f"Observations: {observations_df.shape[0]} rows")

What does this code do?

.dropna(subset=[...]) removes any row where the listed columns have missing values. These are critical identifiers. A condition record without a Patient UUID is useless for joins.

.fillna(pd.NaT) explicitly fills missing date fields with pandas' "Not a Time" value. This prevents errors when you later do date arithmetic on those columns.

.rename() gives the primary key columns distinct names (PatientId, EncounterId) so they won't collide when you join tables that both have a generic Id column.

• Run the cell. You should see row counts for all five DataFrames printed below the cell.

Seeing a KeyError?

If you see a KeyError on a column name like Id or Patient, your CSV files may not have loaded correctly. Confirm that the files in your data/ folder are the ones Synthea generated (not empty or truncated).

help me debug a KeyError when cleaning my Synthea DataFrames

Compute derived columns

Raw dates aren't directly useful for analysis. You need to derive patient age (in years) and encounter length of stay (in days) so you can group and compare them later.

• In a new cell, compute the age and length of stay columns by pasting this code:

# Calculate patient age in years from BirthDate
today = date.today()
patients_df['Age'] = patients_df['BirthDate'].apply(
    lambda x: today.year - x.year - ((today.month, today.day) < (x.month, x.day))
)

# Calculate encounter length of stay in days
encounters_df['LOS_days'] = (
    encounters_df['Stop'] - encounters_df['Start']
).dt.total_seconds() / 86400

# Preview the new columns
print("Patient ages:")
print(patients_df[['PatientId', 'BirthDate', 'Age']].head())
print("\nEncounter LOS:")
print(encounters_df[['EncounterId', 'Start', 'Stop', 'LOS_days']].head())

What does this code do?

The Age column uses a lambda function that subtracts the birth year from today's year, then adjusts down by one if the patient's birthday hasn't happened yet this year. This gives you an accurate integer age.

The LOS_days column subtracts the encounter Start from Stop to get a timedelta, then converts to total seconds divided by 86400 (seconds in a day). This produces a decimal number of days.

• Run the cell. You should see a preview table showing PatientId, BirthDate, and Age columns, followed by EncounterId, Start, Stop, and LOS_days columns.

Seeing NaN in LOS_days?

If some rows show NaN for LOS_days, those encounters have a missing Stop date. The .dropna() in the cleaning step should have caught these. Make sure you ran the cleaning cell before this one.

help me fix NaN values appearing in my LOS_days column

Join tables using pd.merge()

Right now your patient demographics and encounter details live in separate DataFrames. To analyze metrics like length of stay by gender or age, you need them in a single table.

What is pd.merge()?

pd.merge() is the pandas equivalent of a SQL JOIN. You specify a left DataFrame, a right DataFrame, and the key columns to match on. The result is a new DataFrame with columns from both tables aligned by their shared key.

• In a new cell, join the encounters and patients tables by pasting this code:

# Join encounters with patient demographics on Patient UUID
# This is exactly like: SELECT * FROM encounters JOIN patients ON encounters.Patient = patients.PatientId
analysis_df = pd.merge(
    encounters_df,
    patients_df[['PatientId', 'BirthDate', 'Age', 'Gender', 'Race', 'Ethnicity']],
    left_on='Patient',
    right_on='PatientId',
    how='left'
)

# Check the result
print(f"Unified analysis_df shape: {analysis_df.shape}")
print(f"Columns: {analysis_df.columns.tolist()}")
analysis_df.head()

What does this code do?

pd.merge() joins encounters_df (left) with selected columns from patients_df (right) where the encounter's Patient column matches PatientId.

The how='left' parameter means every encounter row is kept, even if there's no matching patient (a LEFT JOIN in SQL terms). Selecting only the columns you need from patients_df keeps the result focused and avoids cluttering it with address or income data you won't use.

• Run the cell. You should see the shape of analysis_df (same number of rows as encounters_df, with additional columns for Age, Gender, Race, and Ethnicity).
• Save your notebook with Cmd+S (macOS) or Ctrl+S (Windows).

analysis_df has zero rows?

If your merged DataFrame is empty, the join keys don't match. Check that encounters_df['Patient'] contains the same UUID format as patients_df['PatientId']. Run encounters_df['Patient'].head() and patients_df['PatientId'].head() in a scratch cell to compare them visually.

help me debug why my pd.merge produced an empty DataFrame

✔️ Awesome, I've got everything!

Great. Double check you've saved 02_clean_transform.ipynb and that all four cells ran without errors.

ⓧ I'd like to double check the full code

Here is the complete notebook code for 02_clean_transform.ipynb. Each section below represents one cell:

import pandas as pd
from datetime import date

# Load raw CSV files into DataFrames
patients_df = pd.read_csv('../data/patients.csv')
encounters_df = pd.read_csv('../data/encounters.csv')
conditions_df = pd.read_csv('../data/conditions.csv')
medications_df = pd.read_csv('../data/medications.csv')
observations_df = pd.read_csv('../data/observations.csv')

# Parse date columns into proper datetime objects
patients_df['BirthDate'] = pd.to_datetime(patients_df['BirthDate'])
patients_df['DeathDate'] = pd.to_datetime(patients_df['DeathDate'])
encounters_df['Start'] = pd.to_datetime(encounters_df['Start'])
encounters_df['Stop'] = pd.to_datetime(encounters_df['Stop'])
conditions_df['Start'] = pd.to_datetime(conditions_df['Start'])
conditions_df['Stop'] = pd.to_datetime(conditions_df['Stop'])
medications_df['Start'] = pd.to_datetime(medications_df['Start'])
medications_df['Stop'] = pd.to_datetime(medications_df['Stop'])
observations_df['Date'] = pd.to_datetime(observations_df['Date'])

# Drop rows where critical identifiers are missing
patients_df = patients_df.dropna(subset=['Id', 'BirthDate', 'Gender', 'Race'])
encounters_df = encounters_df.dropna(subset=['Id', 'Start', 'Stop', 'Patient', 'EncounterClass'])
conditions_df = conditions_df.dropna(subset=['Patient', 'Encounter', 'Code'])
medications_df = medications_df.dropna(subset=['Patient', 'Encounter', 'Code'])
observations_df = observations_df.dropna(subset=['Patient', 'Encounter', 'Code'])

# Fill optional missing values with NaT (Not a Time) for dates
patients_df['DeathDate'] = patients_df['DeathDate'].fillna(pd.NaT)
conditions_df['Stop'] = conditions_df['Stop'].fillna(pd.NaT)
medications_df['Stop'] = medications_df['Stop'].fillna(pd.NaT)

# Rename primary key columns for clarity when joining later
patients_df = patients_df.rename(columns={'Id': 'PatientId'})
encounters_df = encounters_df.rename(columns={'Id': 'EncounterId'})

# Confirm row counts after cleaning
print(f"Patients: {patients_df.shape[0]} rows")
print(f"Encounters: {encounters_df.shape[0]} rows")
print(f"Conditions: {conditions_df.shape[0]} rows")
print(f"Medications: {medications_df.shape[0]} rows")
print(f"Observations: {observations_df.shape[0]} rows")

# Calculate patient age in years from BirthDate
today = date.today()
patients_df['Age'] = patients_df['BirthDate'].apply(
    lambda x: today.year - x.year - ((today.month, today.day) < (x.month, x.day))
)

# Calculate encounter length of stay in days
encounters_df['LOS_days'] = (
    encounters_df['Stop'] - encounters_df['Start']
).dt.total_seconds() / 86400

# Preview the new columns
print("Patient ages:")
print(patients_df[['PatientId', 'BirthDate', 'Age']].head())
print("\nEncounter LOS:")
print(encounters_df[['EncounterId', 'Start', 'Stop', 'LOS_days']].head())

# Join encounters with patient demographics on Patient UUID
# This is exactly like: SELECT * FROM encounters JOIN patients ON encounters.Patient = patients.PatientId
analysis_df = pd.merge(
    encounters_df,
    patients_df[['PatientId', 'BirthDate', 'Age', 'Gender', 'Race', 'Ethnicity']],
    left_on='Patient',
    right_on='PatientId',
    how='left'
)

# Check the result
print(f"Unified analysis_df shape: {analysis_df.shape}")
print(f"Columns: {analysis_df.columns.tolist()}")
analysis_df.head()

Your data is now clean, enriched with derived columns, and unified into a single analysis_df DataFrame. Next up, you'll use this DataFrame to build patient cohorts and compute real clinical metrics like condition prevalence and readmission rates.

Build Patient Cohorts and Compute Clinical Metrics

Your cleaned, joined DataFrames are ready. Now it is time to ask clinically meaningful questions of the data.

In this step, you will segment patients into diagnosis-based cohorts using SNOMED-CT codes, then compute aggregate metrics that mirror the kind of analysis a health data team produces: condition prevalence, average length of stay, and 30-day readmission rates.

In this step, get ready to:

• Define patient cohorts by filtering on SNOMED-CT diagnosis codes.
• Compute condition prevalence rates and average length of stay using groupby aggregations.
• Implement a 30-day all-cause readmission proxy using date arithmetic.

Define patient cohorts using SNOMED-CT codes

A cohort is a group of patients who share a common characteristic. In clinical analytics, cohorts are often defined by diagnosis codes.

The conditions table uses SNOMED-CT codes to identify diagnoses. Filtering for a specific code is exactly like a SQL WHERE clause.

• Create a new Jupyter notebook for this step by running:

jupyter notebook

• In the Jupyter interface, click New, then Python 3.
• Rename the notebook to 03_cohorts_metrics by clicking the title at the top.
• Save the notebook into the notebooks/ directory.

• In the first cell, load your data and parse date columns by pasting:

import pandas as pd

# Load the clinical tables we need for cohort analysis
patients = pd.read_csv('../data/patients.csv')
encounters = pd.read_csv('../data/encounters.csv')
conditions = pd.read_csv('../data/conditions.csv')

# Parse date columns into proper datetime objects
patients['BirthDate'] = pd.to_datetime(patients['BirthDate'])
encounters['Start'] = pd.to_datetime(encounters['Start'])
encounters['Stop'] = pd.to_datetime(encounters['Stop'])
conditions['Start'] = pd.to_datetime(conditions['Start'])

What does this code do?

This cell reloads the raw CSVs and parses dates, giving you a clean starting point independent of the previous notebook. Each notebook in the project is self-contained.

• Run the cell with Shift+Enter.

• In a new cell, define your three cohorts by filtering on SNOMED-CT codes:

# SNOMED-CT codes for our target conditions
DIABETES_CODE = '44054006'       # Type 2 diabetes mellitus
HYPERTENSION_CODE = '59621000'   # Essential hypertension
COPD_CODE = '185086009'          # Chronic obstructive bronchitis

# Build cohorts: unique patient IDs with each diagnosis
diabetes_cohort = conditions[conditions['Code'] == DIABETES_CODE]['Patient'].unique()
hypertension_cohort = conditions[conditions['Code'] == HYPERTENSION_CODE]['Patient'].unique()
copd_cohort = conditions[conditions['Code'] == COPD_CODE]['Patient'].unique()

# Print cohort sizes
print(f"Diabetes cohort: {len(diabetes_cohort)} patients")
print(f"Hypertension cohort: {len(hypertension_cohort)} patients")
print(f"COPD cohort: {len(copd_cohort)} patients")
print(f"Total unique patients: {patients.shape[0]}")

What does this code do?

Each line filters the conditions DataFrame to rows where the Code column matches a specific SNOMED-CT code. The ['Patient'].unique() part extracts the unique patient UUIDs who have that diagnosis.

This is equivalent to SQL: SELECT DISTINCT Patient FROM conditions WHERE Code = '44054006'.

• Run the cell with Shift+Enter.
• You should see three lines showing the number of patients in each cohort, plus the total patient count.

Seeing 0 patients in a cohort?

Check that the code values match exactly. SNOMED-CT codes are stored as strings in the Code column. Run conditions['Code'].dtype to confirm the column type is object (string).

If the column type is numeric, wrap the code in a number comparison instead: conditions[conditions['Code'] == 44054006].

Why is my cohort showing 0 patients even though I used the correct SNOMED-CT code?

Compute condition prevalence and average length of stay

With cohorts defined, you can now compute aggregate metrics. The .groupby() method in pandas is the direct equivalent of SQL's GROUP BY clause.

• In a new cell, compute condition prevalence stratified by gender and race:

# Merge conditions with patient demographics for stratification
conditions_demo = conditions.merge(
    patients[['Id', 'Gender', 'Race']],
    left_on='Patient',
    right_on='Id'
)

# Count how many patients have each condition, grouped by gender
prevalence_by_gender = (
    conditions_demo
    .groupby(['Description', 'Gender'])
    .size()
    .reset_index(name='Count')
    .sort_values('Count', ascending=False)
)

print("Top 10 conditions by gender:")
print(prevalence_by_gender.head(10))

# Count how many patients have each condition, grouped by race
prevalence_by_race = (
    conditions_demo
    .groupby(['Description', 'Race'])
    .size()
    .reset_index(name='Count')
    .sort_values('Count', ascending=False)
)

print("\nTop 10 conditions by race:")
print(prevalence_by_race.head(10))

What does this code do?

First, .merge() joins conditions with patient demographics so you have Gender and Race alongside each diagnosis row. This mirrors a SQL JOIN on the patient UUID.

Then .groupby(['Description', 'Gender']).size() counts how many rows exist for each condition-gender combination. The .value_counts() method does the same thing but .groupby().size() gives you more control over the output structure.

• Run the cell. You should see two tables showing condition counts broken down by gender and race.

• In a new cell, compute average length of stay by encounter class:

# Calculate length of stay in days for each encounter
encounters['LOS_days'] = (encounters['Stop'] - encounters['Start']).dt.total_seconds() / 86400

# Average LOS by encounter class (inpatient, ambulatory, emergency, wellness)
avg_los_by_class = encounters.groupby('EncounterClass')['LOS_days'].mean()

print("Average Length of Stay (days) by Encounter Class:")
print(avg_los_by_class.round(2))

What does this code do?

The LOS_days column subtracts the encounter start from the stop timestamp and converts the result to days. Dividing by 86400 converts seconds to days.

Then .groupby('EncounterClass')['LOS_days'].mean() produces one average per class: inpatient, ambulatory, emergency, wellness, and urgentcare.

• Run the cell. You should see average LOS values for each encounter class, with inpatient stays typically showing the longest duration.

Seeing NaN or unexpected values?

If LOS_days shows NaN for some rows, you likely have encounters with missing Stop dates (ongoing encounters). These are expected. The .mean() method automatically skips NaN values.

Why am I seeing NaN in my length of stay calculations?

Implement the 30-day readmission proxy

Hospital readmissions within 30 days are a key quality metric. You will build a simplified proxy: for each inpatient discharge, check whether the same patient has another inpatient admission within 30 days.

What is a readmission proxy?

Real readmission measures (like those used by CMS) have specific clinical exclusions and risk adjustments. Your proxy is a simplified version that flags any inpatient encounter followed by another inpatient encounter within 30 days for the same patient.

This is a common starting point for analytics teams before layering on clinical business rules.

• In a new cell, implement the 30-day readmission logic:

# Filter to inpatient encounters only
inpatient = encounters[encounters['EncounterClass'] == 'inpatient'].copy()

# Sort by patient and discharge date so we can compare consecutive stays
inpatient = inpatient.sort_values(['Patient', 'Stop'])

# For each patient, get the start date of their NEXT inpatient encounter
inpatient['next_admission'] = inpatient.groupby('Patient')['Start'].shift(-1)

# Calculate days between discharge and next admission
inpatient['days_to_readmit'] = (
    inpatient['next_admission'] - inpatient['Stop']
).dt.days

# Flag encounters where readmission happened within 30 days
inpatient['readmitted_30d'] = inpatient['days_to_readmit'] <= 30

# Summary statistics
total_inpatient = len(inpatient)
readmitted_count = inpatient['readmitted_30d'].sum()
readmission_rate = readmitted_count / total_inpatient * 100

print(f"Total inpatient encounters: {total_inpatient}")
print(f"Readmitted within 30 days: {readmitted_count}")
print(f"30-day readmission rate: {readmission_rate:.1f}%")

What does this code do?

After filtering to inpatient encounters and sorting by patient and discharge date, .groupby('Patient')['Start'].shift(-1) grabs the start date of the next encounter for the same patient. The -1 means "look forward one row within each group."

Subtracting the current Stop from the next Start gives the gap in days. If that gap is 30 or fewer, the readmitted_30d column is set to True.

• Run the cell. You should see the total inpatient encounters, how many had a readmission within 30 days, and the overall readmission rate as a percentage.

Readmission rate seems too high or too low?

Synthea generates synthetic data with artificial distributions. A readmission rate between 5% and 30% is typical depending on the generated population. Extremely low rates (under 1%) may indicate the sort order is wrong.

Verify the sort by running inpatient[['Patient', 'Stop', 'next_admission']].head(10) to confirm that next_admission dates are always after Stop dates for the same patient.

Help me debug my 30-day readmission calculation.

• Save the notebook by pressing Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Double check you have saved 03_cohorts_metrics.ipynb and all cells ran without errors.

ⓧ I'd like to double check the full code

Here is the complete notebook code across all cells:

import pandas as pd

# Load the clinical tables we need for cohort analysis
patients = pd.read_csv('../data/patients.csv')
encounters = pd.read_csv('../data/encounters.csv')
conditions = pd.read_csv('../data/conditions.csv')

# Parse date columns into proper datetime objects
patients['BirthDate'] = pd.to_datetime(patients['BirthDate'])
encounters['Start'] = pd.to_datetime(encounters['Start'])
encounters['Stop'] = pd.to_datetime(encounters['Stop'])
conditions['Start'] = pd.to_datetime(conditions['Start'])

# SNOMED-CT codes for our target conditions
DIABETES_CODE = '44054006'       # Type 2 diabetes mellitus
HYPERTENSION_CODE = '59621000'   # Essential hypertension
COPD_CODE = '185086009'          # Chronic obstructive bronchitis

# Build cohorts: unique patient IDs with each diagnosis
diabetes_cohort = conditions[conditions['Code'] == DIABETES_CODE]['Patient'].unique()
hypertension_cohort = conditions[conditions['Code'] == HYPERTENSION_CODE]['Patient'].unique()
copd_cohort = conditions[conditions['Code'] == COPD_CODE]['Patient'].unique()

# Print cohort sizes
print(f"Diabetes cohort: {len(diabetes_cohort)} patients")
print(f"Hypertension cohort: {len(hypertension_cohort)} patients")
print(f"COPD cohort: {len(copd_cohort)} patients")
print(f"Total unique patients: {patients.shape[0]}")

# Merge conditions with patient demographics for stratification
conditions_demo = conditions.merge(
    patients[['Id', 'Gender', 'Race']],
    left_on='Patient',
    right_on='Id'
)

# Count how many patients have each condition, grouped by gender
prevalence_by_gender = (
    conditions_demo
    .groupby(['Description', 'Gender'])
    .size()
    .reset_index(name='Count')
    .sort_values('Count', ascending=False)
)

print("Top 10 conditions by gender:")
print(prevalence_by_gender.head(10))

# Count how many patients have each condition, grouped by race
prevalence_by_race = (
    conditions_demo
    .groupby(['Description', 'Race'])
    .size()
    .reset_index(name='Count')
    .sort_values('Count', ascending=False)
)

print("\nTop 10 conditions by race:")
print(prevalence_by_race.head(10))

# Calculate length of stay in days for each encounter
encounters['LOS_days'] = (encounters['Stop'] - encounters['Start']).dt.total_seconds() / 86400

# Average LOS by encounter class (inpatient, ambulatory, emergency, wellness)
avg_los_by_class = encounters.groupby('EncounterClass')['LOS_days'].mean()

print("Average Length of Stay (days) by Encounter Class:")
print(avg_los_by_class.round(2))

# Filter to inpatient encounters only
inpatient = encounters[encounters['EncounterClass'] == 'inpatient'].copy()

# Sort by patient and discharge date so we can compare consecutive stays
inpatient = inpatient.sort_values(['Patient', 'Stop'])

# For each patient, get the start date of their NEXT inpatient encounter
inpatient['next_admission'] = inpatient.groupby('Patient')['Start'].shift(-1)

# Calculate days between discharge and next admission
inpatient['days_to_readmit'] = (
    inpatient['next_admission'] - inpatient['Stop']
).dt.days

# Flag encounters where readmission happened within 30 days
inpatient['readmitted_30d'] = inpatient['days_to_readmit'] <= 30

# Summary statistics
total_inpatient = len(inpatient)
readmitted_count = inpatient['readmitted_30d'].sum()
readmission_rate = readmitted_count / total_inpatient * 100

print(f"Total inpatient encounters: {total_inpatient}")
print(f"Readmitted within 30 days: {readmitted_count}")
print(f"30-day readmission rate: {readmission_rate:.1f}%")

You now have three defined patient cohorts, prevalence rates stratified by demographics, average length of stay figures, and a 30-day readmission flag. Next up, you will turn these numbers into publication-quality visualizations with Matplotlib.

Visualize Clinical Insights with Matplotlib

You now have cohorts defined, condition prevalence calculated, length-of-stay metrics computed, and a 30-day readmission proxy in place.

Numbers in a DataFrame tell the story, but a chart communicates it instantly. In this step, you will turn your clinical metrics into four publication-quality visualizations using Matplotlib. Each chart gets saved as a PNG file that you can drop into reports, presentations, or your GitHub README.

In this step, get ready to:

• Learn the matplotlib figure/axes pattern and save charts as PNG files.
• Create a horizontal bar chart of the top 15 conditions and box plots of lab values by cohort.
• Plot a length-of-stay histogram and a readmission rates comparison chart.

Set up the visualization notebook and learn the matplotlib pattern

Matplotlib uses a figure/axes model. A figure is the overall canvas, and axes are the individual plots inside it. You create both at once with plt.subplots(), then call methods on the axes to draw your chart.

• Create a new Jupyter notebook called 04_visualize.ipynb inside the notebooks/ folder.
• In the first cell, import your libraries and create the output directory by running this code:

import pandas as pd
import matplotlib.pyplot as plt
import os

# Create output directory for saved visualizations
os.makedirs('../output', exist_ok=True)

What does this code do?

import matplotlib.pyplot as plt brings in the plotting module under the alias plt. This is the standard convention you will see in every matplotlib tutorial.

os.makedirs('../output', exist_ok=True) creates the output/ directory at the project root. The exist_ok=True flag means it won't error if the folder already exists.

• In the next cell, load your clinical data and prepare the essentials for charting by running this code:

# Load CSV files
patients = pd.read_csv('../data/patients.csv')
encounters = pd.read_csv('../data/encounters.csv')
conditions = pd.read_csv('../data/conditions.csv')
observations = pd.read_csv('../data/observations.csv')

# Parse date columns
encounters['Start'] = pd.to_datetime(encounters['Start'])
encounters['Stop'] = pd.to_datetime(encounters['Stop'])

# Calculate inpatient length of stay in days
inpatient = encounters[encounters['EncounterClass'] == 'inpatient'].copy()
inpatient['LOS_days'] = (inpatient['Stop'] - inpatient['Start']).dt.total_seconds() / 86400

# Define cohorts using SNOMED-CT codes
diabetes_patients = conditions[conditions['Code'] == '44054006']['Patient'].unique()
hypertension_patients = conditions[conditions['Code'] == '59621000']['Patient'].unique()
copd_patients = conditions[conditions['Code'] == '185086009']['Patient'].unique()

Why reload the data?

Each Jupyter notebook runs independently. Variables from 03_cohorts_metrics.ipynb do not carry over, so you reload the CSVs and rebuild the cohort arrays here. This keeps each notebook self-contained and reproducible.

• Run both cells.
• Confirm you see no errors and that the output/ folder appears in your project directory.

Now try a quick dummy chart to see the figure/axes pattern in action. Add a new cell with this code:

# Quick test: create a figure and one axes, then plot dummy data
fig, ax = plt.subplots()
ax.bar(['A', 'B', 'C'], [10, 20, 15])
ax.set_title('Test Chart')
plt.show()

What is the figure/axes model?

plt.subplots() returns two objects: fig (the canvas) and ax (the plotting area). You call drawing methods on ax (like .bar(), .hist(), .boxplot()) and then save or display the fig.

When you need multiple plots side by side, pass a grid size: fig, axes = plt.subplots(1, 2) gives you two axes in a row. You will use this for the lab box plots later.

• Run the cell. You should see a simple bar chart with three bars labelled A, B, and C.

The pattern is always the same: create figure/axes, draw on the axes, save with plt.savefig(). Now apply it to real data.

• Create the top 15 conditions horizontal bar chart by adding a new cell with this code:

# Count unique patients per condition and get the top 15
condition_counts = conditions.groupby('Description')['Patient'].nunique()
top_15 = condition_counts.nlargest(15).sort_values()

# Create horizontal bar chart
fig, ax = plt.subplots(figsize=(10, 8))
ax.barh(top_15.index, top_15.values, color='steelblue')
ax.set_xlabel('Number of Patients')
ax.set_title('Top 15 Conditions by Patient Count')
plt.tight_layout()
plt.savefig('../output/top_conditions.png', dpi=150)
plt.show()

What does this code do?

• .groupby('Description')['Patient'].nunique() counts how many distinct patients have each condition. This is your prevalence measure.
• .nlargest(15).sort_values() takes the top 15 and sorts them ascending so the longest bars appear at the top of the horizontal chart.
• ax.barh() draws horizontal bars. plt.tight_layout() prevents label clipping. plt.savefig() writes the chart to disk as a PNG at 150 DPI.

• Run the cell. You should see a horizontal bar chart with condition names on the left axis and patient counts on the bottom axis.
• Confirm the file output/top_conditions.png exists in your project directory.

Chart looks blank or labels are cut off?

• If the chart is blank, make sure you ran the data-loading cell first. The conditions DataFrame must be in memory.
• If condition names are clipped on the left, check that you have plt.tight_layout() before plt.savefig().
• Help me debug my matplotlib bar chart.

Build lab value box plots stratified by cohort

Box plots reveal the distribution of a value across groups. You will compare HbA1c and systolic blood pressure across the diabetes, hypertension, COPD, and general patient cohorts.

• Filter the observations data and assign cohort labels by adding a new cell with this code:

# Filter observations for HbA1c (LOINC 4548-4) and Systolic BP (LOINC 8480-6)
hba1c = observations[observations['Code'] == '4548-4'].copy()
systolic_bp = observations[observations['Code'] == '8480-6'].copy()

# Convert Value column to numeric
hba1c['Value'] = pd.to_numeric(hba1c['Value'], errors='coerce')
systolic_bp['Value'] = pd.to_numeric(systolic_bp['Value'], errors='coerce')

# Assign cohort labels based on patient membership
def assign_cohort(patient_id):
    if patient_id in set(diabetes_patients):
        return 'Diabetes'
    elif patient_id in set(hypertension_patients):
        return 'Hypertension'
    elif patient_id in set(copd_patients):
        return 'COPD'
    else:
        return 'Other'

hba1c['Cohort'] = hba1c['Patient'].apply(assign_cohort)
systolic_bp['Cohort'] = systolic_bp['Patient'].apply(assign_cohort)

What does this code do?

• LOINC code 4548-4 identifies HbA1c measurements. Code 8480-6 identifies systolic blood pressure readings.
• pd.to_numeric(..., errors='coerce') safely converts the Value column to numbers, turning any non-numeric entries into NaN instead of crashing.
• The assign_cohort() function labels each observation row with the cohort its patient belongs to. Converting cohort arrays to sets speeds up the membership check.

• Create the side-by-side box plots by adding a new cell with this code:

# Create side-by-side box plots for HbA1c and Systolic BP
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

cohort_order = ['Diabetes', 'Hypertension', 'COPD', 'Other']

# HbA1c box plot
hba1c_data = [hba1c[hba1c['Cohort'] == c]['Value'].dropna() for c in cohort_order]
axes[0].boxplot(hba1c_data, labels=cohort_order)
axes[0].set_title('HbA1c by Cohort')
axes[0].set_ylabel('HbA1c (%)')

# Systolic BP box plot
bp_data = [systolic_bp[systolic_bp['Cohort'] == c]['Value'].dropna() for c in cohort_order]
axes[1].boxplot(bp_data, labels=cohort_order)
axes[1].set_title('Systolic Blood Pressure by Cohort')
axes[1].set_ylabel('mmHg')

plt.tight_layout()
plt.savefig('../output/lab_boxplots.png', dpi=150)
plt.show()

What does this code do?

• plt.subplots(1, 2) creates one row of two axes. You access them as axes[0] (left) and axes[1] (right).
• The list comprehension builds one list of values per cohort, which .boxplot() uses to draw one box per group.
• Each box shows the median, quartiles, and outliers for that cohort. This makes it easy to spot whether the diabetes cohort has elevated HbA1c compared to others.

• Run both cells. You should see two box plots side by side: HbA1c on the left and Systolic BP on the right, each with four boxes labelled by cohort.
• Confirm output/lab_boxplots.png exists in your project directory.

Box plots show no data or only one box?

• If a box is missing, that cohort may have no observations for that lab code. Check with hba1c['Cohort'].value_counts() to see how many rows belong to each cohort.
• If you see a warning about empty data, confirm the LOINC codes 4548-4 and 8480-6 match what appears in your observations data by running observations['Code'].unique().
• Help me debug my lab value box plots.

Plot LOS histogram and readmission rates comparison

Length of stay and readmission rates are two metrics that hospital administrators track closely. A histogram shows the distribution shape, and a grouped bar chart lets you compare rates across patient groups.

• Create the inpatient length-of-stay histogram by adding a new cell with this code:

# Histogram of inpatient length of stay
fig, ax = plt.subplots(figsize=(10, 6))
ax.hist(inpatient['LOS_days'].dropna(), bins=30, color='teal', edgecolor='white')
ax.set_xlabel('Length of Stay (days)')
ax.set_ylabel('Number of Encounters')
ax.set_title('Inpatient Length of Stay Distribution')
plt.tight_layout()
plt.savefig('../output/los_histogram.png', dpi=150)
plt.show()

What does this code do?

• ax.hist() draws a histogram. The bins=30 parameter splits the data into 30 buckets so you can see whether most stays are short (left-skewed) or spread out.
• .dropna() removes any NaN values from the LOS column before plotting, preventing matplotlib from raising an error.

• Run the cell. You should see a histogram showing most inpatient stays clustered at shorter durations with a long tail to the right.
• Confirm output/los_histogram.png exists.

• Now compute the 30-day readmission rates and create the comparison chart by adding a new cell with this code:

# Compute 30-day readmission flag
inpatient_sorted = inpatient.sort_values(['Patient', 'Start'])
inpatient_sorted['prev_stop'] = inpatient_sorted.groupby('Patient')['Stop'].shift(1)
inpatient_sorted['days_since_discharge'] = (
    inpatient_sorted['Start'] - inpatient_sorted['prev_stop']
).dt.total_seconds() / 86400
inpatient_sorted['readmitted_30d'] = inpatient_sorted['days_since_discharge'] <= 30

# Calculate readmission rate per cohort
def get_readmission_rate(patient_ids):
    cohort_encounters = inpatient_sorted[inpatient_sorted['Patient'].isin(patient_ids)]
    if len(cohort_encounters) == 0:
        return 0
    return cohort_encounters['readmitted_30d'].mean() * 100

cohorts = {
    'Diabetes': diabetes_patients,
    'Hypertension': hypertension_patients,
    'COPD': copd_patients,
    'Overall': patients['Id'].unique()
}

readmission_rates = {name: get_readmission_rate(ids) for name, ids in cohorts.items()}

What does this code do?

• .shift(1) looks at the previous row within the same patient group to find the prior discharge date.
• If the gap between the prior discharge and the next admission is 30 days or fewer, it counts as a readmission.
• get_readmission_rate() filters the inpatient encounters to just the cohort's patients, then calculates the percentage flagged as readmissions.

• Create the grouped bar chart by adding a final cell with this code:

# Grouped bar chart of readmission rates
fig, ax = plt.subplots(figsize=(8, 6))
bars = ax.bar(readmission_rates.keys(), readmission_rates.values(),
              color=['#e74c3c', '#3498db', '#2ecc71', '#95a5a6'])
ax.set_ylabel('30-Day Readmission Rate (%)')
ax.set_title('30-Day Readmission Rates by Cohort')
ax.set_ylim(0, max(readmission_rates.values()) * 1.3)

# Add percentage labels on each bar
for bar in bars:
    height = bar.get_height()
    ax.text(bar.get_x() + bar.get_width()/2., height,
            f'{height:.1f}%', ha='center', va='bottom')

plt.tight_layout()
plt.savefig('../output/readmission_rates.png', dpi=150)
plt.show()

What does this code do?

• ax.bar() draws one bar per cohort, coloured differently to make comparison easy.
• ax.set_ylim(0, max(...) * 1.3) adds 30% headroom above the tallest bar so the percentage labels fit.
• The ax.text() loop places the numeric percentage above each bar for a polished look.

• Run both cells. You should see a bar chart with four coloured bars showing readmission rates, each labelled with a percentage.
• Confirm output/readmission_rates.png exists.

• Verify all four PNGs are saved by running this in a new cell:

os.listdir('../output')

• You should see top_conditions.png, lab_boxplots.png, los_histogram.png, and readmission_rates.png listed in the output.

Missing PNG files in the output folder?

• Make sure the path uses ../output/ (with two dots). Your notebook runs from the notebooks/ folder, so .. goes up one level to the project root.
• If a specific chart's cell errored, the PNG for that chart won't be saved. Fix the error and re-run that cell.
• Help me debug my missing PNG files.

✔️ Awesome, I've got everything!

Make sure you have saved your 04_visualize.ipynb notebook and that all four PNGs exist in output/.

ⓧ I'd like to double check the full code

Here is the complete code for 04_visualize.ipynb across all cells:

import pandas as pd
import matplotlib.pyplot as plt
import os

# Create output directory for saved visualizations
os.makedirs('../output', exist_ok=True)

# Load CSV files
patients = pd.read_csv('../data/patients.csv')
encounters = pd.read_csv('../data/encounters.csv')
conditions = pd.read_csv('../data/conditions.csv')
observations = pd.read_csv('../data/observations.csv')

# Parse date columns
encounters['Start'] = pd.to_datetime(encounters['Start'])
encounters['Stop'] = pd.to_datetime(encounters['Stop'])

# Calculate inpatient length of stay in days
inpatient = encounters[encounters['EncounterClass'] == 'inpatient'].copy()
inpatient['LOS_days'] = (inpatient['Stop'] - inpatient['Start']).dt.total_seconds() / 86400

# Define cohorts using SNOMED-CT codes
diabetes_patients = conditions[conditions['Code'] == '44054006']['Patient'].unique()
hypertension_patients = conditions[conditions['Code'] == '59621000']['Patient'].unique()
copd_patients = conditions[conditions['Code'] == '185086009']['Patient'].unique()

# Count unique patients per condition and get the top 15
condition_counts = conditions.groupby('Description')['Patient'].nunique()
top_15 = condition_counts.nlargest(15).sort_values()

# Create horizontal bar chart
fig, ax = plt.subplots(figsize=(10, 8))
ax.barh(top_15.index, top_15.values, color='steelblue')
ax.set_xlabel('Number of Patients')
ax.set_title('Top 15 Conditions by Patient Count')
plt.tight_layout()
plt.savefig('../output/top_conditions.png', dpi=150)
plt.show()

# Filter observations for HbA1c (LOINC 4548-4) and Systolic BP (LOINC 8480-6)
hba1c = observations[observations['Code'] == '4548-4'].copy()
systolic_bp = observations[observations['Code'] == '8480-6'].copy()

# Convert Value column to numeric
hba1c['Value'] = pd.to_numeric(hba1c['Value'], errors='coerce')
systolic_bp['Value'] = pd.to_numeric(systolic_bp['Value'], errors='coerce')

# Assign cohort labels based on patient membership
def assign_cohort(patient_id):
    if patient_id in set(diabetes_patients):
        return 'Diabetes'
    elif patient_id in set(hypertension_patients):
        return 'Hypertension'
    elif patient_id in set(copd_patients):
        return 'COPD'
    else:
        return 'Other'

hba1c['Cohort'] = hba1c['Patient'].apply(assign_cohort)
systolic_bp['Cohort'] = systolic_bp['Patient'].apply(assign_cohort)

# Create side-by-side box plots for HbA1c and Systolic BP
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

cohort_order = ['Diabetes', 'Hypertension', 'COPD', 'Other']

# HbA1c box plot
hba1c_data = [hba1c[hba1c['Cohort'] == c]['Value'].dropna() for c in cohort_order]
axes[0].boxplot(hba1c_data, labels=cohort_order)
axes[0].set_title('HbA1c by Cohort')
axes[0].set_ylabel('HbA1c (%)')

# Systolic BP box plot
bp_data = [systolic_bp[systolic_bp['Cohort'] == c]['Value'].dropna() for c in cohort_order]
axes[1].boxplot(bp_data, labels=cohort_order)
axes[1].set_title('Systolic Blood Pressure by Cohort')
axes[1].set_ylabel('mmHg')

plt.tight_layout()
plt.savefig('../output/lab_boxplots.png', dpi=150)
plt.show()

# Histogram of inpatient length of stay
fig, ax = plt.subplots(figsize=(10, 6))
ax.hist(inpatient['LOS_days'].dropna(), bins=30, color='teal', edgecolor='white')
ax.set_xlabel('Length of Stay (days)')
ax.set_ylabel('Number of Encounters')
ax.set_title('Inpatient Length of Stay Distribution')
plt.tight_layout()
plt.savefig('../output/los_histogram.png', dpi=150)
plt.show()

# Compute 30-day readmission flag
inpatient_sorted = inpatient.sort_values(['Patient', 'Start'])
inpatient_sorted['prev_stop'] = inpatient_sorted.groupby('Patient')['Stop'].shift(1)
inpatient_sorted['days_since_discharge'] = (
    inpatient_sorted['Start'] - inpatient_sorted['prev_stop']
).dt.total_seconds() / 86400
inpatient_sorted['readmitted_30d'] = inpatient_sorted['days_since_discharge'] <= 30

# Calculate readmission rate per cohort
def get_readmission_rate(patient_ids):
    cohort_encounters = inpatient_sorted[inpatient_sorted['Patient'].isin(patient_ids)]
    if len(cohort_encounters) == 0:
        return 0
    return cohort_encounters['readmitted_30d'].mean() * 100

cohorts = {
    'Diabetes': diabetes_patients,
    'Hypertension': hypertension_patients,
    'COPD': copd_patients,
    'Overall': patients['Id'].unique()
}

readmission_rates = {name: get_readmission_rate(ids) for name, ids in cohorts.items()}

# Grouped bar chart of readmission rates
fig, ax = plt.subplots(figsize=(8, 6))
bars = ax.bar(readmission_rates.keys(), readmission_rates.values(),
              color=['#e74c3c', '#3498db', '#2ecc71', '#95a5a6'])
ax.set_ylabel('30-Day Readmission Rate (%)')
ax.set_title('30-Day Readmission Rates by Cohort')
ax.set_ylim(0, max(readmission_rates.values()) * 1.3)

# Add percentage labels on each bar
for bar in bars:
    height = bar.get_height()
    ax.text(bar.get_x() + bar.get_width()/2., height,
            f'{height:.1f}%', ha='center', va='bottom')

plt.tight_layout()
plt.savefig('../output/readmission_rates.png', dpi=150)
plt.show()

You now have four polished visualizations saved as PNG files. Next up, you will refactor your notebook logic into reusable Python modules so the pipeline runs from a single script.

Refactor into Modular Python Files

Your notebooks contain the entire pipeline from loading data through visualization. That works for exploration, but notebooks are hard to reuse, test, or share as production code.

In this step, you will extract your notebook logic into standalone .py files organized by responsibility. The result is a modular codebase that anyone can run with a single command.

In this step, get ready to:

• Move notebook logic into reusable Python modules inside a src/ directory.
• Learn how import statements and the if __name__ == '__main__': pattern work.
• Run your full pipeline end-to-end from a single main.py entry point.

Create the src/ package and move notebook logic into modules

A Python module is simply a .py file containing functions you can import elsewhere. A package is a directory containing an __init__.py file, which tells Python to treat that folder as an importable collection of modules.

• Create the src/ directory and its __init__.py file by running these commands in your terminal:

mkdir src
touch src/__init__.py

What is __init__.py?

The __init__.py file can be completely empty. Its presence tells Python that the src/ folder is a package, which means other files can use from src.module_name import function_name to pull in your code.

Think of it like a table of contents page. Even if it is blank, it signals that this directory contains importable modules.

• Confirm the directory exists by running:

ls src/

You should see __init__.py listed in the output.

Now you will create five module files. Each one takes a slice of logic from your notebooks and wraps it in reusable functions.

• Create src/load_data.py in VS Code and paste the following code:

import pandas as pd

def load_all_csvs(data_dir='data/'):
    """Load all 5 clinical CSV files into DataFrames."""
    patients = pd.read_csv(f'{data_dir}patients.csv')
    encounters = pd.read_csv(f'{data_dir}encounters.csv')
    conditions = pd.read_csv(f'{data_dir}conditions.csv')
    medications = pd.read_csv(f'{data_dir}medications.csv')
    observations = pd.read_csv(f'{data_dir}observations.csv')

    return patients, encounters, conditions, medications, observations

What does this code do?

The load_all_csvs() function wraps the pd.read_csv() calls you had in your first notebook. It returns all five DataFrames as a tuple so that any script importing this module gets the raw data in one call.

The data_dir parameter defaults to 'data/' but can be changed if you ever move your CSV files.

• Create src/clean_data.py and paste this code:

import pandas as pd

def clean_patients(patients):
    """Clean patients DataFrame: parse dates, calculate age."""
    patients['BirthDate'] = pd.to_datetime(patients['BirthDate'])
    patients['DeathDate'] = pd.to_datetime(patients['DeathDate'])

    # Calculate age in years from BirthDate
    today = pd.Timestamp.now()
    patients['Age'] = ((today - patients['BirthDate']).dt.days / 365.25).astype(int)

    return patients

def clean_encounters(encounters):
    """Clean encounters DataFrame: parse dates, calculate length of stay."""
    encounters['Start'] = pd.to_datetime(encounters['Start'])
    encounters['Stop'] = pd.to_datetime(encounters['Stop'])

    # Calculate length of stay in days
    encounters['LOS_days'] = (encounters['Stop'] - encounters['Start']).dt.total_seconds() / 86400

    return encounters

def clean_conditions(conditions):
    """Clean conditions DataFrame: parse date columns."""
    conditions['Start'] = pd.to_datetime(conditions['Start'])
    conditions['Stop'] = pd.to_datetime(conditions['Stop'])

    return conditions

def clean_observations(observations):
    """Clean observations DataFrame: parse dates, convert numeric values."""
    observations['Date'] = pd.to_datetime(observations['Date'])

    # Convert Value to numeric where Type is 'numeric'
    numeric_mask = observations['Type'] == 'numeric'
    observations.loc[numeric_mask, 'Value'] = pd.to_numeric(
        observations.loc[numeric_mask, 'Value'], errors='coerce'
    )

    return observations

def build_analysis_df(patients, encounters, conditions):
    """Join tables into a unified analysis DataFrame."""
    # Merge encounters with patients
    analysis_df = encounters.merge(
        patients, left_on='Patient', right_on='Id', suffixes=('', '_patient')
    )

    # Merge with conditions
    analysis_df = analysis_df.merge(
        conditions, left_on='Id', right_on='Encounter', suffixes=('', '_condition'), how='left'
    )

    return analysis_df

What does this code do?

Each function handles one cleaning responsibility from your second notebook. clean_patients() parses dates and calculates age. clean_encounters() parses timestamps and computes length of stay in days.

The build_analysis_df() function performs the same pd.merge() joins you used to create your unified DataFrame. Now it is a single callable function instead of scattered notebook cells.

• Create src/cohort_builder.py and paste this code:

# SNOMED-CT codes for cohort definitions
DIABETES_CODE = '44054006'
HYPERTENSION_CODE = '59621000'
COPD_CODE = '185086009'

def get_cohort_patients(conditions, snomed_code):
    """Get unique patient IDs with a specific condition code."""
    return conditions[conditions['Code'] == snomed_code]['Patient'].unique()

def build_cohorts(conditions):
    """Build patient cohorts for diabetes, hypertension, and COPD."""
    cohorts = {
        'Diabetes': get_cohort_patients(conditions, DIABETES_CODE),
        'Hypertension': get_cohort_patients(conditions, HYPERTENSION_CODE),
        'COPD': get_cohort_patients(conditions, COPD_CODE),
    }

    return cohorts

def label_patients_by_cohort(patients, cohorts):
    """Add cohort membership columns to patients DataFrame."""
    for cohort_name, patient_ids in cohorts.items():
        patients[cohort_name] = patients['Id'].isin(patient_ids)

    return patients

What does this code do?

The SNOMED-CT codes at the top are the same codes you used in your cohort notebook. build_cohorts() returns a dictionary mapping cohort names to arrays of patient IDs.

The label_patients_by_cohort() function adds boolean columns (Diabetes, Hypertension, COPD) to the patients DataFrame so you can filter or group easily.

• Create src/metrics.py and paste this code:

import pandas as pd

def calculate_prevalence(conditions, patients):
    """Calculate condition prevalence rates."""
    total_patients = len(patients)

    # Count distinct patients per condition
    prevalence = (
        conditions.groupby(['Code', 'Description'])['Patient']
        .nunique()
        .reset_index()
        .rename(columns={'Patient': 'PatientCount'})
    )
    prevalence['Prevalence'] = prevalence['PatientCount'] / total_patients
    prevalence = prevalence.sort_values('Prevalence', ascending=False)

    return prevalence

def calculate_los_by_class(encounters):
    """Calculate average length of stay by encounter class."""
    los_by_class = encounters.groupby('EncounterClass')['LOS_days'].mean().reset_index()
    los_by_class.columns = ['EncounterClass', 'AvgLOS']

    return los_by_class

def calculate_readmissions(encounters):
    """Flag 30-day all-cause readmissions for inpatient encounters."""
    # Filter to inpatient encounters only
    inpatient = encounters[encounters['EncounterClass'] == 'inpatient'].copy()
    inpatient = inpatient.sort_values(['Patient', 'Start'])

    # Calculate days between consecutive inpatient encounters per patient
    inpatient['PrevStop'] = inpatient.groupby('Patient')['Stop'].shift(1)
    inpatient['DaysSinceLast'] = (
        inpatient['Start'] - inpatient['PrevStop']
    ).dt.total_seconds() / 86400

    # Flag readmissions within 30 days
    inpatient['Readmission'] = inpatient['DaysSinceLast'] <= 30

    return inpatient

What does this code do?

These three functions mirror the metrics you computed in your third notebook. calculate_prevalence() uses .groupby() and .nunique() to count distinct patients per condition.

The calculate_readmissions() function sorts inpatient encounters by patient and date, then uses .shift(1) to find the previous discharge date. Any new admission within 30 days of a prior discharge gets flagged as a readmission.

• Create src/visualize.py and paste this code:

import matplotlib.pyplot as plt
import pandas as pd

def plot_top_conditions(prevalence, output_dir='output/'):
    """Create horizontal bar chart of top 15 conditions by prevalence."""
    top_15 = prevalence.head(15)

    fig, ax = plt.subplots(figsize=(10, 8))
    ax.barh(top_15['Description'], top_15['Prevalence'])
    ax.set_xlabel('Prevalence')
    ax.set_title('Top 15 Conditions by Prevalence')
    ax.invert_yaxis()
    plt.tight_layout()
    plt.savefig(f'{output_dir}top_conditions.png', dpi=150)
    plt.close()

def plot_los_histogram(encounters, output_dir='output/'):
    """Plot histogram of inpatient length of stay."""
    inpatient = encounters[encounters['EncounterClass'] == 'inpatient']

    fig, ax = plt.subplots(figsize=(10, 6))
    ax.hist(inpatient['LOS_days'], bins=30, edgecolor='black')
    ax.set_xlabel('Length of Stay (days)')
    ax.set_ylabel('Frequency')
    ax.set_title('Inpatient Length of Stay Distribution')
    plt.tight_layout()
    plt.savefig(f'{output_dir}los_histogram.png', dpi=150)
    plt.close()

def plot_lab_boxplots(observations, patients, cohorts, output_dir='output/'):
    """Create box plots of HbA1c and systolic BP by cohort."""
    # Filter to HbA1c (LOINC 4548-4) and Systolic BP (LOINC 8480-6)
    lab_codes = {'4548-4': 'HbA1c', '8480-6': 'Systolic BP'}
    labs = observations[observations['Code'].isin(lab_codes.keys())].copy()
    labs['Value'] = pd.to_numeric(labs['Value'], errors='coerce')
    labs = labs.dropna(subset=['Value'])
    labs['LabName'] = labs['Code'].map(lab_codes)

    # Add cohort labels
    labs['Cohort'] = 'General'
    for cohort_name, patient_ids in cohorts.items():
        labs.loc[labs['Patient'].isin(patient_ids), 'Cohort'] = cohort_name

    fig, axes = plt.subplots(1, 2, figsize=(14, 6))
    for i, lab_name in enumerate(lab_codes.values()):
        lab_data = labs[labs['LabName'] == lab_name]
        cohort_groups = [group['Value'].values for name, group in lab_data.groupby('Cohort')]
        cohort_labels = [name for name, group in lab_data.groupby('Cohort')]
        axes[i].boxplot(cohort_groups, labels=cohort_labels)
        axes[i].set_title(f'{lab_name} by Cohort')
        axes[i].set_ylabel(lab_name)

    plt.tight_layout()
    plt.savefig(f'{output_dir}lab_boxplots.png', dpi=150)
    plt.close()

def plot_readmission_rates(readmissions, cohorts, output_dir='output/'):
    """Create grouped bar chart comparing readmission rates across cohorts."""
    rates = {}
    for cohort_name, patient_ids in cohorts.items():
        cohort_readmissions = readmissions[readmissions['Patient'].isin(patient_ids)]
        if len(cohort_readmissions) > 0:
            rate = cohort_readmissions['Readmission'].mean()
        else:
            rate = 0.0
        rates[cohort_name] = rate

    fig, ax = plt.subplots(figsize=(8, 6))
    ax.bar(rates.keys(), rates.values())
    ax.set_xlabel('Cohort')
    ax.set_ylabel('30-Day Readmission Rate')
    ax.set_title('30-Day Readmission Rates by Cohort')
    plt.tight_layout()
    plt.savefig(f'{output_dir}readmission_rates.png', dpi=150)
    plt.close()

What does this code do?

Each function produces one of your four matplotlib visualizations and saves it to the output/ directory. The plt.close() call after each figure frees memory so plots do not stack up when running the full pipeline.

The plot_lab_boxplots() function filters observations by LOINC codes 4548-4 (HbA1c) and 8480-6 (Systolic BP), then groups values by cohort before creating the box plots.

• Save all five files (src/load_data.py, src/clean_data.py, src/cohort_builder.py, src/metrics.py, src/visualize.py).
• Confirm all files exist by running:

ls src/

You should see __init__.py, clean_data.py, cohort_builder.py, load_data.py, metrics.py, and visualize.py listed.

Don't see all the files?

Make sure you created each file inside the src/ directory, not in the project root. If a file ended up in the wrong spot, move it with mv filename.py src/filename.py.

Also check for typos in the filenames. Python imports are case-sensitive, so Load_Data.py would not match an import of load_data. Help me check that my src/ files are set up correctly.

Understand the if __name__ == '__main__': pattern

Python files can act as both importable modules and standalone scripts. When you run a file directly with python main.py, Python sets a special variable called __name__ to the string '__main__'.

When the same file is imported by another file, __name__ is set to the module name instead. The if __name__ == '__main__': guard ensures that pipeline execution code only runs when you deliberately run the file, not when you import it.

Why does this matter?

Without this guard, importing your module in a notebook or test would immediately execute the entire pipeline. The guard keeps functions importable without triggering side effects.

In SQL terms, it is like having a stored procedure that only executes when you call it, not when you load the schema.

Create main.py and run the full pipeline

Now you will tie all the modules together in a single entry point. This file imports functions from each module and runs them in sequence.

• Create main.py in your project root (not inside src/) and paste this code:

from src.load_data import load_all_csvs
from src.clean_data import clean_patients, clean_encounters, clean_conditions, clean_observations, build_analysis_df
from src.cohort_builder import build_cohorts, label_patients_by_cohort
from src.metrics import calculate_prevalence, calculate_los_by_class, calculate_readmissions
from src.visualize import plot_top_conditions, plot_los_histogram, plot_lab_boxplots, plot_readmission_rates

def run_pipeline():
    """Run the full clinical cohort analytics pipeline."""
    # Step 1: Load raw data
    print("Loading data...")
    patients, encounters, conditions, medications, observations = load_all_csvs()

    # Step 2: Clean and transform
    print("Cleaning data...")
    patients = clean_patients(patients)
    encounters = clean_encounters(encounters)
    conditions = clean_conditions(conditions)
    observations = clean_observations(observations)

    # Step 3: Build cohorts
    print("Building cohorts...")
    cohorts = build_cohorts(conditions)
    patients = label_patients_by_cohort(patients, cohorts)

    # Step 4: Compute metrics
    print("Computing metrics...")
    prevalence = calculate_prevalence(conditions, patients)
    los_by_class = calculate_los_by_class(encounters)
    readmissions = calculate_readmissions(encounters)

    # Step 5: Generate visualizations
    print("Generating visualizations...")
    plot_top_conditions(prevalence)
    plot_los_histogram(encounters)
    plot_lab_boxplots(observations, patients, cohorts)
    plot_readmission_rates(readmissions, cohorts)

    print("Pipeline complete! Check output/ for visualizations.")

if __name__ == '__main__':
    run_pipeline()

What does this code do?

The from src.load_data import load_all_csvs syntax tells Python to look inside the src package, find the load_data module, and pull in the load_all_csvs function. Each import line works the same way for the other modules.

The run_pipeline() function calls each module's functions in order: load, clean, build cohorts, compute metrics, visualize. The print() statements give you progress feedback in the terminal so you can see each stage complete.

The if __name__ == '__main__': guard at the bottom means run_pipeline() only executes when you run this file directly.

• Save main.py.
• Run the full pipeline by executing this command:

python main.py

You should see each stage print in sequence, ending with Pipeline complete! Check output/ for visualizations. in your terminal.

• Verify that the visualizations were regenerated by running:

ls output/

You should see top_conditions.png, los_histogram.png, lab_boxplots.png, and readmission_rates.png with updated timestamps.

Seeing a ModuleNotFoundError?

Make sure you are running python main.py from the clinical-cohort-analytics/ project root, not from inside the src/ directory. Python resolves imports relative to where you run the command.

Also confirm that src/__init__.py exists. Without it, Python does not recognize src as a package. Help me fix my ModuleNotFoundError when running main.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved all files in your project.

ⓧ I'd like to double check the full code

Here is the full main.py for comparison:

from src.load_data import load_all_csvs
from src.clean_data import clean_patients, clean_encounters, clean_conditions, clean_observations, build_analysis_df
from src.cohort_builder import build_cohorts, label_patients_by_cohort
from src.metrics import calculate_prevalence, calculate_los_by_class, calculate_readmissions
from src.visualize import plot_top_conditions, plot_los_histogram, plot_lab_boxplots, plot_readmission_rates

def run_pipeline():
    """Run the full clinical cohort analytics pipeline."""
    # Step 1: Load raw data
    print("Loading data...")
    patients, encounters, conditions, medications, observations = load_all_csvs()

    # Step 2: Clean and transform
    print("Cleaning data...")
    patients = clean_patients(patients)
    encounters = clean_encounters(encounters)
    conditions = clean_conditions(conditions)
    observations = clean_observations(observations)

    # Step 3: Build cohorts
    print("Building cohorts...")
    cohorts = build_cohorts(conditions)
    patients = label_patients_by_cohort(patients, cohorts)

    # Step 4: Compute metrics
    print("Computing metrics...")
    prevalence = calculate_prevalence(conditions, patients)
    los_by_class = calculate_los_by_class(encounters)
    readmissions = calculate_readmissions(encounters)

    # Step 5: Generate visualizations
    print("Generating visualizations...")
    plot_top_conditions(prevalence)
    plot_los_histogram(encounters)
    plot_lab_boxplots(observations, patients, cohorts)
    plot_readmission_rates(readmissions, cohorts)

    print("Pipeline complete! Check output/ for visualizations.")

if __name__ == '__main__':
    run_pipeline()

And here is the full src/load_data.py:

import pandas as pd

def load_all_csvs(data_dir='data/'):
    """Load all 5 clinical CSV files into DataFrames."""
    patients = pd.read_csv(f'{data_dir}patients.csv')
    encounters = pd.read_csv(f'{data_dir}encounters.csv')
    conditions = pd.read_csv(f'{data_dir}conditions.csv')
    medications = pd.read_csv(f'{data_dir}medications.csv')
    observations = pd.read_csv(f'{data_dir}observations.csv')

    return patients, encounters, conditions, medications, observations

Here is src/clean_data.py:

import pandas as pd

def clean_patients(patients):
    """Clean patients DataFrame: parse dates, calculate age."""
    patients['BirthDate'] = pd.to_datetime(patients['BirthDate'])
    patients['DeathDate'] = pd.to_datetime(patients['DeathDate'])

    # Calculate age in years from BirthDate
    today = pd.Timestamp.now()
    patients['Age'] = ((today - patients['BirthDate']).dt.days / 365.25).astype(int)

    return patients

def clean_encounters(encounters):
    """Clean encounters DataFrame: parse dates, calculate length of stay."""
    encounters['Start'] = pd.to_datetime(encounters['Start'])
    encounters['Stop'] = pd.to_datetime(encounters['Stop'])

    # Calculate length of stay in days
    encounters['LOS_days'] = (encounters['Stop'] - encounters['Start']).dt.total_seconds() / 86400

    return encounters

def clean_conditions(conditions):
    """Clean conditions DataFrame: parse date columns."""
    conditions['Start'] = pd.to_datetime(conditions['Start'])
    conditions['Stop'] = pd.to_datetime(conditions['Stop'])

    return conditions

def clean_observations(observations):
    """Clean observations DataFrame: parse dates, convert numeric values."""
    observations['Date'] = pd.to_datetime(observations['Date'])

    # Convert Value to numeric where Type is 'numeric'
    numeric_mask = observations['Type'] == 'numeric'
    observations.loc[numeric_mask, 'Value'] = pd.to_numeric(
        observations.loc[numeric_mask, 'Value'], errors='coerce'
    )

    return observations

def build_analysis_df(patients, encounters, conditions):
    """Join tables into a unified analysis DataFrame."""
    # Merge encounters with patients
    analysis_df = encounters.merge(
        patients, left_on='Patient', right_on='Id', suffixes=('', '_patient')
    )

    # Merge with conditions
    analysis_df = analysis_df.merge(
        conditions, left_on='Id', right_on='Encounter', suffixes=('', '_condition'), how='left'
    )

    return analysis_df

Here is src/cohort_builder.py:

# SNOMED-CT codes for cohort definitions
DIABETES_CODE = '44054006'
HYPERTENSION_CODE = '59621000'
COPD_CODE = '185086009'

def get_cohort_patients(conditions, snomed_code):
    """Get unique patient IDs with a specific condition code."""
    return conditions[conditions['Code'] == snomed_code]['Patient'].unique()

def build_cohorts(conditions):
    """Build patient cohorts for diabetes, hypertension, and COPD."""
    cohorts = {
        'Diabetes': get_cohort_patients(conditions, DIABETES_CODE),
        'Hypertension': get_cohort_patients(conditions, HYPERTENSION_CODE),
        'COPD': get_cohort_patients(conditions, COPD_CODE),
    }

    return cohorts

def label_patients_by_cohort(patients, cohorts):
    """Add cohort membership columns to patients DataFrame."""
    for cohort_name, patient_ids in cohorts.items():
        patients[cohort_name] = patients['Id'].isin(patient_ids)

    return patients

Here is src/metrics.py:

import pandas as pd

def calculate_prevalence(conditions, patients):
    """Calculate condition prevalence rates."""
    total_patients = len(patients)

    # Count distinct patients per condition
    prevalence = (
        conditions.groupby(['Code', 'Description'])['Patient']
        .nunique()
        .reset_index()
        .rename(columns={'Patient': 'PatientCount'})
    )
    prevalence['Prevalence'] = prevalence['PatientCount'] / total_patients
    prevalence = prevalence.sort_values('Prevalence', ascending=False)

    return prevalence

def calculate_los_by_class(encounters):
    """Calculate average length of stay by encounter class."""
    los_by_class = encounters.groupby('EncounterClass')['LOS_days'].mean().reset_index()
    los_by_class.columns = ['EncounterClass', 'AvgLOS']

    return los_by_class

def calculate_readmissions(encounters):
    """Flag 30-day all-cause readmissions for inpatient encounters."""
    # Filter to inpatient encounters only
    inpatient = encounters[encounters['EncounterClass'] == 'inpatient'].copy()
    inpatient = inpatient.sort_values(['Patient', 'Start'])

    # Calculate days between consecutive inpatient encounters per patient
    inpatient['PrevStop'] = inpatient.groupby('Patient')['Stop'].shift(1)
    inpatient['DaysSinceLast'] = (
        inpatient['Start'] - inpatient['PrevStop']
    ).dt.total_seconds() / 86400

    # Flag readmissions within 30 days
    inpatient['Readmission'] = inpatient['DaysSinceLast'] <= 30

    return inpatient

And finally src/visualize.py:

import matplotlib.pyplot as plt
import pandas as pd

def plot_top_conditions(prevalence, output_dir='output/'):
    """Create horizontal bar chart of top 15 conditions by prevalence."""
    top_15 = prevalence.head(15)

    fig, ax = plt.subplots(figsize=(10, 8))
    ax.barh(top_15['Description'], top_15['Prevalence'])
    ax.set_xlabel('Prevalence')
    ax.set_title('Top 15 Conditions by Prevalence')
    ax.invert_yaxis()
    plt.tight_layout()
    plt.savefig(f'{output_dir}top_conditions.png', dpi=150)
    plt.close()

def plot_los_histogram(encounters, output_dir='output/'):
    """Plot histogram of inpatient length of stay."""
    inpatient = encounters[encounters['EncounterClass'] == 'inpatient']

    fig, ax = plt.subplots(figsize=(10, 6))
    ax.hist(inpatient['LOS_days'], bins=30, edgecolor='black')
    ax.set_xlabel('Length of Stay (days)')
    ax.set_ylabel('Frequency')
    ax.set_title('Inpatient Length of Stay Distribution')
    plt.tight_layout()
    plt.savefig(f'{output_dir}los_histogram.png', dpi=150)
    plt.close()

def plot_lab_boxplots(observations, patients, cohorts, output_dir='output/'):
    """Create box plots of HbA1c and systolic BP by cohort."""
    # Filter to HbA1c (LOINC 4548-4) and Systolic BP (LOINC 8480-6)
    lab_codes = {'4548-4': 'HbA1c', '8480-6': 'Systolic BP'}
    labs = observations[observations['Code'].isin(lab_codes.keys())].copy()
    labs['Value'] = pd.to_numeric(labs['Value'], errors='coerce')
    labs = labs.dropna(subset=['Value'])
    labs['LabName'] = labs['Code'].map(lab_codes)

    # Add cohort labels
    labs['Cohort'] = 'General'
    for cohort_name, patient_ids in cohorts.items():
        labs.loc[labs['Patient'].isin(patient_ids), 'Cohort'] = cohort_name

    fig, axes = plt.subplots(1, 2, figsize=(14, 6))
    for i, lab_name in enumerate(lab_codes.values()):
        lab_data = labs[labs['LabName'] == lab_name]
        cohort_groups = [group['Value'].values for name, group in lab_data.groupby('Cohort')]
        cohort_labels = [name for name, group in lab_data.groupby('Cohort')]
        axes[i].boxplot(cohort_groups, labels=cohort_labels)
        axes[i].set_title(f'{lab_name} by Cohort')
        axes[i].set_ylabel(lab_name)

    plt.tight_layout()
    plt.savefig(f'{output_dir}lab_boxplots.png', dpi=150)
    plt.close()

def plot_readmission_rates(readmissions, cohorts, output_dir='output/'):
    """Create grouped bar chart comparing readmission rates across cohorts."""
    rates = {}
    for cohort_name, patient_ids in cohorts.items():
        cohort_readmissions = readmissions[readmissions['Patient'].isin(patient_ids)]
        if len(cohort_readmissions) > 0:
            rate = cohort_readmissions['Readmission'].mean()
        else:
            rate = 0.0
        rates[cohort_name] = rate

    fig, ax = plt.subplots(figsize=(8, 6))
    ax.bar(rates.keys(), rates.values())
    ax.set_xlabel('Cohort')
    ax.set_ylabel('30-Day Readmission Rate')
    ax.set_title('30-Day Readmission Rates by Cohort')
    plt.tight_layout()
    plt.savefig(f'{output_dir}readmission_rates.png', dpi=150)
    plt.close()

Your pipeline now runs from a single command. Next up, you will package this repository with a professional README, requirements.txt, and push it to GitHub so hiring managers can see your work.

Package Your Repository for GitHub

Your pipeline now runs end-to-end from a single main.py entry point, producing clean visualizations and clinically meaningful metrics. The code works, but nobody can find it, reproduce it, or evaluate your skills from it.

In this step, you will package your project into a professional GitHub repository. A well-structured repo with a clear README, pinned dependencies, and a proper .gitignore signals to hiring managers that you understand reproducibility and collaboration.

In this step, get ready to:

• Create a requirements.txt and .gitignore for reproducibility.
• Write a professional README.md with methodology and reproduction instructions.
• Initialize Git, commit, and push to GitHub.

Create requirements.txt and .gitignore

Two small files make the difference between a project others can reproduce and one they give up on. A requirements.txt pins your exact dependency versions so anyone can recreate your environment. A .gitignore keeps clutter out of your repository.

• Create the requirements.txt file by running this command in your terminal:

cat > requirements.txt << 'EOF'
pandas==3.0.3
matplotlib==3.10.9
jupyter
EOF

What does this file do?

Each line pins a library to the exact version you used. When someone else runs pip install -r requirements.txt, they get the same environment you developed with.

Pinning versions prevents the "it works on my machine" problem. Without it, a collaborator might install a newer version of pandas that introduces breaking changes.

• Confirm the file was created by running:

cat requirements.txt

You should see three lines listing pandas, matplotlib, and jupyter with their pinned versions.

• Create the .gitignore file by running this command:

cat > .gitignore << 'EOF'
.venv/
__pycache__/
data/
.DS_Store
EOF

Why exclude these files?

.venv/ is your virtual environment (hundreds of megabytes of installed packages). Anyone can recreate it from requirements.txt.

__pycache__/ holds compiled bytecode that Python regenerates automatically. data/ contains your Synthea CSVs which are too large for Git and can be regenerated. .DS_Store is a macOS metadata file that has no place in a shared repo.

• Confirm the .gitignore was created by running:

cat .gitignore

You should see four lines listing the excluded paths.

Don't see the file?

Files starting with a dot are hidden by default on macOS. In your terminal, cat .gitignore will still display them. If using Finder, press Cmd+Shift+. to toggle hidden file visibility.

help me find my .gitignore file

✔️ Awesome, I've got everything!

Great. Double check you have both requirements.txt and .gitignore saved in your clinical-cohort-analytics folder.

ⓧ I'd like to double check the full code

Here is what each file should contain:

requirements.txt:

pandas==3.0.3
matplotlib==3.10.9
jupyter

.gitignore:

.venv/
__pycache__/
data/
.DS_Store

Write a professional README.md

A README is the first thing anyone sees when they visit your repository. It needs to answer three questions: what does this project do, how was it built, and how can I run it myself.

• Create the README.md file by running this command:

cat > README.md << 'EOF'
# Clinical Cohort Analytics Pipeline

A Python-based clinical data analysis pipeline that processes synthetic patient data (generated by Synthea) to compute clinically meaningful metrics and visualizations.

## Project Overview

This pipeline ingests raw clinical CSV data, cleans and transforms it, builds patient cohorts by diagnosis, and generates insights including:

- Condition prevalence rates stratified by demographics
- Average length of stay by encounter class
- 30-day all-cause readmission proxy rates
- Lab value distributions (HbA1c, systolic blood pressure)

## Methodology

1. **Data Generation**: 1,000 synthetic patients generated using Synthea
2. **Data Ingestion**: Load 5 CSV tables (patients, encounters, conditions, medications, observations)
3. **Cleaning & Transformation**: Handle missing values, parse dates, compute derived columns (age, length of stay)
4. **Cohort Construction**: Segment patients by SNOMED-CT diagnosis codes (diabetes, hypertension, COPD)
5. **Metric Computation**: Prevalence, LOS, readmission rates, lab distributions
6. **Visualization**: Publication-quality charts saved as PNG files

## Sample Outputs

| Visualization | Description |
|---|---|
| `output/top_conditions.png` | Horizontal bar chart of top 15 conditions |
| `output/los_histogram.png` | Inpatient length of stay distribution |
| `output/readmission_rates.png` | 30-day readmission rates by cohort |
| `output/lab_boxplots.png` | HbA1c and systolic BP by cohort |

## How to Reproduce

1. Clone the repository:

```bash
git clone https://github.com/YOUR_USERNAME/clinical-cohort-analytics.git
cd clinical-cohort-analytics
```

2. Create and activate a virtual environment:

```bash
python3 -m venv .venv
source .venv/bin/activate
```

3. Install dependencies:

```bash
pip install -r requirements.txt
```

4. Add your Synthea CSV data to the `data/` directory.

5. Run the pipeline:

```bash
python main.py
```

## Project Structure

```
clinical-cohort-analytics/
├── main.py              # Entry point - runs full pipeline
├── requirements.txt     # Pinned dependencies
├── src/
│   ├── load_data.py     # CSV ingestion
│   ├── clean_data.py    # Cleaning and transformation
│   ├── cohort_builder.py # Cohort construction
│   ├── metrics.py       # Clinical metric calculations
│   └── visualize.py     # Matplotlib visualizations
├── notebooks/           # Jupyter exploration notebooks
├── data/                # Raw CSV data (not tracked in Git)
└── output/              # Generated visualizations
```

## Tools and Technologies

- Python 3.13
- pandas 3.0.3
- matplotlib 3.10.9
- Jupyter Notebook
- Synthea (synthetic patient data generator)
EOF

What makes this README effective?

It opens with a one-line summary so readers know what the project does immediately. The methodology section shows your analytical thinking. The reproduction instructions prove the project actually works.

The sample outputs table lets hiring managers see results without cloning the repo. The project structure section demonstrates you understand code organization.

• Verify the README was created by running:

head -5 README.md

You should see the first five lines including the title and project description.

Customize before publishing

Before pushing to GitHub, update the clone URL in the README to use your actual GitHub username. You will do this after creating the repository in the next substep.

🙋‍♀️ Seeing a syntax error?

If the cat command fails, make sure the EOF markers are on their own lines with no extra spaces. You can also create the file manually in your code editor and paste the content.

help me create my README.md file

✔️ Awesome, I've got everything!

Great. Make sure README.md is saved in your clinical-cohort-analytics folder alongside main.py.

ⓧ I'd like to double check the full code

Your README.md should contain the full markdown content above, starting with # Clinical Cohort Analytics Pipeline and ending with the Tools and Technologies section listing Synthea.

Initialize Git and push to GitHub

Your project files are ready. Now you will turn this folder into a Git repository, commit everything, and push it to GitHub so the world can see your work.

• Confirm Git is available by running:

git --version

✔️ I see a version number

Git is ready. Move on to the next instruction.

ⓧ Command not found

Git was not found. Since you installed Homebrew earlier in this project, run the following to install Git:

brew install git

After installation completes, run git --version again to confirm it works.

• Initialize a new Git repository with main as the default branch by running:

git init -b main

Why use -b main?

GitHub uses main as its default branch name. The -b main flag tells Git to name your initial branch main so it matches GitHub automatically when you push.

• Stage all your project files and create your first commit by running these commands:

git add .
git commit -m "Initial commit: Clinical Cohort Analytics Pipeline"

You should see a summary showing the files that were committed. The data/ folder and .venv/ folder will not appear because your .gitignore excluded them.

Seeing 'please tell me who you are'?

Git needs your identity for commits. Run these two commands with your own name and email, then retry the commit:

git config --global user.email "your-email@example.com"
git config --global user.name "Your Name"

help me configure Git for my first commit

Now you need to create a repository on GitHub to push your code to.

• Open your browser and go to github.com/new.
• Enter clinical-cohort-analytics as the repository name.
• Add a short description like "Clinical data analysis pipeline using Python, pandas, and matplotlib."
• Select Public visibility so hiring managers can see it.
• Leave all other options unchecked (no README, no .gitignore, no license). You already have these files locally.
• Click Create repository.

GitHub will show a page with setup instructions. You need the HTTPS URL displayed at the top.

• Connect your local repository to GitHub and push your code by running these commands:

git remote add origin [[REPO_URL="https://github.com/your-username/clinical-cohort-analytics.git"]]
git push -u origin main

What do these commands do?

git remote add origin tells your local repo where to push code. The URL points to the GitHub repository you just created.

git push -u origin main uploads your committed code to GitHub. The -u flag sets the remote as default so future pushes only need git push.

• Verify the push succeeded by refreshing your GitHub repository page in the browser.

You should see all your files listed: main.py, src/, notebooks/, output/, requirements.txt, .gitignore, and README.md. The README content renders automatically below the file list.

Push failed with authentication error?

GitHub no longer accepts password authentication for Git operations. If prompted for credentials, you need to set up a personal access token or SSH key.

The quickest fix is to generate a personal access token at github.com/settings/tokens and use it as your password when prompted.

help me authenticate Git with GitHub

💡 Update the README clone URL

Now that your repo exists, open README.md in your editor and replace YOUR_USERNAME in the clone URL with your actual GitHub username. Commit and push the update:

git add README.md
git commit -m "Update README with correct clone URL"
git push

Secret mission

Predict 30-Day Readmission with Logistic Regression

You have a pipeline that computes readmission rates. But can you predict which patients will be readmitted before it happens? In this stretch challenge, you will train a logistic regression model on your analysis data, evaluate it honestly, and document why a toy model on synthetic data is not clinically valid.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally with a free GitHub repository. There are no ongoing costs or paid resources to worry about.

Decide whether to keep your resources for future projects, pause them to free up disk space, or delete them entirely.

Resources you used:

• Synthea repository (cloned and built, ~1.5 GB with build artifacts).
• clinical-cohort-analytics/ project directory (includes data/ CSV files, source code, notebooks, and output images).
• healthcare-analytics conda environment (with pandas, matplotlib, and Jupyter installed via pip).
• GitHub repository (free tier, no cost).

Keep everything running

No action needed. Choose this if you plan to extend your pipeline, add new cohorts, or reuse the code as a template for future healthcare data projects.

• Your healthcare-analytics conda environment stays ready. Reactivate it anytime with conda activate healthcare-analytics.
• The Synthea repository remains available if you want to regenerate data with different parameters (e.g., more patients or different demographics).
• Java JDK 17 remains installed via Homebrew. This is useful for other projects that need Java.

Pause - I'll come back to this later

Free up disk space by removing the Synthea build artifacts while keeping your project intact.

• Delete the cloned Synthea repository to reclaim ~1.5 GB by running:

rm -rf ~/Desktop/synthea

• Keep the clinical-cohort-analytics/ directory as-is. Your data files, notebooks, and source code stay intact for when you return.
• Your healthcare-analytics conda environment and GitHub repository remain accessible.

Why keep the project but delete Synthea?

The Synthea repository is only needed to generate new data. Your CSV files are already copied into your project's data/ folder, so you can re-run your entire pipeline without Synthea installed.

Delete - I don't want to use this again

Remove all project resources from your machine and GitHub. This cannot be undone.

• Delete the Synthea repository by running:

rm -rf ~/Desktop/synthea

• Delete your project directory by running:

rm -rf ~/Desktop/clinical-cohort-analytics

• Remove your conda environment by running:

conda env remove -n healthcare-analytics

• Delete your GitHub repository:

• Go to your repository page on GitHub.
• Click Settings in the top navigation.
• Scroll to the bottom to the Danger Zone section.
• Click Delete this repository and follow the confirmation prompts.

• Optionally, uninstall Java JDK 17 if you do not need it for other projects by running:

brew uninstall openjdk@17

What about Miniconda itself?

Miniconda is useful for many projects beyond this one. Only remove it if you are certain you will not need Python environments managed by conda in the future.

Nice Work!

Nice Work!

You just built a complete Clinical Cohort Analytics pipeline from scratch. Along the way, you picked up Python fundamentals and applied them to real healthcare data challenges.

You've learned how to:

• Ingest, clean, and join 5 clinical CSV tables into a unified analysis DataFrame using pandas.
• Build patient cohorts from SNOMED-CT codes and compute clinically meaningful metrics including condition prevalence, length of stay, and 30-day readmission rates.
• Create publication-quality visualizations with matplotlib, refactor notebook code into modular Python files, and package everything into a professional GitHub repository.
• Secret Mission: Trained a logistic regression model predicting 30-day readmission with scikit-learn and wrote an honest limitations disclosure explaining why a toy model on synthetic data is not clinically valid.

Ready to quiz yourself?