Healthcare Data Foundation

Build a reproducible pandas notebook that cleans and validates clinical data.

Introduction

30 Second Summary

A health analysis can look convincing even when blank values are hiding as ordinary text. If that foundation is unreliable, every chart built on it inherits the problem.

In this project, you will build a reproducible clinical-data notebook in JupyterLab with pandas. You will expose hidden missing-value markers before creating a validated handoff for the next coursework project.

What You'll Build

When you run the finished notebook from top to bottom, you'll watch raw hospital encounters become a checked dataset that is ready for the next stage of your analysis.

By the end of this project, you'll have:

  • A reproducible notebook that loads all 101,766 hospital encounters from local CSV data.
  • A visible missing-value audit that exposes hidden placeholders before confirming the cleaned row set.
  • A validated handoff containing the binary readmitted_30d target for Project 2.
  • Secret Mission: Automate the cleaned-data audit as a reusable Windows command.

Are there any prerequisites?

This project assumes basic Python syntax on a Windows computer where the language is already installed. You also need diabetic_data.csv plus IDS_mapping.csv before you begin.

Before We Start

Before the hands-on work begins, commit to building a reproducible notebook that loads, audits, cleans, validates, and exports encounter data. Trustworthy missing-value handling ensures literal placeholder strings do not pass as valid clinical values.

Set Up the Windows Notebook Workspace

Clinical cleaning results become hard to reproduce when Python package versions change between sessions.

A dedicated virtual environment keeps this coursework isolated from unrelated packages. Windows Command Prompt will install pinned JupyterLab and pandas versions inside it.

In this step, get ready to:
  • Organize both coursework CSV files inside a dedicated Windows workspace.
  • Confirm Python 3.11 or newer before creating an isolated environment.
  • Install JupyterLab 4.6.4 plus pandas 3.1.0 before launching the browser workspace.

Why local JupyterLab?

Your raw files stay beside the notebook on your computer. This keeps the workspace available when you reopen the coursework.

A local workspace also avoids uploading the clinical files again for every session. Your data foundation stays in one private project folder.

Build the project folder and pin the packages

The folder layout separates source data from notebooks and generated outputs. The requirements file records the package versions needed to reproduce this workspace.

  • Press the Windows key to open the search bar.
  • Type File Explorer and press Enter.
  • Select Desktop in the left sidebar.
  • Click New in the top toolbar.
  • Select Folder.
  • Type healthcare-coursework and press Enter.
  • Double-click the healthcare-coursework folder.

You should now see an empty healthcare-coursework folder on your Desktop.

  • Click New in the top toolbar.
  • Select Folder.
  • Type data and press Enter.
  • Double-click the data folder.
  • Press Ctrl+N to open a second File Explorer window.
  • Navigate to the folder that contains your original coursework files.
  • Select diabetic_data.csv.
  • Hold Ctrl while selecting IDS_mapping.csv.
  • Press Ctrl+C to copy both selected files.
  • Switch back to the File Explorer window showing the data folder.
  • Press Ctrl+V to paste both files.

You should see diabetic_data.csv and IDS_mapping.csv inside healthcare-coursework\data.

  • Press the Windows key to open the search bar.
  • Type Notepad and press Enter.
  • Add the pinned package requirements by copying this code into the empty Notepad file:
jupyterlab==4.6.4
pandas==3.1.0

What does this file do?

  • The jupyterlab==4.6.4 line locks the browser notebook environment to the project version.
  • The pandas==3.1.0 line locks the data-processing library to the project version.
  • The double equals signs tell the package installer to use those exact releases.
  • Click File in the Notepad menu bar.
  • Click Save As.
  • Select the healthcare-coursework folder on your Desktop.
  • Enter requirements.txt in the File name field.
  • Select All files in the Save as type field.
  • Click Save.
  • Close Notepad.
  • Return to the healthcare-coursework folder in File Explorer.

You should see the data folder beside requirements.txt.

Missing a file or seeing the wrong extension?

  • Confirm that both CSV filenames still end in .csv.
  • Check that requirements.txt sits directly inside healthcare-coursework.
  • Reopen the requirements file in Notepad if its name ends in .txt.txt.

Help me check my Windows coursework folder layout.

✔️ Awesome, I've got everything!

Your source files and pinned requirements are ready. Keep File Explorer available for the next check.

ⓧ I'd like to double check the full code

Compare your saved requirements.txt file with this complete version.

jupyterlab==4.6.4
pandas==3.1.0
Confirm Python and isolate the environment

pandas 3.1.0 requires Python 3.11 or newer. Checking the startup banner now prevents an incompatible environment from being created.

  • Press the Windows key to open the search bar.
  • Type Command Prompt and press Enter.
  • Move Windows Command Prompt to the healthcare-coursework folder by running this command:
cd %USERPROFILE%\Desktop\healthcare-coursework

What does this command do?

The cd command changes the current directory. The next commands now create files inside your Desktop copy of healthcare-coursework.

  • Start the Python interpreter by running this command:
python

What should appear?

Python prints a startup banner containing its version number. The interpreter then waits for Python input.

✔️ I see Python 3.11 or higher

Good. Your Python runtime supports the pinned pandas release.

ⓧ I see an older version

Your existing runtime is below the project minimum. Install the current 64-bit Windows release before creating the environment.

  • Close the Command Prompt window that contains the older interpreter.
  • Open the Python.org Windows downloads page.
  • Select Download Windows installer (64-bit) for Python 3.15.2.
  • Run the downloaded installer.
  • Select Install when the installer opens.
  • Press the Windows key after the installation finishes.
  • Type Command Prompt and press Enter.
  • Return to the coursework folder by running this command:
cd %USERPROFILE%\Desktop\healthcare-coursework

Why return to this folder?

A new Command Prompt starts in your user directory. This command restores the project location before you create its environment.

  • Start the newly installed Python runtime by running this command:
python

Check the new banner

The new startup banner should report Python 3.11 or higher. You can continue once the required version appears.

ⓧ Command not found

Windows cannot currently launch Python from Command Prompt. Install the current 64-bit Windows release before continuing.

  • Close the current Command Prompt window.
  • Open the Python.org Windows downloads page.
  • Select Download Windows installer (64-bit) for Python 3.15.2.
  • Run the downloaded installer.
  • Select Install when the installer opens.
  • Press the Windows key after the installation finishes.
  • Type Command Prompt and press Enter.
  • Return to the coursework folder by running this command:
cd %USERPROFILE%\Desktop\healthcare-coursework

Why return to this folder?

The environment must live beside your data folder and requirements file. This command restores that location in the new Command Prompt.

  • Start the installed Python runtime by running this command:
python

Check the installed runtime

The startup banner should now report Python 3.11 or higher. That confirms Command Prompt can reach the required runtime.

  • Exit the Python interpreter by entering this command:
exit()

What does this command do?

The exit() command closes the interactive interpreter. You return to the project directory in Windows Command Prompt.

  • Create tutorial-env before activating it by running these commands:
python -m venv tutorial-env
tutorial-env\Scripts\activate

What do these commands do?

  • The venv module creates an isolated Python interpreter inside tutorial-env.
  • The activation script updates the current Command Prompt session to use that interpreter.
  • The folder remains part of this workspace after you close Command Prompt.

You should see (tutorial-env) at the start of the Command Prompt line. That prefix confirms the environment is active.

Environment prefix missing?

  • Confirm that Command Prompt is still inside healthcare-coursework.
  • Check File Explorer for a new tutorial-env folder.
  • Run the activation command again if the folder exists without the prompt prefix.

Help me activate my Windows virtual environment.

Install the packages and launch JupyterLab

The activated environment starts with its own package collection. Installing from requirements.txt gives every future notebook session the same tool versions.

The first installation can take a few minutes while Python downloads the packages. A steady stream of installation output means the process is still working.

  • Install the pinned project dependencies by running this command:
python -m pip install -r requirements.txt

What does this command do?

Python runs the package installer from the active environment. The installer reads both exact versions from requirements.txt.

The installed packages remain inside tutorial-env. Other Python projects on your computer keep their own package versions.

Command Prompt should return to a line beginning with (tutorial-env) after the installation completes.

Package installation failed?

  • Confirm that (tutorial-env) still appears at the start of the prompt.
  • Confirm that requirements.txt is stored directly inside healthcare-coursework.
  • Check your internet connection if the download stops before Command Prompt returns.

Help me troubleshoot the pinned package installation.

Before you launch JupyterLab, what do you expect the browser to show? Hold that prediction for the next check.

  • Launch the notebook server from the activated environment by running this command:
jupyter lab

What happens when JupyterLab starts?

JupyterLab starts a local notebook server from the current project folder. It opens the workspace in your default browser.

This command keeps using the current Command Prompt window while the server runs. Leave that window open throughout your notebook work.

You should see the JupyterLab Launcher in your browser. The file browser should show data and requirements.txt inside the workspace.

  • Keep the activated Command Prompt window open while JupyterLab is running.

JupyterLab did not open?

  • Confirm that the launch command was run from a prompt beginning with (tutorial-env).
  • Wait for the server to finish starting if Command Prompt is still printing setup messages.
  • Copy the local browser address printed in Command Prompt if the browser does not open automatically.

Help me launch JupyterLab from my virtual environment.

Your isolated notebook workspace is live with both coursework files ready. Next, you will create the foundation notebook and preview every encounter.

Load and Preview Every Encounter

Your isolated Python workspace is running in JupyterLab. The next proof must come from the data itself.

A successful preview proves that the local path works. It also proves that the notebook kernel can use pandas to read every encounter.

In this step, get ready to:
  • Create coursework_foundation.ipynb with a Python kernel.
  • Confirm that both coursework files are available.
  • Display the first five encounters from all 101,766 rows.
Create and name the notebook

A notebook keeps executable code beside its visible results. The first cell establishes the purpose of this reproducible foundation.

  • Switch back to the JupyterLab browser tab from the previous step.
  • Click the + button in the file browser.
  • Select a Python kernel in the Launcher.
  • Right-click the new notebook in the file browser.
  • Select Rename.
  • Enter coursework_foundation.ipynb.
  • Press Enter.

You should see coursework_foundation.ipynb in the file browser.

  • Select the first cell in coursework_foundation.ipynb.
  • Set the cell type to Markdown using the notebook toolbar.
  • Add the notebook title by pasting this content into the cell:
# Healthcare Analytics Coursework 1 of 4: Data Foundation

This notebook creates a reproducible, validated handoff for the later ICD-9 enrichment, EDA, and VCI projects.

What does this cell do?

  • The heading identifies this notebook as the data foundation project.
  • The description records how later coursework projects depend on this handoff.
  • Save coursework_foundation.ipynb using the save control in the notebook toolbar.
  • Run the Markdown cell using the triangular run control in the notebook toolbar.

You should see a rendered title above the notebook description.

Title still looks like plain text?

  • Select the title cell again.
  • Confirm that the cell type is set to Markdown.
  • Run the cell using the triangular control in the notebook toolbar.

Ask for help if the title still does not render: Help me render a Markdown title cell in JupyterLab.

Validate the files and preview the data

A path check catches misplaced CSV files before the load begins. The next cell stores each location as a reusable path.

  • Select the empty code cell below the rendered title.
  • Add the path validation by pasting this code into the cell:
from pathlib import Path
import pandas as pd

DATA_DIR = Path("data")
RAW_DATA = DATA_DIR / "diabetic_data.csv"
MAPPING_DATA = DATA_DIR / "IDS_mapping.csv"
OUTPUT_DIR = Path("outputs")
OUTPUT_FILE = OUTPUT_DIR / "cleaned_diabetic_data.csv"

for required_file in (RAW_DATA, MAPPING_DATA):
    assert required_file.exists(), f"Missing required file: {required_file}"

print("Both coursework files are available.")

What does this code do?

  • Path creates file locations that work throughout the notebook.
  • The pd name gives the notebook access to pandas.
  • DATA_DIR points to the data folder.
  • RAW_DATA points to diabetic_data.csv.
  • MAPPING_DATA points to IDS_mapping.csv.
  • OUTPUT_DIR records the future export folder.
  • OUTPUT_FILE records the future handoff path.
  • The assertions stop execution when either source file is missing.
  • Save coursework_foundation.ipynb using the notebook toolbar.
  • Run the path validation cell using the triangular run control.

You should see Both coursework files are available. below the cell. This confirms that the notebook can reach both source files.

Did the file check stop?

  • Confirm that the JupyterLab file browser shows the data folder inside healthcare-coursework.
  • Expand data in the file browser.
  • Check that diabetic_data.csv appears inside the folder.
  • Check that IDS_mapping.csv appears inside the folder.

Ask for help if both files appear but the assertion still stops: Help me troubleshoot the notebook file paths.

The next Markdown cell records why this first load preserves placeholder text. That choice prepares the dataset for a later missing-value audit.

  • Add a new cell below the path validation cell.
  • Set the new cell type to Markdown using the notebook toolbar.
  • Add the naive-load heading by pasting this content into the cell:
## 1. Naive load and first visible result

The first load deliberately disables automatic missing-value conversion so we can test whether placeholder text is hiding in the CSV.

Why record the loading choice?

The heading makes this loading decision visible to anyone who reruns the notebook. The description also documents why placeholder text stays unchanged during the first read.

  • Save coursework_foundation.ipynb using the notebook toolbar.
  • Run the Markdown cell using the triangular run control.

You should see a rendered section heading named 1. Naive load and first visible result.

Heading did not render?

  • Select the naive-load heading cell.
  • Confirm that its cell type is Markdown.
  • Run the cell again using the notebook toolbar.

Ask for help if the Markdown cell remains unrendered: Help me fix this notebook heading cell.

The first load creates raw_df as a DataFrame. Its visible shape proves whether the complete encounter file loaded.

  • Add a new code cell below the naive-load heading.
  • Load the encounter file by pasting this code into the cell:
raw_df = pd.read_csv(RAW_DATA, keep_default_na=False, low_memory=False)

print(f"Rows: {raw_df.shape[0]:,}")
print(f"Columns: {raw_df.shape[1]}")
assert raw_df.shape[0] == 101_766, "Unexpected encounter count. Check the input file."
raw_df.head()

What does this code do?

  • pd.read_csv reads the encounter file stored in RAW_DATA.
  • keep_default_na=False preserves textual placeholders during this first load.
  • low_memory=False processes column types across the complete file.
  • raw_df.shape supplies the row count.
  • raw_df.shape also supplies the column count.
  • The assertion confirms that the expected 101,766 encounters loaded.
  • raw_df.head() displays the first five encounters.
  • Save coursework_foundation.ipynb using the notebook toolbar.
  • Before you run the cell, predict whether the full encounter count will appear.
  • Run the naive-load cell using the triangular run control.

You should see Rows: 101,766. You should also see a column count followed by the first five encounter rows.

Preview did not appear?

  • Run the path validation cell before the naive-load cell.
  • Confirm that the notebook kernel is still connected.
  • Check that the load cell ends with raw_df.head().
  • Compare the row-count assertion with 101_766.

Ask for help if the load cell still stops or shows a different count: Help me troubleshoot the encounter preview.

You have connected the local dataset to a working notebook. Every encounter is now available through raw_df for reproducible analysis.

✔️ Awesome, I've got everything!

Your notebook is saved with the title, path validation, full encounter load, row count, column count, and five-row preview.

ⓧ I'd like to double check the full code

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "title",
      "metadata": {},
      "source": [
        "# Healthcare Analytics Coursework 1 of 4: Data Foundation\n",
        "\n",
        "This notebook creates a reproducible, validated handoff for the later ICD-9 enrichment, EDA, and VCI projects."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "paths",
      "metadata": {},
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "\n",
        "DATA_DIR = Path(\"data\")\n",
        "RAW_DATA = DATA_DIR / \"diabetic_data.csv\"\n",
        "MAPPING_DATA = DATA_DIR / \"IDS_mapping.csv\"\n",
        "OUTPUT_DIR = Path(\"outputs\")\n",
        "OUTPUT_FILE = OUTPUT_DIR / \"cleaned_diabetic_data.csv\"\n",
        "\n",
        "for required_file in (RAW_DATA, MAPPING_DATA):\n",
        "    assert required_file.exists(), f\"Missing required file: {required_file}\"\n",
        "\n",
        "print(\"Both coursework files are available.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "naive-heading",
      "metadata": {},
      "source": [
        "## 1. Naive load and first visible result\n",
        "\n",
        "The first load deliberately disables automatic missing-value conversion so we can test whether placeholder text is hiding in the CSV."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "naive-load",
      "metadata": {},
      "outputs": [],
      "source": [
        "raw_df = pd.read_csv(RAW_DATA, keep_default_na=False, low_memory=False)\n",
        "\n",
        "print(f\"Rows: {raw_df.shape[0]:,}\")\n",
        "print(f\"Columns: {raw_df.shape[1]}\")\n",
        "assert raw_df.shape[0] == 101_766, \"Unexpected encounter count. Check the input file.\"\n",
        "raw_df.head()"
      ]
    }
  ],
  "metadata": {},
  "nbformat": 4,
  "nbformat_minor": 5
}

This reference contains the exact title, path validation, naive-load heading, and encounter preview cells for this step.

Your notebook now reaches every encounter in the coursework dataset. Next, you will test whether its first missing-value result can be trusted.

Expose the Hidden Missing-Value Problem

Your JupyterLab notebook now loads every encounter from the local file. That first result confirms the notebook can access the dataset.

The naive load deliberately disabled automatic missing-value parsing in pandas. This step tests whether the apparently complete data contains literal placeholder strings.

In this step, get ready to:
  • Inspect the DataFrame with deep memory reporting.
  • Measure the missing values currently visible to pandas.
  • Scan for literal placeholder markers to expose the disagreement.
Frame the designed obstacle

A Markdown cell records why this audit begins with the naive version. It makes the failure part of the reproducible notebook.

  • Add a Markdown cell beneath the naive-load code cell.
  • Paste the following content into the new cell:
## 2. Designed obstacle: missing values that pandas cannot see yet

We compare pandas' NA mask with a direct scan for common placeholder strings.

Why record the obstacle?

The heading identifies this result as an intentional test. The sentence names the two audits you are about to compare.

  • Run the Markdown cell to render its heading.
  • Save coursework_foundation.ipynb.

You'll see a section titled 2. Designed obstacle: missing values that pandas cannot see yet beneath the encounter preview.

Still seeing Markdown source text?

  • Confirm the new cell uses the Markdown cell type.
  • Run the cell again after changing its type.

Ask for help if the heading still does not render: Help me render this Markdown cell in my JupyterLab notebook.

Measure the naive pandas audit

The first audit checks only the values that pandas already recognizes as missing. Deep memory reporting also confirms that the full DataFrame is being inspected.

  • Add a code cell beneath the obstacle Markdown cell.
  • Paste the first audit into the new code cell:
raw_df.info(memory_usage="deep")

naive_mask = raw_df.isna()
naive_missing_total = sum(
    sum(bool(value) for value in naive_mask[column])
    for column in naive_mask.columns
)
print(f"Naive pandas NA total: {naive_missing_total:,}")

What does this audit do?

  • The raw_df.info(memory_usage="deep") call prints a detailed structural summary.
  • The naive_mask variable stores the missing-value result for every cell.
  • The naive_missing_total calculation converts that mask into one total.

Before you run this cell, do you expect the NA total to match how complete the preview looked?

  • Run the code cell to test the naive audit.
  • Save coursework_foundation.ipynb.

You'll see the deep DataFrame summary first. The audit then prints Naive pandas NA total: 0.

That zero is the intended shortfall. The mask only recognizes values pandas already classifies as missing.

Seeing a different naive total?

  • Rerun the earlier cell that creates raw_df with keep_default_na=False.
  • Confirm this audit uses raw_df instead of another DataFrame.

Ask for help with the unexpected count: Help me debug why my naive pandas missing-value total is not zero.

Scan for literal placeholders

An explicit marker scan asks whether each value is one of four placeholder strings. This tests the contents directly after the pandas audit reports zero.

  • Return to the obstacle code cell.
  • Append the following code after the Naive pandas NA total print statement:
candidate_markers = {"?", "NaN", "NULL", "None"}
marker_counts = {}
for column in raw_df.columns:
    marker_counts[column] = sum(
        str(value).strip() in candidate_markers
        for value in raw_df[column]
    )

nonzero_marker_counts = [
    (column, count)
    for column, count in marker_counts.items()
    if count > 0
]
nonzero_marker_counts.sort(key=lambda item: item[1], reverse=True)

print("Columns containing literal placeholder markers:")
for column, count in nonzero_marker_counts[:10]:
    print(f"  {column}: {count:,}")

assert naive_missing_total == 0, "The designed obstacle did not occur."
assert nonzero_marker_counts, "No configured placeholder markers were found."

How does the marker scan work?

  • The candidate_markers set defines the literal strings the scan recognizes.
  • The marker_counts dictionary stores one count for each column.
  • The nonzero_marker_counts list keeps only columns containing at least one configured marker.
  • The final assertions preserve the designed obstacle as a reproducible test.

Before you rerun the cell, can a zero pandas NA total coexist with nonzero marker counts?

  • Run the completed obstacle code cell.
  • Save coursework_foundation.ipynb.

You'll still see Naive pandas NA total: 0. You'll also see at least one column with a literal placeholder count greater than zero.

You have caught the hidden-data problem. The disagreement proves that literal placeholders remain ordinary values when default NA parsing is disabled.

No placeholder counts printed?

  • Confirm the scan uses the existing raw_df variable.
  • Check that candidate_markers matches the completed code.
  • Rerun the naive-load cell before rerunning the obstacle cell.

Ask for help if the marker list remains empty: Help me debug why my placeholder scan finds no markers in raw_df.

✔️ Awesome, I've got everything!

Your notebook now proves that the naive pandas audit misses literal placeholder strings.

ⓧ I'd like to double check the full code

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "title",
      "metadata": {},
      "source": [
        "# Healthcare Analytics Coursework 1 of 4: Data Foundation\n",
        "\n",
        "This notebook creates a reproducible, validated handoff for the later ICD-9 enrichment, EDA, and VCI projects."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "paths",
      "metadata": {},
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "\n",
        "DATA_DIR = Path(\"data\")\n",
        "RAW_DATA = DATA_DIR / \"diabetic_data.csv\"\n",
        "MAPPING_DATA = DATA_DIR / \"IDS_mapping.csv\"\n",
        "OUTPUT_DIR = Path(\"outputs\")\n",
        "OUTPUT_FILE = OUTPUT_DIR / \"cleaned_diabetic_data.csv\"\n",
        "\n",
        "for required_file in (RAW_DATA, MAPPING_DATA):\n",
        "    assert required_file.exists(), f\"Missing required file: {required_file}\"\n",
        "\n",
        "print(\"Both coursework files are available.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "naive-heading",
      "metadata": {},
      "source": [
        "## 1. Naive load and first visible result\n",
        "\n",
        "The first load deliberately disables automatic missing-value conversion so we can test whether placeholder text is hiding in the CSV."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "naive-load",
      "metadata": {},
      "outputs": [],
      "source": [
        "raw_df = pd.read_csv(RAW_DATA, keep_default_na=False, low_memory=False)\n",
        "\n",
        "print(f\"Rows: {raw_df.shape[0]:,}\")\n",
        "print(f\"Columns: {raw_df.shape[1]}\")\n",
        "assert raw_df.shape[0] == 101_766, \"Unexpected encounter count. Check the input file.\"\n",
        "raw_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "obstacle-heading",
      "metadata": {},
      "source": [
        "## 2. Designed obstacle: missing values that pandas cannot see yet\n",
        "\n",
        "We compare pandas' NA mask with a direct scan for common placeholder strings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "obstacle",
      "metadata": {},
      "outputs": [],
      "source": [
        "raw_df.info(memory_usage=\"deep\")\n",
        "\n",
        "naive_mask = raw_df.isna()\n",
        "naive_missing_total = sum(\n",
        "    sum(bool(value) for value in naive_mask[column])\n",
        "    for column in naive_mask.columns\n",
        ")\n",
        "print(f\"Naive pandas NA total: {naive_missing_total:,}\")\n",
        "\n",
        "candidate_markers = {\"?\", \"NaN\", \"NULL\", \"None\"}\n",
        "marker_counts = {}\n",
        "for column in raw_df.columns:\n",
        "    marker_counts[column] = sum(\n",
        "        str(value).strip() in candidate_markers\n",
        "        for value in raw_df[column]\n",
        "    )\n",
        "\n",
        "nonzero_marker_counts = [\n",
        "    (column, count)\n",
        "    for column, count in marker_counts.items()\n",
        "    if count > 0\n",
        "]\n",
        "nonzero_marker_counts.sort(key=lambda item: item[1], reverse=True)\n",
        "\n",
        "print(\"Columns containing literal placeholder markers:\")\n",
        "for column, count in nonzero_marker_counts[:10]:\n",
        "    print(f\"  {column}: {count:,}\")\n",
        "\n",
        "assert naive_missing_total == 0, \"The designed obstacle did not occur.\"\n",
        "assert nonzero_marker_counts, \"No configured placeholder markers were found.\""
      ]
    }
  ],
  "metadata": {},
  "nbformat": 4,
  "nbformat_minor": 5
}

What should match?

This reference includes every notebook cell through the designed-obstacle audit. The obstacle remains one complete code cell in the saved notebook.

Your notebook now demonstrates why a zero missing-value total can be misleading. Next, you'll reload the data with the correct missing-value parser.

Parse Missing Values and Remove Duplicate Rows

Your obstacle audit proved that raw_df can report zero missing values while literal placeholders remain.

The corrected load gives pandas permission to interpret placeholder text as missing data. This creates an auditable DataFrame before later clinical calculations begin.

In this step, get ready to:
  • Reload diabetic_data.csv with missing-value parsing enabled.
  • Rank the columns by their missing-value counts.
  • Remove exact duplicate encounter rows.
Reload the data with missing-value parsing

Missing-value parsing is decided when a CSV is read. The notebook therefore creates clean_df from RAW_DATA as a fresh DataFrame.

  • In coursework_foundation.ipynb, add a new cell below the obstacle code cell.
  • Change the new cell to Markdown.
  • Add the cleaning explanation by pasting this text:
## 3. Correct missing-value parsing and duplicate handling

The corrected load keeps pandas' default NA markers and adds the question-mark marker used by legacy copies of this dataset.

What does this text explain?

The corrected load keeps the missing markers that pandas already recognizes. It also treats ? as a missing value.

This makes the cleaning decision visible inside the notebook. Anyone rerunning the analysis can see why the missing-value total changes.

  • Run the Markdown cell.
  • Save coursework_foundation.ipynb.

You should see the rendered heading 3. Correct missing-value parsing and duplicate handling above its explanation.

Heading did not render?

  • Confirm that you changed the cell type to Markdown before running it.
  • Check that the heading starts with two hash characters followed by a space.

Help me fix a Markdown cell that stays as plain text in JupyterLab.

The new load combines pandas' default missing-value rules with the extra marker configured for this coursework. The resulting counts reveal the missingness hidden by keep_default_na=False.

  • Add a new code cell below the cleaning explanation.
  • Build the corrected missingness audit by pasting this code:
clean_df = pd.read_csv(
    RAW_DATA,
    na_values=["?"],
    keep_default_na=True,
    low_memory=False,
)

missing_mask = clean_df.isna()
missing_counts = {
    column: sum(bool(value) for value in missing_mask[column])
    for column in missing_mask.columns
}
corrected_missing_total = sum(missing_counts.values())
sorted_missing_counts = sorted(
    missing_counts.items(),
    key=lambda item: item[1],
    reverse=True,
)

print(f"Corrected pandas NA total: {corrected_missing_total:,}")
print("Ten columns with the most missing values:")
for column, count in sorted_missing_counts[:10]:
    print(f"  {column}: {count:,}")

What does this code do?

  • The pd.read_csv call reads RAW_DATA into the independent clean_df DataFrame.
  • The na_values=["?"] setting adds the question-mark placeholder to the missing-value parser.
  • The keep_default_na=True setting preserves pandas' default missing-value markers.
  • The missing_counts dictionary stores one missing-value total for each column.
  • The sorted_missing_counts list ranks those totals from highest to lowest.
  • Save coursework_foundation.ipynb.
  • Run the corrected missingness code cell.

You should see Corrected pandas NA total: followed by a value greater than zero. A ten-column missingness ranking should appear below it.

Good, your audit now exposes the missingness that the naive load concealed.

Corrected total still shows zero?

  • Confirm that the new load uses clean_df instead of reusing raw_df.
  • Check that keep_default_na=True appears exactly inside the pd.read_csv call.
  • Check that na_values=["?"] appears inside the same call.

Help me troubleshoot why my corrected missing-value total is still zero.

Remove exact duplicate rows

An exact duplicate repeats every value from another row. Counting duplicates before removing them preserves evidence of what changed.

  • Append the duplicate audit at the bottom of the corrected missingness code cell by pasting this code:
rows_before = clean_df.shape[0]
duplicate_count = sum(bool(value) for value in clean_df.duplicated())
clean_df = clean_df.drop_duplicates(ignore_index=True)
rows_after = clean_df.shape[0]

print(f"Exact duplicate rows: {duplicate_count:,}")
print(f"Rows before deduplication: {rows_before:,}")
print(f"Rows after deduplication: {rows_after:,}")
assert corrected_missing_total > 0, "Corrected load should detect missing values."

What does this code do?

  • The rows_before value records the encounter count before duplicate removal.
  • The duplicated check identifies rows whose complete contents match an earlier row.
  • The drop_duplicates call removes those repeated rows from clean_df.
  • The ignore_index=True setting gives the cleaned DataFrame a continuous row index.
  • The final assertion protects the corrected missing-value result from silently returning to zero.

Before you run the completed cell, do you expect the row count to stay the same or decrease?

  • Run the completed cleaning cell.

You should see Corrected pandas NA total: followed by a value greater than zero. The ten-column ranking should appear beneath it.

You should also see Exact duplicate rows: with the row totals from before and after deduplication.

That closes the data-quality gap: your notebook now parses missing markers and reports duplicate removal reproducibly.

  • Save coursework_foundation.ipynb.

Cleaning cell did not complete?

  • Run the earlier path setup cell if RAW_DATA is unavailable in the notebook session.
  • Run the corrected missingness code before checking the duplicate section if clean_df is unavailable.
  • Check that the duplicate code remains at the bottom of the same cleaning cell.

Help me debug my corrected missingness and duplicate-removal cell.

✔️ Awesome, I've got everything!

Your notebook now contains the corrected missingness audit and duplicate-removal evidence.

ⓧ I'd like to double check the full code

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "title",
      "metadata": {},
      "source": [
        "# Healthcare Analytics Coursework 1 of 4: Data Foundation\n",
        "\n",
        "This notebook creates a reproducible, validated handoff for the later ICD-9 enrichment, EDA, and VCI projects."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "paths",
      "metadata": {},
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "\n",
        "DATA_DIR = Path(\"data\")\n",
        "RAW_DATA = DATA_DIR / \"diabetic_data.csv\"\n",
        "MAPPING_DATA = DATA_DIR / \"IDS_mapping.csv\"\n",
        "OUTPUT_DIR = Path(\"outputs\")\n",
        "OUTPUT_FILE = OUTPUT_DIR / \"cleaned_diabetic_data.csv\"\n",
        "\n",
        "for required_file in (RAW_DATA, MAPPING_DATA):\n",
        "    assert required_file.exists(), f\"Missing required file: {required_file}\"\n",
        "\n",
        "print(\"Both coursework files are available.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "naive-heading",
      "metadata": {},
      "source": [
        "## 1. Naive load and first visible result\n",
        "\n",
        "The first load deliberately disables automatic missing-value conversion so we can test whether placeholder text is hiding in the CSV."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "naive-load",
      "metadata": {},
      "outputs": [],
      "source": [
        "raw_df = pd.read_csv(RAW_DATA, keep_default_na=False, low_memory=False)\n",
        "\n",
        "print(f\"Rows: {raw_df.shape[0]:,}\")\n",
        "print(f\"Columns: {raw_df.shape[1]}\")\n",
        "assert raw_df.shape[0] == 101_766, \"Unexpected encounter count. Check the input file.\"\n",
        "raw_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "obstacle-heading",
      "metadata": {},
      "source": [
        "## 2. Designed obstacle: missing values that pandas cannot see yet\n",
        "\n",
        "We compare pandas' NA mask with a direct scan for common placeholder strings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "obstacle",
      "metadata": {},
      "outputs": [],
      "source": [
        "raw_df.info(memory_usage=\"deep\")\n",
        "\n",
        "naive_mask = raw_df.isna()\n",
        "naive_missing_total = sum(\n",
        "    sum(bool(value) for value in naive_mask[column])\n",
        "    for column in naive_mask.columns\n",
        ")\n",
        "print(f\"Naive pandas NA total: {naive_missing_total:,}\")\n",
        "\n",
        "candidate_markers = {\"?\", \"NaN\", \"NULL\", \"None\"}\n",
        "marker_counts = {}\n",
        "for column in raw_df.columns:\n",
        "    marker_counts[column] = sum(\n",
        "        str(value).strip() in candidate_markers\n",
        "        for value in raw_df[column]\n",
        "    )\n",
        "\n",
        "nonzero_marker_counts = [\n",
        "    (column, count)\n",
        "    for column, count in marker_counts.items()\n",
        "    if count > 0\n",
        "]\n",
        "nonzero_marker_counts.sort(key=lambda item: item[1], reverse=True)\n",
        "\n",
        "print(\"Columns containing literal placeholder markers:\")\n",
        "for column, count in nonzero_marker_counts[:10]:\n",
        "    print(f\"  {column}: {count:,}\")\n",
        "\n",
        "assert naive_missing_total == 0, \"The designed obstacle did not occur.\"\n",
        "assert nonzero_marker_counts, \"No configured placeholder markers were found.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "clean-heading",
      "metadata": {},
      "source": [
        "## 3. Correct missing-value parsing and duplicate handling\n",
        "\n",
        "The corrected load keeps pandas' default NA markers and adds the question-mark marker used by legacy copies of this dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "clean",
      "metadata": {},
      "outputs": [],
      "source": [
        "clean_df = pd.read_csv(\n",
        "    RAW_DATA,\n",
        "    na_values=[\"?\"],\n",
        "    keep_default_na=True,\n",
        "    low_memory=False,\n",
        ")\n",
        "\n",
        "missing_mask = clean_df.isna()\n",
        "missing_counts = {\n",
        "    column: sum(bool(value) for value in missing_mask[column])\n",
        "    for column in missing_mask.columns\n",
        "}\n",
        "corrected_missing_total = sum(missing_counts.values())\n",
        "sorted_missing_counts = sorted(\n",
        "    missing_counts.items(),\n",
        "    key=lambda item: item[1],\n",
        "    reverse=True,\n",
        ")\n",
        "\n",
        "print(f\"Corrected pandas NA total: {corrected_missing_total:,}\")\n",
        "print(\"Ten columns with the most missing values:\")\n",
        "for column, count in sorted_missing_counts[:10]:\n",
        "    print(f\"  {column}: {count:,}\")\n",
        "\n",
        "rows_before = clean_df.shape[0]\n",
        "duplicate_count = sum(bool(value) for value in clean_df.duplicated())\n",
        "clean_df = clean_df.drop_duplicates(ignore_index=True)\n",
        "rows_after = clean_df.shape[0]\n",
        "\n",
        "print(f\"Exact duplicate rows: {duplicate_count:,}\")\n",
        "print(f\"Rows before deduplication: {rows_before:,}\")\n",
        "print(f\"Rows after deduplication: {rows_after:,}\")\n",
        "assert corrected_missing_total > 0, \"Corrected load should detect missing values.\""
      ]
    }
  ],
  "metadata": {},
  "nbformat": 4,
  "nbformat_minor": 5
}

Your notebook now has an auditable clean_df with corrected missingness and exact duplicates removed. Next, you will turn the original readmission labels into one validated binary target.

Create the Readmission Target and Export the Handoff

Your JupyterLab notebook now turns hidden placeholders into genuine missing values. It also removes exact duplicate rows.

Later analysis needs one validated outcome for readmission within 30 days. A stable cleaned file also prevents each project from reinterpreting the raw labels.

In this step, get ready to:
  • Create a binary readmission target with no unmapped labels.
  • Export the cleaned encounters as the Project 2 handoff.
  • Prove that the reloaded handoff matches the in-memory DataFrame.
Create the binary readmission target

A binary target gives later analysis one consistent outcome. The value 1 represents readmission within 30 days. The value 0 represents every other recorded outcome.

  • Switch back to coursework_foundation.ipynb in JupyterLab.
  • Add a Markdown cell below the cleaning cell.
  • Paste this explanation into the new Markdown cell:
## 4. Binary 30-day readmission target

The later projects need one explicit outcome: 1 for readmission within 30 days and 0 otherwise.

What does this cell add?

This heading records the meaning of the target before the notebook creates it. Anyone reopening the notebook can interpret readmitted_30d without tracing the mapping code first.

  • Run the Markdown cell with the notebook run control.

You should see a rendered section titled 4. Binary 30-day readmission target below the cleaning results.

  • Add a code cell below the rendered target explanation.
  • Paste the target creation code into the new cell:
readmission_map = {"<30": 1, ">30": 0, "NO": 0, "No": 0}
clean_df["readmitted_30d"] = clean_df["readmitted"].map(readmission_map)

target_missing_mask = clean_df[["readmitted_30d"]].isna()
unmapped_target_count = sum(
    bool(value) for value in target_missing_mask["readmitted_30d"]
)
assert unmapped_target_count == 0, "Unexpected readmitted label found."

readmitted_within_30 = sum(
    int(value) == 1 for value in clean_df["readmitted_30d"]
)
not_readmitted_within_30 = clean_df.shape[0] - readmitted_within_30

print(f"Readmitted within 30 days: {readmitted_within_30:,}")
print(f"Not readmitted within 30 days: {not_readmitted_within_30:,}")

What does this code do?

  • The readmission_map dictionary maps <30 to 1. It maps >30, NO, and No to 0.
  • The pandas mapping operation creates readmitted_30d from the existing readmitted labels.
  • The unmapped_target_count assertion stops the notebook if an unexpected label would create a missing target.
  • The final calculations count both target classes so you can see the outcome distribution.
  • Save coursework_foundation.ipynb.

Before you run this cell, predict whether every source label appears in readmission_map.

  • Run the target code cell with the notebook run control.

You should see separate counts for encounters readmitted within 30 days and encounters outside that class. The cell should finish without an assertion error.

Seeing an unmapped target?

  • Confirm that the cleaning cell ran before the target cell so clean_df contains the deduplicated encounters.
  • Compare the values in readmitted with the keys shown in readmission_map.
  • Check the mapping line for missing quotes or changed capitalization.

Help me diagnose an unmapped readmission label in my notebook.

Export and validate the handoff

A CSV export gives Project 2 a stable input outside the current notebook session. Reloading it immediately proves that the file on disk retains the expected shape.

  • Add a Markdown cell below the target code cell.
  • Paste this handoff explanation into the new Markdown cell:
## 5. Export and validate the Project 2 handoff

The exported file is reloaded immediately so the handoff is tested independently of notebook memory.

Why reload the export?

The in-memory clean_df proves that the notebook logic worked. Reloading the exported file checks the artifact that Project 2 actually receives.

  • Run the Markdown cell with the notebook run control.

You should see a rendered section titled 5. Export and validate the Project 2 handoff.

  • Add a code cell below the rendered handoff explanation.
  • Paste the export validation code into the new cell:
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
clean_df.to_csv(OUTPUT_FILE, index=False)

check_df = pd.read_csv(OUTPUT_FILE, low_memory=False)
assert OUTPUT_FILE.exists(), "The cleaned CSV was not created."
assert check_df.shape == clean_df.shape, "Reloaded shape does not match."

print(f"Saved: {OUTPUT_FILE}")
print(f"Validated shape: {check_df.shape}")
print("Project 1 handoff is ready for Project 2.")

What does this code do?

  • The OUTPUT_DIR.mkdir() call creates outputs when the folder is missing.
  • The to_csv() call writes every cleaned column without adding a pandas index.
  • The reload creates check_df from the saved handoff.
  • The shape assertion proves that the exported row count and column count match clean_df.
  • Save coursework_foundation.ipynb.

Before you run this cell, predict whether the reloaded file keeps the new readmitted_30d column.

  • Run the export code cell with the notebook run control.

You should see Saved: outputs\cleaned_diabetic_data.csv, a validated shape, and the Project 2 readiness message. The cell should finish without an assertion error.

Handoff missing or mismatched?

  • Run the path setup cell again if OUTPUT_DIR or OUTPUT_FILE is missing from notebook memory.
  • Run the target cell again if the validated column count does not include readmitted_30d.
  • Confirm that the export cell uses index=False so the saved file does not gain an extra index column.

Help me troubleshoot my cleaned CSV export or shape validation.

The notebook closes with interpretation prompts that connect the audit evidence to later coursework. These notes keep the reasoning beside the reproducible output.

  • Add a Markdown cell below the export code cell.
  • Paste the interpretation prompts into the new Markdown cell:
## Interpretation notes

Complete these in your own words before submission:

1. Why did the naive missing-value audit fail?
2. Which columns have the greatest missingness, and what does that imply for later analysis?
3. How many exact duplicate rows were found?
4. Why is `readmitted_30d` useful for the later EDA and VCI projects?

What do these prompts capture?

The questions preserve your interpretation of the missing-value obstacle, the missingness ranking, the duplicate audit, and the binary target. This turns the notebook into an auditable analysis record.

  • Run the final Markdown cell with the notebook run control.
  • Save coursework_foundation.ipynb.

You should see four numbered interpretation questions at the bottom of the notebook.

✔️ Awesome, I've got everything!

Your notebook now contains the complete loading, auditing, cleaning, target creation, and export workflow.

  • Save coursework_foundation.ipynb before the final full run.

ⓧ I'd like to double check the full code

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "title",
      "metadata": {},
      "source": [
        "# Healthcare Analytics Coursework 1 of 4: Data Foundation\n",
        "\n",
        "This notebook creates a reproducible, validated handoff for the later ICD-9 enrichment, EDA, and VCI projects."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "paths",
      "metadata": {},
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "\n",
        "DATA_DIR = Path(\"data\")\n",
        "RAW_DATA = DATA_DIR / \"diabetic_data.csv\"\n",
        "MAPPING_DATA = DATA_DIR / \"IDS_mapping.csv\"\n",
        "OUTPUT_DIR = Path(\"outputs\")\n",
        "OUTPUT_FILE = OUTPUT_DIR / \"cleaned_diabetic_data.csv\"\n",
        "\n",
        "for required_file in (RAW_DATA, MAPPING_DATA):\n",
        "    assert required_file.exists(), f\"Missing required file: {required_file}\"\n",
        "\n",
        "print(\"Both coursework files are available.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "naive-heading",
      "metadata": {},
      "source": [
        "## 1. Naive load and first visible result\n",
        "\n",
        "The first load deliberately disables automatic missing-value conversion so we can test whether placeholder text is hiding in the CSV."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "naive-load",
      "metadata": {},
      "outputs": [],
      "source": [
        "raw_df = pd.read_csv(RAW_DATA, keep_default_na=False, low_memory=False)\n",
        "\n",
        "print(f\"Rows: {raw_df.shape[0]:,}\")\n",
        "print(f\"Columns: {raw_df.shape[1]}\")\n",
        "assert raw_df.shape[0] == 101_766, \"Unexpected encounter count. Check the input file.\"\n",
        "raw_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "obstacle-heading",
      "metadata": {},
      "source": [
        "## 2. Designed obstacle: missing values that pandas cannot see yet\n",
        "\n",
        "We compare pandas' NA mask with a direct scan for common placeholder strings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "obstacle",
      "metadata": {},
      "outputs": [],
      "source": [
        "raw_df.info(memory_usage=\"deep\")\n",
        "\n",
        "naive_mask = raw_df.isna()\n",
        "naive_missing_total = sum(\n",
        "    sum(bool(value) for value in naive_mask[column])\n",
        "    for column in naive_mask.columns\n",
        ")\n",
        "print(f\"Naive pandas NA total: {naive_missing_total:,}\")\n",
        "\n",
        "candidate_markers = {\"?\", \"NaN\", \"NULL\", \"None\"}\n",
        "marker_counts = {}\n",
        "for column in raw_df.columns:\n",
        "    marker_counts[column] = sum(\n",
        "        str(value).strip() in candidate_markers\n",
        "        for value in raw_df[column]\n",
        "    )\n",
        "\n",
        "nonzero_marker_counts = [\n",
        "    (column, count)\n",
        "    for column, count in marker_counts.items()\n",
        "    if count > 0\n",
        "]\n",
        "nonzero_marker_counts.sort(key=lambda item: item[1], reverse=True)\n",
        "\n",
        "print(\"Columns containing literal placeholder markers:\")\n",
        "for column, count in nonzero_marker_counts[:10]:\n",
        "    print(f\"  {column}: {count:,}\")\n",
        "\n",
        "assert naive_missing_total == 0, \"The designed obstacle did not occur.\"\n",
        "assert nonzero_marker_counts, \"No configured placeholder markers were found.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "clean-heading",
      "metadata": {},
      "source": [
        "## 3. Correct missing-value parsing and duplicate handling\n",
        "\n",
        "The corrected load keeps pandas' default NA markers and adds the question-mark marker used by legacy copies of this dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "clean",
      "metadata": {},
      "outputs": [],
      "source": [
        "clean_df = pd.read_csv(\n",
        "    RAW_DATA,\n",
        "    na_values=[\"?\"],\n",
        "    keep_default_na=True,\n",
        "    low_memory=False,\n",
        ")\n",
        "\n",
        "missing_mask = clean_df.isna()\n",
        "missing_counts = {\n",
        "    column: sum(bool(value) for value in missing_mask[column])\n",
        "    for column in missing_mask.columns\n",
        "}\n",
        "corrected_missing_total = sum(missing_counts.values())\n",
        "sorted_missing_counts = sorted(\n",
        "    missing_counts.items(),\n",
        "    key=lambda item: item[1],\n",
        "    reverse=True,\n",
        ")\n",
        "\n",
        "print(f\"Corrected pandas NA total: {corrected_missing_total:,}\")\n",
        "print(\"Ten columns with the most missing values:\")\n",
        "for column, count in sorted_missing_counts[:10]:\n",
        "    print(f\"  {column}: {count:,}\")\n",
        "\n",
        "rows_before = clean_df.shape[0]\n",
        "duplicate_count = sum(bool(value) for value in clean_df.duplicated())\n",
        "clean_df = clean_df.drop_duplicates(ignore_index=True)\n",
        "rows_after = clean_df.shape[0]\n",
        "\n",
        "print(f\"Exact duplicate rows: {duplicate_count:,}\")\n",
        "print(f\"Rows before deduplication: {rows_before:,}\")\n",
        "print(f\"Rows after deduplication: {rows_after:,}\")\n",
        "assert corrected_missing_total > 0, \"Corrected load should detect missing values.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "target-heading",
      "metadata": {},
      "source": [
        "## 4. Binary 30-day readmission target\n",
        "\n",
        "The later projects need one explicit outcome: 1 for readmission within 30 days and 0 otherwise."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "target",
      "metadata": {},
      "outputs": [],
      "source": [
        "readmission_map = {\"<30\": 1, \">30\": 0, \"NO\": 0, \"No\": 0}\n",
        "clean_df[\"readmitted_30d\"] = clean_df[\"readmitted\"].map(readmission_map)\n",
        "\n",
        "target_missing_mask = clean_df[[\"readmitted_30d\"]].isna()\n",
        "unmapped_target_count = sum(\n",
        "    bool(value) for value in target_missing_mask[\"readmitted_30d\"]\n",
        ")\n",
        "assert unmapped_target_count == 0, \"Unexpected readmitted label found.\"\n",
        "\n",
        "readmitted_within_30 = sum(\n",
        "    int(value) == 1 for value in clean_df[\"readmitted_30d\"]\n",
        ")\n",
        "not_readmitted_within_30 = clean_df.shape[0] - readmitted_within_30\n",
        "\n",
        "print(f\"Readmitted within 30 days: {readmitted_within_30:,}\")\n",
        "print(f\"Not readmitted within 30 days: {not_readmitted_within_30:,}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "export-heading",
      "metadata": {},
      "source": [
        "## 5. Export and validate the Project 2 handoff\n",
        "\n",
        "The exported file is reloaded immediately so the handoff is tested independently of notebook memory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "export",
      "metadata": {},
      "outputs": [],
      "source": [
        "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
        "clean_df.to_csv(OUTPUT_FILE, index=False)\n",
        "\n",
        "check_df = pd.read_csv(OUTPUT_FILE, low_memory=False)\n",
        "assert OUTPUT_FILE.exists(), \"The cleaned CSV was not created.\"\n",
        "assert check_df.shape == clean_df.shape, \"Reloaded shape does not match.\"\n",
        "\n",
        "print(f\"Saved: {OUTPUT_FILE}\")\n",
        "print(f\"Validated shape: {check_df.shape}\")\n",
        "print(\"Project 1 handoff is ready for Project 2.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "interpretation",
      "metadata": {},
      "source": [
        "## Interpretation notes\n",
        "\n",
        "Complete these in your own words before submission:\n",
        "\n",
        "1. Why did the naive missing-value audit fail?\n",
        "2. Which columns have the greatest missingness, and what does that imply for later analysis?\n",
        "3. How many exact duplicate rows were found?\n",
        "4. Why is `readmitted_30d` useful for the later EDA and VCI projects?"
      ]
    }
  ],
  "metadata": {},
  "nbformat": 4,
  "nbformat_minor": 5
}

Before the full run, predict whether reloading the export preserves the final DataFrame shape.

  • Run coursework_foundation.ipynb from its first cell through its final code cell.

You should see Saved: outputs\cleaned_diabetic_data.csv near the end of the run. You should also see a validated shape that includes the new target column.

That completes the handoff. Your exported clinical dataset can now be reopened independently for Project 2.

Secret mission

Turn the Audit into a One-Command Check

Your notebook validates the handoff, but those checks still depend on an interactive session. This mission turns the audit into a standalone command that verifies the exported data directly from disk.

Clean Up Your Resources

Clean Up Your Resources

Your workspace runs entirely on this Windows machine, so there are no cloud resources or ongoing service costs. Choose whether to keep the workspace, pause the running tools, or remove the local folder after backing up your final submission.

Resources you used:

  • The JupyterLab server that is still running from Windows Command Prompt.
  • The healthcare-coursework folder, including its tutorial-env virtual environment, both source CSV files, requirements.txt, coursework_foundation.ipynb, outputs/cleaned_diabetic_data.csv, and audit_snapshot.py.

Keep everything running

No action is needed. Choose this option while you are completing the remaining coursework projects or checking your notebook results.

  • Leave the current JupyterLab server running while you continue working.
  • Keep the healthcare-coursework folder until all four coursework projects are complete.
  • Store the healthcare-coursework folder privately because its data folder contains row-level patient encounter data.

Pause - I'll come back to this later

Stop JupyterLab to free the resources it is using while keeping every project file intact. Your local files remain ready for the next coursework project.

  • Switch back to coursework_foundation.ipynb in JupyterLab.
  • Press Ctrl+S to save the notebook.
  • Return to the Windows Command Prompt window from earlier.
  • Press Ctrl+C to stop JupyterLab.
  • Exit the activated virtual environment by running this command:
deactivate

What Does This Command Do?

The deactivate command exits tutorial-env without deleting its installed packages or project files.

Start Again Later

The saved virtual environment restores the same pinned JupyterLab and pandas versions when you return.

  • Return to the healthcare-coursework project root in Windows Command Prompt.
  • Reactivate the environment and restart JupyterLab by running these commands:
tutorial-env\Scripts\activate
jupyter lab

The first command activates tutorial-env. The second command launches the JupyterLab workspace in your browser.

Delete - I don't want to use this again

Remove the complete local workspace only after your final submission is safely backed up. Deleting the folder permanently removes the notebook and the handoff required by Project 2.

  • Confirm that your final coursework submission is safely backed up.
  • Return to the Windows Command Prompt window from earlier.
  • Press Ctrl+C to stop JupyterLab.
  • Exit the activated virtual environment by running this command:
deactivate

What Does This Command Do?

The deactivate command releases the active environment before you remove its folder.

  • Use File Explorer to go to the folder that contains healthcare-coursework.
  • Right-click the healthcare-coursework folder.
  • Select Delete.
  • Empty the Windows Recycle Bin to complete the removal.

The tutorial-env environment, source CSV copies, notebook, cleaned export, requirements file, and audit script are now removed from this machine.

Nice Work!

Nice Work!

You did it! You built a reproducible clinical-data foundation that turns 101,766 raw hospital encounters into a validated handoff for Project 2.

What you learned:

  • Created a local JupyterLab workspace inside an isolated Python virtual environment. You can reopen the same pinned notebook setup throughout the four-project coursework sequence.
  • Loaded all 101,766 encounter rows with pandas. You proved that a naive missing-value audit can report zero while literal placeholders remain.
  • Parsed hidden missing values. Removed exact duplicate rows. Built the binary readmitted_30d target. Exported the validated outputs/cleaned_diabetic_data.csv handoff for Project 2.
  • Secret Mission: Created a standalone one-command audit with audit_snapshot.py. The script validates the exported handoff without relying on notebook state.

Ready to quiz yourself?