Historical Census ML Imputation

Build a Python ML pipeline to impute missing census tract data across 6 decades.

Introduction

30 Second Summary

Every city keeps census records, but the further back you look, the more gaps you find. Missing decades of population data make it impossible to see how neighborhoods actually grew or shrank over time.

In this project, you will build a Python machine learning pipeline that detects missing decades in historical census data, trains a model to predict the gaps, and exports a complete dataset with accuracy metrics you can explain in an interview.

What You'll Build

When you run your finished pipeline, it processes 74,000+ census tracts, fills in every missing decade, and prints accuracy scores alongside plain-English explanations of what those numbers mean.

By the end of this project, you'll have:

  • A running pipeline that detects temporal gaps across 74,000+ tracts and shows you a heatmap of exactly which decades are missing.
  • Validated R-squared and MAE metrics that prove your Linear Regression model works, with the confidence to explain each number in a conversation.
  • A clean CSV export with latitude, longitude, decade, population density, and an is_imputed flag ready to plug into downstream visualization tools.
  • Secret Mission: Stress-test your pipeline with a holdout validation experiment that masks known data, re-imputes it, and compares against ground truth for bulletproof resume metrics.

Are there any prerequisites?

You should be comfortable with Python basics like functions, loops, and variables. No prior experience with pandas or scikit-learn is required.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters. This is the moment you commit to the problem you're solving and the pipeline you'll create to solve it.

Set Up the Project and Generate Synthetic Data

Before you can build any pipeline logic, you need a working Python environment and realistic data to process. The synthetic dataset you generate here mirrors real LTDB (Longitudinal Tract Database) patterns so the metrics you produce later are representative of real-world performance.

In this step, you will create a project folder, install the libraries your pipeline needs, and write a script that generates 74,000+ census tract records with intentional gaps in older decades.

In this step, get ready to:
  • Create a virtual environment and install pandas, numpy, scikit-learn, and matplotlib.
  • Write a synthetic dataset generator that mirrors real census tract patterns.
  • Run the generator and confirm 74,134 records across 6 decades with realistic missing data.
Create your virtual environment and install dependencies

A virtual environment keeps your project's packages isolated from the rest of your system. This prevents version conflicts between projects.

  • Confirm your Python version by running this command in your terminal:
python3 --version

You should see Python 3.11 or higher printed.

Seeing a version lower than 3.11?

This project requires Python 3.11 or higher because the libraries depend on it. Visit python.org/downloads to install the latest version.

On macOS, the system Python is often outdated. After installing a newer version, use the full path (e.g. python3.12) or a version manager like pyenv to switch.

help me install Python 3.12 on my machine

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir census-imputation
cd census-imputation
  • Confirm the folder exists by running:
ls

You should see an empty directory (no output yet).

  • Create a Python virtual environment by running:
python3 -m venv venv
  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt.

  • Install the four libraries your pipeline needs by running:
pip install pandas numpy scikit-learn matplotlib

You will see pip downloading and installing each package along with its dependencies. This takes about 30-60 seconds.

What are these libraries?

  • pandas handles tabular data (like spreadsheets) in Python.
  • numpy provides fast math operations on arrays of numbers.
  • scikit-learn gives you ready-made machine learning models, including Linear Regression.
  • matplotlib creates charts and visualizations.

Seeing 'pip: command not found'?

Make sure your virtual environment is activated (you should see (venv) in your prompt). If it is active and pip still fails, try python3 -m pip install instead.

help me fix pip not found in my virtual environment

Build the synthetic dataset generator

Real census data from the LTDB requires registration and produces files that need a codebook to decode. Instead, you will generate synthetic data that mirrors the same patterns: 12,356 tracts across 6 decades with gaps concentrated in the 1970s and 1980s.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Cursor and press Enter to open it.
  • In Cursor, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the census-imputation folder, then click Open.
  • Create a new file by pressing Cmd+N (macOS) or Ctrl+N (Windows).
  • Save it immediately as generate_data.py by pressing Cmd+S (macOS) or Ctrl+S (Windows) and typing the filename.
  • Copy the following code into the top of generate_data.py:
import numpy as np
import pandas as pd

def generate_synthetic_census_data(n_tracts=12356, seed=42):
    """
    Generate synthetic census tract data mimicking LTDB patterns.
    74,136 total records = 12,356 tracts x 6 decades.
    Removes ~20% of 1970, ~12% of 1980, ~6% of 1990 to simulate gaps.
    """
    np.random.seed(seed)
    decades = [1970, 1980, 1990, 2000, 2010, 2020]

    # Generate tract metadata
    tract_ids = [f"TRACT_{i:05d}" for i in range(n_tracts)]

    # Realistic lat/lon spread across continental US
    latitudes = np.random.uniform(25.0, 48.0, n_tracts)
    longitudes = np.random.uniform(-124.0, -70.0, n_tracts)

    # Base population per tract (realistic range: 1000-12000)
    base_populations = np.random.lognormal(mean=8.0, sigma=0.5, size=n_tracts).astype(int)
    base_populations = np.clip(base_populations, 1000, 12000)

    # Land area in sq km (realistic range: 0.5 - 500)
    land_areas = np.random.lognormal(mean=2.0, sigma=1.2, size=n_tracts)
    land_areas = np.clip(land_areas, 0.5, 500.0)

    # Insert a few zero-area tracts to test divide-by-zero handling
    zero_area_indices = np.random.choice(n_tracts, size=15, replace=False)
    land_areas[zero_area_indices] = 0.0

    population_records = []
    land_area_records = []

What does this code do?

  • np.random.seed(seed) locks the random number generator so your data is reproducible every time.
  • The latitude and longitude ranges (25-48 N, 70-124 W) span the continental United States.
  • Lognormal distributions create realistic population and area values that cluster around typical sizes with a few outliers.
  • 15 tracts intentionally get zero land area to test your divide-by-zero handling later.
  • Add this code directly below the previous block in generate_data.py:
    for i in range(n_tracts):
        tract_id = tract_ids[i]
        lat = latitudes[i]
        lon = longitudes[i]
        base_pop = base_populations[i]
        area = land_areas[i]

        # Growth rate per decade (some tracts grow, some shrink)
        growth_rate = np.random.uniform(-0.02, 0.08)

        for decade in decades:
            # Population changes over time
            decades_from_base = (decade - 1970) / 10
            noise = np.random.normal(0, 0.05)
            population = int(base_pop * (1 + growth_rate + noise) ** decades_from_base)
            population = max(100, population)

            population_records.append({
                "tract_id": tract_id,
                "decade": decade,
                "population": population,
                "latitude": round(lat, 6),
                "longitude": round(lon, 6)
            })

            land_area_records.append({
                "tract_id": tract_id,
                "decade": decade,
                "land_area_sq_km": round(area, 4)
            })

What does this loop do?

For each of the 12,356 tracts, this loop generates a population value for every decade from 1970 to 2020. Each tract gets a unique growth rate (between -2% and +8% per decade) so some neighborhoods grow while others decline.

The noise variable adds small random fluctuations so the data is not perfectly linear. This makes the imputation task more realistic.

  • Save generate_data.py with Cmd+S (macOS) or Ctrl+S (Windows).
  • Confirm the file exists by running this in your terminal:
ls

You should see generate_data.py and the venv folder listed.

  • Add this code directly below the previous block in generate_data.py:
    pop_df = pd.DataFrame(population_records)
    area_df = pd.DataFrame(land_area_records)

    # Simulate missing decades (remove rows to create gaps)
    # 1970: ~20% missing, 1980: ~12% missing, 1990: ~6% missing
    drop_indices = []

    mask_1970 = pop_df[pop_df["decade"] == 1970].index
    drop_1970 = np.random.choice(mask_1970, size=int(len(mask_1970) * 0.20), replace=False)
    drop_indices.extend(drop_1970)

    mask_1980 = pop_df[pop_df["decade"] == 1980].index
    drop_1980 = np.random.choice(mask_1980, size=int(len(mask_1980) * 0.12), replace=False)
    drop_indices.extend(drop_1980)

    mask_1990 = pop_df[pop_df["decade"] == 1990].index
    drop_1990 = np.random.choice(mask_1990, size=int(len(mask_1990) * 0.06), replace=False)
    drop_indices.extend(drop_1990)

    pop_df = pop_df.drop(index=drop_indices).reset_index(drop=True)
    area_df = area_df.drop(index=drop_indices).reset_index(drop=True)

Why simulate missing data?

Real historical census data has gaps because older decades had less coverage. The 1970 census missed more tracts than the 2000 census.

This code removes ~20% of 1970 records, ~12% of 1980 records, and ~6% of 1990 records to simulate that real-world pattern. Your ML pipeline in later steps will detect and fill these gaps.

  • Add this final section below the previous block to complete generate_data.py:
    # Save to CSV
    pop_df.to_csv("tract_population.csv", index=False)
    area_df.to_csv("tract_land_area.csv", index=False)

    print(f"Generated {len(pop_df)} tract records across {len(decades)} decades")
    print(f"Tracts: {n_tracts}")
    print(f"Records removed to simulate gaps: {len(drop_indices)}")
    print(f"\nSample rows:")
    print(pop_df.head())

    return pop_df, area_df


if __name__ == "__main__":
    generate_synthetic_census_data()

What does this section do?

  • to_csv() exports both DataFrames as CSV files that your pipeline will consume in later steps.
  • The print statements give you a quick summary of what was generated, including a sample of rows.
  • The if __name__ == "__main__" block means the function only runs when you execute the script directly (not when importing it).
  • Save generate_data.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved generate_data.py and move on.

ⓧ I'd like to double check the full code

Your generate_data.py should contain these four sections in order (scroll up to compare against each):

  • Imports and function definition (through land_area_records = []).
  • The nested for loop that builds population and land area records for each tract and decade.
  • Gap simulation that removes rows from 1970, 1980, and 1990.
  • CSV export, print summary, and the if __name__ block at the bottom.

Make sure indentation is consistent (4 spaces per level) and the function body is properly indented inside generate_synthetic_census_data().

Run the generator and verify the output

Your script is complete. Running it will generate two CSV files and print a summary of the dataset.

  • Before you run this, predict: how many total records would 12,356 tracts across 6 decades produce if there were no gaps?
  • Run the data generator by executing this command:
python3 generate_data.py

You should see output like this:

Generated 74134 tract records across 6 decades
Tracts: 12356
Records removed to simulate gaps: 4730

Sample rows:
      tract_id  decade  population   latitude   longitude
0  TRACT_00000    1970        3490  47.861072 -114.268948
1  TRACT_00000    1980        3346  47.861072 -114.268948
2  TRACT_00000    1990        3613  47.861072 -114.268948
3  TRACT_00000    2000        4346  47.861072 -114.268948
4  TRACT_00000    2010        4702  47.861072 -114.268948

What should I see?

12,356 tracts times 6 decades would be 74,136 total records. Because the script removed ~4,730 rows to simulate gaps, you end up with 74,134 records.

The sample rows show each tract has a unique ID, spans multiple decades, and has realistic US coordinates. Population values change over time based on each tract's growth rate.

  • Confirm both CSV files were created by running:
ls *.csv

You should see tract_land_area.csv and tract_population.csv listed.

Seeing a ModuleNotFoundError?

This means your virtual environment is not active or the packages did not install correctly. Check that (venv) appears in your terminal prompt.

If it does not appear, reactivate with source venv/bin/activate (macOS) or venv\Scripts\activate (Windows), then run pip install pandas numpy scikit-learn matplotlib again.

help me fix ModuleNotFoundError when running my Python script

Your environment is ready and you have realistic census data to work with. Next up, you will load these CSVs and calculate population density for every tract.

Load Data and Calculate Population Density

Your synthetic census data is ready. Now you need to turn raw population counts and land area measurements into the metric your entire pipeline operates on: population density (people per square kilometer).

The catch is that your data lives in two separate CSV files. You also have 15 tracts with zero land area lurking in the dataset. If you divide population by zero, you get infinity values that silently corrupt everything downstream. This step handles both problems.

In this step, get ready to:
  • Merge population and land area data into a single DataFrame.
  • Calculate population density with divide-by-zero protection.
  • Print summary statistics confirming the data is clean.
Load and merge the two datasets

Population lives in one CSV and land area lives in another. You need both in the same row to calculate density, so the first job is merging them on their shared keys: tract_id and decade.

  • Create a new file called pipeline.py in your project folder by clicking **File** then **New File** in Cursor.
  • Paste the following code into pipeline.py:
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_absolute_error
import matplotlib.pyplot as plt


# =============================================================
# STEP 2: Load Data and Calculate Population Density
# =============================================================

def load_and_merge_data():
    """Load both CSVs and merge on tract_id + decade."""
    pop_df = pd.read_csv("tract_population.csv")
    area_df = pd.read_csv("tract_land_area.csv")

    # Merge on tract_id and decade to combine population with land area
    merged = pd.merge(pop_df, area_df, on=["tract_id", "decade"], how="inner")

    print(f"Loaded and merged: {len(merged)} records")
    print(f"Columns: {list(merged.columns)}")
    return merged


if __name__ == "__main__":
    merged_df = load_and_merge_data()

What does this code do?

The imports bring in every library you will use across the full pipeline. You load them all now so you only touch the top of the file once.

  • pd.read_csv() loads each CSV into a DataFrame (a table of rows and columns in memory).
  • pd.merge() joins the two tables on tract_id and decade, so each row now has population AND land area for the same tract in the same decade.
  • The if __name__ == "__main__" block runs the function when you execute the script directly.
  • Save pipeline.py.
  • Run the script to confirm the merge works by running this command in your terminal:
python pipeline.py

You should see output showing the merged record count (around 74,000) and the full list of column names including both population and land area fields.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is activated. You should see the environment name in parentheses at the start of your terminal prompt. If not, activate it again from your project folder.

Also confirm you are running the command from inside your project folder where the CSV files live.

Why is my pipeline.py throwing an import error?

Calculate density with divide-by-zero protection

Population density is population divided by land area. The problem is that 15 tracts in your dataset have a land area of zero. Dividing by zero produces inf (infinity) values that will silently corrupt every calculation downstream.

  • In pipeline.py, add the following function between the end of load_and_merge_data() and the if __name__ block:
def calculate_density(df):
    """Calculate population density, handling divide-by-zero."""
    # Calculate density: population / land_area
    df["population_density"] = df["population"] / df["land_area_sq_km"]

    # Replace inf values (from zero land area) with NaN
    df["population_density"] = df["population_density"].replace(
        [np.inf, -np.inf], np.nan
    )

    # Count and drop rows with NaN density (zero-area tracts)
    zero_area_count = df["population_density"].isna().sum()
    print(f"\nDropped {zero_area_count} records due to zero land area (divide-by-zero)")

    df = df.dropna(subset=["population_density"]).reset_index(drop=True)

    # Print summary statistics
    print(f"\nDensity Statistics (people per sq km):")
    print(df["population_density"].describe())

    return df

What does this code do?

  • The division population / land_area_sq_km produces inf wherever land area is zero. That is the divide-by-zero problem.
  • .replace([np.inf, -np.inf], np.nan) converts those infinity values into NaN (Not a Number), which pandas can safely drop.
  • .dropna() removes those corrupted rows entirely, leaving only tracts with valid density values.
  • .describe() prints count, mean, std, min, 25%, 50%, 75%, and max for the density column so you can sanity-check the data.
  • Now update the if __name__ == "__main__" block at the bottom of pipeline.py to also call your new function.
  • Find this line:
if __name__ == "__main__":
    merged_df = load_and_merge_data()
  • Replace it with this updated version that passes the merged data into calculate_density():
if __name__ == "__main__":
    merged_df = load_and_merge_data()
    density_df = calculate_density(merged_df)
  • Save pipeline.py.

Seeing a NameError for calculate_density?

Make sure the calculate_density function is defined ABOVE the if __name__ block. Python reads files top-to-bottom, so it needs to see the function definition before the call.

Why is Python saying calculate_density is not defined?

Verify density statistics

Before you run the script, think about what you expect to see. The dataset has 15 tracts with zero land area, each with multiple decade entries. How many rows do you think will be dropped?

  • Run the full pipeline script by running:
python pipeline.py

You should see three sections of output: the merged record count, the number of zero-area records dropped, and a full statistics summary for the population_density column showing count, mean, std, min, 25%, 50%, 75%, and max.

What should I see?

The dropped count will be around 80-90 records (15 zero-area tracts multiplied by however many decades each has after the gap simulation removed some). The density statistics should show a mean in the hundreds and a max in the thousands (people per square kilometer).

The key confirmation is that the dropped count is greater than zero. That proves your divide-by-zero protection caught the bad data before it could corrupt downstream calculations.

Seeing 0 dropped records?

This means the zero-area tracts were not included in your CSV files. Re-run python generate_data.py to regenerate the data, then run python pipeline.py again.

Why is my density calculation not catching any zero-area tracts?

✔️ Awesome, I've got everything!

Great. Double check you have saved pipeline.py and that your output shows all three sections (merge count, dropped count, density statistics).

ⓧ I'd like to double check the full code

Your pipeline.py should have these sections in order. Scroll up and confirm each is present:

  • Five import lines at the top (numpy, pandas, LinearRegression, r2_score/mean_absolute_error, matplotlib.pyplot).
  • A section comment header for Step 2.
  • The load_and_merge_data() function that reads both CSVs and merges them.
  • The calculate_density(df) function that divides, replaces inf, drops NaN, and prints stats.
  • The if __name__ == "__main__" block calling both functions in sequence.

You now have clean population density values for every valid tract across every available decade. Next up, you will detect exactly which tracts are missing which decades so you know what needs to be imputed.

Detect Temporal Gaps

You now have population density values for most tract-decade combinations. But some decades are missing for certain tracts, and before you can predict those missing values, you need to know exactly what's absent.

Running the pipeline without this step would mean blindly guessing which data to fill. In this step, you'll build an audit system that detects every temporal gap, visualizes the pattern, and filters down to tracts that have enough data points for meaningful regression.

In this step, get ready to:
  • Build gap detection logic that finds missing decades per tract.
  • Visualize the gaps as a bar chart showing missing-data percentages.
  • Filter to tracts with enough data points for imputation.
Build the gap detection logic

Each tract should have data for all six decades (1970 through 2020). The gap detection function groups tracts by ID, checks which decades are present, and reports the missing ones.

  • In pipeline.py, scroll below the calculate_density function and paste the following code:
# =============================================================
# STEP 3: Detect Temporal Gaps
# =============================================================

def detect_gaps(df):
    """Identify which decades are missing for each tract."""
    all_decades = [1970, 1980, 1990, 2000, 2010, 2020]
    # Group by tract_id and find which decades each tract has
    tract_decades = df.groupby("tract_id")["decade"].apply(set)
    # Find missing decades per tract
    gap_records = []
    for tract_id, present_decades in tract_decades.items():
        missing = set(all_decades) - present_decades
        for decade in missing:
            gap_records.append({"tract_id": tract_id, "missing_decade": decade})
    gaps_df = pd.DataFrame(gap_records)
    # Summary: count missing per decade
    if len(gaps_df) > 0:
        gap_summary = gaps_df.groupby("missing_decade").size().reset_index(name="missing_count")
        total_tracts = df["tract_id"].nunique()
        gap_summary["missing_pct"] = (gap_summary["missing_count"] / total_tracts * 100).round(2)
    else:
        gap_summary = pd.DataFrame(columns=["missing_decade", "missing_count", "missing_pct"])
    print(f"\n--- TEMPORAL GAP AUDIT ---")
    print(f"Total unique tracts: {df['tract_id'].nunique()}")
    print(f"Total missing tract-decade combinations: {len(gaps_df)}")
    print(f"\nMissing data by decade:")
    print(gap_summary.to_string(index=False))
    return gaps_df, gap_summary

What does this code do?

The function defines the six expected decades, then groups all records by tract_id to see which decades each tract actually has.

  • tract_decades holds a set of decades present for each tract. Subtracting from the full set reveals the gaps.
  • gap_summary aggregates missing counts by decade so you can see which era lost the most data.
  • missing_pct converts counts into percentages of total tracts for easy comparison.

Now you need to call this function from the main pipeline. Find the if __name__ == "__main__": block at the bottom of pipeline.py.

  • Find this line:
    density_df = calculate_density(merged_df)
  • Add this line directly below it:
    gaps_df, gap_summary = detect_gaps(density_df)
  • Save pipeline.py.
  • Run the pipeline to see the audit output by running this command:
python pipeline.py

You should see the density statistics from Step 2, followed by a new section titled --- TEMPORAL GAP AUDIT --- showing the total missing tract-decade combinations and a table with missing counts per decade.

Seeing a NameError for detect_gaps?

  • Make sure you pasted the function ABOVE the if __name__ block, not inside it.
  • Check that the function name is exactly detect_gaps with an underscore.

Help me debug my detect_gaps function

Visualize the gaps with a bar chart

A table of numbers tells you the facts, but a chart makes the pattern jump out. You'll use matplotlib to create a bar chart that shows exactly how much data each decade is missing.

  • In pipeline.py, paste this function directly below detect_gaps:
def visualize_gaps(gap_summary):
    """Create a bar chart showing missing data percentage per decade."""
    plt.figure(figsize=(10, 6))
    plt.bar(
        gap_summary["missing_decade"].astype(str),
        gap_summary["missing_pct"],
        color="coral",
        edgecolor="black"
    )
    plt.xlabel("Decade")
    plt.ylabel("Missing Tracts (%)")
    plt.title("Temporal Gap Audit: Missing Data by Decade")
    plt.ylim(0, max(gap_summary["missing_pct"]) * 1.3)

    for i, row in gap_summary.iterrows():
        plt.text(
            i, row["missing_pct"] + 0.5,
            f"{row['missing_pct']}%",
            ha="center", fontsize=11
        )

    plt.tight_layout()
    plt.savefig("missing_data_audit.png", dpi=150)
    plt.close()
    print("\nSaved: missing_data_audit.png")

What does this code do?

  • plt.bar() draws a bar for each decade using the missing percentage as bar height.
  • The for loop adds percentage labels above each bar so the exact values are readable.
  • plt.close() prevents the plot window from blocking the script when running in a terminal.
  • In the if __name__ == "__main__": block, find this line:
    gaps_df, gap_summary = detect_gaps(density_df)
  • Add these lines directly below it:
    if len(gap_summary) > 0:
        visualize_gaps(gap_summary)
  • Save pipeline.py and run the pipeline again:
python pipeline.py

You should see Saved: missing_data_audit.png printed in the terminal after the gap audit table.

  • Open missing_data_audit.png from the file explorer in Cursor. You should see a coral-colored bar chart with 1970 showing roughly 20%, 1980 showing roughly 12%, and 1990 showing roughly 6%.

Chart not saving?

  • Make sure import matplotlib.pyplot as plt is at the top of your file. It was added in Step 2.
  • Check that the if len(gap_summary) > 0: condition wraps the visualize_gaps call. Without gaps, the chart has nothing to plot.

Help me debug why my chart isn't saving

Identify tracts eligible for imputation

Not every tract with a gap can be imputed. Linear Regression needs at least 3 data points to fit a meaningful line. Tracts with fewer than 3 recorded decades don't have enough information to predict their missing values reliably.

  • In pipeline.py, paste this function below visualize_gaps:
def filter_eligible_tracts(df, min_decades=3):
    """Filter to tracts with enough data points for regression."""
    decade_counts = df.groupby("tract_id")["decade"].count()
    eligible = decade_counts[decade_counts >= min_decades].index.tolist()
    excluded = decade_counts[decade_counts < min_decades].index.tolist()

    print(f"\nTracts eligible for imputation (>= {min_decades} decades): {len(eligible)}")
    print(f"Tracts excluded (< {min_decades} decades): {len(excluded)}")

    return eligible

What does this code do?

  • decade_counts counts how many decades each tract has recorded data for.
  • Tracts with 3 or more decades go into eligible. Those with fewer go into excluded.
  • The returned list of eligible tract IDs feeds directly into the imputation step.
  • In the if __name__ == "__main__": block, find this line:
    if len(gap_summary) > 0:
        visualize_gaps(gap_summary)
  • Add this line directly below it:
    eligible_tracts = filter_eligible_tracts(density_df)
  • Save pipeline.py.

Before you run this, think about how many tracts you expect to be excluded. With gaps only in 1970, 1980, and 1990, most tracts still have at least 3 decades of data. Do you think many tracts will fail the 3-decade minimum?

  • Run the full pipeline to see the complete audit output:
python pipeline.py

At the bottom of the output, you should see the eligible vs. excluded tract counts. The vast majority of tracts qualify because gaps only affect 1970-1990, leaving most tracts with 4-6 decades of data.

Seeing 0 eligible tracts?

  • Make sure you're passing density_df to filter_eligible_tracts, not gaps_df. The function needs the full dataset to count decades per tract.
  • Confirm your tract_population.csv was generated correctly by checking it has 6 different decade values.

Help me figure out why filter_eligible_tracts returns 0

✔️ Awesome, I've got everything!

Double check you've saved pipeline.py and that missing_data_audit.png exists in your project folder.

ⓧ I'd like to double check the full code

Your pipeline.py should now have these sections in order. Scroll through your file and confirm each is present:

  • Imports at the top: numpy, pandas, LinearRegression, r2_score, mean_absolute_error, matplotlib.pyplot.
  • STEP 2 section: load_and_merge_data() and calculate_density() functions.
  • STEP 3 section: detect_gaps(), visualize_gaps(), and filter_eligible_tracts() functions.
  • The if __name__ == "__main__": block calling all five functions: load_and_merge_data, calculate_density, detect_gaps, visualize_gaps (inside the if len check), and filter_eligible_tracts.

You now have a complete audit of your dataset's temporal gaps. You know exactly which decades are missing, how widespread the gaps are, and which tracts have enough data to impute. Next up, you'll train a Linear Regression model for each eligible tract to predict those missing density values.

Impute Missing Decades with Linear Regression

Your gap detection system from the last step gave you a clear picture: thousands of tracts are missing data from 1970, 1980, and 1990.

A naive approach like filling those gaps with the dataset's average density would ignore each tract's unique growth trajectory. Linear Regression fits a line through each tract's existing data points and predicts where the missing decades fall on that trajectory.

In this step, get ready to:
  • Build a per-tract regression pipeline that trains a model on each tract's known decades and predicts the missing ones.
  • Calculate R-squared and MAE validation metrics with plain-English explanations of what each number means.
  • Combine original and imputed data into one complete DataFrame with an is_imputed flag.
Build the per-tract regression pipeline

Each eligible tract has at least 3 known decades of density data. For each one, you will fit a LinearRegression model on the known decades (X) and density values (y), then predict the missing decades.

Why Linear Regression instead of mean filling?

A tract that grew from 500 people/sq km in 1980 to 2000 people/sq km in 2020 has a clear upward trajectory. Filling its missing 1970 value with the dataset average (say, 1200) would be way off.

Linear Regression draws a best-fit line through the tract's own known points. The prediction for 1970 naturally falls on that line, respecting the tract's individual growth pattern.

  • In pipeline.py, scroll below your filter_eligible_tracts function.
  • Add a section comment and the first part of the imputation function by pasting this code:
# =============================================================
# STEP 4: Temporal ML Imputation with Linear Regression
# =============================================================

def impute_missing_decades(df, gaps_df, eligible_tracts):
    """Fit per-tract Linear Regression and predict missing densities."""
    all_decades = [1970, 1980, 1990, 2000, 2010, 2020]
    imputed_records = []

    for tract_id in eligible_tracts:
        # Get this tract's known data
        tract_data = df[df["tract_id"] == tract_id].sort_values("decade")

        if len(tract_data) < 3:
            continue

        # Get missing decades for this tract
        tract_gaps = gaps_df[gaps_df["tract_id"] == tract_id]["missing_decade"].values
        if len(tract_gaps) == 0:
            continue
  • Still inside the for loop (at the same indentation level as the if len(tract_gaps) == 0 check), add the model fitting and prediction logic:
        # Prepare X (decades) and y (density) for regression
        X_known = tract_data["decade"].values.reshape(-1, 1)
        y_known = tract_data["population_density"].values

        # Fit Linear Regression
        model = LinearRegression()
        model.fit(X_known, y_known)

        # Predict missing decades
        X_missing = np.array(tract_gaps).reshape(-1, 1)
        y_predicted = model.predict(X_missing)

        # Ensure no negative density predictions
        y_predicted = np.maximum(y_predicted, 0.0)

        # Get tract metadata (lat, lon) from existing records
        lat = tract_data["latitude"].iloc[0]
        lon = tract_data["longitude"].iloc[0]

        for decade, density in zip(tract_gaps, y_predicted):
            imputed_records.append({
                "tract_id": tract_id,
                "decade": int(decade),
                "population_density": round(density, 4),
                "latitude": lat,
                "longitude": lon,
                "is_imputed": True
            })

    imputed_df = pd.DataFrame(imputed_records)
    print(f"\nImputed {len(imputed_df)} values across {imputed_df['tract_id'].nunique()} tracts")

    return imputed_df

What does this function do?

  • For each eligible tract, it extracts the known decades and density values.
  • It reshapes decades into a 2D array with .reshape(-1, 1) because scikit-learn expects features as columns, not a flat list.
  • A fresh LinearRegression() model is trained per tract, fitting a line through that tract's specific data points.
  • Predictions for missing decades are clamped to zero minimum (density cannot be negative).
  • Each imputed value is stored with an is_imputed: True flag so you can always distinguish real data from predictions.
  • Now scroll to the bottom of the if __name__ == "__main__": block.
  • Find this line (currently the last line in the block):
    eligible_tracts = filter_eligible_tracts(density_df)
  • Add this line directly below it:
    imputed_df = impute_missing_decades(density_df, gaps_df, eligible_tracts)
  • Save pipeline.py.
  • Run the pipeline in your terminal:
python pipeline.py

After the gap audit output you saw before, you should now see a new line like:

Imputed X values across Y tracts (where X and Y are numbers from your dataset).

Seeing a KeyError or empty output?

  • If you see KeyError: 'missing_decade', make sure you're passing gaps_df (not gap_summary) to the function. The gaps DataFrame has the missing_decade column.
  • If the imputed count is 0, check that generate_data.py actually removed records. Re-run it and confirm you see the "Records removed to simulate gaps" message.
  • Make sure the second code block is indented inside the for tract_id in eligible_tracts: loop, not at the function level.

Help me debug my impute_missing_decades function

Calculate and explain validation metrics

You have imputed values, but how do you know they are accurate? Validation works by holding out one known decade per tract, predicting it as if it were missing, then comparing the prediction against the real value.

  • In pipeline.py, add the first part of the validation function directly below impute_missing_decades:
def validate_imputation(df, eligible_tracts):
    """
    Validate by holding out one known decade per tract,
    predicting it, and comparing with r2_score and MAE.
    """
    all_true = []
    all_pred = []

    for tract_id in eligible_tracts:
        tract_data = df[df["tract_id"] == tract_id].sort_values("decade")

        if len(tract_data) < 4:
            continue

        # Hold out the last known decade
        train = tract_data.iloc[:-1]
        test = tract_data.iloc[-1:]

        X_train = train["decade"].values.reshape(-1, 1)
        y_train = train["population_density"].values
        X_test = test["decade"].values.reshape(-1, 1)
        y_test = test["population_density"].values[0]

        model = LinearRegression()
        model.fit(X_train, y_train)
        y_pred = model.predict(X_test)[0]

        all_true.append(y_test)
        all_pred.append(y_pred)
  • Directly below the loop (back at the function's base indentation level, aligned with all_true = []), add the metrics calculation and explanation:
    r2 = r2_score(all_true, all_pred)
    mae = mean_absolute_error(all_true, all_pred)

    print(f"\n--- IMPUTATION VALIDATION METRICS ---")
    print(f"Tracts validated: {len(all_true)}")
    print(f"R-squared: {r2:.4f}")
    print(f"Mean Absolute Error: {mae:.2f} people per sq km")
    print(f"\nWhat these mean:")
    print(f"  R-squared ({r2:.4f}): This measures how well the model's predictions")
    print(f"  match actual values. 1.0 = perfect prediction. 0.0 = no better than")
    print(f"  guessing the average. Your score means the model explains {r2*100:.1f}%")
    print(f"  of the variance in population density over time.")
    print(f"\n  MAE ({mae:.2f}): On average, the model's prediction is off by")
    print(f"  {mae:.2f} people per square kilometer. Lower is better.")

    return r2, mae

What does this validation do?

  • For each tract with 4+ known decades, it hides the last decade and trains on the rest.
  • It predicts the hidden decade and compares the prediction to the real value.
  • R-squared measures what fraction of the variance in density is explained by the linear time trend. Closer to 1.0 is better.
  • MAE (Mean Absolute Error) tells you the average prediction error in real units (people per sq km). Lower is better.
  • In the if __name__ == "__main__": block, find the line you just added:
    imputed_df = impute_missing_decades(density_df, gaps_df, eligible_tracts)
  • Add this line directly below it:
    r2, mae = validate_imputation(density_df, eligible_tracts)
  • Save pipeline.py and run it again:
python pipeline.py

After the imputation count, you should now see a new block of output starting with --- IMPUTATION VALIDATION METRICS --- showing your R-squared score and MAE value with explanations.

Seeing very low R-squared or a ValueError?

  • A ValueError: Found input variables with inconsistent numbers of samples means all_true and all_pred are empty. Make sure the if len(tract_data) < 4 threshold is correct (4, not 3).
  • Check that the metrics code (r2 = r2_score(...)) is outside the for loop, at the function's base indentation level.

Help me fix my validate_imputation function

Combine original and imputed data

Right now, original data and imputed predictions live in separate DataFrames. You need to merge them into one complete dataset where every tract has all 6 decades.

  • In pipeline.py, add the combine_data function below validate_imputation:
def combine_data(df, imputed_df):
    """Combine original data with imputed predictions."""
    # Add is_imputed flag to original data
    df["is_imputed"] = False

    # Select matching columns from original
    original_cols = ["tract_id", "decade", "population_density", "latitude", "longitude", "is_imputed"]
    original_subset = df[original_cols].copy()

    # Concatenate
    complete_df = pd.concat([original_subset, imputed_df], ignore_index=True)
    complete_df = complete_df.sort_values(["tract_id", "decade"]).reset_index(drop=True)

    # Verify completeness
    tract_decade_counts = complete_df.groupby("tract_id")["decade"].count()
    complete_tracts = (tract_decade_counts == 6).sum()
    total_tracts = complete_df["tract_id"].nunique()

    print(f"\n--- DATA COMPLETENESS CHECK ---")
    print(f"Total records: {len(complete_df)}")
    print(f"Tracts with all 6 decades: {complete_tracts} / {total_tracts}")

    return complete_df

What does this function do?

  • It flags all original records with is_imputed = False so you can always tell real data from predictions.
  • It concatenates the original and imputed DataFrames, then sorts by tract and decade.
  • The completeness check at the end confirms how many tracts now have all 6 decades filled in.
  • In the if __name__ == "__main__": block, find the line:
    r2, mae = validate_imputation(density_df, eligible_tracts)
  • Add this line directly below it:
    complete_df = combine_data(density_df, imputed_df)
  • Save pipeline.py.

Before you run the pipeline, think about this: do you expect every single tract to end up with all 6 decades, or might some still have gaps?

  • Run the full pipeline:
python pipeline.py

At the very end of the output, you should see a --- DATA COMPLETENESS CHECK --- block showing the total record count and how many tracts have all 6 decades. Most tracts will show 6/6 because you only imputed tracts that had enough data to fit a regression.

Seeing a column mismatch error on concat?

  • If pandas raises a column-mismatch error during pd.concat, check that the original_cols list matches exactly the columns in your imputed_df (both should have: tract_id, decade, population_density, latitude, longitude, is_imputed).
  • Make sure df["is_imputed"] = False is added BEFORE selecting the column subset. The line order matters.

Help me debug my combine_data function

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

Your pipeline.py is now quite long. Scroll through your file and confirm these sections are present in order:

  • Imports at the top (numpy, pandas, sklearn, matplotlib).
  • Step 2 section: load_and_merge_data() and calculate_density().
  • Step 3 section: detect_gaps(), visualize_gaps(), and filter_eligible_tracts().
  • Step 4 section: impute_missing_decades(), validate_imputation(), and combine_data().
  • The if __name__ == "__main__": block ending with complete_df = combine_data(density_df, imputed_df).

Your imputation pipeline is working and validated. Next up, you will export this complete dataset to CSV and visualize sample tract trajectories to see the regression lines in action.

Export the Dataset and Visualize Results

Your ML pipeline is functionally complete. It loads raw census data, calculates density safely, detects temporal gaps, trains per-tract regression models, and validates accuracy with concrete metrics.

A CSV of numbers alone is not convincing in an interview. You need a clean export for downstream tools (like a 3D visualization) and visual proof that the imputations make sense when plotted against real data points.

In this step, get ready to:
  • Export the completed dataset to a CSV with clean columns and zero NaN values.
  • Visualize sample tract trajectories showing real vs imputed data points.
  • Print a formatted resume-ready metrics summary for interview prep.
Export the final dataset to CSV

The complete DataFrame has the right data, but its column order and contents need to be production-ready. The export function selects only the columns a downstream tool needs and confirms no NaN values slipped through.

  • In pipeline.py, scroll down below the combine_data() function.
  • Add the Step 5 section header and the export_dataset() function by copying this code below combine_data():
# =============================================================
# STEP 5: Export and Visualization
# =============================================================

def export_dataset(complete_df):
    """Export final dataset to CSV."""
    output_cols = ["tract_id", "latitude", "longitude", "decade", "population_density", "is_imputed"]
    export_df = complete_df[output_cols].copy()

    # Verify no NaN values
    nan_count = export_df.isna().sum().sum()
    print(f"\nNaN values in export: {nan_count}")

    export_df.to_csv("imputed_census_data.csv", index=False)
    print(f"Exported {len(export_df)} rows to imputed_census_data.csv")
    print(f"Columns: {list(export_df.columns)}")

    return export_df

What does this code do?

  • output_cols defines the exact column order a downstream visualization tool expects: tract_id, lat/lon coordinates, decade, density value, and the imputed flag.
  • The NaN check (isna().sum().sum()) is a final safety net. If any NaN values leaked through the pipeline, you would catch them here before exporting a broken file.
  • to_csv() with index=False writes the data without pandas' row numbers, keeping the file clean for other tools.

Now wire this function into your main pipeline so it runs automatically.

  • Scroll down to your if __name__ == "__main__": block at the bottom of the file.
  • Find this line (the last line in the block):
    complete_df = combine_data(density_df, imputed_df)
  • Add this line directly below it:
    export_dataset(complete_df)
  • Save pipeline.py.
  • Run the pipeline to test the export by running:
python pipeline.py

At the end of the output, you should see a line saying NaN values in export: 0 followed by the row count and column list.

  • Confirm the file was created by running:
ls imputed_census_data.csv

You should see imputed_census_data.csv listed in the output.

Don't see the CSV file?

  • Make sure your terminal is in your project folder (the same folder as pipeline.py).
  • Check that you added export_dataset(complete_df) inside the if __name__ block with proper indentation (4 spaces).
  • Why isn't my CSV being exported when I run pipeline.py?
Visualize sample tract trajectories

Numbers alone do not prove your imputations are reasonable. A plot showing real data points alongside imputed ones lets you visually confirm the regression lines are sensible.

  • In pipeline.py, add the first part of the visualize_trajectories() function directly below export_dataset():
def visualize_trajectories(complete_df, n_samples=3):
    """Plot sample tract trajectories showing real vs imputed points."""
    # Pick tracts that have imputed values
    imputed_tracts = complete_df[complete_df["is_imputed"] == True]["tract_id"].unique()

    if len(imputed_tracts) == 0:
        print("No imputed tracts to visualize.")
        return

    sample_tracts = np.random.choice(imputed_tracts, size=min(n_samples, len(imputed_tracts)), replace=False)

    fig, axes = plt.subplots(1, len(sample_tracts), figsize=(5 * len(sample_tracts), 4))
    if len(sample_tracts) == 1:
        axes = [axes]

What does this setup do?

  • The function filters to only tracts that have imputed values (those are the interesting ones to plot).
  • np.random.choice() picks 3 random tracts from that set so you get variety each time you run.
  • plt.subplots() creates a side-by-side figure with one subplot per tract. The single-axes check prevents an error when only one tract is sampled.
  • Now add the plotting loop and save logic directly below the code you just pasted (still inside the same function):
    for ax, tract_id in zip(axes, sample_tracts):
        tract_data = complete_df[complete_df["tract_id"] == tract_id].sort_values("decade")

        real = tract_data[tract_data["is_imputed"] == False]
        imputed = tract_data[tract_data["is_imputed"] == True]

        # Plot real points (filled)
        ax.scatter(real["decade"], real["population_density"],
                   color="blue", zorder=5, s=60, label="Real")

        # Plot imputed points (hollow)
        ax.scatter(imputed["decade"], imputed["population_density"],
                   facecolors="none", edgecolors="red", zorder=5, s=60,
                   linewidths=2, label="Imputed")

        # Draw regression line through all points
        ax.plot(tract_data["decade"], tract_data["population_density"],
                color="gray", linestyle="--", alpha=0.7)

        ax.set_xlabel("Decade")
        ax.set_ylabel("Density (people/sq km)")
        ax.set_title(tract_id, fontsize=9)
        ax.legend(fontsize=8)

    plt.suptitle("Sample Tract Trajectories: Real vs Imputed", fontsize=12)
    plt.tight_layout()
    plt.savefig("sample_trajectories.png", dpi=150)
    plt.close()
    print("Saved: sample_trajectories.png")

What does the plotting loop do?

  • For each tract, it splits data into real (original) and imputed (predicted) subsets.
  • Real values appear as filled blue circles. Imputed values appear as hollow red circles. This makes it immediately clear which points the model generated.
  • A dashed gray line connects all points (real and imputed together), showing the regression trajectory the model fit.
  • The figure is saved as sample_trajectories.png so you can include it in a portfolio or presentation.
  • Add the visualization call to your __main__ block. Find this line at the bottom:
    export_dataset(complete_df)
  • Add this line directly below it:
    visualize_trajectories(complete_df)
  • Save pipeline.py and run the pipeline again:
python pipeline.py

At the end of the output you should see Saved: sample_trajectories.png. Open the file in your file explorer to see 3 tract plots with blue filled circles (real data) and red hollow circles (imputed predictions) connected by dashed lines.

Not seeing the PNG file?

  • Check that both code blocks (the function setup and the plotting loop) are part of the same function. The for loop should be indented at the same level as fig, axes = plt.subplots(...).
  • Make sure visualize_trajectories(complete_df) is inside the if __name__ block with 4-space indentation.
  • Why isn't my sample_trajectories.png being generated?
Print a resume-ready metrics summary

The final function formats everything you built into a single block of text you can reference in interviews. It includes the scale of the project, accuracy metrics, interview talking points, and plain-English metric explanations.

  • In pipeline.py, add the first part of print_resume_summary() below visualize_trajectories():
def print_resume_summary(complete_df, r2, mae):
    """Print formatted resume-ready metrics."""
    total_tracts = complete_df["tract_id"].nunique()
    total_records = len(complete_df)
    imputed_count = complete_df["is_imputed"].sum()
    decades_spanned = complete_df["decade"].nunique()

    print(f"\n{'='*60}")
    print(f"  RESUME METRICS SUMMARY")
    print(f"{'='*60}")
    print(f"\n  Scale:")
    print(f"    - Processed {total_tracts:,} census tracts across {decades_spanned} decades (1970-2020)")
    print(f"    - Total dataset: {total_records:,} records")
    print(f"    - Imputed {imputed_count:,} missing values using per-tract Linear Regression")
    print(f"\n  Model Accuracy:")
    print(f"    - R-squared: {r2:.4f} (model explains {r2*100:.1f}% of density variance)")
    print(f"    - Mean Absolute Error: {mae:.2f} people/sq km")

What does this first section print?

  • The Scale section gives concrete numbers: how many tracts, how many records, how many values your model generated. These are the headline numbers for a resume bullet point.
  • The Model Accuracy section shows R-squared and MAE with formatted precision. The parenthetical after R-squared converts it to a percentage for quick comprehension.
  • Now add the interview explanation and metric definitions directly below (still inside the same function):
    print(f"\n  How to explain these in an interview:")
    print(f"    'I built a Python ML pipeline that processed {total_tracts:,} census tracts")
    print(f"     spanning 50 years. For tracts with missing decades, I trained per-tract")
    print(f"     Linear Regression models on available data points to predict the gaps.")
    print(f"     The model achieved an R-squared of {r2:.3f}, meaning it explained")
    print(f"     {r2*100:.1f}% of the variance in population density over time,")
    print(f"     with an average error of only {mae:.1f} people per square kilometer.'")
    print(f"\n  What each metric means:")
    print(f"    R-squared: Ranges 0 to 1. Measures what proportion of the variation")
    print(f"    in the target variable (density) is predictable from the input (decade).")
    print(f"    Higher = better. A score of {r2:.3f} means the linear time trend captures")
    print(f"    most of how density changes over decades for each tract.")
    print(f"\n    MAE: The average absolute difference between predicted and actual")
    print(f"    density values. Units match the target (people per sq km).")
    print(f"    It tells you: 'On average, my prediction is off by {mae:.1f} people/sq km.'")
    print(f"{'='*60}")

Why include interview talking points in the code?

When you run the pipeline, the summary prints a complete script for how to describe this project in a technical interview. The R-squared and MAE explanations use plain language, not textbook definitions. You can read them directly when someone asks "tell me about a project where you used ML."

  • Add the final call to your __main__ block. Find this line:
    visualize_trajectories(complete_df)
  • Add this line directly below it:
    print_resume_summary(complete_df, r2, mae)
  • Save pipeline.py.

Before you run the final pipeline, predict what you expect to see. Will the resume summary print scale numbers, accuracy metrics, and interview talking points all in one formatted block?

  • Run the complete pipeline end-to-end:
python pipeline.py

At the very end of the output, you should see a formatted block bordered by = characters with "RESUME METRICS SUMMARY" as the header. It shows your tract count, total records, imputed value count, R-squared, MAE, a full interview script, and plain-English metric explanations.

Seeing a NameError for r2 or mae?

  • Make sure your __main__ block has r2, mae = validate_imputation(density_df, eligible_tracts) before the print_resume_summary() call. The function needs both values as arguments.
  • Check that all three new calls (export_dataset, visualize_trajectories, print_resume_summary) are indented at the same level as the other lines inside the if __name__ block.
  • Why am I getting a NameError when running my resume summary function?

Your pipeline now produces three deliverables every time it runs: a clean CSV, a trajectory visualization, and a formatted interview-prep summary.

✔️ Awesome, I've got everything!

Confirm you have saved pipeline.py with all three new functions (export_dataset, visualize_trajectories, print_resume_summary) and the updated __main__ block.

ⓧ I'd like to double check the full code

Your pipeline.py is now too long to show in one block. Scroll through the file and confirm these sections are present in order:

  • Imports at the top (numpy, pandas, sklearn, matplotlib).
  • Step 2 section: load_and_merge_data() and calculate_density().
  • Step 3 section: detect_gaps(), visualize_gaps(), and filter_eligible_tracts().
  • Step 4 section: impute_missing_decades(), validate_imputation(), and combine_data().
  • Step 5 section: export_dataset(), visualize_trajectories(), and print_resume_summary().
  • The __main__ block ending with these three lines:
    export_dataset(complete_df)
    visualize_trajectories(complete_df)
    print_resume_summary(complete_df, r2, mae)

Secret mission

Holdout Validation Experiment

Your Step 4 metrics proved the model works, but they used a narrow test: hold out only the last decade per tract. Can you build a more rigorous experiment that masks 20% of ALL known data (stratified by decade), imputes the gaps, and compares predictions against ground truth? This gives you a second, independently validated set of resume metrics that proves your pipeline holds up under stress.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally on your machine. There are no cloud resources, no paid APIs, and no ongoing costs.

Decide whether to keep your project files for future reference, or delete them to free up disk space.

Resources you used:

  • Project folder on your Desktop containing generate_data.py, pipeline.py, and holdout_validation.py.
  • Generated data files: tract_population.csv, tract_land_area.csv, and imputed_census_data.csv.
  • Visualization outputs: missing_data_audit.png and sample_trajectories.png.
  • Python virtual environment (inside your project folder).

Keep everything running

No action needed. Choose this if you plan to extend the pipeline, adjust parameters, or use the exported CSV in a downstream project.

  • Your project folder stays on your Desktop with all scripts, data, and outputs intact.
  • The virtual environment stays inside the project folder, ready to reactivate whenever you return.
  • The imputed_census_data.csv file is ready to feed into visualization tools whenever you need it.

Pause - I'll come back to this later

Deactivate the virtual environment to free up your terminal. Your files stay on disk for when you return.

  • If your virtual environment is still active, deactivate it by running:
deactivate
  • When you come back, reactivate the virtual environment from inside your project folder by running source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows).

Delete - I don't want to use this again

Remove all project files, generated data, and the virtual environment entirely.

  • If your virtual environment is still active, deactivate it first by running:
deactivate
  • Navigate to your Desktop by running:
cd ~/Desktop
  • Delete the entire project folder (replace the folder name with whatever you named your project) by running:

macOS

rm -rf your-project-folder-name

Windows

rmdir /s /q your-project-folder-name
  • Confirm the folder is gone by running ls (macOS) or dir (Windows). Your project folder should no longer appear in the listing.

Nice Work!

Nice Work!

You just built a complete ML pipeline that processes 74,000+ census tracts across 50 years of data, detects temporal gaps, and fills them with per-tract Linear Regression predictions backed by concrete accuracy metrics.

You learned how to:

  • Build a temporal gap detection system that audits missing data across decades and visualizes exactly where the holes are in a longitudinal dataset.
  • Train per-tract Linear Regression models that predict missing population density values based on each tract's unique growth trajectory, with safe handling of divide-by-zero edge cases.
  • Validate ML predictions with R-squared and MAE metrics and export a clean CSV with an is_imputed flag ready for downstream 3D visualization.
  • Secret Mission: Proved model robustness with a holdout validation experiment that independently verified accuracy under data stress by masking 20% of known values and comparing predictions against ground truth.

Ready to quiz yourself?