Predict Missing Census Data

Use linear regression to impute missing U.S. county population density values.

Introduction

30 Second Summary

Real-world datasets are messy. Government agencies collect data on thousands of counties, but values go missing all the time. Data scientists need reliable ways to fill in those gaps without just guessing.

In this project, you will build a Python pipeline that takes real U.S. county-level census data, detects missing population density values, and uses linear regression to predict accurate replacements from related demographic features.

What You'll Build

You run the script on 3,100+ county records, watch it outperform a naive baseline, and see it produce scatter plots, bar charts, and a printed block of resume-ready statistics with your exact performance numbers.

By the end of this project, you'll have:

  • A working imputation pipeline that feeds in census data with missing values and returns accurate predictions trained on 30+ demographic features.
  • Quantifiable resume statistics (R² score, RMSE reduction percentage) proving your model outperforms naive mean imputation by a measurable margin you can cite in interviews.
  • Publication-quality visualizations comparing imputation methods, saved as image files you can include in a portfolio.
  • Secret Mission: Add KNN imputation as a third method and produce a full comparison table.

Are there any prerequisites?

You need Python installed on your machine and a code editor (any IDE works). Basic Python comfort is helpful, but no prior machine learning experience is required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Libraries and Data

To predict missing census data with machine learning, you need specialized Python libraries for data manipulation, model training, and visualization.

In this step, you'll install those libraries and download the real U.S. county-level census dataset you'll work with throughout the project.

In this step, get ready to:
  • Install pandas, scikit-learn, matplotlib, and numpy.
  • Verify the libraries import correctly.
  • Download the USDA ERS Atlas census data into a project folder.
Install and verify the data science libraries
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Install the required Python libraries by running this command:
pip install pandas scikit-learn matplotlib numpy

You'll see download progress bars as each package installs. This may take a minute.

Seeing 'command not found' for pip?

Try pip3 instead of pip. On some systems, Python 3's package manager uses the pip3 command.

If that also fails, you may need to add Python to your system PATH. Help me fix my pip installation

Before you run the next command, what do you think you'll see if all four libraries installed correctly?

  • Verify everything installed correctly by running this command:
python -c "import pandas; import sklearn; import matplotlib; print('All libraries ready')"

✔️ I see 'All libraries ready'

All four libraries are installed and importable. You're ready to move on.

ⓧ I see an ImportError or ModuleNotFoundError

The error message will name the specific library that failed. Reinstall just that library:

pip install <library-name>

Replace <library-name> with the one that failed (pandas, scikit-learn, matplotlib, or numpy). Then run the verify command again.

If you're using python3 to run scripts on your system, use python3 -c instead of python -c in the verify command.

ⓧ 'python' is not recognized

Your system may use python3 as the command name. Run this instead:

python3 -c "import pandas; import sklearn; import matplotlib; print('All libraries ready')"

If this prints All libraries ready, you're good to go. Use python3 instead of python for all commands in this project.

Create your project folder and download the data

You need a dedicated folder for your project files and the census dataset. You'll also download real county-level data from the USDA Economic Research Service.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir census-imputation
cd census-imputation
  • Confirm the folder was created by running:
pwd

You should see a path ending in census-imputation.

Now you'll download the census data from the USDA ERS Atlas of Rural and Small-Town America.

What is this dataset?

The USDA ERS Atlas of Rural and Small-Town America contains county-level demographic data for all 3,100+ U.S. counties. The People.csv file includes population density, age distributions, migration rates, education levels, and other indicators drawn from the U.S. Census Bureau and American Community Survey.

This is real federal data, not a toy dataset. Working with it gives you a genuine data engineering experience to reference in interviews.

  • Confirm People.csv is in your project folder by running:
ls

You should see People.csv in the output.

Don't see People.csv listed?

Make sure you copied People.csv from the extracted folder into the census-imputation folder on your Desktop. The ZIP extracts multiple CSV files (People, Jobs, Income, Veterans). You only need People.csv for this project.

On Windows, make sure your terminal is pointing at the right folder. Run cd ~/Desktop/census-imputation and try ls (or dir on Command Prompt) again. Help me find my People.csv file

  • Open your census-imputation folder in your IDE so you're ready to create Python files in the next step.

Your environment is ready. Next up, you'll load this census data into pandas and explore what 3,100+ counties of demographic data actually looks like.

Load and Explore the Census Data

Your data science libraries are installed and ready to go. Now you need to load the actual census data and understand what you're working with.

Before you can build any model, you need to know your dataset's structure. What columns are available? Which one is the target you want to predict? How is the data distributed? The answers to these questions will shape every decision in the rest of the pipeline.

In this step, get ready to:
  • Load real county-level census data into a pandas DataFrame.
  • Identify the population density column and select usable numeric features.
  • Examine the data distribution to understand the shape of your target variable.
Create the script and load the data

You will create a single Python file that grows throughout this project. Each step adds new capabilities to the same script.

  • Create a new file called census_imputation.py in the same folder where your People.csv file lives.
  • Add the following code to load the CSV data and print the initial structure:
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
import matplotlib.pyplot as plt

# === CONFIGURATION ===
DATA_FILE = "People.csv"

# === STEP 2: Load and Explore ===
df = pd.read_csv(DATA_FILE, encoding="latin-1")

print("=" * 50)
print("CENSUS DATA EXPLORATION")
print("=" * 50)
print(f"\nDataset shape: {df.shape}")
print(f"\nAll columns:\n{df.columns.tolist()}")

What does this code do?

The imports bring in all the libraries you will need across the full project. You import them now so the file is ready as you add more code in later steps.

The encoding="latin-1" parameter tells pandas how to decode special characters in the USDA data file. Without it, you would likely get a Unicode decode error.

Printing the shape and column list gives you a quick inventory of what the dataset contains.

  • Save census_imputation.py.
  • Run the script to confirm the data loads by running:
python census_imputation.py

You should see the dataset shape (approximately 3,100+ rows and dozens of columns) followed by a full list of column names.

Seeing a FileNotFoundError?

  • Make sure People.csv is in the same folder as your census_imputation.py script. If you extracted the ZIP into a subfolder, either move the CSV file up or update the DATA_FILE path to include the subfolder name.
  • If you see a UnicodeDecodeError instead, double-check that encoding="latin-1" is present in your pd.read_csv() call.
  • Still stuck? Help me debug why my census_imputation.py can't find or load People.csv
Identify the target column and select features

The USDA Atlas CSV uses short column names that vary between releases. Instead of hardcoding a column name, you will search for it programmatically.

  • Add the following code below your existing print statements in census_imputation.py to identify the density column:
density_candidates = [c for c in df.columns if "density" in c.lower() or "popdensity" in c.lower()]
print(f"\nDensity column candidates: {density_candidates}")

target_col = density_candidates[0] if density_candidates else None
if target_col is None:
    print("ERROR: Could not find a density column. Print df.columns and set target_col manually.")
    exit()
print(f"Using target column: '{target_col}'")

What does this code do?

The list comprehension scans every column name for the words "density" or "popdensity" (case-insensitive). This finds the population density column regardless of the exact naming convention used in your download.

If no match is found, the script exits with a helpful message telling you to check the column list manually. This defensive pattern prevents confusing errors later in the pipeline.

  • Now add the feature selection and statistics code directly below in the same file:
id_keywords = ["fips", "state", "county", "name", "code"]
id_cols = [c for c in df.columns if any(kw in c.lower() for kw in id_keywords)]

numeric_cols = df.select_dtypes(include=[np.number]).columns.tolist()
feature_cols = [c for c in numeric_cols if c != target_col and c not in id_cols]
print(f"\nFeature columns selected ({len(feature_cols)}):")
print(feature_cols[:10])
if len(feature_cols) > 10:
    print(f"  ... and {len(feature_cols) - 10} more")

df_complete = df[[target_col] + feature_cols].dropna()
print(f"\nComplete records after dropping NaN: {len(df_complete)}")
print(f"\nPopulation Density Statistics:")
print(df_complete[target_col].describe())

What does this code do?

The id_keywords list filters out identifier columns (FIPS codes, state names, county names) that would leak geographic identity into the model rather than teaching it demographic patterns.

The feature_cols list keeps only numeric columns that are genuine predictors. Finally, df_complete drops any row with missing values in either the target or features, giving you a clean baseline to work from.

  • Save census_imputation.py.

Before you run this, what do you think the population density statistics will look like? Do you expect the mean and median to be close together, or far apart?

  • Run the full script again:
python census_imputation.py

You should now see the complete output: dataset shape, density column identified, feature columns listed, and the descriptive statistics for population density.

Look at the statistics closely. The mean is much higher than the median (the 50% value). This tells you the data is heavily right-skewed.

Why does the skew matter?

Most U.S. counties are rural with fewer than 100 people per square mile. But a handful of urban counties (like Manhattan or San Francisco) exceed 10,000 per square mile.

This extreme skew means a linear regression model trained on raw values will be pulled toward predicting mid-range values, badly missing both the low-density majority and the high-density outliers. You will address this in a later step with log transformation.

Seeing an ERROR about the density column?

  • If the script prints "ERROR: Could not find a density column", scroll up to the full column list printed earlier in the output. Look for any column containing "density" or "pop" and update the search terms in the density_candidates line to match.
  • If you see zero complete records, some columns may have widespread missing values. Try reducing the number of feature columns or check that your CSV file downloaded completely.
  • Need more help? Help me debug why my census script cannot find the density column or is dropping too many rows

✔️ Awesome, I've got everything!

Great. Double check you have saved census_imputation.py and that running it prints the full exploration output without errors.

ⓧ I'd like to double check the full code

Your census_imputation.py should contain these sections in order:

  • Imports (pandas, numpy, sklearn, matplotlib) and the DATA_FILE configuration.
  • The pd.read_csv() call with encoding="latin-1" and the initial shape/column prints.
  • The density_candidates search and target_col assignment with the error guard.
  • The id_keywords filter, feature_cols selection, df_complete creation, and descriptive statistics print.

Scroll up and confirm each chunk is present in your file, in this order.

You now have a clear picture of your data: thousands of counties, dozens of numeric features, and a heavily skewed target variable. Next up, you will simulate missing values and see how well different methods can fill them in.

Simulate Missing Data and Baseline Imputation

Your script now loads thousands of county records, identifies the density column, and prints descriptive statistics. The next challenge is to build a controlled experiment that measures how well different methods can recover missing values.

In real datasets, values go missing all the time. Sensors fail, respondents skip questions, records get corrupted. The trick to evaluating an imputation method is to remove values you already know, predict them with your method, then measure how close your predictions land to the truth.

In this step, get ready to:
  • Simulate missing data by removing 20% of known population density values.
  • Establish a baseline by filling missing values with the mean.
  • Train a linear regression model to predict missing values and compare it against the baseline.
Simulate missing data with a random seed

To test any imputation method fairly, you need ground truth to compare against. You will randomly select 20% of rows, save their real density values, then set those cells to NaN.

  • In census_imputation.py, scroll to the bottom of the file (after the density statistics print statement).
  • Add the following code to simulate missing data:
# === STEP 3: Simulate Missing Data ===
print("\n" + "=" * 50)
print("SIMULATING MISSING DATA")
print("=" * 50)

np.random.seed(42)
n_total = len(df_complete)
n_missing = int(0.2 * n_total)
missing_idx = np.random.choice(df_complete.index, size=n_missing, replace=False)

# Save ground truth before "removing" values
true_values = df_complete.loc[missing_idx, target_col].copy()

# Create the incomplete dataset
df_incomplete = df_complete.copy()
df_incomplete.loc[missing_idx, target_col] = np.nan

print(f"Total records: {n_total}")
print(f"Records with simulated missing density: {n_missing} ({n_missing/n_total*100:.0f}%)")

What does this code do?

np.random.seed(42) locks the random number generator so you get the same "missing" rows every time you run the script. This makes your results reproducible.

np.random.choice with replace=False picks 20% of row indices without duplicates. true_values saves the real density values before you blank them out, giving you the answer key to score predictions against.

  • Save census_imputation.py and run the script:
python census_imputation.py

At the bottom of the output, you should see the new section reporting the total record count and how many records now have simulated missing density (approximately 20% of the total).

Not seeing the missing data section?

Make sure you pasted the new code after the last print(df_complete[target_col].describe()) line. If Python throws a NameError for df_complete, confirm the new code sits below the line where df_complete is created.

Help me debug my missing data simulation code

Compute the mean imputation baseline

The simplest way to fill missing values is to replace every gap with the average. This gives you a baseline to beat.

  • Add this code below the missing data simulation section in census_imputation.py:
# --- Baseline: Mean Imputation ---
mean_density = df_incomplete[target_col].mean()
mean_predictions = np.full(n_missing, mean_density)
rmse_baseline = np.sqrt(mean_squared_error(true_values, mean_predictions))
r2_baseline = r2_score(true_values, mean_predictions)

print(f"\n--- BASELINE: Mean Imputation ---")
print(f"Predicted all missing as: {mean_density:.2f} people/sq mi")
print(f"RMSE: {rmse_baseline:.2f}")
print(f"R\u00b2 Score: {r2_baseline:.4f}")

What does this code do?

mean_predictions fills an array of size n_missing with the single mean value. Every missing county gets the same guess.

RMSE (Root Mean Squared Error) measures average prediction error in the same units as density (people per square mile). R² measures how much variance the method explains. A negative R² means the method is worse than predicting the mean of the test set.

  • Save census_imputation.py and run the script again:
python census_imputation.py

You should see the baseline RMSE printed along with the R² score. The R² will be very low because a single mean cannot capture the variation across rural and urban counties.

Seeing a negative R²?

A negative R² for mean imputation is normal here. It means predicting the overall mean is worse than predicting the test subset's own mean. The important thing is that a number prints without errors.

Why is my R² negative for mean imputation?

Train a linear regression model on the remaining data

Instead of guessing the same number for every county, linear regression can learn relationships between the other demographic features and population density. Counties with high education rates, large household sizes, or high migration might cluster at different density levels.

  • Add this code below the mean imputation section in census_imputation.py:
# --- Linear Regression on Raw Values ---
train_mask = df_incomplete[target_col].notna()
X_train = df_incomplete.loc[train_mask, feature_cols]
y_train = df_incomplete.loc[train_mask, target_col]
X_predict = df_incomplete.loc[missing_idx, feature_cols]

model_raw = LinearRegression()
model_raw.fit(X_train, y_train)
predictions_raw = model_raw.predict(X_predict)

rmse_raw = np.sqrt(mean_squared_error(true_values, predictions_raw))
r2_raw = r2_score(true_values, predictions_raw)

print(f"\n--- LINEAR REGRESSION (Raw Values) ---")
print(f"RMSE: {rmse_raw:.2f}")
print(f"R\u00b2 Score: {r2_raw:.4f}")
print(f"Improvement over baseline: {((rmse_baseline - rmse_raw) / rmse_baseline) * 100:.1f}% RMSE reduction")

What does this code do?

train_mask selects the 80% of rows that still have density values. The model trains on those rows, learning how the feature columns relate to density.

X_predict contains the features for the 20% of rows where you blanked out density. The model uses those features to predict what density should be, and you compare those predictions against true_values to compute RMSE and R².

  • Save census_imputation.py.

Before you run the script, think about this: do you expect the linear regression RMSE to be higher or lower than the mean imputation RMSE?

  • Run the script to check your prediction:
python census_imputation.py

You should see the linear regression R² is noticeably better than the baseline, and the RMSE is lower. The script prints the exact percentage improvement.

However, look closely at the R² value. It is better than mean imputation, but still limited. The model struggles with high-density counties because population density is heavily skewed. A few urban counties have 10,000+ people per square mile while most rural counties sit under 100.

Why is the model still limited?

Population density follows an extreme right-skewed distribution. Most counties cluster near zero, but outliers like Manhattan stretch the scale dramatically.

Linear regression tries to minimise squared errors across the full range. The extreme outliers dominate the error calculation, pulling the model away from accurately predicting the majority of counties. The next step addresses this with a transformation.

Getting a very low or negative R² for linear regression?

Check that feature_cols contains numeric columns only. If your CSV has text columns that slipped through, the model will fail silently or produce bad predictions.

Also verify that train_mask references df_incomplete (not df_complete). Training on the complete data would leak the answers you are trying to predict.

Help me debug my linear regression imputation

✔️ Awesome, I've got everything!

Great. Double check you have saved census_imputation.py and that the script runs without errors, printing both baseline and linear regression results.

ⓧ I'd like to double check the full code

Your census_imputation.py should have these sections in order. Scroll up and confirm each is present:

  • Imports and configuration (DATA_FILE = "People.csv").
  • Step 2: Load and Explore section (loads CSV, identifies density column, selects features, creates df_complete).
  • Step 3: Simulate Missing Data section (random seed, missing_idx, true_values, df_incomplete).
  • Baseline: Mean Imputation section (mean_density, rmse_baseline, r2_baseline).
  • Linear Regression on Raw Values section (model_raw, predictions_raw, rmse_raw, r2_raw).

You now have a working experiment with a clear baseline and a first model that beats it. Next up, you will apply a log transformation to address the skewed density distribution and dramatically improve your predictions.

Improve Predictions with Log Transformation

In the previous step, your raw linear regression improved on mean imputation but still struggled with high-density counties.

The problem is that population density follows an extreme right-skewed distribution. Most counties sit below 100 people per square mile, but a few urban counties exceed 10,000. A log transformation compresses those outliers and lets the model learn patterns across the full range of densities.

In this step, get ready to:
  • Apply a log transformation to normalize the skewed density data.
  • Train a new linear regression model on the transformed values.
  • Measure the improvement over both mean imputation and raw linear regression.
Understand why skewed data breaks raw linear regression

When you looked at your raw linear regression results, the R² was better than baseline but far from ideal. The issue comes from the shape of the data itself.

Why does skewness hurt linear regression?

Linear regression minimizes squared error. When a few urban counties have densities 100x larger than most rural counties, those extreme values dominate the error calculation. The model "chases" the outliers and distorts predictions for the majority of counties.

Log-transforming the target compresses the high end (10,000 becomes ~9.2) while spreading out the low end (10 becomes ~2.4). The model now sees a roughly normal distribution and can learn from all counties equally.

Add the log-transformed linear regression

You will apply np.log1p() to the density values before training, predict in log space, then convert back with np.expm1() to get predictions on the original scale.

  • In census_imputation.py, scroll to the very end of the file (after the raw linear regression print statements).
  • Add the log transformation block by pasting this code at the bottom:
# === STEP 4: Log Transformation ===
print("\n" + "=" * 50)
print("APPLYING LOG TRANSFORMATION")
print("=" * 50)

# Transform skewed density values into log space
y_train_log = np.log1p(y_train)

# Train a new model on the log-transformed target
model_log = LinearRegression()
model_log.fit(X_train, y_train_log)

# Predict in log space, then convert back to original scale
predictions_log_space = model_log.predict(X_predict)
predictions_improved = np.expm1(predictions_log_space)

# Evaluate on the original scale for fair comparison
rmse_improved = np.sqrt(mean_squared_error(true_values, predictions_improved))
r2_improved = r2_score(true_values, predictions_improved)

print(f"--- LINEAR REGRESSION (Log-Transformed Target) ---")
print(f"RMSE: {rmse_improved:.2f}")
print(f"R\u00b2 Score: {r2_improved:.4f}")
print(f"Improvement over baseline: {((rmse_baseline - rmse_improved) / rmse_baseline) * 100:.1f}% RMSE reduction")
print(f"Improvement over raw LR: {((rmse_raw - rmse_improved) / rmse_raw) * 100:.1f}% RMSE reduction")

What does this code do?

  • np.log1p(y_train) computes log(1 + x) for each density value. The "+1" ensures zero-density counties (if any exist) don't break the calculation, since log(0) is undefined.
  • The model trains on these compressed values, where the difference between 100 and 10,000 density is only about 4.6 log units instead of 9,900 raw units.
  • np.expm1() is the exact inverse of log1p(). It converts predictions back to the original people-per-square-mile scale so you can compare fairly against the other methods.
Run the script and measure the improvement

Before you run this, think about what you expect. Do you think the R² will improve slightly, or jump significantly compared to the raw linear regression?

  • Save census_imputation.py.
  • Run the script in your terminal:
python census_imputation.py

At the bottom of the output, you should see the "APPLYING LOG TRANSFORMATION" section with a significantly higher R² score and a large RMSE reduction percentage over both the baseline and raw linear regression.

Why is the jump so dramatic?

By normalizing the distribution, the model no longer wastes its "error budget" chasing a handful of extreme outliers. It can now fit the relationship between demographics and density that holds true for thousands of counties across the full range.

Seeing negative predictions or weird numbers?

  • Make sure you used np.log1p() (not np.log()) and np.expm1() (not np.exp()). These paired functions handle zeros safely.
  • Confirm that y_train_log uses y_train (not y_train_log applied twice). Double-transforming the values produces garbage.
  • Check that you added the code after the raw LR section, not inside it. The Step 4 block should start after the last print statement from Step 3.

Still stuck?

✔️ Awesome, I've got everything!

Great. Double check you have saved census_imputation.py and confirmed the improvement numbers printed correctly.

ⓧ I'd like to double check the full code

Your file should contain these sections in order. Scroll through census_imputation.py and confirm each one is present:

  • Imports (pandas, numpy, sklearn, matplotlib).
  • Configuration: DATA_FILE = "People.csv".
  • Step 2: Load and Explore (CSV read, density column discovery, feature selection, descriptive stats).
  • Step 3: Simulate Missing Data (random seed, 20% removal, mean imputation, raw linear regression).
  • Step 4: Log Transformation (np.log1p on training target, new model fit, np.expm1 on predictions, metrics print).

Your log-transformed model now captures patterns that raw regression missed entirely. Next up, you will turn these numbers into resume-ready statistics and portfolio visualizations.

Generate Resume Statistics and Visualizations

Your log-transformed model achieved a meaningful improvement over both the baseline and raw linear regression. Right now, those numbers only exist as raw terminal output buried in your script's print statements.

Quantifiable results are what make a project stand out on a resume. In this step, you'll format your exact performance numbers into copy-paste-ready resume bullets and create publication-quality plots you can include in a portfolio.

In this step, get ready to:
  • Print a formatted block of resume statistics with your exact numbers.
  • Create a side-by-side scatter plot comparing raw vs. log-transformed predictions.
  • Generate a bar chart comparing RMSE across all three imputation methods.
Print your resume statistics

A hiring manager scanning your resume wants specific numbers: how much data you processed, what technique you applied, and how much you improved the result. Generic claims like "improved accuracy" don't land. Exact percentages do.

  • In census_imputation.py, scroll to the bottom of the file (after the Step 4 print statements).
  • Add the resume statistics block by copying the following code at the end of the file:
# === STEP 5: Resume Statistics and Visualizations ===
print("\n" + "=" * 50)
print("         RESUME-READY STATISTICS")
print("=" * 50)

improvement_over_baseline = ((rmse_baseline - rmse_improved) / rmse_baseline) * 100

print(f"\n  Dataset Scale:")
print(f"    - {n_total:,} U.S. county records processed")
print(f"    - {len(feature_cols)} demographic features engineered")
print(f"    - {n_missing} missing values imputed ({n_missing/n_total*100:.0f}% of dataset)")
print(f"\n  Model Performance:")
print(f"    - R\u00b2 Score: {r2_improved:.4f}")
print(f"    - RMSE: {rmse_improved:.2f} (vs baseline {rmse_baseline:.2f})")
print(f"    - {improvement_over_baseline:.0f}% RMSE improvement over mean imputation")
print(f"\n  Suggested Resume Bullets:")
print(f"    * Built a linear regression pipeline processing {n_total:,}+ U.S. county")
print(f"      census records to predict missing population density values,")
print(f"      achieving R\u00b2 = {r2_improved:.2f}")
print(f"    * Reduced imputation error by {improvement_over_baseline:.0f}% over baseline")
print(f"      through feature engineering across {len(feature_cols)} demographic indicators")
print(f"    * Engineered log-transformation of skewed census data, improving")
print(f"      prediction accuracy for {n_missing}+ missing data points")

print("\n" + "=" * 50)

What does this code do?

  • improvement_over_baseline calculates the percentage RMSE reduction from your best model vs. mean imputation. This is the single most impactful number for your resume.
  • The "Dataset Scale" section quantifies how much data you processed, proving you can handle real federal datasets.
  • The "Suggested Resume Bullets" use your exact computed values. You can copy these directly into a resume or LinkedIn post.
  • Save census_imputation.py.
  • Run the script to see your resume statistics by running:
python census_imputation.py

At the end of the output, you should see a formatted "RESUME-READY STATISTICS" block with your dataset scale, model performance numbers, and three suggested resume bullets filled in with your exact values.

Not seeing the resume statistics block?

  • Make sure you pasted the code AFTER the Step 4 print statements, not inside them.
  • Check that improvement_over_baseline uses rmse_improved (not rmse_raw). This computes improvement from your best model.
  • If you see a NameError, confirm all Step 4 code ran successfully before this block.

Why aren't my resume statistics printing at the end of the script?

Create the actual vs. predicted scatter plot

Numbers tell the story, but a scatter plot shows it. A perfect model would produce points exactly on the diagonal line. The further your points scatter from that line, the worse the predictions.

  • Below the resume statistics block you just added, add the scatter plot code:
# --- Visualization 1: Actual vs Predicted Comparison ---
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

axes[0].scatter(true_values, predictions_raw, alpha=0.4, s=20, color="steelblue")
max_val = max(true_values.max(), max(predictions_raw.max(), predictions_improved.max()))
axes[0].plot([0, max_val], [0, max_val], "r--", linewidth=1.5, label="Perfect Prediction")
axes[0].set_xlabel("Actual Population Density (per sq mi)")
axes[0].set_ylabel("Predicted Population Density")
axes[0].set_title(f"Before: Raw Linear Regression\nR\u00b2 = {r2_raw:.3f}")
axes[0].legend()
axes[0].set_xlim(0, max_val * 1.05)
axes[0].set_ylim(0, max_val * 1.05)

axes[1].scatter(true_values, predictions_improved, alpha=0.4, s=20, color="seagreen")
axes[1].plot([0, max_val], [0, max_val], "r--", linewidth=1.5, label="Perfect Prediction")
axes[1].set_xlabel("Actual Population Density (per sq mi)")
axes[1].set_ylabel("Predicted Population Density")
axes[1].set_title(f"After: Log-Transformed Linear Regression\nR\u00b2 = {r2_improved:.3f}")
axes[1].legend()
axes[1].set_xlim(0, max_val * 1.05)
axes[1].set_ylim(0, max_val * 1.05)

plt.tight_layout()
plt.savefig("improvement_comparison.png", dpi=150)
plt.show()
print("Saved: improvement_comparison.png")

What does this code do?

  • plt.subplots(1, 2) creates a figure with two panels side by side. The left panel shows raw linear regression, the right shows log-transformed.
  • The red dashed diagonal is the "perfect prediction" line. Points on this line mean the model predicted exactly the true value.
  • plt.savefig("improvement_comparison.png", dpi=150) saves a high-resolution image to your project folder. This is the file you can add to a portfolio.
  • Save census_imputation.py and run the script again:
python census_imputation.py

A plot window appears showing two scatter plots side by side. Close the window to let the script continue. You should see "Saved: improvement_comparison.png" printed in the terminal.

What should I see in the scatter plot?

The left panel (raw LR) shows points scattered widely from the diagonal, especially for high-density counties. The right panel (log-transformed LR) shows points hugging the diagonal much more closely. That visual difference IS your improvement.

Plot window not appearing?

  • Some environments (remote servers, WSL) don't have a display backend. If you see a backend error, the file improvement_comparison.png is still saved. Open it from your file browser to see the plot.
  • If your IDE uses an inline plotting pane (like Jupyter or VS Code's interactive window), the plot may appear there instead of a separate window.

My matplotlib plot window isn't appearing when I run the script. How can I fix or work around this?

Create the method comparison bar chart

The scatter plot shows HOW predictions improved. A bar chart shows BY HOW MUCH each method reduced error, making it easy for anyone to see the winner at a glance.

  • Below the scatter plot code, add the bar chart code:
# --- Visualization 2: Method Comparison Bar Chart ---
methods = ["Mean\nImputation", "Raw Linear\nRegression", "Log-Transformed\nLinear Regression"]
rmse_values = [rmse_baseline, rmse_raw, rmse_improved]
colors = ["#d32f2f", "#ff9800", "#4caf50"]

plt.figure(figsize=(8, 5))
bars = plt.bar(methods, rmse_values, color=colors, edgecolor="black", width=0.6)
plt.ylabel("RMSE (Lower is Better)")
plt.title("Imputation Method Comparison - RMSE")
for bar, val in zip(bars, rmse_values):
    plt.text(
        bar.get_x() + bar.get_width() / 2,
        bar.get_height() + max(rmse_values) * 0.02,
        f"{val:.1f}",
        ha="center",
        va="bottom",
        fontweight="bold",
    )
plt.tight_layout()
plt.savefig("method_comparison.png", dpi=150)
plt.show()
print("Saved: method_comparison.png")

What does this code do?

  • Three bars represent mean imputation (red), raw linear regression (orange), and log-transformed linear regression (green). Lower RMSE is better, so the green bar should be the shortest.
  • The plt.text() loop labels each bar with its exact RMSE value, making the chart self-explanatory without needing to read axis ticks.
  • Save census_imputation.py.

Before you run the script, what do you expect to see in the bar chart? Which method should have the lowest (best) RMSE bar?

  • Run the complete script one final time:
python census_imputation.py

Two plot windows appear in sequence. Close the first (scatter plot) to see the second (bar chart). The log-transformed bar is the shortest, confirming it's your best method. Both "Saved:" messages print to the terminal.

  • Confirm both image files were saved by running:
ls *.png

You should see improvement_comparison.png and method_comparison.png listed in the output.

Not seeing both .png files?

  • If only one file appears, check that both plt.savefig() calls are present in your code. Each visualization needs its own save call.
  • Make sure you're running ls from the same folder where your script lives. The images save to the current working directory.
  • If the script errored mid-way, only plots before the error get saved. Check the terminal for any traceback.

My script ran but I can't find the saved .png files. Help me troubleshoot.

✔️ Awesome, I've got everything!

Great. Double check you've saved census_imputation.py and both plot images are in your project folder.

ⓧ I'd like to double check the full code

Your file is too long to show in one block, so confirm each section is present in order by scrolling through census_imputation.py:

  • Imports (pandas, numpy, LinearRegression, mean_squared_error, r2_score, matplotlib.pyplot) and DATA_FILE configuration at the top.
  • Step 2: Load and Explore section (CSV load, column discovery, density statistics).
  • Step 3: Simulate Missing Data section (random 20% removal, mean imputation baseline, raw linear regression).
  • Step 4: Log Transformation section (np.log1p() transform, model training, np.expm1() inverse).
  • Step 5: Resume Statistics block (improvement_over_baseline calculation, dataset scale, model performance, suggested bullets).
  • Visualization 1: Side-by-side scatter plot saved as improvement_comparison.png.
  • Visualization 2: Bar chart saved as method_comparison.png.

Secret mission

Compare Against KNN Imputation

Real-world data scientists never stop at one approach. In this extension, you'll add KNN imputation as a fourth method and produce a full comparison table showing which algorithm wins on your census data. Your updated resume bullets will cite whichever method performed best.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • census_imputation.py (your Python pipeline script).
  • People.csv (downloaded USDA ERS Atlas data file).
  • improvement_comparison.png and method_comparison.png (generated visualizations).
  • Python packages installed via pip: pandas, scikit-learn, matplotlib, numpy.

Keep everything running

No action needed. Choose this if you plan to continue iterating on the pipeline or want to reference your code and visualizations later.

  • Your Python script and downloaded data are just local files on your machine. They consume minimal disk space.
  • The pip packages (pandas, scikit-learn, matplotlib, numpy) remain available for other Python projects.
  • Consider pushing your code and generated plots to a GitHub repository so you can link them from your resume.

Pause - I'll come back to this later

Since everything is local files, there is nothing to pause or shut down. Your script and data will be waiting whenever you come back.

  • No running processes or servers to stop.
  • Your pip packages stay installed and ready for next time.
  • To pick back up, open your terminal, navigate to the folder containing census_imputation.py, and run the script again.

Delete - I don't want to use this again

Remove all project files and uninstall the Python packages you added.

  • Delete your project files by running the following command in the folder that contains them.

On macOS or Linux:

rm census_imputation.py People.csv improvement_comparison.png method_comparison.png

On Windows:

del census_imputation.py, People.csv, improvement_comparison.png, method_comparison.png
  • If you still have the extracted USDA Atlas ZIP folder, delete that folder as well.
  • Uninstall the Python packages by running:
pip uninstall pandas scikit-learn matplotlib numpy -y

Keep packages if you do other data science work

pandas, scikit-learn, matplotlib, and numpy are foundational data science packages. If you plan to do any other Python data work, keep them installed.

Nice Work!

Nice Work!

Nice work! You just built a complete data science pipeline that takes real U.S. census data, simulates missing values, and predicts them using machine learning.

You've learned how to:

  • Processed 3,100+ county-level census records from the USDA ERS Atlas and built a linear regression model to predict missing population density values from correlated demographic features.
  • Applied log transformation to handle extreme right-skewed distributions, dramatically improving prediction accuracy and producing quantifiable R² and RMSE statistics for your resume.
  • Generated portfolio-ready visualizations comparing imputation methods, including actual-vs-predicted scatter plots and a bar chart ranking each approach by error.
  • Secret Mission: Compared linear regression against KNN imputation and produced a full four-method comparison table with dynamically selected best-performing approach.

Ready to quiz yourself?