Analyze GBV Patterns with ML

Apply PCA, K-Means, and Random Forest to real DHS survey data from 34 African countries.

Introduction

30 Second Summary

Gender-based violence affects millions of people across Africa, but the patterns behind it are complex and vary widely between countries. Data science can cut through that complexity, revealing which socioeconomic factors correlate most strongly with violence prevalence and where intervention efforts might have the greatest impact.

In this project, you will build a complete machine learning notebook that analyzes real Demographic and Health Survey data spanning 34 African countries over 20 years. You will uncover hidden patterns through visualization, cluster countries by their GBV profiles, and train a classifier that predicts prevalence levels from socioeconomic indicators.

What You'll Build

By the end, you will have a portfolio-ready notebook you can run top to bottom, producing publication-quality visualizations and ML insights that tell a data-driven story about gender-based violence across an entire continent.

By the end of this project, you'll have:

  • A heatmap visualization showing GBV prevalence across 34 African countries and survey years, instantly revealing which nations have the highest rates.
  • An unsupervised clustering model that automatically groups countries by their GBV profiles using PCA and K-Means, with a scatter plot showing the distinct groupings.
  • A Random Forest classifier that predicts high vs. low GBV prevalence from socioeconomic features, with a feature importance chart showing which factors (education, wealth, urbanization) matter most.
  • Secret Mission: Optimize your clustering with the elbow method and silhouette analysis to find the ideal number of country groups.

Are there any prerequisites?

This project assumes you are comfortable with Python and pandas. Some familiarity with machine learning concepts (what classification and clustering mean) is helpful, but not required. The notebook walks you through each step.

Before We Start

Before we dive into code, let's lock in the question this notebook will answer and why it matters. Every good data science project starts with a clear question, not a dataset.

Set Up and Load the Dataset

You articulated the question your notebook will answer. Now you need the data to answer it.

The dataset you need lives on Hugging Face and requires one library that is not pre-installed in Google Colab. In this step, you will install that library, load 252K rows of real DHS Domestic Violence survey data, and confirm the DataFrame is ready to explore.

In this step, get ready to:
  • Install the Hugging Face datasets library and import all packages.
  • Load the Africa DHS Domestic Violence dataset into a pandas DataFrame.
  • Verify the dataset shape, country count, and year range.
Install the datasets library and import packages

Google Colab already has pandas, numpy, matplotlib, seaborn, and scikit-learn installed. The one missing piece is the datasets library from Hugging Face, which lets you load ML-ready datasets in a single line.

  • In your Colab notebook, click + Code to create a new code cell.
  • Paste the following install command and imports into the cell:
# ============================================================
# Cell 1: Setup - Install and Load Data
# ============================================================
%pip install datasets

from datasets import load_dataset
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report

What does this code do?

%pip install datasets installs the Hugging Face datasets library directly into your Colab kernel. The %pip magic command ensures the package lands in the correct notebook environment.

The import block pulls in every tool you will use across the entire project: pandas for DataFrames, numpy for math, matplotlib and seaborn for plotting, and several scikit-learn modules for machine learning.

Why import everything now?

Loading all imports up front is standard practice in notebook projects. It means you will never hit a missing-import error halfway through a later cell, and anyone reading your notebook can see all dependencies at a glance.

Load the dataset and verify the data

The dataset is hosted on Hugging Face Hub as a Parquet file. One function call downloads and caches it, then you convert it to a familiar pandas DataFrame.

  • Below the imports in the same cell, add the following code to load and verify the dataset:
# Load the Africa DHS Domestic Violence dataset from Hugging Face
ds = load_dataset("electricsheepafrica/africa-dhs-domestic-violence")
df = ds["train"].to_pandas()

# Verify the data
print(f"Shape: {df.shape}")
print(f"Countries: {df['CountryName'].nunique()}")
print(f"Year range: {df['SurveyYear'].min()} - {df['SurveyYear'].max()}")
print(f"\nColumns: {list(df.columns)}")
df.head()

What does this code do?

load_dataset("electricsheepafrica/africa-dhs-domestic-violence") downloads the full DHS Domestic Violence dataset from the Hugging Face Hub. It returns a Dataset object with a single train split.

ds["train"].to_pandas() converts that into a standard pandas DataFrame called df. The print statements then confirm row count, country count, and the year range so you know the data loaded correctly.

Before you run the cell, what shape do you expect? Think about how many rows and columns a dataset covering 34 countries over 20 years of surveys might contain.

  • Click the play button (or press Shift+Enter) to run the cell.

You should see installation output from pip, then the verification prints showing the dataset has loaded.

What should I see?

The cell output starts with pip download and install messages. After that, you should see:

Shape: (252083, 20)
Countries: 34
Year range: 2003 - 2024

This confirms 252,083 rows across 20 columns, covering 34 African countries from 2003 to 2024. You will also see a table preview from df.head() showing the first five rows with columns like CountryName, SurveyYear, Indicator, and Value.

Not seeing the expected output?

  • If you see a ModuleNotFoundError for datasets, make sure the %pip install datasets line is at the top of your cell and you ran the entire cell, not just part of it.
  • If the download seems stuck, Colab may be throttled. Wait 30 seconds and try running the cell again.
  • If your shape does not match (252083, 20), double-check that the dataset name string is exactly electricsheepafrica/africa-dhs-domestic-violence with no typos.

Still stuck? Help me troubleshoot my dataset loading in Google Colab.

✔️ Awesome, I've got everything!

Great. Double check you have saved the cell (Colab auto-saves, but verify the output is still visible).

ⓧ I'd like to double check the full code

Here is the complete Cell 1. Make sure yours matches exactly:

# ============================================================
# Cell 1: Setup - Install and Load Data
# ============================================================
%pip install datasets

from datasets import load_dataset
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report

# Load the Africa DHS Domestic Violence dataset from Hugging Face
ds = load_dataset("electricsheepafrica/africa-dhs-domestic-violence")
df = ds["train"].to_pandas()

# Verify the data
print(f"Shape: {df.shape}")
print(f"Countries: {df['CountryName'].nunique()}")
print(f"Year range: {df['SurveyYear'].min()} - {df['SurveyYear'].max()}")
print(f"\nColumns: {list(df.columns)}")
df.head()

You now have a quarter-million rows of real survey data ready to explore. Next up, you will visualize GBV patterns across countries and years to understand what the data looks like before applying any machine learning.

Explore GBV Patterns with Visualizations

Your dataset is loaded with 252K rows of survey data. Before jumping into machine learning, you need to understand what the data actually looks like.

Skipping Exploratory Data Analysis is the most common beginner mistake in data science. In this step, you will create visualizations that immediately reveal which countries have the highest GBV prevalence and how socioeconomic factors connect to those patterns.

In this step, get ready to:
  • Filter the dataset to national-level physical violence indicators and create a country-by-year prevalence summary.
  • Build a heatmap of GBV prevalence across countries and survey years.
  • Create bar charts comparing GBV prevalence by wealth quintile and education level.
Filter to national-level physical violence indicators

The dataset contains rows for many disaggregation levels (by region, wealth, education). For the heatmap, you only want the national totals so each country has one prevalence value per year.

  • Create a new code cell below your setup cell.
  • Paste the following code to filter for national-level physical violence data and build a country-by-year summary:
# ============================================================
# Cell 2: EDA - Visualize GBV Patterns
# ============================================================

# Filter to national totals for physical violence indicators
national = df[df["IsTotal"] == 1].copy()

# Find indicators related to physical violence (experienced physical violence)
violence_indicators = national[national["Indicator"].str.contains("physical violence", case=False, na=False)]
print(f"Physical violence indicators found: {violence_indicators['Indicator'].nunique()}")
print(violence_indicators["Indicator"].unique()[:5])

# Pick the most common physical violence indicator
top_indicator = violence_indicators["IndicatorId"].value_counts().index[0]
phys_violence = violence_indicators[violence_indicators["IndicatorId"] == top_indicator]

# Create country-year prevalence summary
prevalence_pivot = phys_violence.pivot_table(
    index="CountryName",
    columns="SurveyYear",
    values="Value",
    aggfunc="mean"
)

What does this code do?

  • national = df[df["IsTotal"] == 1].copy() filters to national-level aggregate rows, removing subnational breakdowns.
  • The .str.contains("physical violence") filter narrows to indicators about physical violence specifically.
  • top_indicator picks the most frequently reported physical violence indicator across all countries.
  • The pivot_table reshapes the data so each row is a country and each column is a survey year, with prevalence values as cells.
  • Run the cell by pressing the play button or Shift+Enter.

You should see output showing how many physical violence indicators exist in the dataset, plus a list of their names.

Seeing a NameError for df?

Make sure you ran your Cell 1 (setup) first. Google Colab loses all variables when the runtime disconnects.

Re-run Cell 1, then run Cell 2 again.

Help me debug my NameError in Colab.

Build a heatmap of GBV prevalence across countries and years

The pivot table you just created is the perfect shape for a heatmap. Each cell represents one country's prevalence in one survey year.

  • Add this code below the pivot table code in the same cell:
# Heatmap of GBV prevalence across countries and years
plt.figure(figsize=(14, 10))
sns.heatmap(
    prevalence_pivot,
    cmap="YlOrRd",
    annot=True,
    fmt=".1f",
    linewidths=0.5,
    cbar_kws={"label": "Prevalence (%)"}
)
plt.title("Physical Violence Prevalence by Country and Survey Year", fontsize=14)
plt.xlabel("Survey Year")
plt.ylabel("Country")
plt.tight_layout()
plt.show()

What does this code do?

  • sns.heatmap() takes the pivot table and renders each value as a colored cell. The YlOrRd colormap ranges from yellow (low) to red (high).
  • annot=True prints the actual percentage inside each cell so you can read exact values.
  • Countries appear on the y-axis, survey years on the x-axis. You can immediately spot which countries have the darkest red cells.

Before you run this, what do you expect the heatmap to show? Do you think prevalence will be uniform across countries, or will some stand out?

  • Re-run the entire cell with Shift+Enter.

You should see a large heatmap with countries listed down the left side and survey years across the top. Clear color variation reveals that some countries have much higher physical violence prevalence rates than others.

Heatmap looks empty or all one color?

Check that your prevalence_pivot is not empty. Add print(prevalence_pivot.shape) above the plot to confirm it has rows and columns.

If the shape shows zero rows, the str.contains filter may not have matched anything. Double check the string "physical violence" is lowercase in your code.

Help me debug my empty heatmap.

Create bar charts comparing prevalence by wealth and education

The heatmap shows geographic patterns. Now you want to know whether socioeconomic factors like wealth and education correlate with GBV prevalence.

  • Create a new code cell below Cell 2.
  • Paste the following code to filter by wealth quintile and education level, then plot side-by-side bar charts:
# ============================================================
# Cell 3: EDA - Socioeconomic Patterns
# ============================================================

# Filter for disaggregations by wealth and education
wealth_data = df[
    (df["IndicatorId"] == top_indicator) &
    (df["CharacteristicCategory"] == "Wealth quintile")
]

education_data = df[
    (df["IndicatorId"] == top_indicator) &
    (df["CharacteristicCategory"] == "Education")
]

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Wealth quintile comparison
wealth_summary = wealth_data.groupby("CharacteristicLabel")["Value"].mean().sort_values()
wealth_summary.plot(kind="barh", ax=axes[0], color="coral")
axes[0].set_title("Mean Physical Violence Prevalence by Wealth Quintile")
axes[0].set_xlabel("Prevalence (%)")

# Education level comparison
edu_summary = education_data.groupby("CharacteristicLabel")["Value"].mean().sort_values()
edu_summary.plot(kind="barh", ax=axes[1], color="steelblue")
axes[1].set_title("Mean Physical Violence Prevalence by Education Level")
axes[1].set_xlabel("Prevalence (%)")

plt.tight_layout()
plt.show()

What does this code do?

  • wealth_data filters for rows where the CharacteristicCategory is "Wealth quintile". This gives you prevalence broken down by poorest through richest.
  • education_data does the same for education levels (no education, primary, secondary, higher).
  • plt.subplots(1, 2) creates a figure with two side-by-side panels. Each panel gets its own horizontal bar chart showing mean prevalence per category.
  • Run the cell with Shift+Enter.

You should see two horizontal bar charts side by side. The left chart (coral) shows prevalence by wealth quintile. The right chart (steelblue) shows prevalence by education level.

Seeing an empty chart or a KeyError?

Make sure top_indicator is defined. This variable was created in Cell 2. If you only ran Cell 3 without running Cell 2 first, the variable does not exist yet.

Also verify the string "Wealth quintile" matches exactly (capital W, lowercase q). The filter is case-sensitive.

Help me fix my bar chart error.

What patterns do the bar charts reveal?

Look at whether poorer households show higher or lower GBV prevalence than richer ones. Check if higher education correlates with lower prevalence.

These socioeconomic patterns are exactly the features your machine learning model will use in later steps to predict GBV levels.

✔️ Awesome, I've got everything!

Double check you have both Cell 2 and Cell 3 saved and producing output.

ⓧ I'd like to double check the full code

Your notebook should now have three cells in order. Scroll up and confirm each one is present:

  • Cell 1: Setup - Install and Load Data (from Step 1). Installs datasets, imports all libraries, loads the DataFrame, and prints shape/country/year info.
  • Cell 2: EDA - Visualize GBV Patterns. Filters to national totals, finds the top physical violence indicator, creates prevalence_pivot, and renders the heatmap.
  • Cell 3: EDA - Socioeconomic Patterns. Filters for wealth quintile and education disaggregations, then plots two side-by-side horizontal bar charts.

Your visualizations already tell a story. You can see geographic and socioeconomic patterns in GBV prevalence across Africa. Next up, you will reshape this data into a format that machine learning models can actually consume.

Reshape Data for Machine Learning

Your visualizations revealed clear patterns in the data. Now you want to let a machine learning model discover patterns you might have missed. The natural instinct is to hand your DataFrame directly to a model and see what happens.

The problem is that your data is in long format: one row per indicator per disaggregation. ML models need a feature matrix where each row is one observation and columns are numeric features. In this step, you will pivot the data into that shape.

In this step, get ready to:
  • Attempt to pass the raw DataFrame to KMeans and observe the error.
  • Pivot the data into a feature matrix where each row is one country-year.
  • Handle missing values and standardize the feature matrix using StandardScaler.
Attempt to fit KMeans on the raw DataFrame

You have 252K rows of data and KMeans is already imported. What happens if you just pass the DataFrame straight in?

Before you run the next cell, predict: do you think KMeans will work on df as-is, or will it fail? Why?

  • Create a new code cell below your EDA cells and paste this code:
# ATTEMPT: Try feeding raw data to KMeans (this will fail)
print("Attempting to fit KMeans on raw DataFrame...")
try:
    kmeans_test = KMeans(n_clusters=3, random_state=42, n_init=10)
    kmeans_test.fit(df)
except Exception as e:
    print(f"ERROR: {type(e).__name__}: {e}")
    print("\nThe raw data is in long format - we need to reshape it into a feature matrix!")

What does this code do?

This wraps the KMeans attempt in a try/except block so you can see the error message without crashing your notebook.

KMeans expects a 2D array of numbers. Your df contains text columns like CountryName and Indicator, plus the data is in long format with 252K rows of repeated categories. That is not a valid feature matrix.

  • Run the cell.

You see an error. The DataFrame has non-numeric columns that KMeans cannot process. This is exactly the problem: the raw long-format data is not shaped for ML.

Why does long format break ML models?

In long format, each row represents one indicator value for one disaggregation. A single country-year is spread across dozens of rows. ML models need the opposite: each row should be one observation (a country-year), and each column should be a separate numeric feature.

Reshaping long data into a wide feature matrix is one of the most common data wrangling tasks in real-world data science. It is what separates analysis from production ML work.

Pivot the data into a feature matrix

The fix is a pivot. You will take the national-level totals for each indicator and spread them into columns, so each row becomes one country-year with numeric indicator values as features.

  • Create a new code cell below and paste this code:
# Strategy: pivot so each row = country-year, columns = indicator values by disaggregation
# Use national totals across multiple indicators as features

# Get all national-level indicator values per country-year
feature_data = df[df["IsTotal"] == 1][["CountryName", "SurveyYear", "IndicatorId", "Value"]].copy()
feature_data = feature_data.dropna(subset=["Value"])

# Pivot: each row = country-year, each column = an indicator
feature_matrix = feature_data.pivot_table(
    index=["CountryName", "SurveyYear"],
    columns="IndicatorId",
    values="Value",
    aggfunc="mean"
)

What does this code do?

  • Filters to national totals (IsTotal == 1) and keeps only the four columns needed: country, year, indicator ID, and value.
  • Drops rows where Value is NaN to avoid polluting the pivot.
  • pivot_table reshapes from long to wide: each unique (CountryName, SurveyYear) pair becomes one row, and each IndicatorId becomes its own column filled with the mean Value.
Handle missing values and standardize

Not every country-year has data for every indicator. You need to drop sparse columns and remaining NaN rows, then standardize the features so they are on the same scale for clustering.

  • Add this code to the same cell, directly below the pivot:
# Drop columns with too many missing values (keep columns with >50% data)
threshold = len(feature_matrix) * 0.5
feature_matrix = feature_matrix.dropna(axis=1, thresh=int(threshold))

# Drop rows with any remaining NaN
feature_matrix = feature_matrix.dropna()

print(f"Feature matrix shape: {feature_matrix.shape}")
print(f"Countries represented: {feature_matrix.index.get_level_values('CountryName').nunique()}")
print(f"Features (indicators): {feature_matrix.shape[1]}")

# Standardize
scaler = StandardScaler()
X_scaled = scaler.fit_transform(feature_matrix)
print(f"\nScaled matrix shape: {X_scaled.shape}")
print("Ready for ML!")

What does this code do?

  • Drops any indicator column where more than 50% of rows are missing. This removes indicators that only a few surveys measured.
  • Drops remaining rows with any NaN so the matrix is fully numeric with no gaps.
  • StandardScaler centers each feature to mean 0 and scales to standard deviation 1. This prevents features with large values from dominating the clustering.

Before you run this cell, predict: roughly how many rows do you expect after the pivot? Remember, the original had 252K rows in long format with 34 countries.

  • Run the cell.

You should see a feature matrix shape of roughly 70-100 rows by 10-20 columns. Each row is now one country-year observation, and each column is a numeric indicator value. The final line prints "Ready for ML!" confirming the matrix is scaled and ready for sklearn.

Seeing an empty DataFrame or very few rows?

If your feature matrix has 0 rows, check that your Cell 2 code ran successfully and top_indicator is defined. The pivot depends on earlier cells.

If you see a "name not defined" error, run all cells from the top of the notebook in order. Colab does not automatically re-run prior cells.

Help me debug my feature matrix pivot

✔️ Awesome, I've got everything!

Great. Confirm that Cell 5 is saved in your notebook.

ⓧ I'd like to double check the full code

Scroll through your notebook and confirm these cells exist in order:

  • Cell 1: Setup - Install and Load Data (installs datasets, imports, loads df).
  • Cell 2: EDA - Visualize GBV Patterns (national filter, heatmap).
  • Cell 3: EDA - Socioeconomic Patterns (wealth and education bar charts).
  • Cell 4: Reshape Data for ML - The Obstacle (KMeans attempt in try/except).
  • Cell 5: Proper Feature Matrix Construction (pivot, dropna, StandardScaler, prints "Ready for ML!").

Your data went from 252K rows of long-format survey responses to a compact, numeric feature matrix ready for machine learning. Next up, you will feed this matrix into PCA and K-Means to discover hidden country groupings in the data.

Cluster Countries by GBV Profile

In the last step, you wrestled 252K rows of long-format data into a clean feature matrix where each row is a country-year and each column is an indicator.

Now comes the payoff: are there distinct "types" of GBV profiles across Africa? PCA will compress your features into two dimensions you can plot, and K-Means Clustering will reveal which countries share similar patterns.

In this step, get ready to:
  • Reduce the feature matrix to 2 dimensions with PCA and check how much variance is preserved.
  • Fit K-Means with 3 clusters and visualize country groupings on a scatter plot.
  • Analyze which countries belong to each cluster and what distinguishes them.
Apply PCA to reduce dimensions

Your feature matrix has 10-20 columns. You cannot plot that on a 2D screen. PCA solves this by projecting high-dimensional data down to 2 components while preserving as much variation as possible.

  • Create a new code cell in your notebook (below Cell 5) and add this code to apply PCA and print the explained variance:
# ============================================================
# Cell 6: PCA + K-Means Clustering (Unsupervised)
# ============================================================

# PCA for dimensionality reduction
pca = PCA(n_components=2, random_state=42)
X_pca = pca.fit_transform(X_scaled)

print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance explained: {pca.explained_variance_ratio_.sum():.2%}")

What does this code do?

PCA(n_components=2) creates a transformer that compresses all your indicator columns into just 2 new columns (principal components).

The explained variance ratio tells you how much of the original data's variation is captured in those 2 components. Higher is better. Anything above 40-50% total is workable for visualization.

  • Run the cell.
  • You should see two decimal values for the variance ratio and a total percentage showing how much information the 2 components retain.

Seeing a NameError for X_scaled?

Make sure you ran Cell 5 (the feature matrix construction) first. Colab cells must run in order because each cell depends on variables created in earlier cells.

Help me debug my NameError in the PCA cell.

Fit K-Means and visualize the clusters

Now that you have 2D coordinates for every country-year, you can ask K-Means to find natural groupings in the full feature space. The scatter plot will color each point by its assigned cluster so you can see whether countries genuinely separate.

  • Add this code to the same cell, below the PCA lines:
# K-Means clustering with 3 clusters
kmeans = KMeans(n_clusters=3, init="k-means++", n_init=10, random_state=42)
clusters = kmeans.fit_predict(X_scaled)

# Visualize clusters in PCA space
plt.figure(figsize=(12, 8))
scatter = plt.scatter(
    X_pca[:, 0], X_pca[:, 1],
    c=clusters, cmap="viridis", alpha=0.7, s=80, edgecolors="white"
)

# Label points with country names
country_names = feature_matrix.index.get_level_values("CountryName")
for i, name in enumerate(country_names):
    plt.annotate(
        name[:3],
        (X_pca[i, 0], X_pca[i, 1]),
        fontsize=7, alpha=0.7,
        xytext=(5, 5), textcoords="offset points"
    )

plt.colorbar(scatter, label="Cluster")
plt.xlabel(f"PC1 ({pca.explained_variance_ratio_[0]:.1%} variance)")
plt.ylabel(f"PC2 ({pca.explained_variance_ratio_[1]:.1%} variance)")
plt.title("African Countries Clustered by GBV Profile (PCA + K-Means)")
plt.tight_layout()
plt.show()

What does this code do?

  • KMeans(n_clusters=3) groups all country-years into 3 clusters based on similarity across all indicators. The k-means++ initialization picks smart starting centroids to avoid bad local optima.
  • fit_predict(X_scaled) clusters on the FULL scaled feature matrix (not the 2D PCA version), so the grouping uses all available information.
  • The scatter plot uses the PCA coordinates for positioning and the cluster labels for coloring. Country name abbreviations (name[:3]) label each point so you can identify who belongs where.

Before you run this, what do you expect to see? Do you think the countries will form tight, clearly separated groups, or overlap messily?

  • Run the cell.
  • You should see a 2D scatter plot with countries colored by cluster. Look for visual separation between the color groups and check which country abbreviations appear near each other.

Plot not showing or looks blank?

Make sure plt.show() is at the end of the plotting code. In Colab, plots sometimes render inline automatically, but the explicit call ensures it appears.

If you see an error about X_pca not being defined, the PCA lines at the top of the cell may have a typo. Confirm the PCA block runs without errors before the KMeans block.

Help me figure out why my cluster scatter plot isn't rendering.

Analyze cluster characteristics

A pretty scatter plot is not enough. You need to know which countries ended up in each cluster and whether the groupings make intuitive sense.

  • Add this code at the bottom of the same cell to print cluster membership:
# Analyze cluster characteristics
feature_matrix_with_clusters = feature_matrix.copy()
feature_matrix_with_clusters["Cluster"] = clusters

print("\nCluster sizes:")
print(feature_matrix_with_clusters["Cluster"].value_counts().sort_index())

print("\nCountries per cluster:")
for c in sorted(feature_matrix_with_clusters["Cluster"].unique()):
    countries = feature_matrix_with_clusters[
        feature_matrix_with_clusters["Cluster"] == c
    ].index.get_level_values("CountryName").unique().tolist()
    print(f"  Cluster {c}: {', '.join(countries[:8])}{'...' if len(countries) > 8 else ''}")

What does this code do?

  • It attaches the cluster label to each row in your feature matrix so you can filter by group.
  • value_counts().sort_index() shows how many country-years landed in each cluster. Uneven sizes are normal and often meaningful.
  • The loop lists the actual country names per cluster (up to 8) so you can spot geographic or developmental patterns.
  • Run the cell.
  • You should see cluster sizes printed (the count of country-years in each cluster) followed by lists of country names grouped by cluster assignment.

What should I see?

The three clusters represent distinct GBV profiles. You might notice that countries with similar socioeconomic conditions or geographic proximity end up in the same cluster. The cluster sizes will likely be unequal because some profiles are more common than others across the continent.

Seeing only one or two countries per cluster?

This means your feature matrix ended up very small after dropping NaN rows. Go back to Cell 5 and check that the threshold for dropping columns is set to 0.5 (50%). A stricter threshold removes too many rows.

Help me understand why my clusters have very few data points.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook in Colab (it auto-saves, but you can press **Ctrl+S** to be sure).

ⓧ I'd like to double check the full code

Your Cell 6 should contain all three sections in order. Scroll up through the cell and confirm you have:

  • The PCA block: pca = PCA(n_components=2, random_state=42) and the two print statements for explained variance.
  • The K-Means + visualization block: kmeans = KMeans(n_clusters=3, init="k-means++", n_init=10, random_state=42) followed by the scatter plot with annotations and plt.show().
  • The cluster analysis block: feature_matrix_with_clusters = feature_matrix.copy() with the loop printing countries per cluster.

You now have visual proof that distinct GBV profiles exist across Africa. Next up, you will build a supervised classifier that predicts whether a country-year has high or low prevalence, and discover which socioeconomic features drive that prediction.

Predict GBV Prevalence with Random Forest

Your clustering revealed that distinct GBV profiles exist across African countries. The natural next question is more actionable: given socioeconomic indicators for a country-year, can you predict whether GBV prevalence will be high or low?

A Random Forest classifier can answer this question. Beyond making predictions, it ranks which features matter most. That ranking tells policymakers exactly where to focus interventions.

In this step, get ready to:
  • Create a binary target variable (high/low prevalence) from a median split.
  • Train a Random Forest classifier and evaluate it with a classification report.
  • Visualize feature importances to identify which socioeconomic factors predict GBV levels.
Create a binary target variable

Supervised ML needs a label to predict. You will split countries into "high" and "low" prevalence groups using the median value as the dividing line. This ensures balanced classes for your classifier.

  • Create a new code cell below your clustering cell.
  • Add the following code to define your target variable and feature set:
# ============================================================
# Cell 7: Random Forest Classification (Supervised)
# ============================================================

# Create binary target: high vs low prevalence based on primary violence indicator
if top_indicator in feature_matrix.columns:
    target_col = top_indicator
else:
    target_col = feature_matrix.columns[0]

median_val = feature_matrix[target_col].median()
y = (feature_matrix[target_col] >= median_val).astype(int)
y.name = "high_prevalence"

# Features: all columns except the target
feature_cols = [col for col in feature_matrix.columns if col != target_col]
X = feature_matrix[feature_cols]

What does this code do?

This code transforms a continuous variable into a binary classification problem.

  • The if/else block safely selects the primary violence indicator as the target column. If it is missing from the feature matrix, it falls back to the first available column.
  • The median split (>= median_val) labels each country-year as 1 (high prevalence) or 0 (low prevalence). Using the median guarantees roughly equal class sizes.
  • feature_cols removes the target column from the feature list so you do not leak the answer into the model.
  • Run the cell by pressing Shift+Enter.
  • The cell should execute without error. No printed output is expected yet.

Seeing a KeyError or NameError?

  • If you see NameError: name 'top_indicator' is not defined, re-run Cell 2 (EDA) so the variable exists in memory.
  • If you see NameError: name 'feature_matrix' is not defined, re-run Cell 5 (Feature Matrix Construction).
  • help me fix a NameError in my Random Forest target creation cell
Train a Random Forest and print the classification report

With your features and target defined, you can now standardize, split the data, and train a Random Forest classifier. The classification report will show precision, recall, and accuracy.

  • Add the following code to the same cell, directly below the code you just wrote:
# Standardize features
scaler_supervised = StandardScaler()
X_scaled_sup = scaler_supervised.fit_transform(X)

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X_scaled_sup, y, test_size=0.3, random_state=42, stratify=y
)

# Train Random Forest
rf = RandomForestClassifier(n_estimators=100, random_state=42, max_depth=5)
rf.fit(X_train, y_train)

# Evaluate
y_pred = rf.predict(X_test)
print("Classification Report:")
print(classification_report(y_test, y_pred, target_names=["Low Prevalence", "High Prevalence"]))

What does this code do?

  • StandardScaler() normalizes each feature to zero mean and unit variance. This prevents features with large numeric ranges from dominating the model.
  • train_test_split holds back 30% of data for testing. The stratify=y parameter ensures both classes are proportionally represented in each split.
  • max_depth=5 limits tree depth to prevent overfitting on the small pivoted dataset while still producing interpretable results.
  • classification_report prints precision, recall, and F1-score for each class. These metrics tell you how well the model distinguishes high from low prevalence.

Before you run the cell, predict: do you think the model will achieve above 70% accuracy on data it has never seen?

  • Run the cell with Shift+Enter.

You should see a classification report printed with precision, recall, and F1-score for both "Low Prevalence" and "High Prevalence" classes. The overall accuracy should be above 70%.

Seeing accuracy below 50%?

  • Confirm that your Cell 5 (feature matrix construction) ran successfully and that you did not skip the missing-value cleanup steps.
  • Check that you used the exact parameters shown above. Changing the random_state or removing stratify can produce different results.
  • help me debug my low Random Forest accuracy
Visualize feature importances

A classification report tells you the model works. Feature importance tells you why it works. This chart answers the most actionable question: which socioeconomic factors are the strongest predictors of high GBV prevalence?

  • Add the following code to the bottom of the same cell:
# Feature importance visualization
importances = pd.Series(rf.feature_importances_, index=feature_cols)
top_features = importances.nlargest(10)

plt.figure(figsize=(10, 6))
top_features.sort_values().plot(kind="barh", color="darkgreen")
plt.xlabel("Feature Importance")
plt.title("Top 10 Predictors of High GBV Prevalence (Random Forest)")
plt.tight_layout()
plt.show()

print(f"\nTop predictor: {top_features.index[0]}")
print(f"Model accuracy: {rf.score(X_test, y_test):.1%}")

What does this code do?

  • rf.feature_importances_ extracts how much each feature contributed to the model's decisions, as measured by mean decrease in impurity across all trees.
  • nlargest(10) selects only the top 10 predictors so the chart stays readable.
  • The horizontal bar chart (barh) sorts features from least to most important (bottom to top), making the top predictor visually prominent.

Before you run: which type of factor do you think will rank highest? Education, wealth, or urbanization?

  • Run the cell with Shift+Enter.

You should see a horizontal bar chart with dark green bars ranking the top 10 features by importance. The top predictor stands out clearly at the top of the chart.

Chart not displaying?

  • Make sure plt.show() is included at the end of the plotting section. Without it, Colab may not render the chart inline.
  • If you see an empty chart, confirm that the classification report printed successfully in the section above. The model must be trained before you can extract importances.
  • help me fix my feature importance chart not showing

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook.

ⓧ I'd like to double check the full code

Your Cell 7 should contain all three sections you built above, in this order:

  • The cell header comment and binary target creation (the if/else block, median split, and feature_cols/X definition).
  • The standardization, train/test split, Random Forest training, and classification report print.
  • The feature importance visualization with the horizontal bar chart and final print statements.

Scroll up through your cell and confirm each section is present, in order, with no missing lines.

You now have a complete supervised ML pipeline that predicts GBV prevalence levels and explains which factors drive the predictions. Next up, the Secret Mission challenges you to validate whether your earlier choice of 3 clusters was actually optimal.

Secret mission

Optimize Your Clustering

You chose k=3 clusters arbitrarily. Real data scientists validate their cluster count with data-driven methods. Can you use the elbow method and silhouette analysis to find the true optimal number of country groupings?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely in Google Colab, so there are no ongoing costs.

Resources you used:

  • Google Colab notebook (saved in Google Drive).
  • The datasets library (installed in Colab runtime, resets each session).

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend the notebook.

  • Your Colab notebook stays in Google Drive and is accessible anytime.
  • The datasets library will need to be reinstalled each time you reopen the notebook (Colab environments reset between sessions).
  • There are no ongoing costs for keeping the notebook in Google Drive.

Pause - I'll come back to this later

Close your browser tab to free up Colab compute resources. Your notebook is already saved to Google Drive.

  • Close the Colab browser tab. The runtime shuts down automatically after inactivity.
  • Your notebook file persists in Google Drive. Reopen it anytime to pick up where you left off.
  • You will need to re-run all cells from the top when you reopen (including the %pip install datasets cell) since Colab does not preserve runtime state.

Delete - I don't want to use this again

Remove the notebook from Google Drive to fully clean up.

  • Open Google Drive at drive.google.com.
  • Navigate to the Colab Notebooks folder (or wherever you saved the notebook).
  • Right-click the notebook file and select Move to trash.
  • Empty your Google Drive trash to permanently delete the file.

Nice Work!

Nice Work!

You just built a complete data science portfolio notebook that analyzes gender-based violence patterns across 34 African countries using real DHS survey data. From raw indicators to actionable ML insights, your notebook tells a data-driven story.

You've learned how to:

  • Wrangle 252K rows of institutional survey data into an ML-ready feature matrix using long-to-wide pivots and StandardScaler normalization.
  • Discover hidden country groupings with PCA dimensionality reduction and K-Means clustering, visualized as labeled scatter plots.
  • Train a Random Forest classifier to predict GBV prevalence levels and extract feature importances showing which socioeconomic factors matter most.
  • Secret Mission: Validated your cluster quality with the elbow method and silhouette analysis to find the data-driven optimal number of country groups.

Ready to quiz yourself?