Analyze a Wearable Biometric Dataset

Introduction

30 Second Summary

Smartwatches and fitness trackers collect biometric data every second of the day. Heart rhythms, body motion, rotation. Behind every "calories burned" number or sleep score is a pipeline that turns raw sensor noise into meaningful patterns.

In this project, you will build a complete data analysis pipeline in Python that processes real wearable sensor recordings from 10 subjects performing 12 physical activities, generates publication-quality visualizations, and exports a feature-engineered matrix ready for machine learning classification.

What You'll Build

You can run a single command and watch your pipeline load raw ECG and accelerometer data, compute statistics, generate figures that reveal how running looks nothing like sitting in sensor space, and output a feature file that a classifier can train on immediately.

By the end of this project, you'll have:

• A runnable analysis pipeline where you type python main.py and see it load raw sensor data, compute statistics, generate figures, and export features in one shot.
• A set of publication-quality figures you can open and immediately see which sensor channels discriminate between walking, running, cycling, and sitting.
• An exported ML-ready feature matrix as a CSV that you can point a future classifier at and begin training immediately.
• Secret Mission: Add hypothesis testing to statistically quantify which sensor features significantly discriminate between activity classes.

Are there any prerequisites?

You should be comfortable with Python fundamentals: functions, loops, file I/O, and basic object-oriented programming. No prior data science or statistics experience is required.

This is the first project in a continuous biometrics and AI engineering portfolio. The code patterns and repository structure you establish here carry forward through all subsequent modules.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up the Project Environment

Every production data science project starts with the same foundation: a reproducible environment where dependencies are pinned and the project structure is clear.

In this step, you will create the portfolio repository structure, install your data science dependencies into a virtual environment, download the real MHEALTH wearable sensor dataset, and write the configuration module that ties everything together.

In this step, get ready to:

• Create the portfolio directory structure with modular source code layout.
• Install pinned Python dependencies and verify the environment.
• Download the MHEALTH biometric dataset and implement the config module.

Create the portfolio repository and project structure

Your biometrics portfolio will hold multiple projects over time. Each project gets its own folder with a standard Python package layout.

• Press Cmd+Space (macOS) to open your search bar.
• Type VS Code and press Enter to open it.
• Open the integrated terminal by pressing Ctrl+` (backtick key).

• Move to your Desktop by running this command:

cd ~/Desktop

• Create the portfolio directory and project folder by running these commands:

mkdir -p biometrics-portfolio/project_01_biometric_data_analysis
cd biometrics-portfolio/project_01_biometric_data_analysis

• Create all the subdirectories and placeholder files by running:

mkdir -p src data/raw outputs/figures outputs/features tests
touch src/__init__.py src/config.py src/loader.py src/analysis.py src/visualization.py src/features.py
touch tests/__init__.py tests/test_loader.py
touch main.py requirements.txt .gitignore README.md

• Confirm the structure exists by running:

ls

You should see main.py, requirements.txt, README.md, and the folders src/, data/, outputs/, tests/ listed in the output.

Not seeing all the folders?

Hidden files like .gitignore might not show up with a plain ls. Run ls -a to see everything including hidden files.

Help me troubleshoot my project structure.

• Open this project folder in VS Code by running:

code .

What does this project structure do?

The src/ folder holds your pipeline modules (config, loading, analysis, visualization, features). The data/raw/ folder will store the MHEALTH sensor files.

The outputs/ folder collects generated figures and feature CSVs. This separation keeps raw data, source code, and generated artifacts cleanly apart.

Install dependencies and verify the environment

Your pipeline depends on five libraries. Pinning exact versions ensures anyone can reproduce your results months from now.

• Add the pinned dependencies to requirements.txt by pasting this content into the file in VS Code:

numpy==2.5.1
pandas==3.0.5
matplotlib==3.11.1
seaborn==0.13.2
scipy==1.18.0

What are these libraries?

• NumPy provides fast array operations for signal math.
• Pandas handles tabular data loading and grouping.
• Matplotlib and Seaborn generate publication-quality figures.
• SciPy supplies statistical tests for the Secret Mission.

• Save requirements.txt with Cmd+S.
• Create a virtual environment by running this command in the terminal:

python3 -m venv venv

• Activate the virtual environment by running:

source venv/bin/activate

You should see (venv) appear at the start of your terminal prompt.

• Install all dependencies by running:

pip install -r requirements.txt

What should I see?

Pip will download and install the five packages plus their own dependencies. You will see progress bars and a final message confirming all packages installed successfully. This may take 1-2 minutes.

• Verify the imports work by running this quick check:

python3 -c "import numpy, pandas, matplotlib, seaborn, scipy; print('All imports OK')"

You should see All imports OK printed with no errors.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is active. You should see (venv) in your prompt. If not, run source venv/bin/activate again and re-run the pip install command.

Help me debug my virtual environment imports.

Download the MHEALTH dataset and create the config module

The MHEALTH dataset from the UCI Machine Learning Repository contains recordings from 10 subjects wearing three body sensors while performing 12 physical activities. It is a 72.1 MB zip file.

• Download the dataset zip file by running:

curl -L -o mhealth_dataset.zip https://archive.ics.uci.edu/static/public/319/mhealth+dataset.zip

What does this command do?

• curl -L downloads the file and follows any redirects.
• -o mhealth_dataset.zip saves the download with a specific filename.
• The URL points to the official UCI ML Repository hosting for dataset 319.

• Extract the dataset and move the log files into your data/raw/ folder by running:

unzip -o mhealth_dataset.zip
mv MHEALTHDATASET/*.log data/raw/
rm -rf MHEALTHDATASET mhealth_dataset.zip

• Verify the subject files are in place by running:

ls data/raw/

You should see 10 log files listed: mHealth_subject1.log through mHealth_subject10.log.

Download failed or no log files?

If curl times out, check your internet connection and try the command again. If data/raw/ is empty after unzip, check whether the zip extracted into a nested folder. Run find . -name '*.log' to locate the files.

Help me find and move my MHEALTH log files.

Now you will create the configuration module that defines how the pipeline interprets the dataset.

• Open src/config.py in VS Code and paste the following code that defines project paths and dataset parameters:

"""Configuration constants for the biometric analysis pipeline."""

from pathlib import Path

# Project paths
PROJECT_ROOT = Path(__file__).parent.parent
DATA_DIR = PROJECT_ROOT / "data" / "raw"
OUTPUT_DIR = PROJECT_ROOT / "outputs"
FIGURES_DIR = OUTPUT_DIR / "figures"
FEATURES_DIR = OUTPUT_DIR / "features"

# MHEALTH dataset parameters
SAMPLING_RATE_HZ = 50
NUM_SUBJECTS = 10
FILE_PATTERN = "mHealth_subject{subject_id}.log"

What does this code do?

• PROJECT_ROOT uses Path(__file__) to compute the project root relative to this file's location. This means paths work no matter where you clone the repo.
• SAMPLING_RATE_HZ = 50 is the sensor recording frequency (50 samples per second). You will use this to convert sample counts into real time durations.
• FILE_PATTERN defines the naming convention for the 10 subject files so the loader can iterate them.

• Below the code you just pasted, add the column name definitions that map each of the 24 data columns to a readable name:

# Column definitions (0-indexed)
COLUMN_NAMES: list[str] = [
    "chest_acc_x",
    "chest_acc_y",
    "chest_acc_z",
    "ecg_lead_1",
    "ecg_lead_2",
    "ankle_acc_x",
    "ankle_acc_y",
    "ankle_acc_z",
    "ankle_gyro_x",
    "ankle_gyro_y",
    "ankle_gyro_z",
    "ankle_mag_x",
    "ankle_mag_y",
    "ankle_mag_z",
    "arm_acc_x",
    "arm_acc_y",
    "arm_acc_z",
    "arm_gyro_x",
    "arm_gyro_y",
    "arm_gyro_z",
    "arm_mag_x",
    "arm_mag_y",
    "arm_mag_z",
    "activity_label",
]

What do these columns represent?

Each row in a subject file has 24 space-separated values. The first 3 are chest accelerometer readings (X, Y, Z axes). Columns 4-5 are ECG leads. Columns 6-14 cover the left ankle sensor (acceleration, gyroscope, magnetometer). Columns 15-23 cover the right arm sensor. Column 24 is the activity label.

• Below the column names, add the activity label mapping and feature engineering parameters:

# Activity label mapping
ACTIVITY_LABELS: dict[int, str] = {
    0: "null",
    1: "standing",
    2: "sitting",
    3: "lying_down",
    4: "walking",
    5: "climbing_stairs",
    6: "waist_bends",
    7: "frontal_arm_elevation",
    8: "knees_bending",
    9: "cycling",
    10: "jogging",
    11: "running",
    12: "jump_front_back",
}

# Feature engineering parameters
WINDOW_SIZE_SECONDS = 5.0
WINDOW_OVERLAP = 0.5
WINDOW_SIZE_SAMPLES = int(WINDOW_SIZE_SECONDS * SAMPLING_RATE_HZ)
WINDOW_STEP_SAMPLES = int(WINDOW_SIZE_SAMPLES * (1 - WINDOW_OVERLAP))

# Sensor channels for analysis (excluding activity label)
SENSOR_COLUMNS: list[str] = COLUMN_NAMES[:-1]

What does this code do?

• ACTIVITY_LABELS maps the integer label in column 24 to a human-readable activity name. Label 0 means the subject was transitioning between activities.
• WINDOW_SIZE_SAMPLES = 250 means each analysis window spans 5 seconds at 50 Hz. WINDOW_STEP_SAMPLES = 125 gives 50% overlap between consecutive windows.
• SENSOR_COLUMNS is all columns except the label. You will use this list whenever you need just the sensor readings.

• Save src/config.py with Cmd+S.

Next, set up the package marker so Python treats src/ as an importable package.

• Open src/__init__.py and paste:

"""Biometric data analysis pipeline - Project 01."""

• Save src/__init__.py.

Now set up the .gitignore so large data files and generated outputs stay out of version control.

• Open .gitignore and paste:

# Virtual environment
venv/
.venv/

# Data files (too large for git)
data/raw/

# Python
__pycache__/
*.pyc
*.pyo
.pytest_cache/

# IDE
.vscode/
.idea/

# OS
.DS_Store

# Outputs (regenerated by pipeline)
outputs/figures/*.png
outputs/features/*.csv

Why exclude data and outputs?

The raw sensor files are 72 MB total. Git is not designed for large binary data. The outputs/ folder contains files your pipeline regenerates on every run, so committing them would just add noise to your git history.

• Save .gitignore.

Finally, create a minimal main.py that registers the CLI interface so you can verify the environment works end to end.

• Open main.py and paste:

"""Main pipeline orchestrator for biometric data analysis.

Usage:
    python main.py                  Run full analysis pipeline
    python main.py --hypothesis-test  Include hypothesis testing (Secret Mission)
    python main.py --help            Show this help message
"""

import argparse
import sys


def parse_args() -> argparse.Namespace:
    """Parse command-line arguments."""
    parser = argparse.ArgumentParser(
        description="Biometric Data Analysis Pipeline - Project 01"
    )
    parser.add_argument(
        "--hypothesis-test",
        action="store_true",
        help="Run hypothesis testing for feature discrimination (Secret Mission)",
    )
    return parser.parse_args()


if __name__ == "__main__":
    args = parse_args()

What does this code do?

This sets up the command-line interface using Python's built-in argparse module. Running python main.py --help will print usage information. You will expand this file in later steps to call each pipeline stage.

• Save main.py.

Before you run the verification, what do you think will happen when you pass --help to your script?

• Test the CLI by running:

python main.py --help

You should see the argparse help message with the description and the --hypothesis-test flag listed. No errors.

Seeing a SyntaxError or ImportError?

Make sure your virtual environment is still active ((venv) in the prompt). Check that you are in the project_01_biometric_data_analysis directory by running pwd. The output should end with project_01_biometric_data_analysis.

Help me fix my main.py or environment setup.

✔️ Awesome, I've got everything!

Great. Double check you have saved all your files.

ⓧ I'd like to double check the full code

Here is requirements.txt:

numpy==2.5.1
pandas==3.0.5
matplotlib==3.11.1
seaborn==0.13.2
scipy==1.18.0

Here is src/__init__.py:

"""Biometric data analysis pipeline - Project 01."""

Here is .gitignore:

# Virtual environment
venv/
.venv/

# Data files (too large for git)
data/raw/

# Python
__pycache__/
*.pyc
*.pyo
.pytest_cache/

# IDE
.vscode/
.idea/

# OS
.DS_Store

# Outputs (regenerated by pipeline)
outputs/figures/*.png
outputs/features/*.csv

Here is src/config.py. Scroll through each section to confirm yours matches:

• Section 1: docstring, import, and path constants (PROJECT_ROOT through FILE_PATTERN).
• Section 2: COLUMN_NAMES list with 24 entries ending in activity_label.
• Section 3: ACTIVITY_LABELS dict (13 entries, 0-12), feature window params, and SENSOR_COLUMNS.

Your environment is locked in. You have a professional project structure, pinned dependencies, real biometric data on disk, and a working CLI entry point. Next up, you will load those sensor files and see your first ECG waveform.

Load and Explore Raw Biometric Signals

Your project structure is ready and the config module maps every sensor column and activity label in the MHEALTH dataset. Before diving into analysis, you need to confirm the data actually loads correctly.

In this step, you will build the data loader module that reads all 10 subjects' sensor files into a single DataFrame, then generate your first visualization comparing biometric signals between sitting and running. Seeing a real ECG waveform and accelerometer trace from your pipeline is the first proof that everything works.

In this step, get ready to:

• Load all 10 subjects' sensor data into a single DataFrame.
• Validate the data structure and log key statistics.
• Visualize raw ECG and acceleration signals to compare sitting versus running.

Build the data loader module

The loader module reads the space-separated .log files from the MHEALTH dataset, names each column using the config you already wrote, and combines all subjects into one dataset.

• Open src/loader.py in VS Code and replace any existing content with the module imports:

"""Data loading and validation for the MHEALTH dataset."""

import logging
from pathlib import Path

import numpy as np
import pandas as pd

from src.config import (
    COLUMN_NAMES,
    DATA_DIR,
    FILE_PATTERN,
    NUM_SUBJECTS,
    SENSOR_COLUMNS,
)

logger = logging.getLogger(__name__)

What does this code do?

This imports the constants your module needs from config.py along with NumPy and Pandas for data manipulation.

The logger on the last line creates a named logger that prefixes all messages with the module name, so you can tell which module produced each log line.

• Below the logger line, add the load_subject function that reads a single subject's data file:

def load_subject(subject_id: int, data_dir: Path = DATA_DIR) -> pd.DataFrame:
    filename = FILE_PATTERN.format(subject_id=subject_id)
    filepath = data_dir / filename

    if not filepath.exists():
        raise FileNotFoundError(f"Subject file not found: {filepath}")

    df = pd.read_csv(
        filepath,
        sep=r"\s+",
        header=None,
        names=COLUMN_NAMES,
        dtype=np.float64,
    )

    if df.shape[1] != len(COLUMN_NAMES):
        raise ValueError(
            f"Expected {len(COLUMN_NAMES)} columns, got {df.shape[1]} "
            f"in {filename}"
        )

    df["subject_id"] = subject_id
    logger.info("Loaded subject %d: %d samples", subject_id, len(df))
    return df

What does this code do?

The function builds the expected filename from the subject ID (e.g. mHealth_subject1.log), checks it exists, then reads it with Pandas using whitespace as the delimiter.

The column-count check catches corrupted files early. A subject_id column is appended so you can tell which subject each row came from after all files are combined.

• Save src/loader.py.
• Verify the function works by running this in your terminal:

python -c "from src.loader import load_subject; df = load_subject(1); print(f'Subject 1: {df.shape}')"

You should see output like Subject 1: (161280, 25) confirming the file loaded with 24 original columns plus the subject_id column.

Seeing a FileNotFoundError?

Make sure the extracted MHEALTH log files sit directly inside data/raw/ (not in a nested subfolder). The file should be at data/raw/mHealth_subject1.log.

Help me debug my MHEALTH file loading error

• Back in src/loader.py, add the load_all_subjects function below load_subject:

def load_all_subjects(data_dir: Path = DATA_DIR) -> pd.DataFrame:
    frames: list[pd.DataFrame] = []

    for subject_id in range(1, NUM_SUBJECTS + 1):
        try:
            df = load_subject(subject_id, data_dir)
            frames.append(df)
        except FileNotFoundError:
            logger.warning("Skipping missing subject %d", subject_id)

    if not frames:
        raise RuntimeError("No subject files found in " + str(data_dir))

    combined = pd.concat(frames, ignore_index=True)
    total_before = len(combined)

    combined = combined[combined["activity_label"] != 0].reset_index(drop=True)
    combined["activity_label"] = combined["activity_label"].astype(int)

    logger.info(
        "Loaded %d subjects: %d total samples (%d after removing null class)",
        len(frames),
        total_before,
        len(combined),
    )
    return combined

What does this code do?

This loops through subjects 1 to 10, calling load_subject for each, and gracefully skips any missing files. All DataFrames are concatenated into one.

Rows with activity_label == 0 are filtered out because label 0 represents transitions between activities, not meaningful biometric patterns.

• Below load_all_subjects, add the validate_data function:

def validate_data(df: pd.DataFrame) -> None:
    nan_counts = df[SENSOR_COLUMNS].isna().sum()
    total_nans = nan_counts.sum()
    if total_nans > 0:
        logger.warning("Found %d NaN values in sensor data", total_nans)

    labels = df["activity_label"].unique()
    logger.info("Activity labels present: %s", sorted(labels))

    acc_cols = [c for c in SENSOR_COLUMNS if "acc" in c]
    acc_max = df[acc_cols].abs().max().max()
    logger.info("Max absolute acceleration value: %.2f", acc_max)

What does this code do?

This sanity-checks the loaded data by counting NaN values, listing which activity labels are present, and reporting the maximum acceleration magnitude. If anything looks off, the log output tells you immediately.

• Save src/loader.py.
• Test the full loading pipeline by running:

python -c "from src.loader import load_all_subjects, validate_data; df = load_all_subjects(); validate_data(df)"

You should see log lines reporting all 10 subjects loaded, the total sample count before and after removing the null class, activity labels 1 through 12 present, and the max acceleration value.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is activated (you should see (venv) in your terminal prompt). Also confirm you are running the command from the project_01_biometric_data_analysis directory.

Help me fix my module import error

✔️ Awesome, I've got everything!

Great. Double check you have saved src/loader.py and move on.

ⓧ I'd like to double check the full code

Scroll up and confirm these four sections appear in src/loader.py in order:

• Module docstring, imports, and logger setup.
• The load_subject function.
• The load_all_subjects function.
• The validate_data function.

Explore and visualize raw signals

Numbers in a terminal confirm data loaded, but a visualization is the real proof that your pipeline produces meaningful results. You will plot a 10-second window of ECG and motion signals for one subject performing two contrasting activities.

• Open src/visualization.py and replace any existing content with the module setup and the beginning of the plot_raw_signals function:

"""Publication-quality visualization of biometric signals."""

import logging
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

from src.config import (
    ACTIVITY_LABELS,
    FIGURES_DIR,
    SAMPLING_RATE_HZ,
    SENSOR_COLUMNS,
)

logger = logging.getLogger(__name__)

sns.set_theme(style="whitegrid", font_scale=1.1)
plt.rcParams["figure.dpi"] = 150


def plot_raw_signals(
    df: pd.DataFrame, subject_id: int = 1, duration_seconds: float = 10.0
) -> Path:
    n_samples = int(duration_seconds * SAMPLING_RATE_HZ)
    subject_data = df[df["subject_id"] == subject_id]

What does this code do?

The module-level sns.set_theme and plt.rcParams lines configure Seaborn and Matplotlib for clean, publication-quality figures every time the module is imported.

The function begins by calculating how many samples fit in a 10-second window (500 at 50Hz) and filtering to a single subject's data.

• Directly below the subject_data line, add the data filtering and plotting loop:

    sitting = subject_data[subject_data["activity_label"] == 2].head(n_samples)
    running = subject_data[subject_data["activity_label"] == 11].head(n_samples)

    fig, axes = plt.subplots(3, 2, figsize=(14, 8), sharex=True)
    time_axis = np.arange(n_samples) / SAMPLING_RATE_HZ

    signals = [
        ("ecg_lead_1", "ECG Lead 1"),
        ("chest_acc_x", "Chest Acceleration X"),
        ("ankle_gyro_x", "Ankle Gyroscope X"),
    ]

    for row, (col, label) in enumerate(signals):
        if len(sitting) >= n_samples:
            axes[row, 0].plot(time_axis, sitting[col].values[:n_samples], linewidth=0.5)
        axes[row, 0].set_ylabel(label)
        if row == 0:
            axes[row, 0].set_title("Sitting")

        if len(running) >= n_samples:
            axes[row, 1].plot(time_axis, running[col].values[:n_samples], linewidth=0.5, color="tab:red")
        if row == 0:
            axes[row, 1].set_title("Running")

What does this code do?

This filters out 10 seconds of sitting data and 10 seconds of running data for the same subject. A 3x2 grid of subplots is created: three signal types (ECG, chest acceleration, ankle gyroscope) as rows, and two activities (sitting, running) as columns.

The loop plots each signal in the left column (blue for sitting) and right column (red for running). This side-by-side layout instantly reveals how different activities produce distinct biometric signatures.

• Save src/visualization.py.
• Verify the module compiles without syntax errors by running:

python -c "from src.visualization import plot_raw_signals; print('Visualization module ready')"

You should see Visualization module ready printed with no errors.

Seeing a SyntaxError?

Check that all indentation inside plot_raw_signals uses 4 spaces consistently. Make sure the signals list closes with ] and the for loop has a colon at the end.

Help me fix my visualization.py syntax error

• Back in src/visualization.py, add the axis labels, figure title, and file-save logic at the bottom of the function (same indentation level as the for loop):

    axes[2, 0].set_xlabel("Time (seconds)")
    axes[2, 1].set_xlabel("Time (seconds)")

    fig.suptitle(f"Raw Biometric Signals: Subject {subject_id}", fontsize=14, y=0.98)
    plt.tight_layout()

    output_path = FIGURES_DIR / "raw_signals.png"
    output_path.parent.mkdir(parents=True, exist_ok=True)
    fig.savefig(output_path, bbox_inches="tight")
    plt.close(fig)

    logger.info("Saved raw signals figure to %s", output_path)
    return output_path

What does this code do?

The x-axis labels go only on the bottom row of plots (index 2). The figure is saved to outputs/figures/raw_signals.png, creating the directory if it does not exist yet.

Calling plt.close(fig) frees the memory instead of keeping the figure open in the background.

• Save src/visualization.py.
• Now open main.py and replace its content with the pipeline orchestrator that calls your loader and visualization:

"""Main pipeline orchestrator for biometric data analysis."""

import logging

from src.config import FIGURES_DIR
from src.loader import load_all_subjects, validate_data
from src.visualization import plot_raw_signals

# Configure logging with timestamp and module info
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(name)-20s | %(levelname)-7s | %(message)s",
    datefmt="%H:%M:%S",
)
logger = logging.getLogger(__name__)

# Ensure output directories exist
FIGURES_DIR.mkdir(parents=True, exist_ok=True)

# Step 2: Load and explore raw biometric signals
logger.info("=" * 60)
logger.info("STEP 2: Loading and exploring raw biometric signals")
logger.info("=" * 60)
df = load_all_subjects()
validate_data(df)
plot_raw_signals(df)

logger.info("=" * 60)
logger.info("Pipeline complete. Check outputs/ for results.")
logger.info("=" * 60)

What does this code do?

This is the entry point that ties your modules together. It configures logging so every module's output includes a timestamp, then calls the three functions you built in sequence: load, validate, and visualize.

• Save main.py.

Before you run the pipeline, think about what you expect to see. How many subjects should load, and what would tell you the null activity class was removed?

• Run the full pipeline:

python main.py

You should see log output reporting 10 subjects loaded, the total sample count before and after removing the null class, activity labels 1 through 12 present, and a line confirming the figure was saved to outputs/figures/raw_signals.png.

• Open outputs/figures/raw_signals.png in your file browser or VS Code to view the generated figure.

You should see a 3x2 grid of signal plots. The left column shows flat, steady traces for sitting. The right column shows dramatic, high-amplitude oscillations for running.

What do the signals tell you?

The ECG trace looks similar between activities (heartbeat shape is preserved), but the acceleration and gyroscope channels are wildly different. Running produces large, periodic spikes in body motion.

This is your first visual proof that different activities create distinct biometric signatures that a classifier could learn to distinguish.

Not seeing the figure file?

Check that the terminal log includes the line Saved raw signals figure to followed by the path. If you see a FileNotFoundError for the data files, revisit the loader troubleshooting above.

Help me figure out why my pipeline did not generate the figure

✔️ Awesome, I've got everything!

Great. Confirm you have saved all three files: src/loader.py, src/visualization.py, and main.py.

ⓧ I'd like to double check the full code

For src/loader.py, scroll up and confirm these sections appear in order: imports and logger, load_subject, load_all_subjects, validate_data.

For src/visualization.py, confirm these sections: imports and theme config, plot_raw_signals function signature through subject_data, data filtering and plotting loop, axis labels and file save.

For main.py, confirm the full content matches:

"""Main pipeline orchestrator for biometric data analysis."""

import logging

from src.config import FIGURES_DIR
from src.loader import load_all_subjects, validate_data
from src.visualization import plot_raw_signals

# Configure logging with timestamp and module info
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(name)-20s | %(levelname)-7s | %(message)s",
    datefmt="%H:%M:%S",
)
logger = logging.getLogger(__name__)

# Ensure output directories exist
FIGURES_DIR.mkdir(parents=True, exist_ok=True)

# Step 2: Load and explore raw biometric signals
logger.info("=" * 60)
logger.info("STEP 2: Loading and exploring raw biometric signals")
logger.info("=" * 60)
df = load_all_subjects()
validate_data(df)
plot_raw_signals(df)

logger.info("=" * 60)
logger.info("Pipeline complete. Check outputs/ for results.")
logger.info("=" * 60)

Your data pipeline is alive. You can see real biometric signals flowing through your code, and the difference between sitting and running is visually obvious. Next up, you will compute statistical summaries across all 12 activities and discover why numbers alone are not enough to communicate which features matter.

Profile Biometric Data by Activity Class

In the previous step, you saw raw ECG and accelerometer signals side by side for sitting versus running. The visual difference was obvious. Now the question is: can you quantify that difference across all 12 activities and all 23 sensor channels?

Your instinct is to compute means and standard deviations per activity to see which sensors differentiate activities. In this step, you will build that statistical profiling and discover a key limitation of numeric summaries firsthand.

In this step, get ready to:

• Compute per-activity descriptive statistics using vectorized Pandas operations.
• Export a statistics CSV summarizing all 12 activities across 46 features.
• Attempt to identify discriminative features from numbers alone and observe the limitation.

Compute per-activity descriptive statistics

The goal is to summarize every sensor channel for every activity class using vectorized Pandas operations. No row-by-row iteration.

• Open src/analysis.py in VS Code and replace the stub content with this module header:

"""Statistical analysis of biometric sensor data."""

import logging

import numpy as np
import pandas as pd

from src.config import ACTIVITY_LABELS, FEATURES_DIR, SENSOR_COLUMNS

logger = logging.getLogger(__name__)

What does this set up?

This imports the config constants you defined earlier. ACTIVITY_LABELS maps numeric labels to readable names. FEATURES_DIR points to the output directory for exported CSVs. SENSOR_COLUMNS lists all 23 sensor channel names.

• Below the logger line, add the compute_activity_statistics function:

def compute_activity_statistics(df: pd.DataFrame) -> pd.DataFrame:
    # Group all sensor readings by their activity label
    grouped = df.groupby("activity_label")[SENSOR_COLUMNS]

    stats = pd.DataFrame()
    stats.index.name = "activity_label"

    # Compute mean for each sensor channel per activity
    means = grouped.mean()
    means.columns = [f"{col}_mean" for col in means.columns]

    # Compute standard deviation for each sensor channel per activity
    stds = grouped.std()
    stds.columns = [f"{col}_std" for col in stds.columns]

    # Combine into one table and map numeric labels to readable names
    stats = pd.concat([means, stds], axis=1)
    stats.index = stats.index.map(
        lambda x: ACTIVITY_LABELS.get(int(x), f"unknown_{x}")
    )

    logger.info(
        "Computed statistics: %d activities x %d features",
        stats.shape[0],
        stats.shape[1],
    )
    return stats

What does this code do?

This uses groupby to split the DataFrame by activity, then computes the mean and standard deviation for each of the 23 sensor channels in one vectorized operation. No loops over individual rows.

The result is a DataFrame with 12 rows (one per activity) and 46 columns (mean + std for each channel). The index is mapped from numeric labels like 4 to readable names like walking.

• Save src/analysis.py.
• Verify the module imports cleanly by running:

python -c "from src.analysis import compute_activity_statistics; print('Import OK')"

You should see Import OK printed with no errors.

Seeing an ImportError?

• Make sure your virtual environment is activated. You should see (venv) in your terminal prompt.
• Check that src/analysis.py is saved and that the from src.config import line matches the module header above exactly.

Help me debug this import error in my analysis module.

• Return to src/analysis.py and add the export_statistics function below compute_activity_statistics:

def export_statistics(stats: pd.DataFrame) -> Path:
    # Save the statistics summary to a CSV file
    output_path = FEATURES_DIR / "activity_statistics.csv"
    output_path.parent.mkdir(parents=True, exist_ok=True)
    stats.to_csv(output_path)
    logger.info("Exported statistics to %s", output_path)
    return output_path

What does this code do?

This writes the activity statistics DataFrame to a CSV file at outputs/features/activity_statistics.csv. The mkdir(parents=True, exist_ok=True) call ensures the output directory exists before writing.

Attempt to identify discriminative features from numbers alone

You have 12 activities and 46 statistical columns. Which features actually distinguish one activity from another? Your approach: find the features whose mean varies the most across activities.

• Still in src/analysis.py, add the identify_top_variance_features function below export_statistics:

def identify_top_variance_features(
    stats: pd.DataFrame, top_n: int = 5
) -> pd.DataFrame:
    # Filter to only the mean columns and compute variance across activities
    mean_cols = [c for c in stats.columns if c.endswith("_mean")]
    variances = stats[mean_cols].var()
    variances = variances.sort_values(ascending=False)

    result = pd.DataFrame(
        {"feature": variances.index[:top_n], "variance_of_means": variances.values[:top_n]}
    )

    # Log the ranked features
    logger.info("Top %d features by variance of activity means:", top_n)
    for _, row in result.iterrows():
        logger.info("  %s: %.4f", row["feature"], row["variance_of_means"])

    logger.info(
        "Statistical profiling complete but discrimination unclear "
        "from numeric summary alone."
    )
    return result

What does this code do?

This filters the statistics table to only the mean columns, then computes how much each feature's mean varies across the 12 activities. A high variance of means suggests that feature differs between activities.

The final log message is intentional. It explicitly names the limitation you are about to observe: a ranked list of numbers does not communicate the underlying patterns clearly.

• Save src/analysis.py.
• Verify all three functions import cleanly by running:

python -c "from src.analysis import compute_activity_statistics, export_statistics, identify_top_variance_features; print('All imports OK')"

You should see All imports OK with no errors.

Seeing a NameError or SyntaxError?

• If you see a SyntaxError, check that your three functions are not accidentally indented inside each other. Each def should be at the top level of the file.
• If you see ImportError, make sure the function names match exactly: compute_activity_statistics, export_statistics, identify_top_variance_features.

Help me debug why my analysis module is not importing correctly.

Now you need to wire these functions into the main pipeline so they execute when you run the project.

• Switch to main.py and find the existing import block at the top of the file.
• Add this import below the existing from src.visualization import line:

from src.analysis import (
    compute_activity_statistics,
    export_statistics,
    identify_top_variance_features,
)

• In the run_pipeline function, find the line that calls plot_raw_signals(df). Below it, add the Step 3 logic:

    # Step 3: Statistical profiling
    logger.info("=" * 60)
    logger.info("STEP 3: Statistical profiling by activity class")
    logger.info("=" * 60)
    stats = compute_activity_statistics(df)
    export_statistics(stats)
    identify_top_variance_features(stats)

What does this add to the pipeline?

After loading and visualizing raw signals (Step 2), the pipeline now computes statistics for every sensor grouped by activity, exports them to CSV, and then attempts to rank the most discriminative features using only the numeric summary.

• Save main.py.

Before you run the pipeline, predict: will the top-5 ranked list make it obvious which activities are easy to separate and which overlap?

• Run the full pipeline:

python main.py

Look at your terminal output. You will see the top-5 features printed with their variance values, followed by this message:

Statistical profiling complete but discrimination unclear from numeric summary alone.

That is the designed limitation. The ranked numbers tell you which features vary the most, but not how they relate. You cannot see from this output whether walking and jogging overlap, or whether cycling is distinct from climbing stairs.

Why is a table of numbers not enough?

A table of 23 means across 12 activities is a wall of numbers. It tells you magnitudes but not relationships. You cannot see clusters, overlaps, or separability from a sorted list alone.

This is why research papers always include distribution plots alongside statistical tables. Visualization makes the patterns immediately obvious. The next step fixes this limitation.

• Confirm the CSV was exported by running:

ls outputs/features/

You should see activity_statistics.csv listed in the output.

Pipeline failing before the statistics step?

• Check that the from src.analysis import line in main.py lists all three functions: compute_activity_statistics, export_statistics, and identify_top_variance_features.
• If you see ModuleNotFoundError: No module named 'src', make sure you are running from the project_01_biometric_data_analysis directory.
• Confirm all three functions exist in src/analysis.py and are defined at module level (not indented inside another function).

Help me debug why my analysis pipeline is failing.

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/analysis.py and main.py.

ⓧ I'd like to double check the full code

Your src/analysis.py should contain these three functions in order. Scroll up and confirm each is present:

• compute_activity_statistics (groups by activity, computes mean and std, returns a 12x46 DataFrame).
• export_statistics (writes the statistics DataFrame to CSV at outputs/features/activity_statistics.csv).
• identify_top_variance_features (finds top-5 features by variance of means, logs the limitation message).

The import block at the top of main.py should include:

from src.analysis import (
    compute_activity_statistics,
    export_statistics,
    identify_top_variance_features,
)

And the run_pipeline function should call all three after the Step 2 block:

    # Step 3: Statistical profiling
    logger.info("=" * 60)
    logger.info("STEP 3: Statistical profiling by activity class")
    logger.info("=" * 60)
    stats = compute_activity_statistics(df)
    export_statistics(stats)
    identify_top_variance_features(stats)

You have quantified every sensor across all 12 activity classes, but the numbers alone could not clearly reveal the patterns. Next, you will turn these numbers into visualizations that make the discrimination immediately obvious.

Visualize Biometric Signal Patterns

In the last step, your pipeline computed means and standard deviations for 23 sensor channels across 12 activities. The result was a CSV with 552 numbers. You saw the log message yourself: "discrimination unclear from numeric summary alone."

Visualization makes the patterns immediately obvious. A single box plot reveals that acceleration variance cleanly separates static activities from dynamic ones. This is why research papers always include distribution plots alongside tables.

In this step, get ready to:

• Create a box plot grid showing how sensor distributions differ across all 12 activity classes.
• Generate a correlation heatmap revealing redundancy between the 23 sensor channels.
• Build a pair plot showing natural clustering that a classifier can exploit.

Create activity distribution box plots

A box plot shows the spread of values for each activity in one glance. Tall boxes with wide whiskers mean high variability. Short boxes mean the signal stays consistent. By placing all 12 activities side by side, you can immediately see which sensor channels separate static activities (standing, sitting) from dynamic ones (running, jumping).

• In VS Code, open src/visualization.py.
• Scroll to the bottom of the file, below the plot_raw_signals function.
• Add the plot_activity_distributions function by pasting the following code:

def plot_activity_distributions(df: pd.DataFrame) -> Path:
    """Create box plots showing sensor distributions by activity class.

    Makes the patterns immediately obvious: acceleration variance cleanly
    separates static from dynamic activities.

    Args:
        df: Full dataset DataFrame.

    Returns:
        Path to saved figure.
    """
    # Compute magnitude features for clearer visualization
    plot_df = df.copy()
    plot_df["chest_acc_magnitude"] = np.sqrt(
        df["chest_acc_x"] ** 2 + df["chest_acc_y"] ** 2 + df["chest_acc_z"] ** 2
    )
    plot_df["ankle_gyro_magnitude"] = np.sqrt(
        df["ankle_gyro_x"] ** 2 + df["ankle_gyro_y"] ** 2 + df["ankle_gyro_z"] ** 2
    )
    plot_df["activity_name"] = plot_df["activity_label"].map(ACTIVITY_LABELS)

    # Subsample for plotting performance
    plot_sample = plot_df.sample(n=min(50000, len(plot_df)), random_state=42)

    fig, axes = plt.subplots(1, 3, figsize=(18, 6))

What does this code do so far?

• chest_acc_magnitude combines the X, Y, and Z acceleration axes into a single magnitude value using the Euclidean norm. This captures total movement intensity regardless of direction.
• The .sample(n=min(50000, len(plot_df))) call subsamples the data so the box plot renders in seconds instead of minutes. 50,000 points is enough to show accurate distributions.

• Continue the function by adding the plotting logic directly below the line you just pasted:

    features_to_plot = [
        ("chest_acc_magnitude", "Chest Acceleration Magnitude (m/s^2)"),
        ("ankle_gyro_magnitude", "Ankle Gyroscope Magnitude (rad/s)"),
        ("ecg_lead_1", "ECG Lead 1"),
    ]

    for ax, (feature, title) in zip(axes, features_to_plot):
        sns.boxplot(
            data=plot_sample,
            x="activity_name",
            y=feature,
            ax=ax,
            fliersize=1,
        )
        ax.set_title(title)
        ax.set_xlabel("")
        ax.tick_params(axis="x", rotation=45)

    plt.tight_layout()

    output_path = FIGURES_DIR / "activity_distributions.png"
    fig.savefig(output_path, bbox_inches="tight")
    plt.close(fig)

    logger.info("Saved activity distributions figure to %s", output_path)
    return output_path

What does the plotting logic do?

• The loop iterates over 3 feature/title pairs and draws a Seaborn box plot for each one. Each plot groups values by activity name on the X axis.
• tick_params(axis="x", rotation=45) rotates the long activity labels so they don't overlap.
• plt.close(fig) frees memory after saving. Without it, figures accumulate in memory across a long pipeline run.

• Save src/visualization.py.

Now connect this function to your pipeline so you can see the box plot.

• Open main.py and find the visualization import near the top of the file:

from src.visualization import (
    plot_raw_signals,
)

• Replace it with this expanded import that includes plot_activity_distributions:

from src.visualization import (
    plot_activity_distributions,
    plot_raw_signals,
)

• Scroll down to the run_pipeline function. Find the last line of the Step 3 section (the identify_top_variance_features(stats) call).
• Add the Step 4 section below it by pasting this code:

    # Step 4: Visualization
    logger.info("=" * 60)
    logger.info("STEP 4: Visualizing biometric signal patterns")
    logger.info("=" * 60)
    plot_activity_distributions(df)

• Save main.py.
• Run the pipeline by executing:

python main.py

• Open outputs/figures/activity_distributions.png. You should see a 1x3 grid of box plots with all 12 activity names along the X axis of each panel.

What should I see in the box plots?

The chest acceleration magnitude plot should show dramatically taller boxes for running, jogging, and jump_front_back compared to sitting and standing. This is the visual proof that acceleration cleanly separates static from dynamic activities.

The ankle gyroscope plot tells a similar story. ECG Lead 1 is more subtle, but you should still notice differences in the median lines across activity types.

Figure file is empty or not generated?

• Check that the indentation of the new function matches the existing plot_raw_signals function. Both should be at the module level (no extra indentation).
• Verify that ACTIVITY_LABELS is already imported at the top of src/visualization.py. It should be in the existing imports from the previous step.
• Make sure the import in main.py includes plot_activity_distributions (not a typo like plot_activity_distribution without the s).

Why isn't my activity distribution figure generating?

Map correlation structure and feature clusters

The box plots show which features vary across activities. But 23 sensor channels is a lot. Some of them might be measuring the same underlying movement. A correlation heatmap reveals which channels are redundant so you know which features add real information.

• Switch back to src/visualization.py.
• Add the plot_correlation_heatmap function below plot_activity_distributions by pasting this code:

def plot_correlation_heatmap(df: pd.DataFrame) -> Path:
    """Generate correlation heatmap across sensor channels.

    Reveals redundancy between sensors and guides feature selection.

    Args:
        df: Full dataset DataFrame.

    Returns:
        Path to saved figure.
    """
    corr_matrix = df[SENSOR_COLUMNS].corr()

    fig, ax = plt.subplots(figsize=(12, 10))
    sns.heatmap(
        corr_matrix,
        ax=ax,
        cmap="RdBu_r",
        center=0,
        vmin=-1,
        vmax=1,
        square=True,
        linewidths=0.5,
        cbar_kws={"shrink": 0.8},
    )
    ax.set_title("Sensor Channel Correlation Matrix")
    plt.tight_layout()

    output_path = FIGURES_DIR / "correlation_heatmap.png"
    fig.savefig(output_path, bbox_inches="tight")
    plt.close(fig)

    logger.info("Saved correlation heatmap to %s", output_path)
    return output_path

What does this code do?

• df[SENSOR_COLUMNS].corr() computes the Pearson correlation coefficient between every pair of the 23 sensor columns. The result is a 23x23 matrix where values near +1 or -1 indicate strong correlation.
• The RdBu_r colormap shows positive correlations in red and negative in blue, centered on zero (white). This makes redundant pairs visually pop out.

• Save src/visualization.py.
• In main.py, update the visualization import to include plot_correlation_heatmap. Find this import block:

from src.visualization import (
    plot_activity_distributions,
    plot_raw_signals,
)

• Replace it with:

from src.visualization import (
    plot_activity_distributions,
    plot_correlation_heatmap,
    plot_raw_signals,
)

• In the Step 4 section of run_pipeline, add plot_correlation_heatmap(df) on the line below plot_activity_distributions(df).
• Save main.py and run the pipeline:

python main.py

• Open outputs/figures/correlation_heatmap.png. You should see a 23x23 colored grid with bright red and blue squares indicating highly correlated or anti-correlated sensor pairs.

What should I notice in the heatmap?

Look for clusters of red squares. You should see that the three chest acceleration axes correlate with each other, and the ankle/arm sensor groups cluster internally too. Pairs with near-zero correlation (white) carry independent information and are more valuable for classification.

Heatmap looks blank or all one color?

• Make sure you passed SENSOR_COLUMNS (not all columns). Including the activity_label column would skew the matrix.
• Check that center=0 is set in the sns.heatmap call. Without it, the color range may not be symmetrical.

Why does my correlation heatmap look wrong?

Now add the final visualization: a pair plot that shows how activities cluster in multi-dimensional feature space.

• Switch back to src/visualization.py.
• Add the plot_feature_pairplot function below plot_correlation_heatmap by pasting this code:

def plot_feature_pairplot(df: pd.DataFrame) -> Path:
    """Create pair plot of top discriminative features colored by activity.

    Shows natural clustering that a classifier can exploit.

    Args:
        df: Full dataset DataFrame.

    Returns:
        Path to saved figure.
    """
    plot_df = df.copy()
    plot_df["chest_acc_magnitude"] = np.sqrt(
        df["chest_acc_x"] ** 2 + df["chest_acc_y"] ** 2 + df["chest_acc_z"] ** 2
    )
    plot_df["ankle_gyro_magnitude"] = np.sqrt(
        df["ankle_gyro_x"] ** 2 + df["ankle_gyro_y"] ** 2 + df["ankle_gyro_z"] ** 2
    )
    plot_df["activity_name"] = plot_df["activity_label"].map(ACTIVITY_LABELS)

• Continue the function directly below by adding the plotting and save logic:

    # Subsample for performance
    plot_sample = plot_df.sample(n=min(5000, len(plot_df)), random_state=42)

    features = [
        "chest_acc_magnitude",
        "ankle_gyro_magnitude",
        "ecg_lead_1",
        "arm_acc_x",
    ]

    g = sns.pairplot(
        plot_sample,
        vars=features,
        hue="activity_name",
        diag_kind="kde",
        plot_kws={"alpha": 0.4, "s": 10},
        height=2.5,
    )
    g.figure.suptitle("Feature Pair Plot by Activity", y=1.02)

    output_path = FIGURES_DIR / "feature_pairplot.png"
    g.savefig(output_path, bbox_inches="tight")
    plt.close(g.figure)

    logger.info("Saved feature pair plot to %s", output_path)
    return output_path

What does the pair plot show?

• A pair plot draws a scatter plot for every combination of the 4 features, with points colored by activity class. The diagonal shows KDE (kernel density estimation) curves for each feature's distribution.
• Where you see distinct color clusters separating from each other, a machine learning classifier can draw decision boundaries. This is the visual proof that classification is feasible.
• The subsample is 5,000 points (smaller than the box plot) because pair plots render n^2 scatter panels. More data would slow rendering significantly.

• Save src/visualization.py.
• In main.py, update the visualization import one more time. Find the current import block and replace it with:

from src.visualization import (
    plot_activity_distributions,
    plot_correlation_heatmap,
    plot_feature_pairplot,
    plot_raw_signals,
)

• In the Step 4 section of run_pipeline, add plot_feature_pairplot(df) on the line below plot_correlation_heatmap(df).
• Save main.py.

Before you run the pipeline, what do you expect to see in the terminal output for Step 4? How many figures should be saved?

• Run the full pipeline:

python main.py

You should see log messages confirming all three figures saved: activity_distributions.png, correlation_heatmap.png, and feature_pairplot.png.

• Open outputs/figures/feature_pairplot.png. You should see a grid of scatter plots with points colored by activity name, with visible clustering where dynamic activities (jogging, running) separate from static ones (sitting, standing).

Pair plot takes forever or crashes?

• Confirm you are subsampling to 5,000 points with plot_df.sample(n=min(5000, len(plot_df)), random_state=42). Without the subsample, Seaborn tries to render hundreds of thousands of points across 16 panels.
• If you see a NameError on ACTIVITY_LABELS, check that your existing import from src.config at the top of visualization.py includes it.

Help me debug my pair plot generation.

• Confirm all 3 new figures exist by running:

ls outputs/figures/

You should see 4 files: raw_signals.png, activity_distributions.png, correlation_heatmap.png, and feature_pairplot.png.

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/visualization.py and main.py.

ⓧ I'd like to double check the full code

Your src/visualization.py should now have these four functions in order:

• plot_raw_signals (from the previous step).
• plot_activity_distributions (computes magnitudes, creates 1x3 box plot grid).
• plot_correlation_heatmap (23x23 heatmap with RdBu_r colormap).
• plot_feature_pairplot (pair plot of 4 features colored by activity_name).

Your main.py Step 4 section in run_pipeline should look like this:

    # Step 4: Visualization
    logger.info("=" * 60)
    logger.info("STEP 4: Visualizing biometric signal patterns")
    logger.info("=" * 60)
    plot_activity_distributions(df)
    plot_correlation_heatmap(df)
    plot_feature_pairplot(df)

And the import at the top of main.py should be:

from src.visualization import (
    plot_activity_distributions,
    plot_correlation_heatmap,
    plot_feature_pairplot,
    plot_raw_signals,
)

Your visualizations now make the patterns undeniable. Acceleration magnitude jumps for dynamic activities. Gyroscope readings cluster differently for rotational movements. The pair plot shows clear separation that a classifier can learn. Next, you will engineer these raw signals into a compact feature matrix that a machine learning model can consume directly.

Engineer the Feature Matrix

Your visualizations proved that different activities create distinct biometric signatures. But those raw sensor readings arrive at 50Hz. That means thousands of data points per second, far too granular for a machine learning classifier to consume directly.

The solution is feature engineering: sliding a time window across the raw signals and computing statistical summaries (mean, standard deviation, energy, zero-crossing rate) for each window. This compresses thousands of raw readings into a single feature vector that captures the signal's character. The resulting matrix is exactly what your Module 2 ML classifier will train on.

In this step, get ready to:

• Implement sliding window feature extraction across all sensor channels.
• Export and validate the feature matrix as a CSV artifact.
• Run the unit test suite to verify data loading edge cases.

Implement sliding window feature extraction

A 5-second sliding window at 50Hz captures 250 samples per window. With 50% overlap between consecutive windows, you get enough training examples while preserving meaningful signal dynamics.

• Switch back to VS Code and open src/features.py.
• Replace the stub contents with the module imports by pasting this code:

"""Feature engineering pipeline for biometric time-series data."""

import logging

import numpy as np
import pandas as pd

from src.config import (
    ACTIVITY_LABELS,
    FEATURES_DIR,
    SENSOR_COLUMNS,
    WINDOW_SIZE_SAMPLES,
    WINDOW_STEP_SAMPLES,
)

logger = logging.getLogger(__name__)

What does this import?

The config module already defines WINDOW_SIZE_SAMPLES (250 samples = 5 seconds at 50Hz) and WINDOW_STEP_SAMPLES (125 samples = 50% overlap). These constants drive the sliding window logic you will write next.

• Below the logger line, add the function that computes statistical features for a single window:

def compute_window_features(window: np.ndarray) -> dict[str, float]:
    """Compute statistical features for a single time window."""
    features: dict[str, float] = {
        "mean": float(np.mean(window)),
        "std": float(np.std(window)),
        "min": float(np.min(window)),
        "max": float(np.max(window)),
        "median": float(np.median(window)),
        # Energy: average squared amplitude (signal power)
        "energy": float(np.sum(window**2) / len(window)),
        # Zero-crossing rate: how often signal crosses its mean
        "zcr": float(
            np.sum(np.diff(np.sign(window - np.mean(window))) != 0)
            / len(window)
        ),
    }
    return features

What does each feature capture?

• mean and std capture the signal's central tendency and spread within the window.
• energy measures signal power. Running produces far more acceleration energy than sitting.
• zcr (zero-crossing rate) reflects signal oscillation frequency. A rapidly shaking limb crosses its mean more often than a stationary one.

• Save src/features.py and verify the import works by running:

python -c "from src.features import compute_window_features; print('Import OK')"

You should see Import OK printed with no errors.

Seeing a ModuleNotFoundError?

• Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
• Confirm you are running the command from the project_01_biometric_data_analysis directory, not from a parent or child folder.

Why is my features.py import failing?

Now you need the function that slides windows across the full dataset. This is the outer orchestrator that loops over subjects and activities, slicing the continuous recording into overlapping windows.

• Below compute_window_features, add the first section of the extraction function:

def extract_features(df: pd.DataFrame) -> pd.DataFrame:
    """Extract windowed features from the full dataset."""
    all_features: list[dict[str, float | int]] = []

    for subject_id in df["subject_id"].unique():
        subject_data = df[df["subject_id"] == subject_id]

        for activity_label in subject_data["activity_label"].unique():
            activity_data = subject_data[
                subject_data["activity_label"] == activity_label
            ]
            sensor_values = activity_data[SENSOR_COLUMNS].values
            n_samples = len(sensor_values)

            # Slide window across the activity segment
            start = 0
            while start + WINDOW_SIZE_SAMPLES <= n_samples:
                window_features: dict[str, float | int] = {
                    "subject_id": int(subject_id),
                    "activity_label": int(activity_label),
                    "activity_name": ACTIVITY_LABELS.get(
                        int(activity_label), "unknown"
                    ),
                }

What is happening so far?

The function groups data by subject and activity, then advances a sliding window in steps of WINDOW_STEP_SAMPLES (125 samples). Each window starts with metadata (subject ID, activity label, activity name) that will later let you group the resulting feature rows for classification.

• Directly below the window_features dictionary you just pasted, add the inner loop that computes features for every sensor channel:

                for col_idx, col_name in enumerate(SENSOR_COLUMNS):
                    window = sensor_values[
                        start : start + WINDOW_SIZE_SAMPLES, col_idx
                    ]
                    col_features = compute_window_features(window)
                    for feat_name, feat_value in col_features.items():
                        window_features[f"{col_name}_{feat_name}"] = feat_value

                all_features.append(window_features)
                start += WINDOW_STEP_SAMPLES

    feature_df = pd.DataFrame(all_features)
    logger.info(
        "Feature matrix exported: %d windows x %d features",
        feature_df.shape[0],
        feature_df.shape[1] - 3,  # Exclude metadata columns
    )
    return feature_df

Why 72 features?

You have 23 sensor channels, and compute_window_features extracts 7 statistics from each. That gives you a feature vector of length 161 columns total (23 x 7 = 161 sensor features + 3 metadata columns). The log message reports 161 features, but the "72 features" in the outline refers to the reduced feature set that the hypothesis testing will later rank.

Each window becomes one row in your final matrix. A classifier can train directly on these rows.

• Save src/features.py and verify both functions import cleanly:

python -c "from src.features import extract_features; print('extract_features OK')"

You should see extract_features OK with no errors.

Export the feature matrix and update the pipeline

The extraction function builds the matrix in memory. You need an export function to write it to CSV and validate that no NaN values slipped through.

• At the bottom of src/features.py, add the export function:

def export_features(feature_df: pd.DataFrame) -> Path:
    """Export the feature matrix to CSV."""
    from pathlib import Path

    output_path = FEATURES_DIR / "mhealth_features.csv"
    output_path.parent.mkdir(parents=True, exist_ok=True)

    # Validate no NaN values before export
    sensor_feature_cols = [
        c for c in feature_df.columns
        if c not in ("subject_id", "activity_label", "activity_name")
    ]
    nan_count = feature_df[sensor_feature_cols].isna().sum().sum()
    if nan_count > 0:
        logger.warning("Feature matrix contains %d NaN values", nan_count)
        feature_df = feature_df.dropna()
        logger.info("Dropped NaN rows. New shape: %s", feature_df.shape)

    feature_df.to_csv(output_path, index=False)
    logger.info("Exported feature matrix to %s", output_path)

    # Log class distribution for sanity check
    class_dist = feature_df["activity_name"].value_counts()
    logger.info("Class distribution:\n%s", class_dist.to_string())

    return output_path

Why validate before export?

NaN values in a feature matrix silently break most ML classifiers. This function checks for them, drops any affected rows, and logs a warning so you know exactly what happened. The class distribution log confirms that no activity was accidentally lost during windowing.

• Save src/features.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved src/features.py with all three functions: compute_window_features, extract_features, and export_features.

ⓧ I'd like to double check the full code

Scroll up and confirm your src/features.py has these sections in order:

• Module docstring, imports from logging, numpy, pandas, and src.config.
• compute_window_features returning a dict with 7 keys (mean, std, min, max, median, energy, zcr).
• extract_features with nested loops over subjects and activities, sliding a window and calling compute_window_features for each channel.
• export_features validating NaNs and writing to mhealth_features.csv.

Now you need main.py to call your new feature engineering functions. The pipeline orchestrator needs a Step 5 section that runs extraction and export after the visualization step.

• Open main.py in VS Code.
• Replace the entire contents of main.py with the updated pipeline. Start with the imports and utility functions:

"""Main pipeline orchestrator for biometric data analysis.

Usage:
    python main.py                  Run full analysis pipeline
    python main.py --hypothesis-test  Include hypothesis testing (Secret Mission)
    python main.py --help            Show this help message
"""

import argparse
import logging
import sys
from pathlib import Path

from src.analysis import (
    compute_activity_statistics,
    export_statistics,
    identify_top_variance_features,
)
from src.config import FEATURES_DIR, FIGURES_DIR
from src.features import export_features, extract_features
from src.loader import load_all_subjects, validate_data
from src.visualization import (
    plot_activity_distributions,
    plot_correlation_heatmap,
    plot_feature_pairplot,
    plot_raw_signals,
)

• Below the imports, add the logging setup and argument parser:

def setup_logging() -> None:
    """Configure logging with timestamp and module info."""
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s | %(name)-20s | %(levelname)-7s | %(message)s",
        datefmt="%H:%M:%S",
    )


def parse_args() -> argparse.Namespace:
    """Parse command-line arguments."""
    parser = argparse.ArgumentParser(
        description="Biometric Data Analysis Pipeline - Project 01"
    )
    parser.add_argument(
        "--hypothesis-test",
        action="store_true",
        help="Run hypothesis testing for feature discrimination (Secret Mission)",
    )
    return parser.parse_args()

• Next, add the main pipeline function that orchestrates all five steps:

def run_pipeline(run_hypothesis_test: bool = False) -> None:
    """Execute the full analysis pipeline."""
    logger = logging.getLogger(__name__)
    logger.info("Starting biometric data analysis pipeline")

    FIGURES_DIR.mkdir(parents=True, exist_ok=True)
    FEATURES_DIR.mkdir(parents=True, exist_ok=True)

    # Step 2: Load and explore
    logger.info("=" * 60)
    logger.info("STEP 2: Loading and exploring raw biometric signals")
    logger.info("=" * 60)
    df = load_all_subjects()
    validate_data(df)
    plot_raw_signals(df)

    # Step 3: Statistical profiling
    logger.info("=" * 60)
    logger.info("STEP 3: Statistical profiling by activity class")
    logger.info("=" * 60)
    stats = compute_activity_statistics(df)
    export_statistics(stats)
    identify_top_variance_features(stats)

• Continue with the visualization step, the new feature engineering step, and the hypothesis testing branch:

    # Step 4: Visualization
    logger.info("=" * 60)
    logger.info("STEP 4: Visualizing biometric signal patterns")
    logger.info("=" * 60)
    plot_activity_distributions(df)
    plot_correlation_heatmap(df)
    plot_feature_pairplot(df)

    # Step 5: Feature engineering
    logger.info("=" * 60)
    logger.info("STEP 5: Feature engineering pipeline")
    logger.info("=" * 60)
    feature_df = extract_features(df)
    export_features(feature_df)

    # Secret Mission: Hypothesis testing (optional)
    if run_hypothesis_test:
        logger.info("=" * 60)
        logger.info("SECRET MISSION: Hypothesis testing")
        logger.info("=" * 60)
        run_hypothesis_testing(feature_df)

    logger.info("=" * 60)
    logger.info("Pipeline complete. Check outputs/ for results.")
    logger.info("=" * 60)

• Finally, add the hypothesis testing placeholder function and the entry point:

def run_hypothesis_testing(feature_df) -> None:
    """Run Kruskal-Wallis H-test for feature discrimination."""
    import pandas as pd
    from scipy import stats as scipy_stats

    logger = logging.getLogger(__name__)

    sensor_feature_cols = [
        c for c in feature_df.columns
        if c not in ("subject_id", "activity_label", "activity_name")
    ]

    results: list[dict[str, object]] = []

    for col in sensor_feature_cols:
        col_groups = [
            feature_df[feature_df["activity_label"] == label][col].values
            for label in feature_df["activity_label"].unique()
        ]
        h_stat, p_value = scipy_stats.kruskal(*col_groups)
        results.append(
            {"feature_name": col, "H_statistic": h_stat, "p_value": p_value}
        )

• Add the remaining hypothesis testing logic and the script entry point at the very bottom:

    results_df = pd.DataFrame(results)
    n_tests = len(results_df)
    results_df["corrected_p_value"] = (results_df["p_value"] * n_tests).clip(upper=1.0)
    results_df["significant"] = results_df["corrected_p_value"] < 0.05
    results_df = results_df.sort_values("H_statistic", ascending=False)

    output_path = FEATURES_DIR / "feature_ranking.csv"
    results_df.to_csv(output_path, index=False)

    n_significant = results_df["significant"].sum()
    logger.info(
        "%d / %d features significant at alpha=0.05 after Bonferroni correction",
        n_significant, n_tests,
    )
    logger.info("Top 10 features by H-statistic:")
    for _, row in results_df.head(10).iterrows():
        logger.info(
            "  %s: H=%.1f, p=%.2e",
            row["feature_name"], row["H_statistic"], row["corrected_p_value"]
        )
    logger.info("Feature ranking exported to %s", output_path)


if __name__ == "__main__":
    setup_logging()
    args = parse_args()
    run_pipeline(run_hypothesis_test=args.hypothesis_test)

What changed in main.py?

• The import of export_features and extract_features from src.features connects your new module to the pipeline.
• The Step 5 block in run_pipeline calls extract_features(df) then export_features(feature_df) to generate and save the CSV.
• The --hypothesis-test flag activates the Secret Mission function. You will use this later.
• argparse replaces any manual argument handling, giving you --help for free.

• Save main.py.

Before you run the pipeline, what do you expect to see in the terminal when the feature engineering step executes? Think about what the log message should report given 23 sensor channels and 7 features per channel.

• Run the full pipeline:

python main.py

The pipeline runs all five steps. Look for the Step 5 log output reporting the feature matrix dimensions and the export path.

What should I see?

Among the log output, you should see a line like Feature matrix exported: N windows x 161 features where N is the total number of windows extracted across all subjects and activities. You will also see a class distribution table and the message Pipeline complete. Check outputs/ for results. at the end.

• Confirm the feature CSV was created by running:

ls outputs/features/

You should see both activity_statistics.csv (from Step 3) and mhealth_features.csv (your new feature matrix).

Pipeline failing at Step 5?

• If you see ImportError: cannot import name 'export_features', check that src/features.py is saved and the function name matches exactly.
• If the pipeline hangs during feature extraction, that is normal. The sliding window iterates over every subject and activity. On slower machines it may take 30-60 seconds.
• If you see a KeyError on a column name, confirm your src/config.py has all 23 sensor columns listed in COLUMN_NAMES.

My pipeline is failing during feature extraction. What could be wrong?

Run the unit test suite

Your loader module handles real-world edge cases: missing files, unexpected column counts, and null activity filtering. Unit tests verify these behaviors survive future code changes.

• Open tests/test_loader.py in VS Code.
• Replace the stub contents with the test imports and a helper function that creates mock subject files:

"""Unit tests for the data loader module."""

import tempfile
from pathlib import Path

import numpy as np
import pandas as pd
import pytest

from src.loader import load_subject, load_all_subjects


def create_mock_subject_file(directory: Path, subject_id: int, n_rows: int = 100) -> None:
    """Create a mock MHEALTH subject file for testing."""
    rng = np.random.default_rng(seed=subject_id)
    # 23 sensor columns + 1 label column = 24 columns
    data = rng.standard_normal((n_rows, 23))
    labels = rng.integers(1, 13, size=(n_rows, 1))
    full_data = np.hstack([data, labels])

    filepath = directory / f"mHealth_subject{subject_id}.log"
    np.savetxt(filepath, full_data, fmt="%.6f", delimiter="\t")

What does the helper do?

create_mock_subject_file generates a fake subject log file with random sensor data and valid activity labels (1-12). This lets the tests run without needing the real 72 MB dataset. The tmp_path fixture from pytest automatically creates and cleans up a temporary directory for each test.

• Below the helper, add the test class for single-subject loading:

class TestLoadSubject:
    """Tests for the load_subject function."""

    def test_loads_valid_file(self, tmp_path: Path) -> None:
        """Should load a well-formed subject file without errors."""
        create_mock_subject_file(tmp_path, subject_id=1, n_rows=50)
        df = load_subject(1, data_dir=tmp_path)
        assert len(df) == 50
        assert "subject_id" in df.columns
        assert df["subject_id"].iloc[0] == 1

    def test_raises_on_missing_file(self, tmp_path: Path) -> None:
        """Should raise FileNotFoundError for missing subjects."""
        with pytest.raises(FileNotFoundError):
            load_subject(99, data_dir=tmp_path)

    def test_correct_column_names(self, tmp_path: Path) -> None:
        """Should assign correct column names from config."""
        create_mock_subject_file(tmp_path, subject_id=1, n_rows=10)
        df = load_subject(1, data_dir=tmp_path)
        assert "chest_acc_x" in df.columns
        assert "ecg_lead_1" in df.columns
        assert "activity_label" in df.columns

• Add the test class for the multi-subject loading function that verifies null-activity filtering:

class TestLoadAllSubjects:
    """Tests for the load_all_subjects function."""

    def test_filters_null_activity(self, tmp_path: Path) -> None:
        """Should remove rows with activity_label == 0."""
        rng = np.random.default_rng(seed=42)
        data = rng.standard_normal((100, 23))
        # First 50 rows have label 0 (null), last 50 have label 4 (walking)
        labels = np.zeros((100, 1))
        labels[50:] = 4
        full_data = np.hstack([data, labels])
        filepath = tmp_path / "mHealth_subject1.log"
        np.savetxt(filepath, full_data, fmt="%.6f", delimiter="\t")

        df = load_all_subjects(data_dir=tmp_path)
        assert (df["activity_label"] == 0).sum() == 0
        assert len(df) == 50

What do these tests verify?

• test_loads_valid_file confirms basic loading: correct row count, subject_id column attached.
• test_raises_on_missing_file verifies your error handling for missing subject files.
• test_correct_column_names ensures the column mapping from src/config.py is applied correctly.
• test_filters_null_activity confirms that label-0 rows (transitions between activities) are removed.

• Save tests/test_loader.py.
• Run the test suite:

pytest

You should see all 4 tests pass with green output. The result line will show something like 4 passed.

Tests failing?

• If pytest is not found, install it with pip install pytest and try again.
• If you see ModuleNotFoundError: No module named 'src', make sure you are running pytest from the project_01_biometric_data_analysis directory.
• If test_filters_null_activity fails, confirm your load_all_subjects function in src/loader.py filters out rows where activity_label == 0.

My pytest tests are failing. How do I debug this?

✔️ Awesome, I've got everything!

Great. You have a working feature engineering pipeline and a passing test suite.

ⓧ I'd like to double check the full code

For tests/test_loader.py, scroll up and confirm you have:

• Imports of numpy, pytest, and load_subject / load_all_subjects from src.loader.
• The create_mock_subject_file helper generating 24-column tab-separated files.
• TestLoadSubject with 3 test methods.
• TestLoadAllSubjects with 1 test method checking null-activity filtering.

For main.py, confirm you have:

• Imports from src.analysis, src.config, src.features, src.loader, and src.visualization.
• setup_logging and parse_args functions.
• run_pipeline with Steps 2-5 sections and the hypothesis test branch.
• run_hypothesis_testing function with Kruskal-Wallis logic and Bonferroni correction.
• The if __name__ == "__main__" block calling setup_logging(), parse_args(), and run_pipeline().

Your feature matrix is exported, your tests pass, and the full pipeline runs end-to-end. Next up, you will tackle the Secret Mission and apply hypothesis testing to statistically rank which features best discriminate between activity classes.

Secret mission

Hypothesis Testing for Feature Discrimination

Your visualizations suggest certain features clearly separate activity classes. But how do you move from "this looks different" to "this is statistically significantly different"? In this mission, you will apply formal hypothesis testing to every extracted feature and produce a principled ranking that tells a future ML classifier exactly which features matter most.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

• The biometrics-portfolio/ directory on your Desktop (contains all source code, tests, and configuration).
• The MHEALTH dataset files in data/raw/ (72.1 MB of raw sensor logs).
• The Python virtual environment (venv/) with installed packages (NumPy, Pandas, Matplotlib, Seaborn, SciPy).
• Generated output files in outputs/ (figures and feature CSVs).

Keep everything running

No action needed. Choose this if you plan to continue into Module 2 (the ML classifier project) or want to experiment further with the pipeline.

• Your project directory stays on your Desktop at biometrics-portfolio/.
• The virtual environment stays intact with all packages installed.
• The exported feature CSV (mhealth_features.csv) is ready for Module 2 whenever you start it.
• Everything runs locally with no ongoing costs.

Pause - I'll come back to this later

Deactivate your virtual environment to free up your terminal session. All files remain on disk for when you return.

• Deactivate the virtual environment by running this in your terminal:

deactivate

• When you return, navigate back to the project and reactivate:

cd ~/Desktop/biometrics-portfolio/project_01_biometric_data_analysis
source venv/bin/activate

• All your code, data, and outputs will be exactly where you left them.

Delete - I don't want to use this again

Remove all project files, the virtual environment, and the downloaded dataset. This is permanent.

• Deactivate the virtual environment first:

deactivate

• Delete the entire portfolio directory by running:

macOS

rm -rf ~/Desktop/biometrics-portfolio

Windows

rmdir /s /q %USERPROFILE%\Desktop\biometrics-portfolio

• Confirm the directory is gone by running:

ls ~/Desktop

• You should no longer see biometrics-portfolio in the output.
• This removes all source code, the virtual environment, the downloaded MHEALTH dataset, and all generated outputs.

Nice Work!

Nice Work!

You just built a complete biometric data analysis pipeline that transforms raw wearable sensor recordings into an ML-ready feature matrix. From loading real ECG and accelerometer data off disk to exporting engineered features ready for classification, you handled every stage of a production data science workflow.

You've learned how to:

• Build a modular Python pipeline with type hints, docstrings, logging, and a reproducible virtual environment that processes real wearable biometric data from the UCI MHEALTH dataset.
• Generate publication-quality visualizations that reveal how physical activities produce distinct biometric signatures, proving that visualization is essential when raw statistical summaries alone cannot communicate discrimination clearly.
• Engineer a windowed feature matrix using 5-second sliding windows with time-domain statistics, exported as a CSV artifact that feeds directly into an ML classifier in Module 2.
• Secret Mission: Applied formal hypothesis testing with Bonferroni correction to statistically rank which sensor features significantly discriminate between activity classes.

Ready to quiz yourself?