Train a Group-Safe Classifier

Introduction

30 Second Summary

A model can look accurate while quietly memorizing the people it has already seen. The real test is whether its rules still work for runners left out of training.

In this project, you will compare three running-readiness classifiers with group-aware cross-validation. Your reloadable Pipelines will form a Level 3 handoff without touching the Level 1 test partition.

What You'll Build

By the end, your terminal will show a group-safe comparison of three classifiers with reloadable Pipelines ready for Level 3.

By the end of this project, you'll have:

• A three-model comparison showing fold-level accuracy plus validation variability. You will also see each model's generalization gap.
• A group-safety proof showing every runner in validation exactly once. Each fold will record zero runner overlap.
• A reloadable Level 3 handoff containing three trained Pipeline artifacts. Its contract will record validation metrics plus reproducibility evidence.
• Secret Mission: Compare three decision-tree capacities across the same group folds. Explain the observed balance between training fit and generalization.

Are there any prerequisites?

You need the completed Level 1 project with its existing .venv. This project continues in that repository to preserve the immutable v1 handoff.

Before We Start

This project depends on a clean experimental boundary before any model training begins. You will rotate only Level 1 training runners through GroupKFold validation while keeping data/v1/processed/test.csv unread and unscored until Level 3.

Verify the v1 Handoff and Runtime

Level 2 depends on an immutable Level 1 handoff. A missing artifact or changed runtime would make the model comparison impossible to reproduce.

In this step, you will verify the repository's four handoff files with VS Code. You will also activate the existing virtual environment before installing the pinned dependency set.

In this step, get ready to:

• Confirm that the four immutable v1 handoff files still exist.
• Pin the Level 2 dependencies in requirements.txt.
• Prove that the original Level 1 data tests still pass.

Open the Level 1 repository

The existing repository defines the boundary between Level 1 data preparation and Level 2 model training. Keeping this workspace intact preserves the v1 handoff that later experiments depend on.

• Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type VS Code and press Enter to open it.
• Click File in the top menu bar.
• Click Open Folder.
• Select the existing Level 1 repository folder from the previous project.
• Click Open.

You should see the Level 1 repository in the Explorer sidebar.

• Click Terminal in the top menu bar.
• Click New Terminal.
• Activate the repository's existing environment by running this command:

source .venv/bin/activate

What does activation change?

Activation directs Python and pip to the packages inside .venv. This keeps the Level 2 dependency set isolated inside the existing repository.

Environment not activating?

• Confirm that the VS Code Explorer shows .venv inside the Level 1 repository.
• Confirm that the integrated terminal started inside the Level 1 repository folder.

If the environment still does not activate, help me diagnose the existing .venv activation problem.

• Expand data/v1/processed/ in the Explorer sidebar.
• Confirm that data/v1/processed/train.csv is listed.
• Confirm that data/v1/processed/test.csv is listed.
• Expand contracts/v1/ in the Explorer sidebar.
• Confirm that contracts/v1/schema_contract.json is listed.
• Expand artifacts/v1/ in the Explorer sidebar.
• Confirm that artifacts/v1/preprocessing.json is listed.

You can now see all four v1 handoff files. These files remain unchanged because they define the fixed input boundary for Level 2.

Pin the Level 2 runtime

Exact package pins ensure that later training runs use the same numerical behavior and persistence format. Level 2 adds the serializer required to hand fitted pipelines to Level 3.

• Click requirements.txt in the Explorer sidebar.
• Replace the file's contents with the complete pinned dependency set below:

numpy==2.5.4
pandas==3.1.0
scikit-learn==1.9.1
joblib==1.6.0

Why pin all four packages?

• NumPy 2.5.4 provides the numerical arrays used during model training.
• pandas 3.1.0 loads the immutable training CSV into a tabular structure.
• scikit-learn 1.9.1 supplies the grouped validation tools and classifiers used in Level 2.
• joblib 1.6.0 preserves fitted pipeline objects for Level 3.

• Save requirements.txt.
• Install the exact dependency set into the active environment by running this command:

python -m pip install -r requirements.txt

What does this command do?

Python runs pip inside the active environment. Pip reads each exact pin from requirements.txt before installing or confirming the required version.

Pip should confirm that the four pinned packages are installed in the active environment.

Dependency installation failing?

• Confirm that your terminal prompt identifies the active .venv environment.
• Confirm that the active Python interpreter is version 3.12 or newer.
• Compare every dependency line with the full-file reference below.

For environment-specific installation help, help me diagnose the pinned requirements installation.

✔️ Awesome, I've got everything!

Your saved requirements.txt now records the complete Level 2 runtime.

ⓧ I'd like to double check the full code

• Compare your requirements.txt file with this complete reference:

numpy==2.5.4
pandas==3.1.0
scikit-learn==1.9.1
joblib==1.6.0

What should match?

Your file should contain these four lines in this order. Each version uses an exact equality pin so a later install cannot silently select a different release.

Prove the handoff still passes

The Level 1 tests protect the original runner split and preprocessing state. Passing them after the dependency installation proves that the Level 2 setup has preserved that boundary.

Before you run the suite, do you expect the dependency update to preserve all seven Level 1 checks?

• Run the original Level 1 data pipeline tests with verbose output by running this command:

python -m unittest -v tests/test_data_pipeline.py

What does this command check?

Python's test runner executes the existing data pipeline test file. Verbose output names each check so you can identify the exact Level 1 contract that passes or fails.

You should see seven tests finish. The final status is OK.

Strong start. The immutable v1 handoff and pinned Level 2 runtime now agree.

Level 1 test failing?

• Read the first failing test name in the verbose output.
• Confirm that all four v1 handoff paths still exist in the repository.
• Confirm that requirements.txt matches the full-file reference above.

If a test still fails, help me trace the failing Level 1 contract without modifying the v1 handoff.

Your Level 1 evidence is intact. Next, you will use GroupKFold to measure how a majority-class baseline behaves across unseen runner groups.

Build the Group-Safe Baseline

Accuracy needs a comparison point that respects the runner boundary established in Level 1. This step tests how credible a constant classifier can look across unseen runner groups.

You will build a majority-class baseline inside four shared folds. The experiment consumes only the immutable training handoff.

In this step, get ready to:

• Load and validate the three approved Level 1 training inputs.
• Create four reproducible folds that keep runners separated.
• Evaluate a fresh majority-class Pipeline inside every fold.

Validate the Level 1 handoff

The training module treats the Level 1 outputs as an immutable data product. A SHA-256 fingerprint identifies the exact bytes consumed by the experiment.

• In the VS Code Explorer sidebar from earlier, select the existing src folder.
• Click the New File button in the Explorer toolbar.
• Enter train_models.py as the file name.
• Add the imports and experiment constants to src/train_models.py by pasting this code:

from __future__ import annotations

import json
from hashlib import sha256
from pathlib import Path
from statistics import fmean, pstdev
from typing import Any

import numpy as np
import pandas as pd
from sklearn.base import clone
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import GroupKFold
from sklearn.pipeline import Pipeline

DATASET_VERSION = "v1"
RANDOM_STATE = 42
GROUP_FOLDS = 4

What do these imports support?

• The standard-library imports read JSON files. They also calculate fingerprints and summarize fold scores.
• NumPy provides the index-array type returned by the fold splitter.
• pandas loads the training CSV into a DataFrame.
• scikit-learn supplies the splitter, estimator, cloning utility, metric, and Pipeline.
• The three constants keep the dataset version, random seed, and fold count consistent.

• Save src/train_models.py.
• Check that Python can parse the first chunk by running this command in the active terminal:

python -m src.train_models

What should you see?

The active environment prompt returns without a traceback. This confirms that Python can import the pinned libraries and parse the constants.

Seeing an import failure?

Confirm that the terminal prompt still shows the active .venv. Compare each import against the code block above.

Keep the dependency pins unchanged while you investigate. Help me diagnose the import failure.

• Add the fingerprint and handoff path helpers below the constants by pasting this code:

def fingerprint(path: Path) -> str:
    return sha256(path.read_bytes()).hexdigest()


def handoff_paths(project_root: Path) -> dict[str, Path]:
    return {
        "train": project_root / "data" / DATASET_VERSION / "processed" / "train.csv",
        "schema": project_root / "contracts" / DATASET_VERSION / "schema_contract.json",
        "preprocessing": project_root / "artifacts" / DATASET_VERSION / "preprocessing.json",
    }

Why centralize the handoff paths?

• The fingerprint() helper turns the bytes of one consumed input into a reproducible digest.
• The handoff_paths() helper defines only the training CSV, schema contract, and preprocessing artifact.
• The held-out test CSV does not appear in this mapping.

• Save src/train_models.py.
• Check that Python can parse both helpers by running this command:

python -m src.train_models

What does this check prove?

The prompt returns without a traceback. Python can now parse both helpers without reading any handoff file.

• Add the handoff loader below handoff_paths() by pasting this code:

def load_training_handoff(
    project_root: Path,
) -> tuple[pd.DataFrame, dict[str, Any], dict[str, Any]]:
    paths = handoff_paths(project_root)
    contract = json.loads(paths["schema"].read_text(encoding="utf-8"))
    preprocessing = json.loads(paths["preprocessing"].read_text(encoding="utf-8"))
    train = pd.read_csv(paths["train"])

    if contract["dataset_version"] != DATASET_VERSION:
        raise ValueError("Schema dataset version does not match Level 2")
    if preprocessing["dataset_version"] != DATASET_VERSION:
        raise ValueError("Preprocessing dataset version does not match Level 2")
    if preprocessing["feature_columns"] != contract["feature_columns"]:
        raise ValueError("Preprocessing feature order does not match the schema")

    required = [*contract["feature_columns"], contract["target_column"], contract["group_column"]]
    missing = sorted(set(required) - set(train.columns))
    if missing:
        raise ValueError(f"Training handoff is missing columns: {missing}")
    if int(train[contract["feature_columns"]].isna().sum().sum()) != 0:
        raise ValueError("Training features must satisfy v1 preprocessing")
    if int(train[contract["target_column"]].nunique()) < 2:
        raise ValueError("Training target must contain at least two classes")

    return train, contract, preprocessing

What does the loader protect?

• The version checks prevent Level 2 from mixing artifacts from different dataset releases.
• The feature-order check preserves the exact input sequence defined by the schema.
• The required-column check confirms that every feature, target, and group column exists.
• The missing-value check confirms that the frozen Level 1 preprocessing contract still holds.
• The class-count check confirms that the target supports binary classification.

• Save src/train_models.py.
• Check the completed loader for syntax errors by running this command:

python -m src.train_models

What should happen now?

The active prompt returns without a traceback. The loader is ready to validate the three approved inputs when the experiment starts.

Seeing a loader error?

Check the indentation inside load_training_handoff(). Every validation condition belongs inside the function.

Compare the contract keys character for character. Help me repair the handoff loader.

Create the runner-group folds

Repeated measurements from one runner can resemble each other. GroupKFold keeps every runner on one side of a fold boundary.

• Add the shared fold generator below load_training_handoff() by pasting this code:

def make_group_folds(
    train: pd.DataFrame,
    contract: dict[str, Any],
) -> list[tuple[np.ndarray, np.ndarray]]:
    splitter = GroupKFold(
        n_splits=GROUP_FOLDS,
        shuffle=True,
        random_state=RANDOM_STATE,
    )
    return list(
        splitter.split(
            train[contract["feature_columns"]],
            train[contract["target_column"]],
            groups=train[contract["group_column"]],
        )
    )

How are the folds kept group-safe?

• The GROUP_FOLDS constant creates four validation rotations.
• The shuffle=True setting shuffles whole runner groups before assignment.
• The random_state=RANDOM_STATE setting makes the assignment reproducible.
• The groups argument uses the group column named by the schema.

• Save src/train_models.py.
• Check the fold generator for syntax errors by running this command:

python -m src.train_models

What does this check confirm?

The prompt returns without a traceback. The module can now construct the same four runner-group folds on every run.

• Add the fold manifest builder below make_group_folds() by pasting this code:

def build_fold_manifest(
    train: pd.DataFrame,
    contract: dict[str, Any],
    folds: list[tuple[np.ndarray, np.ndarray]],
) -> list[dict[str, Any]]:
    group_column = contract["group_column"]
    manifest = []
    for fold_number, (fit_indices, validation_indices) in enumerate(folds, start=1):
        fit_groups = sorted(str(value) for value in train.iloc[fit_indices][group_column].unique())
        validation_groups = sorted(
            str(value) for value in train.iloc[validation_indices][group_column].unique()
        )
        overlap = sorted(set(fit_groups).intersection(validation_groups))
        manifest.append(
            {
                "fold": fold_number,
                "fit_rows": int(len(fit_indices)),
                "validation_rows": int(len(validation_indices)),
                "fit_runners": fit_groups,
                "validation_runners": validation_groups,
                "runner_overlap": overlap,
            }
        )
    return manifest

What evidence does the manifest capture?

• Each entry records the fit-row count and validation-row count for one fold.
• The runner lists make group membership inspectable in the final report.
• The runner_overlap field calculates the intersection between the fit and validation runner sets.
• Four validation-runner lists show that each training runner receives one validation turn.

• Save src/train_models.py.
• Check the manifest builder for syntax errors by running this command:

python -m src.train_models

What should you see?

The terminal returns to the active environment prompt without a traceback. The final report can now expose runner membership for every fold.

Having trouble with the folds?

Confirm that make_group_folds() appears before build_fold_manifest(). Both function definitions must align at the left edge.

Check the indentation inside the manifest loop. Help me debug the group-fold code.

Evaluate the majority baseline

A Pipeline gives every candidate one fitting interface. This first Pipeline contains only a classifier because its strategy does not transform the features.

• Add the baseline candidate builder below build_fold_manifest() by pasting this code:

def build_model_candidates() -> dict[str, Pipeline]:
    return {
        "majority_baseline": Pipeline(
            steps=[
                ("classifier", DummyClassifier(strategy="most_frequent")),
            ]
        ),
    }

What does the baseline learn?

The DummyClassifier(strategy="most_frequent") learns the most common target label among its fit rows. It predicts that label without using a relationship between the input features and target.

Wrapping the estimator in a Pipeline gives it the same interface as later candidates. This keeps the comparison structure consistent.

• Save src/train_models.py.
• Check the candidate builder for syntax errors by running this command:

python -m src.train_models

What does this check prove?

The terminal returns without a traceback. The baseline Pipeline is ready to be cloned inside the fold loop.

• Add the fold evaluation loop below build_model_candidates() by pasting this code:

def evaluate_candidates(train: pd.DataFrame, contract: dict[str, Any], folds: list[tuple[np.ndarray, np.ndarray]], candidates: dict[str, Pipeline]) -> dict[str, dict[str, Any]]:
    features = contract["feature_columns"]
    target = contract["target_column"]
    results: dict[str, dict[str, Any]] = {}

    for name, candidate in candidates.items():
        fold_results = []
        for fold_number, (fit_indices, validation_indices) in enumerate(folds, start=1):
            fit = train.iloc[fit_indices]
            validation = train.iloc[validation_indices]
            model = clone(candidate)
            model.fit(fit[features], fit[target])
            fit_predictions = model.predict(fit[features])
            validation_predictions = model.predict(validation[features])
            training_accuracy = float(accuracy_score(fit[target], fit_predictions))
            validation_accuracy = float(accuracy_score(validation[target], validation_predictions))
            fold_results.append({"fold": fold_number, "training_accuracy": round(training_accuracy, 6), "validation_accuracy": round(validation_accuracy, 6), "generalization_gap": round(training_accuracy - validation_accuracy, 6), "unique_validation_predictions": sorted(int(value) for value in set(validation_predictions.tolist()))})

        training_scores = [fold["training_accuracy"] for fold in fold_results]
        validation_scores = [fold["validation_accuracy"] for fold in fold_results]
        mean_training = fmean(training_scores)
        mean_validation = fmean(validation_scores)
        results[name] = {"folds": fold_results, "aggregate": {"mean_training_accuracy": round(mean_training, 6), "mean_validation_accuracy": round(mean_validation, 6), "validation_accuracy_std": round(pstdev(validation_scores), 6), "generalization_gap": round(mean_training - mean_validation, 6)}}

    return results

How does the evaluation stay independent?

• The clone(candidate) call creates a fresh unfitted Pipeline for every fold.
• The model fits only the rows selected by fit_indices.
• Training accuracy measures performance on the fit runners.
• Validation accuracy measures transfer to the held-out runners in that fold.
• The generalization_gap field subtracts validation accuracy from training accuracy.
• The unique_validation_predictions field records every distinct label emitted in the fold.

• Save src/train_models.py.
• Check the evaluation loop for syntax errors by running this command:

python -m src.train_models

What should happen?

The prompt returns without a traceback. The module now contains the baseline fitting and scoring logic.

Seeing an evaluation error?

Confirm that clone(candidate) sits inside the inner fold loop. Both prediction calls must use the cloned model.

Review the closing brackets on the nested result dictionaries. Help me debug the evaluation loop.

• Add the executable report entry point at the bottom of src/train_models.py by pasting this code:

def main() -> None:
    project_root = Path(__file__).resolve().parents[1]
    train, contract, preprocessing = load_training_handoff(project_root)
    folds = make_group_folds(train, contract)
    paths = handoff_paths(project_root)
    results = evaluate_candidates(train, contract, folds, build_model_candidates())
    report = {
        "input_fingerprints": {
            name: {
                "path": path.relative_to(project_root).as_posix(),
                "sha256": fingerprint(path),
            }
            for name, path in paths.items()
        },
        "cross_validation": {
            "method": "GroupKFold",
            "n_splits": GROUP_FOLDS,
            "shuffle": True,
            "random_state": RANDOM_STATE,
            "fold_manifest": build_fold_manifest(train, contract, folds),
        },
        "results": results,
    }
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main()

What does the report prove?

• The input_fingerprints section records the paths and SHA-256 values for the three consumed inputs.
• The cross_validation section records the fold settings and runner membership.
• The results section records per-fold scores and aggregate evidence.
• The report contains no path, fingerprint, or score for the held-out test CSV.

• Save src/train_models.py.

Before you run the completed experiment, predict whether each validation fold will contain one predicted label or several.

• Run the completed baseline experiment with this command:

python -m src.train_models

What should the report show?

• The fold manifest contains four entries.
• Every runner_overlap list is empty.
• Each training runner appears in one validation_runners list across the four folds.
• Every baseline fold contains one value in unique_validation_predictions.
• The aggregate section reports mean training accuracy, mean validation accuracy, validation variability, and the generalization gap.

That result exposes the intended shortfall. The baseline can earn accuracy while every running feature remains irrelevant to its prediction.

Why is this failure useful?

The baseline establishes an empirical floor under the same runner-safe validation design used for later comparisons. Its score shows how far class frequency alone can go.

A single prediction value confirms that the classifier did not learn a feature-dependent rule. Accuracy alone cannot prove meaningful feature learning.

Report missing folds or empty overlaps?

Confirm that main() calls make_group_folds() once. The same folds list must reach the manifest and evaluation functions.

If an overlap list contains a runner, inspect the groups argument passed to the splitter. Help me diagnose the baseline report.

✔️ Awesome, I've got everything!

Your report proves that the majority baseline ran across four group-safe folds. Double check that you saved src/train_models.py.

ⓧ I'd like to double check the full code

Compare your complete src/train_models.py file with this baseline implementation.

from __future__ import annotations

import json
from hashlib import sha256
from pathlib import Path
from statistics import fmean, pstdev
from typing import Any

import numpy as np
import pandas as pd
from sklearn.base import clone
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import GroupKFold
from sklearn.pipeline import Pipeline

DATASET_VERSION = "v1"
RANDOM_STATE = 42
GROUP_FOLDS = 4


def fingerprint(path: Path) -> str:
    return sha256(path.read_bytes()).hexdigest()


def handoff_paths(project_root: Path) -> dict[str, Path]:
    return {
        "train": project_root / "data" / DATASET_VERSION / "processed" / "train.csv",
        "schema": project_root / "contracts" / DATASET_VERSION / "schema_contract.json",
        "preprocessing": project_root / "artifacts" / DATASET_VERSION / "preprocessing.json",
    }


def load_training_handoff(
    project_root: Path,
) -> tuple[pd.DataFrame, dict[str, Any], dict[str, Any]]:
    paths = handoff_paths(project_root)
    contract = json.loads(paths["schema"].read_text(encoding="utf-8"))
    preprocessing = json.loads(paths["preprocessing"].read_text(encoding="utf-8"))
    train = pd.read_csv(paths["train"])

    if contract["dataset_version"] != DATASET_VERSION:
        raise ValueError("Schema dataset version does not match Level 2")
    if preprocessing["dataset_version"] != DATASET_VERSION:
        raise ValueError("Preprocessing dataset version does not match Level 2")
    if preprocessing["feature_columns"] != contract["feature_columns"]:
        raise ValueError("Preprocessing feature order does not match the schema")

    required = [*contract["feature_columns"], contract["target_column"], contract["group_column"]]
    missing = sorted(set(required) - set(train.columns))
    if missing:
        raise ValueError(f"Training handoff is missing columns: {missing}")
    if int(train[contract["feature_columns"]].isna().sum().sum()) != 0:
        raise ValueError("Training features must satisfy v1 preprocessing")
    if int(train[contract["target_column"]].nunique()) < 2:
        raise ValueError("Training target must contain at least two classes")

    return train, contract, preprocessing


def make_group_folds(
    train: pd.DataFrame,
    contract: dict[str, Any],
) -> list[tuple[np.ndarray, np.ndarray]]:
    splitter = GroupKFold(
        n_splits=GROUP_FOLDS,
        shuffle=True,
        random_state=RANDOM_STATE,
    )
    return list(
        splitter.split(
            train[contract["feature_columns"]],
            train[contract["target_column"]],
            groups=train[contract["group_column"]],
        )
    )


def build_fold_manifest(
    train: pd.DataFrame,
    contract: dict[str, Any],
    folds: list[tuple[np.ndarray, np.ndarray]],
) -> list[dict[str, Any]]:
    group_column = contract["group_column"]
    manifest = []
    for fold_number, (fit_indices, validation_indices) in enumerate(folds, start=1):
        fit_groups = sorted(str(value) for value in train.iloc[fit_indices][group_column].unique())
        validation_groups = sorted(
            str(value) for value in train.iloc[validation_indices][group_column].unique()
        )
        overlap = sorted(set(fit_groups).intersection(validation_groups))
        manifest.append(
            {
                "fold": fold_number,
                "fit_rows": int(len(fit_indices)),
                "validation_rows": int(len(validation_indices)),
                "fit_runners": fit_groups,
                "validation_runners": validation_groups,
                "runner_overlap": overlap,
            }
        )
    return manifest


def build_model_candidates() -> dict[str, Pipeline]:
    return {
        "majority_baseline": Pipeline(
            steps=[
                ("classifier", DummyClassifier(strategy="most_frequent")),
            ]
        ),
    }


def evaluate_candidates(train: pd.DataFrame, contract: dict[str, Any], folds: list[tuple[np.ndarray, np.ndarray]], candidates: dict[str, Pipeline]) -> dict[str, dict[str, Any]]:
    features = contract["feature_columns"]
    target = contract["target_column"]
    results: dict[str, dict[str, Any]] = {}

    for name, candidate in candidates.items():
        fold_results = []
        for fold_number, (fit_indices, validation_indices) in enumerate(folds, start=1):
            fit = train.iloc[fit_indices]
            validation = train.iloc[validation_indices]
            model = clone(candidate)
            model.fit(fit[features], fit[target])
            fit_predictions = model.predict(fit[features])
            validation_predictions = model.predict(validation[features])
            training_accuracy = float(accuracy_score(fit[target], fit_predictions))
            validation_accuracy = float(accuracy_score(validation[target], validation_predictions))
            fold_results.append({"fold": fold_number, "training_accuracy": round(training_accuracy, 6), "validation_accuracy": round(validation_accuracy, 6), "generalization_gap": round(training_accuracy - validation_accuracy, 6), "unique_validation_predictions": sorted(int(value) for value in set(validation_predictions.tolist()))})

        training_scores = [fold["training_accuracy"] for fold in fold_results]
        validation_scores = [fold["validation_accuracy"] for fold in fold_results]
        mean_training = fmean(training_scores)
        mean_validation = fmean(validation_scores)
        results[name] = {"folds": fold_results, "aggregate": {"mean_training_accuracy": round(mean_training, 6), "mean_validation_accuracy": round(mean_validation, 6), "validation_accuracy_std": round(pstdev(validation_scores), 6), "generalization_gap": round(mean_training - mean_validation, 6)}}

    return results


def main() -> None:
    project_root = Path(__file__).resolve().parents[1]
    train, contract, preprocessing = load_training_handoff(project_root)
    folds = make_group_folds(train, contract)
    paths = handoff_paths(project_root)
    results = evaluate_candidates(train, contract, folds, build_model_candidates())
    report = {
        "input_fingerprints": {
            name: {
                "path": path.relative_to(project_root).as_posix(),
                "sha256": fingerprint(path),
            }
            for name, path in paths.items()
        },
        "cross_validation": {
            "method": "GroupKFold",
            "n_splits": GROUP_FOLDS,
            "shuffle": True,
            "random_state": RANDOM_STATE,
            "fold_manifest": build_fold_manifest(train, contract, folds),
        },
        "results": results,
    }
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main()

Your group-safe baseline now provides a reproducible floor without touching the held-out test partition. Next, you will test whether feature-dependent models transfer better across the same runner groups.

Compare Feature-Learning Models

Your majority-class baseline established a useful accuracy floor. Its single repeated prediction also proved that it ignores every runner measurement.

Now you will compare logistic regression with a decision tree on the same folds. Each Pipeline reuses the four GroupKFold splits so model behavior is the only experimental change.

In this step, get ready to:

• Add scaled logistic regression as a feature-learning candidate.
• Add an unrestricted decision tree as a nonlinear candidate.
• Compare validation performance with variability and generalization gaps.

Add the feature-learning candidates

Logistic regression needs fold-local scaling because its regularization acts on coefficient size. StandardScaler learns the fit runners' feature means and standard deviations before the classifier learns its weights.

The decision tree works directly with the feature values. It learns thresholds that reduce Gini impurity without using a scaler.

• In src/train_models.py, locate the existing import section.
• Add the comparison imports by copying these lines into that section:

from statistics import fmean, pstdev

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

What Do These Imports Add?

• LogisticRegression supplies the regularized linear probability model.
• StandardScaler supplies the learned fold-local transformation.
• DecisionTreeClassifier supplies the nonlinear threshold model.
• fmean and pstdev summarize model performance across the four folds.

• Inside build_model_candidates(), keep the existing majority_baseline entry.
• Add these two entries below the baseline entry:

        "logistic_regression": Pipeline(
            steps=[
                ("scaler", StandardScaler()),
                (
                    "classifier",
                    LogisticRegression(
                        C=1.0,
                        solver="lbfgs",
                        max_iter=1000,
                    ),
                ),
            ]
        ),
        "decision_tree": Pipeline(
            steps=[
                (
                    "classifier",
                    DecisionTreeClassifier(random_state=RANDOM_STATE),
                ),
            ]
        ),

How Do These Models Learn?

• StandardScaler() fits only on the current fold's fit runners because it lives inside the Pipeline.
• LogisticRegression() learns a linear log-odds score with regularized log loss.
• The sigmoid converts that score into a probability. The default 0.5 threshold converts the probability into a class prediction.
• DecisionTreeClassifier() recursively chooses feature thresholds that reduce Gini impurity.

• Save src/train_models.py.
• Check that all three candidates reach the existing fold engine by running:

python -m src.train_models

What Should You See?

The JSON output now includes majority_baseline, logistic_regression, and decision_tree. Each model has results from the shared runner-group folds.

Good. The same fold engine now trains three fresh models without touching the held-out test partition.

Missing One of the Models?

Check that both new dictionary entries remain inside the dictionary returned by build_model_candidates(). Confirm that the surrounding commas and parentheses match the reference code.

If Python reports an import problem, confirm that your existing virtual environment still uses the pinned requirements from the previous step.

Help me debug the model candidate additions in src/train_models.py.

Record settings and generalization metrics

A fair comparison records the settings that produced each result. It also separates training fit from performance on unseen runner groups.

The generalization gap is mean training accuracy minus mean validation accuracy. A larger positive gap shows that more of the training fit failed to transfer to validation runners.

• Add model_settings() directly below build_model_candidates() by copying this function:

def model_settings() -> dict[str, dict[str, Any]]:
    return {
        "majority_baseline": {
            "pipeline_steps": ["classifier"],
            "hyperparameters": {"strategy": "most_frequent"},
        },
        "logistic_regression": {
            "pipeline_steps": ["scaler", "classifier"],
            "hyperparameters": {
                "scaler": "StandardScaler",
                "C": 1.0,
                "solver": "lbfgs",
                "max_iter": 1000,
            },
        },
        "decision_tree": {
            "pipeline_steps": ["classifier"],
            "hyperparameters": {"random_state": RANDOM_STATE},
        },
    }

Why Record Model Settings?

The settings connect each metric to the exact Pipeline that produced it. Level 3 can use this record without guessing how scaling or classifier parameters were configured.

• In evaluate_candidates(), locate the summary section that begins with training_scores.
• Replace that summary section with this aggregation logic:

        training_scores = [
            fold["training_accuracy"] for fold in fold_results
        ]
        validation_scores = [
            fold["validation_accuracy"] for fold in fold_results
        ]
        mean_training = fmean(training_scores)
        mean_validation = fmean(validation_scores)
        results[name] = {
            "folds": fold_results,
            "aggregate": {
                "mean_training_accuracy": round(mean_training, 6),
                "mean_validation_accuracy": round(mean_validation, 6),
                "validation_accuracy_std": round(
                    pstdev(validation_scores),
                    6,
                ),
                "generalization_gap": round(
                    mean_training - mean_validation,
                    6,
                ),
            },
        }

What Does This Summary Measure?

• mean_training_accuracy measures average fit on the runners used to train each fold model.
• mean_validation_accuracy measures average accuracy on unseen runner groups.
• validation_accuracy_std measures how sensitive performance is to the selected validation runners.
• generalization_gap measures how much average training accuracy exceeds average validation accuracy.

• Save src/train_models.py.

Run the three-model comparison

The shared folds now support a controlled comparison. The remaining question is whether each model's training fit transfers to runners it did not fit.

Before you run the comparison, which model do you expect to have the largest positive gap between training accuracy and validation accuracy?

• Run the complete grouped comparison with this command:

python -m src.train_models

What Should the Final Report Contain?

• Each of the three model keys contains four fold records.
• Each fold records training accuracy and validation accuracy.
• Each fold records its training-minus-validation generalization gap.
• Each aggregate records mean training accuracy, mean validation accuracy, validation standard deviation, and the overall generalization gap.

You should also see one unique validation prediction in every baseline fold. The learned models can produce feature-dependent predictions.

That is the comparison working: three fresh Pipelines now face the same unseen runner groups under one experimental boundary.

How Should You Select a Model?

Use mean_validation_accuracy to select the current winner. Use validation_accuracy_std to judge how sensitive that result is to the fold composition.

Compare mean_training_accuracy with mean_validation_accuracy to identify underfitting or overfitting. The selected model is validation-selected because the held-out test set remains unscored.

Comparison Command Failing?

Check that evaluate_candidates() calls clone(candidate) inside the fold loop. Reusing one fitted object would carry state across folds.

Check that the logistic Pipeline lists the scaler before the classifier. This order keeps scaling inside each fold's fit boundary.

Help me debug the grouped model comparison.

✔️ Awesome, I've got everything!

Your report compares all three models across the same four group-safe folds. Double check that you saved src/train_models.py.

ⓧ I'd like to double check the full code

Compare the complete candidate-comparison sections below with src/train_models.py.

from statistics import fmean, pstdev
from typing import Any

import numpy as np
import pandas as pd
from sklearn.base import clone
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

RANDOM_STATE = 42


def build_model_candidates() -> dict[str, Pipeline]:
    return {
        "majority_baseline": Pipeline(
            steps=[
                ("classifier", DummyClassifier(strategy="most_frequent")),
            ]
        ),
        "logistic_regression": Pipeline(
            steps=[
                ("scaler", StandardScaler()),
                (
                    "classifier",
                    LogisticRegression(
                        C=1.0,
                        solver="lbfgs",
                        max_iter=1000,
                    ),
                ),
            ]
        ),
        "decision_tree": Pipeline(
            steps=[
                (
                    "classifier",
                    DecisionTreeClassifier(random_state=RANDOM_STATE),
                ),
            ]
        ),
    }


def model_settings() -> dict[str, dict[str, Any]]:
    return {
        "majority_baseline": {
            "pipeline_steps": ["classifier"],
            "hyperparameters": {"strategy": "most_frequent"},
        },
        "logistic_regression": {
            "pipeline_steps": ["scaler", "classifier"],
            "hyperparameters": {
                "scaler": "StandardScaler",
                "C": 1.0,
                "solver": "lbfgs",
                "max_iter": 1000,
            },
        },
        "decision_tree": {
            "pipeline_steps": ["classifier"],
            "hyperparameters": {"random_state": RANDOM_STATE},
        },
    }


def evaluate_candidates(
    train: pd.DataFrame,
    contract: dict[str, Any],
    folds: list[tuple[np.ndarray, np.ndarray]],
    candidates: dict[str, Pipeline],
) -> dict[str, dict[str, Any]]:
    features = contract["feature_columns"]
    target = contract["target_column"]
    results: dict[str, dict[str, Any]] = {}

    for name, candidate in candidates.items():
        fold_results = []
        for fold_number, (fit_indices, validation_indices) in enumerate(
            folds, start=1
        ):
            fit = train.iloc[fit_indices]
            validation = train.iloc[validation_indices]
            model = clone(candidate)
            model.fit(fit[features], fit[target])
            fit_predictions = model.predict(fit[features])
            validation_predictions = model.predict(validation[features])
            training_accuracy = float(
                accuracy_score(fit[target], fit_predictions)
            )
            validation_accuracy = float(
                accuracy_score(validation[target], validation_predictions)
            )
            fold_results.append(
                {
                    "fold": fold_number,
                    "training_accuracy": round(training_accuracy, 6),
                    "validation_accuracy": round(validation_accuracy, 6),
                    "generalization_gap": round(
                        training_accuracy - validation_accuracy,
                        6,
                    ),
                    "unique_validation_predictions": sorted(
                        int(value)
                        for value in set(validation_predictions.tolist())
                    ),
                }
            )

        training_scores = [
            fold["training_accuracy"] for fold in fold_results
        ]
        validation_scores = [
            fold["validation_accuracy"] for fold in fold_results
        ]
        mean_training = fmean(training_scores)
        mean_validation = fmean(validation_scores)
        results[name] = {
            "folds": fold_results,
            "aggregate": {
                "mean_training_accuracy": round(mean_training, 6),
                "mean_validation_accuracy": round(mean_validation, 6),
                "validation_accuracy_std": round(
                    pstdev(validation_scores),
                    6,
                ),
                "generalization_gap": round(
                    mean_training - mean_validation,
                    6,
                ),
            },
        }

    return results

What Should Match?

Your candidate names, Pipeline step order, classifier settings, fold-level fields, and aggregate fields should match this reference exactly.

Your validation comparison is complete. Next, you will refit fresh Pipelines on every allowed training row and package the Level 3 handoff.

Package the Level 3 Handoff

Your grouped comparison now shows how three Pipeline candidates perform across unseen runners. The held-out test partition still has no influence on model selection.

Cross-validation selects settings without producing the final Level 3 artifacts. This step refits fresh Pipelines on every allowed training row. It also packages the evidence that Level 3 needs for evaluation without retraining.

In this step, get ready to:

• Refit three fresh Pipelines on the complete Level 1 training partition.
• Persist the models with deterministic experiment metadata.
• Document the evaluation boundary for Level 3.

Refit fresh full-training Pipelines

The estimators fitted inside GroupKFold belong to individual folds. Final artifacts need fresh estimators fitted on the entire Level 1 training partition after model comparison is complete.

Why refit after validation?

Each validation model misses the runners assigned to its validation fold. Rebuilding the candidate Pipelines gives every final model access to all permitted training rows.

The selected hyperparameters stay fixed during this refit. The held-out test data remains outside the process.

• Return to src/train_models.py in VS Code.
• Bring the file to its complete Level 2 state by using the full-file reference below.
• Preserve the existing candidate comparison code exactly.
• Save src/train_models.py.

✔️ Awesome, I've got everything!

Your training module now refits fresh candidates. It also builds both handoff documents.

ⓧ I'd like to double check the full code

Compare your complete src/train_models.py file against this reference.

from __future__ import annotations

import json
import platform
from hashlib import sha256
from pathlib import Path
from statistics import fmean, pstdev
from typing import Any

import joblib
import numpy as np
import pandas as pd
import sklearn
from sklearn.base import clone
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import GroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

DATASET_VERSION = "v1"
RANDOM_STATE = 42
GROUP_FOLDS = 4


def fingerprint(path: Path) -> str:
    return sha256(path.read_bytes()).hexdigest()


def handoff_paths(project_root: Path) -> dict[str, Path]:
    return {
        "train": project_root / "data" / DATASET_VERSION / "processed" / "train.csv",
        "schema": project_root / "contracts" / DATASET_VERSION / "schema_contract.json",
        "preprocessing": project_root / "artifacts" / DATASET_VERSION / "preprocessing.json",
    }


def load_training_handoff(
    project_root: Path,
) -> tuple[pd.DataFrame, dict[str, Any], dict[str, Any]]:
    paths = handoff_paths(project_root)
    contract = json.loads(paths["schema"].read_text(encoding="utf-8"))
    preprocessing = json.loads(
        paths["preprocessing"].read_text(encoding="utf-8")
    )
    train = pd.read_csv(paths["train"])

    if contract["dataset_version"] != DATASET_VERSION:
        raise ValueError("Schema dataset version does not match Level 2")
    if preprocessing["dataset_version"] != DATASET_VERSION:
        raise ValueError("Preprocessing dataset version does not match Level 2")
    if preprocessing["feature_columns"] != contract["feature_columns"]:
        raise ValueError("Preprocessing feature order does not match the schema")

    required = [
        *contract["feature_columns"],
        contract["target_column"],
        contract["group_column"],
    ]
    missing = sorted(set(required) - set(train.columns))
    if missing:
        raise ValueError(f"Training handoff is missing columns: {missing}")
    if int(train[contract["feature_columns"]].isna().sum().sum()) != 0:
        raise ValueError("Training features must satisfy v1 preprocessing")
    if int(train[contract["target_column"]].nunique()) < 2:
        raise ValueError("Training target must contain at least two classes")

    return train, contract, preprocessing


def build_model_candidates() -> dict[str, Pipeline]:
    return {
        "majority_baseline": Pipeline(
            steps=[
                ("classifier", DummyClassifier(strategy="most_frequent")),
            ]
        ),
        "logistic_regression": Pipeline(
            steps=[
                ("scaler", StandardScaler()),
                (
                    "classifier",
                    LogisticRegression(
                        C=1.0,
                        solver="lbfgs",
                        max_iter=1000,
                    ),
                ),
            ]
        ),
        "decision_tree": Pipeline(
            steps=[
                (
                    "classifier",
                    DecisionTreeClassifier(random_state=RANDOM_STATE),
                ),
            ]
        ),
    }


def model_settings() -> dict[str, dict[str, Any]]:
    return {
        "majority_baseline": {
            "pipeline_steps": ["classifier"],
            "hyperparameters": {"strategy": "most_frequent"},
        },
        "logistic_regression": {
            "pipeline_steps": ["scaler", "classifier"],
            "hyperparameters": {
                "scaler": "StandardScaler",
                "C": 1.0,
                "solver": "lbfgs",
                "max_iter": 1000,
            },
        },
        "decision_tree": {
            "pipeline_steps": ["classifier"],
            "hyperparameters": {"random_state": RANDOM_STATE},
        },
    }


def make_group_folds(
    train: pd.DataFrame,
    contract: dict[str, Any],
) -> list[tuple[np.ndarray, np.ndarray]]:
    splitter = GroupKFold(
        n_splits=GROUP_FOLDS,
        shuffle=True,
        random_state=RANDOM_STATE,
    )
    return list(
        splitter.split(
            train[contract["feature_columns"]],
            train[contract["target_column"]],
            groups=train[contract["group_column"]],
        )
    )


def build_fold_manifest(
    train: pd.DataFrame,
    contract: dict[str, Any],
    folds: list[tuple[np.ndarray, np.ndarray]],
) -> list[dict[str, Any]]:
    group_column = contract["group_column"]
    manifest = []
    for fold_number, (fit_indices, validation_indices) in enumerate(folds, start=1):
        fit_groups = sorted(
            str(value) for value in train.iloc[fit_indices][group_column].unique()
        )
        validation_groups = sorted(
            str(value)
            for value in train.iloc[validation_indices][group_column].unique()
        )
        overlap = sorted(set(fit_groups).intersection(validation_groups))
        manifest.append(
            {
                "fold": fold_number,
                "fit_rows": int(len(fit_indices)),
                "validation_rows": int(len(validation_indices)),
                "fit_runners": fit_groups,
                "validation_runners": validation_groups,
                "runner_overlap": overlap,
            }
        )
    return manifest


def evaluate_candidates(
    train: pd.DataFrame,
    contract: dict[str, Any],
    folds: list[tuple[np.ndarray, np.ndarray]],
    candidates: dict[str, Pipeline],
) -> dict[str, dict[str, Any]]:
    features = contract["feature_columns"]
    target = contract["target_column"]
    results: dict[str, dict[str, Any]] = {}

    for name, candidate in candidates.items():
        fold_results = []
        for fold_number, (fit_indices, validation_indices) in enumerate(
            folds, start=1
        ):
            fit = train.iloc[fit_indices]
            validation = train.iloc[validation_indices]
            model = clone(candidate)
            model.fit(fit[features], fit[target])
            fit_predictions = model.predict(fit[features])
            validation_predictions = model.predict(validation[features])
            training_accuracy = float(
                accuracy_score(fit[target], fit_predictions)
            )
            validation_accuracy = float(
                accuracy_score(validation[target], validation_predictions)
            )
            fold_results.append(
                {
                    "fold": fold_number,
                    "training_accuracy": round(training_accuracy, 6),
                    "validation_accuracy": round(validation_accuracy, 6),
                    "generalization_gap": round(
                        training_accuracy - validation_accuracy,
                        6,
                    ),
                    "unique_validation_predictions": sorted(
                        int(value)
                        for value in set(validation_predictions.tolist())
                    ),
                }
            )

        training_scores = [
            fold["training_accuracy"] for fold in fold_results
        ]
        validation_scores = [
            fold["validation_accuracy"] for fold in fold_results
        ]
        mean_training = fmean(training_scores)
        mean_validation = fmean(validation_scores)
        results[name] = {
            "folds": fold_results,
            "aggregate": {
                "mean_training_accuracy": round(mean_training, 6),
                "mean_validation_accuracy": round(mean_validation, 6),
                "validation_accuracy_std": round(
                    pstdev(validation_scores),
                    6,
                ),
                "generalization_gap": round(
                    mean_training - mean_validation,
                    6,
                ),
            },
        }

    return results


def fit_and_save_models(
    train: pd.DataFrame,
    contract: dict[str, Any],
    project_root: Path,
) -> dict[str, str]:
    features = contract["feature_columns"]
    target = contract["target_column"]
    model_directory = project_root / "models" / DATASET_VERSION
    model_directory.mkdir(parents=True, exist_ok=True)
    artifact_paths: dict[str, str] = {}

    for name, model in build_model_candidates().items():
        model.fit(train[features], train[target])
        path = model_directory / f"{name}.joblib"
        joblib.dump(model, path, compress=3)
        artifact_paths[name] = path.relative_to(project_root).as_posix()

    return artifact_paths


def build_metadata(
    train: pd.DataFrame,
    contract: dict[str, Any],
    preprocessing: dict[str, Any],
    folds: list[tuple[np.ndarray, np.ndarray]],
    results: dict[str, dict[str, Any]],
    artifact_paths: dict[str, str],
    project_root: Path,
) -> dict[str, Any]:
    paths = handoff_paths(project_root)
    group_column = contract["group_column"]
    target_column = contract["target_column"]
    settings = model_settings()
    selected_model = max(
        results,
        key=lambda name: results[name]["aggregate"][
            "mean_validation_accuracy"
        ],
    )

    return {
        "experiment_id": "level-2-train-v1",
        "dataset_version": contract["dataset_version"],
        "schema_version": contract["schema_version"],
        "input_fingerprints": {
            name: {
                "path": path.relative_to(project_root).as_posix(),
                "sha256": fingerprint(path),
            }
            for name, path in paths.items()
        },
        "training_data": {
            "row_count": int(len(train)),
            "runner_count": int(train[group_column].nunique()),
            "class_counts": {
                str(int(label)): int(count)
                for label, count in train[target_column]
                .value_counts()
                .sort_index()
                .items()
            },
        },
        "level_1_preprocessing": {
            "strategy": preprocessing["strategy"],
            "fit_scope": preprocessing["fit_scope"],
            "feature_columns": preprocessing["feature_columns"],
        },
        "held_out_test_set": {
            "path": "data/v1/processed/test.csv",
            "status": "not loaded, fingerprinted, or scored in Level 2",
            "reserved_for": "Level 3 Evaluate",
        },
        "feature_columns": list(contract["feature_columns"]),
        "target_column": target_column,
        "group_column": group_column,
        "cross_validation": {
            "method": "GroupKFold",
            "n_splits": GROUP_FOLDS,
            "shuffle": True,
            "random_state": RANDOM_STATE,
            "fold_manifest": build_fold_manifest(train, contract, folds),
        },
        "model_settings": settings,
        "comparison_metric": "accuracy",
        "results": results,
        "selected_by_mean_validation_accuracy": selected_model,
        "selected_hyperparameters": settings[selected_model]["hyperparameters"],
        "final_fit_scope": "full Level 1 training partition only",
        "model_artifacts": artifact_paths,
        "runtime": {
            "python": platform.python_version(),
            "numpy": np.__version__,
            "pandas": pd.__version__,
            "scikit_learn": sklearn.__version__,
            "joblib": joblib.__version__,
        },
        "persistence_warning": "Load joblib artifacts only from trusted sources.",
        "level3_contract_path": "artifacts/v1/level3_evaluation_contract.json",
    }


def build_level3_contract(metadata: dict[str, Any]) -> dict[str, Any]:
    settings = metadata["model_settings"]
    results = metadata["results"]
    return {
        "contract_version": "1.0.0",
        "experiment_id": metadata["experiment_id"],
        "dataset_version": metadata["dataset_version"],
        "schema_version": metadata["schema_version"],
        "input_fingerprints": metadata["input_fingerprints"],
        "held_out_test_set": metadata["held_out_test_set"],
        "feature_columns": metadata["feature_columns"],
        "target_column": metadata["target_column"],
        "group_column": metadata["group_column"],
        "cross_validation": {
            "method": metadata["cross_validation"]["method"],
            "n_splits": metadata["cross_validation"]["n_splits"],
            "shuffle": metadata["cross_validation"]["shuffle"],
            "random_state": metadata["cross_validation"]["random_state"],
        },
        "random_seeds": {
            "group_kfold": RANDOM_STATE,
            "decision_tree": RANDOM_STATE,
        },
        "selected_model": metadata[
            "selected_by_mean_validation_accuracy"
        ],
        "selected_hyperparameters": metadata["selected_hyperparameters"],
        "models": {
            name: {
                "artifact_path": metadata["model_artifacts"][name],
                "pipeline_steps": settings[name]["pipeline_steps"],
                "hyperparameters": settings[name]["hyperparameters"],
                "validation_metrics": results[name]["aggregate"],
            }
            for name in metadata["model_artifacts"]
        },
        "runtime": metadata["runtime"],
        "evaluation_rule": (
            "Load a trusted Pipeline and call predict on test.csv without "
            "calling fit; record the first held-out metrics in Level 3."
        ),
        "persistence_warning": metadata["persistence_warning"],
    }


def run_training(project_root: Path) -> dict[str, Any]:
    train, contract, preprocessing = load_training_handoff(project_root)
    folds = make_group_folds(train, contract)
    results = evaluate_candidates(
        train,
        contract,
        folds,
        build_model_candidates(),
    )
    artifact_paths = fit_and_save_models(train, contract, project_root)
    metadata = build_metadata(
        train,
        contract,
        preprocessing,
        folds,
        results,
        artifact_paths,
        project_root,
    )
    level3_contract = build_level3_contract(metadata)

    report_path = (
        project_root / "reports" / DATASET_VERSION / "training_experiment.json"
    )
    level3_path = (
        project_root
        / "artifacts"
        / DATASET_VERSION
        / "level3_evaluation_contract.json"
    )
    report_path.parent.mkdir(parents=True, exist_ok=True)
    level3_path.parent.mkdir(parents=True, exist_ok=True)
    report_path.write_text(json.dumps(metadata, indent=2), encoding="utf-8")
    level3_path.write_text(
        json.dumps(level3_contract, indent=2),
        encoding="utf-8",
    )
    return metadata


def main() -> None:
    project_root = Path(__file__).resolve().parents[1]
    metadata = run_training(project_root)
    print(json.dumps(metadata, indent=2))


if __name__ == "__main__":
    main()

How does the packaging flow work?

• The fit_and_save_models() function rebuilds all three candidates before fitting them on every Level 1 training row.
• The build_metadata() function records the grouped validation evidence. It fingerprints only the three inputs consumed by training.
• The build_level3_contract() function reduces the report to the machine-readable context needed for evaluation.
• The run_training() function writes the model artifacts. It also writes both JSON handoff files.

Generate the machine-readable handoff

Model persistence turns each fitted Pipeline into a reloadable local artifact. The experiment report preserves the full training evidence. The smaller evaluation contract gives Level 3 an explicit prediction boundary.

Before you run the training program, which files do you expect it to create without reading the held-out test CSV?

• Generate the full Level 3 handoff by running this command from the existing repository terminal:

python -m src.train_models

What should you see?

The terminal prints the complete experiment metadata. The output includes grouped validation results for majority_baseline. It includes the same evidence for logistic_regression plus decision_tree.

The held_out_test_set entry reports that data/v1/processed/test.csv was not loaded. The three model paths point into models/v1/.

• Confirm models/v1/ contains majority_baseline.joblib.
• Confirm the same directory contains logistic_regression.joblib.
• Confirm the same directory contains decision_tree.joblib.
• Confirm reports/v1/training_experiment.json records fold metrics plus aggregate metrics.
• Confirm artifacts/v1/level3_evaluation_contract.json records the selected model plus every artifact path.

Why persist every candidate?

The validation winner is the selected model for this experiment. Saving every candidate lets Level 3 compare their first held-out results without fitting again.

That separation preserves the meaning of the test set. A surprising test result cannot influence Level 2 retroactively.

Missing an artifact?

Check that your terminal is using the existing repository directory. A different directory gives run_training() the wrong project root.

Compare the bottom of src/train_models.py with the reference above. A missing main() call prevents the module from starting.

Check that the Level 1 training CSV remains at data/v1/processed/train.csv.

Ask for help with the exact traceback: help me debug why src.train_models did not create every Level 3 handoff artifact

Load only trusted artifacts

Joblib persistence uses Python's pickle mechanism. Loading an untrusted artifact can execute arbitrary code.

Level 3 must load only these local artifacts with the recorded package versions. Its first held-out evaluation calls predict without calling fit.

Document the Level 3 boundary

A machine-readable contract supports automation. A portfolio document explains the experiment to a person reviewing the repository.

• Create docs/level-2-training.md in the existing repository with the reference content below.
• Save docs/level-2-training.md.

✔️ Awesome, I've got everything!

Your document now explains the learning objectives. It also sets the Level 3 evaluation boundary.

ⓧ I'd like to double check the full code

Compare your complete docs/level-2-training.md file against this reference.

# Level 2 Train: Group-Safe Running-Readiness Classification

## Problem

Predict the synthetic `needs_recovery_next_day` label from the five features defined by the Level 1 v1 schema. Level 2 performs model selection only inside the existing training partition. The original test partition remains unread and unscored for Level 3.

## How the models learn

- Majority baseline: learns only the most common training label and ignores all features. It is the minimum reference a feature-learning model should justify exceeding.
- Logistic regression: learns a regularized linear log-odds score. A sigmoid converts the score to a probability, and the default 0.5 threshold converts probability to class. `StandardScaler` is fitted inside each fold's Pipeline.
- Decision tree: greedily chooses feature thresholds that reduce Gini impurity. Unrestricted depth increases capacity, which can improve training fit while increasing sensitivity to the particular runners seen during fitting.

## Validation design

Four-fold `GroupKFold` uses only `data/v1/processed/train.csv`. Every runner appears in validation exactly once and never appears on both sides of one fold. Mean validation accuracy supports model selection. Validation standard deviation shows fold sensitivity. The difference between mean training and validation accuracy is the reported generalization gap.

The v1 mean-imputation artifact is a frozen upstream input. Level 2 does not refit it. Every new Level 2 learned transform stays inside a Pipeline and is fitted separately on each fold's training runners.

## Level 3 handoff

- `models/v1/majority_baseline.joblib`
- `models/v1/logistic_regression.joblib`
- `models/v1/decision_tree.joblib`
- `reports/v1/training_experiment.json`
- `artifacts/v1/level3_evaluation_contract.json`

The contract records input fingerprints, feature order, validation metrics, selected model and hyperparameters, seeds, runtime versions, and Pipeline paths. Level 3 must load a trusted Pipeline and call `predict` on `data/v1/processed/test.csv` without calling `fit` first.

## Reproduce

```bash
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m src.train_models
python -m unittest -v tests/test_training.py
```

## Security and limitations

Joblib is pickle-based. Never load an artifact from an untrusted source, and use the recorded package versions.

The data, target logic, and thresholds are synthetic educational examples. Accuracy is sufficient for the Level 2 baseline and generalization lesson, but Level 3 should use a broader metric set for final evaluation. Validation-selected means best among these candidates on grouped training folds, not proven best on future real runners.

What does the document capture?

The model section links each classifier to its learning behavior. The validation section explains why runner groups define the fold boundary.

The handoff section names every persisted artifact. The final section records the trust warning plus the limits of synthetic educational data.

Before the final run, do you expect the selected model to change when the data plus settings are unchanged?

• Regenerate the deterministic handoff by running the training command again:

python -m src.train_models

What should you see now?

The terminal prints the same deterministic metadata. The selected model comes from mean validation accuracy.

You can now inspect three reloadable Pipeline artifacts. You can also inspect the experiment report plus the Level 3 contract.

That is the handoff packaged. Level 3 can reproduce the context before making its first unbiased prediction on the held-out runners.

Your models now travel with their validation evidence. Next, you will prove this boundary with an isolated automated test suite.

Test the Training Boundary

Your three classifiers now have grouped validation evidence. Their fitted Pipelines are ready for the Level 3 handoff.

A written promise cannot prove that the held-out test set stayed untouched. In this step, Python unittest rebuilds the experiment inside isolated directories that contain only the three permitted training inputs.

In this step, get ready to:

• Build an isolated training fixture without the held-out test file.
• Test grouped folds plus saved Pipeline behavior.
• Prove contract completeness plus deterministic output.

Create an isolated training fixture

Each test starts with a fresh temporary project directory. The fixture copies only train.csv, schema_contract.json, and preprocessing.json into that directory.

Why omit the test file?

A successful training run without test.csv proves that model selection does not depend on the held-out partition. The missing file turns the evaluation boundary into an executable constraint.

• Switch back to VS Code from earlier.
• Select the existing tests folder in the Explorer sidebar.
• Click the New File icon in the Explorer toolbar.
• Enter test_training.py as the file name.
• Add the test dependencies to tests/test_training.py by pasting this code:

import json
import shutil
import unittest
from pathlib import Path
from tempfile import TemporaryDirectory

import joblib
from sklearn.pipeline import Pipeline

from src.train_models import (
    build_model_candidates,
    load_training_handoff,
    run_training,
)

What do these imports support?

• TemporaryDirectory creates a disposable project directory for each test.
• shutil copies the three permitted Level 1 inputs into that directory.
• joblib reloads each saved classifier artifact.
• The imported training helpers let the suite exercise the same production path used by the main command.

• Save tests/test_training.py.
• Check that the new test module loads by running this command in the terminal from earlier:

python -m unittest -v tests/test_training.py

What does this check prove?

Python imports the new module through unittest discovery. A successful status confirms that the imports resolve before the fixture is added.

The command completes without discovered tests. That is expected because the module currently contains dependencies only.

Does the module fail to load?

Confirm that the file sits at tests/test_training.py inside the existing repository. Run the command from the directory that contains src and tests.

Check each import against the code above. Help me diagnose an import failure in tests/test_training.py.

The fixture uses setUp() to create a clean directory before every test. It runs the complete training workflow inside that directory.

• Add the test class below the import block in tests/test_training.py by pasting this code:

class TrainingTests(unittest.TestCase):
    def setUp(self) -> None:
        self.source_root = Path(__file__).resolve().parents[1]
        self.temporary_directory = TemporaryDirectory()
        self.addCleanup(self.temporary_directory.cleanup)
        self.project_root = Path(self.temporary_directory.name)
        self.copy_handoff(self.project_root)
        self.metadata = run_training(self.project_root)

How does the fixture isolate training?

• source_root points to the completed repository that holds the original Level 1 handoff.
• temporary_directory creates a separate project root for the current test.
• addCleanup registers automatic deletion after the test finishes.
• run_training() exercises loading, validation, cross-validation, final fitting, persistence, and contract creation inside the isolated root.

• Save tests/test_training.py.
• Confirm that the class definition still loads by running:

python -m unittest -v tests/test_training.py

What does this run check?

The command parses the new class plus its setup method. Test methods arrive next, so unittest still has no assertions to execute.

The module completes successfully. Your fixture structure is valid.

Seeing a fixture error?

Check that setUp uses the same capitalization shown above. Confirm that every line inside the method has eight leading spaces.

Compare the imported helper name with run_training in src/train_models.py. Help me fix the TrainingTests fixture definition.

The copy helper maps each source artifact to its matching location inside the temporary project. The source mapping deliberately contains no path for test.csv.

• Add the source mapping below setUp() inside the TrainingTests class by pasting this code:

    def copy_handoff(self, destination: Path) -> None:
        sources = {
            "train": self.source_root
            / "data"
            / "v1"
            / "processed"
            / "train.csv",
            "schema": self.source_root
            / "contracts"
            / "v1"
            / "schema_contract.json",
            "preprocessing": self.source_root
            / "artifacts"
            / "v1"
            / "preprocessing.json",
        }

What does the source mapping enforce?

Each dictionary entry identifies one input consumed by Level 2 training. The exact three-entry boundary matches the fingerprints recorded in the experiment metadata.

• Save tests/test_training.py.
• Confirm that the source mapping parses by running:

python -m unittest -v tests/test_training.py

What does this run check?

Unittest imports the class with the new path mapping. A successful result confirms that the multiline Path expressions are valid Python.

The module loads successfully. The three allowed source paths are now defined.

Does the path mapping fail?

Check the indentation of each path segment. Confirm that the closing brace aligns with the sources assignment.

Make sure every source path matches the existing Level 1 repository. Help me troubleshoot the copy_handoff source mapping.

The destination mapping recreates the expected v1 directory structure under the temporary root. The final loop creates parent folders before copying each allowed input.

• Add the destination mapping directly below the sources dictionary by pasting this code:

        destinations = {
            "train": destination
            / "data"
            / "v1"
            / "processed"
            / "train.csv",
            "schema": destination
            / "contracts"
            / "v1"
            / "schema_contract.json",
            "preprocessing": destination
            / "artifacts"
            / "v1"
            / "preprocessing.json",
        }
        for name, source in sources.items():
            destinations[name].parent.mkdir(parents=True, exist_ok=True)
            shutil.copy2(source, destinations[name])

How are the isolated inputs created?

• destinations mirrors the paths expected by load_training_handoff().
• mkdir creates only the parent directories needed by each permitted artifact.
• copy2 places the three source files into the isolated project root.
• The helper never creates a held-out test path.

• Save tests/test_training.py.
• Check the completed fixture by running:

python -m unittest -v tests/test_training.py

What does this fixture check prove?

The module now contains the complete isolated setup. The next test method triggers that setup before its assertion runs.

The command completes successfully. Your temporary handoff builder is ready.

Does the fixture stop loading?

Confirm that destinations stays inside copy_handoff(). Check that the loop aligns with the dictionary assignment.

Verify that the tests folder remains inside the repository from earlier. Help me repair the temporary fixture paths.

Prove groups, predictions, and persistence

The first behavioral test inspects the recorded fold manifest. Every fold must have empty runner overlap.

The combined validation-runner set must also equal every training runner. Together these assertions prove group separation plus complete validation coverage.

• Add the grouped-fold test below copy_handoff() inside the TrainingTests class by pasting this code:

    def test_group_folds_have_no_overlap_and_cover_all_runners(self) -> None:
        train, contract, _ = load_training_handoff(self.project_root)
        manifest = self.metadata["cross_validation"]["fold_manifest"]
        validation_runners = set()
        for fold in manifest:
            self.assertEqual(fold["runner_overlap"], [])
            validation_runners.update(fold["validation_runners"])
        self.assertEqual(
            validation_runners,
            set(str(value) for value in train[contract["group_column"]].unique()),
        )

What does this test prove?

• The first assertion rejects any fold that places one runner on both sides.
• validation_runners accumulates every runner used for validation.
• The final assertion rejects a manifest that skips a runner or introduces an unknown runner.

• Save tests/test_training.py.
• Run the suite to execute the first boundary test:

python -m unittest -v tests/test_training.py

What should this run establish?

The test invokes the full training workflow inside a temporary project root. It passes only when all four folds remain group-safe plus collectively complete.

You should see one passing test. The temporary training run used no held-out test file.

Does the grouped-fold test fail?

Compare the manifest keys with cross_validation, fold_manifest, runner_overlap, and validation_runners in src/train_models.py.

Check that the test uses the contract's group_column when collecting the expected runners. Help me diagnose the grouped-fold assertion failure.

The majority baseline has one valid behavior to test. Every validation fold must receive predictions from exactly one class.

• Add the baseline behavior test below the grouped-fold test by pasting this code:

    def test_majority_baseline_predicts_one_class_in_every_fold(self) -> None:
        folds = self.metadata["results"]["majority_baseline"]["folds"]
        self.assertTrue(
            all(len(fold["unique_validation_predictions"]) == 1 for fold in folds)
        )

Why test one unique prediction?

The majority classifier ignores every feature. One unique prediction per validation fold confirms that the implementation still behaves as a constant-class floor.

• Save tests/test_training.py.
• Run the suite to check the baseline behavior:

python -m unittest -v tests/test_training.py

What does this run add?

The suite now checks group safety plus the baseline's constant output. Both tests rebuild the experiment from the isolated three-file handoff.

You should see two passing tests. The baseline remains a deliberately limited reference model.

Does the baseline test fail?

Confirm that the result path includes majority_baseline plus unique_validation_predictions. Check the spelling against src/train_models.py.

Make sure the assertion checks the length of each prediction list. Help me debug the majority baseline test.

Persistence only helps Level 3 when each saved artifact reloads as a complete Pipeline. A prediction-length check proves that the reconstructed object can process the full training feature frame.

• Add the Pipeline reload test below the baseline test by pasting this code:

    def test_saved_pipelines_reload_and_predict(self) -> None:
        train, contract, _ = load_training_handoff(self.project_root)
        for name in build_model_candidates():
            artifact = self.project_root / self.metadata["model_artifacts"][name]
            model = joblib.load(artifact)
            self.assertIsInstance(model, Pipeline)
            predictions = model.predict(train[contract["feature_columns"]])
            self.assertEqual(len(predictions), len(train))

What does the persistence test cover?

• build_model_candidates() supplies the exact three artifact names expected from training.
• joblib.load reconstructs each trusted local Pipeline.
• The type assertion rejects an artifact that does not contain a Pipeline.
• The length assertion proves that each reloaded model predicts one output per input row.

• Save tests/test_training.py.
• Run the suite to verify all three saved Pipelines:

python -m unittest -v tests/test_training.py

What does this run add?

The new test reloads the baseline, logistic regression, and decision tree artifacts. It calls predict without fitting again.

You should see three passing tests. Every persisted classifier is ready for direct Level 3 prediction.

Does a saved Pipeline fail to reload?

Confirm that run_training() completes before the test reads model_artifacts. Check that the active environment uses the pinned dependencies from requirements.txt.

Only load the artifacts created inside this trusted temporary fixture. Help me diagnose a Pipeline reload or prediction failure.

The strongest holdout check examines the temporary filesystem itself. It also verifies the explicit boundary statement written by the training program.

• Add the held-out-file independence test below the reload test by pasting this code:

    def test_training_does_not_require_the_held_out_test_file(self) -> None:
        test_path = self.project_root / "data" / "v1" / "processed" / "test.csv"
        self.assertFalse(test_path.exists())
        self.assertEqual(
            self.metadata["held_out_test_set"]["status"],
            "not loaded, fingerprinted, or scored in Level 2",
        )

How does this make the boundary executable?

The first assertion proves that the isolated training root has no held-out test file. The second assertion checks that the generated metadata preserves the Level 3 reservation.

• Save tests/test_training.py.
• Run the suite to prove that training succeeds without the held-out file:

python -m unittest -v tests/test_training.py

What does this run establish?

The suite reaches persisted models plus metadata while test.csv is absent from the fixture. That proves Level 2 has no runtime dependency on the final holdout.

You should see four passing tests. The held-out test boundary now has direct automated evidence.

Does training look for test.csv?

Inspect handoff_paths() in src/train_models.py. Its returned mapping must contain only the training CSV plus the schema and preprocessing artifacts.

Check that copy_handoff() has no test destination. Help me find why isolated training depends on test.csv.

Verify contract completeness and determinism

Level 3 needs a contract that names every consumed input plus every persisted model. The contract test checks those required fields directly.

• Add the Level 3 contract test below the held-out-file test by pasting this code:

    def test_level3_contract_is_complete(self) -> None:
        contract_path = (
            self.project_root
            / "artifacts"
            / "v1"
            / "level3_evaluation_contract.json"
        )
        level3 = json.loads(contract_path.read_text(encoding="utf-8"))
        self.assertEqual(
            set(level3["input_fingerprints"]),
            {"train", "schema", "preprocessing"},
        )
        self.assertEqual(set(level3["models"]), set(build_model_candidates()))
        self.assertIn(level3["selected_model"], level3["models"])
        self.assertEqual(level3["random_seeds"]["group_kfold"], 42)

What does the contract test require?

• The fingerprint keys must name exactly the three consumed Level 1 inputs.
• The model keys must match all three training candidates.
• The selected model must point to one of those persisted candidates.
• The grouped-fold seed must preserve reproducible split ordering.

• Save tests/test_training.py.
• Run the suite to validate the machine-readable handoff:

python -m unittest -v tests/test_training.py

What does this contract check prove?

The fifth test reads the generated contract from the isolated project. It rejects missing fingerprints, missing models, an invalid selection, or a changed fold seed.

You should see five passing tests. The Level 3 contract contains the required experiment boundary.

Is the contract test failing?

Compare the contract path with artifacts/v1/level3_evaluation_contract.json. Confirm that the parsed object uses input_fingerprints, models, selected_model, and random_seeds.

Check those keys against build_level3_contract() in src/train_models.py. Help me diagnose the Level 3 contract assertion failure.

Reproducibility requires more than using the same random seed. Two clean project roots must produce identical metadata plus identical Level 3 contracts.

• Add the deterministic metadata comparison below the contract test by pasting this code:

    def test_experiment_and_contract_are_deterministic(self) -> None:
        first_contract_path = (
            self.project_root
            / "artifacts"
            / "v1"
            / "level3_evaluation_contract.json"
        )
        first_contract = json.loads(
            first_contract_path.read_text(encoding="utf-8")
        )
        with TemporaryDirectory() as second_directory:
            second_root = Path(second_directory)
            self.copy_handoff(second_root)
            second_metadata = run_training(second_root)
            second_contract = json.loads(
                (
                    second_root
                    / "artifacts"
                    / "v1"
                    / "level3_evaluation_contract.json"
                ).read_text(encoding="utf-8")
            )
        self.assertEqual(
            json.dumps(self.metadata, sort_keys=True),
            json.dumps(second_metadata, sort_keys=True),
        )

How does the repeat run test metadata?

The method builds a second isolated root from the same three inputs. It reruns training before comparing canonical JSON strings with sorted keys.

• Save tests/test_training.py.
• Run the suite to compare metadata across two temporary roots:

python -m unittest -v tests/test_training.py

What does this repeat run establish?

The sixth test now reruns the complete experiment in a second directory. Its first assertion rejects any metadata drift between those clean runs.

You should see six passing tests. The experiment metadata is deterministic across isolated roots.

Does the metadata comparison differ?

Inspect runtime values plus artifact paths for machine-specific or temporary-directory content. Confirm that persisted paths remain relative to each project root.

Check that every stochastic training component uses the recorded integer seed. Help me locate nondeterministic metadata fields.

The final assertion applies the same deterministic requirement to the Level 3 contract. The module entry point also keeps the file runnable through Python unittest.

• Complete the deterministic contract check at the bottom of tests/test_training.py by pasting this code:

        self.assertEqual(
            json.dumps(first_contract, sort_keys=True),
            json.dumps(second_contract, sort_keys=True),
        )


if __name__ == "__main__":
    unittest.main()

What does the final assertion add?

The assertion rejects drift in the machine-readable Level 3 contract. The final guard starts unittest when the module runs as the main Python program.

• Save tests/test_training.py.

Before you run the final check, do you expect all six tests to pass without reading the held-out test file?

• Run the complete training-boundary suite with verbose output:

python -m unittest -v tests/test_training.py

What does the final command verify?

The verbose unittest run executes every method in TrainingTests. Each method receives a new three-input fixture plus a fresh training run.

You should see six tests followed by OK. That result proves group separation, baseline behavior, Pipeline reloadability, holdout independence, contract completeness, and deterministic output.

That is the training boundary locked down. Level 3 can now perform the first held-out evaluation without retraining or guessing how these artifacts were produced.

Does the final suite fail?

Use the first failing test name to identify the affected boundary. Compare that test against its matching method in the full-file reference below.

Confirm that the active environment still uses the pinned dependencies in requirements.txt. Help me debug the failing training-boundary suite.

✔️ Awesome, I've got everything!

Great work. Double check that tests/test_training.py is saved beside the existing Level 1 tests.

ⓧ I'd like to double check the full code

Compare your complete tests/test_training.py file against this reference.

import json
import shutil
import unittest
from pathlib import Path
from tempfile import TemporaryDirectory

import joblib
from sklearn.pipeline import Pipeline

from src.train_models import (
    build_model_candidates,
    load_training_handoff,
    run_training,
)


class TrainingTests(unittest.TestCase):
    def setUp(self) -> None:
        self.source_root = Path(__file__).resolve().parents[1]
        self.temporary_directory = TemporaryDirectory()
        self.addCleanup(self.temporary_directory.cleanup)
        self.project_root = Path(self.temporary_directory.name)
        self.copy_handoff(self.project_root)
        self.metadata = run_training(self.project_root)

    def copy_handoff(self, destination: Path) -> None:
        sources = {
            "train": self.source_root
            / "data"
            / "v1"
            / "processed"
            / "train.csv",
            "schema": self.source_root
            / "contracts"
            / "v1"
            / "schema_contract.json",
            "preprocessing": self.source_root
            / "artifacts"
            / "v1"
            / "preprocessing.json",
        }
        destinations = {
            "train": destination
            / "data"
            / "v1"
            / "processed"
            / "train.csv",
            "schema": destination
            / "contracts"
            / "v1"
            / "schema_contract.json",
            "preprocessing": destination
            / "artifacts"
            / "v1"
            / "preprocessing.json",
        }
        for name, source in sources.items():
            destinations[name].parent.mkdir(parents=True, exist_ok=True)
            shutil.copy2(source, destinations[name])

    def test_group_folds_have_no_overlap_and_cover_all_runners(self) -> None:
        train, contract, _ = load_training_handoff(self.project_root)
        manifest = self.metadata["cross_validation"]["fold_manifest"]
        validation_runners = set()
        for fold in manifest:
            self.assertEqual(fold["runner_overlap"], [])
            validation_runners.update(fold["validation_runners"])
        self.assertEqual(
            validation_runners,
            set(str(value) for value in train[contract["group_column"]].unique()),
        )

    def test_majority_baseline_predicts_one_class_in_every_fold(self) -> None:
        folds = self.metadata["results"]["majority_baseline"]["folds"]
        self.assertTrue(
            all(len(fold["unique_validation_predictions"]) == 1 for fold in folds)
        )

    def test_saved_pipelines_reload_and_predict(self) -> None:
        train, contract, _ = load_training_handoff(self.project_root)
        for name in build_model_candidates():
            artifact = self.project_root / self.metadata["model_artifacts"][name]
            model = joblib.load(artifact)
            self.assertIsInstance(model, Pipeline)
            predictions = model.predict(train[contract["feature_columns"]])
            self.assertEqual(len(predictions), len(train))

    def test_training_does_not_require_the_held_out_test_file(self) -> None:
        test_path = self.project_root / "data" / "v1" / "processed" / "test.csv"
        self.assertFalse(test_path.exists())
        self.assertEqual(
            self.metadata["held_out_test_set"]["status"],
            "not loaded, fingerprinted, or scored in Level 2",
        )

    def test_level3_contract_is_complete(self) -> None:
        contract_path = (
            self.project_root
            / "artifacts"
            / "v1"
            / "level3_evaluation_contract.json"
        )
        level3 = json.loads(contract_path.read_text(encoding="utf-8"))
        self.assertEqual(
            set(level3["input_fingerprints"]),
            {"train", "schema", "preprocessing"},
        )
        self.assertEqual(set(level3["models"]), set(build_model_candidates()))
        self.assertIn(level3["selected_model"], level3["models"])
        self.assertEqual(level3["random_seeds"]["group_kfold"], 42)

    def test_experiment_and_contract_are_deterministic(self) -> None:
        first_contract_path = (
            self.project_root
            / "artifacts"
            / "v1"
            / "level3_evaluation_contract.json"
        )
        first_contract = json.loads(
            first_contract_path.read_text(encoding="utf-8")
        )
        with TemporaryDirectory() as second_directory:
            second_root = Path(second_directory)
            self.copy_handoff(second_root)
            second_metadata = run_training(second_root)
            second_contract = json.loads(
                (
                    second_root
                    / "artifacts"
                    / "v1"
                    / "level3_evaluation_contract.json"
                ).read_text(encoding="utf-8")
            )
        self.assertEqual(
            json.dumps(self.metadata, sort_keys=True),
            json.dumps(second_metadata, sort_keys=True),
        )
        self.assertEqual(
            json.dumps(first_contract, sort_keys=True),
            json.dumps(second_contract, sort_keys=True),
        )


if __name__ == "__main__":
    unittest.main()

How should I use this reference?

Compare the imports, fixture paths, six test methods, indentation, and final entry point. Save your file after resolving every difference.

Secret mission

Control Tree Capacity

An unrestricted decision tree can keep splitting until it captures small details from its training runners. In this mission, you will compare three capacity settings across the same four group-safe folds. You will use the observed training fit and generalization gaps to explain the bias-overfitting trade-off.

Clean Up Your Resources

Clean Up Your Resources

Everything in this project runs locally, so there are no ongoing costs. Choose whether to keep the Level 2 handoff, pause the virtual environment, or delete the Level 2 additions.

Resources you used:

• Three trusted scikit-learn Pipeline artifacts at models/v1/majority_baseline.joblib, models/v1/logistic_regression.joblib, and models/v1/decision_tree.joblib.
• Experiment reports at reports/v1/training_experiment.json and reports/v1/tree_capacity_experiment.json.
• The Level 3 evaluation contract at artifacts/v1/level3_evaluation_contract.json.
• The training document at docs/level-2-training.md.
• Level 2 source modules at src/train_models.py and src/capacity_experiment.py.
• The automated training test module at tests/test_training.py.

Keep everything running

No action needed. Choose this if you want to preserve the complete handoff for Level 3 or continue comparing model behavior.

• Keep the three trusted model artifacts so Level 3 can evaluate them without retraining.
• Preserve both experiment reports with the Level 3 contract so the validation evidence remains reproducible.
• Leave requirements.txt pinned so the saved models stay paired with their recorded runtime versions.

Pause - I'll come back to this later

Exit the active Python virtual environment to return your shell to its normal context. Every local file remains unchanged.

• Switch back to the terminal you used for this project.
• Exit the active .venv by running this command:

deactivate

What Does This Command Do?

This exits the active virtual environment. Your models, reports, source code, tests, and Level 3 contract stay on disk.

• Confirm that your terminal prompt no longer starts with (.venv).

Is the Virtual Environment Already Inactive?

If your prompt does not show (.venv), there is no active environment to exit. Your project is already paused.

Help me confirm whether my Python virtual environment is active.

Delete - I don't want to use this again

Remove only the files introduced in Level 2 to return to the Level 1 handoff. Keep the original v1 data, schema contract, preprocessing artifact, and held-out test file.

Protect the Level 1 Handoff

This deletion is deliberately narrow, so the immutable Level 1 inputs remain available for a clean Level 2 restart.

• Switch back to the Level 1 repository in VS Code.
• Use the Explorer sidebar context menu to delete majority_baseline.joblib, logistic_regression.joblib, and decision_tree.joblib from models/v1/.
• Use the Explorer sidebar context menu to delete training_experiment.json and tree_capacity_experiment.json from reports/v1/.
• Use the Explorer sidebar context menu to delete level3_evaluation_contract.json from artifacts/v1/.
• Use the Explorer sidebar context menu to delete level-2-training.md from docs/.
• Use the Explorer sidebar context menu to delete train_models.py and capacity_experiment.py from src/.
• Use the Explorer sidebar context menu to delete test_training.py from tests/.
• Refresh the Explorer sidebar to verify the deletion.

You should no longer see the model artifacts, reports, Level 3 contract, training document, Level 2 source modules, or training test module. You should still see the Level 1 v1 handoff files and the pinned requirements.txt.

Nice Work!

Nice Work!

Excellent work! Your three-classifier training system now has group-safe evidence, reloadable artifacts, and an untouched Level 3 holdout.

You've learned how to:

• Designed group-aware cross-validation with GroupKFold across four shared folds. Every runner appears in validation exactly once without overlapping that fold's fit runners.
• Compared a majority-class baseline, scaled logistic regression, and an unrestricted decision tree across identical folds. Training accuracy shows fit. Validation accuracy shows transfer to unseen runners. Their difference exposes the generalization gap.
• Implemented model persistence through three trusted Pipeline artifacts. The Level 3 contract records fingerprints, model settings, seeds, metrics, runtime versions, and the predict-without-fit boundary. Six isolated tests prove the training workflow succeeds without test.csv.
• Secret Mission: Compared three decision-tree capacities on the same grouped folds. Observed depths reveal the effective complexity. Validation variability and generalization gaps show how capacity limits change transfer. The report avoids assuming monotonic improvement.

Ready to quiz yourself?