Evaluate a Running-Readiness Classifier

Audit a running-readiness classifier with one-time holdout evaluation.

Introduction

30 Second Summary

A model can look accurate on paper while making the mistakes that matter most. If final test results influence another round of choices, that evidence stops being trustworthy.

In this project, you will evaluate the Level 2 running-readiness binary classifier against its untouched held-out runners exactly once. The write-once workflow preserves complete metrics, row-level errors, fingerprints, limitations, plus a consumed-holdout contract as reproducible evidence.

What You'll Build

You will demo a timestamped evaluation report that compares the frozen model with its majority baseline without reopening the final test set for tuning.

By the end of this project, you'll have:

  • A selection lock that proves the validation winner was frozen before test access. Its dataset plus model fingerprints expose later changes to the evidence.
  • A model-versus-baseline report that shows accuracy, precision, recall, F1, class support, plus both confusion matrices. Its error CSV separates false positives from false negatives while preserving their feature values.
  • A consumed-holdout contract that blocks Level 4 from using the final labels for feature selection, threshold tuning, model comparison, or model selection.
  • Secret Mission: Preserve group-safe out-of-fold probabilities. Compare thresholds 0.30 through 0.70. Propose a validation-selected F1 threshold without reading test.csv.

Do I need any experience before starting?

This project assumes you completed Levels 1 and 2 of the running-readiness repository. You will reuse the existing .venv with its pinned dependencies.

The repository must also contain the fitted Pipeline artifacts, datasets, experiment metadata, plus the Level 3 evaluation contract.

Before We Start

Level 3 begins from your existing running-readiness repository in VS Code with its current .venv and pinned dependencies preserved for verification. The Level 3 boundary freezes the model named by artifacts/v1/level3_evaluation_contract.json from Level 2 grouped validation evidence before opening data/v1/processed/test.csv for final evaluation.

Verify the Level 2 Runtime

Final evidence only deserves trust when the Level 2 Python runtime still reproduces its earlier behavior. That proof must happen before any held-out label is read.

This is a read-only preflight for the existing repository. Nothing in the repository is regenerated or edited.

The held-out data/v1/processed/test.csv file remains closed throughout this step.

In this step, get ready to:
  • Activate the existing .venv for this repository.
  • Confirm the four pinned dependency versions from imported modules.
  • Run the preserved Level 1 and Level 2 test suites.
Prepare a read-only terminal session

The existing virtual environment contains the dependency set used by Level 2. Activating it keeps this verification tied to that original runtime.

  • In the existing VS Code window, create an integrated terminal from the top menu.
  • Activate the existing .venv by running this command:
source .venv/bin/activate

What does this command do?

  • The source command applies the activation script to your current terminal session.
  • The .venv/bin/activate path selects the environment already stored inside this repository.

Environment not activating?

  • Confirm the terminal is attached to the existing repository.
  • Confirm the repository still contains the .venv directory.
  • Stop before creating a replacement environment.

Ask for help with the existing environment if activation still fails: help me diagnose why the existing .venv does not activate from my repository.

Match the pinned dependency set

The dependency set is part of the reproducibility boundary. NumPy 2.5.4, pandas 3.1.0, scikit-learn 1.9.1, and joblib 1.6.0 must import from the activated environment.

  • Print the imported package versions by running this command:
python -c "import joblib, numpy, pandas, sklearn; print(numpy.__version__, pandas.__version__, sklearn.__version__, joblib.__version__)"

What does this check prove?

  • The imports prove that all four packages are available to the active Python interpreter.
  • The __version__ values reveal the exact library versions loaded by that interpreter.
  • Matching all four pins confirms that the current terminal uses the Level 2 dependency set.

You should see 2.5.4 3.1.0 1.9.1 1.6.0 in that order.

That is the runtime boundary secured. All four libraries match the Level 2 handoff.

Imports failing or versions different?

A missing import or different version means the active terminal does not match the preserved Level 2 runtime.

  • Return to the activation check above.
  • Stop before opening data/v1/processed/test.csv.
  • Leave every v1 dataset unchanged.
  • Leave every model artifact unchanged.

Use this prompt to investigate the mismatch without replacing the environment: help me trace why the active environment loads different pinned versions.

Check the preserved test boundary

The existing Python unittest suites cover the Level 1 data pipeline behavior and the Level 2 training behavior. Passing both suites confirms that the pinned runtime still supports the preserved repository.

Before you run the suites, do you expect the preserved checks to pass without rebuilding any artifact?

  • Run both existing test modules with verbose output by running this command:
python -m unittest -v tests/test_data_pipeline.py tests/test_training.py

What does this test command cover?

  • The python -m unittest runner executes the preserved automated checks.
  • The -v flag prints each test result so failures are easy to locate.
  • The tests/test_data_pipeline.py module checks the Level 1 data pipeline.
  • The tests/test_training.py module checks the Level 2 training workflow.

You should see every existing test pass. The final summary ends with OK.

That is the preflight complete. Your Level 1 data behavior and Level 2 training behavior still pass unchanged.

A test did not pass?

A failure means the preflight has caught a repository or runtime mismatch before held-out access.

  • Return to the dependency check if an import or version differs.
  • Stop before regenerating any v1 dataset.
  • Stop before replacing any model artifact.
  • Keep data/v1/processed/test.csv closed.

Use the failing test output to request targeted help: help me diagnose the failing Level 1 or Level 2 test without modifying evaluation-only data.

Your repository has passed its read-only preflight. Next, you will freeze the validation winner before the held-out test is opened.

Lock the Validation Winner

Your Level 2 runtime has passed every check. The next boundary freezes the validation winner before test.csv can reveal its final performance.

Validation accuracy can select a candidate. It cannot reveal the candidate's false-positive burden or false-negative burden. This step preserves that missing evidence for the held-out evaluation.

In this step, get ready to:
  • Add the write-once evaluation module to the existing repository.
  • Verify the Level 2 handoff through SHA-256 fingerprints and grouped-fold evidence.
  • Create the selection lock before opening the held-out test set.
Create the evaluation module

The evaluation module owns the complete Level 3 boundary. Its lock path uses exclusive file creation so a later command cannot silently replace the original selection decision.

  • Switch back to the existing repository in VS Code.
  • Create evaluate_models.py inside the existing src/ folder using the file sidebar.
  • Use the full-code reference below to replace the empty file contents.

✔️ Awesome, I've got everything!

Your evaluation module now contains the lock boundary plus the functions used by later steps.

ⓧ I'd like to double check the full code

  • Compare src/evaluate_models.py with this complete file:
from __future__ import annotations

import ast
import inspect
import json
import sys
from datetime import datetime, timezone
from hashlib import sha256
from pathlib import Path
from typing import Any

import joblib
import pandas as pd
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

DATASET_VERSION = "v1"
BASELINE_NAME = "majority_baseline"
LOCK_RELATIVE_PATH = Path("artifacts/v1/level3_selection_lock.json")
REPORT_RELATIVE_PATH = Path("reports/v1/level3_evaluation.json")
MARKDOWN_RELATIVE_PATH = Path("reports/v1/level3_evaluation.md")
ERRORS_RELATIVE_PATH = Path("reports/v1/level3_errors.csv")
HOLDOUT_MARKER_RELATIVE_PATH = Path("artifacts/v1/holdout_consumed.json")
LEVEL4_RELATIVE_PATH = Path("artifacts/v1/level4_features_contract.json")

FORBIDDEN_HOLDOUT_USES = [
    "feature_selection",
    "threshold_tuning",
    "model_comparison",
    "model_selection",
]
LIMITATIONS = [
    "The runner records, readiness rule, and labels are synthetic educational constructs, not evidence about real physiology.",
    "One finite v1 holdout does not prove performance across new populations, devices, seasons, or training plans.",
    "Precision and recall depend on class prevalence and the operating threshold.",
    "False positives and false negatives have no validated clinical or coaching cost in this synthetic task.",
    "This evaluation supports an educational software-engineering claim, not a medical, safety, or causal claim.",
]


def fingerprint(path: Path) -> str:
    return sha256(path.read_bytes()).hexdigest()


def read_json(path: Path) -> dict[str, Any]:
    return json.loads(path.read_text(encoding="utf-8"))


def write_json_exclusive(path: Path, payload: dict[str, Any]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("x", encoding="utf-8") as handle:
        json.dump(payload, handle, indent=2)
        handle.write("\n")


def level2_paths(root: Path) -> dict[str, Path]:
    return {
        "contract": root / "artifacts/v1/level3_evaluation_contract.json",
        "training_report": root / "reports/v1/training_experiment.json",
    }


def verify_level2_handoff(root: Path) -> tuple[dict[str, Any], dict[str, Any], pd.DataFrame]:
    paths = level2_paths(root)
    missing = [path.as_posix() for path in paths.values() if not path.exists()]
    if missing:
        raise FileNotFoundError(f"Missing Level 2 handoff files: {missing}")
    contract = read_json(paths["contract"])
    metadata = read_json(paths["training_report"])
    if contract["dataset_version"] != DATASET_VERSION:
        raise ValueError("Expected dataset v1")
    if contract["experiment_id"] != metadata["experiment_id"]:
        raise ValueError("Experiment identifiers do not match")
    for name, evidence in contract["input_fingerprints"].items():
        path = root / evidence["path"]
        if not path.exists() or fingerprint(path) != evidence["sha256"]:
            raise ValueError(f"Fingerprint mismatch for Level 2 input: {name}")
    train = pd.read_csv(root / contract["input_fingerprints"]["train"]["path"])
    group = contract["group_column"]
    manifest = metadata["cross_validation"]["fold_manifest"]
    if len(manifest) != metadata["cross_validation"]["n_splits"]:
        raise ValueError("Fold count mismatch")
    if any(fold["runner_overlap"] for fold in manifest):
        raise ValueError("Runner overlap exists in Level 2 folds")
    validation_groups = [str(value) for fold in manifest for value in fold["validation_runners"]]
    if len(validation_groups) != len(set(validation_groups)):
        raise ValueError("A runner appears in multiple validation folds")
    if set(validation_groups) != {str(value) for value in train[group].unique()}:
        raise ValueError("Validation folds do not cover all training runners")
    results = metadata["results"]
    winner = max(results, key=lambda name: results[name]["aggregate"]["mean_validation_accuracy"])
    if metadata["selected_by_mean_validation_accuracy"] != winner:
        raise ValueError("Training report selection is not the validation winner")
    if contract["selected_model"] != winner:
        raise ValueError("Contract selection differs from validation evidence")
    if BASELINE_NAME not in contract["models"]:
        raise ValueError("Majority baseline is missing")
    for name, spec in contract["models"].items():
        if not (root / spec["artifact_path"]).exists():
            raise FileNotFoundError(f"Missing model artifact: {name}")
    return contract, metadata, train


def paths_as_relative(path: Path, root: Path) -> str:
    return path.relative_to(root).as_posix()


def lock_selection(root: Path) -> dict[str, Any]:
    path = root / LOCK_RELATIVE_PATH
    if path.exists():
        raise FileExistsError("The selection lock cannot be rewritten")
    contract, metadata, _ = verify_level2_handoff(root)
    selected = contract["selected_model"]
    names = [selected] if selected == BASELINE_NAME else [selected, BASELINE_NAME]
    paths = level2_paths(root)
    payload = {
        "lock_version": "1.0.0",
        "dataset_version": DATASET_VERSION,
        "experiment_id": contract["experiment_id"],
        "selection_basis": "Level 2 mean_validation_accuracy only",
        "selection_metric": "mean_validation_accuracy",
        "selected_model": selected,
        "selected_validation_metrics": contract["models"][selected]["validation_metrics"],
        "majority_baseline_validation_metrics": contract["models"][BASELINE_NAME]["validation_metrics"],
        "level2_contract": {"path": paths_as_relative(paths["contract"], root), "sha256": fingerprint(paths["contract"])},
        "training_report": {"path": paths_as_relative(paths["training_report"], root), "sha256": fingerprint(paths["training_report"])},
        "input_fingerprints": contract["input_fingerprints"],
        "model_artifacts": {name: {"path": contract["models"][name]["artifact_path"], "sha256": fingerprint(root / contract["models"][name]["artifact_path"])} for name in names},
        "held_out_test_status": "not read during selection lock",
        "test_metrics_available": False,
        "selection_is_final": True,
        "level2_selected_field": metadata["selected_by_mean_validation_accuracy"],
    }
    write_json_exclusive(path, payload)
    return payload


def verify_selection_lock(root: Path) -> tuple[dict[str, Any], dict[str, Any], pd.DataFrame, dict[str, Any]]:
    contract, metadata, train = verify_level2_handoff(root)
    path = root / LOCK_RELATIVE_PATH
    if not path.exists():
        raise FileNotFoundError("Create the Level 3 selection lock first")
    lock = read_json(path)
    if not lock["selection_is_final"] or lock["selected_model"] != contract["selected_model"]:
        raise ValueError("Locked selection is invalid")
    paths = level2_paths(root)
    if lock["level2_contract"]["sha256"] != fingerprint(paths["contract"]):
        raise ValueError("Level 3 contract changed after locking")
    if lock["training_report"]["sha256"] != fingerprint(paths["training_report"]):
        raise ValueError("Training report changed after locking")
    for name, evidence in lock["model_artifacts"].items():
        if evidence["sha256"] != fingerprint(root / evidence["path"]):
            raise ValueError(f"Model changed after locking: {name}")
    return contract, metadata, train, lock


def metric_bundle(y_true: Any, y_pred: Any) -> dict[str, Any]:
    matrix = confusion_matrix(y_true, y_pred, labels=[0, 1])
    report = classification_report(y_true, y_pred, labels=[0, 1], target_names=["class_0", "class_1"], output_dict=True, zero_division=0.0)
    tn, fp, fn, tp = [int(value) for value in matrix.ravel().tolist()]
    return {"accuracy": float(accuracy_score(y_true, y_pred)), "positive_class": {"precision": float(report["class_1"]["precision"]), "recall": float(report["class_1"]["recall"]), "f1": float(report["class_1"]["f1-score"]), "support": int(report["class_1"]["support"])}, "class_support": {"0": int(report["class_0"]["support"]), "1": int(report["class_1"]["support"])}, "confusion_matrix": {"labels": [0, 1], "layout": "[[TN, FP], [FN, TP]]", "matrix": [[int(value) for value in row] for row in matrix.tolist()], "true_negatives": tn, "false_positives": fp, "false_negatives": fn, "true_positives": tp}}


def build_error_rows(test: pd.DataFrame, predictions: list[int], features: list[str], target: str, group: str) -> pd.DataFrame:
    rows = test.reset_index(drop=True).copy()
    rows.insert(0, "row_id", list(range(len(rows))))
    rows["actual"] = rows[target].astype(int)
    rows["predicted"] = predictions
    rows["error_type"] = ["false_positive" if actual == 0 and predicted == 1 else "false_negative" if actual == 1 and predicted == 0 else "correct" for actual, predicted in zip(rows["actual"], rows["predicted"])]
    return rows[rows["error_type"] != "correct"][["row_id", group, *features, "actual", "predicted", "error_type"]].copy()


def markdown_report(report: dict[str, Any]) -> str:
    selected = report["models"]["selected"]
    baseline = report["models"]["majority_baseline"]
    sm, bm = selected["metrics"], baseline["metrics"]
    limitations = "\n".join(f"- {item}" for item in report["limitations"])
    return f"""# Level 3 Evaluate: Running-Readiness Classifier

Evaluation timestamp, UTC: `{report['evaluated_at_utc']}`

The model `{selected['name']}` was selected from Level 2 validation before `test.csv` was read. The held-out operating point is the frozen Pipeline `predict` rule. It was not tuned on test labels.

## Held-out metrics

| Model | Accuracy | Precision 1 | Recall 1 | F1 1 | Support 0 | Support 1 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `{selected['name']}` | {sm['accuracy']:.3f} | {sm['positive_class']['precision']:.3f} | {sm['positive_class']['recall']:.3f} | {sm['positive_class']['f1']:.3f} | {sm['class_support']['0']} | {sm['class_support']['1']} |
| `{baseline['name']}` | {bm['accuracy']:.3f} | {bm['positive_class']['precision']:.3f} | {bm['positive_class']['recall']:.3f} | {bm['positive_class']['f1']:.3f} | {bm['class_support']['0']} | {bm['class_support']['1']} |

Selected `[[TN, FP], [FN, TP]]`: `{sm['confusion_matrix']['matrix']}`

Baseline `[[TN, FP], [FN, TP]]`: `{bm['confusion_matrix']['matrix']}`

Undefined precision, recall, and F1 values use `zero_division=0.0`. A false positive flags recovery when the synthetic label is 0. A false negative clears a runner when it is 1. Accuracy alone can reward a majority predictor that misses the minority class.

## Reproducibility

- Test SHA-256: `{report['test_data']['sha256']}`
- Selected model SHA-256: `{selected['sha256']}`
- Selection lock SHA-256: `{report['selection_lock']['sha256']}`
- Consumed marker: `{report['holdout_consumed_marker']['path']}`
- Train-test runner overlap: `{report['group_safety']['runner_overlap_count']}`

## Limitations

{limitations}
"""


def run_evaluation(root: Path) -> dict[str, Any]:
    outputs = [REPORT_RELATIVE_PATH, MARKDOWN_RELATIVE_PATH, ERRORS_RELATIVE_PATH, HOLDOUT_MARKER_RELATIVE_PATH, LEVEL4_RELATIVE_PATH]
    existing = [(root / path).as_posix() for path in outputs if (root / path).exists()]
    if existing:
        raise FileExistsError(f"Refusing repeated held-out scoring: {existing}")
    contract, metadata, train, lock = verify_selection_lock(root)
    selected_name = lock["selected_model"]
    features, target, group = list(contract["feature_columns"]), contract["target_column"], contract["group_column"]
    test_path = root / contract["held_out_test_set"]["path"]
    test_sha256 = fingerprint(test_path)
    test = pd.read_csv(test_path)
    missing = sorted(set([*features, target, group]) - set(test.columns))
    if missing:
        raise ValueError(f"Held-out test is missing columns: {missing}")
    labels = {int(value) for value in test[target].unique()}
    if not labels.issubset({0, 1}):
        raise ValueError(f"Expected binary labels, found {labels}")
    overlap = sorted({str(value) for value in train[group].unique()}.intersection({str(value) for value in test[group].unique()}))
    if overlap:
        raise ValueError(f"Train and test runners overlap: {overlap}")
    selected_spec, baseline_spec = contract["models"][selected_name], contract["models"][BASELINE_NAME]
    selected_model = joblib.load(root / selected_spec["artifact_path"])
    baseline_model = joblib.load(root / baseline_spec["artifact_path"])
    selected_predictions = [int(value) for value in selected_model.predict(test[features])]
    baseline_predictions = [int(value) for value in baseline_model.predict(test[features])]
    y_true = test[target].astype(int)
    selected_metrics, baseline_metrics = metric_bundle(y_true, selected_predictions), metric_bundle(y_true, baseline_predictions)
    errors = build_error_rows(test, selected_predictions, features, target, group)
    evaluated_at_utc = datetime.now(timezone.utc).isoformat()
    report = {"report_version": "1.1.0", "experiment_id": "level-3-evaluate-v1", "evaluated_at_utc": evaluated_at_utc, "dataset_version": DATASET_VERSION, "holdout_consumed": True, "evaluation_status": "first and only planned held-out evaluation completed", "selection_basis": lock["selection_basis"], "selection_lock": {"path": LOCK_RELATIVE_PATH.as_posix(), "sha256": fingerprint(root / LOCK_RELATIVE_PATH)}, "operating_point": {"held_out_prediction_method": "Pipeline.predict", "held_out_threshold_role": "frozen default operating rule, not tuned on test labels", "exploratory_threshold_analysis_is_separate": True}, "metric_policy": {"labels": [0, 1], "positive_class": 1, "zero_division": 0.0, "confusion_matrix_layout": "[[TN, FP], [FN, TP]]"}, "test_data": {"path": paths_as_relative(test_path, root), "sha256": test_sha256, "row_count": int(len(test)), "runner_count": int(test[group].nunique()), "class_support": selected_metrics["class_support"]}, "group_safety": {"runner_overlap": overlap, "runner_overlap_count": len(overlap)}, "models": {"selected": {"name": selected_name, "path": selected_spec["artifact_path"], "sha256": fingerprint(root / selected_spec["artifact_path"]), "fit_calls_during_evaluation": 0, "metrics": selected_metrics}, "majority_baseline": {"name": BASELINE_NAME, "path": baseline_spec["artifact_path"], "sha256": fingerprint(root / baseline_spec["artifact_path"]), "fit_calls_during_evaluation": 0, "metrics": baseline_metrics}}, "comparison_to_majority_baseline": {"accuracy_delta": selected_metrics["accuracy"] - baseline_metrics["accuracy"], "positive_precision_delta": selected_metrics["positive_class"]["precision"] - baseline_metrics["positive_class"]["precision"], "positive_recall_delta": selected_metrics["positive_class"]["recall"] - baseline_metrics["positive_class"]["recall"], "positive_f1_delta": selected_metrics["positive_class"]["f1"] - baseline_metrics["positive_class"]["f1"]}, "error_analysis": {"path": ERRORS_RELATIVE_PATH.as_posix(), "row_count": int(len(errors)), "false_positive_count": selected_metrics["confusion_matrix"]["false_positives"], "false_negative_count": selected_metrics["confusion_matrix"]["false_negatives"]}, "holdout_consumed_marker": {"path": HOLDOUT_MARKER_RELATIVE_PATH.as_posix()}, "limitations": LIMITATIONS, "level4_contract_path": LEVEL4_RELATIVE_PATH.as_posix(), "level2_validation_results_preserved": metadata["results"]}
    errors_path = root / ERRORS_RELATIVE_PATH
    errors_path.parent.mkdir(parents=True, exist_ok=True)
    errors.to_csv(errors_path, index=False, mode="x")
    write_json_exclusive(root / REPORT_RELATIVE_PATH, report)
    markdown_path = root / MARKDOWN_RELATIVE_PATH
    markdown_path.parent.mkdir(parents=True, exist_ok=True)
    with markdown_path.open("x", encoding="utf-8") as handle:
        handle.write(markdown_report(report))
    consumed_marker = {"marker_version": "1.0.0", "holdout_consumed": True, "consumed_at_utc": evaluated_at_utc, "consumption_reason": "Level 3 first and only planned final evaluation", "locked_selection": {"model": selected_name, "selection_basis": lock["selection_basis"], "selection_lock_path": LOCK_RELATIVE_PATH.as_posix(), "selection_lock_sha256": fingerprint(root / LOCK_RELATIVE_PATH)}, "dataset": {"path": paths_as_relative(test_path, root), "sha256": test_sha256}, "selected_model": {"path": selected_spec["artifact_path"], "sha256": fingerprint(root / selected_spec["artifact_path"])}, "evaluation_report": {"path": REPORT_RELATIVE_PATH.as_posix(), "sha256": fingerprint(root / REPORT_RELATIVE_PATH)}, "forbidden_future_uses": FORBIDDEN_HOLDOUT_USES, "allowed_future_uses": ["historical_reporting", "audit"]}
    write_json_exclusive(root / HOLDOUT_MARKER_RELATIVE_PATH, consumed_marker)
    level4_contract = {"handoff_version": "1.1.0", "next_level": "Level 4: Features", "source_dataset_version": DATASET_VERSION, "recommended_next_dataset_version": "v2", "evaluated_at_utc": evaluated_at_utc, "feature_columns": features, "target_column": target, "group_column": group, "locked_selection": consumed_marker["locked_selection"], "selected_model": consumed_marker["selected_model"], "holdout_policy": {"holdout_consumed": True, "marker_path": HOLDOUT_MARKER_RELATIVE_PATH.as_posix(), "marker_sha256": fingerprint(root / HOLDOUT_MARKER_RELATIVE_PATH), "forbidden_future_uses": FORBIDDEN_HOLDOUT_USES, "required_level4_selection_data": "grouped validation predictions from training data only"}, "evaluation_evidence": {"json_report": {"path": REPORT_RELATIVE_PATH.as_posix(), "sha256": fingerprint(root / REPORT_RELATIVE_PATH)}, "markdown_report": {"path": MARKDOWN_RELATIVE_PATH.as_posix(), "sha256": fingerprint(root / MARKDOWN_RELATIVE_PATH)}, "error_analysis": {"path": ERRORS_RELATIVE_PATH.as_posix(), "sha256": fingerprint(root / ERRORS_RELATIVE_PATH)}}, "selected_test_metrics": selected_metrics, "majority_baseline_test_metrics": baseline_metrics, "limitations": LIMITATIONS}
    write_json_exclusive(root / LEVEL4_RELATIVE_PATH, level4_contract)
    return report


def verify_no_fit_calls_in_this_module() -> list[int]:
    tree = ast.parse(inspect.getsource(sys.modules[__name__]))
    return [node.lineno for node in ast.walk(tree) if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr in {"fit", "fit_transform"}]


def readme_section(report: dict[str, Any]) -> str:
    selected, baseline = report["models"]["selected"], report["models"]["majority_baseline"]
    sm, bm = selected["metrics"], baseline["metrics"]
    return f"""<!-- LEVEL-3-EVALUATE:START -->
## Level 3: Evaluate

Evaluated at `{report['evaluated_at_utc']}` after `{selected['name']}` was locked from grouped validation evidence.

| Model | Accuracy | Precision 1 | Recall 1 | F1 1 |
| --- | ---: | ---: | ---: | ---: |
| `{selected['name']}` | {sm['accuracy']:.3f} | {sm['positive_class']['precision']:.3f} | {sm['positive_class']['recall']:.3f} | {sm['positive_class']['f1']:.3f} |
| `{baseline['name']}` | {bm['accuracy']:.3f} | {bm['positive_class']['precision']:.3f} | {bm['positive_class']['recall']:.3f} | {bm['positive_class']['f1']:.3f} |

Selected confusion matrix `[[TN, FP], [FN, TP]]`: `{sm['confusion_matrix']['matrix']}`

- [Evaluation report](reports/v1/level3_evaluation.md)
- [Machine-readable metrics](reports/v1/level3_evaluation.json)
- [Error analysis](reports/v1/level3_errors.csv)
- [Consumed-holdout marker](artifacts/v1/holdout_consumed.json)
- [Level 4 handoff](artifacts/v1/level4_features_contract.json)

Reproduce safeguards with `python -m unittest -v tests/test_data_pipeline.py tests/test_training.py tests/test_evaluation.py`. The holdout command is intentionally excluded because v1 test labels are consumed.
<!-- LEVEL-3-EVALUATE:END -->"""


def update_readme(root: Path) -> Path:
    report_path = root / REPORT_RELATIVE_PATH
    if not report_path.exists():
        raise FileNotFoundError("Run evaluation before documentation")
    readme = root / "README.md"
    existing = readme.read_text(encoding="utf-8") if readme.exists() else ""
    start, end = "<!-- LEVEL-3-EVALUATE:START -->", "<!-- LEVEL-3-EVALUATE:END -->"
    section = readme_section(read_json(report_path))
    if start in existing:
        if end not in existing:
            raise ValueError("Incomplete Level 3 README marker block")
        before, after = existing.split(start, 1)[0].rstrip(), existing.split(end, 1)[1].lstrip()
        updated = f"{before}\n\n{section}\n" + (f"\n{after.rstrip()}\n" if after else "")
    else:
        prefix = existing.rstrip()
        updated = f"{prefix}\n\n{section}\n" if prefix else f"{section}\n"
    readme.write_text(updated, encoding="utf-8")
    return readme


def main() -> None:
    if len(sys.argv) != 2 or sys.argv[1] not in {"lock", "evaluate", "document"}:
        raise SystemExit("Usage: python -m src.evaluate_models [lock|evaluate|document]")
    root = Path(__file__).resolve().parents[1]
    if sys.argv[1] == "lock":
        print(json.dumps(lock_selection(root), indent=2))
    elif sys.argv[1] == "evaluate":
        report = run_evaluation(root)
        print(json.dumps({"evaluated_at_utc": report["evaluated_at_utc"], "holdout_consumed": report["holdout_consumed"], "selected": report["models"]["selected"], "majority_baseline": report["models"]["majority_baseline"], "comparison": report["comparison_to_majority_baseline"]}, indent=2))
    else:
        print(update_readme(root).relative_to(root).as_posix())


if __name__ == "__main__":
    main()

How does this module enforce the boundary?

The handoff verifier recomputes Level 2 fingerprints. It also proves that every runner appears in exactly one validation fold.

The lock function creates its JSON file through exclusive mode. The remaining functions prepare later evaluation steps without reading the held-out CSV during locking.

  • Save src/evaluate_models.py.
  • Check the module's syntax by running this command in the active repository terminal:
python -m py_compile src/evaluate_models.py

What does this check prove?

Python compiles the source without executing the module's lock or evaluation commands. A clean return to the prompt proves that the file has valid Python syntax.

You should return to the terminal prompt without a syntax error. The module is ready to verify the handoff.

Seeing a syntax error?

Compare the reported line in src/evaluate_models.py with the full-file reference. Pay close attention to indentation around the function named in the traceback.

Check that every multiline string starts and ends with three double quotes.

Ask for help with the exact line Python reported.

Freeze the validation-selected model

The lock command reads the Level 2 contract plus training report. It recalculates the winner from grouped validation accuracy before recording the selected model and majority baseline fingerprints.

Why lock before evaluation?

Final-test labels can influence future decisions once you see their results. A write-once selection record proves that the model choice predates that evidence.

Before you run the lock command, which evidence do you expect it to use without opening test.csv?

  • Create the write-once selection lock by running this command exactly once:
python -m src.evaluate_models lock

What does the lock command do?

  • It verifies the Level 2 input fingerprints before trusting the recorded validation results.
  • It checks grouped-fold coverage for every training runner.
  • It recalculates the winner from mean_validation_accuracy.
  • It fingerprints the selected model plus the predeclared majority baseline.
  • It creates artifacts/v1/level3_selection_lock.json through exclusive file creation.

That boundary is now frozen. You should see JSON containing mean_validation_accuracy as the selection metric.

Did the lock command stop?

A missing-file failure means a Level 2 handoff artifact is absent from its original path. Restore that unchanged artifact before trying again.

A fingerprint failure means an input changed after Level 2 recorded it. Restore the original v1 artifact instead of regenerating the dataset or models.

Get help interpreting the failure before touching the holdout.

Inspect the write-once lock

The lock is deliberately incomplete as an evaluation report. Its job is to preserve the decision boundary before richer test evidence exists.

Before you inspect the file, do you expect to find any held-out precision or recall values inside it?

  • Switch back to the repository in VS Code.
  • Select artifacts/v1/level3_selection_lock.json from the file sidebar.
  • Confirm that selection_metric contains mean_validation_accuracy.
  • Confirm that held_out_test_status contains not read during selection lock.
  • Confirm that test_metrics_available contains false.
  • Confirm that model_artifacts contains fingerprints for the selected model plus majority_baseline.
  • Confirm that the lock contains no test fingerprint.

You should find validation evidence plus immutable model fingerprints. You should also find the held-out status recorded as not read during selection lock.

The missing error picture is intentional

The lock contains no test precision or recall. It also contains no F1 score or class support. No confusion matrix appears either.

Validation accuracy selected the model without exposing its final error burden. The evaluation boundary must be tested before that missing evidence is revealed.

Your validation winner is now fixed before held-out access. Next, you will prove through isolated tests that evaluation cannot fit a model or replace this decision.

Prove the Evaluation Boundary

Your validation winner is now frozen in an immutable selection lock. That gives the evaluation workflow a fixed model choice before any held-out label is read.

The missing error picture can only be filled after executable guards protect the boundary. In this step, isolated Python unittest fixtures prove that evaluation cannot retrain the model, replace the winner, accept changed inputs, or score the real holdout twice.

In this step, get ready to:
  • Build a temporary fixture repository that never uses the real v1 holdout.
  • Add six guards for selection integrity, fingerprint enforcement, fit-free evaluation, and one-time scoring.
  • Run the isolated suite and confirm the real evaluation outputs remain absent.
Build the isolated fixture repository

A test fixture creates a disposable copy of the Level 2 handoff inside a temporary directory. Every guard can exercise locking and evaluation behavior without touching the repository's real data/v1/processed/test.csv file.

  • Switch back to VS Code from the previous step.
  • Select the tests folder in the Explorer sidebar.
  • Create test_evaluation.py inside the tests folder.
  • Add the fixture dependencies by pasting this import group into tests/test_evaluation.py:
import ast
import inspect
import json
import unittest
from pathlib import Path
from tempfile import TemporaryDirectory

import joblib
import pandas as pd

import src.evaluate_models as evaluate_models
from src.evaluate_models import ERRORS_RELATIVE_PATH, HOLDOUT_MARKER_RELATIVE_PATH, LEVEL4_RELATIVE_PATH, LOCK_RELATIVE_PATH, REPORT_RELATIVE_PATH, fingerprint, lock_selection, run_evaluation

What do these imports support?

  • The standard-library imports create temporary repositories, inspect source code, parse its syntax tree, and define the test suite.
  • The project imports expose the exact paths and functions that each guard exercises.
  • The data libraries create fixture CSV files and serialize temporary model artifacts.
  • Save tests/test_evaluation.py.
  • Return to the active terminal from the previous step.
  • Check that the import group loads by running:
python -m unittest -v tests/test_evaluation.py

What does this check prove?

The test runner imports the new module without accessing the real held-out data. At this point, successful completion confirms that every imported project identifier resolves correctly.

You should see a successful run with zero discovered tests. The fixture structure comes next.

Does the import check fail?

Confirm that test_evaluation.py sits directly inside the existing tests folder. Check the long project import against the reference block above.

Ask for help with the import failure.

  • Add the deterministic fixture model and test helpers below the imports by pasting:
class FixtureModel:
    def __init__(self, strategy: str) -> None:
        self.strategy = strategy

    def predict(self, features: pd.DataFrame) -> list[int]:
        if self.strategy == "majority":
            return [0] * len(features)
        return [int(value >= 0.5) for value in features["signal"]]


class EvaluationTests(unittest.TestCase):
    def setUp(self) -> None:
        self.temp = TemporaryDirectory()
        self.addCleanup(self.temp.cleanup)
        self.root = Path(self.temp.name)
        self.build_fixture()

    def write_json(self, path: Path, payload: dict) -> None:
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(json.dumps(payload, indent=2), encoding="utf-8")

How does the fixture stay isolated?

  • FixtureModel produces predictable learned-model or majority-baseline predictions from a small signal column.
  • setUp creates a new temporary directory for every test case.
  • addCleanup removes that temporary repository after the test finishes.
  • write_json creates the parent directories required by each fixture artifact.
  • Save tests/test_evaluation.py.
  • Confirm the helper classes load by running:
python -m unittest -v tests/test_evaluation.py

What does this run check?

The runner constructs the module without executing a real evaluation. Successful completion proves that the model class and test helpers have valid syntax.

You should still see zero discovered tests with a successful result.

Does fixture setup fail?

Check that EvaluationTests inherits from unittest.TestCase. Confirm that write_json remains indented inside the class.

Ask for help with the fixture structure.

  • Add build_fixture() below write_json() by pasting:
    def build_fixture(self) -> None:
        train_path = self.root / "data/v1/processed/train.csv"
        test_path = self.root / "data/v1/processed/test.csv"
        schema_path = self.root / "contracts/v1/schema_contract.json"
        preprocessing_path = self.root / "artifacts/v1/preprocessing.json"
        train_path.parent.mkdir(parents=True, exist_ok=True)
        train = pd.DataFrame({"signal": [0.1, 0.9, 0.2, 0.8], "needs_recovery_next_day": [0, 1, 0, 1], "runner_id": ["runner-a", "runner-b", "runner-c", "runner-d"]})
        test = pd.DataFrame({"signal": [0.2, 0.8, 0.7, 0.3], "needs_recovery_next_day": [0, 1, 0, 1], "runner_id": ["runner-e", "runner-e", "runner-f", "runner-f"]})
        train.to_csv(train_path, index=False)
        test.to_csv(test_path, index=False)
        self.write_json(schema_path, {"dataset_version": "v1", "feature_columns": ["signal"], "target_column": "needs_recovery_next_day", "group_column": "runner_id"})
        self.write_json(preprocessing_path, {"dataset_version": "v1", "feature_columns": ["signal"], "fit_scope": "training only"})
        models = self.root / "models/v1"
        models.mkdir(parents=True, exist_ok=True)
        joblib.dump(FixtureModel("learned"), models / "logistic_regression.joblib")
        joblib.dump(FixtureModel("majority"), models / "majority_baseline.joblib")
        fingerprints = {"train": {"path": "data/v1/processed/train.csv", "sha256": fingerprint(train_path)}, "schema": {"path": "contracts/v1/schema_contract.json", "sha256": fingerprint(schema_path)}, "preprocessing": {"path": "artifacts/v1/preprocessing.json", "sha256": fingerprint(preprocessing_path)}}
        results = {"majority_baseline": {"aggregate": {"mean_validation_accuracy": 0.5}}, "logistic_regression": {"aggregate": {"mean_validation_accuracy": 0.75}}}
        self.write_json(self.root / "reports/v1/training_experiment.json", {"experiment_id": "level-2-train-v1", "dataset_version": "v1", "cross_validation": {"n_splits": 2, "fold_manifest": [{"runner_overlap": [], "validation_runners": ["runner-a", "runner-b"]}, {"runner_overlap": [], "validation_runners": ["runner-c", "runner-d"]}]}, "results": results, "selected_by_mean_validation_accuracy": "logistic_regression"})
        self.write_json(self.root / "artifacts/v1/level3_evaluation_contract.json", {"experiment_id": "level-2-train-v1", "dataset_version": "v1", "input_fingerprints": fingerprints, "held_out_test_set": {"path": "data/v1/processed/test.csv"}, "feature_columns": ["signal"], "target_column": "needs_recovery_next_day", "group_column": "runner_id", "selected_model": "logistic_regression", "models": {"majority_baseline": {"artifact_path": "models/v1/majority_baseline.joblib", "validation_metrics": results["majority_baseline"]["aggregate"]}, "logistic_regression": {"artifact_path": "models/v1/logistic_regression.joblib", "validation_metrics": results["logistic_regression"]["aggregate"]}}})

What does the fixture reproduce?

  • The training and test rows use different runner groups so the group-safety check can pass.
  • The fingerprints bind the temporary training data, schema contract, and preprocessing record to the fixture handoff.
  • The validation evidence makes logistic_regression the deterministic winner over majority_baseline.
  • The persisted fixture models let the evaluator produce complete evidence without using the real model artifacts.
  • Save tests/test_evaluation.py.
  • Confirm the complete fixture initializes by running:
python -m unittest -v tests/test_evaluation.py

What does this fixture check prove?

The temporary repository can now create its synthetic Level 2 handoff without touching the real repository. The successful run establishes the sandbox for all six guards.

You should see zero discovered tests with successful completion once more.

Does the fixture fail to initialize?

Check the indentation of build_fixture() first. Confirm that every path begins with self.root so no fixture artifact escapes the temporary directory.

Ask for help with the temporary handoff.

Add the boundary guard tests

Each guard targets one way evaluation evidence could lose credibility. The first three protect selection from held-out access, rewrites, and changed Level 2 inputs.

  • Add the no-test-file selection guard below build_fixture() by pasting:
    def test_selection_lock_succeeds_without_test_csv(self) -> None:
        (self.root / "data/v1/processed/test.csv").unlink()
        lock = lock_selection(self.root)
        self.assertEqual(lock["selected_model"], "logistic_regression")
        self.assertEqual(lock["held_out_test_status"], "not read during selection lock")

What does this guard enforce?

The test deletes the fixture's held-out CSV before selection begins. A successful lock proves that model selection depends only on grouped validation evidence.

  • Save tests/test_evaluation.py.
  • Run the isolated suite with:
python -m unittest -v tests/test_evaluation.py

What does the first passing test mean?

One passing test means the selection lock succeeds after the temporary test.csv is removed. The real held-out file remains untouched.

You should see one test pass with OK at the end.

Does the selection-only guard fail?

Confirm that the deleted path belongs to self.root. Check that the fixture contract does not include the test file among input_fingerprints.

Ask for help tracing the selection dependency.

  • Add the immutable-lock guard below the first test by pasting:
    def test_selection_lock_uses_validation_winner_and_cannot_be_rewritten(self) -> None:
        lock = lock_selection(self.root)
        self.assertEqual(lock["selection_metric"], "mean_validation_accuracy")
        with self.assertRaises(FileExistsError):
            lock_selection(self.root)

What does immutability protect?

The first call records the validation-selected winner. The second call must fail because rewriting the lock could replace that winner after new evidence became visible.

  • Save tests/test_evaluation.py.
  • Run the expanded suite with:
python -m unittest -v tests/test_evaluation.py

What does the second passing test mean?

Two passing tests prove that selection remains validation-only and write-once. The expected rewrite failure is contained inside the test.

You should see two tests pass with OK at the end.

Does the rewrite guard fail?

Check that both lock_selection(self.root) calls remain inside the same test. Confirm that the second call is nested under assertRaises.

Ask for help with the expected exception.

  • Add the fingerprint-drift guard below the immutable-lock test by pasting:
    def test_level2_fingerprint_mutation_is_rejected(self) -> None:
        path = self.root / "data/v1/processed/train.csv"
        path.write_text(path.read_text(encoding="utf-8") + "\n", encoding="utf-8")
        with self.assertRaises(ValueError):
            lock_selection(self.root)

Why does a blank line matter?

A SHA-256 fingerprint changes when even one byte changes. Appending a newline proves that the lock rejects drift before trusting the Level 2 handoff.

  • Save tests/test_evaluation.py.
  • Run all three selection guards with:
python -m unittest -v tests/test_evaluation.py

What have the first three guards proved?

The selection boundary now works without held-out data, resists rewrites, and rejects changed Level 2 inputs. These controls preserve the history that produced the chosen model.

You should see three tests pass with OK at the end.

Does fingerprint drift go unnoticed?

Confirm that the newline is appended after the fixture records the original fingerprint. Check that lock_selection(self.root) remains inside the expected ValueError context.

Ask for help tracing fingerprint verification.

Prove six guards without consuming the holdout

The remaining guards constrain the evaluator itself. They prove that scoring is fit-free, lock-dependent, and write-once while still producing complete evidence inside a disposable repository.

  • Add the source-level no-fit guard below the fingerprint test by pasting:
    def test_evaluation_source_contains_no_fit_call(self) -> None:
        tree = ast.parse(inspect.getsource(evaluate_models))
        lines = [node.lineno for node in ast.walk(tree) if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr in {"fit", "fit_transform"}]
        self.assertEqual(lines, [])
        self.assertEqual(evaluate_models.verify_no_fit_calls_in_this_module(), [])

How does the source guard work?

The test parses src.evaluate_models into an abstract syntax tree. It searches actual method calls for fit or fit_transform instead of relying on a text search.

The module's own verification helper must independently return the same empty result.

  • Save tests/test_evaluation.py.
  • Run the suite with the source guard by running:
python -m unittest -v tests/test_evaluation.py

What does fit-free evaluation prove?

Four passing tests show that the held-out evaluator can only load frozen models and call prediction methods. Retraining remains outside the evaluation boundary.

You should see four tests pass with OK at the end.

Does the source inspection fail?

Compare the set of searched method names with the reference block. Confirm that inspect.getsource(evaluate_models) receives the imported module.

Ask for help inspecting the syntax tree.

  • Add the required-lock guard below the no-fit test by pasting:
    def test_evaluation_requires_a_selection_lock(self) -> None:
        self.assertFalse((self.root / LOCK_RELATIVE_PATH).exists())
        with self.assertRaises(FileNotFoundError):
            run_evaluation(self.root)

Why must scoring require a lock?

Evaluation needs proof that model selection predates test access. This guard blocks scoring when the temporary repository has no immutable selection record.

  • Save tests/test_evaluation.py.
  • Run the suite with the required-lock guard by running:
python -m unittest -v tests/test_evaluation.py

What does the missing-lock result prove?

Five passing tests prove that held-out scoring cannot begin from an unlocked model choice. The expected missing-lock failure stays inside the disposable fixture.

You should see five tests pass with OK at the end.

Does evaluation run without a lock?

Confirm that this test does not call lock_selection(self.root) first. Check that run_evaluation(self.root) is nested under the expected missing-file assertion.

Ask for help with the lock precondition.

  • Add the once-only evidence guard and module entry point below the required-lock test by pasting:
    def test_evaluation_is_once_only_and_writes_complete_evidence(self) -> None:
        lock_selection(self.root)
        report = run_evaluation(self.root)
        selected = report["models"]["selected"]["metrics"]
        baseline = report["models"]["majority_baseline"]["metrics"]
        self.assertEqual(selected["confusion_matrix"]["matrix"], [[1, 1], [1, 1]])
        self.assertEqual(selected["class_support"], {"0": 2, "1": 2})
        self.assertEqual(selected["positive_class"]["f1"], 0.5)
        self.assertEqual(baseline["positive_class"]["recall"], 0.0)
        self.assertEqual(report["metric_policy"]["zero_division"], 0.0)
        self.assertTrue(report["evaluated_at_utc"].endswith("+00:00"))
        marker = json.loads((self.root / HOLDOUT_MARKER_RELATIVE_PATH).read_text(encoding="utf-8"))
        self.assertTrue(marker["holdout_consumed"])
        self.assertEqual(set(marker["forbidden_future_uses"]), {"feature_selection", "threshold_tuning", "model_comparison", "model_selection"})
        self.assertEqual(len(pd.read_csv(self.root / ERRORS_RELATIVE_PATH)), 2)
        self.assertTrue((self.root / REPORT_RELATIVE_PATH).exists())
        self.assertTrue((self.root / LEVEL4_RELATIVE_PATH).exists())
        with self.assertRaises(FileExistsError):
            run_evaluation(self.root)


if __name__ == "__main__":
    unittest.main()

What does the final guard cover?

  • The selected fixture model produces one true negative, one false positive, one false negative, and one true positive.
  • The majority baseline exposes zero positive-class recall under the explicit zero-division policy.
  • The evaluator writes timestamped reports, an error CSV, a consumed marker, and a Level 4 contract inside the temporary repository.
  • A second evaluation attempt fails because those write-once outputs already exist.
  • Save tests/test_evaluation.py.

Before you run the complete suite, do you expect any real Level 3 report or consumed marker to appear in your repository?

  • Run all six isolated guards by running:
python -m unittest -v tests/test_evaluation.py

What does the complete suite prove?

All six tests execute against fresh temporary repositories. Their reports and consumed markers disappear during cleanup, so the real v1 holdout remains unscored.

You should see six tests pass with OK at the end. That's the boundary secured: the evaluator is now tested before it receives final-test evidence.

Do any of the six guards fail?

Read the first failing test name before changing code. Compare that method with the matching reference chunk because each guard isolates one boundary condition.

Ask for help diagnosing the failing guard.

  • Expand reports/v1 in the VS Code Explorer sidebar.
  • Confirm that level3_evaluation.json is absent.
  • Expand artifacts/v1 in the VS Code Explorer sidebar.
  • Confirm that holdout_consumed.json is absent.

Why must both files remain absent?

Their absence proves that the real evaluation has not run. The temporary tests exercised complete scoring behavior without consuming the repository's v1 holdout.

✔️ Awesome, I've got everything!

Great. Save tests/test_evaluation.py and keep the immutable selection lock unchanged.

ⓧ I'd like to double check the full code

Compare your test module with this complete isolated suite.

import ast
import inspect
import json
import unittest
from pathlib import Path
from tempfile import TemporaryDirectory

import joblib
import pandas as pd

import src.evaluate_models as evaluate_models
from src.evaluate_models import ERRORS_RELATIVE_PATH, HOLDOUT_MARKER_RELATIVE_PATH, LEVEL4_RELATIVE_PATH, LOCK_RELATIVE_PATH, REPORT_RELATIVE_PATH, fingerprint, lock_selection, run_evaluation


class FixtureModel:
    def __init__(self, strategy: str) -> None:
        self.strategy = strategy

    def predict(self, features: pd.DataFrame) -> list[int]:
        if self.strategy == "majority":
            return [0] * len(features)
        return [int(value >= 0.5) for value in features["signal"]]


class EvaluationTests(unittest.TestCase):
    def setUp(self) -> None:
        self.temp = TemporaryDirectory()
        self.addCleanup(self.temp.cleanup)
        self.root = Path(self.temp.name)
        self.build_fixture()

    def write_json(self, path: Path, payload: dict) -> None:
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(json.dumps(payload, indent=2), encoding="utf-8")

    def build_fixture(self) -> None:
        train_path = self.root / "data/v1/processed/train.csv"
        test_path = self.root / "data/v1/processed/test.csv"
        schema_path = self.root / "contracts/v1/schema_contract.json"
        preprocessing_path = self.root / "artifacts/v1/preprocessing.json"
        train_path.parent.mkdir(parents=True, exist_ok=True)
        train = pd.DataFrame({"signal": [0.1, 0.9, 0.2, 0.8], "needs_recovery_next_day": [0, 1, 0, 1], "runner_id": ["runner-a", "runner-b", "runner-c", "runner-d"]})
        test = pd.DataFrame({"signal": [0.2, 0.8, 0.7, 0.3], "needs_recovery_next_day": [0, 1, 0, 1], "runner_id": ["runner-e", "runner-e", "runner-f", "runner-f"]})
        train.to_csv(train_path, index=False)
        test.to_csv(test_path, index=False)
        self.write_json(schema_path, {"dataset_version": "v1", "feature_columns": ["signal"], "target_column": "needs_recovery_next_day", "group_column": "runner_id"})
        self.write_json(preprocessing_path, {"dataset_version": "v1", "feature_columns": ["signal"], "fit_scope": "training only"})
        models = self.root / "models/v1"
        models.mkdir(parents=True, exist_ok=True)
        joblib.dump(FixtureModel("learned"), models / "logistic_regression.joblib")
        joblib.dump(FixtureModel("majority"), models / "majority_baseline.joblib")
        fingerprints = {"train": {"path": "data/v1/processed/train.csv", "sha256": fingerprint(train_path)}, "schema": {"path": "contracts/v1/schema_contract.json", "sha256": fingerprint(schema_path)}, "preprocessing": {"path": "artifacts/v1/preprocessing.json", "sha256": fingerprint(preprocessing_path)}}
        results = {"majority_baseline": {"aggregate": {"mean_validation_accuracy": 0.5}}, "logistic_regression": {"aggregate": {"mean_validation_accuracy": 0.75}}}
        self.write_json(self.root / "reports/v1/training_experiment.json", {"experiment_id": "level-2-train-v1", "dataset_version": "v1", "cross_validation": {"n_splits": 2, "fold_manifest": [{"runner_overlap": [], "validation_runners": ["runner-a", "runner-b"]}, {"runner_overlap": [], "validation_runners": ["runner-c", "runner-d"]}]}, "results": results, "selected_by_mean_validation_accuracy": "logistic_regression"})
        self.write_json(self.root / "artifacts/v1/level3_evaluation_contract.json", {"experiment_id": "level-2-train-v1", "dataset_version": "v1", "input_fingerprints": fingerprints, "held_out_test_set": {"path": "data/v1/processed/test.csv"}, "feature_columns": ["signal"], "target_column": "needs_recovery_next_day", "group_column": "runner_id", "selected_model": "logistic_regression", "models": {"majority_baseline": {"artifact_path": "models/v1/majority_baseline.joblib", "validation_metrics": results["majority_baseline"]["aggregate"]}, "logistic_regression": {"artifact_path": "models/v1/logistic_regression.joblib", "validation_metrics": results["logistic_regression"]["aggregate"]}}})

    def test_selection_lock_succeeds_without_test_csv(self) -> None:
        (self.root / "data/v1/processed/test.csv").unlink()
        lock = lock_selection(self.root)
        self.assertEqual(lock["selected_model"], "logistic_regression")
        self.assertEqual(lock["held_out_test_status"], "not read during selection lock")

    def test_selection_lock_uses_validation_winner_and_cannot_be_rewritten(self) -> None:
        lock = lock_selection(self.root)
        self.assertEqual(lock["selection_metric"], "mean_validation_accuracy")
        with self.assertRaises(FileExistsError):
            lock_selection(self.root)

    def test_level2_fingerprint_mutation_is_rejected(self) -> None:
        path = self.root / "data/v1/processed/train.csv"
        path.write_text(path.read_text(encoding="utf-8") + "\n", encoding="utf-8")
        with self.assertRaises(ValueError):
            lock_selection(self.root)

    def test_evaluation_source_contains_no_fit_call(self) -> None:
        tree = ast.parse(inspect.getsource(evaluate_models))
        lines = [node.lineno for node in ast.walk(tree) if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr in {"fit", "fit_transform"}]
        self.assertEqual(lines, [])
        self.assertEqual(evaluate_models.verify_no_fit_calls_in_this_module(), [])

    def test_evaluation_requires_a_selection_lock(self) -> None:
        self.assertFalse((self.root / LOCK_RELATIVE_PATH).exists())
        with self.assertRaises(FileNotFoundError):
            run_evaluation(self.root)

    def test_evaluation_is_once_only_and_writes_complete_evidence(self) -> None:
        lock_selection(self.root)
        report = run_evaluation(self.root)
        selected = report["models"]["selected"]["metrics"]
        baseline = report["models"]["majority_baseline"]["metrics"]
        self.assertEqual(selected["confusion_matrix"]["matrix"], [[1, 1], [1, 1]])
        self.assertEqual(selected["class_support"], {"0": 2, "1": 2})
        self.assertEqual(selected["positive_class"]["f1"], 0.5)
        self.assertEqual(baseline["positive_class"]["recall"], 0.0)
        self.assertEqual(report["metric_policy"]["zero_division"], 0.0)
        self.assertTrue(report["evaluated_at_utc"].endswith("+00:00"))
        marker = json.loads((self.root / HOLDOUT_MARKER_RELATIVE_PATH).read_text(encoding="utf-8"))
        self.assertTrue(marker["holdout_consumed"])
        self.assertEqual(set(marker["forbidden_future_uses"]), {"feature_selection", "threshold_tuning", "model_comparison", "model_selection"})
        self.assertEqual(len(pd.read_csv(self.root / ERRORS_RELATIVE_PATH)), 2)
        self.assertTrue((self.root / REPORT_RELATIVE_PATH).exists())
        self.assertTrue((self.root / LEVEL4_RELATIVE_PATH).exists())
        with self.assertRaises(FileExistsError):
            run_evaluation(self.root)


if __name__ == "__main__":
    unittest.main()

Your evaluation boundary now has executable proof behind it. Next, you can perform the first and only held-out evaluation with the frozen model choice and a write-once consumed-holdout record.

Evaluate the Held-Out Runners

Your boundary tests have proved that evaluation requires the immutable selection lock. They used temporary repositories. The real data/v1/processed/test.csv remains untouched.

This step resolves the missing error picture with one comparison between the locked scikit-learn Pipeline and the majority baseline. The run records final evidence before marking the holdout as consumed.

In this step, get ready to:
  • Run the locked model and majority baseline against the held-out runners exactly once.
  • Interpret accuracy, class-1 precision, recall, F1, support, and the confusion matrix.
  • Verify the consumed-holdout marker and the Level 4 restrictions.
Run the one-time evaluation

The evaluator verifies the selection lock before it opens the held-out CSV. It also checks every protected fingerprint before loading the trusted local model artifacts.

This Run Is Write-Once

This command permanently consumes the v1 holdout. The evaluator uses exclusive file creation to block a second successful run.

Before you run it, do you expect the selected model and majority baseline to produce the same minority-class error pattern? Hold that prediction.

  • Evaluate the held-out runners exactly once by running this command:
python -m src.evaluate_models evaluate

What Does This Command Do?

  • Verifies that the selected model still matches the immutable selection lock.
  • Checks that the Level 2 fingerprints have not changed.
  • Confirms that no runner appears in both the training data and held-out data.
  • Loads the trusted local joblib model artifacts.
  • Calls predict without fitting either model.
  • Writes the reports, error evidence, consumed marker, and Level 4 contract with exclusive creation.
  • Verify the completed run by locating evaluated_at_utc in the terminal output.

You should see a UTC timestamp followed by held-out evidence for selected and majority_baseline. Each model includes accuracy, class-1 metrics, class support, and a matrix in [[TN, FP], [FN, TP]] format.

That is the sensitive part complete. The real v1 holdout now has one frozen evaluation.

Evaluation Stopped Before Scoring?

  • Leave the holdout untouched if the command reports a fingerprint problem.
  • Compare the affected Level 2 file with its committed version before trying again.
  • Avoid rerunning the command if any Level 3 report or consumed marker already exists.

Ask for help before changing the lock or deleting evidence files: help me diagnose why the one-time held-out evaluation stopped

Inspect metrics and row-level errors

The JSON report preserves machine-readable evidence. The Markdown report presents the comparison for reviewers.

The error CSV preserves the selected model's misclassified runners. This supports error analysis without another prediction pass.

  • Switch back to VS Code from earlier.
  • Open reports/v1/level3_evaluation.md from the repository file sidebar.
  • Confirm the metric table compares the selected model with majority_baseline.
  • Confirm both confusion matrices use [[TN, FP], [FN, TP]].
  • Confirm the report records zero_division=0.0.
  • Confirm the report includes the synthetic-label limitations.
  • Open reports/v1/level3_evaluation.json from the file sidebar.
  • Confirm comparison_to_majority_baseline records the selected model's metric deltas.

How Do These Metrics Expose Errors?

  • Accuracy measures the share of all predictions that are correct. A majority classifier can still look strong when class 1 is rare.
  • Precision measures how often a predicted class-1 result is correct.
  • Recall measures how many actual class-1 rows the model identifies.
  • F1 balances precision with recall through their harmonic mean.
  • Support records how many held-out rows belong to each class.
  • The confusion matrix exposes true negatives, false positives, false negatives, and true positives directly.
  • Confirm group_safety records a runner_overlap_count of 0.
  • Open reports/v1/level3_errors.csv from the file sidebar.
  • Confirm every row has an error_type of false_positive or false_negative.
  • Confirm the CSV preserves row_id, the runner group column, all feature columns, actual, predicted, and error_type.

A false positive flags recovery when the synthetic label is 0. A false negative clears a runner when the synthetic label is 1.

Verify the consumed-holdout contract

The consumed marker turns the evaluation timestamp into a machine-readable boundary. Its SHA-256 fingerprints bind the test data, selection lock, selected model, and final report to the same run.

The Level 4 contract carries that boundary forward. Future feature or model decisions must use grouped training validation evidence.

Before you inspect the marker, which future uses do you expect it to forbid? Hold that prediction.

  • Open artifacts/v1/holdout_consumed.json from the repository file sidebar.
  • Confirm holdout_consumed is true.
  • Confirm consumed_at_utc contains one UTC timestamp.
  • Confirm the marker fingerprints the test data, selection lock, selected model, and evaluation report.
  • Confirm forbidden_future_uses contains feature_selection, threshold_tuning, model_comparison, and model_selection.
  • Open artifacts/v1/level4_features_contract.json from the file sidebar.
  • Confirm holdout_policy fingerprints artifacts/v1/holdout_consumed.json.
  • Confirm required_level4_selection_data requires grouped validation predictions from training data only.

You should see holdout_consumed set to true alongside the immutable evidence from this run. The Level 4 contract should forbid every listed selection use of the consumed v1 test labels.

Your selected model and majority baseline now have one audit-ready held-out evaluation. Next, you will publish the evidence without replacing the Level 1 or Level 2 documentation.

Publish the Evidence

Your locked selected Pipeline has now faced the v1 holdout exactly once. The resulting files form the final evidence for this dataset version.

Raw metrics become portfolio evidence when a reviewer can reproduce the safeguards. This step publishes the evaluation trail while preserving the boundary that protects future experiments from test-driven decisions.

In this step, get ready to:
  • Generate a marker-delimited Level 3 section in README.md without replacing earlier content.
  • Audit every evidence link plus the consumed-holdout rule.
  • Run all three test suites together to prove the repository remains reproducible.
Generate the README evidence section

The update_readme() function reads the frozen evaluation report. It replaces only the content between the Level 3 markers.

Before you run this, do you expect the existing Level 1 and Level 2 documentation to remain in place?

  • Generate the Level 3 section by running this command in your repository terminal:
python -m src.evaluate_models document

What does this command do?

The document command dispatches to update_readme(). The function loads the existing evaluation report before rendering its frozen timestamp plus metrics.

The marker pair gives the function a safe replacement boundary. Existing Level 1 or Level 2 content outside that boundary stays untouched.

You should see README.md printed in the terminal. That output identifies the file updated by the documentation command.

  • Select README.md from your code editor's file sidebar.
  • Find the <!-- LEVEL-3-EVALUATE:START --> marker.
  • Confirm the previous Level 1 and Level 2 content still appears above the generated section.
  • Find the closing <!-- LEVEL-3-EVALUATE:END --> marker.

Between the markers, you should see timestamped selected-versus-baseline metrics plus the selected confusion matrix. You should also see five evidence links plus the consumed-holdout reproduction rule.

Level 3 section missing?

Confirm that reports/v1/level3_evaluation.json still exists. The documentation command reads that frozen report before it updates the README.

Check for an unmatched Level 3 marker if the command stops during replacement. Both marker lines must remain intact.

Help me diagnose the README documentation command.

Audit the evidence links and boundary

Each generated link connects a published claim to its underlying artifact. The final link carries the consumed-holdout restriction into Level 4.

  • Open reports/v1/level3_evaluation.md from the README to confirm the human-readable metric table.
  • Open reports/v1/level3_evaluation.json from the README to locate the machine-readable evaluation record.
  • Open reports/v1/level3_errors.csv from the README to confirm the selected-model error evidence.
  • Open artifacts/v1/holdout_consumed.json from the README to confirm the consumed status.
  • Open artifacts/v1/level4_features_contract.json from the README to confirm the future-use restrictions.
  • Read the reproduction sentence at the bottom of the generated section to confirm the held-out evaluation command is excluded.

The links should resolve to the same immutable evidence created during evaluation. The Level 4 handoff should reserve future selection work for grouped training-validation evidence.

Why exclude the holdout command?

The three test suites reproduce the safeguards without scoring the v1 holdout again. Another held-out run would reuse evidence that has already influenced the project record.

The consumed marker permits historical reporting plus auditing. Future feature selection or threshold tuning must use grouped training-validation evidence.

✔️ Awesome, I've got everything!

Your README preserves its earlier content. Its generated Level 3 section now links every final artifact while keeping the consumed-holdout boundary visible.

ⓧ I'd like to double check the full code

This step changes README.md only. Compare your repository with the following cumulative state.

  • The generated README.md section begins with <!-- LEVEL-3-EVALUATE:START --> and ends with <!-- LEVEL-3-EVALUATE:END -->.
  • The generated section contains timestamped selected-versus-baseline metrics plus the selected confusion matrix.
  • The generated section links the Markdown report plus the JSON report.
  • The generated section links the error CSV plus the consumed marker.
  • The generated section links the Level 4 handoff plus the consumed-holdout reproduction rule.
  • The files reports/v1/level3_evaluation.json plus reports/v1/level3_evaluation.md remain immutable.
  • The files reports/v1/level3_errors.csv plus artifacts/v1/holdout_consumed.json remain immutable.
  • The files artifacts/v1/level3_selection_lock.json plus artifacts/v1/level4_features_contract.json remain immutable.
  • The files src/evaluate_models.py plus tests/test_evaluation.py remain unchanged.
Run the final regression suite

A combined regression run checks the complete data pipeline plus training boundary plus evaluation boundary in one process. This proves the published documentation did not disturb the earlier levels.

Before you run the suite, do you expect documentation generation to change any tested model behavior?

  • Run all preserved test modules together with this command:
python -m unittest -v tests/test_data_pipeline.py tests/test_training.py tests/test_evaluation.py

What does this command verify?

  • The tests/test_data_pipeline.py module protects the Level 1 data contract.
  • The tests/test_training.py module protects the Level 2 grouped-training handoff.
  • The tests/test_evaluation.py module protects validation-only selection plus one-time scoring.

You should see the complete suite finish with OK. This confirms the published evidence still rests on all three protected levels.

A test did not pass?

Stop at the first failing module. Keep the consumed holdout untouched while you compare the affected Level 1 or Level 2 artifact with its preserved version.

Confirm that src/evaluate_models.py plus tests/test_evaluation.py remain unchanged. The documentation command should modify only README.md.

Help me diagnose the failing combined test suite.

That closes the audit trail: your published evidence is reproducible without turning the consumed holdout into tuning feedback.

Secret mission

Secret Mission: Trace the Threshold Tradeoff on Grouped Validation Predictions

The frozen prediction rule gives you one operating point. Can you rebuild the original grouped training folds to trace five threshold tradeoffs while keeping the consumed v1 holdout sealed?

Clean Up Your Resources

Clean Up Your Resources

All Level 3 artifacts stay on your Mac at no ongoing cost. Decide whether to keep the evidence, pause your environment, or delete only this level.

Resources you used:

  • Level 3 source modules at src/evaluate_models.py and src/threshold_analysis.py.
  • Evaluation boundary tests at tests/test_evaluation.py.
  • Selection and handoff artifacts at artifacts/v1/level3_selection_lock.json, artifacts/v1/holdout_consumed.json, and artifacts/v1/level4_features_contract.json.
  • Evaluation evidence at reports/v1/level3_evaluation.json, reports/v1/level3_evaluation.md, reports/v1/level3_errors.csv, and reports/v1/threshold_analysis.json.
  • The marker-delimited Level 3 Evaluate section in README.md.

Keep everything running

No action needed. Choose this if you want to continue into Level 4 or retain the complete evaluation record.

  • Keep the selection lock as proof that validation selected the model before held-out access.
  • Preserve the reports and consumed-holdout marker as the immutable record of the first held-out evaluation.
  • Retain reports/v1/threshold_analysis.json as evidence that threshold exploration used only group-safe training validation predictions.
  • Keep the shared .venv plus every Level 1 and Level 2 file for future work.

Pause - I'll come back to this later

Leave the active virtual environment while keeping every local file intact.

  • Leave the existing .venv by running this command:
deactivate

What Does This Command Do?

The command removes the virtual environment from your current terminal session. It does not delete .venv or any project evidence.

  • Confirm that your terminal prompt no longer shows the active .venv environment.

That cleanly pauses the environment while every evidence file stays ready for Level 4.

Delete - I don't want to use this again

Deleting this evidence is irreversible. The command below targets only the Level 3 paths listed here.

Every Level 1 and Level 2 file stays untouched.

  • Switch back to the terminal from this project.
  • Remove only the Level 3 files by running this command:
rm -f \
  src/evaluate_models.py \
  src/threshold_analysis.py \
  tests/test_evaluation.py \
  reports/v1/level3_evaluation.json \
  reports/v1/level3_evaluation.md \
  reports/v1/level3_errors.csv \
  reports/v1/threshold_analysis.json \
  artifacts/v1/level3_selection_lock.json \
  artifacts/v1/holdout_consumed.json \
  artifacts/v1/level4_features_contract.json

What Does This Remove?

The rm -f command removes each named file from the existing repository. The explicit paths keep the deletion boundary at Level 3.

The command preserves .venv plus every Level 1 and Level 2 source file, test, model, report, and artifact.

  • Return to README.md in VS Code.
  • Find the opening marker <!-- LEVEL-3-EVALUATE:START -->.
  • Select from the opening marker through <!-- LEVEL-3-EVALUATE:END -->.
  • Delete the selected Level 3 block.
  • Save README.md.
  • Confirm that the deleted Level 3 paths no longer appear in the repository file tree.
  • Search README.md for <!-- LEVEL-3-EVALUATE:START -->.

You should find no Level 3 marker block. Your earlier README content should still be present.

Nice Work!

Nice Work!

You did it! You turned the untouched v1 holdout into an audit-ready binary-classifier evaluation without letting final-test labels steer the model.

You've learned how to:

  • Locked the validation-selected model before opening test.csv. Ran one held-out comparison against the majority baseline. The evidence captures accuracy, class-1 precision, recall, F1, support, explicit zero-division handling, and a stable confusion matrix.
  • Produced reproducible false-positive and false-negative evidence with the original feature values. Sealed the result with a UTC timestamp plus dataset, model, lock, and report fingerprints inside a machine-readable consumed-holdout marker.
  • Published a preserved Level 3 README section with links to the reports, error evidence, consumed marker, and handoff. Created a versioned Level 4 contract that blocks v1 holdout reuse for feature selection, threshold tuning, model comparison, or model selection.
  • Completed the Secret Mission by preserving one group-safe out-of-fold probability per training row. Compared decision thresholds from 0.30 through 0.70. Separated the exploratory sweep from the validation-selected F1 threshold proposal without reopening test.csv.

Ready to quiz yourself?