Build a Leakage-Aware PyTorch MLP

Train a reproducible PyTorch MLP with runner-safe cross-validation.

Introduction

30 Second Summary

A result can look convincing even when the experiment quietly gave itself clues it should never have seen. Reusing the same final test makes each new comparison less trustworthy.

In this project, you will add a CPU-only PyTorch multilayer perceptron to your existing running-readiness repository. The experiment compares two configurations on locked development folds while preserving the consumed v1 holdout boundary.

What You'll Build

Run the finished workflow to see two neural networks train on the same runner-safe folds before a reloadable checkpoint becomes the input for Level 6.

By the end of this project, you'll have:

  • Use a one-neuron trace to follow a tensor through its changed loss. The printed gradients reveal the backpropagation update.
  • Compare two MLP configurations across four inherited runner-safe folds. Review development-only F1, precision, recall, plus generalization gaps.
  • Reload a CPU checkpoint on demand. Inspect the exact preprocessing statistics behind it. A fingerprinted Level 6 handoff carries the approved recipe forward.
  • Secret Mission: make memorization visible by training a 128-unit network on two runners. Compare its validation curve with dropout, weight decay, plus early stopping.

Are there any prerequisites?

You need the completed Level 4 running-readiness repository with its report plus handoff. You also need Docker Desktop running because the project uses a Linux amd64 CPU container for PyTorch.

Before We Start

Before the hands-on work begins, lock in the evidence boundary for the CPU-only PyTorch binary MLP you are adding to your existing running-readiness repository. The experiment compares only with Level 4's raw and engineered development controls on inherited runner-safe folds because data/test_v1.csv is a consumed v1 holdout that must remain untouched.

Build the CPU Environment and Audit Level 4

Level 5 depends on evidence produced by Level 4. Any changed fingerprint or fold signature would make the neural-network comparison unreliable.

On an Intel Mac, PyTorch 2.15.0+cpu runs inside a Linux amd64 Docker container. A fail-closed audit protects the consumed holdout before any training begins.

In this step, get ready to:
  • Pin the Level 5 dependencies inside a Linux amd64 CPU image.
  • Add the Level 5 module with its Level 4 contract audit.
  • Build the image and prove that the inherited evidence remains unchanged.
Define the Linux CPU image

A requirements file gives the experiment a fixed numerical stack. These pins keep local reruns inside the same dependency line.

  • Create requirements-level5.txt beside the existing root-level files using your editor's new-file control.
  • Paste the dependency pins below into requirements-level5.txt.
numpy==2.5.4
pandas==3.1.0
scikit-learn==1.9.1

What do these pins control?

  • NumPy 2.5.4 supplies the numerical arrays passed into tensors.
  • pandas 3.1.0 loads the existing development data.
  • scikit-learn 1.9.1 provides fold-local imputation and scaling.
  • Save requirements-level5.txt.
  • Confirm that requirements-level5.txt appears beside the root-level Level 4 files in your editor's file tree.

Dependency file missing?

Check that the filename ends with .txt. Confirm that your editor did not place it inside src/.

Still stuck? Help me place the Level 5 requirements file correctly.

✔️ Awesome, I've got everything!

Your dependency file is saved with all three version pins.

ⓧ I'd like to double check the full code

numpy==2.5.4
pandas==3.1.0
scikit-learn==1.9.1

What should match?

Your file should contain these three lines in this order. Each package uses an exact version pin.

The image also needs a repeatable build recipe. Dockerfile.level5 starts from Python 3.15.2 on Linux amd64 before installing CPU-only PyTorch.

  • Create Dockerfile.level5 beside requirements-level5.txt using your editor's new-file control.
  • Paste the image definition below into Dockerfile.level5.
# syntax=docker/dockerfile:1
FROM --platform=linux/amd64 python:3.15.2-slim

ENV PYTHONHASHSEED=42 \
    LEVEL5_BASE_IMAGE=python:3.15.2-slim

WORKDIR /workspace
COPY requirements-level5.txt /tmp/requirements-level5.txt
RUN python -m pip install --no-cache-dir -r /tmp/requirements-level5.txt && \
    python -m pip install --no-cache-dir torch==2.15.0+cpu \
      --index-url https://download.pytorch.org/whl/cpu

CMD ["python", "-m", "src.neural_network_experiment", "audit"]

How does the image protect the environment?

  • FROM fixes the base environment to Python 3.15.2 on Linux amd64.
  • ENV fixes Python's hash seed and records the base-image identity.
  • RUN installs the pinned numerical stack before adding PyTorch 2.15.0+cpu from its CPU wheel index.
  • CMD makes the Level 4 audit the image's default operation.
  • Save Dockerfile.level5.
  • Confirm that Dockerfile.level5 appears beside requirements-level5.txt in your editor's file tree.

Dockerfile in the wrong location?

Both Level 5 environment files belong in the repository's top-level folder. The build cannot copy the requirements file if either file sits inside src/.

Need another pair of eyes? Help me check my Level 5 Docker build files.

✔️ Awesome, I've got everything!

Your Dockerfile is ready to build the pinned CPU environment.

ⓧ I'd like to double check the full code

# syntax=docker/dockerfile:1
FROM --platform=linux/amd64 python:3.15.2-slim

ENV PYTHONHASHSEED=42 \
    LEVEL5_BASE_IMAGE=python:3.15.2-slim

WORKDIR /workspace
COPY requirements-level5.txt /tmp/requirements-level5.txt
RUN python -m pip install --no-cache-dir -r /tmp/requirements-level5.txt && \
    python -m pip install --no-cache-dir torch==2.15.0+cpu \
      --index-url https://download.pytorch.org/whl/cpu

CMD ["python", "-m", "src.neural_network_experiment", "audit"]

What should match?

Compare the base image and dependency installation lines carefully. The final command must target the audit entry point.

Add the fail-closed audit module

A fail-closed audit treats any contract drift as a stopping condition. The experiment cannot continue when Level 4 evidence differs from the approved handoff.

Why raw PyTorch?

Raw PyTorch keeps logits and gradients visible throughout the project. Higher-level training frameworks would hide the loop that later steps examine.

  • Create src/neural_network_experiment.py inside the existing src/ folder using your editor's new-file control.
  • Select the full-code tab below.
  • Copy the complete module into src/neural_network_experiment.py.

✔️ Awesome, I've got everything!

Your Level 5 module contains the audit entry point and the later experiment helpers.

ⓧ I'd like to double check the full code

from __future__ import annotations

import copy
import json
import os
import platform
import random
import sys
from importlib.metadata import version
from pathlib import Path
from statistics import fmean, pstdev
from typing import Any

import numpy as np
import pandas as pd
import torch
from sklearn.impute import SimpleImputer
from sklearn.model_selection import GroupShuffleSplit
from sklearn.preprocessing import StandardScaler
from torch import nn

from src.feature_experiment import (
    FORBIDDEN_HOLDOUT_USES,
    HOLDOUT_MARKER_PATH,
    LEVEL5_HANDOFF_PATH,
    RAW_FEATURES,
    TARGET_COLUMN,
    TRAIN_PATH,
    V2_REPORT_PATH,
    V2_SCHEMA_PATH,
    SchemaFeatureTransformer,
    audit_handoff,
    fingerprint,
    fold_signature,
    metric_values,
    read_json,
    write_json_exclusive,
)
from src.train_models import GROUP_COLUMN, SCHEMA_PATH, make_group_folds

REPORT_PATH = Path("reports/v3/neural_network_experiment.json")
CHECKPOINT_PATH = Path("artifacts/v3/level5_mlp_state.pt")
PREPROCESSOR_PATH = Path("artifacts/v3/level5_preprocessor.json")
LEVEL6_HANDOFF_PATH = Path("artifacts/v3/level6_finetune_handoff.json")
DOC_PATH = Path("docs/level-5-networks.md")
SEED = 42
THRESHOLD = 0.5
CONFIGS = (
    {"name": "compact", "hidden_units": 8, "dropout": 0.0,
     "learning_rate": 0.01, "weight_decay": 0.0},
    {"name": "regularized", "hidden_units": 16, "dropout": 0.25,
     "learning_rate": 0.01, "weight_decay": 0.001},
)


def _sha(root: Path, relative: Path) -> str:
    return fingerprint(root / relative)["sha256"]


def validate_level5_contract(
    handoff: dict[str, Any], feature_schema: dict[str, Any],
    report: dict[str, Any], marker: dict[str, Any]
) -> None:
    if handoff.get("handoff_version") != "2.0.0":
        raise ValueError("Expected Level 4 handoff version 2.0.0")
    if handoff.get("next_level") != "Level 5: Networks":
        raise ValueError("Handoff does not target Level 5: Networks")
    approved = handoff.get("selected_candidate_features", [])
    if not approved or not set(approved).issubset(feature_schema.get("candidate_features", [])):
        raise ValueError("Approved features do not match the v2 feature schema")
    if handoff.get("selected_variant") != report.get("selected_variant"):
        raise ValueError("Handoff and report disagree on selected variant")
    expected = report.get("identical_fold_signatures", [])
    if len(expected) != 4:
        raise ValueError("Exactly four inherited fold signatures are required")
    for variant in ("raw", "engineered"):
        observed = [row["fold_signature"] for row in report["results"][variant]["folds"]]
        if observed != expected:
            raise ValueError(f"Level 4 {variant} folds changed")
    if report.get("consumed_v1_holdout_read") is not False:
        raise ValueError("Level 4 report does not prove zero holdout reads")
    if marker.get("forbidden_uses") != list(FORBIDDEN_HOLDOUT_USES):
        raise ValueError("Consumed-holdout restrictions changed")


def audit_level4(root: Path) -> dict[str, Any]:
    base = audit_handoff(root)
    handoff = read_json(root / LEVEL5_HANDOFF_PATH)
    feature_schema = read_json(root / V2_SCHEMA_PATH)
    report = read_json(root / V2_REPORT_PATH)
    marker = read_json(root / HOLDOUT_MARKER_PATH)
    validate_level5_contract(handoff, feature_schema, report, marker)
    checks = {
        "feature_schema": (V2_SCHEMA_PATH, handoff["fingerprints"]["feature_schema"]["sha256"]),
        "feature_report": (V2_REPORT_PATH, handoff["fingerprints"]["feature_experiment"]["sha256"]),
        "development_data": (TRAIN_PATH, handoff["fingerprints"]["development_data"]["sha256"]),
    }
    for label, (path, expected) in checks.items():
        if _sha(root, path) != expected:
            raise ValueError(f"Fingerprint mismatch: {label}")
    return {
        "status": "passed",
        "holdout_reads": base["holdout_reads"],
        "selected_variant": handoff["selected_variant"],
        "approved_features": handoff["selected_candidate_features"],
        "fold_signatures": report["identical_fold_signatures"],
    }


def seed_everything(seed: int) -> None:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.use_deterministic_algorithms(True)


class BinaryMLP(nn.Module):
    def __init__(self, input_dim: int, hidden_units: int, dropout: float):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(input_dim, hidden_units), nn.ReLU(), nn.Dropout(dropout),
            nn.Linear(hidden_units, 1),
        )

    def forward(self, inputs: torch.Tensor) -> torch.Tensor:
        return self.network(inputs).squeeze(1)


def approved_matrix(root: Path) -> tuple[pd.DataFrame, pd.Series, pd.Series, dict[str, Any]]:
    handoff = read_json(root / LEVEL5_HANDOFF_PATH)
    schema = read_json(root / V2_SCHEMA_PATH)
    frame = pd.read_csv(root / TRAIN_PATH)
    transformer = SchemaFeatureTransformer(
        mode=handoff["selected_variant"],
        engineered_features=tuple(schema["engineered_feature_columns"]),
    )
    transformed = transformer.fit_transform(frame[list(RAW_FEATURES)])
    approved = handoff["selected_candidate_features"]
    missing = sorted(set(approved) - set(transformed.columns))
    if missing:
        raise ValueError(f"Approved features unavailable: {missing}")
    return transformed[approved], frame[TARGET_COLUMN], frame[GROUP_COLUMN], handoff


def fit_preprocessor(x_fit: pd.DataFrame) -> tuple[SimpleImputer, StandardScaler]:
    imputer = SimpleImputer(strategy="mean")
    imputed = imputer.fit_transform(x_fit)
    scaler = StandardScaler()
    scaler.fit(imputed)
    return imputer, scaler


def transform(x: pd.DataFrame, imputer: SimpleImputer, scaler: StandardScaler) -> np.ndarray:
    return scaler.transform(imputer.transform(x)).astype(np.float32)


def inner_group_split(x: pd.DataFrame, y: pd.Series, groups: pd.Series, seed: int) -> tuple[np.ndarray, np.ndarray]:
    splitter = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=seed)
    fit_idx, stop_idx = next(splitter.split(x, y, groups))
    fit_groups = set(groups.iloc[fit_idx])
    stop_groups = set(groups.iloc[stop_idx])
    if fit_groups & stop_groups:
        raise ValueError("Inner runner leakage detected")
    return fit_idx, stop_idx


def _tensors(x: np.ndarray, y: np.ndarray) -> tuple[torch.Tensor, torch.Tensor]:
    return torch.tensor(x, dtype=torch.float32), torch.tensor(y, dtype=torch.float32)


def _loss(model: BinaryMLP, x: torch.Tensor, y: torch.Tensor) -> float:
    model.eval()
    with torch.no_grad():
        return float(nn.BCEWithLogitsLoss()(model(x), y).item())


def train_with_early_stopping(
    x_fit: np.ndarray, y_fit: np.ndarray, x_stop: np.ndarray, y_stop: np.ndarray,
    config: dict[str, Any], seed: int, max_epochs: int = 200, patience: int = 15,
) -> tuple[BinaryMLP, list[dict[str, float]], int]:
    seed_everything(seed)
    tx, ty = _tensors(x_fit, y_fit)
    sx, sy = _tensors(x_stop, y_stop)
    model = BinaryMLP(tx.shape[1], config["hidden_units"], config["dropout"])
    criterion = nn.BCEWithLogitsLoss()
    optimizer = torch.optim.Adam(
        model.parameters(), lr=config["learning_rate"],
        weight_decay=config["weight_decay"],
    )
    history: list[dict[str, float]] = []
    best_loss, best_epoch, stale = float("inf"), 0, 0
    best_state: dict[str, torch.Tensor] | None = None
    for epoch in range(1, max_epochs + 1):
        model.train()
        optimizer.zero_grad(set_to_none=True)
        loss = criterion(model(tx), ty)
        loss.backward()
        optimizer.step()
        train_loss = _loss(model, tx, ty)
        stop_loss = _loss(model, sx, sy)
        history.append({"epoch": epoch, "train_loss": train_loss, "validation_loss": stop_loss})
        if stop_loss < best_loss - 1e-6:
            best_loss, best_epoch, stale = stop_loss, epoch, 0
            best_state = copy.deepcopy(model.state_dict())
        else:
            stale += 1
            if stale >= patience:
                break
    if best_state is None:
        raise RuntimeError("Early stopping never recorded a checkpoint")
    model.load_state_dict(best_state)
    return model, history, best_epoch


def train_fixed(
    x_fit: np.ndarray, y_fit: np.ndarray, config: dict[str, Any], seed: int,
    epochs: int, x_watch: np.ndarray | None = None, y_watch: np.ndarray | None = None,
) -> tuple[BinaryMLP, list[dict[str, float]]]:
    seed_everything(seed)
    tx, ty = _tensors(x_fit, y_fit)
    model = BinaryMLP(tx.shape[1], config["hidden_units"], config["dropout"])
    criterion = nn.BCEWithLogitsLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=config["learning_rate"], weight_decay=config["weight_decay"])
    history: list[dict[str, float]] = []
    watch = _tensors(x_watch, y_watch) if x_watch is not None and y_watch is not None else None
    for epoch in range(1, epochs + 1):
        model.train()
        optimizer.zero_grad(set_to_none=True)
        loss = criterion(model(tx), ty)
        loss.backward()
        optimizer.step()
        if watch is not None:
            history.append({"epoch": epoch, "train_loss": _loss(model, tx, ty),
                            "validation_loss": _loss(model, watch[0], watch[1])})
    return model, history


def evaluate_model(model: BinaryMLP, x: np.ndarray, y: np.ndarray) -> dict[str, float]:
    tx, ty = _tensors(x, y)
    model.eval()
    with torch.no_grad():
        probabilities = torch.sigmoid(model(tx)).cpu().numpy()
    values = metric_values(y.astype(int), (probabilities >= THRESHOLD).astype(int))
    values["loss"] = _loss(model, tx, ty)
    return values


def summarize_folds(folds: list[dict[str, Any]]) -> dict[str, Any]:
    metrics = ("accuracy", "precision", "recall", "f1", "loss")
    return {
        "validation_mean": {m: fmean(row["validation"][m] for row in folds) for m in metrics},
        "validation_stdev": {m: pstdev(row["validation"][m] for row in folds) for m in metrics},
        "gap_mean": {m: fmean(row["generalization_gap"][m] for row in folds) for m in metrics},
    }


def evaluate_config(
    x: pd.DataFrame, y: pd.Series, groups: pd.Series,
    folds: list[tuple[np.ndarray, np.ndarray]], config: dict[str, Any],
) -> dict[str, Any]:
    records = []
    for fold_number, (fit_rows, validation_rows) in enumerate(folds, start=1):
        outer_fit_groups = set(groups.iloc[fit_rows])
        outer_validation_groups = set(groups.iloc[validation_rows])
        if outer_fit_groups & outer_validation_groups:
            raise ValueError("Outer runner leakage detected")
        x_outer, y_outer, g_outer = x.iloc[fit_rows], y.iloc[fit_rows], groups.iloc[fit_rows]
        inner_fit, inner_stop = inner_group_split(x_outer, y_outer, g_outer, SEED + fold_number)
        inner_imp, inner_scale = fit_preprocessor(x_outer.iloc[inner_fit])
        _, history, best_epoch = train_with_early_stopping(
            transform(x_outer.iloc[inner_fit], inner_imp, inner_scale), y_outer.iloc[inner_fit].to_numpy(np.float32),
            transform(x_outer.iloc[inner_stop], inner_imp, inner_scale), y_outer.iloc[inner_stop].to_numpy(np.float32),
            config, SEED + fold_number,
        )
        outer_imp, outer_scale = fit_preprocessor(x_outer)
        model, _ = train_fixed(
            transform(x_outer, outer_imp, outer_scale), y_outer.to_numpy(np.float32),
            config, SEED + fold_number, best_epoch,
        )
        train_metrics = evaluate_model(model, transform(x_outer, outer_imp, outer_scale), y_outer.to_numpy(np.float32))
        validation_metrics = evaluate_model(
            model, transform(x.iloc[validation_rows], outer_imp, outer_scale),
            y.iloc[validation_rows].to_numpy(np.float32),
        )
        gap = {m: (validation_metrics[m] - train_metrics[m] if m == "loss" else train_metrics[m] - validation_metrics[m])
               for m in train_metrics}
        records.append({
            "fold": fold_number, "fold_signature": fold_signature(groups, fit_rows, validation_rows),
            "outer_runner_overlap": [], "inner_runner_overlap": [], "best_epoch": best_epoch,
            "loss_history": history, "train": train_metrics,
            "validation": validation_metrics, "generalization_gap": gap,
        })
    return {"config": config, "folds": records, "aggregate": summarize_folds(records)}


def dependency_versions() -> dict[str, str]:
    return {name: version(name) for name in ("numpy", "pandas", "scikit-learn", "torch")}


def serialize_preprocessor(imputer: SimpleImputer, scaler: StandardScaler, features: list[str]) -> dict[str, Any]:
    return {
        "feature_names": features, "imputation": "mean", "feature_selection": "none",
        "imputer_statistics": imputer.statistics_.tolist(),
        "scaler_mean": scaler.mean_.tolist(), "scaler_scale": scaler.scale_.tolist(),
    }


def run_experiment(root: Path) -> dict[str, Any]:
    audit = audit_level4(root)
    for relative in (REPORT_PATH, CHECKPOINT_PATH, PREPROCESSOR_PATH, LEVEL6_HANDOFF_PATH):
        if (root / relative).exists():
            raise FileExistsError(f"Refusing to overwrite {relative}")
    x, y, groups, handoff = approved_matrix(root)
    folds = make_group_folds(pd.read_csv(root / TRAIN_PATH), read_json(root / SCHEMA_PATH))
    signatures = [fold_signature(groups, fit, valid) for fit, valid in folds]
    if signatures != audit["fold_signatures"]:
        raise ValueError("Current runner-safe folds do not match Level 4")
    results = {config["name"]: evaluate_config(x, y, groups, folds, config) for config in CONFIGS}
    selected_name = max(
        results,
        key=lambda name: (
            results[name]["aggregate"]["validation_mean"]["f1"],
            -abs(results[name]["aggregate"]["gap_mean"]["f1"]),
            name == "regularized",
        ),
    )
    selected = results[selected_name]["config"]
    inner_fit, inner_stop = inner_group_split(x, y, groups, SEED)
    inner_imp, inner_scale = fit_preprocessor(x.iloc[inner_fit])
    _, final_history, best_epoch = train_with_early_stopping(
        transform(x.iloc[inner_fit], inner_imp, inner_scale), y.iloc[inner_fit].to_numpy(np.float32),
        transform(x.iloc[inner_stop], inner_imp, inner_scale), y.iloc[inner_stop].to_numpy(np.float32),
        selected, SEED,
    )
    imputer, scaler = fit_preprocessor(x)
    model, _ = train_fixed(transform(x, imputer, scaler), y.to_numpy(np.float32), selected, SEED, best_epoch)
    checkpoint = {
        "model_state": model.state_dict(), "input_dim": x.shape[1], "config": selected,
        "feature_names": list(x.columns), "threshold": THRESHOLD,
        "seed": SEED, "best_epoch": best_epoch,
    }
    (root / CHECKPOINT_PATH).parent.mkdir(parents=True, exist_ok=True)
    torch.save(checkpoint, root / CHECKPOINT_PATH)
    write_json_exclusive(root / PREPROCESSOR_PATH, serialize_preprocessor(imputer, scaler, list(x.columns)))
    level4_report = read_json(root / V2_REPORT_PATH)
    report = {
        "report_version": "3.0.0", "evidence_scope": "development_only",
        "consumed_v1_holdout_read": False, "threshold": THRESHOLD,
        "determinism": {"seed": SEED, "torch_use_deterministic_algorithms": True,
                        "cross_platform_bitwise_guarantee": False},
        "dependencies": dependency_versions(), "runtime": {"python": sys.version, "platform": platform.platform(),
                        "base_image": os.environ.get("LEVEL5_BASE_IMAGE", "unknown")},
        "approved_features": list(x.columns), "feature_selection": "none",
        "fold_signatures": signatures,
        "classical_development_controls": {
            variant: level4_report["results"][variant]["aggregate"] for variant in ("raw", "engineered")
        },
        "mlp_results": results, "selected_config": selected_name,
        "final_development_fit": {"best_epoch": best_epoch, "loss_history": final_history,
                                  "train_metrics": evaluate_model(model, transform(x, imputer, scaler), y.to_numpy(np.float32))},
        "limitations": [
            "All Level 5 comparisons are development-only because the v1 holdout is already consumed.",
            "Level 5 fits imputation and scaling inside each relevant training boundary.",
            "The inherited v1 process used development-wide mean imputation before splitting, so the historical pipeline is not claimed to be leakage-free.",
            "PyTorch does not guarantee identical results across releases, commits, or platforms even with deterministic settings.",
        ],
    }
    write_json_exclusive(root / REPORT_PATH, report)
    handoff6 = {
        "handoff_version": "3.0.0", "created_by": "Level 5: Networks",
        "next_level": "Level 6: Fine-tune", "consumed_v1_holdout_read": False,
        "forbidden_uses": list(FORBIDDEN_HOLDOUT_USES),
        "approved_features": list(x.columns), "selected_config": selected,
        "threshold": THRESHOLD, "seed": SEED, "best_epoch": best_epoch,
        "fingerprints": {
            "level4_handoff": fingerprint(root / LEVEL5_HANDOFF_PATH),
            "development_data": fingerprint(root / TRAIN_PATH),
            "feature_schema": fingerprint(root / V2_SCHEMA_PATH),
            "level5_report": fingerprint(root / REPORT_PATH),
            "checkpoint": fingerprint(root / CHECKPOINT_PATH),
            "preprocessor": fingerprint(root / PREPROCESSOR_PATH),
        },
        "limitations": report["limitations"],
    }
    write_json_exclusive(root / LEVEL6_HANDOFF_PATH, handoff6)
    return {"selected_config": selected_name, "best_epoch": best_epoch,
            "report": str(REPORT_PATH), "handoff": str(LEVEL6_HANDOFF_PATH)}


def concept_demo(root: Path) -> dict[str, Any]:
    audit_level4(root)
    seed_everything(SEED)
    inputs = torch.tensor([[0.5, -1.0]], dtype=torch.float32)
    target = torch.tensor([1.0], dtype=torch.float32)
    neuron = nn.Linear(2, 1)
    criterion = nn.BCEWithLogitsLoss()
    optimizer = torch.optim.Adam(neuron.parameters(), lr=0.1)
    before_logit = neuron(inputs).squeeze(1)
    before_loss = criterion(before_logit, target)
    optimizer.zero_grad(set_to_none=True)
    before_loss.backward()
    gradients = {name: parameter.grad.detach().tolist() for name, parameter in neuron.named_parameters()}
    optimizer.step()
    after_logit = neuron(inputs).squeeze(1)
    after_loss = criterion(after_logit, target)
    return {"tensor_shape": list(inputs.shape), "neuron_equation": "z = xW^T + b",
            "before_logit_not_probability": float(before_logit.item()),
            "gradients": gradients, "before_loss": float(before_loss.item()),
            "after_logit_not_probability": float(after_logit.item()), "after_loss": float(after_loss.item())}


def document_experiment(root: Path) -> None:
    report = read_json(root / REPORT_PATH)
    selected = report["selected_config"]
    aggregate = report["mlp_results"][selected]["aggregate"]
    metrics = aggregate["validation_mean"]
    text = f"""# Level 5: Networks

## Result

Selected configuration: `{selected}` on development-only runner-safe folds.

| Metric | Mean outer validation |
| --- | ---: |
| F1 | {metrics['f1']:.3f} |
| Precision | {metrics['precision']:.3f} |
| Recall | {metrics['recall']:.3f} |
| Loss | {metrics['loss']:.3f} |

## Architecture and training

The model is `Linear -> ReLU -> Dropout -> Linear`. Training uses raw logits with binary cross-entropy, explicit backpropagation, Adam updates, nested runner-safe early stopping, and threshold `{report['threshold']}` only for metrics.

## Evidence boundary

The v1 holdout is consumed and was not read. These are development-only comparisons against Level 4's raw and engineered controls. Level 5 preprocessing is fold-local, but v1 historically used development-wide mean imputation before splitting. The full historical pipeline is not described as leakage-free.

## Artifacts

- `{REPORT_PATH}`
- `{CHECKPOINT_PATH}`
- `{PREPROCESSOR_PATH}`
- `{LEVEL6_HANDOFF_PATH}`

## Reproduce

```bash
docker build --platform linux/amd64 -f Dockerfile.level5 -t running-readiness-level5:2.15.0 .
docker run --rm --platform linux/amd64 --mount type=bind,src="$(pwd)",dst=/workspace -w /workspace running-readiness-level5:2.15.0 python -m src.neural_network_experiment audit
docker run --rm --platform linux/amd64 --mount type=bind,src="$(pwd)",dst=/workspace -w /workspace running-readiness-level5:2.15.0 python -m unittest -v tests/test_neural_networks.py
```
"""
    (root / DOC_PATH).parent.mkdir(parents=True, exist_ok=True)
    (root / DOC_PATH).write_text(text, encoding="utf-8")
    readme = root / "README.md"
    marker = "<!-- LEVEL_5_NETWORKS -->"
    existing = readme.read_text(encoding="utf-8") if readme.exists() else "# Running Readiness\n"
    line = f"{marker}\n- [Level 5: Networks]({DOC_PATH.as_posix()})\n"
    if marker in existing:
        existing = existing.split(marker)[0].rstrip() + "\n\n" + line
    else:
        existing = existing.rstrip() + "\n\n" + line
    readme.write_text(existing, encoding="utf-8")


def main() -> None:
    root = Path.cwd()
    command = sys.argv[1] if len(sys.argv) > 1 else "audit"
    if command == "audit":
        payload = audit_level4(root)
    elif command == "concepts":
        payload = concept_demo(root)
    elif command == "run":
        payload = run_experiment(root)
    elif command == "document":
        document_experiment(root)
        payload = {"documentation": str(DOC_PATH), "status": "written"}
    else:
        raise SystemExit("Use: audit | concepts | run | document")
    print(json.dumps(payload, indent=2, sort_keys=True))


if __name__ == "__main__":
    main()

How is this module organized?

  • validate_level5_contract() checks the handoff version and target label. It also checks approved features plus inherited fold signatures.
  • audit_level4() verifies the schema and report fingerprints. It also verifies the development-data fingerprint plus holdout restrictions.
  • seed_everything() activates the fixed seed and deterministic algorithms used by later commands.
  • main() dispatches the audit before exposing the later concept and experiment commands.
  • Save src/neural_network_experiment.py.
  • Confirm that neural_network_experiment.py appears inside the existing src/ folder in your editor's file tree.

Module path look wrong?

The file must sit directly inside src/. A nested folder or altered filename prevents Python from finding the module.

Need help comparing the module? Help me find a mismatch in my Level 5 source file.

Build and verify the CPU image

The image build turns the two environment files into a reusable Linux amd64 runtime. The first build takes longer because Docker downloads the base image and Python wheels.

  • Build the Level 5 image from the running-readiness repository by running this command:
docker build --platform linux/amd64 -f Dockerfile.level5 -t running-readiness-level5:2.15.0 .

What does this build do?

  • --platform linux/amd64 selects the architecture supported by the CPU wheel.
  • -f Dockerfile.level5 selects the Level 5 build recipe.
  • -t running-readiness-level5:2.15.0 gives the finished image its reusable name and tag.

When the build completes, your terminal confirms that the image tagged running-readiness-level5:2.15.0 is ready.

Image build failed?

Confirm that Docker Desktop is running. Check that the terminal is currently inside the existing running-readiness repository.

Check that Dockerfile.level5 and requirements-level5.txt are beside each other. A missing file prevents the copy step from completing.

Still blocked? Help me diagnose my Level 5 Docker image build.

Before you run the audit, do you expect unchanged Level 4 evidence to pass or stop?

  • Run the fail-closed audit inside the new image with this command:
docker run --rm --platform linux/amd64 --mount type=bind,src="$(pwd)",dst=/workspace -w /workspace running-readiness-level5:2.15.0 python -m src.neural_network_experiment audit

What does the audit prove?

  • The handoff still uses version 2.0.0 for Level 5: Networks.
  • The selected feature list and selected variant still agree with the Level 4 schema plus report.
  • The schema and report fingerprints still match. The development-data fingerprint must also match.
  • The consumed-holdout marker still forbids model selection and feature selection. It also forbids threshold tuning plus model comparison.

You should see status: passed plus holdout_reads: 0. You should also see four inherited fold signatures.

That is the safety boundary established. Your CPU environment can now run Level 5 code without treating the consumed v1 holdout as fresh evidence.

Audit stopped the run?

A stopped audit means the repository no longer matches the locked Level 4 handoff. Read the final audit message to identify the changed contract or fingerprint.

Restore the affected Level 4 artifact from its trusted version before rerunning the command. Do not weaken the audit or regenerate the fingerprint around changed evidence.

Need help locating the drift? Help me interpret the failed Level 4 audit.

Your Linux CPU environment is ready. Next, you will trace one neural update from its raw logit through loss and backpropagation.

Trace One Neural Update

Your Level 4 audit now protects every experiment from changed evidence or a consumed-holdout read. PyTorch can now focus on a single transparent update.

A classifier eventually needs a probability. A bare affine neuron produces a raw logit with no probability interpretation.

You will trace one input through the neuron equation. You will then inspect how loss flows backward before Adam changes the parameters.

In this step, get ready to:
  • Trace an input tensor through the affine neuron equation.
  • Inspect how backpropagation produces weight gradients plus bias gradients.
  • Run one Adam update to observe the loss change.
Trace the affine forward pass

A tensor stores the feature values that flow into the neuron. The affine layer multiplies those values by learned weights before adding a bias.

What does the neuron equation mean?

The equation z = xW^T + b describes the forward pass. Its result is the raw logit stored in before_logit.

  • The x term represents the input features in inputs.
  • The W term represents the learnable weights in neuron.weight.
  • The b term represents the learnable bias in neuron.bias.
  • Find concept_demo() in src/neural_network_experiment.py.
  • Locate the inputs tensor that contains one row with two feature values.
  • Locate the target tensor that contains the positive binary label.
  • Match inputs to x in the neuron equation.
  • Match neuron.weight to W in the neuron equation.
  • Match neuron.bias to b in the neuron equation.
Follow the loss into the parameters

Binary cross-entropy measures how poorly the logit supports the target label. BCEWithLogitsLoss performs the stable loss calculation directly from that logit.

Backpropagation applies the chain rule from the loss back to each parameter. The resulting gradients tell Adam how to update the neuron.

  • Locate the criterion that creates BCEWithLogitsLoss.
  • Follow before_loss.backward() to the backward pass.
  • Inspect the gradients dictionary that captures each named parameter gradient.
  • Follow optimizer.step() to the parameter update.
  • Compare before_logit with after_logit after the update.
  • Compare before_loss with after_loss after the update.

How does one update work?

  • The forward pass produces a logit from the current weights plus bias.
  • The loss measures the logit's error against the target.
  • The backward pass stores a gradient on each learnable parameter.
  • The optimizer uses those gradients to change the parameter values.
Run the audited neuron trace

Before you run the trace, do you expect one optimizer update to leave the loss unchanged or move it?

  • Run the neuron trace in your existing Docker image by entering this command:
docker run --rm --platform linux/amd64 --mount type=bind,src="$(pwd)",dst=/workspace -w /workspace running-readiness-level5:2.15.0 python -m src.neural_network_experiment concepts

What does this command do?

The container mounts your current repository at /workspace. It runs the concepts CLI path from that location.

The command calls audit_level4() before creating any demo tensors. Training stays blocked if the locked evidence has changed.

The temporary container is removed after the JSON output is printed. Your repository remains available through the bind mount.

You will see tensor_shape report one row with two features. The output also lists gradients for the weight plus bias.

The before_logit_not_probability field makes the missing probability interpretation explicit. The after_logit_not_probability field shows the result after Adam changes the parameters.

Compare before_loss with after_loss. Their change proves that backward() plus optimizer.step() changed the neuron.

You have now watched a neuron turn a backward pass into a measurable parameter update. The consumed v1 holdout remained unread throughout the trace.

Neuron trace not running?

Confirm that your terminal is inside the running-readiness repository. The bind mount uses your terminal's current location.

If the audit stops the demo, restore the locked Level 4 evidence before trying again. The failure is protecting your evidence boundary.

If the container cannot start, return to Docker Desktop from earlier. Wait until its engine reports that it is running.

Help me diagnose my neuron trace command.

The one-neuron update is now visible from input tensor to changed loss. Next, you will place the same mechanics inside a multilayer network with ReLU plus probability evaluation.

Compare MLPs on Locked Folds

Your one-neuron trace made the update mechanics in PyTorch visible. That trace still produces one affine logit with no nonlinear feature interactions.

This step compares two multilayer perceptrons on the exact four grouped cross-validation folds inherited from Level 4. Fold-local preprocessing keeps validation runners outside every fitted statistic.

In this step, get ready to:
  • Materialize only the handoff-approved prediction-time features from the selected Level 4 variant.
  • Compare a compact ReLU network with a dropout and weight-decay configuration.
  • Verify loss histories, classification metrics, generalization gaps, and locked runner boundaries.
Protect the development-only boundary

The evidence boundary starts before model training. approved_matrix() transforms only TRAIN_PATH data using the Level 4 selected variant.

The report records feature_selection as none. This keeps the approved feature list fixed throughout the comparison.

  • In your code editor's file explorer, select src/neural_network_experiment.py.
  • Press Cmd+F on macOS or Ctrl+F on Windows to open the find box.
  • Enter def approved_matrix in the find box.
  • Confirm that pd.read_csv(root / TRAIN_PATH) supplies the development rows.
  • Confirm that selected_candidate_features supplies the approved feature list.
  • Replace the search term with data/test_v1.csv.
  • Confirm that your editor reports no matches.

Why does the consumed holdout stay absent?

The v1 holdout has already been consumed. Reading it here would reuse evidence that cannot support neural-network selection or comparison.

The Level 5 report therefore labels its evidence as development-only. It includes Level 4's raw and engineered development controls without presenting a fresh holdout result.

Inspect the runner-safe training design

Every outer fold contains an inner stopping split built from disjoint runner groups. That inner split chooses the epoch without touching the outer validation runners.

After the epoch is chosen, preprocessing is refitted on the full outer training fold. The model then scores the untouched outer validation fold.

  • Replace the search term with CONFIGS = (.
  • Confirm that compact uses 8 hidden units with zero dropout and zero weight decay.
  • Confirm that regularized uses 16 hidden units with 0.25 dropout and 0.001 weight decay.
  • Replace the search term with def evaluate_config.
  • Confirm that each outer fold checks runner separation before training.
  • Confirm that every fold records empty outer_runner_overlap and inner_runner_overlap lists.

What happens inside each fold?

  • The inner_group_split() function creates a runner-disjoint stopping split with GroupShuffleSplit.
  • The fit_preprocessor() function learns mean imputation statistics from the relevant fit rows.
  • The same function learns standard scaling statistics from those fit rows.
  • The train_with_early_stopping() function records per-epoch training loss and stopping-validation loss.
  • The train_fixed() function refits the network for the chosen epoch count on the full outer training fold.
  • The evaluate_model() function applies sigmoid before calculating metrics at the fixed 0.5 threshold.
Prove the folds stayed locked

The experiment audits Level 4 again before it trains either configuration. It also rejects any current fold signature that differs from the four inherited signatures.

  • Predict whether changing hidden units and regularization should change the runner assignments.
  • Train both MLP configurations inside the audited container by running this command:
python -m src.neural_network_experiment run

What does this run do?

  • The command repeats the Level 4 contract audit before loading development data.
  • It recreates the inherited outer folds before comparing their signatures with the locked signatures.
  • It trains both configurations with deterministic seed 42 behavior.
  • It writes the development-only comparison to reports/v3/neural_network_experiment.json.

When the run finishes, your terminal prints a JSON summary containing selected_config. It also prints best_epoch and the report path.

  • Confirm that the terminal returns to its prompt after printing the summary.
  • In your code editor's file explorer, select reports/v3/neural_network_experiment.json.

Did the experiment stop early?

If the run reports Refusing to overwrite, a Level 5 output already exists. Preserve that output before deciding whether this repository should be rerun from scratch.

If the audit fails, restore the locked Level 4 evidence before running any training. The failed audit is protecting the comparison boundary.

Help me diagnose why the locked-fold experiment cannot create its report.

  • Predict which fields must remain identical across both configurations before you inspect their fold records.
  • Confirm that report_version is 3.0.0.
  • Confirm that evidence_scope is development_only.
  • Confirm that consumed_v1_holdout_read is false.
  • Expand mlp_results to inspect compact and regularized.
  • Confirm that each configuration contains exactly four fold records.
  • Compare each configuration's fold signatures with the top-level fold_signatures list.
  • Confirm that every outer and inner runner overlap list is empty.

What should the report prove?

  • Both configurations reuse the same four inherited fold signatures.
  • Every fold includes training and stopping-validation loss histories.
  • Every fold includes train and validation accuracy, precision, recall, F1, and loss values.
  • Every fold includes generalization gaps between its train and validation evidence.
  • The aggregate sections include validation means and standard deviations.
  • The classical controls include Level 4 raw and engineered development evidence.
  • The limitations state that the comparison is development-only.

Strong work. Your MLP evidence now uses the inherited runner boundaries without reopening the consumed holdout.

Your two neural configurations now have comparable development evidence. Next, you'll trace how the selected configuration becomes a reloadable checkpoint and a fingerprinted Level 6 handoff.

Checkpoint the Selected Network

Your two MLP configurations now have development-only evidence from the four inherited runner-safe folds. Their cross-validation results identify which configuration generalizes best across runners.

Those metrics cannot preserve learned parameters for fine-tuning. A PyTorch checkpoint carries the selected network into Level 6 with its preprocessing recipe and evidence lineage intact.

In this step, get ready to:
  • Trace how development evidence selected the final network.
  • Confirm the checkpoint and preprocessing metadata.
  • Verify the Level 6 fingerprint chain and permanent holdout prohibition.
Trace the deterministic selection

Mean outer-fold F1 ranks the configurations using development evidence. A smaller absolute F1 generalization gap resolves a tie before grouped early stopping chooses the final epoch count.

  • In VS Code Explorer, select reports/v3/neural_network_experiment.json.
  • Locate the selected_config field.
  • Compare each configuration's validation_mean value for f1.
  • Compare each configuration's absolute gap_mean value for f1.
  • Locate the final_development_fit object.
  • Read the selected best_epoch value.
  • Confirm that loss_history records the grouped inner stopping run.
  • Confirm that dependencies records the package version inventory.

You'll see one selected_config plus one fixed best_epoch. These values connect the fold comparison to the all-development training recipe.

How Selection Becomes a Final Model

  • Mean outer-fold F1 rewards stronger validation performance across unseen runners.
  • The absolute F1-gap tie-break favors closer training and validation behavior when mean F1 is tied.
  • A runner-safe grouped inner split chooses the epoch count without reading the consumed holdout.
  • The final refit learns mean imputation and standard scaling statistics from every development row.
Inspect checkpoint and preprocessing metadata

The checkpoint stores the selected network state and the fields needed to rebuild its architecture. A separate JSON artifact preserves the exact transformations required before inference.

  • Select src/neural_network_experiment.py in the VS Code Explorer.
  • Locate the checkpoint dictionary inside run_experiment().
  • Confirm that the dictionary includes model_state, input_dim, and config.
  • Confirm that the dictionary also includes feature_names, threshold, seed, and best_epoch.
  • Locate the torch.save call that writes to CHECKPOINT_PATH.
  • In the VS Code Explorer, confirm that artifacts/v3/level5_mlp_state.pt is listed.
  • Select artifacts/v3/level5_preprocessor.json.
  • Verify the feature_names, imputation, feature_selection, imputer_statistics, scaler_mean, and scaler_scale fields.

You'll see mean recorded for imputation. You'll also see none recorded for feature selection.

That is the reusable bundle secured: the network state has its exact preprocessing recipe beside it.

How the Files Work Together

The model_state object holds the learned weights and biases. The architecture fields provide everything required to recreate the same BinaryMLP.

PyTorch supports restricted checkpoint loading with weights_only=True. Using map_location="cpu" keeps the reload independent of accelerator availability.

Missing a Generated Artifact?

  • Return to the terminal output from the run command.
  • Find the first exception before checking later output.
  • Refresh the VS Code file tree if the successful run completed without an exception.

Need help tracing the artifact failure?

Verify the Level 6 handoff

The Level 6 handoff binds the final model to the evidence that produced it. File fingerprints expose changed bytes before a later fine-tuning workflow trusts the artifacts.

Before the final check, consider this question: can filenames alone prove that each artifact still contains its original bytes.

  • Select artifacts/v3/level6_finetune_handoff.json in the VS Code Explorer.
  • Confirm that handoff_version is 3.0.0.
  • Confirm that created_by is Level 5: Networks and next_level is Level 6: Fine-tune.
  • Confirm that consumed_v1_holdout_read is false.
  • Verify that forbidden_uses still lists model selection, feature selection, threshold tuning, and model comparison.
  • Compare approved_features, selected_config, threshold, seed, and best_epoch with the Level 5 report.
  • Inspect fingerprints for level4_handoff, development_data, feature_schema, level5_report, checkpoint, and preprocessor.
  • Confirm that limitations preserves the development-only evidence boundary.

You'll see six fingerprint entries tied to the exact evidence and model artifacts. You'll also see the consumed-holdout read fixed at false.

That closes the reproducibility gap. Your selected network now has a CPU checkpoint, matching preprocessing metadata, and a fingerprinted path into Level 6.

Your selected network is ready to survive beyond one experiment process. Next, you'll prove its group safety, reproducibility, and reload behavior with executable tests.

Prove the Workflow and Publish Evidence

Your selected MLP now has a reloadable CPU checkpoint. Its preprocessing metadata is serialized.

A portfolio claim is credible when its safeguards are executable. You will prove those safeguards with tests before generating the Level 5 evidence page.

In this step, get ready to:
  • Create focused tests for runner separation plus schema validation.
  • Prove same-seed reproducibility plus CPU checkpoint reload behavior.
  • Generate the Level 5 documentation from the development-only report.
Build the focused test suite

The Python unittest framework turns the workflow's safety claims into executable checks. The first pair proves group separation plus fail-closed schema validation.

  • In the Explorer sidebar in VS Code, create tests/test_neural_networks.py inside the repository.
  • Add the test dependencies plus project imports by pasting this code into tests/test_neural_networks.py:
import tempfile
import unittest
from pathlib import Path

import numpy as np
import pandas as pd
import torch

from src.neural_network_experiment import (
    BinaryMLP,
    CONFIGS,
    inner_group_split,
    train_fixed,
    train_with_early_stopping,
    validate_level5_contract,
)

What do these imports support?

  • The standard-library imports create isolated checkpoint files plus define the test case.
  • NumPy plus pandas provide deterministic test matrices plus grouped labels.
  • PyTorch loads model state for prediction comparison.
  • The project imports exercise the same training functions used by the experiment.
  • Save tests/test_neural_networks.py.
  • Confirm tests/test_neural_networks.py appears under the tests folder in the Explorer sidebar.

File missing from the Explorer?

Check that you created the file inside the running-readiness repository. Confirm that the path is exactly tests/test_neural_networks.py.

Need help locating it?

  • Add the shared deterministic fixture below the import block:
class NeuralNetworkTests(unittest.TestCase):
    def setUp(self):
        self.x = np.array([
            [-2.0, -1.0], [-1.5, -0.5], [-1.0, -1.5], [-0.5, -2.0],
            [0.5, 2.0], [1.0, 1.5], [1.5, 0.5], [2.0, 1.0],
        ], dtype=np.float32)
        self.y = np.array([0, 0, 0, 0, 1, 1, 1, 1], dtype=np.float32)
        self.config = CONFIGS[0]

What does the fixture provide?

  • The input matrix contains two features across eight synthetic observations.
  • The target array gives the tests a balanced binary outcome.
  • The compact configuration keeps every training check fast on the CPU.
  • Save tests/test_neural_networks.py.
  • Check that the fixture imports successfully by running the existing container command:
docker run --rm --platform linux/amd64 --mount type=bind,src="$(pwd)",dst=/workspace -w /workspace running-readiness-level5:2.15.0 python -m unittest -v tests/test_neural_networks.py

What does this check prove?

The audited Docker image can import the test module plus its Level 5 dependencies. This catches path or syntax problems before the assertions are added.

The test runner should complete without an import failure. Your shared fixture is ready for focused assertions.

Seeing an import failure?

Confirm that your terminal is using the running-readiness repository. Check that the image name is running-readiness-level5:2.15.0.

Still blocked?

  • Add the runner-separation test below setUp():
    def test_group_separation(self):
        frame = pd.DataFrame(self.x, columns=["a", "b"])
        target = pd.Series(self.y)
        groups = pd.Series(["r1", "r1", "r2", "r2", "r3", "r3", "r4", "r4"])
        fit, stop = inner_group_split(frame, target, groups, 42)
        self.assertFalse(set(groups.iloc[fit]) & set(groups.iloc[stop]))

What does this test prove?

The test passes four runner identities into inner_group_split(). Its assertion requires the fitting groups plus stopping groups to remain disjoint.

  • Save tests/test_neural_networks.py.
  • Run the test module inside the container:
docker run --rm --platform linux/amd64 --mount type=bind,src="$(pwd)",dst=/workspace -w /workspace running-readiness-level5:2.15.0 python -m unittest -v tests/test_neural_networks.py

What does this run prove?

The test runner executes the group-safety assertion against the real inner-split helper. A passing result proves that the stopping split shares no runner with its fit split.

You should see test_group_separation pass. The run ends with OK.

Runner-separation test failing?

Compare the group labels with the code block above. A missing duplicate can change the grouped split behavior.

Investigate the failure with help.

  • Add the schema-drift test below test_group_separation():
    def test_schema_validation_rejects_unapproved_feature(self):
        handoff = {"handoff_version": "2.0.0", "next_level": "Level 5: Networks",
                   "selected_candidate_features": ["distance_km"], "selected_variant": "raw"}
        schema = {"candidate_features": ["distance_km"]}
        fold = {"fold_signature": "same"}
        report = {"selected_variant": "raw", "identical_fold_signatures": ["same"] * 4,
                  "consumed_v1_holdout_read": False,
                  "results": {"raw": {"folds": [fold] * 4}, "engineered": {"folds": [fold] * 4}}}
        marker = {"forbidden_uses": ["model selection", "feature selection", "threshold tuning", "model comparison"]}
        validate_level5_contract(handoff, schema, report, marker)
        handoff["selected_candidate_features"] = ["future_weather"]
        with self.assertRaises(ValueError):
            validate_level5_contract(handoff, schema, report, marker)

How does this test fail closed?

  • The first validation uses an approved feature plus four inherited fold signatures.
  • The test replaces that feature with future_weather.
  • The final assertion requires validate_level5_contract() to reject the changed schema.
  • Save tests/test_neural_networks.py.
  • Run both focused tests inside the container:
docker run --rm --platform linux/amd64 --mount type=bind,src="$(pwd)",dst=/workspace -w /workspace running-readiness-level5:2.15.0 python -m unittest -v tests/test_neural_networks.py

What does the second passing test add?

The contract accepts its approved schema before rejecting the injected feature. That sequence proves the guard responds to drift instead of failing for unrelated fixture data.

You should see two passing tests. The final status remains OK.

Schema test failing too early?

Check the handoff version plus Level 5 target against the code block. Confirm that the marker contains all four forbidden uses in the same order.

Trace the rejected field with help.

Prove determinism and checkpoint reloads

Reproducibility requires more than recording seed 42. The same-seed test compares every early-stopping record plus every tensor in the resulting state dictionaries.

  • Add the same-seed reproducibility test below the schema test:
    def test_same_seed_reproduces_history_and_state(self):
        first, history_a, epoch_a = train_with_early_stopping(
            self.x[:6], self.y[:6], self.x[6:], self.y[6:], self.config, 42, 30, 5)
        second, history_b, epoch_b = train_with_early_stopping(
            self.x[:6], self.y[:6], self.x[6:], self.y[6:], self.config, 42, 30, 5)
        self.assertEqual(epoch_a, epoch_b)
        self.assertEqual(history_a, history_b)
        for key in first.state_dict():
            self.assertTrue(torch.equal(first.state_dict()[key], second.state_dict()[key]))

What does equality cover?

  • The two training calls use the same data plus configuration.
  • The epoch assertion checks that early stopping makes the same choice.
  • The history assertion compares every recorded training loss plus validation loss.
  • The tensor assertion checks every learned parameter in both state dictionaries.
  • Save tests/test_neural_networks.py.
  • Run the expanded test module inside the container:
docker run --rm --platform linux/amd64 --mount type=bind,src="$(pwd)",dst=/workspace -w /workspace running-readiness-level5:2.15.0 python -m unittest -v tests/test_neural_networks.py

What does this run establish?

The third test trains the compact network twice under identical conditions. Matching histories plus model tensors demonstrate same-environment reproducibility.

You should see test_same_seed_reproduces_history_and_state pass. The suite now contains three passing tests.

Same seed producing different state?

Confirm that both training calls use seed 42. Check that the compact configuration still comes from CONFIGS[0].

Inspect the first mismatch with help.

  • Add the checkpoint reload test below the reproducibility test:
    def test_checkpoint_reload_preserves_predictions(self):
        model, _ = train_fixed(self.x, self.y, self.config, 42, 10)
        with tempfile.TemporaryDirectory() as directory:
            path = Path(directory) / "model.pt"
            torch.save({"model_state": model.state_dict(), "input_dim": 2,
                        "config": self.config}, path)
            saved = torch.load(path, map_location="cpu", weights_only=True)
            reloaded = BinaryMLP(saved["input_dim"], saved["config"]["hidden_units"], saved["config"]["dropout"])
            reloaded.load_state_dict(saved["model_state"])
            model.eval()
            reloaded.eval()
            inputs = torch.tensor(self.x)
            with torch.no_grad():
                self.assertTrue(torch.equal(model(inputs), reloaded(inputs)))


if __name__ == "__main__":
    unittest.main()

How does the reload test work?

  • The test saves a temporary checkpoint outside the project artifacts.
  • The reload uses weights_only=True to restrict checkpoint loading.
  • The reload uses map_location="cpu" to preserve CPU portability.
  • The final assertion requires the original model plus reloaded model to produce identical tensors.
  • Save tests/test_neural_networks.py.

Before you run the full suite, which four workflow guarantees do you expect the tests to cover?

  • Run the completed test suite inside the audited container:
docker run --rm --platform linux/amd64 --mount type=bind,src="$(pwd)",dst=/workspace -w /workspace running-readiness-level5:2.15.0 python -m unittest -v tests/test_neural_networks.py

What does the complete suite prove?

The suite exercises runner boundaries plus schema rejection. It also checks deterministic training plus portable checkpoint behavior.

You should see all four tests pass. The final status is OK.

One of the four tests failing?

Use the verbose output to identify the first failing test. Compare only that test with its matching code block before changing the experiment module.

Get targeted debugging help.

You now have executable proof for the workflow's core safeguards. Each claim can be rerun inside the same audited environment.

✔️ My tests match

Your test file now proves group separation plus schema rejection. It also proves same-seed reproducibility plus CPU checkpoint reload behavior.

ⓧ I'd like to double check the full code

import tempfile
import unittest
from pathlib import Path

import numpy as np
import pandas as pd
import torch

from src.neural_network_experiment import (
    BinaryMLP,
    CONFIGS,
    inner_group_split,
    train_fixed,
    train_with_early_stopping,
    validate_level5_contract,
)


class NeuralNetworkTests(unittest.TestCase):
    def setUp(self):
        self.x = np.array([
            [-2.0, -1.0], [-1.5, -0.5], [-1.0, -1.5], [-0.5, -2.0],
            [0.5, 2.0], [1.0, 1.5], [1.5, 0.5], [2.0, 1.0],
        ], dtype=np.float32)
        self.y = np.array([0, 0, 0, 0, 1, 1, 1, 1], dtype=np.float32)
        self.config = CONFIGS[0]

    def test_group_separation(self):
        frame = pd.DataFrame(self.x, columns=["a", "b"])
        target = pd.Series(self.y)
        groups = pd.Series(["r1", "r1", "r2", "r2", "r3", "r3", "r4", "r4"])
        fit, stop = inner_group_split(frame, target, groups, 42)
        self.assertFalse(set(groups.iloc[fit]) & set(groups.iloc[stop]))

    def test_schema_validation_rejects_unapproved_feature(self):
        handoff = {"handoff_version": "2.0.0", "next_level": "Level 5: Networks",
                   "selected_candidate_features": ["distance_km"], "selected_variant": "raw"}
        schema = {"candidate_features": ["distance_km"]}
        fold = {"fold_signature": "same"}
        report = {"selected_variant": "raw", "identical_fold_signatures": ["same"] * 4,
                  "consumed_v1_holdout_read": False,
                  "results": {"raw": {"folds": [fold] * 4}, "engineered": {"folds": [fold] * 4}}}
        marker = {"forbidden_uses": ["model selection", "feature selection", "threshold tuning", "model comparison"]}
        validate_level5_contract(handoff, schema, report, marker)
        handoff["selected_candidate_features"] = ["future_weather"]
        with self.assertRaises(ValueError):
            validate_level5_contract(handoff, schema, report, marker)

    def test_same_seed_reproduces_history_and_state(self):
        first, history_a, epoch_a = train_with_early_stopping(
            self.x[:6], self.y[:6], self.x[6:], self.y[6:], self.config, 42, 30, 5)
        second, history_b, epoch_b = train_with_early_stopping(
            self.x[:6], self.y[:6], self.x[6:], self.y[6:], self.config, 42, 30, 5)
        self.assertEqual(epoch_a, epoch_b)
        self.assertEqual(history_a, history_b)
        for key in first.state_dict():
            self.assertTrue(torch.equal(first.state_dict()[key], second.state_dict()[key]))

    def test_checkpoint_reload_preserves_predictions(self):
        model, _ = train_fixed(self.x, self.y, self.config, 42, 10)
        with tempfile.TemporaryDirectory() as directory:
            path = Path(directory) / "model.pt"
            torch.save({"model_state": model.state_dict(), "input_dim": 2,
                        "config": self.config}, path)
            saved = torch.load(path, map_location="cpu", weights_only=True)
            reloaded = BinaryMLP(saved["input_dim"], saved["config"]["hidden_units"], saved["config"]["dropout"])
            reloaded.load_state_dict(saved["model_state"])
            model.eval()
            reloaded.eval()
            inputs = torch.tensor(self.x)
            with torch.no_grad():
                self.assertTrue(torch.equal(model(inputs), reloaded(inputs)))


if __name__ == "__main__":
    unittest.main()
Generate the Level 5 documentation

The documentation command converts the development-only JSON report into a concise Markdown evidence page. It also links that page from the repository README.

Why the evidence boundary matters

Level 5 fits preprocessing inside each relevant training boundary. The inherited v1 process used development-wide mean imputation before splitting.

The historical pipeline therefore cannot be called leakage-free. Precise wording keeps the neural-network evidence defensible.

Before you generate the page, which existing report values do you expect it to summarize?

  • Generate the documentation from the report by running this inside the audited container environment:
python -m src.neural_network_experiment document

What does the document command do?

  • The command reads the selected configuration plus aggregate validation metrics.
  • It writes docs/level-5-networks.md with the architecture plus evidence boundary.
  • It updates README.md at the <!-- LEVEL_5_NETWORKS --> marker.
  • Switch back to VS Code.
  • Select docs/level-5-networks.md in the Explorer sidebar.
  • Confirm the page contains the selected development-only metrics table.
  • Confirm the architecture is recorded as Linear -> ReLU -> Dropout -> Linear.
  • Confirm the evidence boundary records fold-local Level 5 preprocessing.
  • Confirm the evidence boundary records the historical v1 imputation limitation.
  • Select README.md in the Explorer sidebar.
  • Confirm <!-- LEVEL_5_NETWORKS --> appears before the link to docs/level-5-networks.md.

That closes the evidence loop. Your audited workflow now has passing tests plus generated documentation.

Documentation files missing?

Confirm that reports/v3/neural_network_experiment.json still exists. The document command reads that report before creating the evidence page.

Trace the missing output with help.

Secret mission

Make the Generalization Gap Visible

Force a 128-unit MLP to memorize rows from exactly two runners. Compare its validation behavior with a second network that uses dropout, weight decay, and early stopping.

Clean Up Your Resources

Clean Up Your Resources

Your Level 5 work runs locally through Docker, so there are no cloud resources or recurring costs. The source files and documentation remain part of the repository in every option below.

Resources you used:

  • Local Docker image running-readiness-level5:2.15.0.
  • Generated neural-network and Secret Mission reports under reports/v3/.
  • Selected MLP checkpoint, preprocessing metadata, and Level 6 handoff under artifacts/v3/.

Keep everything running

No action needed. Choose this if you plan to continue into Level 6 or rerun the overfitting comparison.

  • Retain the running-readiness-level5:2.15.0 image for reproducible container runs.
  • Keep artifacts/v3/ so Level 6 can use the selected checkpoint and fingerprinted handoff.
  • Keep reports/v3/ so the development-only comparison and generalization-gap evidence remain available.
  • Leave the Level 5 source files and documentation in the repository.

Pause - I'll come back to this later

Pausing closes Docker Desktop while preserving every reproducible artifact for your return.

  • Stop Docker Desktop through its desktop menu.
  • Leave the running-readiness-level5:2.15.0 image installed.
  • Keep artifacts/v3/ in the repository.
  • Keep reports/v3/ in the repository.

The project leaves no container running because every container command used --rm.

Delete - I don't want to use this again

Deleting the generated outputs is permanent. The locked Level 4 evidence remains safe because the commands below target only the Level 5 image and v3 directories.

  • Return to the terminal at the repository from earlier.
  • Remove the Level 5 Docker image by running this command:
docker image rm running-readiness-level5:2.15.0

What Does This Command Remove?

This removes the local Level 5 image with the 2.15.0 tag. Your repository files remain unchanged.

  • Confirm the image is absent from Docker Desktop's local image list.

Image Still Present?

If Docker cannot remove the image, confirm Docker Desktop is running. A stopped Docker engine cannot process the removal request.

If Docker reports that the image is in use, confirm no Level 5 container remains active before retrying the command.

Ask for help with the cleanup error.

  • Remove the generated Level 5 artifacts and reports by running this command:
rm -rf artifacts/v3/ reports/v3/

What Does This Cleanup Remove?

This removes the selected checkpoint, preprocessing metadata, Level 6 handoff, neural-network report, and overfitting report.

The command leaves artifacts/v2/, reports/v2/, Level 5 source files, and documentation untouched.

  • Use your code editor's file explorer to confirm artifacts/v3/ no longer appears.
  • Use your code editor's file explorer to confirm reports/v3/ no longer appears.

Directories Still Present?

Confirm the terminal is positioned at the repository before retrying the cleanup command. Running it from another location targets different relative paths.

Ask for help verifying the cleanup location.

Nice Work!

Nice Work!

Level 5 is complete. You built a leakage-aware PyTorch MLP that produces reproducible development-only evidence on locked runner-safe folds.

What you learned:

  • Built an explicit neural-network training loop that exposes logits, loss, backpropagation, gradients, and optimizer updates.
  • Compared two MLP configurations with grouped cross-validation on the exact inherited folds. Kept imputation, scaling, and early stopping inside the relevant training boundaries.
  • Published a reproducible Level 5 evidence package with passing tests plus reloadable artifacts. Documented the consumed v1 holdout boundary plus the inherited imputation limitation.
  • Secret Mission: Made the generalization gap visible by contrasting a wide memorizing network with dropout, weight decay, and early stopping.

Ready to quiz yourself?