Build an AI Leakage Audit

Expose model leakage with group-aware validation and reproducible evidence.

Introduction

30 Second Summary

A high score can make a model look ready for the real world. If the test quietly rewards recognition of familiar cases, that confidence can disappear as soon as new cases arrive.

In this project, you will build Model Reality Check, a deterministic Python audit that exposes entity leakage. You will replace a misleading row-wise score with group-aware evidence that supports a deployment decision.

What You'll Build

You will run one command to see a deceptively strong score collapse on unseen accounts before a repaired feature set earns a defensible deployment decision.

By the end of this project, you'll have:

  • A one-command model audit that shows the gap between memorizing familiar accounts and generalizing to unseen accounts.
  • A repaired evaluation path that uses grouped cross-validation to test performance on entirely new accounts before issuing a deployment decision.
  • A public GitHub case study that gives reviewers reproducible evidence. Its source includes tests plus JSON metrics plus a Markdown decision report.
  • Secret Mission: Add a GitHub Actions workflow that reruns the audit and unit tests on every push or pull request.

Are there any prerequisites?

No prior model-evaluation experience is required. The guide includes setup for every required tool and account.

Before We Start

A strong model score is only useful when the evaluation behind it deserves trust. Before any hands-on work begins, define the audit you are building and the judgment it will prove.

Set Up the Audit Workspace

A leakage audit only earns trust when another person can recreate it from the same tools. Hidden machine differences can turn a model result into evidence nobody else can check.

A public GitHub destination gives the case study somewhere visible to land. An isolated Python environment prevents packages elsewhere on your Mac from silently shaping the result.

In this step, get ready to:
  • Confirm your GitHub account plus Visual Studio Code command access.
  • Verify Python 3.13.17 plus Git in macOS Terminal.
  • Create a virtual environment with scikit-learn 1.9.1 installed.
Confirm your publishing tools

GitHub becomes the public destination for your finished case study. Your account must be ready before you publish the audit in a later step.

GitHub handles your password directly. This project never asks you to paste that credential into Terminal or Visual Studio Code.

Choose the GitHub account state that matches your screen:

✔️ I already have an account

  • Visit the GitHub home page.
  • Sign in with your existing account.
  • Confirm your profile image appears in the page header.

Your GitHub account is ready to receive the public repository later in the project.

ⓧ I need an account

  • Visit the GitHub account creation page.
  • Follow the on-screen prompts to create your account.
  • Complete any verification GitHub requests.
  • Return to the GitHub home page.
  • Confirm your profile image appears in the page header.

Your new account is ready for the case study you publish later.

Visual Studio Code gives you one place to edit the audit files. Its terminal command also lets you open the correct folder without hunting through Finder.

  • Press Cmd+Space to open Spotlight.
  • Type Terminal into Spotlight.
  • Press Enter to open Terminal.
  • Check whether the Visual Studio Code terminal command is available by running:
code --version

What does this check prove?

A version number proves that Terminal can find the code command. You use that command to open the audit folder directly in Visual Studio Code.

✔️ I see a version number

Your Visual Studio Code installation can be launched from Terminal. Keep this Terminal window open for the next checks.

ⓧ Visual Studio Code opens, but the command is missing

Finding this command is a little fiddly because it lives inside the Command Palette. Visual Studio Code installs it into your shell path from there.

  • Press Cmd+Space to open Spotlight.
  • Type Visual Studio Code into Spotlight.
  • Press Enter to open the app.
  • Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows) to open the Command Palette.
  • Enter Shell Command: Install 'code' and optionally 'copilot' commands in PATH.
  • Select the matching command.
  • Choose No for the optional Copilot CLI.
  • Quit Terminal.
  • Press Cmd+Space to open Spotlight again.
  • Type Terminal into Spotlight.
  • Press Enter to open a fresh Terminal window.
  • Repeat the Visual Studio Code version check above.

You should now see a Visual Studio Code version number.

ⓧ Visual Studio Code is not installed

  • Visit the official Visual Studio Code macOS setup guide.
  • Download the official macOS build.
  • Open the downloaded .dmg file.
  • Drag Visual Studio Code.app into the Applications folder.
  • Press Cmd+Space to open Spotlight.
  • Type Visual Studio Code into Spotlight.
  • Press Enter to open the app.
  • Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows) to open the Command Palette.
  • Enter Shell Command: Install 'code' and optionally 'copilot' commands in PATH.
  • Select the matching command.
  • Choose No for the optional Copilot CLI.
  • Quit Terminal.
  • Open a fresh Terminal window through Spotlight.
  • Repeat the Visual Studio Code version check above.

You should now see a Visual Studio Code version number.

Verify Python and Git

The audit depends on one exact Python release. Pinning the interpreter now makes later model scores easier to reproduce.

  • Return to the Terminal window from the Visual Studio Code check.
  • Check your available Python version by running:
python3 --version

What does this check prove?

This command reports which Python installation the python3 command currently reaches. The project needs the output to identify Python 3.13.17.

✔️ I see Python 3.13.17

Your Mac has the exact Python release used by this project. Leave Terminal open for the Git check.

ⓧ I see another version

macOS may include an older Python for Apple development tools. The official installer keeps that system copy intact while adding Python 3.13.17 for your projects.

  • Visit the official Python releases for macOS page.
  • Find Python 3.13.17 in the stable releases list.
  • Download its macOS installer.
  • Open the downloaded installer package.
  • Complete the installer with its default options.
  • Quit Terminal after the installer finishes.
  • Open a fresh Terminal window through Spotlight.
  • Repeat the Python version check above.

You should now see Python 3.13.17 in Terminal.

ⓧ Command not found

The official Python installer adds the interpreter plus its terminal links. The default options also update your shell path.

  • Visit the official Python releases for macOS page.
  • Find Python 3.13.17 in the stable releases list.
  • Download its macOS installer.
  • Open the downloaded installer package.
  • Complete the installer with its default options.
  • Quit Terminal after the installer finishes.
  • Open a fresh Terminal window through Spotlight.
  • Repeat the Python version check above.

You should now see Python 3.13.17 in Terminal.

Git records each meaningful change to the audit. The final repository uses that history to show how the evaluation developed.

  • Check whether Git is available by running:
git --version

What does this check prove?

A version number confirms that Terminal can reach Git. You use it later to record the audit files before publishing them.

✔️ I see a Git version

Git is ready for the project. Keep this Terminal window open.

ⓧ Command not found

Apple supplies Git through its Xcode Command Line Tools installer. The installer opens a separate macOS window after you run the command below.

The download can take a few minutes. A quiet progress window during that time is expected.

  • Start the Apple installer by running:
xcode-select --install

What does this command do?

This command asks macOS to install the Xcode Command Line Tools. That package includes Git.

  • Complete the Apple installer.
  • Repeat the Git version check above.

You should now see a Git version in Terminal.

Git still unavailable?

Quit Terminal after the Apple installer completes. Open a fresh Terminal window through Spotlight before repeating the check.

If the command still fails, help me diagnose why Git is unavailable after installing the Xcode Command Line Tools.

Create the isolated workspace

The project folder keeps the audit source plus its evidence together. The virtual environment keeps the project dependency separate from packages used by other Python projects.

  • Move to your Desktop before creating the project folder by running these commands:
cd ~/Desktop
mkdir model-reality-check
cd model-reality-check

What do these commands do?

  • The first command moves Terminal to your Desktop.
  • The second command creates the model-reality-check folder.
  • The final command moves Terminal into that folder.
  • Switch to Finder.
  • Confirm the model-reality-check folder appears on your Desktop.

Folder missing from your Desktop?

Check that Terminal accepted each command without reporting a folder permission problem. Confirm that you typed the folder name as model-reality-check.

Need a second pair of eyes? Help me find the model-reality-check folder created from Terminal.

  • Return to Terminal.
  • Open the current project folder in Visual Studio Code by running:
code .

What does this command do?

The period represents the current folder. Visual Studio Code opens model-reality-check as the active workspace.

  • Approve workspace access if Visual Studio Code asks whether you trust the folder you created.
  • Confirm model-reality-check appears at the top of the Explorer sidebar.

Your project now has a fixed home on the Desktop. The next command creates its isolated Python environment inside that folder.

  • Return to the Terminal window that is already inside model-reality-check.
  • Create the virtual environment by running these commands:
python3 -m venv .venv
source .venv/bin/activate
python --version

What do these commands do?

  • The first command creates the isolated environment inside .venv.
  • The second command activates that environment for the current Terminal session.
  • The final command confirms which Python release the activated environment uses.

You should see Python 3.13.17. Your Terminal prompt should also show (.venv) near the folder name.

Virtual environment not active?

Confirm that Terminal is inside the model-reality-check folder. Check that the activation command uses .venv/bin/activate.

If the Python version is unexpected, help me diagnose my macOS virtual environment.

A requirements file records the exact library release used by the audit. Anyone reproducing the project can install the same dependency from this file.

  • Create requirements.txt inside model-reality-check by running:
printf "scikit-learn==1.9.1\n" > requirements.txt

What does this command do?

This command writes one pinned dependency into requirements.txt.

The double equals sign requires scikit-learn 1.9.1 whenever the environment is rebuilt.

  • Switch back to Visual Studio Code.
  • Select requirements.txt in the Explorer sidebar.
  • Confirm the file contains exactly one dependency line.

✔️ Awesome, I've got everything!

Your dependency pin is correct. Save requirements.txt if Visual Studio Code shows an unsaved change.

ⓧ I'd like to double check the full code

scikit-learn==1.9.1

Installing from requirements.txt turns the recorded dependency into the package available inside your virtual environment.

The first installation can take a few minutes while Python downloads the package files. Terminal output during that wait means the installation is still progressing.

  • Return to the activated Terminal session.
  • Install the pinned dependency by running:
python -m pip install -r requirements.txt

What does this command do?

The activated Python interpreter runs pip against requirements.txt.

This keeps scikit-learn 1.9.1 inside the project environment.

The installation finishes when Terminal returns to the prompt without reporting a failure.

Dependency installation failed?

Confirm that (.venv) appears in the Terminal prompt. Confirm that requirements.txt contains the exact pinned dependency shown above.

A network interruption can also stop the package download. Restore the connection before repeating the install command.

Still blocked? Help me troubleshoot the scikit-learn installation in my virtual environment.

Before you run these checks, predict the four version results you expect to see:

  • Verify the complete workspace from the activated environment by running:
python --version
git --version
code --version
python -c "import sklearn; print(sklearn.__version__)"

What should you see?

  • The Python check reports Python 3.13.17.
  • The Git check reports an installed Git version.
  • The Visual Studio Code check reports an installed version.
  • The scikit-learn check reports 1.9.1.

That is the setup locked down. Every later score now comes from the pinned project environment.

One version check failed?

Confirm that your Terminal prompt still shows (.venv). Repeat the matching setup path above for the tool whose version is missing.

For a mismatched library version, inspect requirements.txt before repeating the dependency installation.

Need help reading the outputs? Help me diagnose my final audit workspace checks.

Your audit workspace is reproducible. Next, you will build the classifier whose first score looks convincing enough to fool a careless review.

Build the Misleading Model

Your isolated Python workspace is ready for its first experiment. You can now create the model evaluation that starts this audit.

A validation score can look excellent when the experiment tests the wrong claim. This step creates that trap with repeated observations from the same accounts.

In this step, get ready to:
  • Create a deterministic dataset with repeated account observations.
  • Evaluate both features with row-wise cross-validation.
  • Run the audit to investigate its first score.
Generate repeated account observations

Each account contributes five rows. Every row carries an account-like behavior_signature plus a transferable stable_signal.

  • Create audit.py inside model-reality-check by pasting this code into a new file:
from sklearn.model_selection import KFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier

N_GROUPS = 100
OBSERVATIONS_PER_GROUP = 5


def build_dataset():
    """Create repeated account observations with one shortcut and one stable feature."""
    x_leaky = []
    targets = []

    for observation in range(OBSERVATIONS_PER_GROUP):
        observation_offset = observation / 1000

        for group_id in range(N_GROUPS):
            pair_id = group_id // 2
            target = group_id % 2

            behavior_signature = pair_id * 10 + target + observation_offset
            stable_signal = target + observation_offset

            x_leaky.append([behavior_signature, stable_signal])
            targets.append(target)

    return x_leaky, targets

What does this code build?

  • The constants define 100 accounts with five observations per account.
  • The outer loop keeps rows ordered by observation number.
  • The inner loop creates one row for every account during each observation round.
  • The target alternates between 0 and 1 across the accounts.
  • The returned feature matrix contains behavior_signature plus stable_signal for every row.
  • Save audit.py.

The file sidebar now lists audit.py inside model-reality-check. Its dataset function contains both feature values for each account observation.

Can't find the dataset code?

  • Save audit.py directly inside the model-reality-check directory.
  • Check that build_dataset() sits below the imports plus the two constants.

Still stuck? Help me check the dataset section in audit.py.

Evaluate rows with consecutive folds

The scikit-learn KFold splitter creates consecutive folds because shuffling is disabled. A one-neighbor k-nearest neighbors classifier predicts each validation row from the closest training row.

  • Add the evaluator below return x_leaky, targets by pasting this code:
def evaluate_model(x_leaky, targets):
    model = KNeighborsClassifier(n_neighbors=1)
    row_wise = KFold(n_splits=OBSERVATIONS_PER_GROUP, shuffle=False)
    scores = cross_val_score(model, x_leaky, targets, cv=row_wise)
    return scores.mean()

What does this evaluator do?

  • The classifier uses one neighbor so the closest training row determines each prediction.
  • The splitter creates five consecutive folds to match the five observation rounds.
  • The cross_val_score() call trains the classifier once per fold.
  • The function returns the mean accuracy across all five folds.
  • Save audit.py.

You can now see evaluate_model() directly below build_dataset(). The file contains a complete row-wise evaluation path.

Does the evaluator look incomplete?

  • Check that KNeighborsClassifier is imported at the top of audit.py.
  • Check that cross_val_score appears beside KFold in the model-selection import.

Need another pair of eyes? Help me check the evaluate_model function.

Run the first evaluation

The entry point connects the generated dataset to the evaluator. It also formats the resulting mean accuracy to three decimal places.

  • Add the runnable entry point below evaluate_model() by pasting this code:
def main():
    x_leaky, targets = build_dataset()
    score = evaluate_model(x_leaky, targets)
    print("Row-wise score with shortcut:", f"{score:.3f}")


if __name__ == "__main__":
    main()

What does the entry point do?

  • The main() function builds all 500 observations.
  • The evaluator receives the two-feature rows plus their binary targets.
  • The final line runs main() when you execute audit.py directly.
  • Save audit.py.

Before you run the audit, how convincing do you expect its first score to look?

  • Run the row-wise evaluation in the activated terminal with this command:
python audit.py

What should you see?

You'll see Row-wise score with shortcut: 1.000 in the terminal.

Each 100-row fold contains one observation from every account. The training rows still contain four observations from each of those accounts.

This creates entity leakage. The score only measures predictions for accounts represented during training.

That impressive 1.000 is the designed obstacle. You have reproduced false confidence with a deterministic experiment.

Didn't get the row-wise score?

  • Confirm the terminal prompt still shows the activated .venv environment.
  • Check that main() calls build_dataset() before evaluate_model().
  • Check that the final guard calls main().

Still seeing a different result? Help me debug the first audit run.

✔️ Awesome, I've got everything!

Your saved file now contains every piece of the row-wise audit.

ⓧ I'd like to double check the full code

from sklearn.model_selection import KFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier

N_GROUPS = 100
OBSERVATIONS_PER_GROUP = 5


def build_dataset():
    """Create repeated account observations with one shortcut and one stable feature."""
    x_leaky = []
    targets = []

    for observation in range(OBSERVATIONS_PER_GROUP):
        observation_offset = observation / 1000

        for group_id in range(N_GROUPS):
            pair_id = group_id // 2
            target = group_id % 2

            behavior_signature = pair_id * 10 + target + observation_offset
            stable_signal = target + observation_offset

            x_leaky.append([behavior_signature, stable_signal])
            targets.append(target)

    return x_leaky, targets


def evaluate_model(x_leaky, targets):
    model = KNeighborsClassifier(n_neighbors=1)
    row_wise = KFold(n_splits=OBSERVATIONS_PER_GROUP, shuffle=False)
    scores = cross_val_score(model, x_leaky, targets, cv=row_wise)
    return scores.mean()


def main():
    x_leaky, targets = build_dataset()
    score = evaluate_model(x_leaky, targets)
    print("Row-wise score with shortcut:", f"{score:.3f}")


if __name__ == "__main__":
    main()

You have caught a model looking brilliant under the wrong test. Next, you'll make it face accounts it has never seen.

Test Unseen Accounts

The row-wise score you produced looks convincing. It measures new rows from accounts the model already encountered.

Your audit now changes the validation unit from rows to accounts. Group-wise validation keeps each account intact. This exposes entity leakage hiding behind the strong score.

In this step, get ready to:
  • Track the account group attached to every observation.
  • Compare row-wise validation with account-level validation.
  • Expose the shortcut model's failure on unseen accounts.
Track account identity

A group-aware splitter needs an account label for every observation. The dataset also keeps the safe feature separate so the next evaluation can distinguish the shortcut from the transferable signal.

  • In audit.py, replace the imports and constants at the top of the file with this block:
import json
from pathlib import Path
from statistics import mean

from sklearn.model_selection import GroupKFold, KFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier

N_GROUPS = 100
OBSERVATIONS_PER_GROUP = 5
PROJECT_GATE = 0.80
ROOT = Path(__file__).resolve().parent

What do these foundations add?

  • GroupKFold supplies the account-aware splitting strategy.
  • mean supports consistent score summaries across folds.
  • PROJECT_GATE centralizes the audit's approval threshold.
  • ROOT identifies the folder containing audit.py.
  • Add the score summarizer below build_dataset() by copying this function:
def summarize(scores):
    values = [float(score) for score in scores]
    return {
        "folds": len(values),
        "mean_accuracy": round(mean(values), 3),
        "minimum_accuracy": round(min(values), 3),
        "maximum_accuracy": round(max(values), 3),
    }

What does this function do?

  • values converts the fold results into standard Python numbers.
  • mean_accuracy captures the overall result across every fold.
  • minimum_accuracy records the weakest fold.
  • maximum_accuracy records the strongest fold.
  • Save audit.py.
  • Confirm the existing row-wise audit still runs by using this command:
python audit.py

What should you see?

You should still see the strong row-wise shortcut score. This confirms the new audit foundations have not changed the original experiment.

Does the audit stop before printing?

Check that each import appears once at the top of audit.py. Make sure summarize() sits outside every other function.

Still stuck? Help me diagnose why audit.py stopped running after I added GroupKFold and summarize().

  • Replace the existing build_dataset() function with this account-aware version:
def build_dataset():
    """Create repeated account observations with one shortcut and one stable feature."""
    x_leaky = []
    x_safe = []
    targets = []
    groups = []

    for observation in range(OBSERVATIONS_PER_GROUP):
        observation_offset = observation / 1000

        for group_id in range(N_GROUPS):
            pair_id = group_id // 2
            target = group_id % 2

            behavior_signature = pair_id * 10 + target + observation_offset
            stable_signal = target + observation_offset

            x_leaky.append([behavior_signature, stable_signal])
            x_safe.append([stable_signal])
            targets.append(target)
            groups.append(group_id)

    return x_leaky, x_safe, targets, groups

How does the dataset change?

  • x_leaky retains both the account-like signature and the stable signal.
  • x_safe isolates the stable signal for the repaired evaluation path.
  • groups records the account attached to every row.
  • build_dataset() returns both feature views with the targets and account groups.
Add group-wise validation

GroupKFold assigns every row from one account to the same validation fold. Using N_GROUPS folds turns this audit into a leave-one-account-out stress test.

  • Replace the existing evaluate_model() function with this comparison:
def evaluate_model(x_leaky, x_safe, targets, groups):
    model = KNeighborsClassifier(n_neighbors=1)
    row_wise = KFold(n_splits=OBSERVATIONS_PER_GROUP, shuffle=False)
    group_wise = GroupKFold(n_splits=N_GROUPS)

    row_scores = cross_val_score(model, x_leaky, targets, cv=row_wise)
    grouped_leaky_scores = cross_val_score(
        model,
        x_leaky,
        targets,
        groups=groups,
        cv=group_wise,
    )

    return {
        "row_wise_leaky": summarize(row_scores),
        "group_wise_leaky": summarize(grouped_leaky_scores),
    }

How does this test unseen accounts?

  • row_wise preserves the original comparison where accounts cross fold boundaries.
  • group_wise creates one validation fold for each account group.
  • groups=groups tells the splitter which rows belong to the same account.
  • grouped_leaky_scores measures whether the shortcut feature transfers to entirely unseen accounts.
  • Replace the existing main() function with this two-score output:
def main():
    x_leaky, x_safe, targets, groups = build_dataset()
    metrics = evaluate_model(x_leaky, x_safe, targets, groups)
    print(
        "Row-wise score with shortcut:",
        f"{metrics['row_wise_leaky']['mean_accuracy']:.3f}",
    )
    print(
        "Group-wise score with shortcut:",
        f"{metrics['group_wise_leaky']['mean_accuracy']:.3f}",
    )

What changes in the output?

  • main() passes both feature views with the targets and account groups into the evaluator.
  • metrics keeps each validation result under a descriptive key.
  • The two printed scores place the original claim beside the unseen-account result.
  • Save audit.py.
Compare the two claims

Before you run the audit, which score do you expect to change when every validation account is absent from training?

  • Test both validation strategies by running this command:
python audit.py

What should you see?

  • The row-wise shortcut score remains deceptively strong because account observations cross fold boundaries.
  • The group-wise shortcut score drops dramatically because each validation account is absent from training.
  • The gap proves that behavior_signature acts like an account identifier instead of a pattern that transfers to unseen accounts.

You caught the misleading claim. Your audit now separates account recognition from genuine generalization.

Do the scores look too similar?

Check that groups.append(group_id) runs inside the inner account loop. Confirm that the grouped evaluation passes groups=groups into cross_val_score().

Need another pair of eyes? Help me find why my group-wise shortcut score is not dramatically lower than my row-wise score.

✔️ Awesome, I've got everything!

Your audit now tests the shortcut model against entirely unseen accounts.

ⓧ I'd like to double check the full code

import json
from pathlib import Path
from statistics import mean

from sklearn.model_selection import GroupKFold, KFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier

N_GROUPS = 100
OBSERVATIONS_PER_GROUP = 5
PROJECT_GATE = 0.80
ROOT = Path(__file__).resolve().parent


def build_dataset():
    """Create repeated account observations with one shortcut and one stable feature."""
    x_leaky = []
    x_safe = []
    targets = []
    groups = []

    for observation in range(OBSERVATIONS_PER_GROUP):
        observation_offset = observation / 1000

        for group_id in range(N_GROUPS):
            pair_id = group_id // 2
            target = group_id % 2

            behavior_signature = pair_id * 10 + target + observation_offset
            stable_signal = target + observation_offset

            x_leaky.append([behavior_signature, stable_signal])
            x_safe.append([stable_signal])
            targets.append(target)
            groups.append(group_id)

    return x_leaky, x_safe, targets, groups


def summarize(scores):
    values = [float(score) for score in scores]
    return {
        "folds": len(values),
        "mean_accuracy": round(mean(values), 3),
        "minimum_accuracy": round(min(values), 3),
        "maximum_accuracy": round(max(values), 3),
    }


def evaluate_model(x_leaky, x_safe, targets, groups):
    model = KNeighborsClassifier(n_neighbors=1)
    row_wise = KFold(n_splits=OBSERVATIONS_PER_GROUP, shuffle=False)
    group_wise = GroupKFold(n_splits=N_GROUPS)

    row_scores = cross_val_score(model, x_leaky, targets, cv=row_wise)
    grouped_leaky_scores = cross_val_score(
        model,
        x_leaky,
        targets,
        groups=groups,
        cv=group_wise,
    )

    return {
        "row_wise_leaky": summarize(row_scores),
        "group_wise_leaky": summarize(grouped_leaky_scores),
    }


def main():
    x_leaky, x_safe, targets, groups = build_dataset()
    metrics = evaluate_model(x_leaky, x_safe, targets, groups)
    print(
        "Row-wise score with shortcut:",
        f"{metrics['row_wise_leaky']['mean_accuracy']:.3f}",
    )
    print(
        "Group-wise score with shortcut:",
        f"{metrics['group_wise_leaky']['mean_accuracy']:.3f}",
    )


if __name__ == "__main__":
    main()

What should match?

This reference contains the complete account-aware audit for this step. The final output includes one row-wise score and one group-wise score.

Your honest evaluation has exposed the shortcut. Next, you will remove that shortcut and turn the repaired result into inspectable evidence.

Remove the Shortcut and Generate Evidence

Your group-aware evaluation exposed the shortcut. The impressive row-wise score collapsed when the model faced unseen accounts.

Finding leakage only identifies the unsafe evidence. A defensible deployment decision needs a repaired evaluation plus inspectable JSON metrics plus a Markdown report that states the result's limits.

In this step, get ready to:
  • Evaluate the stable feature on entirely unseen accounts.
  • Apply a deployment gate that blocks the shortcut feature.
  • Generate machine-readable metrics plus a model-audit report.
Evaluate the stable signal

The repaired path uses only stable_signal. The same group-wise splitter tests whether this feature transfers to accounts that never appear in training.

  • In audit.py, locate the existing return dictionary inside evaluate_model().
  • Delete the existing return dictionary.
  • Paste this code directly after the grouped_leaky_scores evaluation:
    grouped_safe_scores = cross_val_score(
        model,
        x_safe,
        targets,
        groups=groups,
        cv=group_wise,
    )

    metrics = {
        "dataset": {
            "groups": N_GROUPS,
            "observations_per_group": OBSERVATIONS_PER_GROUP,
            "samples": len(targets),
        },
        "row_wise_leaky": summarize(row_scores),
        "group_wise_leaky": summarize(grouped_leaky_scores),
        "group_wise_safe": summarize(grouped_safe_scores),
    }

    safe_score = metrics["group_wise_safe"]["mean_accuracy"]
    metrics["decision"] = {
        "project_gate": PROJECT_GATE,
        "status": "PASS" if safe_score >= PROJECT_GATE else "BLOCK",
        "approved_feature_set": ["stable_signal"] if safe_score >= PROJECT_GATE else [],
        "blocked_feature": "behavior_signature",
    }
    return metrics

What does this code do?

  • The grouped_safe_scores evaluation tests x_safe with account isolation preserved.
  • The metrics dictionary records the dataset size plus summaries for all three evaluation paths.
  • The decision dictionary approves stable_signal when its mean accuracy reaches PROJECT_GATE.
  • The blocked feature remains behavior_signature because its account-like pattern failed the unseen-account test.
  • Save audit.py.
  • Check that the extended evaluation still runs by using this command:
python audit.py

What does this check prove?

The command now computes the safe group-wise evaluation before returning the metrics. A successful run proves the expanded metrics structure still supplies the two score summaries used by the current output code.

You should still see the row-wise shortcut score plus the group-wise shortcut score. The script should finish without a traceback.

Seeing an error after the evaluation change?

Check that grouped_safe_scores sits after the closing parenthesis for grouped_leaky_scores.

Confirm that the old return dictionary was replaced. Leaving both dictionaries in place makes the new decision code unreachable.

Still stuck? Help me debug the repaired evaluate_model function.

Render the audit report

Raw scores show what happened. The report connects those scores to the leakage finding plus the deployment decision plus the experiment's limitation.

  • In audit.py, add the start of render_report() below evaluate_model():
def render_report(metrics):
    row_score = metrics["row_wise_leaky"]["mean_accuracy"]
    grouped_leaky_score = metrics["group_wise_leaky"]["mean_accuracy"]
    grouped_safe_score = metrics["group_wise_safe"]["mean_accuracy"]
    decision = metrics["decision"]

Why cache these values?

These local variables give each evaluation result a clear role in the narrative. The report can compare misleading row-wise confidence with honest unseen-account performance without repeating long dictionary lookups.

  • Inside render_report(), paste this report builder below the decision assignment:
    return "\n".join(
        [
            "# Model Evaluation Audit",
            "",
            "## Executive finding",
            "",
            f"The row-wise evaluation reports {row_score:.3f} accuracy while repeated observations from the same accounts appear across training and validation. Keeping accounts separate drops the shortcut model to {grouped_leaky_score:.3f}. Removing `behavior_signature` and evaluating only `stable_signal` produces {grouped_safe_score:.3f} on unseen accounts.",
            "",
            "## Evidence",
            "",
            "| Evaluation | Mean accuracy | What it tests |",
            "| --- | ---: | --- |",
            f"| Row-wise with shortcut | {row_score:.3f} | New rows from accounts represented in training |",
            f"| Group-wise with shortcut | {grouped_leaky_score:.3f} | Entirely unseen accounts |",
            f"| Group-wise without shortcut | {grouped_safe_score:.3f} | Unseen accounts using the stable feature |",
            "",
            "## Decision",
            "",
            f"{decision['status']}: use `{decision['approved_feature_set'][0]}` and block `{decision['blocked_feature']}` from this model path.",
            "",
            "## Limitation",
            "",
            "This is a deterministic synthetic stress test. It demonstrates an evaluation failure mode and its repair, not production performance on a real population.",
            "",
        ]
    )

How is the report structured?

  • The executive finding explains why the first score was misleading.
  • The evidence table compares the questions answered by all three evaluation paths.
  • The decision names the approved feature set plus the blocked shortcut.
  • The limitation prevents the synthetic stress test from being presented as production evidence.
  • Save audit.py.
  • Confirm that the report function has valid Python syntax by running the audit again:
python audit.py

What does this syntax check prove?

Python parses every function before it starts main(). Seeing the established score lines confirms the new report builder contains valid Python syntax.

You should see the two established score lines again. The command should return to the terminal prompt without a traceback.

Does the report function fail to parse?

Check that every report line remains inside the list passed to "\n".join().

Match the indentation of the closing square bracket plus the closing parenthesis to the reference code.

Need another pair of eyes? Help me find the syntax problem in render_report.

Write and inspect the evidence

The final path turns each run into reproducible evidence. It writes compact metrics for tools plus a readable report for reviewers.

  • In audit.py, add write_artifacts() below render_report():
def write_artifacts(metrics):
    metrics_path = ROOT / "artifacts" / "metrics.json"
    report_path = ROOT / "reports" / "model_audit.md"
    metrics_path.parent.mkdir(exist_ok=True)
    report_path.parent.mkdir(exist_ok=True)

    metrics_path.write_text(
        json.dumps(metrics, indent=2) + "\n",
        encoding="utf-8",
    )
    report_path.write_text(render_report(metrics), encoding="utf-8")
    return metrics_path, report_path

What does this code do?

  • The path objects keep both generated files inside the model-reality-check workspace.
  • The directory calls create artifacts plus reports when they do not already exist.
  • The JSON writer preserves the complete metrics dictionary with readable indentation.
  • The report writer saves the decision narrative produced by render_report().
  • In audit.py, locate the existing main() function.
  • Replace main() plus the launch condition beneath it with this completed version:
def main():
    x_leaky, x_safe, targets, groups = build_dataset()
    metrics = evaluate_model(x_leaky, x_safe, targets, groups)
    metrics_path, report_path = write_artifacts(metrics)

    print(
        "Row-wise score with shortcut:",
        f"{metrics['row_wise_leaky']['mean_accuracy']:.3f}",
    )
    print(
        "Group-wise score with shortcut:",
        f"{metrics['group_wise_leaky']['mean_accuracy']:.3f}",
    )
    print(
        "Group-wise score without shortcut:",
        f"{metrics['group_wise_safe']['mean_accuracy']:.3f}",
    )
    print("Deployment decision:", metrics["decision"]["status"])
    print("Wrote", metrics_path.relative_to(ROOT))
    print("Wrote", report_path.relative_to(ROOT))


if __name__ == "__main__":
    main()

How does the completed run work?

The completed main() runs every evaluation before writing both evidence files. Its terminal output exposes the three scores plus the deployment decision plus the generated paths.

  • Save audit.py.

Before you run this, consider whether the stable feature reaches the project gate after the shortcut is removed.

  • Run the completed audit by using this command:
python audit.py

What does the completed command verify?

This run executes the misleading baseline plus the honest shortcut test plus the repaired evaluation. It also rebuilds both evidence files from the same metrics used for the deployment decision.

You should see all three score labels. You should also see Deployment decision: PASS plus confirmation that artifacts/metrics.json and reports/model_audit.md were written.

That is the repair proven. The audit now blocks the account-like shortcut while approving the feature that transfers to unseen accounts.

  • In the VS Code file list, select artifacts/metrics.json.
  • Confirm that the decision status is PASS.
  • Confirm that behavior_signature is the blocked feature.
  • In the VS Code file list, select reports/model_audit.md.
  • Confirm that the report compares all three evaluations.
  • Confirm that the limitation describes the result as a deterministic synthetic stress test.

Missing the evidence files or PASS decision?

Check that main() calls write_artifacts(metrics) before printing the generated paths.

Confirm that grouped_safe_scores evaluates x_safe with groups=groups.

Still missing the expected result? Help me trace the final audit output.

✔️ Awesome, I've got everything!

Your saved audit.py now matches the complete leakage audit. Both generated evidence files come from the same deterministic run.

ⓧ I'd like to double check the full code

import json
from pathlib import Path
from statistics import mean

from sklearn.model_selection import GroupKFold, KFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier

N_GROUPS = 100
OBSERVATIONS_PER_GROUP = 5
PROJECT_GATE = 0.80
ROOT = Path(__file__).resolve().parent


def build_dataset():
    """Create repeated account observations with one shortcut and one stable feature."""
    x_leaky = []
    x_safe = []
    targets = []
    groups = []

    for observation in range(OBSERVATIONS_PER_GROUP):
        observation_offset = observation / 1000

        for group_id in range(N_GROUPS):
            pair_id = group_id // 2
            target = group_id % 2

            behavior_signature = pair_id * 10 + target + observation_offset
            stable_signal = target + observation_offset

            x_leaky.append([behavior_signature, stable_signal])
            x_safe.append([stable_signal])
            targets.append(target)
            groups.append(group_id)

    return x_leaky, x_safe, targets, groups


def summarize(scores):
    values = [float(score) for score in scores]
    return {
        "folds": len(values),
        "mean_accuracy": round(mean(values), 3),
        "minimum_accuracy": round(min(values), 3),
        "maximum_accuracy": round(max(values), 3),
    }


def evaluate_model(x_leaky, x_safe, targets, groups):
    model = KNeighborsClassifier(n_neighbors=1)
    row_wise = KFold(n_splits=OBSERVATIONS_PER_GROUP, shuffle=False)
    group_wise = GroupKFold(n_splits=N_GROUPS)

    row_scores = cross_val_score(model, x_leaky, targets, cv=row_wise)
    grouped_leaky_scores = cross_val_score(
        model,
        x_leaky,
        targets,
        groups=groups,
        cv=group_wise,
    )
    grouped_safe_scores = cross_val_score(
        model,
        x_safe,
        targets,
        groups=groups,
        cv=group_wise,
    )

    metrics = {
        "dataset": {
            "groups": N_GROUPS,
            "observations_per_group": OBSERVATIONS_PER_GROUP,
            "samples": len(targets),
        },
        "row_wise_leaky": summarize(row_scores),
        "group_wise_leaky": summarize(grouped_leaky_scores),
        "group_wise_safe": summarize(grouped_safe_scores),
    }

    safe_score = metrics["group_wise_safe"]["mean_accuracy"]
    metrics["decision"] = {
        "project_gate": PROJECT_GATE,
        "status": "PASS" if safe_score >= PROJECT_GATE else "BLOCK",
        "approved_feature_set": ["stable_signal"] if safe_score >= PROJECT_GATE else [],
        "blocked_feature": "behavior_signature",
    }
    return metrics


def render_report(metrics):
    row_score = metrics["row_wise_leaky"]["mean_accuracy"]
    grouped_leaky_score = metrics["group_wise_leaky"]["mean_accuracy"]
    grouped_safe_score = metrics["group_wise_safe"]["mean_accuracy"]
    decision = metrics["decision"]

    return "\n".join(
        [
            "# Model Evaluation Audit",
            "",
            "## Executive finding",
            "",
            f"The row-wise evaluation reports {row_score:.3f} accuracy while repeated observations from the same accounts appear across training and validation. Keeping accounts separate drops the shortcut model to {grouped_leaky_score:.3f}. Removing `behavior_signature` and evaluating only `stable_signal` produces {grouped_safe_score:.3f} on unseen accounts.",
            "",
            "## Evidence",
            "",
            "| Evaluation | Mean accuracy | What it tests |",
            "| --- | ---: | --- |",
            f"| Row-wise with shortcut | {row_score:.3f} | New rows from accounts represented in training |",
            f"| Group-wise with shortcut | {grouped_leaky_score:.3f} | Entirely unseen accounts |",
            f"| Group-wise without shortcut | {grouped_safe_score:.3f} | Unseen accounts using the stable feature |",
            "",
            "## Decision",
            "",
            f"{decision['status']}: use `{decision['approved_feature_set'][0]}` and block `{decision['blocked_feature']}` from this model path.",
            "",
            "## Limitation",
            "",
            "This is a deterministic synthetic stress test. It demonstrates an evaluation failure mode and its repair, not production performance on a real population.",
            "",
        ]
    )


def write_artifacts(metrics):
    metrics_path = ROOT / "artifacts" / "metrics.json"
    report_path = ROOT / "reports" / "model_audit.md"
    metrics_path.parent.mkdir(exist_ok=True)
    report_path.parent.mkdir(exist_ok=True)

    metrics_path.write_text(
        json.dumps(metrics, indent=2) + "\n",
        encoding="utf-8",
    )
    report_path.write_text(render_report(metrics), encoding="utf-8")
    return metrics_path, report_path


def main():
    x_leaky, x_safe, targets, groups = build_dataset()
    metrics = evaluate_model(x_leaky, x_safe, targets, groups)
    metrics_path, report_path = write_artifacts(metrics)

    print(
        "Row-wise score with shortcut:",
        f"{metrics['row_wise_leaky']['mean_accuracy']:.3f}",
    )
    print(
        "Group-wise score with shortcut:",
        f"{metrics['group_wise_leaky']['mean_accuracy']:.3f}",
    )
    print(
        "Group-wise score without shortcut:",
        f"{metrics['group_wise_safe']['mean_accuracy']:.3f}",
    )
    print("Deployment decision:", metrics["decision"]["status"])
    print("Wrote", metrics_path.relative_to(ROOT))
    print("Wrote", report_path.relative_to(ROOT))


if __name__ == "__main__":
    main()

How to compare this file

This reference is the exact finished state of audit.py. A line-by-line match confirms that each evaluation path feeds the same metrics plus evidence pipeline.

Your audit now turns a misleading score into a documented deployment decision. Next, you will test the split invariant before publishing the case study.

Test and Publish the Case Study

Your audit now exposes entity leakage across account groups. It also produces evidence that supports a deployment decision.

A metric becomes easier to trust when its split invariant is tested. This step adds those tests before publishing the reasoning as a public GitHub case study.

In this step, get ready to:
  • Test the dataset shape and account isolation.
  • Document the experiment for a technical reviewer.
  • Publish the complete case study in a public repository.
Test the split invariant

A strong metric is only credible when the validation split matches the claim. These unit tests prove that the dataset has the expected shape and that each validation fold contains a completely unseen account.

  • In VS Code from earlier, create tests/test_audit.py inside the model-reality-check folder.
  • Add the first dataset-shape test by pasting this code into tests/test_audit.py:
import unittest

from sklearn.model_selection import GroupKFold

from audit import N_GROUPS, build_dataset, evaluate_model


class ModelAuditTests(unittest.TestCase):
    def setUp(self):
        self.x_leaky, self.x_safe, self.targets, self.groups = build_dataset()

    def test_dataset_shape(self):
        self.assertEqual(len(self.targets), 500)
        self.assertEqual(len(set(self.groups)), N_GROUPS)
        self.assertEqual(len(self.x_leaky[0]), 2)
        self.assertEqual(len(self.x_safe[0]), 1)


if __name__ == "__main__":
    unittest.main()

What Does This Test Prove?

  • The shared setup builds the same deterministic dataset before each test.
  • The assertions confirm that the dataset contains 500 targets across 100 account groups.
  • The feature-width assertions distinguish the two-feature shortcut view from the one-feature safe view.
  • Save tests/test_audit.py.
  • Run the first test from the active virtual environment with this command:
python -m unittest -v tests/test_audit.py

What Should You See?

The terminal runs one dataset-shape test. The final result reports OK.

Test File Not Running?

Confirm that test_audit.py is inside the tests folder. Confirm that your terminal is still inside model-reality-check.

If Python cannot import the audit module, check that audit.py sits beside the tests folder.

Still stuck? Help me diagnose why my first unittest cannot import or run the audit.

Dataset shape checks catch accidental changes to sample counts or feature views. The next test protects the account boundary that makes the evaluation honest.

  • In tests/test_audit.py, find the if __name__ == "__main__": block.
  • Insert this group-isolation test directly above that block:
    def test_group_splits_never_overlap(self):
        splitter = GroupKFold(n_splits=N_GROUPS)

        for train_index, test_index in splitter.split(
            self.x_leaky,
            self.targets,
            self.groups,
        ):
            train_groups = {self.groups[index] for index in train_index}
            test_groups = {self.groups[index] for index in test_index}
            self.assertTrue(train_groups.isdisjoint(test_groups))
            self.assertEqual(len(test_groups), 1)

How Does the Isolation Check Work?

  • The splitter creates one validation fold for each account group.
  • The set comparison proves that no account appears in both sides of a fold.
  • The final assertion proves that each validation fold contains exactly one account.
  • Save tests/test_audit.py.
  • Run the expanded test file with this command:
python -m unittest -v tests/test_audit.py

What Has the Suite Proved?

The terminal now runs two tests. A final OK confirms that every account remains isolated from its paired training fold.

Group Test Failing?

Check that splitter.split() receives self.groups as its third argument. Missing group labels removes the account boundary from this test.

Need a second pair of eyes? Help me debug the non-overlapping GroupKFold test.

Account isolation tests the split mechanics. The next test checks the behavioral result that matters to the audit.

  • In tests/test_audit.py, return to the line above the if __name__ == "__main__": block.
  • Insert this shortcut-detection test above that block:
    def test_audit_detects_and_repairs_shortcut(self):
        metrics = evaluate_model(
            self.x_leaky,
            self.x_safe,
            self.targets,
            self.groups,
        )

        self.assertGreaterEqual(
            metrics["row_wise_leaky"]["mean_accuracy"],
            0.95,
        )
        self.assertLessEqual(
            metrics["group_wise_leaky"]["mean_accuracy"],
            0.05,
        )
        self.assertGreaterEqual(
            metrics["group_wise_safe"]["mean_accuracy"],
            0.95,
        )

Why Test the Score Gap?

  • The row-wise assertion preserves the deliberately misleading result.
  • The group-wise shortcut assertion proves that the account-like feature fails on unseen accounts.
  • The safe-feature assertion proves that the stable signal transfers across account boundaries.
  • Save tests/test_audit.py.
  • Run the three-test suite with this command:
python -m unittest -v tests/test_audit.py

What Should the Gap Test Show?

The terminal now runs three tests. The final OK confirms that the audit still exposes the shortcut before verifying its repair.

Accuracy Assertion Failing?

Confirm that the test passes self.x_leaky before self.x_safe when calling evaluate_model().

Compare each metrics key with the matching evaluation path in audit.py.

Still seeing a failure? Help me trace which model-audit score violates its unittest threshold.

The final test connects the repaired evaluation to the project gate. This prevents a future code change from silently approving the wrong feature set.

  • In tests/test_audit.py, locate the line above the if __name__ == "__main__": block again.
  • Insert this deployment-decision test above that block:
    def test_repaired_model_passes_project_gate(self):
        metrics = evaluate_model(
            self.x_leaky,
            self.x_safe,
            self.targets,
            self.groups,
        )
        self.assertEqual(metrics["decision"]["status"], "PASS")
        self.assertEqual(
            metrics["decision"]["approved_feature_set"],
            ["stable_signal"],
        )

What Does the Gate Test Protect?

  • The first assertion requires the repaired model to produce a passing decision.
  • The second assertion limits approval to stable_signal.
  • Together they stop behavior_signature from returning to the approved model path.
  • Save tests/test_audit.py.

Before the full run, predict whether all four tests finish successfully.

  • Run the complete test file with this command:
python -m unittest -v tests/test_audit.py

What Does Success Look Like?

The terminal lists four passing tests. The run finishes with OK.

That is the hard proof in place. Your audit now checks both account isolation and the final deployment decision.

Full Suite Not Passing?

Use the verbose test names to identify the failing invariant. Compare only that method with the matching section in the full-file check below.

Need help reading the failure? Help me interpret my unittest output and repair the failing model-audit test.

✔️ Awesome, I've got everything!

Your four tests pass. Save tests/test_audit.py before continuing.

ⓧ I'd like to double check the full code

import unittest

from sklearn.model_selection import GroupKFold

from audit import N_GROUPS, build_dataset, evaluate_model


class ModelAuditTests(unittest.TestCase):
    def setUp(self):
        self.x_leaky, self.x_safe, self.targets, self.groups = build_dataset()

    def test_dataset_shape(self):
        self.assertEqual(len(self.targets), 500)
        self.assertEqual(len(set(self.groups)), N_GROUPS)
        self.assertEqual(len(self.x_leaky[0]), 2)
        self.assertEqual(len(self.x_safe[0]), 1)

    def test_group_splits_never_overlap(self):
        splitter = GroupKFold(n_splits=N_GROUPS)

        for train_index, test_index in splitter.split(
            self.x_leaky,
            self.targets,
            self.groups,
        ):
            train_groups = {self.groups[index] for index in train_index}
            test_groups = {self.groups[index] for index in test_index}
            self.assertTrue(train_groups.isdisjoint(test_groups))
            self.assertEqual(len(test_groups), 1)

    def test_audit_detects_and_repairs_shortcut(self):
        metrics = evaluate_model(
            self.x_leaky,
            self.x_safe,
            self.targets,
            self.groups,
        )

        self.assertGreaterEqual(
            metrics["row_wise_leaky"]["mean_accuracy"],
            0.95,
        )
        self.assertLessEqual(
            metrics["group_wise_leaky"]["mean_accuracy"],
            0.05,
        )
        self.assertGreaterEqual(
            metrics["group_wise_safe"]["mean_accuracy"],
            0.95,
        )

    def test_repaired_model_passes_project_gate(self):
        metrics = evaluate_model(
            self.x_leaky,
            self.x_safe,
            self.targets,
            self.groups,
        )
        self.assertEqual(metrics["decision"]["status"], "PASS")
        self.assertEqual(
            metrics["decision"]["approved_feature_set"],
            ["stable_signal"],
        )


if __name__ == "__main__":
    unittest.main()
Write the recruiter-facing evidence

The tests make the audit defensible to a machine. A focused README.md makes the same reasoning easy for a technical reviewer to inspect.

The case study leads with the failure mode before showing the repair. Its limitation keeps the synthetic result from being mistaken for a production benchmark.

  • In VS Code, create README.md inside the model-reality-check folder.
  • Add the project overview and experiment table by pasting this first section:
# Model Reality Check

A deterministic AI evaluation case study that catches entity leakage, replaces a shortcut feature, and produces inspectable model-audit evidence.

## The problem

Repeated observations from the same account can land in both training and validation. A model may then recognize account-specific patterns while appearing to generalize. This project makes that failure visible and tests a repair.

## Experiment

| Evaluation | Question | Expected signal |
| --- | --- | --- |
| Row-wise with shortcut | Can the model predict another row from accounts represented in training? | Misleadingly strong |
| Group-wise with shortcut | Can the same feature set handle entirely unseen accounts? | Shortcut exposed |
| Group-wise without shortcut | Can the stable feature transfer to unseen accounts? | Repair verified |

Why Lead With the Experiment?

  • The problem statement gives the first high score a clear source of risk.
  • The table separates the three evaluation questions before presenting any implementation detail.
  • A reviewer can see the full diagnostic arc without reading the Python source first.
  • Save README.md.
  • Confirm that the editor shows three evaluation rows beneath the Experiment heading.

The overview now frames the misleading score as an experiment. The next section gives a visitor a reproducible path through the evidence.

  • In README.md, place your cursor below the experiment table.
  • Append the Quickstart and repository map with this section:

## Quickstart

Requires Python 3.13.17.

```zsh
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python audit.py
python -m unittest -v tests/test_audit.py
```

The audit writes:

- `artifacts/metrics.json` for machine-readable evidence
- `reports/model_audit.md` for the decision narrative

## Repository map

- `audit.py`: deterministic dataset, three evaluation paths, deployment gate, and report generation
- `tests/test_audit.py`: dataset, split-integrity, leakage-detection, and decision tests
- `artifacts/metrics.json`: generated metrics from the latest run
- `reports/model_audit.md`: generated model-audit case study

What Makes This Reproducible?

  • The Quickstart gives a visitor the complete route from an empty environment to tested evidence.
  • The repository map connects each claim with the file that supports it.
  • The generated paths separate machine-readable metrics from the decision narrative.
  • Save README.md.
  • Confirm that the editor shows the five Quickstart commands and four repository-map entries.

A reviewer can now reproduce the result. The closing sections turn the technical sequence into an interview explanation with a clear boundary around its claims.

  • In README.md, place your cursor below the repository map.
  • Append the interview story and limitation with this final section:

## Interview story

The first score looked excellent, but its split allowed repeated observations from the same accounts on both sides. I changed the validation unit from rows to accounts, exposed the shortcut, removed the account-like feature, and added tests so future changes cannot silently reintroduce the flaw.

## Limitation

The dataset is synthetic and deliberately adversarial. The project demonstrates evaluation judgment and reproducibility, not production performance on a real population.

Why End With a Limitation?

The interview story shows how you challenged an impressive metric and repaired the experiment. The limitation defines exactly what the synthetic evidence supports.

  • Save README.md.
  • Read the final two sections in the editor.
  • Confirm that the last sentence avoids any production-performance claim.

✔️ Awesome, I've got everything!

Your README covers the problem, experiment, Quickstart, evidence map, interview story, and limitation.

ⓧ I'd like to double check the full code

# Model Reality Check

A deterministic AI evaluation case study that catches entity leakage, replaces a shortcut feature, and produces inspectable model-audit evidence.

## The problem

Repeated observations from the same account can land in both training and validation. A model may then recognize account-specific patterns while appearing to generalize. This project makes that failure visible and tests a repair.

## Experiment

| Evaluation | Question | Expected signal |
| --- | --- | --- |
| Row-wise with shortcut | Can the model predict another row from accounts represented in training? | Misleadingly strong |
| Group-wise with shortcut | Can the same feature set handle entirely unseen accounts? | Shortcut exposed |
| Group-wise without shortcut | Can the stable feature transfer to unseen accounts? | Repair verified |

## Quickstart

Requires Python 3.13.17.

```zsh
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python audit.py
python -m unittest -v tests/test_audit.py
```

The audit writes:

- `artifacts/metrics.json` for machine-readable evidence
- `reports/model_audit.md` for the decision narrative

## Repository map

- `audit.py`: deterministic dataset, three evaluation paths, deployment gate, and report generation
- `tests/test_audit.py`: dataset, split-integrity, leakage-detection, and decision tests
- `artifacts/metrics.json`: generated metrics from the latest run
- `reports/model_audit.md`: generated model-audit case study

## Interview story

The first score looked excellent, but its split allowed repeated observations from the same accounts on both sides. I changed the validation unit from rows to accounts, exposed the shortcut, removed the account-like feature, and added tests so future changes cannot silently reintroduce the flaw.

## Limitation

The dataset is synthetic and deliberately adversarial. The project demonstrates evaluation judgment and reproducibility, not production performance on a real population.

The public repository should exclude local environments and generated cache files. A .gitignore keeps those machine-specific files out of the case study.

  • In VS Code, create .gitignore inside the model-reality-check folder.
  • Add the local exclusions by pasting this content:
.venv/
__pycache__/
*.pyc
.DS_Store

What Does This File Exclude?

  • The virtual environment stays local because another machine can rebuild it from requirements.txt.
  • Python cache directories and compiled files stay out of the commit.
  • The macOS metadata file stays out of the repository.
  • Save .gitignore.
  • Confirm that the file contains four exclusion patterns.

✔️ Awesome, I've got everything!

Your local environment and machine-generated files are excluded from the repository.

ⓧ I'd like to double check the full code

.venv/
__pycache__/
*.pyc
.DS_Store

The repository should contain evidence from the latest version of the audit. Rebuilding it now keeps the generated metrics aligned with the code and tests you publish.

Before you rebuild the evidence, predict which deployment decision the current audit should print.

  • Refresh both generated evidence files by running this command:
python audit.py

What Should the Audit Refresh?

The terminal prints all three evaluation scores. It also prints Deployment decision: PASS.

The final two lines confirm that artifacts/metrics.json and reports/model_audit.md were written.

Evidence Files Missing?

Confirm that the active terminal is inside model-reality-check. Check the first error line if the script stops before writing the files.

Need help tracing the run? Help me diagnose why audit.py did not regenerate both evidence files.

  • Open artifacts/metrics.json from the VS Code file list.
  • Confirm that its decision status is PASS.
  • Open reports/model_audit.md from the VS Code file list.
  • Confirm that its limitation avoids claiming production performance.

Your repository evidence is current. A visitor can now reproduce the audit from the README and inspect the same decision artifacts.

Publish the repository

A local case study can only be reviewed on your machine. A short Git history preserves the finished state before you publish it.

Publishing makes the repository visible to anyone with its URL. The ignored virtual environment and cache files remain on your Mac.

  • Initialize the repository on main with this command group:
git init -b main
git add .
git commit -m "Build a group-aware model audit"

What Does This Commit Capture?

  • The first command creates the local repository on the main branch.
  • The second command stages the project files allowed by .gitignore.
  • The third command records the tested audit with a meaningful summary.

The terminal reports a new root commit with the project files it recorded. Your local case study now has a reproducible starting point.

Commit Not Created?

If Git asks for author details, follow its terminal guidance to configure the missing identity. Run the commit group again after the identity is available.

Need help with the output? Help me diagnose why my first Git commit was not created.

The remote repository must start empty so your existing files remain the source of truth. You will connect it only after GitHub creates the public destination.

  • Sign in to GitHub with the account you confirmed earlier.
  • Use GitHub's repository creation page to start a new repository.
  • Enter model-reality-check as the repository name.
  • Set the repository visibility to public.
  • Leave the README initialization option unselected.
  • Leave the license initialization option unselected.
  • Leave the gitignore initialization option unselected.
  • Create the repository.

GitHub now shows an empty public repository with instructions for existing local code. This empty state avoids a conflicting starter commit.

  • Copy the remote URL from the new repository page.
  • Record the copied value here: REMOTE-URL.

The next command group connects your local commit to the public repository. Git may ask you to complete GitHub authentication before the push can finish.

  • Replace REMOTE-URL below with the remote URL you recorded.
  • Connect and push the repository by running this command group:
git remote add origin REMOTE-URL
git remote -v
git push -u origin main

What Does the Push Do?

  • The remote command names the GitHub destination origin.
  • The verification command prints the destination before any files are uploaded.
  • The push uploads main and records it as the branch used for future pushes.

The terminal reports that the local branch now tracks the public branch. Your tested audit is published.

Push Not Completing?

Confirm that you replaced REMOTE-URL with the URL copied from your own repository. Complete the browser authentication flow if Git requests access.

If the remote contains starter files, recreate it without a README, license, or gitignore file before connecting it again.

Still blocked? Help me diagnose why my local main branch will not push to origin.

Before the public check, predict whether a signed-out visitor can see every evidence file.

  • Open a private browser window.
  • Paste your repository URL into the address bar: REMOTE-URL.
  • Confirm that the repository opens without a signed-in GitHub session.
  • Confirm that the repository page displays the README.md case study.
  • Confirm that audit.py is visible in the repository.
  • Confirm that tests/test_audit.py is visible in the repository.
  • Confirm that artifacts/metrics.json is visible in the repository.
  • Confirm that reports/model_audit.md is visible in the repository.

You should see the README rendered beneath the public file list. The source, tests, metrics, and report should all open while you remain signed out.

Repository Not Publicly Visible?

Return to the signed-in repository page and confirm that you created the intended public repository. Check that the browser URL belongs to model-reality-check.

If a file is missing, confirm that it existed before the commit. Create a new commit after restoring the file.

Need help with the final check? Help me diagnose why a signed-out visitor cannot see my complete GitHub case study.

You have turned a local model audit into public evidence. A reviewer can now inspect the split invariant, reproduce the metrics, and understand the deployment decision.

Secret mission

Add Continuous Model-Audit Checks

Your audit already proves that its evaluation split is honest. Add a GitHub Actions workflow that reruns the unit tests before rebuilding the evidence on every push or pull request.

Clean Up Your Resources

Clean Up Your Resources

Everything here has no ongoing cost because your GitHub Actions workflow uses a standard GitHub-hosted runner in a public repository. Decide whether to keep both resources, pause your work, or delete them.

Resources you used:

  • Local model-reality-check directory with its disposable virtual environment in .venv plus all source and evidence files.
  • Public model-reality-check GitHub repository with its Git history plus the Model Audit CI workflow.

Keep everything running

No action needed. Choose this if you want the repository to remain available as evidence of your model-evaluation work.

  • Keep the local model-reality-check directory so its environment and generated evidence remain available.
  • Keep the public model-reality-check repository visible so visitors can inspect the source, tests, metrics, report, and Git history.
  • Push future changes whenever you want the Model Audit CI workflow to rerun the audit.

Pause - I'll come back to this later

Shut down the active virtual environment to return this terminal to its normal shell. Your project files and public repository stay available.

  • Deactivate the active .venv environment by running this command:
deactivate

What does this command do?

This removes the virtual environment from your current terminal session. You should see its environment indicator disappear from the terminal prompt.

  • Leave the local model-reality-check directory on your Mac.
  • Leave the public model-reality-check repository unchanged on GitHub.
  • Stop pushing commits until you want the Model Audit CI workflow to run again.

Delete - I don't want to use this again

Remove all project resources to start fresh. GitHub's confirmation sequence gives you a final chance to stop before the public repository is deleted.

  • Deactivate the active .venv environment by running this command:
deactivate

What does this command do?

This ends the active virtual environment in your current terminal session. It leaves the local directory untouched for the next cleanup action.

  • Use Finder to locate the model-reality-check directory.
  • Move the model-reality-check directory to Trash.
  • Return to your public model-reality-check repository on GitHub.
  • Select Settings in the repository navigation.
  • Select General in the sidebar.
  • Scroll to Danger Zone.
  • Click Delete this repository.
  • Click I want to delete this repository.
  • Click I have read and understand these effects.
  • Enter model-reality-check when GitHub asks for the repository name.
  • Click the final Delete this repository button.

You should no longer find the repository at its previous GitHub URL. The local directory remains in Trash until you empty it.

Nice Work!

Nice Work!

You did it! Model Reality Check is now a public AI evaluation case study backed by reproducible evidence.

You've learned how to:

  • Expose entity leakage when a deceptively strong row-wise score collapses under an unseen-account evaluation.
  • Enforce account isolation with grouped cross-validation. Approve stable_signal using the project gate. Block behavior_signature from the model path.
  • Publish the case study on GitHub with reproducible evidence. Prove the split invariant through unit tests. State the synthetic-data limitation in the report.
  • Secret Mission: Added continuous model-audit checks with GitHub Actions. The workflow runs the unit tests on every push or pull request. It rebuilds the audit evidence only after the tests pass.

Ready to quiz yourself?