Build an LLM Regression Detector

Introduction

30 Second Summary

A prompt can look finished until one small edit changes dozens of model decisions. Without a benchmark, that regression reaches users before anyone can explain what moved.

In this project, you will build an LLM regression detector for a support-ticket classifier powered by Gemini 3.5 Flash-Lite. You will version prompts, validate structured outputs, evaluate human-reviewed cases, compare saved baselines, and block harmful behavior changes.

What You'll Build

You will corrupt a prompt on purpose, watch the detector trace the resulting label regressions, and see GitHub Actions block the change before a release.

By the end of this project, you'll have:

• A typed Gemini classifier that produces validated labels, confidence, rationale, latency, and token evidence for fictional support tickets.
• A human-reviewed golden dataset that exposes overall accuracy, changed cases, label regressions, and slice-level behavior.
• A deterministic regression gate that turns correctness, token, and p95 latency changes into PASS, WARN, or CRITICAL release outcomes.
• Secret Mission: Add a local trend dashboard that compares saved run history without making new Gemini requests.

Are there any prerequisites?

You need a Windows computer, Python 3.12, Git, a GitHub account, and a Google AI Studio project that can access Gemini 3.5 Flash-Lite.

The live evaluation uses fictional tickets and can fit within Google’s free tier when it is available to your project. Access, quotas, and any resulting cost depend on the limits shown in your own account.

Before We Start

Before We Start...

Before the build begins, lock in the failure you are preventing and the evidence your merge gate must produce.

Set Up the Windows Project

A regression detector needs a reproducible environment before it can judge reproducibility in a model. This step gives every later command the same Python runtime and package set.

You’ll prepare a Windows workspace in Visual Studio Code with Python, Git, and an isolated virtual environment. You’ll run every project command from VS Code’s integrated PowerShell terminal.

In this step, get ready to:

• Verify Python 3.12 and Git.
• Create the project workspace and virtual environment.
• Install a pinned dependency set.

Verify Python and Git

• Press the Windows key to open Search.
• Type Visual Studio Code.
• Press Enter to open VS Code.
• Click Terminal in the top menu.
• Select New Terminal.
• Check the required Python version in the integrated terminal by running:

py -3.12 --version

You should see Python 3.12 followed by a patch version.

• Check Git by running:

git --version

You should see a Git version number. Both commands now point to the tools used throughout the project.

Missing Python or Git?

Install Python 3.12 from python.org or Git from git-scm.com. After either installer changes your PATH, close the current VS Code terminal and open a fresh integrated terminal before checking again.

Still stuck? Help me diagnose why Python 3.12 or Git is unavailable in the VS Code terminal.

Create the isolated workspace

• Move to your Desktop and create the llm-regression-detector folder by running:

Set-Location "$HOME\Desktop"
New-Item -ItemType Directory -Force llm-regression-detector
Set-Location llm-regression-detector

• Click File in the VS Code menu.
• Select Open Folder.
• Choose Desktop\llm-regression-detector.
• Click Select Folder.
• Select Yes, I trust the authors if VS Code displays the Workspace Trust prompt.
• Click Terminal in the VS Code menu.
• Select New Terminal.
• Create the virtual environment and project folders by running:

py -3.12 -m venv .venv
New-Item -ItemType Directory -Force src\regression_detector, tests, prompts, data, runs, reports
New-Item -ItemType File -Force src\regression_detector\__init__.py

• Activate the virtual environment by running:

.\.venv\Scripts\Activate.ps1

You should see (.venv) at the start of the VS Code terminal prompt.

Activation blocked?

The integrated PowerShell terminal can block local activation scripts. Apply a process-scoped bypass by running:

Set-ExecutionPolicy -Scope Process Bypass

Activate .venv again in the same VS Code terminal.

Need a hand? Help me activate a Python virtual environment in the VS Code terminal.

Pin and install the dependencies

• Click the Explorer icon in the VS Code Activity Bar.
• Click New File at the top of the Explorer.
• Enter pyproject.toml to create it inside llm-regression-detector.
• Paste this configuration into the new file:

[build-system]
requires = ["setuptools==84.2.1"]
build-backend = "setuptools.build_meta"

[project]
name = "llm-regression-detector"
version = "0.1.0"
requires-python = ">=3.12,<3.13"
dependencies = [
  "google-genai==2.33.0",
  "pydantic==2.14.1",
  "PyYAML==6.0.3",
  "python-dotenv==1.2.4",
  "typer==0.27.3",
  "Jinja2==3.1.6",
]

[project.optional-dependencies]
dev = ["pytest==9.1.1"]

[tool.setuptools.packages.find]
where = ["src"]
include = ["regression_detector*"]
namespaces = false

[tool.pytest.ini_options]
testpaths = ["tests"]
markers = ["live: requires a Gemini API key and network access"]

Why pin every package?

A version pin makes installation repeatable across your laptop and CI. It also prevents a future SDK release from changing the evaluator while you are measuring model changes.

The Google Gen AI SDK warns about breaking behavior in its next major version. The 2.33.0 pin keeps this project on the verified API surface.

• Save pyproject.toml.
• Install the application and development dependencies by running:

python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

The first installation can take two or three minutes while Windows downloads wheels. A quiet pause during that download is expected.

• Before you check the environment, predict whether every import resolves from the pinned installation.
• Verify the toolchain by running:

python -c "from pathlib import Path; from google import genai; import regression_detector, pydantic, yaml, typer, jinja2, pytest; print('package:', Path(regression_detector.__file__).resolve()); print('toolchain ready')"

You should see toolchain ready. That confirms the Windows environment is ready for the detector.

Dependency installation failed?

Confirm the prompt starts with (.venv) and that python --version reports Python 3.12. An older interpreter cannot install this exact set.

Share the failure safely through Help me fix my pinned Python dependency installation.

Your Windows package now installs from the explicit src/ layout. Next, you’ll store Gemini credentials safely without making a live request yet.

Authenticate with Gemini

Your pinned Windows environment can now run the detector’s code. The next risk is credential handling because every live evaluation needs a Gemini API key.

You’ll protect the key before any request reaches Gemini 3.5 Flash-Lite. Then you’ll prepare a smoke client that stays offline until the local contracts and retry tests pass.

In this step, get ready to:

• Store the Gemini API key outside source control.
• Create a validated Gemini 3.5 Flash-Lite smoke client.
• Compile the client without making a network request.

Protect the API key

A leaked key can consume your project quota or create charges. You’ll keep it in .env and prove Git ignores that file before using it.

• Open the Google AI Studio API keys page.
• Click Create API key and follow the project dialog.
• Copy the key into your password manager, then close the reveal dialog.

Keep evaluation data fictional

Free-tier input and output tokens are available when the model is enabled for your project. Google states that content sent through unpaid services may be used to improve its products and may be reviewed by people after privacy protections are applied.

Use only the fictional tickets supplied in this project. Never evaluate customer records, private source code, credentials, or personal information on the free tier.

• Initialize the local Git repository by running:

git init

• In the VS Code Explorer, create .gitignore inside llm-regression-detector.
• Paste these ignore rules into the new file:

.env
.venv/
__pycache__/
.pytest_cache/
*.pyc
runs/*.json
reports/*.html
.cache/

• In the VS Code Explorer, create .env beside pyproject.toml.
• Paste this line into the new file:

GEMINI_API_KEY=your-key-here

• Replace your-key-here with the key from your password manager.
• Save .env and close the file.
• Verify Git ignores the secret file by running:

git check-ignore .env

You should see .env in the output. That result proves the key file stays outside normal commits.

Git does not list .env?

Make sure .gitignore is saved beside .env and that the first line is exactly .env.

Need help? Help me protect my .env file with Git.

Create the smoke-test client

• In the VS Code Explorer, expand src\regression_detector.
• Create smoke.py inside that folder.
• Paste this client into the new file:

import os

from dotenv import load_dotenv
from google import genai
from google.genai import types

# Load the ignored local key only when this module is executed.
load_dotenv()
api_key = os.environ.get("GEMINI_API_KEY")
if not api_key:
    raise RuntimeError("GEMINI_API_KEY is missing")

with genai.Client(api_key=api_key) as client:
    response = client.models.generate_content(
        model="gemini-3.5-flash-lite",
        contents="Return only the word READY.",
        config=types.GenerateContentConfig(
            thinking_config=types.ThinkingConfig(thinking_level="minimal"),
            max_output_tokens=20,
        ),
    )

text = (response.text or "").strip()
candidates = response.candidates or []
usage = response.usage_metadata
if not candidates or text != "READY":
    raise RuntimeError("Gemini returned an invalid smoke response")
counts = None if usage is None else (
    usage.prompt_token_count, usage.candidates_token_count, usage.total_token_count
)
if counts is None or any(value is None for value in counts):
    raise RuntimeError("Gemini omitted required usage metadata")
print(f"response={text} input={counts[0]} output={counts[1]} total={counts[2]}")

What does this client prove?

• load_dotenv() reads the ignored local secret only when the smoke module runs.
• thinking_level="minimal" uses the model’s supported low-cost classification setting without deprecated sampling controls.
• usage_metadata must contain input, output, and total token counts before the response is accepted.

• Save src\regression_detector\smoke.py with Ctrl+S.
• Confirm the file exists in the VS Code Explorer under src\regression_detector.

You should see smoke.py listed beneath src\regression_detector in the Explorer.

Check the client safely

Compare the saved file with this smoke-test checklist before the live call.

Validate the client offline

The smoke client now contains a live call, but you will only compile it here. Compilation checks Python syntax without loading the key or opening a network connection.

• Compile smoke.py without executing its live request by running:

python -m py_compile src\regression_detector\smoke.py

The command should finish without output. That quiet success proves the module parses while your API quota remains untouched.

Compilation failed?

A syntax error includes the failing line number. Compare that line with the saved code before continuing.

Need help? Help me fix the offline smoke-client check.

Your key is protected and the live client compiles without spending quota. Next, you’ll define the typed records that every dataset, run, cache entry, and comparison must obey.

Define the Typed Data Contract

The protected smoke client is ready, but free-form dictionaries would let malformed records spread through the detector. A trustworthy gate needs one explicit contract before any live evaluation begins.

You’ll use Pydantic models to reject invalid data at each boundary. This turns silent shape changes into immediate validation errors.

In this step, get ready to:

• Model tickets and predictions.
• Model evaluation runs and usage.
• Model comparisons and gate decisions.

Model tickets and predictions

The domain model fixes the allowed labels and requires every case to carry an expected answer. The evaluator can only score a prediction after both sides pass this contract.

• In the VS Code Explorer, create domain.py inside src\regression_detector.
• Paste this contract into the new file:

from enum import StrEnum

from pydantic import BaseModel, ConfigDict, Field


# Restrict classification to the support team's approved labels.
class Label(StrEnum):
    BILLING = "billing"
    TECHNICAL = "technical"
    ACCOUNT = "account"


# Store one fictional ticket and its human-approved answer.
class TicketCase(BaseModel):
    model_config = ConfigDict(extra="forbid")

    case_id: str = Field(pattern=r"^case-[0-9]{3}$")
    text: str = Field(min_length=10, max_length=500)
    expected_label: Label
    slice: str = Field(min_length=2, max_length=40)


# Validate the structured answer returned by the model.
class Prediction(BaseModel):
    model_config = ConfigDict(extra="forbid")

    label: Label
    confidence: float = Field(ge=0, le=1)
    rationale: str = Field(min_length=3, max_length=200)

What does the domain contract enforce?

• StrEnum makes labels readable in JSON while keeping the allowed set closed.
• extra="forbid" rejects unexpected keys instead of silently discarding them.
• Field constraints catch invalid IDs, empty tickets, and impossible confidence values.

• Save src\regression_detector\domain.py.
• Confirm a valid case loads by running:

python -c "from regression_detector.domain import TicketCase; print(TicketCase.model_validate({'case_id':'case-001','text':'I cannot sign in to my account','expected_label':'account','slice':'clear'}).expected_label)"

You should see account.

Domain validation failed?

Check that every class body is indented with four spaces and that the pattern contains [0-9]{3}. Pydantic reports the exact field that violates the contract.

Ask for a focused review through this model-contract prompt.

Model evaluation runs

A regression result needs more than accuracy. Each run also records the exact model, prompt hash, dataset hash, latency, token usage, and per-case output.

• In the VS Code Explorer, create run_models.py inside src\regression_detector.
• Paste these run models into the new file:

from datetime import datetime
from typing import Literal

from pydantic import BaseModel, ConfigDict, Field, model_validator

from .domain import Label, Prediction


class UsageStats(BaseModel):
    input_tokens: int = Field(ge=0)
    output_tokens: int = Field(ge=0)
    thought_tokens: int = Field(ge=0)
    total_tokens: int = Field(ge=0)

    @classmethod
    def zero(cls) -> "UsageStats":
        return cls(input_tokens=0, output_tokens=0, thought_tokens=0, total_tokens=0)

    @model_validator(mode="after")
    def totals_are_consistent(self) -> "UsageStats":
        subtotal = self.input_tokens + self.output_tokens + self.thought_tokens
        if self.total_tokens < subtotal:
            raise ValueError("total_tokens is smaller than its components")
        return self


class ApiMeasurement(BaseModel):
    latency_ms: float = Field(gt=0)
    usage: UsageStats


class EvaluationSettings(BaseModel):
    api_version: str
    sdk_version: str
    thinking_level: Literal["minimal"]
    max_output_tokens: int = Field(gt=0)
    schema_version: str


class CaseResult(BaseModel):
    case_id: str = Field(pattern=r"^case-[0-9]{3}$")
    expected_label: Label
    prediction: Prediction
    correct: bool
    api_measurement: ApiMeasurement
    run_api_usage: UsageStats
    cache_hit: bool

    @model_validator(mode="after")
    def evidence_is_consistent(self) -> "CaseResult":
        if self.correct != (self.prediction.label == self.expected_label):
            raise ValueError("correct disagrees with the labels")
        expected = UsageStats.zero() if self.cache_hit else self.api_measurement.usage
        if self.run_api_usage != expected:
            raise ValueError("run_api_usage disagrees with cache provenance")
        return self


class EvaluationRun(BaseModel):
    model_config = ConfigDict(extra="forbid")

    run_id: str
    created_at: datetime
    model_id: str
    prompt_version: str
    prompt_hash: str
    dataset_hash: str
    settings: EvaluationSettings
    results: list[CaseResult] = Field(min_length=1)

    @model_validator(mode="after")
    def case_ids_are_unique(self) -> "EvaluationRun":
        case_ids = [result.case_id for result in self.results]
        if len(case_ids) != len(set(case_ids)):
            raise ValueError("evaluation run contains duplicate case IDs")
        return self

Why keep complete run artifacts?

• UsageStats makes token consumption comparable between runs.
• CaseResult keeps each prediction next to its expected label and request evidence.
• EvaluationRun stores the compatibility fields required before two runs can be compared.

• Save src\regression_detector\run_models.py.
• Confirm the models import by running:

python -c "from regression_detector.run_models import EvaluationRun; print(EvaluationRun.__name__)"

You should see EvaluationRun.

Model comparisons and test the contract

• In the VS Code Explorer, create comparison_models.py inside src\regression_detector.
• Paste these comparison models into the new file:

from enum import StrEnum

from pydantic import BaseModel, Field


# Use stable severity values in the CLI and CI exit policy.
class Severity(StrEnum):
    PASS = "PASS"
    WARN = "WARN"
    CRITICAL = "CRITICAL"


# Describe one changed case between compatible runs.
class CaseDelta(BaseModel):
    case_id: str
    before_label: str
    after_label: str
    was_correct: bool
    is_correct: bool


# Summarize the metrics and changed cases used by the gate.
class ComparisonSummary(BaseModel):
    baseline_run_id: str
    candidate_run_id: str
    accuracy_before: float = Field(ge=0, le=1)
    accuracy_after: float = Field(ge=0, le=1)
    accuracy_delta: float
    regressions: list[CaseDelta]
    improvements: list[CaseDelta]
    changed: list[CaseDelta]


# Return both a machine decision and a human-readable reason.
class GateDecision(BaseModel):
    severity: Severity
    exit_code: int = Field(ge=0, le=2)
    reasons: list[str]

• Save src\regression_detector\comparison_models.py with Ctrl+S.
• In the VS Code Explorer, create test_models.py inside tests.
• Paste these contract tests into the new file:

from datetime import UTC, datetime

import pytest
from pydantic import ValidationError

from regression_detector.domain import Prediction, TicketCase
from regression_detector.run_models import EvaluationRun


def test_valid_prediction() -> None:
    prediction = Prediction(
        label="technical", confidence=0.9, rationale="Device fails"
    )
    assert prediction.label == "technical"


def test_unknown_expected_label_is_rejected() -> None:
    with pytest.raises(ValidationError):
        TicketCase.model_validate({
            "case_id": "case-001",
            "text": "Please change the delivery address",
            "expected_label": "shipping",
            "slice": "clear",
        })


def test_empty_evaluation_run_is_rejected() -> None:
    with pytest.raises(ValidationError):
        EvaluationRun(
            run_id="empty", created_at=datetime.now(UTC), model_id="model",
            prompt_version="v1", prompt_hash="a", dataset_hash="b",
            settings={"api_version": "v1beta", "sdk_version": "2.33.0",
                      "thinking_level": "minimal", "max_output_tokens": 200,
                      "schema_version": "prediction-v2"},
            results=[],
        )

• In the VS Code Explorer, create conftest.py inside tests.
• Paste this fixture into the new file to make every pytest run offline by default:

import socket

import pytest


@pytest.fixture(autouse=True)
def offline_test_environment(monkeypatch: pytest.MonkeyPatch) -> None:
    # Prevent tests from loading credentials or opening sockets.
    monkeypatch.delenv("GEMINI_API_KEY", raising=False)
    monkeypatch.delenv("GOOGLE_API_KEY", raising=False)
    monkeypatch.setenv("PYTHON_DOTENV_DISABLED", "1")

    def deny_network(*args: object, **kwargs: object) -> None:
        raise AssertionError("offline tests must not use the network")

    monkeypatch.setattr(socket, "create_connection", deny_network)
    monkeypatch.setattr(socket.socket, "connect", deny_network)

What do these tests protect?

• test_valid_prediction checks the happy path.
• test_unknown_expected_label_is_rejected proves the closed label set fails safely.

• Save tests\test_models.py.
• Before you run the tests, predict whether the invalid shipping label can pass validation.
• Run the contract tests:

python -m pytest tests\test_models.py -q

You should see every contract test pass. The suite proves unknown labels and empty evaluation runs fail before metrics can be calculated.

Model tests failed?

Run the command from the folder containing pyproject.toml. If an import fails, confirm the virtual environment is active and reinstall with python -m pip install -e ".[dev]".

Use this test-debugging prompt for a focused fix.

Your detector now has typed boundaries for every important artifact. Next, you’ll move model behavior into versioned prompt and configuration files.

Load Versioned Prompts and Configuration

Typed records protect the detector’s data shape, but model behavior still depends on instructions, model ID, thinking level, API version, and token limits. Those choices must be reviewable before a run can be reproduced.

You’ll move behavior into versioned YAML files and compute a prompt hash from the exact bytes used. A candidate change then becomes a diffable artifact instead of an invisible code edit.

In this step, get ready to:

• Write the baseline and candidate prompts.
• Record deterministic runtime settings.
• Load and fingerprint the configuration.

Write versioned prompts

The baseline prompt defines the behavior you approve. The candidate prompt starts as a copy so later changes have a clean reference point.

• In the VS Code Explorer, create classifier-v1.yaml inside prompts.
• Paste this baseline prompt into the new file:

version: classifier-v1
system: |
  You classify fictional support tickets for an internal evaluation.
  Choose billing for charges, invoices, refunds, or payments.
  Choose technical for device, software, connection, or performance failures.
  Choose account for sign-in, profile, access, or identity requests.
  Use the ticket text only.
user_template: |
  Classify this ticket:
  {{ ticket_text }}

Why keep schema out of the prompt?

The instructions define the label boundary without copying the response schema into the prompt. Gemini receives the schema separately through the SDK, which avoids competing format instructions.

• Save prompts\classifier-v1.yaml.
• Create the candidate as an exact copy by running:

Copy-Item prompts\classifier-v1.yaml prompts\classifier-v2.yaml
(Get-Content prompts\classifier-v2.yaml) -replace 'classifier-v1', 'classifier-v2' | Set-Content prompts\classifier-v2.yaml

• Confirm both prompt files exist by running:

Get-ChildItem prompts\classifier-v*.yaml

You should see classifier-v1.yaml and classifier-v2.yaml.

Prompt copy failed?

If the copy command reports a missing path, run Get-Location and confirm it ends with llm-regression-detector.

Use this prompt-copying help if the filenames still differ.

Record runtime settings

• In the VS Code Explorer, create config.yaml beside pyproject.toml.
• Paste these runtime settings into the new file:

model_id: gemini-3.5-flash-lite
api_version: v1beta
thinking_level: minimal
max_output_tokens: 120
requests_per_minute: 5
retry_attempts: 4
max_retry_wait_seconds: 60
schema_version: prediction-v2
cache_path: .cache/responses.sqlite3
dataset_path: data/golden.json
baseline_path: runs/baseline.json
report_path: reports/regression-report.html

Why these defaults?

Gemini 3.5 Flash-Lite deprecates and ignores custom sampling controls such as temperature. The supported minimal thinking level keeps this classification workload fast without relying on a silent no-op.

Four total attempts bound transient retries. A 60-second ceiling prevents the evaluator from sleeping indefinitely or retrying before a longer server-requested delay.

• Save config.yaml.
• Check the YAML syntax by running:

python -c "import yaml; print(yaml.safe_load(open('config.yaml', encoding='utf-8'))['model_id'])"

You should see gemini-3.5-flash-lite.

Load and fingerprint the files

• In the VS Code Explorer, create config.py inside src\regression_detector.
• Paste this loader into the new file:

import hashlib
from pathlib import Path
from typing import Literal

import yaml
from pydantic import BaseModel, ConfigDict, Field


class Settings(BaseModel):
    model_config = ConfigDict(extra="forbid")

    model_id: str
    api_version: str
    thinking_level: Literal["minimal"]
    max_output_tokens: int = Field(gt=0)
    requests_per_minute: int = Field(gt=0)
    retry_attempts: int = Field(ge=1, le=10)
    max_retry_wait_seconds: int = Field(ge=1, le=300)
    schema_version: str
    cache_path: Path
    dataset_path: Path
    baseline_path: Path
    report_path: Path


def load_settings(path: Path = Path("config.yaml")) -> Settings:
    # Reject unknown or invalid runtime settings at startup.
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    return Settings.model_validate(data)


def file_hash(path: Path) -> str:
    # Fingerprint exact bytes so every behavior change is traceable.
    return hashlib.sha256(path.read_bytes()).hexdigest()

What does the loader guarantee?

• Settings rejects missing or impossible runtime values.
• load_settings() converts paths into Windows-safe Path objects.
• file_hash() creates the compatibility fingerprint stored with every run.

• Save src\regression_detector\config.py with Ctrl+S.
• In the VS Code Explorer, create test_config.py inside tests.
• Paste these configuration tests into the new file:

from pathlib import Path

from regression_detector.config import file_hash, load_settings


def test_load_settings() -> None:
    # Confirm every reproducibility control reaches the typed loader.
    settings = load_settings()
    assert settings.model_id == "gemini-3.5-flash-lite"
    assert settings.api_version == "v1beta"
    assert settings.thinking_level == "minimal"
    assert settings.schema_version == "prediction-v2"
    assert settings.max_retry_wait_seconds == 60
    assert not hasattr(settings, "temperature")


def test_prompt_hashes_differ() -> None:
    # A version-only prompt change must still produce a new fingerprint.
    first = file_hash(Path("prompts/classifier-v1.yaml"))
    second = file_hash(Path("prompts/classifier-v2.yaml"))
    assert len(first) == 64
    assert first != second

• Save tests\test_config.py.
• Before you run the tests, predict whether changing only the prompt version produces a new hash.
• Run the configuration tests:

python -m pytest tests\test_config.py -q

You should see 2 passed. The distinct hashes prove a one-line prompt change is traceable.

Configuration tests failed?

If the hash test fails, open prompts\classifier-v2.yaml from the VS Code Explorer. Confirm its first line says version: classifier-v2.

Use this configuration-debugging prompt for the smallest correction.

Your model behavior is now versioned and fingerprinted. Next, you’ll build the human-approved golden dataset that gives those prompts a stable target.

Create the Golden Dataset

Versioned prompts are only meaningful when they face the same approved examples. A shifting or weakly labeled dataset can make a worse model look better.

You’ll create a balanced golden dataset of fictional support tickets. Then you’ll validate every record and fingerprint the normalized cases used by each run.

In this step, get ready to:

• Create balanced fictional ticket cases.
• Review the expected labels and slices.
• Validate and fingerprint the dataset.

Create the fictional cases

Each label gets six cases across clear, paraphrased, and ambiguous wording. The tickets contain no customer data, credentials, or production text.

• In the VS Code Explorer, create golden.json inside data.
• Paste these fictional cases into the new file:

[
  {"case_id":"case-001","text":"I was charged twice for the same monthly plan.","expected_label":"billing","slice":"clear"},
  {"case_id":"case-002","text":"My refund was approved but has not reached my card.","expected_label":"billing","slice":"clear"},
  {"case_id":"case-003","text":"Please send a copy of last month's invoice.","expected_label":"billing","slice":"paraphrase"},
  {"case_id":"case-004","text":"There is a purchase on my statement that I do not recognize.","expected_label":"billing","slice":"ambiguous"},
  {"case_id":"case-005","text":"Can the renewal date for my subscription be moved?","expected_label":"billing","slice":"paraphrase"},
  {"case_id":"case-006","text":"The checkout page rejects every payment method I try.","expected_label":"billing","slice":"ambiguous"},
  {"case_id":"case-007","text":"The desktop app closes whenever I open settings.","expected_label":"technical","slice":"clear"},
  {"case_id":"case-008","text":"The device connects to Wi-Fi but never loads data.","expected_label":"technical","slice":"clear"},
  {"case_id":"case-009","text":"Updates on my phone do not appear on my laptop.","expected_label":"technical","slice":"paraphrase"},
  {"case_id":"case-010","text":"The display flashes after waking from sleep.","expected_label":"technical","slice":"paraphrase"},
  {"case_id":"case-011","text":"Everything works eventually, but startup takes several minutes.","expected_label":"technical","slice":"ambiguous"},
  {"case_id":"case-012","text":"I see an error after installing the latest version.","expected_label":"technical","slice":"ambiguous"},
  {"case_id":"case-013","text":"I forgot my password and cannot sign in.","expected_label":"account","slice":"clear"},
  {"case_id":"case-014","text":"Please change the email address on my profile.","expected_label":"account","slice":"clear"},
  {"case_id":"case-015","text":"Too many login attempts locked me out.","expected_label":"account","slice":"paraphrase"},
  {"case_id":"case-016","text":"I want my profile and saved details removed.","expected_label":"account","slice":"paraphrase"},
  {"case_id":"case-017","text":"The verification code never arrives when I log in.","expected_label":"account","slice":"ambiguous"},
  {"case_id":"case-018","text":"My legal name changed and the old one still appears.","expected_label":"account","slice":"ambiguous"}
]

What makes this dataset golden?

Balance prevents the overall score from hiding a weak label. Slices expose failures that only appear with paraphrases or ambiguous wording.

The expected label is a human decision. Review every case before treating this file as the source of truth.

• Save data\golden.json.
• Confirm the JSON parses and contains 18 cases by running:

python -c "import json; print(len(json.load(open('data/golden.json', encoding='utf-8'))))"

You should see 18.

Golden dataset does not parse?

A parser error names the line and column containing an invalid comma or quote. Compare that line with the supplied object and keep the outer square brackets.

Use this JSON-repair prompt if the file still fails.

Review the approved labels

• Open data\golden.json from the VS Code Explorer.
• Read the six billing cases to confirm each concerns money or payment.
• Read the six technical cases to confirm each concerns product behavior.
• Read the six account cases to confirm each concerns identity or access.
• Review each slice value against the wording difficulty.

The review makes you the approver of the target behavior. If you disagree with a label, change it now and keep the case ID stable.

Validate and fingerprint the dataset

• In the VS Code Explorer, create dataset.py inside src\regression_detector.
• Paste this validator into the new file:

import hashlib
import json
from collections import Counter
from pathlib import Path

from pydantic import TypeAdapter

from .domain import TicketCase

CASE_LIST = TypeAdapter(list[TicketCase])


def load_cases(path: Path) -> list[TicketCase]:
    # Validate the complete file before an evaluator can use it.
    raw = json.loads(path.read_text(encoding="utf-8"))
    cases = CASE_LIST.validate_python(raw)
    if not cases:
        raise ValueError("golden dataset must contain at least one case")
    ids = [case.case_id for case in cases]
    if len(ids) != len(set(ids)):
        raise ValueError("case_id values must be unique")
    return cases


def dataset_hash(cases: list[TicketCase]) -> str:
    # Reject meaningless fingerprints for empty in-memory datasets.
    if not cases:
        raise ValueError("cannot hash an empty dataset")
    payload = [case.model_dump(mode="json") for case in cases]
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


def label_counts(cases: list[TicketCase]) -> Counter[str]:
    # Expose label balance as an observable dataset check.
    return Counter(case.expected_label.value for case in cases)

What does dataset validation prove?

• TypeAdapter validates the list and every nested TicketCase.
• case_id uniqueness keeps comparison joins unambiguous.
• Canonical JSON makes indentation-only edits invisible to the dataset fingerprint.

• Save src\regression_detector\dataset.py with Ctrl+S.
• In the VS Code Explorer, create test_dataset.py inside tests.
• Paste these dataset tests into the new file:

from pathlib import Path

import pytest

from regression_detector.dataset import dataset_hash, label_counts, load_cases


def test_golden_dataset() -> None:
    # Prove the checked-in set is balanced and identifiable.
    cases = load_cases(Path("data/golden.json"))
    assert len(cases) == 18
    assert label_counts(cases) == {
        "billing": 6,
        "technical": 6,
        "account": 6,
    }
    assert len(dataset_hash(cases)) == 64


def test_empty_dataset_file_is_rejected(tmp_path: Path) -> None:
    # Prevent an empty file from producing meaningless perfect metrics.
    path = tmp_path / "empty.json"
    path.write_text("[]", encoding="utf-8")
    with pytest.raises(ValueError, match="at least one case"):
        load_cases(path)


def test_empty_in_memory_dataset_is_rejected() -> None:
    with pytest.raises(ValueError, match="empty dataset"):
        dataset_hash([])

• Save tests\test_dataset.py.
• Before you run the test, predict whether a mistyped label or duplicate ID would reach the evaluator.
• Run the dataset test:

python -m pytest tests\test_dataset.py -q

You should see all dataset tests pass. The validated file contains six approved cases per label, while empty files and empty fingerprints fail closed.

Dataset test failed?

Pydantic names the invalid field and case position. Correct that record in golden.json without changing unrelated cases.

Use this dataset-debugging prompt if the check remains red.

Your approved target is balanced, typed, and fingerprinted. Next, you’ll add a response cache and bounded retries so repeated evaluations stay cheap and recover from transient API failures.

Add Caching and Safe Retries

The golden dataset makes repeated runs comparable, but it also repeats identical API work. Without controls, retries can multiply cost and rapid loops can collide with account-specific rate limits.

You’ll add a local SQLite cache, deterministic cache keys, request pacing, and bounded exponential backoff. These controls preserve evidence while keeping failures visible.

In this step, get ready to:

• Cache structured responses in SQLite.
• Pace requests and calculate bounded backoff.
• Test cache and retry behavior offline.

Cache responses with complete keys

A cache hit is valid only when every behavior-changing input matches. The key therefore includes the model, prompt hash, case text, generation settings, and schema version.

• In the VS Code Explorer, create cache.py inside src\regression_detector.
• Paste this cache implementation into the new file:

import hashlib
import json
import sqlite3
from pathlib import Path


def make_cache_key(
    *, model_id: str, prompt_hash: str, ticket_text: str,
    api_version: str, thinking_level: str, max_output_tokens: int,
    schema_version: str, sdk_version: str,
) -> str:
    # Include every input that can change the validated response.
    payload = {
        "api_version": api_version,
        "max_output_tokens": max_output_tokens,
        "model_id": model_id,
        "prompt_hash": prompt_hash,
        "schema_version": schema_version,
        "sdk_version": sdk_version,
        "thinking_level": thinking_level,
        "ticket_text": ticket_text,
    }
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


class ResponseCache:
    def __init__(self, path: Path) -> None:
        path.parent.mkdir(parents=True, exist_ok=True)
        self.connection = sqlite3.connect(path)
        self.connection.execute(
            "CREATE TABLE IF NOT EXISTS responses "
            "(cache_key TEXT PRIMARY KEY, response_json TEXT NOT NULL)"
        )

    def get(self, cache_key: str) -> str | None:
        row = self.connection.execute(
            "SELECT response_json FROM responses WHERE cache_key = ?", (cache_key,)
        ).fetchone()
        return row[0] if row else None

    def put(self, cache_key: str, response_json: str) -> None:
        self.connection.execute(
            "INSERT OR REPLACE INTO responses VALUES (?, ?)",
            (cache_key, response_json),
        )
        self.connection.commit()

    def close(self) -> None:
        self.connection.close()

What does the cache protect?

• make_cache_key() sorts keys so dictionary order cannot change the fingerprint.
• responses stores only validated JSON keyed by its complete request identity.
• Parameterized SQL keeps cache content separate from the query.

• Save src\regression_detector\cache.py.
• Confirm SQLite can open the cache by running:

python -c "from pathlib import Path; from regression_detector.cache import ResponseCache; c=ResponseCache(Path('.cache/test.sqlite3')); c.close(); print('cache ready')"

You should see cache ready and a new local .cache\test.sqlite3 file.

Cache file was not created?

Confirm the .cache directory is writable and the SQL string contains both cache_key and response_json columns.

Use this Windows cache prompt for a focused diagnosis.

Pace and retry transient requests

Rate limits are evaluated per project and vary by model and account. The local setting provides conservative pacing, while retries apply only to transient server or quota responses.

• In the VS Code Explorer, create controls.py inside src\regression_detector.
• Paste these pacing and retry controls into the new file:

import time
from collections.abc import Callable

RETRYABLE_SERVER_CODES = {500, 502, 503, 504}
PERMANENT_CLIENT_CODES = {400, 401, 403, 404, 422}


class RetryBudgetExceeded(RuntimeError):
    pass


def should_retry(status_code: int, retry_after: float | None) -> bool:
    # Retry servers and explicitly paced quota responses only.
    if status_code in RETRYABLE_SERVER_CODES:
        return True
    return status_code == 429 and retry_after is not None


def retry_delay(
    attempt: int, retry_after: float | None, max_wait: float
) -> float:
    # Honor a server delay when it fits the configured wait budget.
    delay = retry_after if retry_after is not None else 2.0 ** max(0, attempt - 1)
    if delay < 0:
        raise ValueError("retry delay cannot be negative")
    if delay > max_wait:
        raise RetryBudgetExceeded(
            f"required retry delay {delay}s exceeds {max_wait}s"
        )
    return delay


class RequestPacer:
    def __init__(
        self,
        requests_per_minute: int,
        clock: Callable[[], float] = time.monotonic,
        sleeper: Callable[[float], None] = time.sleep,
    ) -> None:
        if requests_per_minute <= 0:
            raise ValueError("requests_per_minute must be positive")
        self.interval = 60.0 / requests_per_minute
        self.clock = clock
        self.sleeper = sleeper
        self.last_request: float | None = None

    def wait(self) -> None:
        now = self.clock()
        if self.last_request is not None:
            remaining = self.interval - (now - self.last_request)
            if remaining > 0:
                self.sleeper(remaining)
        self.last_request = self.clock()

Why separate pacing from retries?

• RETRYABLE_SERVER_CODES contains 500, 502, 503, and 504. Permanent authentication and validation failures stay outside the retry path.
• should_retry() also permits 429 only when the response supplies a retry delay.
• retry_delay() honors that server delay or calculates 1, 2, 4, 8 seconds of exponential backoff. It raises RetryBudgetExceeded before a wait exceeds the configured budget.
• RequestPacer accepts injected clock and sleep functions so tests can simulate an early request without real sleeping.

• Save src\regression_detector\controls.py.
• Check the bounded schedule by running:

python -c "from regression_detector.controls import retry_delay; print([retry_delay(i, None, 60) for i in range(1, 7)])"

You should see [1.0, 2.0, 4.0, 8.0, 16.0, 32.0]. A later attempt that exceeds the 60-second budget fails instead of sleeping longer.

Test the controls offline

• In the VS Code Explorer, create test_controls.py inside tests.
• Paste these offline control tests into the new file:

from pathlib import Path

import pytest

from regression_detector.cache import ResponseCache, make_cache_key
from regression_detector.controls import (
    RETRYABLE_SERVER_CODES, RequestPacer, RetryBudgetExceeded,
    retry_delay, should_retry,
)


def key(**changes: object) -> str:
    values = {
        "model_id": "model", "prompt_hash": "prompt", "ticket_text": "ticket",
        "api_version": "v1beta", "thinking_level": "minimal",
        "max_output_tokens": 120, "schema_version": "prediction-v2",
        "sdk_version": "2.33.0",
    }
    values.update(changes)
    return make_cache_key(**values)


def test_cache_round_trip(tmp_path: Path) -> None:
    cache = ResponseCache(tmp_path / "responses.sqlite3")
    cache.put("abc", '{"prediction":{"label":"billing"},"api_measurement":{}}')
    assert cache.get("abc") is not None
    assert cache.get("missing") is None
    cache.close()


@pytest.mark.parametrize("field,value", [
    ("api_version", "v1"), ("thinking_level", "low"),
    ("max_output_tokens", 121), ("schema_version", "prediction-v3"),
    ("sdk_version", "2.34.0"),
])
def test_every_behavior_setting_changes_the_key(field: str, value: object) -> None:
    assert key() != key(**{field: value})


def test_retry_policy_respects_server_delay_and_budget() -> None:
    assert RETRYABLE_SERVER_CODES == {500, 502, 503, 504}
    assert should_retry(503, None)
    assert should_retry(429, 12)
    assert not should_retry(429, None)
    assert not should_retry(401, None)
    assert retry_delay(2, 12, 60) == 12
    with pytest.raises(RetryBudgetExceeded):
        retry_delay(2, 120, 60)


def test_pacer_uses_fake_time_without_real_sleep() -> None:
    now = [0.0]
    simulated_waits: list[float] = []

    def fake_sleep(seconds: float) -> None:
        simulated_waits.append(seconds)
        now[0] += seconds

    pacer = RequestPacer(60, clock=lambda: now[0], sleeper=fake_sleep)
    pacer.wait()
    now[0] = 0.25
    pacer.wait()

    assert simulated_waits == [pytest.approx(0.75)]
    assert now[0] == pytest.approx(1.0)
    assert pacer.last_request == pytest.approx(1.0)

What do the offline tests cover?

• test_cache_round_trip verifies persisted responses and cache misses.
• test_every_behavior_setting_changes_the_key verifies that each behavior-changing setting changes the request identity.
• test_retry_policy_respects_server_delay_and_budget verifies RETRYABLE_SERVER_CODES, quota-delay handling, permanent failures, and the bound enforced by retry_delay().
• test_pacer_uses_fake_time_without_real_sleep sends a second request before the permitted interval and verifies the simulated 0.75-second wait.

• Save tests\test_controls.py.
• Before you run the tests, predict whether any test needs the Gemini API key.
• Run the cache and retry tests:

python -m pytest tests\test_controls.py -q

You should see 8 passed without a Gemini API key or network access. The suite verifies cache invalidation, RETRYABLE_SERVER_CODES, server-directed delays, the budget enforced by retry_delay(), and a simulated pacing wait without real sleeping.

Control tests failed?

Delete .cache\test.sqlite3 if a previous manual check left the file locked, then close any Python process still using it.

Use this controls-debugging prompt if the tests remain red.

Your detector can now reuse identical responses and recover from transient failures without hiding persistent ones. Next, you’ll connect these controls to the structured evaluation engine.

Run the Evaluation Engine

The detector has validated cases, versioned prompts, and request controls. It still needs one engine that connects them without losing evidence between the API response and the saved run.

You’ll render prompts with Jinja, request schema-constrained predictions, reuse complete cache entries, and save one portable run artifact. The final command evaluates all 18 golden cases.

In this step, get ready to:

• Load and render the selected prompt.
• Request and cache structured predictions.
• Evaluate every golden case into one run artifact.

Load and render a prompt

• In the VS Code Explorer, create prompts.py inside src\regression_detector.
• Paste this prompt loader into the new file:

from pathlib import Path

import yaml
from jinja2 import Template
from pydantic import BaseModel, ConfigDict


# Validate the versioned prompt file before rendering it.
class PromptSpec(BaseModel):
    model_config = ConfigDict(extra="forbid")

    version: str
    system: str
    user_template: str


# Parse one YAML prompt into the typed prompt contract.
def load_prompt(path: Path) -> PromptSpec:
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    return PromptSpec.model_validate(data)


# Render only the fictional ticket into the user template.
def render_ticket(spec: PromptSpec, ticket_text: str) -> str:
    return Template(spec.user_template).render(ticket_text=ticket_text)

What does prompt rendering control?

• PromptSpec rejects misspelled or extra prompt fields.
• load_prompt() preserves the version stored in the YAML file.
• render_ticket() injects one fictional ticket without changing the system instruction.

• Save src\regression_detector\prompts.py.
• Render one case by running:

python -c "from pathlib import Path; from regression_detector.prompts import load_prompt, render_ticket; p=load_prompt(Path('prompts/classifier-v1.yaml')); print(render_ticket(p, 'I was charged twice'))"

You should see Classify this ticket: followed by the fictional ticket.

Ticket text did not render?

Confirm the YAML key is user_template and the placeholder is exactly {{ ticket_text }}.

Use this prompt-rendering help if the output is empty.

Request and cache structured predictions

Gemini receives the Pydantic JSON schema through the SDK. The returned text is still validated locally because syntactic JSON can contain semantically invalid values.

• In the VS Code Explorer, create api.py inside src\regression_detector.
• Paste this structured request client into the new file:

import time
from typing import Any

from google import genai
from google.genai import errors, types

from .config import Settings
from .controls import RequestPacer, retry_delay, should_retry
from .domain import Prediction
from .prompts import PromptSpec, render_ticket
from .run_models import ApiMeasurement, UsageStats


def _seconds(value: object) -> float | None:
    if not isinstance(value, str):
        return None
    text = value.strip().removesuffix("s")
    try:
        return float(text)
    except ValueError:
        return None


def _retry_after(exc: errors.APIError) -> float | None:
    # Prefer HTTP metadata, then Google's RetryInfo payload.
    response = getattr(exc, "response", None)
    headers = getattr(response, "headers", {}) if response else {}
    header_delay = _seconds(headers.get("Retry-After"))
    if header_delay is not None:
        return header_delay
    details: Any = getattr(exc, "details", {})
    error = details.get("error", details) if isinstance(details, dict) else {}
    for item in error.get("details", []):
        if isinstance(item, dict) and "retryDelay" in item:
            return _seconds(item["retryDelay"])
    return None


def _usage(response: object) -> UsageStats:
    metadata = getattr(response, "usage_metadata", None)
    if metadata is None:
        raise RuntimeError("Gemini omitted usage metadata")
    values = {
        "input_tokens": getattr(metadata, "prompt_token_count", None),
        "output_tokens": getattr(metadata, "candidates_token_count", None),
        "thought_tokens": getattr(metadata, "thoughts_token_count", 0) or 0,
        "total_tokens": getattr(metadata, "total_token_count", None),
    }
    if values["input_tokens"] is None or values["output_tokens"] is None:
        raise RuntimeError("Gemini returned incomplete usage metadata")
    if values["total_tokens"] is None:
        raise RuntimeError("Gemini omitted total token usage")
    return UsageStats.model_validate(values)


def request_prediction(
    client: genai.Client, settings: Settings, spec: PromptSpec,
    ticket_text: str, pacer: RequestPacer,
) -> tuple[Prediction, ApiMeasurement]:
    for attempt in range(1, settings.retry_attempts + 1):
        pacer.wait()
        started = time.perf_counter()
        try:
            response = client.models.generate_content(
                model=settings.model_id,
                contents=render_ticket(spec, ticket_text),
                config=types.GenerateContentConfig(
                    system_instruction=spec.system,
                    thinking_config=types.ThinkingConfig(
                        thinking_level=settings.thinking_level
                    ),
                    max_output_tokens=settings.max_output_tokens,
                    response_mime_type="application/json",
                    response_json_schema=Prediction.model_json_schema(),
                ),
            )
            candidates = response.candidates or []
            if not candidates:
                raise RuntimeError("Gemini returned no candidates")
            reason = getattr(candidates[0], "finish_reason", None)
            if getattr(reason, "value", reason) != "STOP":
                raise RuntimeError(f"Gemini stopped with finish reason {reason}")
            prediction = Prediction.model_validate_json(response.text or "")
            measurement = ApiMeasurement(
                latency_ms=(time.perf_counter() - started) * 1000,
                usage=_usage(response),
            )
            return prediction, measurement
        except errors.APIError as exc:
            delay = _retry_after(exc)
            if attempt == settings.retry_attempts or not should_retry(exc.code, delay):
                raise
            time.sleep(retry_delay(attempt, delay, settings.max_retry_wait_seconds))
    raise RuntimeError("retry loop ended unexpectedly")

How is one request made safe?

• response_json_schema constrains the response to the prediction shape.
• Prediction.model_validate_json() enforces confidence and label rules locally.
• Only transient status codes enter the bounded retry path.

• Save src\regression_detector\api.py with Ctrl+S.
• In the VS Code Explorer, create classification.py in the same folder.
• Paste this cache-aware classifier into the new file:

from importlib.metadata import version

from google import genai
from pydantic import BaseModel

from .api import request_prediction
from .cache import ResponseCache, make_cache_key
from .config import Settings
from .controls import RequestPacer
from .domain import Prediction, TicketCase
from .prompts import PromptSpec
from .run_models import ApiMeasurement, UsageStats


class CachedResponse(BaseModel):
    prediction: Prediction
    api_measurement: ApiMeasurement


def classify_case(
    client: genai.Client, settings: Settings, spec: PromptSpec,
    prompt_hash: str, case: TicketCase, cache: ResponseCache,
    pacer: RequestPacer,
) -> tuple[Prediction, ApiMeasurement, UsageStats, bool]:
    sdk_version = version("google-genai")
    key = make_cache_key(
        model_id=settings.model_id,
        prompt_hash=prompt_hash,
        ticket_text=case.text,
        api_version=settings.api_version,
        thinking_level=settings.thinking_level,
        max_output_tokens=settings.max_output_tokens,
        schema_version=settings.schema_version,
        sdk_version=sdk_version,
    )
    cached_json = cache.get(key)
    if cached_json is not None:
        cached = CachedResponse.model_validate_json(cached_json)
        return (
            cached.prediction,
            cached.api_measurement,
            UsageStats.zero(),
            True,
        )

    prediction, measurement = request_prediction(
        client, settings, spec, case.text, pacer
    )
    cached = CachedResponse(
        prediction=prediction,
        api_measurement=measurement,
    )
    cache.put(key, cached.model_dump_json())
    return prediction, measurement, measurement.usage, False

• Save src\regression_detector\classification.py.
• Check both modules compile by running:

python -m py_compile src\regression_detector\api.py src\regression_detector\classification.py

You should return to the VS Code integrated terminal with no error.

Request modules do not compile?

A compile error reports the exact file and line. Check indentation around the multiline return values and ensure every opening parenthesis has a closing partner.

Use this compile-error prompt if the modules still fail.

Evaluate the golden dataset

• In the VS Code Explorer, create evaluator.py inside src\regression_detector.
• Paste this evaluation engine into the new file:

import os
from datetime import UTC, datetime
from importlib.metadata import version
from pathlib import Path

from dotenv import load_dotenv
from google import genai
from google.genai import types

from .cache import ResponseCache
from .classification import classify_case
from .config import file_hash, load_settings
from .controls import RequestPacer
from .dataset import dataset_hash, load_cases
from .prompts import load_prompt
from .run_models import CaseResult, EvaluationRun, EvaluationSettings


def run_evaluation(prompt_path: Path, output_path: Path) -> EvaluationRun:
    load_dotenv()
    api_key = os.environ.get("GEMINI_API_KEY")
    if not api_key:
        raise RuntimeError("GEMINI_API_KEY is missing")
    settings = load_settings()
    cases = load_cases(settings.dataset_path)
    spec = load_prompt(prompt_path)
    prompt_digest = file_hash(prompt_path)
    cache = ResponseCache(settings.cache_path)
    pacer = RequestPacer(settings.requests_per_minute)
    results: list[CaseResult] = []
    http_options = types.HttpOptions(
        api_version=settings.api_version,
        retry_options=types.HttpRetryOptions(attempts=1),
    )
    try:
        with genai.Client(api_key=api_key, http_options=http_options) as client:
            for case in cases:
                prediction, measurement, run_usage, hit = classify_case(
                    client, settings, spec, prompt_digest, case, cache, pacer
                )
                results.append(CaseResult(
                    case_id=case.case_id,
                    expected_label=case.expected_label,
                    prediction=prediction,
                    correct=prediction.label == case.expected_label,
                    api_measurement=measurement,
                    run_api_usage=run_usage,
                    cache_hit=hit,
                ))
    finally:
        cache.close()

    created = datetime.now(UTC)
    run = EvaluationRun(
        run_id=f"{spec.version}-{created:%Y%m%dT%H%M%SZ}",
        created_at=created,
        model_id=settings.model_id,
        prompt_version=spec.version,
        prompt_hash=prompt_digest,
        dataset_hash=dataset_hash(cases),
        settings=EvaluationSettings(
            api_version=settings.api_version,
            sdk_version=version("google-genai"),
            thinking_level=settings.thinking_level,
            max_output_tokens=settings.max_output_tokens,
            schema_version=settings.schema_version,
        ),
        results=results,
    )
    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(run.model_dump_json(indent=2), encoding="utf-8")
    return run

• In the VS Code Explorer, create test_api.py inside tests.
• Paste these offline API tests into the new file:

from types import SimpleNamespace as NS

import pytest
from google.genai import errors

from regression_detector.api import _retry_after, _usage, request_prediction
from regression_detector.config import load_settings
from regression_detector.controls import RequestPacer
from regression_detector.prompts import PromptSpec


def test_retry_info_delay_is_extracted() -> None:
    exc = errors.ClientError(429, {
        "error": {"details": [{"retryDelay": "12s"}]}
    })
    assert _retry_after(exc) == 12


def test_missing_total_usage_fails_closed() -> None:
    response = NS(usage_metadata=NS(
        prompt_token_count=2, candidates_token_count=1,
        thoughts_token_count=0, total_token_count=None,
    ))
    with pytest.raises(RuntimeError, match="total token"):
        _usage(response)


def test_malformed_prediction_is_not_accepted() -> None:
    response = NS(candidates=[NS(finish_reason=NS(value="STOP"))], text="not json")
    client = NS(models=NS(generate_content=lambda **_: response))
    pacer = RequestPacer(60, clock=lambda: 0.0, sleeper=lambda _: None)
    spec = PromptSpec(version="v1", system="Classify.", user_template="{{ ticket_text }}")
    with pytest.raises(ValueError):
        request_prediction(client, load_settings(), spec, "fictional ticket", pacer)

• Save src\regression_detector\evaluator.py.
• Before you run the engine, predict whether the first pass reports any cache hits.
• Evaluate the baseline prompt by running:

python -c "from pathlib import Path; from regression_detector.evaluator import run_evaluation; r=run_evaluation(Path('prompts/classifier-v1.yaml'), Path('runs/baseline-candidate.json')); print(f'correct={sum(x.correct for x in r.results)}/18 cache_hits={sum(x.cache_hit for x in r.results)} run_api_tokens={sum(x.run_api_usage.total_tokens for x in r.results)}')"

Expect roughly four minutes while the first 18 live requests follow the configured pacing. You should see a correctness count, zero cache hits, positive current-run API tokens, and a new runs\baseline-candidate.json artifact.

Run the same command again. You should see cache_hits=18 and run_api_tokens=0 in seconds. Each cache hit still retains the original API latency and usage as historical evidence.

Evaluation stopped early?

A schema validation error means Gemini returned a value outside the local contract. Keep the raw error, do not cache the response, and inspect the prompt plus schema before retrying.

Use this evaluation-debugging prompt for a safe diagnosis.

You now have a structured, cached evaluation artifact for every golden case. Next, you’ll approve it as a baseline and compare a separately evaluated candidate.

Compare the Baseline and Candidate

The evaluator produced a complete run, but one run alone cannot identify a regression. You need an approved reference and a candidate produced under compatible conditions.

You’ll inspect the first run, freeze it as the baseline, evaluate classifier-v2, and compare predictions case by case. Compatibility checks prevent unrelated datasets or models from entering the same result.

In this step, get ready to:

• Review and approve the baseline run.
• Evaluate the candidate prompt independently.
• Compare accuracy and changed cases.

Review and approve the baseline

• Summarize the baseline artifact by running:

python -c "import json; r=json.load(open('runs/baseline-candidate.json')); ok=sum(x['correct'] for x in r['results']); original=sum(x['api_measurement']['usage']['total_tokens'] for x in r['results']); current=sum(x['run_api_usage']['total_tokens'] for x in r['results']); print(f'accuracy={ok/len(r[\"results\"]):.1%} correct={ok}/18 original_api_tokens={original} current_run_api_tokens={current}')"

You should see the accuracy and correct-case count for all 18 cases. original_api_tokens preserves the token evidence from the live requests, while current_run_api_tokens is zero when this artifact came from cache hits.

• Inspect every incorrect case by running:

python -c "import json; r=json.load(open('runs/baseline-candidate.json')); [print(x['case_id'], x['expected_label'], '->', x['prediction']['label']) for x in r['results'] if not x['correct']]"

The command prints only mismatches. No output means the baseline classified every golden case correctly.

• Freeze the reviewed run as runs\baseline.json by running:

Copy-Item runs\baseline-candidate.json runs\baseline.json

• Confirm the approved artifact exists by running:

Get-Item runs\baseline.json

You should see the file path, size, and modification time for baseline.json.

Baseline evidence looks wrong?

Do not approve a run with unexplained errors, a partial case list, or the wrong model. Re-run Step 7 after correcting the source issue.

Use this baseline-review prompt if the summary looks inconsistent.

Evaluate the independent candidate

The candidate uses the same model, settings, dataset, and schema. Only the versioned prompt artifact changes, which isolates prompt behavior from unrelated variables.

Before you run it, do you expect two prompts with the same instructions but different version fields to produce the same labels?

• Evaluate classifier-v2.yaml by running:

python -c "from pathlib import Path; from regression_detector.evaluator import run_evaluation; r=run_evaluation(Path('prompts/classifier-v2.yaml'), Path('runs/candidate.json')); print(f'correct={sum(x.correct for x in r.results)}/18 cache_hits={sum(x.cache_hit for x in r.results)}')"

Expect roughly four minutes because the changed prompt hash creates a separate request identity. You should see a correctness count and a new runs\candidate.json artifact.

Candidate reused every baseline response?

If every request unexpectedly reports a cache hit, compare the prompt hashes stored in baseline.json and candidate.json. They should differ because the prompt version line changed.

Use this cache-isolation prompt for a focused check.

Compare compatible runs

• In the VS Code Explorer, create compare.py inside src\regression_detector.
• Paste this comparison engine into the new file:

from pathlib import Path

from .comparison_models import CaseDelta, ComparisonSummary
from .run_models import EvaluationRun


def load_run(path: Path) -> EvaluationRun:
    # Revalidate saved evidence before comparison.
    return EvaluationRun.model_validate_json(path.read_text(encoding="utf-8"))


def compare_runs(
    baseline: EvaluationRun, candidate: EvaluationRun
) -> ComparisonSummary:
    # Refuse comparisons where any execution control changed.
    compatibility = {
        "model_id": (baseline.model_id, candidate.model_id),
        "dataset_hash": (baseline.dataset_hash, candidate.dataset_hash),
        "settings": (baseline.settings, candidate.settings),
    }
    mismatches = [name for name, pair in compatibility.items() if pair[0] != pair[1]]
    if mismatches:
        raise ValueError("incompatible runs: " + ", ".join(mismatches))

    before = {result.case_id: result for result in baseline.results}
    after = {result.case_id: result for result in candidate.results}
    if before.keys() != after.keys():
        raise ValueError("case_id set mismatch")
    changed: list[CaseDelta] = []
    for case_id, old in before.items():
        new = after[case_id]
        if old.prediction.label != new.prediction.label:
            changed.append(CaseDelta(
                case_id=case_id,
                before_label=old.prediction.label.value,
                after_label=new.prediction.label.value,
                was_correct=old.correct,
                is_correct=new.correct,
            ))
    regressions = [item for item in changed if item.was_correct and not item.is_correct]
    improvements = [item for item in changed if not item.was_correct and item.is_correct]
    before_accuracy = sum(item.correct for item in baseline.results) / len(before)
    after_accuracy = sum(item.correct for item in candidate.results) / len(after)
    return ComparisonSummary(
        baseline_run_id=baseline.run_id,
        candidate_run_id=candidate.run_id,
        accuracy_before=before_accuracy,
        accuracy_after=after_accuracy,
        accuracy_delta=after_accuracy - before_accuracy,
        regressions=regressions,
        improvements=improvements,
        changed=changed,
    )

How does the comparison stay honest?

• Model and dataset checks stop invalid comparisons before metric calculation.
• changed includes every case whose predicted label moved.
• regressions includes only cases that moved from correct to incorrect.

• Save src\regression_detector\compare.py.
• Before you compare the runs, predict whether any changed label appears.
• Compare the approved baseline with the candidate by running:

python -c "from pathlib import Path; from regression_detector.compare import compare_runs, load_run; c=compare_runs(load_run(Path('runs/baseline.json')), load_run(Path('runs/candidate.json'))); print(f'accuracy={c.accuracy_before:.1%}->{c.accuracy_after:.1%} delta={c.accuracy_delta:+.1%} changed={len(c.changed)} regressions={len(c.regressions)}')"

You should see both accuracy values, the delta, changed-case count, and regression count. Identical labels produce changed=0 regressions=0.

Runs are incompatible?

A compatibility error is deliberate. Compare the two stored model_id and dataset_hash values, then regenerate whichever artifact used the wrong input.

Use this comparison-debugging prompt if the source mismatch is unclear.

Your baseline and candidate now produce a typed, compatibility-checked comparison. Next, you’ll introduce a controlled prompt defect and watch the detector expose the regression.

Detect a Deliberate Regression

The first candidate should behave like the baseline because its instructions are equivalent. A regression detector earns trust only after it catches a controlled defect with a known cause.

You’ll deliberately corrupt the billing rule in classifier-v2.yaml, run the full dataset, and inspect the affected cases and slices. The failure happens on the main path so you can see the detector distinguish a real regression from ordinary variation.

In this step, get ready to:

• Inject a controlled billing-label defect.
• Evaluate the degraded prompt independently.
• Trace regressions to cases and slices.

Inject the controlled defect

The candidate will intentionally map billing language to the account label. This is a reversible test mutation, not an instruction you would approve for production.

• Open prompts\classifier-v2.yaml from the VS Code Explorer.
• Find this approved billing rule:

  Choose billing for charges, invoices, refunds, or payments.

• Replace it with this deliberately incorrect line:

  Choose account for charges, invoices, refunds, payments, renewals, checkout, or statements.

• Save prompts\classifier-v2.yaml.
• Confirm the mutation is present by running:

Select-String -Path prompts\classifier-v2.yaml -Pattern "Choose account for charges"

You should see the filename, line number, and incorrect account rule. That visible mutation is the known cause you expect the detector to find.

Mutation not found?

If the search returns nothing, check that you edited classifier-v2.yaml rather than the approved classifier-v1.yaml.

Use this mutation-check prompt if the wrong file changed.

Run the degraded candidate

Before you run it, which label do you expect the six billing cases to receive under the corrupted rule?

• Evaluate the mutated candidate by running:

python -c "from pathlib import Path; from regression_detector.evaluator import run_evaluation; r=run_evaluation(Path('prompts/classifier-v2.yaml'), Path('runs/degraded.json')); print(f'correct={sum(x.correct for x in r.results)}/18 cache_hits={sum(x.cache_hit for x in r.results)}')"

Expect roughly four minutes because the mutation creates a new prompt hash. The correctness count should fall if the model follows the faulty rule.

• Save the typed comparison and print its summary by running:

python -c "from pathlib import Path; from regression_detector.compare import compare_runs, load_run; c=compare_runs(load_run(Path('runs/baseline.json')), load_run(Path('runs/degraded.json'))); Path('runs/degraded-comparison.json').write_text(c.model_dump_json(indent=2)); print(f'delta={c.accuracy_delta:+.1%} changed={len(c.changed)} regressions={len(c.regressions)}')"

You should see a negative accuracy delta and at least one regression. The exact count remains evidence from your live model run, not a hard-coded expected score.

No regression detected?

If no label changes, strengthen only the controlled line by adding Always follow this mapping at its start, then run the evaluation again. Keep the approved baseline untouched.

Use this designed-obstacle prompt if the model ignores the mutation.

Trace the impact by slice

• In the VS Code Explorer, create analysis.py inside src\regression_detector.
• Paste these slice-analysis helpers into the new file:

from collections import defaultdict

from .domain import TicketCase
from .run_models import EvaluationRun


# Calculate correct and total counts for each human-defined slice.
def slice_scores(
    run: EvaluationRun, cases: list[TicketCase]
) -> dict[str, tuple[int, int]]:
    slice_by_id = {case.case_id: case.slice for case in cases}
    totals: dict[str, list[int]] = defaultdict(lambda: [0, 0])
    for result in run.results:
        name = slice_by_id[result.case_id]
        totals[name][1] += 1
        totals[name][0] += int(result.correct)
    return {name: (values[0], values[1]) for name, values in totals.items()}


# Format changed cases for terminal and CI logs.
def regression_rows(run: EvaluationRun, case_ids: set[str]) -> list[str]:
    rows: list[str] = []
    for result in run.results:
        if result.case_id in case_ids:
            rows.append(
                f"{result.case_id}: {result.expected_label} -> "
                f"{result.prediction.label.value}"
            )
    return rows

Why inspect slices and cases?

• slice_scores() separates clear, paraphrased, and ambiguous performance.
• regression_rows() turns typed run records into concise evidence rows.

• Save src\regression_detector\analysis.py.
• Print the degraded slice scores and regression rows by running:

python -c "from pathlib import Path; from regression_detector.analysis import regression_rows, slice_scores; from regression_detector.compare import compare_runs, load_run; from regression_detector.dataset import load_cases; b=load_run(Path('runs/baseline.json')); d=load_run(Path('runs/degraded.json')); c=compare_runs(b,d); print(slice_scores(d,load_cases(Path('data/golden.json')))); print(*regression_rows(d,{x.case_id for x in c.regressions}),sep='\n')"

You should see correct-versus-total counts for each slice followed by the exact case IDs that moved from correct to incorrect. The rows connect the aggregate drop to reviewable examples.

Evidence does not line up?

An empty regression list with a negative delta indicates the runs or comparison artifact do not match. Regenerate degraded-comparison.json directly from baseline.json and degraded.json.

Use this evidence-reconciliation prompt if the rows still disagree.

You have watched a known prompt defect become a measurable set of case regressions. Next, you’ll turn that evidence into PASS, WARN, and CRITICAL release policy.

Build the Regression Quality Gate

The detector found a real drop, but a list of changes does not yet tell automation what to do. Release policy must convert evidence into the same outcome every time.

You’ll define reviewed thresholds for accuracy, case regressions, token growth, and p95 latency. Then a quality gate will return PASS, WARN, or CRITICAL with deterministic exit codes.

In this step, get ready to:

• Record explicit release thresholds.
• Evaluate correctness, cost, and latency signals.
• Expose deterministic CLI exit codes.

Define the release policy

Thresholds are product decisions, not universal truths. This policy treats a five-point accuracy drop or two regressed cases as critical, while smaller drops and operational increases produce warnings.

• In the VS Code Explorer, create policy.yaml beside config.yaml.
• Paste these release thresholds into the new file:

warning_accuracy_drop: 0.01
critical_accuracy_drop: 0.05
critical_regression_count: 2
warning_token_increase: 0.25
warning_p95_latency_increase: 0.50

Why use three severities?

Accuracy and regression count protect behavior. Token and latency ratios catch candidates that remain correct but become expensive or slow.

PASS maps to exit code 0, WARN to 1, and CRITICAL to 2. Both nonzero outcomes block the default CI job until someone reviews the evidence.

• Save policy.yaml.
• Confirm the policy parses by running:

python -c "import yaml; p=yaml.safe_load(open('policy.yaml')); print(p['critical_accuracy_drop'], p['critical_regression_count'])"

You should see 0.05 2.

Policy does not parse?

YAML decimals must stay numeric. Remove quotation marks around thresholds and keep one space after each colon.

Use this policy-syntax prompt if parsing still fails.

Evaluate gate signals

• In the VS Code Explorer, create run_signals.py inside src\regression_detector.
• Paste these run-signal helpers into the new file:

import math

from .run_models import EvaluationRun


def total_tokens(run: EvaluationRun) -> int:
    # Compare original API measurements, including cache hits.
    total = sum(
        result.api_measurement.usage.total_tokens
        for result in run.results
    )
    if total <= 0:
        raise ValueError("run has no original token measurements")
    return total


def p95_latency_ms(run: EvaluationRun) -> float:
    # Use original live latencies so warm-cache speed cannot hide a slowdown.
    values = sorted(
        result.api_measurement.latency_ms
        for result in run.results
    )
    if not values:
        raise ValueError("run has no latency measurements")
    index = math.ceil(0.95 * len(values)) - 1
    return values[index]


def increase_ratio(before: float, after: float) -> float:
    if before <= 0:
        raise ValueError("baseline signal must be positive")
    if after < 0:
        raise ValueError("candidate signal cannot be negative")
    return (after - before) / before

• Save src\regression_detector\run_signals.py with Ctrl+S.
• In the VS Code Explorer, create gate.py in the same folder.
• Paste this gate implementation into the new file:

from pathlib import Path

import yaml
from pydantic import BaseModel, ConfigDict, Field, model_validator

from .comparison_models import ComparisonSummary, GateDecision, Severity
from .run_models import EvaluationRun
from .run_signals import increase_ratio, p95_latency_ms, total_tokens


class GatePolicy(BaseModel):
    model_config = ConfigDict(extra="forbid")

    warning_accuracy_drop: float = Field(ge=0, le=1)
    critical_accuracy_drop: float = Field(ge=0, le=1)
    critical_regression_count: int = Field(ge=1)
    warning_token_increase: float = Field(ge=0)
    warning_p95_latency_increase: float = Field(ge=0)

    @model_validator(mode="after")
    def validate_order(self) -> "GatePolicy":
        if self.critical_accuracy_drop < self.warning_accuracy_drop:
            raise ValueError("critical accuracy drop must be at least the warning drop")
        return self


def load_policy(path: Path = Path("policy.yaml")) -> GatePolicy:
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    return GatePolicy.model_validate(data)


def evaluate_gate(
    summary: ComparisonSummary, baseline: EvaluationRun,
    candidate: EvaluationRun, policy: GatePolicy,
) -> GateDecision:
    if summary.baseline_run_id != baseline.run_id:
        raise ValueError("summary baseline_run_id mismatch")
    if summary.candidate_run_id != candidate.run_id:
        raise ValueError("summary candidate_run_id mismatch")
    accuracy_drop = max(0.0, -summary.accuracy_delta)
    token_growth = increase_ratio(total_tokens(baseline), total_tokens(candidate))
    latency_growth = increase_ratio(
        p95_latency_ms(baseline), p95_latency_ms(candidate)
    )
    critical = (
        accuracy_drop >= policy.critical_accuracy_drop
        or len(summary.regressions) >= policy.critical_regression_count
    )
    warning = (
        accuracy_drop >= policy.warning_accuracy_drop
        or token_growth >= policy.warning_token_increase
        or latency_growth >= policy.warning_p95_latency_increase
    )
    reasons = [
        f"accuracy_drop={accuracy_drop:.1%}",
        f"regressions={len(summary.regressions)}",
        f"token_growth={token_growth:.1%}",
        f"p95_latency_growth={latency_growth:.1%}",
    ]
    if critical:
        return GateDecision(severity=Severity.CRITICAL, exit_code=2, reasons=reasons)
    if warning:
        return GateDecision(severity=Severity.WARN, exit_code=1, reasons=reasons)
    return GateDecision(severity=Severity.PASS, exit_code=0, reasons=reasons)

How does the gate decide?

• Correctness signals can produce CRITICAL because they represent behavior regressions.
• Operational growth produces WARN so a reviewer can inspect cost or latency before release.
• Every decision carries the measured reasons used to reach it.

• Save src\regression_detector\gate.py.
• Check both modules compile by running:

python -m py_compile src\regression_detector\run_signals.py src\regression_detector\gate.py

You should return to the VS Code integrated terminal without an error.

Expose the gate through a CLI

• In the VS Code Explorer, create gate_cli.py inside src\regression_detector.
• Paste this command-line gate into the new file:

from pathlib import Path

import typer

from .compare import compare_runs, load_run
from .gate import evaluate_gate, load_policy

app = typer.Typer(add_completion=False)


# Print the decision, preserve its evidence, and return its exit code.
@app.command()
def main(
    baseline_path: Path = Path("runs/baseline.json"),
    candidate_path: Path = Path("runs/degraded.json"),
    output_path: Path = Path("reports/gate.json"),
) -> None:
    baseline = load_run(baseline_path)
    candidate = load_run(candidate_path)
    summary = compare_runs(baseline, candidate)
    decision = evaluate_gate(summary, baseline, candidate, load_policy())
    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(decision.model_dump_json(indent=2), encoding="utf-8")
    typer.echo(decision.severity.value)
    for reason in decision.reasons:
        typer.echo(f"- {reason}")
    raise typer.Exit(code=decision.exit_code)


if __name__ == "__main__":
    app()

• Save src\regression_detector\gate_cli.py.
• Before you run the gate, predict which severity the controlled defect should receive.
• Run the gate against the degraded candidate:

python -m regression_detector.gate_cli
Write-Host "exit_code=$LASTEXITCODE"

You should see CRITICAL, the four measured reasons, and exit_code=2. You should also see reports\gate.json on disk.

• In the VS Code Explorer, create test_gate.py inside tests.
• Paste these gate tests into the new file:

from datetime import UTC, datetime

from regression_detector.compare import compare_runs
from regression_detector.domain import Label, Prediction
from regression_detector.gate import GatePolicy, evaluate_gate
from regression_detector.run_models import (
    ApiMeasurement, CaseResult, EvaluationRun, EvaluationSettings, UsageStats,
)


def make_run(run_id: str, label: Label, tokens: int, latency: float) -> EvaluationRun:
    usage = UsageStats(
        input_tokens=tokens - 1,
        output_tokens=1,
        thought_tokens=0,
        total_tokens=tokens,
    )
    return EvaluationRun(
        run_id=run_id,
        created_at=datetime.now(UTC),
        model_id="model",
        prompt_version=run_id,
        prompt_hash=run_id,
        dataset_hash="dataset",
        settings=EvaluationSettings(
            api_version="v1beta",
            sdk_version="2.33.0",
            thinking_level="minimal",
            max_output_tokens=120,
            schema_version="prediction-v2",
        ),
        results=[CaseResult(
            case_id="case-001",
            expected_label=Label.BILLING,
            prediction=Prediction(label=label, confidence=0.9, rationale="test"),
            correct=label == Label.BILLING,
            api_measurement=ApiMeasurement(latency_ms=latency, usage=usage),
            run_api_usage=usage,
            cache_hit=False,
        )],
    )


def policy() -> GatePolicy:
    return GatePolicy(
        warning_accuracy_drop=0.01,
        critical_accuracy_drop=0.5,
        critical_regression_count=1,
        warning_token_increase=0.25,
        warning_p95_latency_increase=0.5,
    )


def decision(baseline: EvaluationRun, candidate: EvaluationRun):
    return evaluate_gate(compare_runs(baseline, candidate), baseline, candidate, policy())


def test_pass_warn_and_critical_paths() -> None:
    baseline = make_run("base", Label.BILLING, 10, 100)
    assert decision(baseline, make_run("pass", Label.BILLING, 10, 100)).exit_code == 0
    assert decision(baseline, make_run("warn", Label.BILLING, 13, 100)).exit_code == 1
    assert decision(baseline, make_run("critical", Label.TECHNICAL, 10, 100)).exit_code == 2


def test_cached_run_uses_original_measurement() -> None:
    baseline = make_run("base", Label.BILLING, 10, 100)
    candidate = make_run("cached", Label.BILLING, 13, 100)
    candidate.results[0].cache_hit = True
    candidate.results[0].run_api_usage = UsageStats.zero()

    assert candidate.results[0].api_measurement.usage.total_tokens == 13
    assert candidate.results[0].run_api_usage == UsageStats.zero()
    assert decision(baseline, candidate).exit_code == 1

• Save tests\test_gate.py.
• Run the policy test:

python -m pytest tests\test_gate.py -q

You should see both gate tests pass. The suite covers PASS, WARN, CRITICAL, and a cached run whose current API usage is zero while its original measurement still drives policy.

Gate result is inconsistent?

If the CLI severity and test disagree, delete reports\gate.json and confirm both commands load runs\degraded.json.

Use this gate-debugging prompt to reconcile the evidence.

Your evidence now drives a deterministic release decision. Next, you’ll generate a readable report and run the same blocking command in GitHub Actions.

Generate Reports and Block CI

The gate can block a release, but reviewers also need a readable explanation and a repeatable automation path. Terminal output alone is easy to lose and hard to discuss in a pull request.

You’ll render an HTML evidence report, preserve offline fixtures, and run the gate in GitHub Actions. You’ll watch the designed defect fail before restoring the prompt and proving the workflow passes.

In this step, get ready to:

• Generate a human-readable regression report.
• Run the degraded fixture in GitHub Actions.
• Restore the prompt and turn the workflow green.

Generate the evidence report

• Create the report and fixture folders by running:

New-Item -ItemType Directory -Force templates, tests\fixtures, .github\workflows

• In the VS Code Explorer, create report.html inside templates.
• Paste this report template into the new file:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>LLM Regression Report</title>
  <style>
    body { font-family: Segoe UI, sans-serif; margin: 2rem; color: #172033; }
    .CRITICAL { color: #b42318; }
    .WARN { color: #b54708; }
    .PASS { color: #067647; }
    table { border-collapse: collapse; width: 100%; }
    th, td { border: 1px solid #d0d5dd; padding: .6rem; text-align: left; }
    code { background: #f2f4f7; padding: .1rem .3rem; }
  </style>
</head>
<body>
  <h1>LLM Regression Report</h1>
  <h2 class="{{ decision.severity.value }}">{{ decision.severity.value }}</h2>
  <p>Accuracy: {{ "%.1f%%"|format(summary.accuracy_before * 100) }} to {{ "%.1f%%"|format(summary.accuracy_after * 100) }}</p>
  <p>Changed cases: {{ summary.changed|length }}. Regressions: {{ summary.regressions|length }}.</p>
  <h2>Gate evidence</h2>
  <ul>{% for reason in decision.reasons %}<li><code>{{ reason }}</code></li>{% endfor %}</ul>
  <h2>Changed cases</h2>
  <table><tr><th>Case</th><th>Before</th><th>After</th><th>Regressed</th></tr>
  {% for item in summary.changed %}<tr><td>{{ item.case_id }}</td><td>{{ item.before_label }}</td><td>{{ item.after_label }}</td><td>{{ item.was_correct and not item.is_correct }}</td></tr>{% endfor %}
  </table>
</body>
</html>

• Save templates\report.html with Ctrl+S.
• In the VS Code Explorer, create report.py inside src\regression_detector.
• Paste this report renderer into the new file:

from pathlib import Path

from jinja2 import Environment, FileSystemLoader, select_autoescape

from .compare import compare_runs, load_run
from .gate import evaluate_gate, load_policy


# Render typed comparison evidence into a portable HTML report.
def render_report(
    baseline_path: Path,
    candidate_path: Path,
    output_path: Path,
) -> Path:
    baseline = load_run(baseline_path)
    candidate = load_run(candidate_path)
    summary = compare_runs(baseline, candidate)
    decision = evaluate_gate(summary, baseline, candidate, load_policy())
    environment = Environment(
        loader=FileSystemLoader("templates"),
        autoescape=select_autoescape(["html"]),
    )
    html = environment.get_template("report.html").render(
        summary=summary,
        decision=decision,
    )
    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(html, encoding="utf-8")
    return output_path

Why generate the report from artifacts?

• Autoescaping keeps model-generated rationale from becoming executable markup.
• The report reads the same typed artifacts used by the CLI gate.
• The changed-case table lets a reviewer trace aggregate metrics to examples.

• Save src\regression_detector\report.py.
• Render the degraded report by running:

python -c "from pathlib import Path; from regression_detector.report import render_report; print(render_report(Path('runs/baseline.json'), Path('runs/degraded.json'), Path('reports/regression-report.html')))"
Start-Process reports\regression-report.html

Your browser should show a CRITICAL heading, the accuracy change, gate reasons, and one row per changed case.

Report is blank or incomplete?

If the browser opens a blank page, confirm reports\regression-report.html has a nonzero file size and run the rendering command from the folder containing templates.

Use this report-debugging prompt if the page remains incomplete.

Prove CI blocks the defect

• Copy the approved and degraded synthetic artifacts into test fixtures by running:

Copy-Item runs\baseline.json tests\fixtures\baseline.json
Copy-Item runs\degraded.json tests\fixtures\degraded.json
Copy-Item runs\degraded.json tests\fixtures\candidate.json

• In the VS Code Explorer, expand .github\workflows.
• Create regression-gate.yml inside that folder.
• Paste this GitHub Actions workflow into the new file:

name: regression-gate

on:
  push:
  pull_request:

jobs:
  evaluate:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - name: Install dependencies
        run: python -m pip install -e ".[dev]"
      - name: Prepare local run artifacts
        shell: pwsh
        run: |
          New-Item -ItemType Directory -Force runs, reports
          Copy-Item tests/fixtures/baseline.json runs/baseline.json
          Copy-Item tests/fixtures/degraded.json runs/degraded.json
      - name: Run offline tests
        run: python -m pytest -q
      - name: Enforce the regression gate
        run: python -m regression_detector.gate_cli --baseline-path tests/fixtures/baseline.json --candidate-path tests/fixtures/candidate.json

Why use committed fixtures in CI?

The workflow never calls Gemini or needs an API key. It replays committed, typed evidence so release policy remains deterministic.

The first push is supposed to fail because candidate.json contains the controlled regression. That red check proves the nonzero exit code reaches CI.

• Save .github\workflows\regression-gate.yml with Ctrl+S.
• Open the new GitHub repository page.
• Sign in if GitHub prompts you.
• Enter llm-regression-detector in the Repository name field.
• Select Private.
• Leave every initialization option unchecked.
• Click Create repository.
• Record the repository’s HTTPS URL as https://github.com/your-user/llm-regression-detector.git.

The first push may open Windows Credential Manager or a browser sign-in. Complete that GitHub authentication flow, and never paste a password or token into the project files.

• Commit the project and push the designed failure by running:

git add .
git commit -m "Build LLM regression gate"
git branch -M main
git remote add origin [[YOUR_REPO_URL="https://github.com/your-user/llm-regression-detector.git"]]
git push -u origin main

• Open your repository on GitHub.
• Click the Actions tab.
• Open the newest regression-gate run.
• Confirm Enforce the regression gate is red and its log shows CRITICAL.

That failure is intentional. Your repository now proves a release-blocking model regression cannot slip through as a green build.

Push or workflow missing?

If Git rejects the push, confirm the remote URL with git remote -v and finish the browser authentication prompt. If Actions does not start, confirm the workflow path is exactly .github\workflows\regression-gate.yml.

Use this CI-setup prompt for safe help.

Restore the prompt and turn CI green

• Open prompts\classifier-v2.yaml from the VS Code Explorer.
• Find this deliberately incorrect rule:

  Choose account for charges, invoices, refunds, payments, renewals, checkout, or statements.

• Replace it with the approved billing rule:

  Choose billing for charges, invoices, refunds, or payments.

• Save prompts\classifier-v2.yaml.
• Re-evaluate the restored prompt by running:

python -c "from pathlib import Path; from regression_detector.evaluator import run_evaluation; r=run_evaluation(Path('prompts/classifier-v2.yaml'), Path('runs/fixed.json')); print(f'correct={sum(x.correct for x in r.results)}/18 cache_hits={sum(x.cache_hit for x in r.results)}')"

You should see cache_hits=18 because this exact approved candidate ran in Step 8. The restored prompt reuses its validated responses instead of spending more API quota.

• Replace the degraded CI candidate fixture with the restored runs\fixed.json artifact by running:

Copy-Item runs\fixed.json tests\fixtures\candidate.json -Force
git add prompts\classifier-v2.yaml tests\fixtures\candidate.json
git commit -m "Fix billing prompt regression"
git push

• Open the newest regression-gate run in the repository’s Actions tab.
• Confirm every workflow step is green.

That’s the full control loop working. The bad prompt was blocked, the restored prompt reused safe cache entries, and the approved offline fixture makes the same CI policy pass deterministically.

Fixed workflow still red?

If the fixed run stays red, compare the candidate fixture’s prompt hash with runs\fixed.json and inspect the gate reasons in the Actions log.

Use this green-build prompt to find stale evidence.

Your detector now produces human-readable evidence and enforces it in CI. Next, you’ll harden the project with offline contract tests and one final clean verification.

Harden and Verify the Detector

The CI loop blocks the defect and accepts the restored prompt. The final hardening task is to keep both outcomes reproducible as the project evolves.

You’ll preserve separate failing and passing fixtures, test the compatibility guards, and run every offline check from a clean command. The finished project can verify policy without network access or credentials.

In this step, get ready to:

• Preserve permanent failing and passing fixtures.
• Test schema and compatibility failure paths.
• Run the complete offline verification suite.

Preserve both gate outcomes

A fixed candidate should not erase the evidence that proved the gate can fail. Separate fixtures keep the designed regression available for tests while CI evaluates the current candidate.

• Copy the degraded run into its permanent fixture by running:

Copy-Item runs\degraded.json tests\fixtures\degraded.json -Force

• Copy the restored runs\fixed.json artifact into the permanent passing candidate fixture by running:

Copy-Item runs\fixed.json tests\fixtures\candidate.json -Force

• Prove the preserved degraded fixture still blocks release by running:

python -m regression_detector.gate_cli --baseline-path tests\fixtures\baseline.json --candidate-path tests\fixtures\degraded.json
Write-Host "exit_code=$LASTEXITCODE"

• Run the synthetic PASS, WARN, CRITICAL, and cache-evidence tests:

python -m pytest tests\test_gate.py -q

You should first see CRITICAL with exit_code=2. You should then see both synthetic gate tests pass without reading live run files.

Degraded fixture no longer fails?

If the degraded fixture reports PASS, compare tests\fixtures\degraded.json with runs\degraded.json and confirm their prompt hashes match.

Use this fixture-integrity prompt if the failure case was overwritten.

Test contract failure paths

• In the VS Code Explorer, create test_regression_contracts.py inside tests.
• Paste these regression-contract tests into the new file:

from pathlib import Path

import pytest
from pydantic import ValidationError

from regression_detector.compare import compare_runs, load_run
from regression_detector.gate import evaluate_gate, load_policy


def runs():
    return (
        load_run(Path("tests/fixtures/baseline.json")),
        load_run(Path("tests/fixtures/candidate.json")),
    )


def test_approved_fixture_passes() -> None:
    baseline, candidate = runs()
    summary = compare_runs(baseline, candidate)
    assert evaluate_gate(summary, baseline, candidate, load_policy()).exit_code == 0


@pytest.mark.parametrize("field", ["dataset_hash", "settings"])
def test_incompatible_runs_are_rejected(field: str) -> None:
    baseline, candidate = runs()
    value = "different" if field == "dataset_hash" else candidate.settings.model_copy(update={"schema_version": "v3"})
    candidate = candidate.model_copy(update={field: value})
    with pytest.raises(ValueError, match="mismatch"):
        compare_runs(baseline, candidate)


def test_duplicate_case_ids_are_rejected() -> None:
    _, candidate = runs()
    data = candidate.model_dump()
    data["results"].append(data["results"][0])
    with pytest.raises(ValidationError, match="duplicate"):
        candidate.__class__.model_validate(data)

Which comparison boundaries do these tests protect?

• test_approved_fixture_passes protects the green release path.
• test_incompatible_runs_are_rejected blocks dataset and runtime-setting mismatches.
• test_duplicate_case_ids_are_rejected blocks ambiguous case matching.

• In the VS Code Explorer, create test_validation_contracts.py inside tests.
• Paste these strict-validation tests into the new file:

import pytest
from pydantic import ValidationError

from regression_detector.classification import CachedResponse
from regression_detector.config import Settings, load_settings
from regression_detector.domain import Prediction


def test_unknown_config_key_is_rejected() -> None:
    data = load_settings().model_dump()
    data["temperature"] = 0
    with pytest.raises(ValidationError, match="temperature"):
        Settings.model_validate(data)


def test_stale_cache_payload_is_rejected() -> None:
    stale = {
        "prediction": {
            "label": "billing", "confidence": 0.9, "rationale": "test"
        }
    }
    with pytest.raises(ValidationError, match="api_measurement"):
        CachedResponse.model_validate(stale)


def test_invalid_prediction_is_rejected() -> None:
    with pytest.raises(ValidationError):
        Prediction(label="billing", confidence=1.5, rationale="Outside range")

Which validation boundaries do these tests protect?

• Unknown configuration keys fail instead of becoming silent no-ops.
• Cache entries without their original API measurement fail closed.
• Structured predictions still enforce semantic confidence limits.

• Save tests\test_regression_contracts.py and tests\test_validation_contracts.py.
• Run both contract suites:

python -m pytest tests\test_regression_contracts.py tests\test_validation_contracts.py -q

You should see 7 passed. The suite covers green policy, compatibility mismatches, duplicate IDs, unknown configuration, stale cache metadata, and invalid model output.

Contract tests failed?

If a compatibility test does not raise an error, confirm compare_runs() checks the dataset hash, full runtime settings, and identical case sets before calculating metrics. Duplicate case IDs should fail earlier when EvaluationRun validates the fixture.

Use this contract-test prompt for a focused fix.

Run the final offline verification

Before you run the suite, do you expect the tests to prove both a CRITICAL degraded path and a PASS fixed path in the same offline run?

• Compile every application module by running:

python -m compileall -q src

You should return to the VS Code integrated terminal with no output.

• Run the complete offline test suite:

python -m pytest -q

You should see the complete suite pass without a Gemini API key or network access. The exact test count can grow as you add more contract cases.

• Run the release gate against the current CI fixtures:

python -m regression_detector.gate_cli --baseline-path tests\fixtures\baseline.json --candidate-path tests\fixtures\candidate.json
Write-Host "exit_code=$LASTEXITCODE"

You should see PASS and exit_code=0.

• Render the final passing report by running:

python -c "from pathlib import Path; from regression_detector.report import render_report; render_report(Path('tests/fixtures/baseline.json'), Path('tests/fixtures/candidate.json'), Path('reports/final-report.html'))"
Start-Process reports\final-report.html

Your browser should show a PASS heading with the final accuracy and evidence reasons.

• Commit the permanent fixtures and hardening tests by running:

git add tests\fixtures\degraded.json tests\fixtures\candidate.json tests\test_gate.py tests\test_regression_contracts.py tests\test_validation_contracts.py
git commit -m "Harden regression contract tests"
git push

• Open the newest workflow run in the repository’s Actions tab.
• Confirm the complete offline suite and regression gate are green.

You’ve closed the loop. The project now preserves the approved baseline, a known regression, the current candidate, deterministic policy, and the evidence needed to reproduce every gate outcome.

Final verification failed?

Run each final command from the folder containing pyproject.toml and confirm (.venv) appears in the prompt. A missing fixture error means the copy or Git add step did not include all three JSON files.

Use this final-verification prompt if any check remains red.

Secret mission

Build a Local Trend Dashboard

Build a local trend dashboard that compares the approved, degraded, and fixed run artifacts without making new Gemini requests.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep the project ready, pause its GitHub workflow, or delete its local and remote resources. Nothing runs continuously, but future live evaluations can consume Gemini quota or incur account-specific API charges.

Resources you used:

• The local llm-regression-detector folder containing source code, the virtual environment, cache, reports, and synthetic fixtures.
• The private GitHub repository containing the regression-gate workflow and project history.
• The Gemini API key managed through Google AI Studio.

Keep everything running

No action is needed. Choose this if you are still testing prompts, reviewing model changes, or extending the release policy.

• Keep the local llm-regression-detector folder for offline tests and reports.
• Keep the private GitHub repository for CI demonstrations.
• Keep the Gemini API key only while you plan to run more live evaluations.

Pause - I'll come back to this later

Stop local activity and disable automatic workflow runs while keeping every file ready for another session.

• Press Ctrl+C in the VS Code integrated terminal to stop any running evaluator or test command.
• Close VS Code.
• Open the repository’s Actions tab on GitHub.
• Select the regression-gate workflow.
• Open the workflow menu.
• Select Disable workflow.
• Leave the local folder, private repository, and API key in place.

Re-enable the workflow from the same page when you return. No model request runs until you start one locally.

Delete - I don't want to use this again

Deleting these resources is permanent. Save any report or fixture you want before starting.

• Open the project repository on GitHub.
• Select the repository’s Settings tab.
• Select General in the left sidebar.
• Scroll to Danger Zone.
• Click Delete this repository.
• Follow GitHub’s confirmation prompts.

You should no longer be able to open the repository or its workflow history.

• Open the Google AI Studio API keys page.
• Locate the key created for this project.
• Open its actions menu.
• Select the delete option.
• Confirm the key deletion.

The key should disappear from the list and can no longer authorize Gemini requests.

Removing the local folder looks final because it is. Confirm your committed fixtures are safely stored elsewhere before running the command.

• Close VS Code.
• Close every terminal using the llm-regression-detector folder.
• Press the Windows key to open search.
• Type PowerShell.
• Press Enter to open PowerShell.
• Remove the project folder from your Desktop by running:

Set-Location $HOME\Desktop
Remove-Item -Recurse -Force llm-regression-detector
Test-Path llm-regression-detector

You should see False, which confirms the local folder is gone.

Nice Work!

Nice Work!

You did it! You built an LLM regression detector that traces prompt defects to individual cases and blocks unsafe changes in CI.

You've learned how to:

• Build an 18-case golden dataset with approved labels, stable IDs, behavioral slices, and a deterministic fingerprint.
• Run reproducible Gemini evaluations with structured outputs, semantic validation, bounded retries, content-addressed caching, latency, and token evidence.
• Enforce a reviewable regression verdict with compatibility checks, PASS/WARN/CRITICAL policy, HTML evidence, offline fixtures, and GitHub Actions.
• Secret Mission: Build a local trend dashboard that filters the approved, degraded, and fixed run artifacts without new API calls.

Ready to quiz yourself?