Build an Agent Evaluation Harness

Design a versioned eval schema, 25-case dataset, and Typer CLI from scratch.

Introduction

30 Second Summary

Every AI agent needs a way to prove it actually works, and that proof starts with a structured set of test cases and a tool to validate them.

In this project, you will build an evaluation harness scaffold from scratch: a versioned Pydantic v2 schema for eval cases, a 25-case JSONL dataset targeting a hello-world MCP server, and a Typer CLI that loads, validates, and summarizes the dataset.

What You'll Build

A self-contained evaluation harness scaffold inside your agentic-systems-engineering repo, with a publication-grade eval schema, a hand-written dataset, and a CLI to work with it.

By the end of this project, you'll have:

  • A versioned Pydantic v2 eval case schema with built-in provenance tracking that validates 25 hand-written cases cleanly.
  • A working Typer CLI (eval-harness load) with --validate and --summary flags producing output in under 500ms.
  • ADR-006 documenting the schema as a one-way-door decision, committed alongside the code.
  • A lint subcommand that catches duplicate IDs, missing tags, and schema version drift before they reach the eval runner.

Are there any prerequisites?

You should be comfortable with Python 3.12+, Pydantic, JSONL, and pytest. You should also understand eval concepts like accuracy, pass@k, and F1. Familiarity with MCP and the series-01 hello-world MCP server is expected.

Before We Start

Before We Start...

Scaffold the Project Structure

Before you write a single line of schema or CLI code, the evaluation harness needs a clean project structure.

This step creates the full directory tree inside your agentic-systems-engineering repo, wires up a pyproject.toml with Pydantic and Typer as dependencies, and lays down the documentation stubs that every serious engineering project needs from day one.

In This Step, Get Ready To:
  • Create the full directory tree and placeholder files for the evaluation harness.
  • Configure pyproject.toml with dependencies and a CLI entry point.
  • Write the README, data-handling declaration, and measurement stubs.
Create the Directory Tree

The evaluation harness lives inside systems/02-agent-evaluation-harness/ in your repo.

This directory contains five subdirectories: src/eval_harness/ for the Python package, datasets/ for JSONL eval data, tests/ for pytest tests, docs/adr/ for architecture decision records, and measurements/ for tracking dataset growth over iterations.

  • Open your terminal and navigate to the root of your agentic-systems-engineering repo.
  • Run the following commands to create every directory and placeholder file.
mkdir -p systems/02-agent-evaluation-harness/{src/eval_harness,datasets,tests,docs/adr,measurements}
touch systems/02-agent-evaluation-harness/src/eval_harness/__init__.py
touch systems/02-agent-evaluation-harness/src/eval_harness/schema.py
touch systems/02-agent-evaluation-harness/src/eval_harness/cli.py
touch systems/02-agent-evaluation-harness/datasets/.gitkeep
touch systems/02-agent-evaluation-harness/tests/__init__.py
touch systems/02-agent-evaluation-harness/tests/test_schema.py
touch systems/02-agent-evaluation-harness/README.md
touch systems/02-agent-evaluation-harness/data-handling.md
touch systems/02-agent-evaluation-harness/measurements/dataset-growth.md

What do these commands do?

The mkdir -p command creates the entire nested directory tree in one shot. The -p flag means it won't error if directories already exist.

The touch commands create empty placeholder files. The __init__.py files mark directories as Python packages. The .gitkeep file ensures Git tracks the otherwise-empty datasets/ directory.

  • Verify the structure by running find systems/02-agent-evaluation-harness -type f from the repo root.

You should see all 10 files listed across the directory tree.

Set Up Dependencies and pyproject.toml

The pyproject.toml file is the single source of truth for your project's metadata, dependencies, and build configuration. It also defines the CLI entry point that maps the eval-harness command to your Typer app.

  • Open systems/02-agent-evaluation-harness/pyproject.toml in your editor.
  • Paste the following configuration.
[project]
name = "eval-harness"
version = "0.1.0"
description = "Agent evaluation harness - schema, dataset, and CLI"
requires-python = ">=3.12"
dependencies = [
    "pydantic>=2.12",
    "typer>=0.24",
]

[project.scripts]
eval-harness = "eval_harness.cli:app"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/eval_harness"]

[project.optional-dependencies]
dev = ["pytest>=8.0"]

What does this configuration do?

  • The [project.scripts] section maps the eval-harness CLI command to the app object in eval_harness.cli. This is how Typer exposes subcommands like eval-harness load.
  • The [tool.hatch.build.targets.wheel] section tells Hatchling (the build backend) to find your package inside the src/ directory.
  • The [project.optional-dependencies] section keeps pytest as a dev-only dependency, separate from production dependencies.

Now install the project in editable mode so that changes to your source code take effect immediately without reinstalling.

  • Navigate into the project directory and install with the dev extras.
cd systems/02-agent-evaluation-harness
pip install -e ".[dev]"

What does editable mode do?

The -e flag installs the package as a symlink to your source code. When you edit schema.py or cli.py, the eval-harness command picks up changes automatically. The ".[dev]" part installs the optional dev dependencies (pytest) alongside the core ones.

  • Verify the installation by running eval-harness --help.

You should see Typer's auto-generated help output. It will show an error about missing commands for now, which is expected since cli.py is still empty.

Command not found?

If your terminal says eval-harness: command not found, the entry point script was not added to your PATH. This usually happens when pip installs to a user-level directory that is not on PATH, or when you are not inside a virtual environment.

Try creating and activating a virtual environment first, then re-run pip install -e ".[dev]". If you are still stuck, help me debug why eval-harness is not found after pip install -e.

Write the README, Data-Handling Declaration, and Stubs

Every project in this series carries three documentation files from the start: a README for orientation, a data-handling declaration for trust, and a measurements stub for tracking growth over iterations.

  • Open README.md and add the following content.
# Agent Evaluation Harness (Iteration 01)

A self-contained evaluation harness for the agentic-systems-engineering series.
This iteration delivers a versioned Pydantic v2 schema for eval cases, a 25-case
JSONL dataset targeting the hello-world MCP server, and a Typer CLI to load,
validate, and summarize the dataset.

See [SERIES.md](../../SERIES.md) for the full series roadmap.

## Quick Start

```bash
pip install -e ".[dev]"
eval-harness load datasets/hello-world-v0.jsonl --validate --summary
```
  • Open data-handling.md and add the following content.
# Data Handling Declaration

**Classification**: Class 0 - Public Synthetic Data

All data in this system is public synthetic data. It is safe to publish, contains
no PII, and includes no proprietary content. Every eval case is hand-written or
programmatically generated for testing purposes only.
  • Open measurements/dataset-growth.md and add the following content.
# Dataset Growth Tracking

This file tracks the size and composition of eval datasets across iterations.

| Iteration | Dataset File | Total Cases | Easy | Medium | Hard |
|-----------|-------------|-------------|------|--------|------|
| 01        | hello-world-v0.jsonl | 25 | 10 | 8 | 7 |

Why declare data handling on day one?

The data-handling.md file is a trust signal. It tells anyone who opens this repo exactly what kind of data lives here and whether it is safe to use. Class 0 means everything is synthetic and publishable. This matters when your eval datasets eventually grow to include real user interactions or proprietary model outputs in later iterations.

Your project scaffold is complete. You have a clean directory tree, a working pyproject.toml with the CLI entry point wired up, and documentation stubs in place. Next up, you will define the Pydantic schema that gives your eval cases their structure and validation rules.

Define the Eval Case Schema

Your project scaffold is in place. Now you need to define the data contract that every eval case in your pipeline will conform to. This is the most consequential decision in the entire harness because once datasets are published against this schema, changing it requires a new version and migration tooling. In this step, you'll build the Pydantic v2 models that make your eval cases publication-grade: self-describing, immutable, and provenance-tracked.

In This Step, Get Ready To:
  • Build Provenance and EvalCase Pydantic models with schema versioning and immutability.
  • Add a cross-field model validator that enforces consistency between output type and output value.
Build the Provenance and EvalCase Models

A publication-grade eval schema is one you can ship to another team, publish in a paper, or open-source without ambiguity. Three properties make it publication-grade.

  • Every case carries its own provenance, so you know who wrote it and how it was generated.
  • The schema is versioned in every record, so a reader can always determine which contract a case conforms to.
  • The expected output type is explicit (exact match, regex, JSONPath), so evaluation logic never has to guess what "correct" means.

You'll encode all of this into two Pydantic v2 models inside src/eval_harness/schema.py.

  • Open src/eval_harness/schema.py and add the following code.
from __future__ import annotations

from typing import Literal

from pydantic import BaseModel, ConfigDict, Field, model_validator

# Type aliases using Literal for compile-time-like validation
SourceType = Literal["hand_written", "synthetic", "adversarial"]
Difficulty = Literal["easy", "medium", "hard"]


class Provenance(BaseModel):
    """Tracks who created an eval case and how it was generated."""

    author: str
    source_type: SourceType


class EvalCase(BaseModel):
    """A single evaluation case with provenance and schema versioning."""

    model_config = ConfigDict(frozen=True)

    schema_version: Literal["v0"]
    case_id: str = Field(pattern=r"^[a-z0-9][a-z0-9_-]{2,63}$")
    prompt: str = Field(min_length=1)
    tool_name: str
    expected_output_type: Literal["exact", "regex", "jsonpath"]
    expected_output: str | None = None
    difficulty: Difficulty
    provenance: Provenance
    version: int = Field(ge=1)
    tags: list[str] = Field(default_factory=list)

What does this code do?

This defines two Pydantic models that form the core data contract for your eval harness.

  • Provenance records who authored each case and whether it was hand-written, synthetic, or adversarial.
  • EvalCase is the full eval record. The schema_version field is locked to "v0" using a Literal type, so Pydantic rejects any record that claims a different version.
  • Field(pattern=...) on case_id enforces URL-safe, filesystem-safe identifiers at the schema level.
  • ConfigDict(frozen=True) makes every EvalCase instance immutable after creation. Eval cases should never be mutated during processing.

Why Literal Types Instead of Enums?

Using Literal types for schema_version, difficulty, and expected_output_type is idiomatic Pydantic v2. Literals give you the same validation guarantees as enums (Pydantic raises a literal_error on mismatch) without the serialization overhead. Your JSONL records stay as plain strings.

Add a Model-Level Validator for Output Consistency

The schema has a cross-field dependency that field-level validation alone cannot catch. If expected_output_type is "regex", then expected_output must contain the regex pattern. A regex match with no pattern is meaningless. The same applies to "jsonpath". For "exact", expected_output may be None, meaning "any output is acceptable, we only check the tool was called."

  • Add the following @model_validator method to the EvalCase class, below the field definitions.
    @model_validator(mode="after")
    def _check_expected_output(self) -> EvalCase:
        """Ensure regex and jsonpath types have a non-None expected_output."""
        if self.expected_output_type in ("regex", "jsonpath") and self.expected_output is None:
            raise ValueError(
                f"expected_output must not be None when "
                f"expected_output_type is '{self.expected_output_type}'"
            )
        return self

What does this validator do?

The @model_validator(mode="after") decorator runs after all individual field validators have passed. This gives you access to the fully constructed model, so you can check relationships between fields. If someone tries to create a regex eval case without providing a regex pattern, Pydantic raises a ValidationError before the object ever exists.

Your complete schema.py should now contain both models and the validator. Here is a chance to compare your file against the full solution.

I'll try it first

Combine the two code blocks above into a single file. Make sure the @model_validator method is indented inside the EvalCase class body.

Let me check my work

from __future__ import annotations

from typing import Literal

from pydantic import BaseModel, ConfigDict, Field, model_validator

# Type aliases for constrained string values
SourceType = Literal["hand_written", "synthetic", "adversarial"]
Difficulty = Literal["easy", "medium", "hard"]


class Provenance(BaseModel):
    """Tracks who created an eval case and how it was generated."""

    author: str
    source_type: SourceType


class EvalCase(BaseModel):
    """A single evaluation case with provenance and schema versioning."""

    model_config = ConfigDict(frozen=True)

    # Schema version locked to v0 for this iteration
    schema_version: Literal["v0"]

    # Case identifier: lowercase alphanumeric, 3-64 chars, hyphens/underscores allowed
    case_id: str = Field(pattern=r"^[a-z0-9][a-z0-9_-]{2,63}$")

    # The prompt sent to the agent
    prompt: str = Field(min_length=1)

    # Which tool the agent should call
    tool_name: str

    # How to evaluate the output: exact match, regex, or JSONPath
    expected_output_type: Literal["exact", "regex", "jsonpath"]
    expected_output: str | None = None

    # Difficulty tier for reporting and filtering
    difficulty: Difficulty

    # Who wrote this case and how
    provenance: Provenance

    # Case version, starts at 1, increments on edits
    version: int = Field(ge=1)

    # Freeform tags for filtering and grouping
    tags: list[str] = Field(default_factory=list)

    @model_validator(mode="after")
    def _check_expected_output(self) -> EvalCase:
        """Ensure regex and jsonpath types have a non-None expected_output."""
        if self.expected_output_type in ("regex", "jsonpath") and self.expected_output is None:
            raise ValueError(
                f"expected_output must not be None when "
                f"expected_output_type is '{self.expected_output_type}'"
            )
        return self

Pro tip: version the schema on day one

Schema versioning is cheap to add on day one and expensive to retrofit. The schema_version field in every JSONL record means you can evolve the schema (add fields, change semantics) in v1 while still loading v0 records. Without it, you end up with ambiguity about whether a record was written before or after a schema change. That ambiguity poisons your eval results.

Version the schema before you have two records, not after you have two thousand.

You can quickly verify the schema works by running a smoke test from your project directory.

  • Run the following as a quick script to confirm a valid case parses and an invalid one is rejected.
from eval_harness.schema import EvalCase, Provenance

# This should succeed
case = EvalCase(
    schema_version="v0",
    case_id="test-case-001",
    prompt="Greet Alice",
    tool_name="hello_world",
    expected_output_type="exact",
    expected_output="Hello, Alice!",
    difficulty="easy",
    provenance=Provenance(author="you", source_type="hand_written"),
    version=1,
    tags=["greeting"],
)
print(f"Created: {case.case_id}")

# This should raise ValidationError (regex with no pattern)
try:
    bad_case = EvalCase(
        schema_version="v0",
        case_id="bad-case-001",
        prompt="Greet Bob",
        tool_name="hello_world",
        expected_output_type="regex",
        expected_output=None,
        difficulty="easy",
        provenance=Provenance(author="you", source_type="hand_written"),
        version=1,
    )
except Exception as e:
    print(f"Caught expected error: {e}")

What should I see?

The first case should print Created: test-case-001. The second case should raise a ValidationError with a message about expected_output being None when the output type is "regex". If both behave as expected, your schema is working correctly.

Seeing an ImportError?

If Python cannot find eval_harness.schema, make sure you ran pip install -e ".[dev]" from the systems/02-agent-evaluation-harness/ directory in Step 1. The editable install is what makes the src/eval_harness package importable. Help me debug this import error.

Your eval case schema is locked down and validated. Next up, you'll write 25 hand-crafted eval cases against this schema and load them into a JSONL dataset.

Write the 25-Case Dataset

Your schema is locked and validated. Now you need data to feed it. In this step, you'll hand-write 25 eval cases targeting the hello-world MCP server from series 01, covering a deliberate mix of difficulty tiers and output types. A well-designed dataset isn't random. It's structured to expose gaps in your eval runner before that runner even exists.

In This Step, Get Ready To:
  • Design a case distribution across easy, medium, and hard tiers.
  • Write all 25 cases as a JSONL file that validates against the EvalCase schema.
  • Smoke-test the dataset by parsing every line through Pydantic before the CLI exists.
Design the Case Distribution

Before writing a single JSON line, you need a plan. A good eval dataset isn't 25 copies of the same test with different names. It's a deliberate spread across difficulty levels and output verification strategies.

Here is the distribution you'll target:

  • Easy (10 cases): Basic greeting tool calls. Verify the tool returns a greeting string. Mix of exact and regex output types.
  • Medium (8 cases): Schema validation cases. Verify the returned object has expected keys using jsonpath. Test different input names, empty strings, and special characters.
  • Hard (7 cases): Edge cases. Timestamp format validation with regex, Unicode handling, max-length inputs, missing optional fields, and concurrent-style prompts.

Every case uses schema_version: "v0", version: 1, and source_type: "hand_written" in its provenance. Replace the author field with your own name.

Why plan the distribution before writing cases?

Without a plan, you'll naturally gravitate toward easy cases that all look the same. A deliberate distribution forces coverage across output types (exact, regex, jsonpath) and difficulty tiers. When your eval runner lands in iteration 02, you'll immediately know whether it handles all three verification strategies correctly.

Write the JSONL File

Each line in a JSONL file is a standalone JSON object. No wrapping array, no commas between lines. This format is ideal for eval datasets because you can append cases without rewriting the file, and you can stream-parse without loading everything into memory.

  • Create datasets/hello-world-v0.jsonl in the systems/02-agent-evaluation-harness/ directory.
  • Write one JSON object per line, with no trailing commas and no blank lines between records.

Here are the first three cases to get you started:

{"schema_version":"v0","case_id":"greet-alice-exact","prompt":"Greet Alice","tool_name":"hello_world","expected_output_type":"exact","expected_output":"Hello, Alice!","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","basic"]}
{"schema_version":"v0","case_id":"greet-bob-regex","prompt":"Say hello to Bob","tool_name":"hello_world","expected_output_type":"regex","expected_output":"^Hello,\\s+Bob","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","regex-match"]}
{"schema_version":"v0","case_id":"greet-response-has-name-key","prompt":"Greet Charlie","tool_name":"hello_world","expected_output_type":"jsonpath","expected_output":"$.name","difficulty":"medium","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["schema-check","jsonpath"]}

What does each field do?

  • case_id uniquely identifies the case. It must match the pattern ^[a-z0-9][a-z0-9_-]{2,63}$. Lowercase, hyphens and underscores only, 3 to 64 characters.
  • expected_output_type tells the future eval runner how to compare the actual output against expected_output. For exact, it's a string equality check. For regex, it's a pattern match. For jsonpath, it checks that a path exists in the response object.
  • tags allow filtering and grouping cases. Give every case at least one tag so the dataset is filterable from day one.

Now write the remaining 22 cases. Your full 25 cases should cover these scenarios:

  • Standard names (Alice, Bob, Charlie, Diana, etc.) with exact and regex matching.
  • Empty string input to test how the tool handles missing names.
  • Unicode names (e.g., characters with accents, CJK characters) to test encoding.
  • Very long names (64+ characters) to test boundary behavior.
  • Special characters in names (quotes, angle brackets, ampersands).
  • Response object structure checks using jsonpath (e.g., $.name, $.greeting, $.timestamp).
  • Timestamp ISO 8601 format validation using regex.
  • Multi-word names (e.g., "Mary Jane", "Jean-Pierre").
  • Missing optional greeting field in the response.

I'll give it a go!

Write all 25 cases yourself using the distribution plan above. Remember: every case_id must be unique, every regex and jsonpath case needs a non-None expected_output, and every case needs at least one tag.

Let me check my work

Here is a complete 25-case dataset. Replace "you" in the author field with your own name.

{"schema_version":"v0","case_id":"greet-alice-exact","prompt":"Greet Alice","tool_name":"hello_world","expected_output_type":"exact","expected_output":"Hello, Alice!","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","basic"]}
{"schema_version":"v0","case_id":"greet-bob-regex","prompt":"Say hello to Bob","tool_name":"hello_world","expected_output_type":"regex","expected_output":"^Hello,\\s+Bob","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","regex-match"]}
{"schema_version":"v0","case_id":"greet-charlie-exact","prompt":"Greet Charlie","tool_name":"hello_world","expected_output_type":"exact","expected_output":"Hello, Charlie!","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","basic"]}
{"schema_version":"v0","case_id":"greet-diana-exact","prompt":"Say hi to Diana","tool_name":"hello_world","expected_output_type":"exact","expected_output":"Hello, Diana!","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","basic"]}
{"schema_version":"v0","case_id":"greet-eve-regex","prompt":"Greet Eve","tool_name":"hello_world","expected_output_type":"regex","expected_output":"Hello,?\\s*Eve","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","regex-match"]}
{"schema_version":"v0","case_id":"greet-frank-exact","prompt":"Greet Frank","tool_name":"hello_world","expected_output_type":"exact","expected_output":"Hello, Frank!","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","basic"]}
{"schema_version":"v0","case_id":"greet-grace-regex","prompt":"Say hello to Grace","tool_name":"hello_world","expected_output_type":"regex","expected_output":"(?i)hello.*grace","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","regex-match","case-insensitive"]}
{"schema_version":"v0","case_id":"greet-hank-exact","prompt":"Greet Hank","tool_name":"hello_world","expected_output_type":"exact","expected_output":"Hello, Hank!","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","basic"]}
{"schema_version":"v0","case_id":"greet-iris-regex","prompt":"Greet Iris please","tool_name":"hello_world","expected_output_type":"regex","expected_output":"^Hello,\\s+Iris","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","regex-match"]}
{"schema_version":"v0","case_id":"greet-jack-exact","prompt":"Greet Jack","tool_name":"hello_world","expected_output_type":"exact","expected_output":"Hello, Jack!","difficulty":"easy","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["greeting","basic"]}
{"schema_version":"v0","case_id":"greet-response-has-name-key","prompt":"Greet Charlie","tool_name":"hello_world","expected_output_type":"jsonpath","expected_output":"$.name","difficulty":"medium","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["schema-check","jsonpath"]}
{"schema_version":"v0","case_id":"greet-response-has-greeting-key","prompt":"Greet Diana","tool_name":"hello_world","expected_output_type":"jsonpath","expected_output":"$.greeting","difficulty":"medium","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["schema-check","jsonpath"]}
{"schema_version":"v0","case_id":"greet-response-has-timestamp","prompt":"Greet Eve","tool_name":"hello_world","expected_output_type":"jsonpath","expected_output":"$.timestamp","difficulty":"medium","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["schema-check","jsonpath","timestamp"]}
{"schema_version":"v0","case_id":"greet-multiword-name","prompt":"Greet Mary Jane","tool_name":"hello_world","expected_output_type":"regex","expected_output":"Mary\\s+Jane","difficulty":"medium","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["multi-word","regex-match"]}
{"schema_version":"v0","case_id":"greet-hyphenated-name","prompt":"Greet Jean-Pierre","tool_name":"hello_world","expected_output_type":"regex","expected_output":"Jean-Pierre","difficulty":"medium","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["multi-word","regex-match","hyphenated"]}
{"schema_version":"v0","case_id":"greet-special-chars-ampersand","prompt":"Greet Tom & Jerry","tool_name":"hello_world","expected_output_type":"regex","expected_output":"Tom.*Jerry","difficulty":"medium","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["special-chars","regex-match"]}
{"schema_version":"v0","case_id":"greet-special-chars-quotes","prompt":"Greet O'Brien","tool_name":"hello_world","expected_output_type":"regex","expected_output":"O'Brien","difficulty":"medium","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["special-chars","regex-match"]}
{"schema_version":"v0","case_id":"greet-empty-string-structure","prompt":"Greet with an empty name","tool_name":"hello_world","expected_output_type":"jsonpath","expected_output":"$.greeting","difficulty":"medium","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["edge-case","empty-input","jsonpath"]}
{"schema_version":"v0","case_id":"greet-unicode-accented","prompt":"Greet Ren\u00e9e","tool_name":"hello_world","expected_output_type":"regex","expected_output":"Ren\u00e9e","difficulty":"hard","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["unicode","regex-match"]}
{"schema_version":"v0","case_id":"greet-unicode-cjk","prompt":"Greet \u592a\u90ce","tool_name":"hello_world","expected_output_type":"regex","expected_output":"\u592a\u90ce","difficulty":"hard","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["unicode","cjk","regex-match"]}
{"schema_version":"v0","case_id":"greet-unicode-emoji-name","prompt":"Greet \ud83d\ude00User","tool_name":"hello_world","expected_output_type":"regex","expected_output":"\ud83d\ude00User","difficulty":"hard","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["unicode","emoji","regex-match"]}
{"schema_version":"v0","case_id":"greet-long-name-boundary","prompt":"Greet Abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijkl","tool_name":"hello_world","expected_output_type":"regex","expected_output":"Abcdefghij","difficulty":"hard","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["edge-case","long-input","regex-match"]}
{"schema_version":"v0","case_id":"greet-timestamp-iso8601","prompt":"Greet Frank and check timestamp","tool_name":"hello_world","expected_output_type":"regex","expected_output":"\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}","difficulty":"hard","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["timestamp","iso8601","regex-match"]}
{"schema_version":"v0","case_id":"greet-tool-called-no-output-check","prompt":"Just call the greeting tool for anyone","tool_name":"hello_world","expected_output_type":"exact","expected_output":null,"difficulty":"hard","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["tool-call-only","no-output-check"]}
{"schema_version":"v0","case_id":"greet-missing-optional-field","prompt":"Greet with minimal response","tool_name":"hello_world","expected_output_type":"jsonpath","expected_output":"$.name","difficulty":"hard","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["edge-case","missing-field","jsonpath"]}
{"schema_version":"v0","case_id":"greet-concurrent-style-prompt","prompt":"Greet Alice and Bob at the same time","tool_name":"hello_world","expected_output_type":"regex","expected_output":"Alice|Bob","difficulty":"hard","provenance":{"author":"you","source_type":"hand_written"},"version":1,"tags":["concurrent","multi-name","regex-match"]}

What does this dataset cover?

Lines 1-10 are easy cases testing basic greetings with exact and regex matching. Lines 11-18 are medium cases testing response structure with jsonpath, multi-word names, and special characters. Lines 19-25 are hard cases covering Unicode, long inputs, ISO 8601 timestamps, null output checks, and concurrent prompts.

Pro tip: keep case IDs descriptive

A case_id like greet-timestamp-iso8601 tells you exactly what the case tests at a glance. Avoid generic IDs like test-001. When your dataset grows to hundreds of cases, descriptive IDs save you from opening the file to understand what failed.

Validate the Dataset by Hand

The CLI doesn't exist yet. You still need to confirm every line in your JSONL file parses cleanly through the EvalCase schema before moving on. A quick smoke test in the Python REPL catches typos and schema violations now, when they're cheap to fix.

  • Open a Python REPL or create a temporary script in the project root.
  • Run the following validation code.
from pathlib import Path
from eval_harness.schema import EvalCase

dataset = Path("datasets/hello-world-v0.jsonl")
cases = []
for i, line in enumerate(dataset.read_text().splitlines(), 1):
    case = EvalCase.model_validate_json(line)
    cases.append(case)
    print(f"  Line {i}: {case.case_id} OK")

print(f"\nValidated {len(cases)} cases")
assert len(cases) == 25, f"Expected 25 cases, got {len(cases)}"

What does this script do?

The script reads every line of the JSONL file and passes it through EvalCase.model_validate_json(). This method parses the JSON string directly into a Pydantic model, running all field validators and the model-level validator you wrote in Step 2. If any line fails, Pydantic raises a ValidationError with the exact field and reason. The final assertion confirms you have exactly 25 cases.

  • Run the script from the systems/02-agent-evaluation-harness/ directory.
python validate_dataset.py

You should see 25 lines of output, each showing the case_id followed by "OK", and a final line reading "Validated 25 cases".

Seeing a ValidationError?

Check these common causes:

  • case_id pattern mismatch. IDs must be lowercase alphanumeric, 3-64 characters, starting with a letter or digit. Only hyphens and underscores are allowed. No spaces, no uppercase.
  • expected_output is null but expected_output_type is regex or jsonpath. Your model validator requires a non-None value for these types.
  • schema_version is not exactly "v0". The Literal type rejects any other value.
  • JSON syntax errors in the JSONL. Trailing commas, unescaped quotes, or missing closing braces. Use python -m json.tool on individual lines to debug.
  • version is 0 or negative. The ge=1 constraint requires version >= 1.

If you're stuck, help me debug my JSONL validation error.

Your dataset is validated and ready. Next up, you'll build the Typer CLI that loads this file, validates it with proper error reporting, and prints summary statistics.

Build the Typer CLI

You have a validated schema and a 25-case dataset. Now you need a way to interact with them from the command line. Typer is a modern Python CLI library that uses type hints for argument declaration, automatic help generation, and Rich-formatted output. In this step, you will build the eval-harness CLI with a fully implemented load command and stub commands for future iterations.

In This Step, Get Ready To:
  • Create a Typer app with load, run, and report commands.
  • Implement the load command with --validate and --summary flags.
  • Verify the CLI loads and validates all 25 cases in under 500ms.
Create the App With Load, Run, and Report Commands

Typer uses the @app.command() decorator to register Python functions as CLI subcommands. Each function's type-hinted parameters become the command's arguments and options automatically. You will create three commands, but only load gets a real implementation in this iteration.

  • Open src/eval_harness/cli.py and add the following code.
import typer
from pathlib import Path
from eval_harness.schema import EvalCase

app = typer.Typer(name="eval-harness", no_args_is_help=True)


@app.command()
def load(
    path: Path,
    validate: bool = typer.Option(False, "--validate", help="Validate every case against the schema"),
    summary: bool = typer.Option(False, "--summary", help="Print dataset summary statistics"),
) -> None:
    """Load and optionally validate/summarize an eval dataset."""
    # Full implementation coming in the next substep
    typer.echo(f"Loading {path}...")


@app.command()
def run() -> None:
    """Run eval cases against an MCP server. (Not implemented in iteration 01.)"""
    typer.echo("Not implemented in iteration 01. See SERIES.md.")
    raise typer.Exit(code=0)


@app.command()
def report() -> None:
    """Generate an eval report. (Not implemented in iteration 01.)"""
    typer.echo("Not implemented in iteration 01. See SERIES.md.")
    raise typer.Exit(code=0)

What does this code do?

typer.Typer(no_args_is_help=True) creates the CLI app and shows the help text when no subcommand is provided. The load command takes a Path argument (the JSONL file) and two boolean flags. typer.Option(False, "--validate") creates a boolean flag that defaults to False. The run and report commands are stubs that print a message and exit cleanly.

How Does the Entry Point Work?

The [project.scripts] section in your pyproject.toml maps the CLI name eval-harness to eval_harness.cli:app. This tells Python to look for the app object inside eval_harness.cli and use it as the CLI entrypoint. Because you installed the package in editable mode earlier, the eval-harness command is already available in your terminal.

Implement the Load Command Logic

The load command needs to do three things: parse every line through EvalCase, report validation results when --validate is set, and print dataset statistics when --summary is set. You will use collections.Counter for the breakdowns.

  • Replace the entire contents of src/eval_harness/cli.py with the full implementation below.

I'll give it a go!

Implement the load command body yourself. Here is what it needs to do:

  • Read the file and split into lines, filtering out empty lines.
  • For each line, attempt EvalCase.model_validate_json(line). Catch ValidationError and record failures with the line number and error message.
  • If --validate is set, print each line as PASS or FAIL, then print totals. Exit with code 1 if any failures.
  • If --summary is set, print total cases, breakdown by difficulty, by output type, by source type, and unique tags (sorted, comma-separated).
  • If neither flag is set, just print the count of loaded cases.

Let me check my work

import typer
from collections import Counter
from pathlib import Path
from pydantic import ValidationError
from eval_harness.schema import EvalCase

app = typer.Typer(name="eval-harness", no_args_is_help=True)


@app.command()
def load(
    path: Path,
    validate: bool = typer.Option(False, "--validate", help="Validate every case against the schema"),
    summary: bool = typer.Option(False, "--summary", help="Print dataset summary statistics"),
) -> None:
    """Load and optionally validate/summarize an eval dataset."""
    # Read the file and filter out empty lines
    lines = [line for line in path.read_text().splitlines() if line.strip()]

    cases: list[EvalCase] = []
    failures: list[tuple[int, str]] = []

    # Parse each line through the EvalCase schema
    for i, line in enumerate(lines, 1):
        try:
            case = EvalCase.model_validate_json(line)
            cases.append(case)
            if validate:
                typer.echo(f"  Line {i}: PASS ({case.case_id})")
        except ValidationError as e:
            failures.append((i, str(e)))
            if validate:
                typer.echo(f"  Line {i}: FAIL - {e}")

    # Print validation totals
    if validate:
        typer.echo(f"\nValidation: {len(cases)} passed, {len(failures)} failed out of {len(lines)} lines")
        if failures:
            raise typer.Exit(code=1)

    # Print summary statistics using Counter for breakdowns
    if summary:
        typer.echo(f"\n--- Dataset Summary ---")
        typer.echo(f"Total cases: {len(cases)}")

        diff_counts = Counter(c.difficulty for c in cases)
        typer.echo(f"By difficulty: easy={diff_counts.get('easy', 0)}, medium={diff_counts.get('medium', 0)}, hard={diff_counts.get('hard', 0)}")

        out_counts = Counter(c.expected_output_type for c in cases)
        typer.echo(f"By output type: exact={out_counts.get('exact', 0)}, regex={out_counts.get('regex', 0)}, jsonpath={out_counts.get('jsonpath', 0)}")

        src_counts = Counter(c.provenance.source_type for c in cases)
        typer.echo(f"By source type: hand_written={src_counts.get('hand_written', 0)}, synthetic={src_counts.get('synthetic', 0)}, adversarial={src_counts.get('adversarial', 0)}")

        # Collect and sort all unique tags
        all_tags = sorted({tag for c in cases for tag in c.tags})
        typer.echo(f"Unique tags: {', '.join(all_tags)}")

    # Default output when no flags are set
    if not validate and not summary:
        typer.echo(f"Loaded {len(cases)} cases from {path}")


@app.command()
def run() -> None:
    """Run eval cases against an MCP server. (Not implemented in iteration 01.)"""
    typer.echo("Not implemented in iteration 01. See SERIES.md.")
    raise typer.Exit(code=0)


@app.command()
def report() -> None:
    """Generate an eval report. (Not implemented in iteration 01.)"""
    typer.echo("Not implemented in iteration 01. See SERIES.md.")
    raise typer.Exit(code=0)

What does this implementation do?

The load command reads the JSONL file and parses each line through EvalCase.model_validate_json(), which parses a JSON string directly into a Pydantic model. This is faster than calling json.loads() followed by model_validate() because Pydantic v2's Rust-based core handles the JSON parsing and validation in a single pass.

The Counter class from collections tallies each field value in a single pass over the cases list. The summary output gives you a quick snapshot of your dataset's composition without opening the JSONL file manually.

Verify the CLI End-to-End

Time to see your CLI in action. You will run both flags against your 25-case dataset and confirm everything passes validation, the summary breakdowns are correct, and the whole command completes in under 500ms.

  • Make sure you are in the systems/02-agent-evaluation-harness directory.
  • Run the load command with the --validate flag.
eval-harness load datasets/hello-world-v0.jsonl --validate

You should see 25 lines of PASS output, followed by a totals line showing 25 passed and 0 failed.

  • Run the load command with the --summary flag.
eval-harness load datasets/hello-world-v0.jsonl --summary

The summary should show 25 total cases, a breakdown of 10 easy, 8 medium, and 7 hard, and all cases listed as hand_written.

  • Run both flags together and time the execution.
time eval-harness load datasets/hello-world-v0.jsonl --validate --summary

What should I see?

The time command prints the elapsed wall-clock time after the command finishes. The "real" value should be well under 500ms (likely under 200ms). Pydantic v2's Rust-based core validates thousands of models per second, so 25 cases is trivial. The bottleneck would be Python startup time, not validation.

Seeing Pydantic Validation Errors?

If you see ValidationError on load, check these common causes:

  • case_id pattern mismatch. IDs must be lowercase alphanumeric, 3-64 characters, starting with a letter or digit. Only hyphens and underscores are allowed. No spaces, no uppercase.
  • expected_output is None but expected_output_type is "regex" or "jsonpath". The model validator requires a non-None value for these types.
  • schema_version is not exactly "v0". The Literal type rejects any other value.
  • JSON syntax errors in the JSONL. Trailing commas, unescaped quotes, or missing closing braces. Use python -m json.tool on individual lines to debug.
  • version is 0 or negative. The ge=1 constraint requires version >= 1.

If you are stuck, help me debug my validation errors.

Your CLI is working. You can load, validate, and summarize your entire dataset from the command line. Next up, you will write tests to prove the schema rejects malformed input and the dataset stays clean.

Write Tests

Your CLI loads and validates the dataset, but you need automated proof that the schema catches what it should catch and passes what it should pass. In this step, you will write pytest tests that verify valid construction, rejection of bad input, and a full round-trip of all 25 cases.

In This Step, Get Ready To:
  • Write round-trip serialization tests proving EvalCase survives JSON export and re-import without data loss.
  • Write negative tests proving the schema rejects malformed input with ValidationError.
  • Write an integration test that loads the full 25-case dataset and checks for duplicates.
Test Valid Case Construction and Round-Trip Serialization

A round-trip test proves that no data is lost or corrupted when a model is serialized to JSON and parsed back. This is critical for eval cases because the JSONL file is the source of truth. If serialization silently drops a field, your eval results become unreliable.

  • Open tests/test_schema.py.
  • Add the following test functions.
import pytest
from pydantic import ValidationError
from eval_harness.schema import EvalCase, Provenance


def _make_case(**overrides) -> EvalCase:
    """Helper that builds a valid EvalCase, letting tests override specific fields."""
    defaults = {
        "schema_version": "v0",
        "case_id": "test-case-001",
        "prompt": "Greet Alice",
        "tool_name": "hello_world",
        "expected_output_type": "exact",
        "expected_output": "Hello, Alice!",
        "difficulty": "easy",
        "provenance": Provenance(author="tester", source_type="hand_written"),
        "version": 1,
        "tags": ["greeting"],
    }
    defaults.update(overrides)
    return EvalCase(**defaults)


def test_valid_case_round_trip():
    """A valid case survives JSON serialization and deserialization."""
    original = _make_case()
    json_str = original.model_dump_json()
    restored = EvalCase.model_validate_json(json_str)
    assert original == restored


def test_frozen_model_rejects_mutation():
    """Cases are immutable after creation."""
    case = _make_case()
    with pytest.raises(ValidationError):
        case.prompt = "Changed prompt"


@pytest.mark.parametrize("difficulty", ["easy", "medium", "hard"])
def test_all_difficulty_levels_accepted(difficulty):
    """Each difficulty literal is valid."""
    case = _make_case(difficulty=difficulty)
    assert case.difficulty == difficulty


@pytest.mark.parametrize("output_type", ["exact", "regex", "jsonpath"])
def test_all_output_types_accepted(output_type):
    """Each expected_output_type literal is valid."""
    expected = "Hello" if output_type == "exact" else "^Hello" if output_type == "regex" else "$.name"
    case = _make_case(expected_output_type=output_type, expected_output=expected)
    assert case.expected_output_type == output_type

What does the _make_case helper do?

The _make_case() function builds a valid EvalCase with sensible defaults. Tests override only the field they care about. This keeps each test focused on one behavior instead of repeating the full constructor every time.

Test Validation Failures

Positive tests prove the schema accepts good data. Negative tests prove it rejects bad data. Without negative tests, you cannot be confident that the schema is actually enforcing its constraints. A schema that accepts everything is no schema at all.

  • Add the following test functions to the same tests/test_schema.py file, below the existing tests.
@pytest.mark.parametrize(
    "bad_id",
    [
        "UPPERCASE",        # no uppercase allowed
        "has spaces",       # no spaces allowed
        "ab",               # too short (min 3 chars)
    ],
)
def test_invalid_case_id_rejected(bad_id):
    """case_id must match the pattern ^[a-z0-9][a-z0-9_-]{2,63}$."""
    with pytest.raises(ValidationError):
        _make_case(case_id=bad_id)


def test_invalid_schema_version_rejected():
    """Only 'v0' is accepted for schema_version."""
    with pytest.raises(ValidationError):
        _make_case(schema_version="v1")


def test_regex_type_requires_expected_output():
    """A regex match with no pattern is meaningless."""
    with pytest.raises(ValidationError):
        _make_case(expected_output_type="regex", expected_output=None)


def test_jsonpath_type_requires_expected_output():
    """A jsonpath check with no expression is meaningless."""
    with pytest.raises(ValidationError):
        _make_case(expected_output_type="jsonpath", expected_output=None)


def test_version_zero_rejected():
    """Version must be >= 1."""
    with pytest.raises(ValidationError):
        _make_case(version=0)


def test_invalid_source_type_rejected():
    """source_type must be one of the allowed literals."""
    with pytest.raises(ValidationError):
        _make_case(
            provenance=Provenance(author="tester", source_type="unknown")
        )

Why test each failure mode individually?

Each test targets exactly one constraint. If you bundled multiple invalid fields into a single test, a passing test would only prove that at least one constraint fired. Individual tests prove each constraint works independently.

Test the Full Dataset Loads Cleanly

Unit tests cover the schema in isolation. This integration test proves that your actual JSONL file conforms to the schema. It also checks for duplicate case_id values, which the schema alone cannot catch because each line is validated independently.

  • Add the following test to tests/test_schema.py.
from pathlib import Path


def test_full_dataset_loads_without_errors():
    """Every line in the JSONL dataset must parse as a valid EvalCase."""
    dataset_path = Path(__file__).resolve().parent.parent / "datasets" / "hello-world-v0.jsonl"
    assert dataset_path.exists(), f"Dataset not found at {dataset_path}"

    cases = []
    for i, line in enumerate(dataset_path.read_text().splitlines(), 1):
        if not line.strip():
            continue
        case = EvalCase.model_validate_json(line)
        cases.append(case)

    # Exactly 25 cases expected
    assert len(cases) == 25, f"Expected 25 cases, got {len(cases)}"

    # No duplicate case_id values
    ids = [c.case_id for c in cases]
    assert len(ids) == len(set(ids)), f"Duplicate case_ids found: {[x for x in ids if ids.count(x) > 1]}"

What does this test do?

This test reads the actual hello-world-v0.jsonl file from disk and parses every line through EvalCase.model_validate_json(). It asserts two things: exactly 25 cases load successfully, and no two cases share the same case_id. The path resolution uses __file__ so the test works regardless of which directory you run pytest from.

Now run the full test suite.

  • Make sure you are in the systems/02-agent-evaluation-harness directory.
  • Run the tests with verbose output.
cd systems/02-agent-evaluation-harness
pytest tests/ -v

What should I see?

You should see all tests passing with green output. The exact count depends on the parametrized tests, but expect around 15 or more individual test results. Every line should show PASSED.

Tests failing unexpectedly?

If the dataset integration test fails with a FileNotFoundError, confirm you are running pytest from the systems/02-agent-evaluation-harness directory and that the JSONL file exists at datasets/hello-world-v0.jsonl. If a validation test fails with no error raised, double-check that your model_validator in schema.py raises ValueError for the regex/jsonpath cases. Help me debug my failing tests.

Your schema, dataset, and CLI are now backed by automated tests. Next up, you will document the schema as a one-way-door decision in ADR-006.

Write ADR-006

Your schema is built, your dataset validates, your CLI works, and your tests pass. There is one critical piece missing: the decision record that locks this schema as the contract. ADR-006 documents the eval dataset schema as a one-way-door decision. Once this ADR is committed alongside the code, breaking changes to the schema require a new version and migration tooling.

In This Step, Get Ready To:
  • Draft ADR-006 using the Nygard lightweight ADR format.
  • Verify every claim in the ADR matches the actual implementation.
Draft ADR-006 for the Eval Dataset Schema

An Architecture Decision Record (ADR) captures a significant design decision alongside its context and consequences. The Nygard lightweight ADR format uses five sections: Title, Status, Context, Decision, and Consequences. It fits on one page and is the most widely adopted lightweight format.

The schema you built in Step 2 is a one-way-door decision. Once datasets are published against it, breaking changes require a new schema version and migration tooling. ADR-006 makes this commitment explicit.

  • Create docs/adr/ADR-006-eval-dataset-schema-v1.md and add the following content:
# ADR-006: Eval Dataset Schema v1

## Status

Accepted

## Context

The evaluation harness needs a stable, versioned schema for eval cases
before any runner or reporter is built. The schema is a one-way-door
decision: once datasets are published against it, breaking changes
require a new schema version and migration tooling.

We need a schema that is:

- **Self-describing**: every record declares its own schema version.
- **Traceable**: every case carries provenance (who wrote it, how it was generated).
- **Unambiguous**: the expected output type (exact, regex, jsonpath) is explicit,
  not inferred.

## Decision

We adopt the `EvalCase` schema defined in `src/eval_harness/schema.py`
as the v0 contract. Key design choices:

1. **`schema_version` in every record** for forward compatibility. A single
   dataset file can contain mixed versions during migration because each
   record self-describes.
2. **`provenance` as a required nested object** for traceability. Every case
   must declare its author and source type (`hand_written`, `synthetic`,
   or `adversarial`).
3. **`expected_output_type` as an explicit discriminator** (`exact`, `regex`,
   `jsonpath`) rather than implicit type inference. Evaluation logic never
   has to guess what "correct" means.
4. **`case_id` pattern-validated** with `^[a-z0-9][a-z0-9_-]{2,63}$` for
   filesystem and URL safety.
5. **Immutable instances** via `frozen=True` to prevent mutation after parsing.

## Consequences

- Schema changes after publication require a new version (v1, v2, etc.).
  The CLI must support loading multiple schema versions.
- Provenance is mandatory, which adds friction to case authoring but
  prevents untraceable data from entering the pipeline.
- The `expected_output_type` discriminator means adding a new match type
  (e.g., `semantic`) requires a schema version bump.
- `case_id` constraints prevent natural-language IDs, which is intentional.
  IDs are references, not descriptions.

What is a one-way-door decision?

A one-way-door decision is one that is difficult or impossible to reverse once made. Publishing a schema that other code depends on is a classic example. If you change the schema later, every existing dataset and every downstream consumer breaks. That is why you document it in an ADR before committing: you are making the cost of reversal explicit.

Why Nygard over MADR?

The MADR template includes additional sections like Decision Drivers, Considered Options, and Decision Outcome. For a focused schema decision where the alternatives are straightforward, the five-section Nygard format captures everything needed without ceremony. You can always upgrade to MADR for more complex architectural decisions later.

Review the ADR Against the Implementation

An ADR that drifts from the actual code is worse than no ADR at all. Before committing, you need to verify every claim in ADR-006 matches what is in schema.py.

  • Open src/eval_harness/schema.py and docs/adr/ADR-006-eval-dataset-schema-v1.md side by side.
  • Confirm the case_id regex in the ADR matches the Field(pattern=...) value in the schema: ^[a-z0-9][a-z0-9_-]{2,63}$.
  • Confirm the three expected_output_type values listed in the ADR (exact, regex, jsonpath) match the Literal type in the schema.
  • Confirm the Provenance model has author and source_type fields as described in the ADR.
  • Confirm frozen=True is set in the model config, matching the "Immutable instances" claim.
  • Confirm the schema_version field is Literal["v0"], matching the "v0 contract" language in the Decision section.

If anything does not match, update the ADR to reflect the actual implementation. The code is the source of truth. The ADR documents the code, not the other way around.

Pro tip: ADRs live in source control

ADRs should be stored in source control alongside the code they document, numbered sequentially. This means the ADR and the schema travel together in every commit, branch, and pull request. A reviewer can see both the decision and the implementation in the same diff.

ADR-006 is written and verified. Your schema contract is now locked. Every piece of the harness scaffold is in place: schema, dataset, CLI, tests, and decision record. Time to wrap up and look ahead to what comes next.

Secret mission

Add a Lint Subcommand

Your schema validates structure, but it can't catch quality issues like duplicate case IDs or missing tags. Add a lint command that enforces dataset hygiene before bad data reaches the eval runner.

Before you go

Clean Up Your Resources

This project runs entirely locally with no cloud resources, so there are no ongoing costs. Decide whether to keep your resources as-is, pause them to come back later, or delete them entirely.

Resources you used:

  • The systems/02-agent-evaluation-harness/ directory and all files inside it.
  • The Python virtual environment (if you created one) for this project.
  • The editable pip install of the eval-harness package.

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue with iteration 02 soon.

  • Your project files, dataset, and CLI entry point all remain in place.
  • The editable install means any code changes take effect immediately without reinstalling.

Pause - I'll come back to this later

Deactivate your virtual environment to free up shell resources. Your files stay on disk and you can resume anytime.

  • Deactivate your virtual environment by running deactivate in your terminal.
  • When you return, reactivate the environment and run pip install -e ".[dev]" again from the systems/02-agent-evaluation-harness/ directory to restore the CLI entry point.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is irreversible.

  • Uninstall the package by running pip uninstall eval-harness from within your virtual environment.
  • Delete the entire project directory by running rm -rf systems/02-agent-evaluation-harness/ from the repo root.
  • If you created a dedicated virtual environment for this project, delete it by removing the environment directory (e.g., rm -rf .venv).

Mission accomplished

That's a Wrap!

Nice work! You just built the foundation of an agent evaluation pipeline from scratch. You have a validated, versioned eval dataset and a CLI that loads and summarizes 25 cases in under 500ms. This is the infrastructure every serious eval pipeline needs before a single eval can run.

Here's what you accomplished:

  • Designed a publication-grade eval case schema with Pydantic v2, baking in provenance tracking and schema versioning from day one.
  • Hand-wrote a 25-case JSONL dataset targeting the hello-world MCP server across easy, medium, and hard difficulty tiers, with full validation tests proving every case conforms to the schema.
  • Built a Typer CLI with load, --validate, and --summary flags, and authored ADR-006 documenting the schema as a one-way-door decision.
  • Added a lint subcommand that catches duplicate case_id values, missing tags, and schema version drift before they reach the eval runner.

Ready to quiz yourself?