Calibrate Claude Code Effort

Introduction

30 Second Summary

Default settings often stay in place because nobody has measured whether a lighter option can do the same job. Without a fair comparison, choosing between speed and confidence becomes guesswork.

In this project, you will build a local calibration harness that compares high effort with medium effort on one known repair using Claude Code. Its report records ten measurements before applying a frozen rule to select USE_MEDIUM or KEEP_HIGH.

What You'll Build

Picture opening results/report.md after two isolated repairs and seeing matching starting hashes, ten measured values, every control confirmed, and a final verdict backed by the workspace checker.

By the end of this project, you'll have:

• An auditable preflight that records your exact eligible plan tier before any model call. It also captures the literal Claude Code client version.
• A frozen repair fixture backed by a passing Python control. Its preregistration proves all three breaks existed before either byte-identical workspace was created.
• An isolated comparison report that records pass or fail, turns, whole-tree tokens, client-estimated cost, and wall-clock minutes for each fresh session. It ends with the script-generated USE_MEDIUM or KEEP_HIGH decision.
• Secret Mission: Tamper with a copy of the measurements and prove that weaker lower-effort evidence forces KEEP_HIGH.

Are there any prerequisites?

The guide assumes no local tools are installed, so it checks each requirement from the start.

Model runs add no cost only while included Fable allowance remains available through a Max plan, a premium seat on Team plan, or a premium seat on seat-based Enterprise plan, with no payment card or usage-credit setup.

Before We Start

A controlled experiment starts with a clear commitment before any results can influence your judgment. This is your moment to define the narrow comparison you will trust.

Gate Access and Tools

A metered credential or hidden effort override would invalidate the comparison before either repair begins. This gate confirms the account path before any model call can start.

You will verify included Claude Fable 5.1 access. You will also prepare Python, Visual Studio Code, and Claude Code for the controlled repair.

In this step, get ready to:

• Confirm an eligible included plan tier with usage credits disabled.
• Install or verify Python, Visual Studio Code, and Claude Code.
• Confirm subscription authentication with no active credential or effort override.

Check included Fable access

This account check touches billing settings. The project parks before every model call when your current access requires usage credits or another metered path.

• Open Claude in your browser.
• Sign in with the account you plan to use with Claude Code.
• Open Settings.
• Select Usage.
• Check whether the page shows remaining Fable allowance inside your regular plan usage.
• Identify the exact plan tier shown on the page.

✔️ I have eligible included access

Continue only when the exact tier is Max plan, premium seat on Team plan, or premium seat on seat-based Enterprise plan.

• Record the exact eligible tier here: your exact eligible plan tier.
• Record the date of this check here: your plan check date.
• Confirm that usage credits remain disabled for this project.

That clears the cost gate. Your comparison can use included subscription access without requesting a payment card.

ⓧ My access is metered or exhausted

Park the project if you have Pro or a standard Team seat. Also park for a standard Enterprise seat or usage-based Enterprise access.

Exhausted included Fable allowance also requires you to park. These paths can use metered credits.

• Stop before creating the access attestation.
• Decline any request to enable usage credits.
• Return when the Usage page shows an eligible included tier with remaining allowance.
• Keep the project parked if the page asks for billing setup.

Keep the experiment unmetered

Do not add a payment card or enroll in usage credits for this project. The gate protects the comparison by stopping before a paid model call.

Verify the local tools and save the attestation

The harness depends on three local tools. Capturing their installed versions makes the final comparison traceable to the environment that produced it.

• Press Cmd+Space on macOS or the Windows key on Windows to open system search.
• Open your applications menu if you use Linux.
• Search for your terminal application.
• Open the terminal application.
• Check the three local tools by running the commands for your platform:

macOS

python3 --version
code --version
claude --version

What do these commands check?

• The first command prints the installed Python version.
• The second command prints the Visual Studio Code version details.
• The third command prints the literal installed Claude Code version without starting a model call.

Windows

py --version
code --version
claude --version

What do these commands check?

• The first command asks the Windows Python launcher for its installed version.
• The second command prints the Visual Studio Code version details.
• The third command prints the literal installed Claude Code version without starting a model call.

Linux

python3 --version
code --version
claude --version

What do these commands check?

• The first command prints the installed Python version.
• The second command prints the Visual Studio Code version details.
• The third command prints the literal installed Claude Code version without starting a model call.

Start with the Python result. This project uses Python 3.14.7 and its standard library.

✔️ I see version 3.14.7 or higher

Python is ready for the control checker and benchmark harness. Keep the literal version output for the checkpoint at the end of this step.

ⓧ I see an older version

Install Python 3.14.7 through the official path for your platform. The new version must be the one used by this project.

macOS

• Open the official Python downloads page.
• Download the macOS installer for Python 3.14.7.
• Run the installer.
• Open a fresh terminal after installation.
• Return to the version check above.

Still seeing the older Python?

An earlier Python installation may still be first on your command path. Help me select the new Python installation.

Windows

• Open the official Python downloads page.
• Download the Windows installer for Python 3.14.7.
• Run the installer.
• Open a fresh terminal after installation.
• Return to the version check above.

Still seeing the older Python?

The Windows launcher may still select an earlier installation. Help me verify the Python launcher.

Linux

• Open the official Python downloads page.
• Follow the official installation path for your Linux distribution.
• Install Python 3.14.7.
• Open a fresh terminal after installation.
• Return to the version check above.

Still seeing the older Python?

Your distribution may keep its system Python as the default command. Help me verify the newer Python installation.

ⓧ Python is not installed

Install Python 3.14.7 through the official path for your platform.

macOS

• Open the official Python downloads page.
• Download the macOS installer for Python 3.14.7.
• Run the installer.
• Open a fresh terminal after installation.
• Return to the version check above.

Python command still missing?

A fresh terminal may be required before the installed command becomes available. Help me find the Python command.

Windows

• Open the official Python downloads page.
• Download the Windows installer for Python 3.14.7.
• Run the installer.
• Open a fresh terminal after installation.
• Return to the version check above.

Python command still missing?

A fresh terminal may be required before the Windows launcher becomes available. Help me find the Python launcher.

Linux

• Open the official Python downloads page.
• Follow the official installation path for your Linux distribution.
• Install Python 3.14.7.
• Open a fresh terminal after installation.
• Return to the version check above.

Python command still missing?

A fresh terminal may be required before the installed command becomes available. Help me find the Python command.

Next, use the result from the Visual Studio Code version check to choose your path.

✔️ I see Visual Studio Code version details

Visual Studio Code is installed with its command available to the terminal. Keep the literal output for the checkpoint at the end of this step.

ⓧ The code command is missing

macOS

• Open the official Visual Studio Code setup guide.
• Download the macOS installer.
• Drag Visual Studio Code.app to Applications.
• Open Visual Studio Code through system search.
• Run Shell Command: Install 'code' command in PATH from the Command Palette.
• Open a fresh terminal.
• Return to the version check above.

The code command is still missing?

A fresh terminal may be required after adding the command to your path. Help me fix the Visual Studio Code command.

Windows

• Open the official Visual Studio Code setup guide.
• Download the User Setup installer.
• Run the installer.
• Open a fresh terminal.
• Return to the version check above.

The code command is still missing?

The User Setup installer adds the command to your path. Help me fix the Visual Studio Code command.

Linux

• Open the official Visual Studio Code Linux setup guide.
• Select the documented package for your Linux distribution.
• Install Visual Studio Code.
• Open a fresh terminal.
• Return to the version check above.

The code command is still missing?

Your package installation may need a fresh terminal before the command becomes available. Help me fix the Visual Studio Code command.

Finally, use the Claude Code version result to choose your installation path.

✔️ I see a Claude Code version

Claude Code is installed. Keep the literal output because it becomes part of the experiment's provenance.

ⓧ The claude command is missing

The installer may take a few minutes while it downloads Claude Code. A quiet terminal during that time does not mean the installation has failed.

macOS

• Install Claude Code by running the official command:

curl -fsSL https://claude.ai/install.sh | bash

What does this command do?

The command downloads the official installer from Claude. Your shell runs the installer locally.

• Open a fresh terminal after installation.
• Return to the version check above.

Claude Code command still missing?

The installed command may require a fresh terminal before it appears on your path. Help me verify the Claude Code installation.

Windows

• Install Claude Code in PowerShell by running the official command:

irm https://claude.ai/install.ps1 | iex

What does this command do?

The command retrieves the official Windows installer from Claude. PowerShell runs the installer locally.

• Open a fresh PowerShell window after installation.
• Return to the version check above.

Claude Code command still missing?

The installed command may require a fresh PowerShell window before it appears on your path. Help me verify the Claude Code installation.

Linux

• Install Claude Code by running the official command:

curl -fsSL https://claude.ai/install.sh | bash

What does this command do?

The command downloads the official installer from Claude. Your shell runs the installer locally.

• Open a fresh terminal after installation.
• Return to the version check above.

Claude Code command still missing?

The installed command may require a fresh terminal before it appears on your path. Help me verify the Claude Code installation.

Your plan evidence belongs in a small JSON attestation. The harness reads this file before allowing either model run.

• Press Cmd+Space on macOS or the Windows key on Windows to open system search.
• Open your applications menu if you use Linux.
• Type Visual Studio Code into the search field.
• Press Enter to open Visual Studio Code.
• Select New File.
• Paste the attestation template into the new file:

{
  "plan_tier": "replace with: Max plan, premium seat on Team plan, or premium seat on seat-based Enterprise plan",
  "fable_access": "included_subscription",
  "usage_credits_enabled_for_project": false,
  "subscription_login_confirmed": true,
  "checked_at": "replace-with-check-date",
  "evidence": "Settings > Usage showed remaining included Fable allowance"
}

What does this file prove?

The attestation records the account path that qualifies for this experiment. The false usage-credit value keeps the project on the included subscription path.

The evidence field records where you performed the check. The harness later parks if the tier or date still contains a placeholder.

• Replace the plan_tier placeholder with your exact eligible plan tier.
• Replace the checked_at placeholder with your plan check date.
• Press Cmd+Shift+S on macOS or Ctrl+Shift+S on Windows and Linux.
• Choose Desktop as the save location.
• Save the file as access-attestation.json.
• Confirm that access-attestation.json appears in the Visual Studio Code Explorer.

Attestation file not saving?

Confirm that the filename ends with .json only once. Keep every key in double quotes.

Check that the selected plan tier matches one eligible value exactly. Help me check my attestation structure.

✔️ Awesome, I've got everything!

Good work. Your attestation now preserves the included account path that this experiment depends on.

ⓧ I'd like to double check the full code

Compare your file structure with this source template. Keep your captured tier and date in place.

{
  "plan_tier": "replace with: Max plan, premium seat on Team plan, or premium seat on seat-based Enterprise plan",
  "fable_access": "included_subscription",
  "usage_credits_enabled_for_project": false,
  "subscription_login_confirmed": true,
  "checked_at": "replace-with-check-date",
  "evidence": "Settings > Usage showed remaining included Fable allowance"
}

Confirm subscription authentication and remove overrides

Claude Code can use subscription authentication or a metered credential. This experiment requires the subscription path recorded in your attestation.

Which overrides must be absent?

• Credential variables include ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_AWS_API_KEY, ANTHROPIC_FOUNDRY_API_KEY, ANTHROPIC_FOUNDRY_AUTH_TOKEN, and AWS_BEARER_TOKEN_BEDROCK.
• Routing variables include ANTHROPIC_BASE_URL and ANTHROPIC_PROFILE.
• Provider selectors include CLAUDE_CODE_USE_ANTHROPIC_AWS, CLAUDE_CODE_USE_BEDROCK, CLAUDE_CODE_USE_FOUNDRY, CLAUDE_CODE_USE_MANTLE, and CLAUDE_CODE_USE_VERTEX.
• The CLAUDE_CODE_EFFORT_LEVEL variable must be absent because it takes precedence over the client effort flag.

• Review the environment variables active in your terminal.
• Remove any active value for the listed credential variables.
• Remove any active value for the listed routing variables.
• Disable any active provider selector from the listed set.
• Remove CLAUDE_CODE_EFFORT_LEVEL if it is set.
• Close the current terminal after clearing the settings.
• Open a fresh terminal through system search.
• Check the subscription authentication state by running:

claude auth status --text

What does this command check?

The command reports whether Claude Code is logged in. It exits successfully when authentication is active.

This check does not start a model conversation. It only verifies the local authentication state.

✔️ Subscription authentication succeeds

Your local client is authenticated. The experiment can continue on the subscription path recorded in your attestation.

ⓧ I need to sign in

The sign-in command opens a browser flow. Use the same eligible Claude account whose Usage page you checked.

• Start subscription sign-in by running:

claude auth login

What does this command do?

The command signs Claude Code in to your Anthropic account. Omitting the console option keeps the flow on Claude subscription authentication.

• Complete the browser sign-in flow.
• Return to the authentication check above.

Browser sign-in not completing?

Confirm that the browser uses the same account whose Usage page you checked. Help me troubleshoot Claude Code subscription login.

Before you run the final gate, what version details and authentication state do you expect to see?

macOS

• Run the final local gate:

python3 --version
code --version
claude --version
claude auth status --text

What should you see?

You should see the installed Python version and Visual Studio Code details. You should also see the literal Claude Code version followed by a successful authenticated state.

Windows

• Run the final local gate:

py --version
code --version
claude --version
claude auth status --text

What should you see?

You should see the installed Python version and Visual Studio Code details. You should also see the literal Claude Code version followed by a successful authenticated state.

Linux

• Run the final local gate:

python3 --version
code --version
claude --version
claude auth status --text

What should you see?

You should see the installed Python version and Visual Studio Code details. You should also see the literal Claude Code version followed by a successful authenticated state.

• Return to access-attestation.json in Visual Studio Code.
• Confirm that plan_tier contains your exact eligible plan tier.
• Confirm that usage_credits_enabled_for_project remains false.
• Confirm that subscription_login_confirmed remains true.
• Confirm that checked_at contains your plan check date.

That is the gate cleared. Your included account path and local toolchain are now ready for a comparison with no model call made yet.

Your experiment is protected from metered access and hidden overrides. Next, you will create a clean invoice function and prove its checker passes before introducing any fault.

Establish a Passing Control

You cleared the first gate: your included access path is verified. Claude Code remains untouched while you establish a trustworthy starting point.

A passing control proves that the original invoice function satisfies its checker before any fault exists. Later failures can then be traced to deliberate mutations.

In this step, get ready to:

• Create the clean invoice calculation with integer-cent arithmetic.
• Encode each invoice rule in an automated test.
• Run the checker to prove the clean fixture passes.

Create the clean invoice rules

Integer cents keep every expected total exact. The clean function encodes the invoice rules that every later repair must restore.

• Press Cmd+Space on macOS or the Windows key on Windows to open your search bar.
• On Linux, open your application launcher.
• Type Visual Studio Code into the search bar.
• Press Enter to open Visual Studio Code.

• Click File in the top menu.
• Click Open Folder....
• Select the folder that contains access-attestation.json.
• Click Open.

You should see access-attestation.json in the Explorer. Keep this attestation unchanged throughout the step.

• Right-click the opened project folder in the Explorer.
• Select New Folder.
• Enter fixtures.
• Press Enter.

You should see the new fixtures folder beneath the opened project folder.

• Right-click the fixtures folder in the Explorer.
• Select New Folder.
• Enter clean.
• Press Enter.

You should see the nested path fixtures/clean in the Explorer.

• Right-click the clean folder in the Explorer.
• Select New File.
• Enter invoice.py.
• Press Enter.
• Add the clean calculation rules to fixtures/clean/invoice.py by pasting this code:

def calculate_total(subtotal_cents, is_member, destination):
    if subtotal_cents < 0:
        raise ValueError("subtotal_cents must be non-negative")

    discount_cents = 0
    if is_member and subtotal_cents >= 5000:
        discount_cents = subtotal_cents // 10

    discounted_subtotal = subtotal_cents - discount_cents

    if destination == "domestic":
        shipping_cents = 500
    elif destination == "international":
        shipping_cents = 1500
    else:
        raise ValueError("destination must be domestic or international")

    if discounted_subtotal >= 10000:
        shipping_cents = 0

    return discounted_subtotal + shipping_cents

What does this code do?

• The subtotal_cents < 0 guard rejects a negative invoice subtotal.
• The subtotal_cents >= 5000 condition includes the exact member-discount threshold.
• The discount_cents value holds the member discount in integer cents.
• The discounted_subtotal value holds the subtotal after the discount.
• The destination branch assigns the shipping charge in integer cents.
• The free-shipping rule checks discounted_subtotal before returning the final total.

• Save fixtures/clean/invoice.py by pressing Cmd+S on macOS or Ctrl+S on Windows.
• Confirm that invoice.py is listed inside fixtures/clean in the Explorer.

Invoice File in the Wrong Place?

Move invoice.py into the clean folder if the Explorer shows it elsewhere. Its full path must be fixtures/clean/invoice.py.

Ask for help if the folder tree does not match: Help me place invoice.py inside fixtures/clean in Visual Studio Code.

✔️ Awesome, I've got everything!

Your clean invoice calculation is saved inside fixtures/clean/invoice.py.

ⓧ I'd like to double check the full code

• Compare fixtures/clean/invoice.py with this complete file.

def calculate_total(subtotal_cents, is_member, destination):
    if subtotal_cents < 0:
        raise ValueError("subtotal_cents must be non-negative")

    discount_cents = 0
    if is_member and subtotal_cents >= 5000:
        discount_cents = subtotal_cents // 10

    discounted_subtotal = subtotal_cents - discount_cents

    if destination == "domestic":
        shipping_cents = 500
    elif destination == "international":
        shipping_cents = 1500
    else:
        raise ValueError("destination must be domestic or international")

    if discounted_subtotal >= 10000:
        shipping_cents = 0

    return discounted_subtotal + shipping_cents

Add and run the workspace checker

Automated tests turn each invoice rule into an executable requirement. Python's unittest framework reports whether all three requirements remain true.

• Right-click the clean folder in the Explorer.
• Select New File.
• Enter check_invoice.py.
• Press Enter.
• Add the three rule checks to fixtures/clean/check_invoice.py by pasting this code:

import unittest

from invoice import calculate_total


class InvoiceRulesTest(unittest.TestCase):
    def test_negative_subtotal_is_rejected(self):
        with self.assertRaises(ValueError):
            calculate_total(-1, False, "domestic")

    def test_member_discount_starts_at_inclusive_threshold(self):
        self.assertEqual(calculate_total(5000, True, "domestic"), 5000)

    def test_free_shipping_uses_discounted_subtotal(self):
        self.assertEqual(calculate_total(10000, True, "domestic"), 9500)


if __name__ == "__main__":
    unittest.main(verbosity=2)

How Does the Checker Grade the Fixture?

• The import gives the checker access to calculate_total() from the file beside it.
• The first test expects a negative subtotal to raise ValueError.
• The second test checks that the member discount starts at exactly 5000 cents.
• The third test proves that free shipping uses the discounted subtotal.
• The final block runs every test with detailed output when you execute the checker directly.

• Save fixtures/clean/check_invoice.py by pressing Cmd+S on macOS or Ctrl+S on Windows.
• Right-click the fixtures/clean folder in the Explorer.
• Select Open in Integrated Terminal.

The terminal should open with fixtures/clean as its current folder.

Before you run the checker, do you expect every clean invoice rule to pass?

macOS

• Run the clean checker with the current Python executable by running this command:

python3 check_invoice.py

You should see Ran 3 tests followed by OK.

Windows

• Run the clean checker with the current Python executable by running this command:

py check_invoice.py

You should see Ran 3 tests followed by OK.

Linux

• Run the clean checker with the current Python executable by running this command:

python3 check_invoice.py

You should see Ran 3 tests followed by OK.

Checker Not Passing?

Confirm that the terminal is running inside fixtures/clean. Both Python files must be in that folder.

Compare each file with the full-code tabs if a test fails. You can also ask for help diagnosing the clean checker.

That baseline is secured: your clean fixture now proves all three invoice rules before any mutation.

✔️ Awesome, I've got everything!

Your workspace checker passes all three tests from fixtures/clean.

ⓧ I'd like to double check the full code

• Compare fixtures/clean/check_invoice.py with this complete file.

import unittest

from invoice import calculate_total


class InvoiceRulesTest(unittest.TestCase):
    def test_negative_subtotal_is_rejected(self):
        with self.assertRaises(ValueError):
            calculate_total(-1, False, "domestic")

    def test_member_discount_starts_at_inclusive_threshold(self):
        self.assertEqual(calculate_total(5000, True, "domestic"), 5000)

    def test_free_shipping_uses_discounted_subtotal(self):
        self.assertEqual(calculate_total(10000, True, "domestic"), 9500)


if __name__ == "__main__":
    unittest.main(verbosity=2)

Your passing control is ready. Next, you will preregister the three breaks before generating the saved broken fixture.

Preregister the Broken Fixture

The clean invoice checker now passes all three rules. That gives your comparison a proven starting point.

Before either Claude Code repair, the experiment needs a frozen list of faults. Preregistration fixes those faults plus the acceptance rule before model behavior can influence the test.

In this step, get ready to:

• Record the three exact mutations before generating the broken fixture.
• Build a guarded Python script that generates the fixture from the passing control.
• Run the generator to prove the fixture fails exactly three tests.

Write the preregistration first

Choosing failures after seeing the repairs would invalidate the comparison. Changing the acceptance rule after a run would create the same problem.

• Create preregistered_failures.md beside access-attestation.json using your editor's file-creation control.
• Paste this preregistration into preregistered_failures.md:

# Preregistered Failures and Decision Rule

The following mutations will be applied before either repair run.

1. Negative subtotal rule
   - Mutation: change `subtotal_cents < 0` to `subtotal_cents < -1000`.
   - Expected proof: `test_negative_subtotal_is_rejected` fails.

2. Inclusive member threshold rule
   - Mutation: change `subtotal_cents >= 5000` to `subtotal_cents > 5000`.
   - Expected proof: `test_member_discount_starts_at_inclusive_threshold` fails.

3. Post-discount free-shipping rule
   - Mutation: change `discounted_subtotal >= 10000` to `subtotal_cents >= 10000`.
   - Expected proof: `test_free_shipping_uses_discounted_subtotal` fails.

Decision rule written before the runs:

- Use medium only if the high run passes, the medium run passes, and medium takes no more than one additional turn compared with high.
- Keep high for every other result, including a failed check, changed pre-run evidence, mismatched inputs, a missing measurement, or sessions that were not fresh.

What does this record protect?

• Each mutation maps to one existing test method.
• The high-effort repair must pass.
• The medium-effort repair must pass.
• The medium-effort repair may use at most one additional turn.
• Every other outcome keeps high.

• Save preregistered_failures.md.
• Confirm preregistered_failures.md appears beside access-attestation.json in your editor.

Your three faults now exist before mutation. The decision gate is also fixed before either repair.

Preregistration file missing or incomplete?

• Check that the filename is exactly preregistered_failures.md.
• Check that each mutation includes its original expression.
• Check that each mutation includes its replacement expression.
• Compare the file with the reference using help me check my preregistration.

✔️ Awesome, I've got everything!

Your preregistration now locks the three mutations plus the decision rule before fixture generation.

ⓧ I'd like to double check the full code

# Preregistered Failures and Decision Rule

The following mutations will be applied before either repair run.

1. Negative subtotal rule
   - Mutation: change `subtotal_cents < 0` to `subtotal_cents < -1000`.
   - Expected proof: `test_negative_subtotal_is_rejected` fails.

2. Inclusive member threshold rule
   - Mutation: change `subtotal_cents >= 5000` to `subtotal_cents > 5000`.
   - Expected proof: `test_member_discount_starts_at_inclusive_threshold` fails.

3. Post-discount free-shipping rule
   - Mutation: change `discounted_subtotal >= 10000` to `subtotal_cents >= 10000`.
   - Expected proof: `test_free_shipping_uses_discounted_subtotal` fails.

Decision rule written before the runs:

- Use medium only if the high run passes, the medium run passes, and medium takes no more than one additional turn compared with high.
- Keep high for every other result, including a failed check, changed pre-run evidence, mismatched inputs, a missing measurement, or sessions that were not fresh.

What should match?

Compare every mutation plus both decision bullets with your saved file. These exact statements become the experiment's pre-run evidence.

Build the guarded mutation script

The generator turns the written plan into a reproducible broken workspace. Its safeguards stop generation when the plan is incomplete or the clean control fails.

• Create make_fixture.py beside preregistered_failures.md using your editor's file-creation control.
• Paste the imports plus fixture definitions into make_fixture.py:

from pathlib import Path
import shutil
import subprocess
import sys


ROOT = Path(__file__).resolve().parent
CLEAN = ROOT / "fixtures" / "clean"
BROKEN = ROOT / "fixtures" / "broken"
PREREGISTRATION = ROOT / "preregistered_failures.md"

REQUIRED_TEXT = (
    "subtotal_cents < 0` to `subtotal_cents < -1000",
    "subtotal_cents >= 5000` to `subtotal_cents > 5000",
    "discounted_subtotal >= 10000` to `subtotal_cents >= 10000",
    "medium takes no more than one additional turn",
)

MUTATIONS = (
    ("if subtotal_cents < 0:", "if subtotal_cents < -1000:"),
    ("subtotal_cents >= 5000", "subtotal_cents > 5000"),
    ("if discounted_subtotal >= 10000:", "if subtotal_cents >= 10000:"),
)

What do these definitions control?

• The path constants locate the clean control plus the generated broken fixture.
• The REQUIRED_TEXT tuple describes the evidence that must already exist.
• The MUTATIONS tuple pairs each clean expression with its exact broken replacement.

• Save make_fixture.py.
• Confirm make_fixture.py appears beside preregistered_failures.md in your editor.

Definitions showing a syntax warning?

• Check that every path uses the capitalized constant names shown above.
• Check that both tuples end with a closing parenthesis.
• Ask for targeted help using help me find the syntax problem.

• Place your cursor below the closing parenthesis of MUTATIONS in make_fixture.py.
• Paste the preregistration plus clean-control gate:

def main():
    if not PREREGISTRATION.exists():
        raise SystemExit("Create preregistered_failures.md before making the fixture.")

    preregistered = PREREGISTRATION.read_text(encoding="utf-8")
    missing = [text for text in REQUIRED_TEXT if text not in preregistered]
    if missing:
        raise SystemExit("Preregistration is incomplete: " + ", ".join(missing))

    clean_check = subprocess.run(
        [sys.executable, "check_invoice.py"],
        cwd=CLEAN,
        capture_output=True,
        text=True,
    )
    if clean_check.returncode != 0:
        print(clean_check.stdout)
        print(clean_check.stderr)
        raise SystemExit("The clean control must pass before mutation.")


if __name__ == "__main__":
    main()

How does the preflight work?

• The first check stops if preregistered_failures.md is missing.
• The second check stops if any required mutation or decision phrase is absent.
• The subprocess uses the current Python executable to run the clean checker.
• A failing clean control stops mutation before the broken fixture is created.

• Save make_fixture.py.
• Select your platform below to run the preregistration plus clean-control gate.

macOS

• Run the gate from the project directory with this command:

python3 make_fixture.py

What should happen?

Your terminal returns to its prompt without a stop message. This proves the preregistration plus clean-control checks passed.

🙋‍♀️ Did the preflight stop?

Read the stop message to identify the failed gate. Use help me diagnose the fixture preflight for targeted help.

Windows

• Run the gate from the project directory with this command:

py make_fixture.py

What should happen?

Your terminal returns to its prompt without a stop message. This proves the preregistration plus clean-control checks passed.

🙋‍♀️ Did the preflight stop?

Read the stop message to identify the failed gate. Use help me diagnose the fixture preflight for targeted help.

Linux

• Run the gate from the project directory with this command:

python3 make_fixture.py

What should happen?

Your terminal returns to its prompt without a stop message. This proves the preregistration plus clean-control checks passed.

🙋‍♀️ Did the preflight stop?

Read the stop message to identify the failed gate. Use help me diagnose the fixture preflight for targeted help.

The gate is working. Your written plan plus passing control now protect the mutation process.

• Find if __name__ == "__main__": near the bottom of make_fixture.py.
• Insert the fixture-copy plus mutation block immediately above that line:

    if BROKEN.exists():
        shutil.rmtree(BROKEN)
    shutil.copytree(CLEAN, BROKEN)

    invoice_path = BROKEN / "invoice.py"
    source = invoice_path.read_text(encoding="utf-8")
    for old, new in MUTATIONS:
        if source.count(old) != 1:
            raise SystemExit("Mutation target was not unique: " + old)
        source = source.replace(old, new, 1)
    invoice_path.write_text(source, encoding="utf-8")

How is the broken copy generated?

• An existing fixtures/broken folder is removed before each generation.
• The passing fixtures/clean folder becomes the source for a fresh copy.
• Each original expression must occur exactly once before replacement.
• Only fixtures/broken/invoice.py receives the three substitutions.

• Save make_fixture.py.
• Select your platform below to generate the first broken copy.

macOS

• Generate fixtures/broken with this command:

python3 make_fixture.py

What changed?

The command recreates fixtures/broken from the clean fixture. It applies the three preregistered substitutions to the copied invoice file.

Windows

• Generate fixtures/broken with this command:

py make_fixture.py

What changed?

The command recreates fixtures/broken from the clean fixture. It applies the three preregistered substitutions to the copied invoice file.

Linux

• Generate fixtures/broken with this command:

python3 make_fixture.py

What changed?

The command recreates fixtures/broken from the clean fixture. It applies the three preregistered substitutions to the copied invoice file.

• Expand fixtures/broken in your editor's file explorer.

You should see invoice.py plus check_invoice.py inside the copied fixture.

Broken fixture folder missing?

• Check the terminal for a message about a missing preregistration entry.
• Check that fixtures/clean still contains both source files.
• Use help me trace the missing fixture for targeted help.

Generate and verify the broken fixture

Creating a broken copy is only half the control. The generator must reject every result except the three failures you preregistered.

• Find invoice_path.write_text(source, encoding="utf-8") in make_fixture.py.
• Insert the grading plus readiness block immediately below that line:

    broken_check = subprocess.run(
        [sys.executable, "check_invoice.py"],
        cwd=BROKEN,
        capture_output=True,
        text=True,
    )
    output = broken_check.stdout + broken_check.stderr
    print(output)

    if broken_check.returncode == 0:
        raise SystemExit("The broken fixture unexpectedly passed.")
    if "FAILED (failures=3)" not in output:
        raise SystemExit("Expected exactly three failures from the saved fixture.")

    cache_dir = BROKEN / "__pycache__"
    if cache_dir.exists():
        shutil.rmtree(cache_dir)

    print("BROKEN FIXTURE READY: exactly 3 failures")

How is the fixture graded?

• The copied checker runs inside fixtures/broken.
• An unexpected pass stops the generator.
• Any failure count other than three also stops the generator.
• The generated cache folder is removed before the readiness message is printed.

• Save the completed make_fixture.py.

Before you run the completed generator, do you expect the checker to report one combined failure or three independent failures?

macOS

• Generate and verify the saved broken fixture with this command:

python3 make_fixture.py

What should you see?

Your output lists test_negative_subtotal_is_rejected, test_member_discount_starts_at_inclusive_threshold, plus test_free_shipping_uses_discounted_subtotal as failures.

The final line reads BROKEN FIXTURE READY: exactly 3 failures.

Windows

• Generate and verify the saved broken fixture with this command:

py make_fixture.py

What should you see?

Your output lists test_negative_subtotal_is_rejected, test_member_discount_starts_at_inclusive_threshold, plus test_free_shipping_uses_discounted_subtotal as failures.

The final line reads BROKEN FIXTURE READY: exactly 3 failures.

Linux

• Generate and verify the saved broken fixture with this command:

python3 make_fixture.py

What should you see?

Your output lists test_negative_subtotal_is_rejected, test_member_discount_starts_at_inclusive_threshold, plus test_free_shipping_uses_discounted_subtotal as failures.

The final line reads BROKEN FIXTURE READY: exactly 3 failures.

That intentional failure is now controlled. Your saved fixture contains exactly the three faults declared before mutation.

Did the generator reject the fixture?

• Compare each MUTATIONS pair with the clean expressions in fixtures/clean/invoice.py.
• Confirm the copied checker remains unchanged.
• Remove any manual edits from fixtures/broken.
• Use help me diagnose the failure count for targeted help.

✔️ Awesome, I've got everything!

Your preregistration plus broken fixture now form a reproducible input for both repair runs.

ⓧ I'd like to double check the full code

Compare your completed make_fixture.py with this reference.

from pathlib import Path
import shutil
import subprocess
import sys


ROOT = Path(__file__).resolve().parent
CLEAN = ROOT / "fixtures" / "clean"
BROKEN = ROOT / "fixtures" / "broken"
PREREGISTRATION = ROOT / "preregistered_failures.md"

REQUIRED_TEXT = (
    "subtotal_cents < 0` to `subtotal_cents < -1000",
    "subtotal_cents >= 5000` to `subtotal_cents > 5000",
    "discounted_subtotal >= 10000` to `subtotal_cents >= 10000",
    "medium takes no more than one additional turn",
)

MUTATIONS = (
    ("if subtotal_cents < 0:", "if subtotal_cents < -1000:"),
    ("subtotal_cents >= 5000", "subtotal_cents > 5000"),
    ("if discounted_subtotal >= 10000:", "if subtotal_cents >= 10000:"),
)


def main():
    if not PREREGISTRATION.exists():
        raise SystemExit("Create preregistered_failures.md before making the fixture.")

    preregistered = PREREGISTRATION.read_text(encoding="utf-8")
    missing = [text for text in REQUIRED_TEXT if text not in preregistered]
    if missing:
        raise SystemExit("Preregistration is incomplete: " + ", ".join(missing))

    clean_check = subprocess.run(
        [sys.executable, "check_invoice.py"],
        cwd=CLEAN,
        capture_output=True,
        text=True,
    )
    if clean_check.returncode != 0:
        print(clean_check.stdout)
        print(clean_check.stderr)
        raise SystemExit("The clean control must pass before mutation.")

    if BROKEN.exists():
        shutil.rmtree(BROKEN)
    shutil.copytree(CLEAN, BROKEN)

    invoice_path = BROKEN / "invoice.py"
    source = invoice_path.read_text(encoding="utf-8")
    for old, new in MUTATIONS:
        if source.count(old) != 1:
            raise SystemExit("Mutation target was not unique: " + old)
        source = source.replace(old, new, 1)
    invoice_path.write_text(source, encoding="utf-8")

    broken_check = subprocess.run(
        [sys.executable, "check_invoice.py"],
        cwd=BROKEN,
        capture_output=True,
        text=True,
    )
    output = broken_check.stdout + broken_check.stderr
    print(output)

    if broken_check.returncode == 0:
        raise SystemExit("The broken fixture unexpectedly passed.")
    if "FAILED (failures=3)" not in output:
        raise SystemExit("Expected exactly three failures from the saved fixture.")

    cache_dir = BROKEN / "__pycache__"
    if cache_dir.exists():
        shutil.rmtree(cache_dir)

    print("BROKEN FIXTURE READY: exactly 3 failures")


if __name__ == "__main__":
    main()

What should this script guarantee?

The script validates the preconditions before copying the clean fixture. It accepts only a generated fixture with exactly three failures.

Compare your generated fixtures/broken/invoice.py with this reference.

def calculate_total(subtotal_cents, is_member, destination):
    if subtotal_cents < -1000:
        raise ValueError("subtotal_cents must be non-negative")

    discount_cents = 0
    if is_member and subtotal_cents > 5000:
        discount_cents = subtotal_cents // 10

    discounted_subtotal = subtotal_cents - discount_cents

    if destination == "domestic":
        shipping_cents = 500
    elif destination == "international":
        shipping_cents = 1500
    else:
        raise ValueError("destination must be domestic or international")

    if subtotal_cents >= 10000:
        shipping_cents = 0

    return discounted_subtotal + shipping_cents

Which rules changed?

The negative guard is weaker. The member threshold is exclusive.

Free shipping now uses the pre-discount subtotal. These are the three preregistered faults.

Compare your generated fixtures/broken/check_invoice.py with this unchanged checker.

import unittest

from invoice import calculate_total


class InvoiceRulesTest(unittest.TestCase):
    def test_negative_subtotal_is_rejected(self):
        with self.assertRaises(ValueError):
            calculate_total(-1, False, "domestic")

    def test_member_discount_starts_at_inclusive_threshold(self):
        self.assertEqual(calculate_total(5000, True, "domestic"), 5000)

    def test_free_shipping_uses_discounted_subtotal(self):
        self.assertEqual(calculate_total(10000, True, "domestic"), 9500)


if __name__ == "__main__":
    unittest.main(verbosity=2)

Why must the checker match?

The generator copies this checker without modification. Its three existing rules remain the grading authority for both repairs.

Your experiment now has a frozen plan plus a reproducible three-failure fixture. Next, you will build the harness that gives both effort settings the same starting bytes.

Build and Audit the Harness

Your saved broken fixture now fails exactly three preregistered checks. That gives both repair runs a controlled starting problem.

The next risk is experimental drift between the two runs. In this step, you will build a harness that protects the evidence before Claude Code receives either repair prompt.

In this step, get ready to:

• Build fail-closed preflight checks for access and authentication.
• Configure isolated high-effort and medium-effort repair processes with identical inputs.
• Prepare matching workspaces and verify the evidence snapshots before any model call.

Assemble preflight and provenance controls

A trustworthy harness must stop when any required control is missing. This fail-closed decision prevents a metered route or mismatched fixture from reaching a model call.

• In the Explorer sidebar in Visual Studio Code, create benchmark.py beside make_fixture.py.
• Confirm that benchmark.py appears beside the existing fixture folders in the Explorer sidebar.
• Use the full-file reference below to populate benchmark.py.

✔️ Awesome, I've got everything!

Your harness file contains the preflight, isolation, grading, telemetry, reporting, and decision logic. Save it before continuing.

ⓧ I'd like to double check the full code

• Compare benchmark.py with this complete file.

1. Controls and preflight

import argparse
import hashlib
import json
import os
import platform
from pathlib import Path
import shutil
import subprocess
import sys
import time

ROOT = Path(__file__).resolve().parent
ATTESTATION = ROOT / "access-attestation.json"
PREREGISTRATION = ROOT / "preregistered_failures.md"
BROKEN = ROOT / "fixtures" / "broken"
RUNS = ROOT / "runs"
RESULTS = ROOT / "results"
MODEL = "claude-fable-5-1"
MAX_TURNS = "8"
TIMEOUT_SECONDS = 420

INCLUDED_PLAN_TIERS = {
    "Max plan",
    "premium seat on Team plan",
    "premium seat on seat-based Enterprise plan",
}

BLOCKED_VALUE_ENV_VARS = (
    "ANTHROPIC_API_KEY",
    "ANTHROPIC_AUTH_TOKEN",
    "ANTHROPIC_AWS_API_KEY",
    "ANTHROPIC_FOUNDRY_API_KEY",
    "ANTHROPIC_FOUNDRY_AUTH_TOKEN",
    "AWS_BEARER_TOKEN_BEDROCK",
    "ANTHROPIC_BASE_URL",
    "ANTHROPIC_PROFILE",
)

BLOCKED_PROVIDER_ENV_VARS = (
    "CLAUDE_CODE_USE_ANTHROPIC_AWS",
    "CLAUDE_CODE_USE_BEDROCK",
    "CLAUDE_CODE_USE_FOUNDRY",
    "CLAUDE_CODE_USE_MANTLE",
    "CLAUDE_CODE_USE_VERTEX",
)

REQUIRED_PREREGISTRATION_TEXT = (
    "subtotal_cents < 0` to `subtotal_cents < -1000",
    "subtotal_cents >= 5000` to `subtotal_cents > 5000",
    "discounted_subtotal >= 10000` to `subtotal_cents >= 10000",
    "medium takes no more than one additional turn",
)

PROMPT = (
    "Use the BENCHMARK_PYTHON environment variable to run check_invoice.py "
    "in this directory. Repair only invoice.py until the check passes. "
    "Do not edit check_invoice.py. Keep running the check and repairing "
    "invoice.py until the check passes, then stop."
)


def write_json(path, value):
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(value, indent=2) + "\n", encoding="utf-8")


def file_hash(path):
    return hashlib.sha256(path.read_bytes()).hexdigest()


def workspace_hash(path):
    digest = hashlib.sha256()
    for name in ("invoice.py", "check_invoice.py"):
        file_path = path / name
        digest.update(name.encode("utf-8"))
        digest.update(file_path.read_bytes())
    return digest.hexdigest()


def run_check(path):
    completed = subprocess.run(
        [sys.executable, "check_invoice.py"],
        cwd=path,
        capture_output=True,
        text=True,
    )
    return {
        "passed": completed.returncode == 0,
        "returncode": completed.returncode,
        "output": completed.stdout + completed.stderr,
    }


def park(reason):
    RESULTS.mkdir(parents=True, exist_ok=True)
    message = "PARKED\n\n" + reason + "\nNo model call was started.\n"
    (RESULTS / "PARKED.md").write_text(message, encoding="utf-8")
    raise SystemExit(message)


def truthy_environment_value(value):
    return str(value).strip().lower() in {"1", "true", "yes", "on"}


def validate_preregistration():
    if not PREREGISTRATION.exists():
        park("The preregistration is missing.")
    text = PREREGISTRATION.read_text(encoding="utf-8")
    missing = [item for item in REQUIRED_PREREGISTRATION_TEXT if item not in text]
    if missing:
        park("The preregistration does not contain all three breaks and the decision rule.")


def preflight():
    if not ATTESTATION.exists():
        park("The included-access attestation is missing.")

    attestation = json.loads(ATTESTATION.read_text(encoding="utf-8"))
    plan_tier = attestation.get("plan_tier")
    if plan_tier not in INCLUDED_PLAN_TIERS:
        park("Replace the plan_tier placeholder with one exact eligible included tier.")
    if attestation.get("fable_access") != "included_subscription":
        park("Fable access was not confirmed as subscription-included.")
    if attestation.get("usage_credits_enabled_for_project") is not False:
        park("Usage credits must stay disabled for this project.")
    if attestation.get("subscription_login_confirmed") is not True:
        park("Claude Code subscription authentication was not confirmed.")
    checked_at = attestation.get("checked_at")
    if not isinstance(checked_at, str) or checked_at.startswith("replace"):
        park("Replace checked_at with the date of the plan check.")

    active_routes = [name for name in BLOCKED_VALUE_ENV_VARS if os.environ.get(name)]
    active_routes.extend(
        name
        for name in BLOCKED_PROVIDER_ENV_VARS
        if truthy_environment_value(os.environ.get(name, ""))
    )
    if active_routes:
        park("A metered or routed credential path is active: " + ", ".join(active_routes))
    if os.environ.get("CLAUDE_CODE_EFFORT_LEVEL"):
        park("CLAUDE_CODE_EFFORT_LEVEL would override the --effort flag.")

    claude = shutil.which("claude")
    if not claude:
        park("Claude Code was not found on PATH.")

    version = subprocess.run(
        [claude, "--version"], capture_output=True, text=True
    )
    if version.returncode != 0:
        park("claude --version did not complete successfully.")

    auth = subprocess.run(
        [claude, "auth", "status", "--text"],
        capture_output=True,
        text=True,
    )
    if auth.returncode != 0:
        park("Claude Code is not authenticated.")

    record = {
        "plan_tier": plan_tier,
        "fable_access": "included_subscription",
        "usage_credits_enabled_for_project": False,
        "known_metered_override_detected": False,
        "billing_setup_requested_by_project": False,
        "platform": platform.platform(),
        "python_version": platform.python_version(),
        "claude_version_from_command": version.stdout.strip(),
        "authenticated": True,
        "attestation": attestation,
    }
    write_json(RESULTS / "preflight.json", record)
    return claude, record

2. Preparation and runs

Prepare the workspaces

def prepare():
    claude, preflight_record = preflight()
    validate_preregistration()

    if not BROKEN.exists():
        raise SystemExit("Run make_fixture.py before preparing the benchmark.")

    broken_check = run_check(BROKEN)
    if broken_check["passed"]:
        raise SystemExit("The saved broken fixture must fail before repair.")

    snapshots = RESULTS / "pre-artifacts"
    if snapshots.exists():
        shutil.rmtree(snapshots)
    snapshots.mkdir(parents=True)
    shutil.copy2(ATTESTATION, snapshots / ATTESTATION.name)
    shutil.copy2(PREREGISTRATION, snapshots / PREREGISTRATION.name)

    pre_artifacts = {
        "attestation_sha256": file_hash(ATTESTATION),
        "preregistration_sha256": file_hash(PREREGISTRATION),
        "snapshotted_before_first_model_call": True,
        "three_breaks_present_before_first_model_call": True,
        "decision_rule_present_before_first_model_call": True,
    }

    RUNS.mkdir(parents=True, exist_ok=True)
    destinations = {
        "default_high": RUNS / "default_high",
        "lower_medium": RUNS / "lower_medium",
    }

    for destination in destinations.values():
        if destination.exists():
            shutil.rmtree(destination)
        shutil.copytree(BROKEN, destination)
        cache_dir = destination / "__pycache__"
        if cache_dir.exists():
            shutil.rmtree(cache_dir)

    hashes = {name: workspace_hash(path) for name, path in destinations.items()}
    if len(set(hashes.values())) != 1:
        raise SystemExit("Repair workspaces are not byte-identical.")

    for name, path in destinations.items():
        if run_check(path)["passed"]:
            raise SystemExit(name + " unexpectedly passes before repair.")

    pre_run_evidence = {
        "preflight": preflight_record,
        "pre_artifacts": pre_artifacts,
        "starting_workspace_hashes": hashes,
        "identical_starting_workspaces": True,
        "both_starting_checks_failed": True,
    }
    write_json(RESULTS / "pre-run-evidence.json", pre_run_evidence)

    print("Plan tier:", preflight_record["plan_tier"])
    print("Claude Code version:", preflight_record["claude_version_from_command"])
    print("Pre-artifacts snapshotted before the first model call.")
    print("Workspace hashes match:", hashes["default_high"])
    print("Both pre-run checks fail.")
    print("PREPARE OK")
    return claude, destinations, hashes, preflight_record, pre_artifacts

Run one isolated repair

def integer(value):
    try:
        return int(value)
    except (TypeError, ValueError):
        return 0


def whole_tree_tokens(payload):
    model_usage = payload.get("modelUsage")
    if not isinstance(model_usage, dict) or not model_usage:
        return None

    total = 0
    for usage in model_usage.values():
        if not isinstance(usage, dict):
            return None
        total += integer(usage.get("inputTokens"))
        total += integer(usage.get("outputTokens"))
        total += integer(usage.get("cacheReadInputTokens"))
        total += integer(usage.get("cacheCreationInputTokens"))
    return total


def decode_output(value):
    if value is None:
        return ""
    if isinstance(value, bytes):
        return value.decode("utf-8", errors="replace")
    return value


def run_one(claude, label, effort, workspace):
    tools = "PowerShell,Bash,Edit,Read" if platform.system() == "Windows" else "Bash,Edit,Read"
    command = [
        claude,
        "--restricted",
        "--model",
        MODEL,
        "--effort",
        effort,
        "--tools",
        tools,
        "--allowedTools",
        tools,
        "--disallowedTools",
        "mcp__*",
        "--permission-mode",
        "dontAsk",
        "--output-format",
        "json",
        "--no-session-persistence",
        "--max-turns",
        MAX_TURNS,
        "-p",
        PROMPT,
    ]

    environment = os.environ.copy()
    environment.pop("CLAUDE_CODE_EFFORT_LEVEL", None)
    environment["CLAUDE_CODE_SKIP_PROMPT_HISTORY"] = "1"
    environment["BENCHMARK_PYTHON"] = sys.executable

    started = time.perf_counter()
    timed_out = False
    try:
        completed = subprocess.run(
            command,
            cwd=workspace,
            env=environment,
            capture_output=True,
            text=True,
            timeout=TIMEOUT_SECONDS,
        )
        stdout = completed.stdout
        stderr = completed.stderr
        returncode = completed.returncode
    except subprocess.TimeoutExpired as error:
        timed_out = True
        stdout = decode_output(error.stdout)
        stderr = decode_output(error.stderr)
        returncode = None
    wall_clock_minutes = (time.perf_counter() - started) / 60

    raw_dir = RESULTS / "raw"
    raw_dir.mkdir(parents=True, exist_ok=True)
    (raw_dir / (label + ".stdout.json")).write_text(stdout, encoding="utf-8")
    (raw_dir / (label + ".stderr.txt")).write_text(stderr, encoding="utf-8")

    try:
        payload = json.loads(stdout)
    except json.JSONDecodeError:
        payload = {}

    protected_hash = file_hash(BROKEN / "check_invoice.py")
    checker = workspace / "check_invoice.py"
    tests_unchanged = checker.exists() and file_hash(checker) == protected_hash
    check = run_check(workspace)
    cli_success = (
        not timed_out
        and returncode == 0
        and payload.get("subtype") == "success"
        and payload.get("is_error") is False
    )
    passed = bool(cli_success and tests_unchanged and check["passed"])

    summary = {
        "label": label,
        "effort": effort,
        "passed": passed,
        "turns": payload.get("num_turns"),
        "total_tokens": whole_tree_tokens(payload),
        "cost_estimate_usd": payload.get("total_cost_usd"),
        "wall_clock_minutes": round(wall_clock_minutes, 3),
        "session_id": payload.get("session_id"),
        "models_used": sorted((payload.get("modelUsage") or {}).keys()),
        "cli_success": cli_success,
        "tests_unchanged": tests_unchanged,
        "checker_returncode": check["returncode"],
        "pass_fail_source": "check_invoice.py process return code",
        "timed_out": timed_out,
    }
    write_json(RESULTS / (label + ".json"), summary)
    return summary

3. Decision and CLI

Calculate and report the result

def observed_deltas(data):
    default = data["runs"]["default_high"]
    lower = data["runs"]["lower_medium"]

    def subtract(left, right, digits=None):
        if not isinstance(left, (int, float)) or not isinstance(right, (int, float)):
            return None
        value = left - right
        return round(value, digits) if digits is not None else value

    return {
        "tokens_saved_high_minus_medium": subtract(
            default.get("total_tokens"), lower.get("total_tokens")
        ),
        "client_estimated_usd_saved_high_minus_medium": subtract(
            default.get("cost_estimate_usd"), lower.get("cost_estimate_usd"), 6
        ),
        "wall_clock_minutes_saved_high_minus_medium": subtract(
            default.get("wall_clock_minutes"), lower.get("wall_clock_minutes"), 3
        ),
        "medium_turn_delta": subtract(lower.get("turns"), default.get("turns")),
    }


def decision_for(data):
    default = data["runs"]["default_high"]
    lower = data["runs"]["lower_medium"]
    turns_exist = isinstance(default.get("turns"), int) and isinstance(
        lower.get("turns"), int
    )
    metrics_exist = all(
        run.get(field) is not None
        for run in (default, lower)
        for field in (
            "turns",
            "total_tokens",
            "cost_estimate_usd",
            "wall_clock_minutes",
        )
    )
    criterion_met = bool(
        default.get("passed") is True
        and lower.get("passed") is True
        and data.get("fresh_sessions") is True
        and data.get("identical_starting_workspaces") is True
        and data.get("pre_artifacts_unchanged") is True
        and data.get("script_graded") is True
        and turns_exist
        and metrics_exist
        and lower["turns"] <= default["turns"] + 1
    )
    return "USE_MEDIUM" if criterion_met else "KEEP_HIGH"


def write_report(data):
    default = data["runs"]["default_high"]
    lower = data["runs"]["lower_medium"]
    deltas = observed_deltas(data)
    decision = decision_for(data)
    preflight_record = data["preflight"]

    lines = [
        "# Controlled Effort Comparison",
        "",
        "## Operator evidence",
        "",
        "- Plan tier returned by the manual check: " + preflight_record["plan_tier"],
        "- Claude Code version returned by `claude --version`: "
        + preflight_record["claude_version_from_command"],
        "- Known metered override detected: "
        + str(preflight_record["known_metered_override_detected"]),
        "- Billing setup requested by this project: "
        + str(preflight_record["billing_setup_requested_by_project"]),
        "- Pre-artifacts snapshotted before first model call: "
        + str(data["pre_artifacts"]["snapshotted_before_first_model_call"]),
        "- Three breaks present before first model call: "
        + str(data["pre_artifacts"]["three_breaks_present_before_first_model_call"]),
        "- Decision rule present before first model call: "
        + str(data["pre_artifacts"]["decision_rule_present_before_first_model_call"]),
        "- Pre-artifacts unchanged through both runs: "
        + str(data["pre_artifacts_unchanged"]),
        "- Identical starting workspaces: "
        + str(data["identical_starting_workspaces"]),
        "- Fresh session IDs: " + str(data["fresh_sessions"]),
        "- Pass/fail source: check_invoice.py process return code",
        "",
        "## Ten recorded values",
        "",
        "| Run | Script pass | Turns | Total processed tokens | Client cost estimate USD | Wall-clock minutes |",
        "| --- | --- | ---: | ---: | ---: | ---: |",
    ]
    for run in (default, lower):
        lines.append(
            "| {label} | {passed} | {turns} | {total_tokens} | "
            "{cost_estimate_usd} | {wall_clock_minutes} |".format(**run)
        )

    lines.extend(
        [
            "",
            "## Observed deltas for this repair",
            "",
            "- Tokens saved, high minus medium: "
            + str(deltas["tokens_saved_high_minus_medium"]),
            "- Client-estimated USD saved, high minus medium: "
            + str(deltas["client_estimated_usd_saved_high_minus_medium"]),
            "- Wall-clock minutes saved, high minus medium: "
            + str(deltas["wall_clock_minutes_saved_high_minus_medium"]),
            "- Medium turn delta: " + str(deltas["medium_turn_delta"]),
            "",
            "The deltas describe this controlled repair. The cost figures are Claude Code client estimates, not billing statements or guaranteed future savings.",
            "",
            "Decision: " + decision,
        ]
    )
    (RESULTS / "report.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
    return decision

Run the benchmark from the CLI

def run_benchmark():
    claude, destinations, hashes, preflight_record, pre_artifacts = prepare()
    default = run_one(claude, "default_high", "high", destinations["default_high"])
    lower = run_one(claude, "lower_medium", "medium", destinations["lower_medium"])

    fresh_sessions = bool(
        default.get("session_id")
        and lower.get("session_id")
        and default["session_id"] != lower["session_id"]
    )
    pre_artifacts_unchanged = bool(
        file_hash(ATTESTATION) == pre_artifacts["attestation_sha256"]
        and file_hash(PREREGISTRATION) == pre_artifacts["preregistration_sha256"]
    )
    data = {
        "model": MODEL,
        "criterion": "medium passes within one extra turn",
        "preflight": preflight_record,
        "pre_artifacts": pre_artifacts,
        "pre_artifacts_unchanged": pre_artifacts_unchanged,
        "workspace_hashes": hashes,
        "identical_starting_workspaces": len(set(hashes.values())) == 1,
        "fresh_sessions": fresh_sessions,
        "script_graded": True,
        "runs": {
            "default_high": default,
            "lower_medium": lower,
        },
    }
    data["observed_deltas"] = observed_deltas(data)
    data["decision"] = decision_for(data)
    write_json(RESULTS / "results.json", data)
    decision = write_report(data)
    print("DECISION:", decision)
    print("Report:", RESULTS / "report.md")


def decide_existing(path):
    data = json.loads(path.read_text(encoding="utf-8"))
    print(decision_for(data))


def main():
    parser = argparse.ArgumentParser(description="Controlled Claude Code effort check")
    parser.add_argument(
        "action",
        nargs="?",
        default="run",
        choices=("prepare", "run", "decide"),
    )
    parser.add_argument("result_file", nargs="?")
    args = parser.parse_args()

    if args.action == "prepare":
        prepare()
    elif args.action == "run":
        run_benchmark()
    else:
        path = Path(args.result_file) if args.result_file else RESULTS / "results.json"
        decide_existing(path)


if __name__ == "__main__":
    main()

How is the harness organized?

• The path constants keep the attestation and preregistration tied to one project directory.
• The preflight functions park the experiment when an access or environment control fails.
• The preparation functions snapshot the pre-run records and create byte-identical repair workspaces.
• The run functions isolate each Claude Code process and preserve its raw output.
• The reporting functions compile the measurements.
• The decision function applies the preregistered rule.

• Save benchmark.py.

Why snapshot the evidence?

The harness records a SHA-256 digest for the attestation and preregistration. A changed digest proves that a pre-run record no longer matches the version used to prepare the experiment.

The snapshots also preserve readable copies under results/pre-artifacts. You can audit the evidence without relying on memory.

Audit isolation and telemetry

The two repairs must differ only by effort. The run_one() function enforces that boundary through one prompt and separate processes.

• In benchmark.py, locate MODEL = "claude-fable-5-1" to confirm both runs use the pinned model.
• Locate MAX_TURNS = "8" to confirm both processes share the eight-turn cap.
• Locate --restricted to confirm the evaluation boundary is enabled.
• Locate --no-session-persistence to confirm neither process saves a resumable session.
• Locate mcp__* to confirm every MCP tool is denied.
• Locate check_invoice.py process return code to confirm the workspace checker controls pass or fail.

Which values does each run record?

• The passed field records the protected checker result after the client exits.
• The turns field comes from num_turns in the client result.
• The total_tokens field sums the processed tokens from modelUsage.
• The cost_estimate_usd field preserves the client estimate without using it as an acceptance rule.
• The wall_clock_minutes field measures elapsed time around the complete client process.

💡 Keep the checker authoritative

The harness hashes the saved check_invoice.py before grading each workspace. A repair cannot earn a pass by editing or deleting its test.

Prepare identical workspaces

The prepare action runs every preflight control and creates both repair directories. It stops before either Claude Code repair invocation.

Before you run the audit, do you expect the two starting workspace hashes to match?

macOS

• Prepare the macOS workspaces by running this command from the project directory:

python3 benchmark.py prepare

What does this command do?

The command checks included access and authentication. It then snapshots the records and creates both run directories.

🙋‍♀️ Did the harness park?

• Read results/PARKED.md to identify the failed preflight control.
• Correct the named access or environment issue before running preparation again.
• Use this prompt for help: Help me diagnose why the harness parked before PREPARE OK.

Windows

• Prepare the Windows workspaces by running this command from the project directory:

py benchmark.py prepare

What does this command do?

The command checks included access and authentication. It then snapshots the records and creates both run directories.

🙋‍♀️ Did the harness park?

• Read results/PARKED.md to identify the failed preflight control.
• Correct the named access or environment issue before running preparation again.
• Use this prompt for help: Help me diagnose why the harness parked before PREPARE OK.

Linux

• Prepare the Linux workspaces by running this command from the project directory:

python3 benchmark.py prepare

What does this command do?

The command checks included access and authentication. It then snapshots the records and creates both run directories.

🙋‍♀️ Did the harness park?

• Read results/PARKED.md to identify the failed preflight control.
• Correct the named access or environment issue before running preparation again.
• Use this prompt for help: Help me diagnose why the harness parked before PREPARE OK.

You should see the exact plan tier and the literal Claude Code version. You should also see matching workspace hashes followed by PREPARE OK.

You have now proved the control boundary. Both repair directories contain the same broken bytes while the original fixture remains unchanged.

• In the Explorer sidebar, expand results/pre-artifacts to confirm both evidence snapshots exist.
• Open results/pre-run-evidence.json to confirm identical_starting_workspaces is true.
• Check both_starting_checks_failed to confirm its value is true.
• Compare the two values under starting_workspace_hashes to confirm they match exactly.

Your harness is ready to run both repairs under controlled conditions. Next, you will launch the high and medium processes and accept the script's measured decision.

Run the Controlled Comparison

The prepared harness has already proved that both repair workspaces start from the same broken bytes. It has also preserved the plan record from before either model call.

In this step, Claude Code runs the high and medium repairs in separate restricted processes. The workspace checker grades both repairs.

The final run converts an inherited effort default into a measured local decision. The preregistered rule decides whether medium is safe for this repair shape.

In this step, get ready to:

• Run the high-effort repair before the medium-effort repair.
• Audit the controls and ten recorded values in the operator report.
• Accept the preregistered decision for this repair shape.

Run both isolated repairs

This command uses your included Fable allowance. The harness checks the account path again and parks before a model call if any gate fails.

Each repair can take up to seven minutes because the harness limits it to 420 seconds. Quiet terminal output during that window is expected.

Before you run this, which outcome do you expect: medium passes within one extra turn of high, or it does not?

macOS

• Launch the controlled comparison from the terminal you used for preparation by running this command:

python3 benchmark.py run

What does this command do?

• The run action calls run_benchmark().
• The harness calls prepare() again so both workspaces are recreated from the saved broken fixture.
• It starts default_high before lower_medium in separate processes.
• It preserves the raw client output before writing each run summary.

• Leave the command running until the terminal prompt returns.

At completion, you'll see DECISION: followed by either USE_MEDIUM or KEEP_HIGH. You'll also see Report: with the path to results/report.md.

Comparison did not finish?

If the harness parks, read results/PARKED.md to identify the failed preflight control. Resolve that control before running the comparison again.

If a repair reaches its timeout, inspect the matching files in results/raw to find the last client output.

Help me diagnose my controlled comparison run.

Windows

• Launch the controlled comparison from the PowerShell terminal you used for preparation by running this command:

py benchmark.py run

What does this command do?

• The run action calls run_benchmark().
• The harness calls prepare() again so both workspaces are recreated from the saved broken fixture.
• It starts default_high before lower_medium in separate processes.
• It preserves the raw client output before writing each run summary.

• Leave the command running until the PowerShell prompt returns.

At completion, you'll see DECISION: followed by either USE_MEDIUM or KEEP_HIGH. You'll also see Report: with the path to results/report.md.

Comparison did not finish?

If the harness parks, read results/PARKED.md to identify the failed preflight control. Resolve that control before running the comparison again.

If a repair reaches its timeout, inspect the matching files in results/raw to find the last client output.

Help me diagnose my controlled comparison run.

Linux

• Launch the controlled comparison from the terminal you used for preparation by running this command:

python3 benchmark.py run

What does this command do?

• The run action calls run_benchmark().
• The harness calls prepare() again so both workspaces are recreated from the saved broken fixture.
• It starts default_high before lower_medium in separate processes.
• It preserves the raw client output before writing each run summary.

• Leave the command running until the terminal prompt returns.

At completion, you'll see DECISION: followed by either USE_MEDIUM or KEEP_HIGH. You'll also see Report: with the path to results/report.md.

Comparison did not finish?

If the harness parks, read results/PARKED.md to identify the failed preflight control. Resolve that control before running the comparison again.

If a repair reaches its timeout, inspect the matching files in results/raw to find the last client output.

Help me diagnose my controlled comparison run.

The core experiment is complete. You now have two isolated repairs graded by the same protected checker.

Audit the operator report

The results/report.md file keeps the decision beside the evidence that produced it. Its controls show whether the comparison stayed faithful to the preregistered design.

Pass or fail comes from check_invoice.py. The cost fields remain client estimates for this pair of runs.

• In the Visual Studio Code file sidebar, expand results.
• Expand raw inside results.
• Confirm it contains default_high.stdout.json, default_high.stderr.txt, lower_medium.stdout.json, and lower_medium.stderr.txt.
• Confirm default_high.json, lower_medium.json, results.json, and report.md are listed inside results.
• Select report.md.

• Under Operator evidence, confirm the plan tier matches access-attestation.json.
• Confirm the Claude Code version matches the literal preflight capture.
• Confirm Known metered override detected shows False.
• Confirm Billing setup requested by this project shows False.
• Confirm the provenance and isolation controls report True.
• Confirm the pass or fail source names the check_invoice.py process return code.

How to read the ten values

• The passed value records the protected checker's result.
• The turns value comes from the client's num_turns field.
• The total_tokens value sums whole-tree processed tokens from modelUsage.
• The cost_estimate_usd value records the client's total_cost_usd estimate.
• The wall_clock_minutes value records elapsed time for that repair.

• Compare the default_high row with the lower_medium row under Ten recorded values.
• Read the measurements under Observed deltas for this repair.
• Accept the verdict shown after Decision.

How to apply the decision

A USE_MEDIUM verdict supports medium for comparable deterministic repairs. A KEEP_HIGH verdict preserves high for this repair shape.

The result does not cover unrelated workloads. The client cost remains an estimate for these two runs.

You have replaced an inherited setting with an auditable local decision. Every control and measurement now stays with the verdict.

Secret mission

Prove the Decision Fails Closed

Deliberately worsen a copy of the medium-run evidence without touching the original experiment. Use the harness's local decision path to prove that unacceptable lower-effort evidence always keeps high.

Clean Up Your Resources

Clean Up Your Resources

Your local Claude Code experiment has no ongoing resource cost. Choose whether to keep its files for later calibration, pause here, or delete them.

Resources you used:

• The access record in access-attestation.json.
• The preregistered experiment rules in preregistered_failures.md.
• The fixture generator in make_fixture.py.
• The comparison harness in benchmark.py.
• The passing control plus the saved broken fixture in fixtures/.
• The high-effort plus medium-effort repair workspaces in runs/.
• The original experiment evidence plus the tampered fail-closed proof in results/.

Keep everything running

No action is needed. Choose this option if you want to preserve the complete calibration record for future review.

• Leave every listed artifact unchanged.
• Use results/report.md when you want to review the measured decision.
• Compare results/results.json with results/tampered.json when you want to revisit the fail-closed proof.

Pause - I'll come back to this later

Both isolated repair processes have finished. Pausing means keeping every local artifact in its current location.

• Close any terminal or editor windows that still show this experiment.
• Leave fixtures/, runs/, plus results/ unchanged.
• Return to results/report.md when you resume.

Delete - I don't want to use this again

Deleting these files becomes permanent after you empty your system bin. These actions target only the experiment artifacts listed above.

macOS

• Use Finder to locate the experiment folder that contains benchmark.py.
• Select access-attestation.json, preregistered_failures.md, make_fixture.py, benchmark.py, fixtures/, runs/, plus results/.
• Move the selected items to your system bin.
• Empty your system bin.
• Confirm the experiment folder no longer lists any selected item.

Windows

• Use File Explorer to locate the experiment folder that contains benchmark.py.
• Select access-attestation.json, preregistered_failures.md, make_fixture.py, benchmark.py, fixtures/, runs/, plus results/.
• Delete the selected items.
• Empty your system bin.
• Confirm the experiment folder no longer lists any selected item.

Linux

• Use your file manager to locate the experiment folder that contains benchmark.py.
• Select access-attestation.json, preregistered_failures.md, make_fixture.py, benchmark.py, fixtures/, runs/, plus results/.
• Move the selected items to your system bin.
• Empty your system bin.
• Confirm the experiment folder no longer lists any selected item.

Nice Work!

Nice Work!

You did it! You built an auditable harness that calibrates Claude Code effort through one controlled Python repair.

You've learned how to:

• Build a fail-closed access gate that parks before any model call when the route is ineligible. Its preflight record preserves the eligible tier. The same record preserves authentication state. It also preserves the installed client version.
• Use preregistration to define three faults before mutation. Byte-identical workspaces kept the inputs fixed. Fresh-session isolation prevented memory carryover.
• Apply deterministic grading through the protected Python checker. Your ten-value report converts both isolated runs into one auditable USE_MEDIUM or KEEP_HIGH decision.
• Complete the optional Secret Mission by worsening a copy of the measurements. The decision returns KEEP_HIGH without launching Claude Code. The original experiment remains unchanged.

Ready to quiz yourself?