Build a Multi-Agent Change Review Board

Build a Claude review board with specialist agents and deterministic gates.

Introduction

30 Second Summary

A software change can look safe from one angle while hiding a security gap, an outage risk, or a missing test. When one reviewer has to catch everything, those blind spots are difficult to prove.

In this project, you will build a command-line release review board with the Claude Agent SDK. Three specialist agents inspect the same change before deterministic Python decides whether the release can proceed.

What You'll Build

Your terminal will show three specialist reviewers inspecting the same risky change before a deterministic Python gate writes an auditable decision to JSON.

By the end of this project, you'll have:

  • A parallel review board that launches named security, reliability, and testing reviewers from one command.
  • A JSON audit trail you can open to trace which reviewers ran, what they found, and why the release was blocked or allowed.
  • A local policy test suite that proves high-risk findings and missing reviewers cannot produce a passing decision.
  • Secret Mission: Remove one worker temporarily and prove the board fails closed instead of treating an incomplete review as safe.

Are there any prerequisites?

You should be comfortable reading Python functions, dictionaries, async code, and JSON. No prior Agent SDK experience is required.

You need a Claude API key with API billing configured because a Claude subscription does not authenticate this SDK.

Before We Start

Before the hands-on work begins, lock in the release board's control boundary. The security reviewer, reliability reviewer, and test reviewer provide specialist advice. Deterministic Python alone makes the final release decision of GO, NEEDS_CHANGES, or BLOCK.

Set Up a Bounded Agent Workspace

Your release review board needs a reproducible workspace before it can coordinate specialist agents through the Claude Agent SDK.

A project-local virtual environment isolates the SDK from global Python packages. A session-scoped API key keeps authentication inside one PowerShell process.

In this step, get ready to:
  • Create the multi-agent-review-board workspace with an isolated Python environment.
  • Pin and install Claude Agent SDK 0.2.174.
  • Add the risky release candidate plus session-scoped API authentication.
Create the isolated workspace

The .venv folder gives this project its own Python package environment. This boundary prevents unrelated projects from changing the SDK version used by your review board.

  • Press the Windows key to open the search bar.
  • Type Windows PowerShell into the search bar.
  • Press Enter to open Windows PowerShell.
  • Move to your Desktop by running this command:
cd ~/Desktop

What does this command do?

This command makes your Desktop the starting location for the project. You will know where to find the folder after the terminal creates it.

  • Create the project folder and move into it by running these commands:
mkdir multi-agent-review-board
cd multi-agent-review-board

What do these commands do?

  • The first command creates the multi-agent-review-board folder on your Desktop.
  • The second command makes that folder the current PowerShell location.

Your PowerShell prompt should now end with the multi-agent-review-board folder name.

  • Create the project-local virtual environment by running this command:
py -m venv .venv

How does this isolate the project?

Python creates the .venv folder inside multi-agent-review-board. Packages installed after activation stay within this environment.

  • Activate the virtual environment by running this command:
.venv\Scripts\Activate.ps1

What does activation change?

Activation points Python package commands at the environment inside .venv. The prompt gains a virtual-environment prefix when activation succeeds.

✔️ I see the virtual-environment prefix

Your environment is active. Keep this PowerShell window open for the rest of the project.

ⓧ PowerShell blocked the activation script

PowerShell can restrict local scripts. A process-scoped policy change applies only to this PowerShell window.

  • Allow local scripts in this PowerShell process by running this command:
Set-ExecutionPolicy -Scope Process RemoteSigned

Why is this process-scoped?

The policy change lasts only until you close this PowerShell window. It lets the activation script run without changing the permanent machine policy.

  • Activate the virtual environment again by running this command:
.venv\Scripts\Activate.ps1

What should change now?

The activation script should now complete. Your prompt should gain a virtual-environment prefix.

  • Open the current project folder in Visual Studio Code by running this command:
code .

What does this command open?

The period represents the current folder. Visual Studio Code opens multi-agent-review-board as the workspace.

You should see multi-agent-review-board as the open folder in Visual Studio Code.

PowerShell cannot open Visual Studio Code?

If PowerShell cannot find the command, start Visual Studio Code through Windows search. Open the multi-agent-review-board folder from your Desktop.

Help me open my project folder in Visual Studio Code on Windows.

Pin and install the Agent SDK

A pinned dependency gives every future install the same SDK version. The requirements.txt file records that exact package contract.

  • Switch back to the activated PowerShell window.
  • Create requirements.txt inside multi-agent-review-board by running this command:
code requirements.txt

What does this command create?

Visual Studio Code opens requirements.txt in the current workspace. The editor creates the file when it does not already exist.

  • Add the pinned SDK dependency to requirements.txt by pasting this line:
claude-agent-sdk==0.2.174

Why pin this version?

The double equals sign requires Claude Agent SDK version 0.2.174. A future package release cannot silently change this project's behavior.

  • Save requirements.txt.
  • Confirm that requirements.txt appears in the Visual Studio Code Explorer sidebar.

Does the file look empty after saving?

Check that the package line is inside the editor for requirements.txt. Confirm that the filename does not include an extra text extension.

Help me check my requirements file in Visual Studio Code.

✔️ Awesome, I've got everything!

Great. Your requirements.txt file now pins the SDK version used by this project.

ⓧ I'd like to double check the full code

Compare your complete requirements.txt file with this reference.

claude-agent-sdk==0.2.174

What should match?

The filename must be requirements.txt. Its only line must contain the pinned package entry shown above.

The installer reads requirements.txt and places the package inside the active virtual environment. Installation output gives you an immediate record of the packages added.

  • Install the pinned SDK into .venv by running this command in the activated PowerShell window:
py -m pip install -r requirements.txt

What does this installation command do?

Python asks pip to read the package pin from requirements.txt. The active environment keeps the installed SDK inside .venv.

The terminal should finish with successful installation output. That output confirms the dependency is available to this environment.

Did the SDK installation fail?

Confirm that the PowerShell prompt still shows the virtual-environment prefix. Check that requirements.txt contains the exact pinned package line.

Help me troubleshoot the Agent SDK installation in my virtual environment.

Add the risky fixture and session key

The board needs a concrete release candidate that exposes risks across security, reliability, and testing. This intentionally weak proposal gives each future specialist evidence within its assigned domain.

  • Create release_candidate.md inside multi-agent-review-board by running this command in the activated PowerShell window:
code release_candidate.md

What does this command prepare?

Visual Studio Code opens a Markdown file named release_candidate.md. This file becomes the shared evidence source for every reviewer.

  • Add the complete release candidate to release_candidate.md by pasting this content:
# Release Candidate: Synchronous Search Index Rebuild

## Proposed change

Add `POST /admin/rebuild-index` to the maintenance service. The endpoint starts a full search-index rebuild and returns `200 OK` when the rebuild completes.

## Access check

The handler treats a request as internal when the header `X-Internal-Request` has the value `true`. No user identity, service identity, or role is checked.

## Diagnostics

When a rebuild fails, the handler writes every request header and the exception text to the application log.

## Runtime behavior

The request stays open until the full rebuild finishes. The implementation has no timeout, cancellation, concurrency limit, or duplicate-request protection.

## Rollout and rollback

Deploy to all production instances at once. Rollback means restoring the previous application container. The plan does not describe how to reverse a partially rebuilt index.

## Test plan

Send one request with `X-Internal-Request: true` and verify that the response status is `200`.

Why is this candidate useful?

  • The access check plus diagnostic logging give the security reviewer trust-boundary evidence.
  • The synchronous runtime plus incomplete rollback plan give the reliability reviewer operational evidence.
  • The single positive test gives the test reviewer a visible coverage gap.
  • Save release_candidate.md.
  • Confirm that release_candidate.md appears beside requirements.txt in the Explorer sidebar.

You should see the release title plus six review sections in the editor. The fixture now contains evidence for all three review domains.

Is part of the fixture missing?

Compare each heading with the reference below. Pay particular attention to the access check, runtime behavior, and test plan sections.

Help me compare my release candidate fixture with the required version.

✔️ Awesome, I've got everything!

Your risky release candidate is saved. All three specialist domains now have concrete evidence to inspect.

ⓧ I'd like to double check the full code

Compare your complete release_candidate.md file with this reference.

# Release Candidate: Synchronous Search Index Rebuild

## Proposed change

Add `POST /admin/rebuild-index` to the maintenance service. The endpoint starts a full search-index rebuild and returns `200 OK` when the rebuild completes.

## Access check

The handler treats a request as internal when the header `X-Internal-Request` has the value `true`. No user identity, service identity, or role is checked.

## Diagnostics

When a rebuild fails, the handler writes every request header and the exception text to the application log.

## Runtime behavior

The request stays open until the full rebuild finishes. The implementation has no timeout, cancellation, concurrency limit, or duplicate-request protection.

## Rollout and rollback

Deploy to all production instances at once. Rollback means restoring the previous application container. The plan does not describe how to reverse a partially rebuilt index.

## Test plan

Send one request with `X-Internal-Request: true` and verify that the response status is `200`.

What should match?

Every heading plus every risk statement should match the reference. These details become the evidence that the agents inspect later.

Credentials deserve care. The next command keeps your Claude API key only in the current PowerShell process.

  • Return to the activated PowerShell window.
  • Set ANTHROPIC_API_KEY for this session by running this command after replacing your-api-key with your Claude API key:
$env:ANTHROPIC_API_KEY = "your-api-key"

Why keep the key session-scoped?

The environment variable gives the SDK access to your API credentials without placing the key in a project file. Closing this PowerShell window removes the session value.

Before you run the final checks, predict whether the SDK import plus the session-key check will both succeed.

  • Verify the SDK import plus the session-scoped key by running these commands:
python -c "import claude_agent_sdk; print('SDK ready')"
Test-Path Env:ANTHROPIC_API_KEY

What do these checks prove?

  • The first command imports claude_agent_sdk from the active environment.
  • The second command checks whether ANTHROPIC_API_KEY exists without printing its secret value.

You should see SDK ready on the first line. You should see True on the next line.

Did either final check fail?

If the import fails, confirm that the virtual environment is active. Run the requirements installation again from the multi-agent-review-board folder.

If the key check returns a false result, set ANTHROPIC_API_KEY again in this PowerShell window. Keep the secret out of screenshots plus project files.

Help me diagnose my final workspace verification.

Solid work. Your isolated SDK environment now imports successfully. PowerShell also detects the session-scoped credential without revealing it.

Your bounded workspace plus risky fixture are ready. Next up, you will run one broad review and expose the coverage evidence it cannot guarantee.

Expose the Single-Agent Coverage Gap

Your isolated workspace is ready. The Claude Agent SDK imports successfully inside the active PowerShell session.

A broad review can sound convincing while leaving no proof that security, reliability, and testing each received independent attention. In this step, you'll run that single-reviewer baseline and identify the production evidence it cannot provide.

In this step, get ready to:
  • Create a read-only single-agent baseline.
  • Run the baseline against release_candidate.md.
  • Identify the missing production evidence.
Define the review boundary

The baseline gives one agent a broad objective. Its tool boundary allows file inspection while preventing terminal commands or file changes.

  • Use the new-file button in the VS Code Explorer sidebar to create app.py inside the multi-agent-review-board folder.
  • Fill app.py with the baseline by pasting this code:
import argparse
import asyncio

from claude_agent_sdk import ClaudeAgentOptions, ResultMessage, query


async def run_review(candidate: str) -> None:
    options = ClaudeAgentOptions(
        tools=["Read", "Grep", "Glob"],
        allowed_tools=["Read", "Grep", "Glob"],
        disallowed_tools=["Bash", "Write", "Edit"],
        permission_mode="dontAsk",
    )
    prompt = f"Review the release candidate at {candidate}."
    async for message in query(prompt=prompt, options=options):
        if isinstance(message, ResultMessage):
            print(message.result)


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Run a release review."
    )
    parser.add_argument("candidate", help="Path to the release candidate Markdown file")
    args = parser.parse_args()
    asyncio.run(run_review(args.candidate))


if __name__ == "__main__":
    main()

What does this code do?

  • The argparse parser collects the candidate file path from the command line.
  • ClaudeAgentOptions limits the agent to the read-only Read, Grep, and Glob tools.
  • The disallowed tool list removes access to terminal commands and file edits.
  • query() returns an asynchronous stream of messages from the review session.
  • ResultMessage.result holds the final free-form review that the script prints.
  • Save app.py.

✔️ Awesome, I've got everything!

Great. Double check that app.py is saved inside the multi-agent-review-board folder.

ⓧ I'd like to double check the full code

Compare your saved app.py with the complete baseline below:

import argparse
import asyncio

from claude_agent_sdk import ClaudeAgentOptions, ResultMessage, query


async def run_review(candidate: str) -> None:
    options = ClaudeAgentOptions(
        tools=["Read", "Grep", "Glob"],
        allowed_tools=["Read", "Grep", "Glob"],
        disallowed_tools=["Bash", "Write", "Edit"],
        permission_mode="dontAsk",
    )
    prompt = f"Review the release candidate at {candidate}."
    async for message in query(prompt=prompt, options=options):
        if isinstance(message, ResultMessage):
            print(message.result)


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Run a release review."
    )
    parser.add_argument("candidate", help="Path to the release candidate Markdown file")
    args = parser.parse_args()
    asyncio.run(run_review(args.candidate))


if __name__ == "__main__":
    main()
Run the baseline review

This request uses the Claude API account linked to your key, so it can add usage charges. You only need one baseline run for this comparison.

Before you run it, consider whether one broad answer can prove that each review domain completed its own inspection.

  • Run the baseline against the risky release candidate with this command:
python app.py release_candidate.md

You'll see a free-form review in the terminal. Its wording can vary, but it should discuss risks in the release candidate.

Your first model-backed review is working. The terminal still gives you no evidence that three independent specialists completed the job.

Review command failed?

Confirm that the virtual environment remains active in this PowerShell window. Confirm that the API key is still available in the same session.

Help me troubleshoot the baseline review command.

The response proves that one model can review the file. The next check is whether the process produced auditable coverage.

What evidence is missing?

One broad query provides no guaranteed division of labor among security-reviewer, reliability-reviewer, and test-reviewer.

The free-form response provides no structured output contract that requires named findings or complete reviewer coverage.

There is no worker lifecycle audit showing which specialists started or stopped.

There is no deterministic policy gate that owns the final release decision.

Your baseline has done its job by exposing the evidence gap. Next, you'll make three named specialists visible from start to finish.

Fan Out to Specialist Reviewers

Your single-agent baseline can inspect the release candidate. However, its free-form review cannot prove that security, reliability, plus testing received independent coverage.

This step uses the Claude Agent SDK to create three programmatic subagents. Each specialist receives an isolated context plus a narrow review domain.

In this step, get ready to:
  • Define three read-only specialist reviewers with bounded turns.
  • Configure a coordinator that launches every reviewer concurrently.
  • Record visible lifecycle evidence when each specialist starts or stops.
Define the specialist contracts

A delegation contract tells the coordinator which specialists must participate. It also prevents one broad review from silently replacing independent domain analysis.

  • In the app.py file from the previous step, replace everything above the current run_review() function with this foundation:
import argparse
import asyncio
from datetime import datetime, timezone

from claude_agent_sdk import (
    AgentDefinition,
    ClaudeAgentOptions,
    HookMatcher,
    ResultMessage,
    query,
)

REQUIRED_REVIEWERS = {
    "security-reviewer",
    "reliability-reviewer",
    "test-reviewer",
}

COORDINATOR_PROMPT = """You are a release-review coordinator, not a reviewer.
Delegate all analysis to exactly these named subagents: security-reviewer,
reliability-reviewer, and test-reviewer. Launch all three in one batch so their
independent reviews can run concurrently. Give each subagent the candidate file
path and its role-specific objective. Wait for all three reports. Never invent
or replace a missing report.
"""

What does this foundation define?

  • The expanded import group provides programmatic agent definitions plus lifecycle hook matching.
  • The REQUIRED_REVIEWERS set gives the board one canonical list of required specialist names.
  • The COORDINATOR_PROMPT contract assigns delegation to the coordinator. It forbids the coordinator from replacing a missing report.
  • Save app.py.
  • Confirm that the top section lists security-reviewer, reliability-reviewer, plus test-reviewer.

Seeing an import warning?

Check that AgentDefinition plus HookMatcher remain inside the parenthesized import from claude_agent_sdk.

Compare the capitalization of each imported name. Help me check the expanded imports in app.py.

Each AgentDefinition gives one worker a focused prompt plus a limited tool surface. A four-turn bound keeps every review contained.

  • Add the following build_agents() function below COORDINATOR_PROMPT:
def build_agents() -> dict[str, AgentDefinition]:
    output_contract = """Read only the candidate file passed by the coordinator.
Return a concise report to the coordinator. Every finding must include an id,
severity, title, evidence, and a concrete recommendation. Do not review outside
your assigned domain and do not modify files.
"""
    return {
        "security-reviewer": AgentDefinition(
            description="Reviews release candidates only for security and trust-boundary risk.",
            prompt="You are the security specialist. Examine authentication, authorization, secret exposure, logging, abuse paths, and privilege boundaries.\n\n" + output_contract,
            tools=["Read", "Grep", "Glob"], maxTurns=4, background=True,
            permissionMode="dontAsk",
        ),
        "reliability-reviewer": AgentDefinition(
            description="Reviews release candidates only for reliability and operational risk.",
            prompt="You are the reliability specialist. Examine timeouts, retries, concurrency, idempotency, rollout, rollback, partial failure, and observability.\n\n" + output_contract,
            tools=["Read", "Grep", "Glob"], maxTurns=4, background=True,
            permissionMode="dontAsk",
        ),
        "test-reviewer": AgentDefinition(
            description="Reviews release candidates only for verification and test-coverage risk.",
            prompt="You are the test specialist. Examine positive, negative, failure, rollback, concurrency, and security test coverage.\n\n" + output_contract,
            tools=["Read", "Grep", "Glob"], maxTurns=4, background=True,
            permissionMode="dontAsk",
        ),
    }

How are the workers bounded?

  • The shared output contract gives every specialist the same finding fields. Its domain boundary keeps reports focused.
  • The Read, Grep, plus Glob tools support read-only inspection.
  • The maxTurns value limits each worker to 4 turns.
  • Background execution allows the coordinator to fan out work across the three independent contexts.
  • Save app.py.
  • Confirm that build_agents() returns exactly three named dictionary entries.

Reviewer names not matching?

Compare each dictionary key with REQUIRED_REVIEWERS. Even a small spelling difference creates a different agent type.

Check that each definition uses the camelCase fields maxTurns plus permissionMode. Help me compare my three AgentDefinition entries.

Wire the coordinator and lifecycle hooks

The worker definitions become active when the coordinator receives access to the Agent tool. Lifecycle hooks add observable evidence for every worker that starts or completes.

  • Replace the existing run_review() function in app.py with this orchestrated version:
async def run_review(candidate: str) -> None:
    events: list[dict] = []
    completed_reviewers: set[str] = set()

    async def track_subagent(input_data, tool_use_id, context):
        event_name = input_data["hook_event_name"]
        agent_type = input_data["agent_type"]
        events.append({"timestamp": datetime.now(timezone.utc).isoformat(), "event": event_name, "agent_type": agent_type, "agent_id": input_data["agent_id"]})
        if event_name == "SubagentStop":
            completed_reviewers.add(agent_type)
            print(f"[STOP] {agent_type}")
        else:
            print(f"[START] {agent_type}")
        return {}

    options = ClaudeAgentOptions(
        tools=["Read", "Grep", "Glob", "Agent"],
        allowed_tools=["Read", "Grep", "Glob", "Agent"],
        disallowed_tools=["Bash", "Write", "Edit"],
        permission_mode="dontAsk", setting_sources=[], system_prompt=COORDINATOR_PROMPT,
        agents=build_agents(),
        hooks={"SubagentStart": [HookMatcher(hooks=[track_subagent])], "SubagentStop": [HookMatcher(hooks=[track_subagent])]},
        max_turns=12,
        env={"CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH": "1", "CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS": "3"},
    )
    async for message in query(prompt=f"Review {candidate}. Use all three named reviewers exactly once, launch them concurrently, and synthesize their findings.", options=options):
        if isinstance(message, ResultMessage):
            print(message.result)

What does the coordinator control?

  • The track_subagent() hook records each event timestamp plus the worker type and ID.
  • A completed worker enters completed_reviewers only after its stop event.
  • The top-level coordinator can use Agent. Every worker remains limited to read-only inspection tools.
  • The environment settings cap concurrent workers at 3. They also prevent first-level workers from spawning more subagents.
  • The final ResultMessage remains free-form in this step. Structured validation plus the policy gate arrive next.
  • Replace the existing main() function plus the final entry-point check with this version:
def main() -> None:
    parser = argparse.ArgumentParser(description="Run a bounded multi-agent release review.")
    parser.add_argument("candidate", help="Path to the release candidate Markdown file")
    args = parser.parse_args()
    asyncio.run(run_review(args.candidate))


if __name__ == "__main__":
    main()

What does the entry point do?

The entry point still accepts one candidate path. Its updated description now reflects the bounded multi-agent review that run_review() performs.

  • Save app.py.
Run the three-reviewer board

The lifecycle output is your first auditable sign of fan-out plus fan-in. This API-backed run can take longer than the baseline because three specialists inspect the candidate before the coordinator responds.

Before you run this, which three reviewer names do you expect to appear in both lifecycle states?

  • Run the multi-agent review from the activated PowerShell session with this command:
python app.py release_candidate.md

What should you see?

You should see [START] plus [STOP] lifecycle lines for security-reviewer, reliability-reviewer, plus test-reviewer.

The exact line order can vary because the specialists run concurrently. Their lifecycle lines appear before the coordinator prints the combined review.

That coverage gap is now visible: your board can prove which specialist contexts ran before synthesis.

Missing a lifecycle line?

Check that the coordinator tools include Agent. Confirm that both hook names match SubagentStart plus SubagentStop exactly.

If the run ends before every stop event, read the terminal message for the failed worker. Help me troubleshoot a missing specialist lifecycle event.

✔️ Awesome, I've got everything!

Your saved app.py now launches three named reviewers. The terminal proves that each specialist starts plus stops before the combined review.

ⓧ I'd like to double check the full code

Compare your complete app.py file with this version:

import argparse
import asyncio
from datetime import datetime, timezone

from claude_agent_sdk import (
    AgentDefinition,
    ClaudeAgentOptions,
    HookMatcher,
    ResultMessage,
    query,
)

REQUIRED_REVIEWERS = {
    "security-reviewer",
    "reliability-reviewer",
    "test-reviewer",
}

COORDINATOR_PROMPT = """You are a release-review coordinator, not a reviewer.
Delegate all analysis to exactly these named subagents: security-reviewer,
reliability-reviewer, and test-reviewer. Launch all three in one batch so their
independent reviews can run concurrently. Give each subagent the candidate file
path and its role-specific objective. Wait for all three reports. Never invent
or replace a missing report.
"""


def build_agents() -> dict[str, AgentDefinition]:
    output_contract = """Read only the candidate file passed by the coordinator.
Return a concise report to the coordinator. Every finding must include an id,
severity, title, evidence, and a concrete recommendation. Do not review outside
your assigned domain and do not modify files.
"""
    return {
        "security-reviewer": AgentDefinition(
            description="Reviews release candidates only for security and trust-boundary risk.",
            prompt="You are the security specialist. Examine authentication, authorization, secret exposure, logging, abuse paths, and privilege boundaries.\n\n" + output_contract,
            tools=["Read", "Grep", "Glob"], maxTurns=4, background=True,
            permissionMode="dontAsk",
        ),
        "reliability-reviewer": AgentDefinition(
            description="Reviews release candidates only for reliability and operational risk.",
            prompt="You are the reliability specialist. Examine timeouts, retries, concurrency, idempotency, rollout, rollback, partial failure, and observability.\n\n" + output_contract,
            tools=["Read", "Grep", "Glob"], maxTurns=4, background=True,
            permissionMode="dontAsk",
        ),
        "test-reviewer": AgentDefinition(
            description="Reviews release candidates only for verification and test-coverage risk.",
            prompt="You are the test specialist. Examine positive, negative, failure, rollback, concurrency, and security test coverage.\n\n" + output_contract,
            tools=["Read", "Grep", "Glob"], maxTurns=4, background=True,
            permissionMode="dontAsk",
        ),
    }


async def run_review(candidate: str) -> None:
    events: list[dict] = []
    completed_reviewers: set[str] = set()

    async def track_subagent(input_data, tool_use_id, context):
        event_name = input_data["hook_event_name"]
        agent_type = input_data["agent_type"]
        events.append({"timestamp": datetime.now(timezone.utc).isoformat(), "event": event_name, "agent_type": agent_type, "agent_id": input_data["agent_id"]})
        if event_name == "SubagentStop":
            completed_reviewers.add(agent_type)
            print(f"[STOP] {agent_type}")
        else:
            print(f"[START] {agent_type}")
        return {}

    options = ClaudeAgentOptions(
        tools=["Read", "Grep", "Glob", "Agent"],
        allowed_tools=["Read", "Grep", "Glob", "Agent"],
        disallowed_tools=["Bash", "Write", "Edit"],
        permission_mode="dontAsk", setting_sources=[], system_prompt=COORDINATOR_PROMPT,
        agents=build_agents(),
        hooks={"SubagentStart": [HookMatcher(hooks=[track_subagent])], "SubagentStop": [HookMatcher(hooks=[track_subagent])]},
        max_turns=12,
        env={"CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH": "1", "CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS": "3"},
    )
    async for message in query(prompt=f"Review {candidate}. Use all three named reviewers exactly once, launch them concurrently, and synthesize their findings.", options=options):
        if isinstance(message, ResultMessage):
            print(message.result)


def main() -> None:
    parser = argparse.ArgumentParser(description="Run a bounded multi-agent release review.")
    parser.add_argument("candidate", help="Path to the release candidate Markdown file")
    args = parser.parse_args()
    asyncio.run(run_review(args.candidate))


if __name__ == "__main__":
    main()

How to compare the file

Check the imports plus reviewer definitions first. Then compare run_review() plus main() before saving any corrections.

Your coordinator now delegates one change review across three observable specialists. Next, you will turn their combined output into validated data plus a deterministic release decision.

Add a Fail-Closed Policy Gate

Your coordinator now gathers three specialist perspectives. However, agent synthesis remains probabilistic. A polished answer cannot guarantee complete coverage or make the release decision safely.

In this step, you will give the Claude Agent SDK a validated output contract. You will place a deterministic Python policy after that output so missing or risky reviews fail closed.

In this step, get ready to:
  • Require structured reviewer reports through a strict JSON Schema.
  • Enforce read-only tools plus runtime and estimated-spend limits.
  • Write auditable JSON artifacts with a deterministic release decision.
Define the structured review contract

A JSON Schema defines the exact shape that every combined review must follow. It turns free-form synthesis into named reports with bounded severity values and evidence fields.

  • In the open app.py file, replace the import section at the top with this version:
import argparse
import asyncio
import json
from datetime import datetime, timezone
from pathlib import Path

from claude_agent_sdk import (
    AgentDefinition,
    ClaudeAgentOptions,
    HookMatcher,
    ResultMessage,
    query,
)

What Do These Imports Add?

  • The json module serializes the decision and audit records.
  • The Path class validates the candidate location and writes both output files.
  • Find the closing brace of REQUIRED_REVIEWERS.
  • Insert the first part of REVIEW_SCHEMA directly below it:
REVIEW_SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "properties": {
        "summary": {"type": "string"},
        "reviewers": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "properties": {
                    "name": {
                        "type": "string",
                        "enum": sorted(REQUIRED_REVIEWERS),
                    },
                    "summary": {"type": "string"},
                    "findings": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "additionalProperties": False,
                            "properties": {
                                "id": {"type": "string"},
                                "severity": {
                                    "type": "string",
                                    "enum": ["low", "medium", "high", "critical"],
                                },
                                "title": {"type": "string"},
                                "evidence": {"type": "string"},
                                "recommendation": {"type": "string"},

How Is the Contract Bounded?

This first panel opens the schema and defines each reviewer report. The reviewer name must match the required set. Severity must use one of four policy values.

  • Keep the cursor after the final comma in the first panel.
  • Complete the same REVIEW_SCHEMA dictionary by pasting these remaining lines directly below it:
                            },
                            "required": [
                                "id",
                                "severity",
                                "title",
                                "evidence",
                                "recommendation",
                            ],
                        },
                    },
                },
                "required": ["name", "summary", "findings"],
            },
        },
    },
    "required": ["summary", "reviewers"],
}

What Does the Schema Require?

Every finding needs an identifier plus severity. It also needs a title plus evidence. A recommendation completes the record.

Unknown properties are rejected at both report levels. The coordinator must return data that Python can evaluate predictably.

  • Save app.py.
  • Check the VS Code Problems panel for unmatched braces in REVIEW_SCHEMA.

The schema should now appear as one complete dictionary with no Python syntax problems.

Seeing a Schema Syntax Problem?

Check that the second panel starts immediately after the final comma in the first panel. Confirm that every opening brace has a matching closing brace.

Ask for help checking the nesting without changing the field names: Help me find the unmatched brace in my REVIEW_SCHEMA dictionary.

  • Replace the existing COORDINATOR_PROMPT with this stricter contract:
COORDINATOR_PROMPT = """You are a release-review coordinator, not a reviewer.
Delegate all analysis to exactly these named subagents: security-reviewer,
reliability-reviewer, and test-reviewer. Launch all three in one batch so their
independent reviews can run concurrently. Give each subagent the candidate file
path, its role-specific objective, and the required finding fields. Wait for all
three reports. Never invent or replace a missing report. Synthesize only evidence
returned by the named reviewers, preserve each reviewer name exactly, and return
the final response through the configured structured-output schema.
"""

Why Tighten the Coordinator Contract?

The coordinator must preserve each specialist name and use only returned evidence. It must also send its final answer through the configured schema.

  • Inside build_agents(), replace the existing output_contract string with this version:
    output_contract = """Read only the candidate file passed by the coordinator.
Return a concise report to the coordinator. Every finding must include an id,
severity from low, medium, high, or critical, a title, evidence quoted or closely
paraphrased from the candidate, and a concrete recommendation. Do not review
outside your assigned domain and do not modify files.
"""

How Do Workers Support the Schema?

Each specialist now receives the same finding-field contract. The role prompt still limits each worker to its assigned review domain.

  • Save app.py.
  • Confirm the VS Code Problems panel shows no syntax problem for the two updated prompt strings.
Add deterministic policy and runtime bounds

Validated model output gives Python reliable input. The fail-closed policy now decides whether incomplete coverage or unresolved risk can pass.

  • Find the closing brace of build_agents().
  • Add the first part of apply_policy() directly below the function:
def apply_policy(review: dict, completed_reviewers: set[str]) -> dict:
    reported_reviewers = {
        reviewer["name"] for reviewer in review.get("reviewers", [])
    }
    missing_reviewers = sorted(
        (REQUIRED_REVIEWERS - reported_reviewers)
        | (REQUIRED_REVIEWERS - set(completed_reviewers))
    )
    findings = [
        finding
        for reviewer in review.get("reviewers", [])
        for finding in reviewer.get("findings", [])
    ]
    severities = {finding.get("severity") for finding in findings}

How Is Coverage Measured?

  • The reported set comes from the validated review payload.
  • The completed set comes from observed lifecycle events.
  • A reviewer missing from either set appears in missing_reviewers.
  • Keep the cursor inside apply_policy() after the severities assignment.
  • Complete the policy by pasting this decision logic directly below it:
    if missing_reviewers:
        decision = "BLOCK"
        reasons = [
            "Required reviewer coverage is incomplete: "
            + ", ".join(missing_reviewers)
        ]
    elif severities & {"critical", "high"}:
        decision = "BLOCK"
        reasons = ["At least one high or critical finding remains."]
    elif "medium" in severities:
        decision = "NEEDS_CHANGES"
        reasons = ["At least one medium finding must be addressed."]
    else:
        decision = "GO"
        reasons = ["All required reviewers completed with only low-severity findings."]

    return {
        "decision": decision,
        "reasons": reasons,
        "missing_reviewers": missing_reviewers,
    }

Who Owns the Decision?

Python blocks incomplete coverage first. High or critical findings also block. Medium findings require changes.

Only complete reviews containing low-severity findings can return GO.

  • Replace the opening of run_review() through the two audit collections with this version:
async def run_review(candidate: str) -> int:
    project_root = Path.cwd().resolve()
    candidate_path = Path(candidate).resolve()

    if not candidate_path.is_file():
        raise FileNotFoundError(f"Candidate file not found: {candidate_path}")

    try:
        relative_candidate = candidate_path.relative_to(project_root)
    except ValueError as error:
        raise ValueError("Candidate file must be inside the project folder") from error

    events: list[dict] = []
    completed_reviewers: set[str] = set()

Why Validate the Candidate Path?

The coordinator receives a path relative to the current project. A missing file or a path outside the project stops before any agent work begins.

  • Inside track_subagent(), replace the one-line event append with this event record:
        event = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "event": event_name,
            "agent_type": agent_type,
            "agent_id": input_data["agent_id"],
        }
        events.append(event)

What Makes the Audit Useful?

Each lifecycle record stores the event type plus reviewer identity. Its timestamp and agent ID let you distinguish separate worker runs.

  • Replace the existing ClaudeAgentOptions block with this bounded configuration:
    options = ClaudeAgentOptions(
        tools=["Read", "Grep", "Glob", "Agent"],
        allowed_tools=["Read", "Grep", "Glob", "Agent"],
        disallowed_tools=["Bash", "Write", "Edit"],
        permission_mode="dontAsk",
        setting_sources=[],
        system_prompt=COORDINATOR_PROMPT,
        agents=build_agents(),
        output_format={"type": "json_schema", "schema": REVIEW_SCHEMA},
        hooks={
            "SubagentStart": [HookMatcher(hooks=[track_subagent])],
            "SubagentStop": [HookMatcher(hooks=[track_subagent])],
        },
        cwd=project_root,
        max_turns=12,
        max_budget_usd=2.0,
        env={
            "CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH": "1",
            "CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS": "3",
        },
    )

How Is the Run Bounded?

  • The coordinator can read files or delegate through Agent.
  • The coordinator cannot use shell or file-writing tools.
  • Filesystem settings are excluded through setting_sources=[].
  • Turn count plus concurrency plus spawn depth constrain the orchestration tree.

What Does the Spend Guard Mean?

The max_budget_usd=2.0 setting is a client-side estimated-spend guard. The estimate can pass the configured value. Claude Console remains the authoritative source for billing data.

  • Save app.py.
  • Confirm the VS Code Problems panel shows no syntax problems in apply_policy() or run_review().

Seeing an Indentation Problem?

Keep apply_policy() at the left edge of the file. Keep the event dictionary inside track_subagent() with eight leading spaces.

Ask for an indentation comparison if the editor still reports a problem: Help me compare the indentation of apply_policy and track_subagent with the surrounding app.py functions.

Persist the decision and verify the gate

The final part of the control plane separates agent execution from release governance. It stores observed lifecycle evidence even when the agent run fails.

  • Replace the existing query loop at the bottom of run_review() with this prompt and result-capture logic:
    prompt = (
        "Review the release candidate at "
        f"{relative_candidate.as_posix()}. "
        "Use all three named reviewers exactly once, launch them concurrently, "
        "wait for every report, and then synthesize their findings."
    )

    review = None
    last_result = None
    execution_error = None

    try:
        async for message in query(prompt=prompt, options=options):
            if isinstance(message, ResultMessage):
                last_result = message
                if message.total_cost_usd is not None:
                    print(f"Estimated API cost: ${message.total_cost_usd:.4f}")
                if message.subtype == "success" and message.structured_output:
                    review = message.structured_output
    except Exception as error:
        execution_error = str(error)
        print(f"Agent run ended with an error: {error}")

How Does Result Capture Fail Closed?

Only a successful ResultMessage with validated structured_output becomes the review. An execution error is recorded instead of being treated as approval.

  • Add the audit write and fallback review directly below the exception handler:
    Path("audit.json").write_text(
        json.dumps(events, indent=2),
        encoding="utf-8",
    )

    if review is None:
        review = {
            "summary": "No validated structured review was produced.",
            "reviewers": [],
        }

Why Write the Audit First?

The lifecycle audit survives a missing structured result. The empty fallback report causes every required reviewer to count as missing.

  • Add the policy result and JSON output directly below the fallback review:
    policy = apply_policy(review, completed_reviewers)
    result_payload = {
        **policy,
        "completed_reviewers": sorted(completed_reviewers),
        "estimated_cost_usd": (
            last_result.total_cost_usd if last_result is not None else None
        ),
        "execution_error": execution_error,
        "review": review,
    }
    Path("review_result.json").write_text(
        json.dumps(result_payload, indent=2),
        encoding="utf-8",
    )

What Goes Into the Result File?

The payload combines deterministic policy fields with observed worker completion. It also stores the validated review plus estimated cost and any execution error.

  • Finish run_review() with the terminal summary and decision-specific return code:
    print(f"Decision: {policy['decision']}")
    for reason in policy["reasons"]:
        print(f"Reason: {reason}")
    print("Wrote review_result.json and audit.json")

    return {"GO": 0, "NEEDS_CHANGES": 1, "BLOCK": 2}[policy["decision"]

Why Return Different Codes?

The process return code mirrors the policy outcome. That makes the board usable as a gate in later automation.

  • Replace the existing main() function with this version:
def main() -> None:
    parser = argparse.ArgumentParser(
        description="Run a bounded multi-agent release review."
    )
    parser.add_argument("candidate", help="Path to the release candidate Markdown file")
    args = parser.parse_args()
    raise SystemExit(asyncio.run(run_review(args.candidate)))

How Does Main Preserve the Decision?

The return value from run_review() now becomes the process exit status. The command-line interface still accepts one candidate path.

  • Save app.py.

✔️ Awesome, I've got everything!

Great. Confirm that app.py is saved before you run the bounded review board.

ⓧ I'd like to double check the full code

Compare your complete app.py file with this cumulative version:

import argparse
import asyncio
import json
from datetime import datetime, timezone
from pathlib import Path

from claude_agent_sdk import (
    AgentDefinition,
    ClaudeAgentOptions,
    HookMatcher,
    ResultMessage,
    query,
)

REQUIRED_REVIEWERS = {
    "security-reviewer",
    "reliability-reviewer",
    "test-reviewer",
}

REVIEW_SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "properties": {
        "summary": {"type": "string"},
        "reviewers": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "properties": {
                    "name": {
                        "type": "string",
                        "enum": sorted(REQUIRED_REVIEWERS),
                    },
                    "summary": {"type": "string"},
                    "findings": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "additionalProperties": False,
                            "properties": {
                                "id": {"type": "string"},
                                "severity": {
                                    "type": "string",
                                    "enum": ["low", "medium", "high", "critical"],
                                },
                                "title": {"type": "string"},
                                "evidence": {"type": "string"},
                                "recommendation": {"type": "string"},
                            },
                            "required": [
                                "id",
                                "severity",
                                "title",
                                "evidence",
                                "recommendation",
                            ],
                        },
                    },
                },
                "required": ["name", "summary", "findings"],
            },
        },
    },
    "required": ["summary", "reviewers"],
}

COORDINATOR_PROMPT = """You are a release-review coordinator, not a reviewer.
Delegate all analysis to exactly these named subagents: security-reviewer,
reliability-reviewer, and test-reviewer. Launch all three in one batch so their
independent reviews can run concurrently. Give each subagent the candidate file
path, its role-specific objective, and the required finding fields. Wait for all
three reports. Never invent or replace a missing report. Synthesize only evidence
returned by the named reviewers, preserve each reviewer name exactly, and return
the final response through the configured structured-output schema.
"""


def build_agents() -> dict[str, AgentDefinition]:
    output_contract = """Read only the candidate file passed by the coordinator.
Return a concise report to the coordinator. Every finding must include an id,
severity from low, medium, high, or critical, a title, evidence quoted or closely
paraphrased from the candidate, and a concrete recommendation. Do not review
outside your assigned domain and do not modify files.
"""

    return {
        "security-reviewer": AgentDefinition(
            description="Reviews release candidates only for security and trust-boundary risk.",
            prompt=(
                "You are the security specialist. Examine authentication, authorization, "
                "secret exposure, logging, abuse paths, and privilege boundaries.\n\n"
                + output_contract
            ),
            tools=["Read", "Grep", "Glob"],
            maxTurns=4,
            background=True,
            permissionMode="dontAsk",
        ),
        "reliability-reviewer": AgentDefinition(
            description="Reviews release candidates only for reliability and operational risk.",
            prompt=(
                "You are the reliability specialist. Examine timeouts, retries, concurrency, "
                "idempotency, rollout, rollback, partial failure, and observability.\n\n"
                + output_contract
            ),
            tools=["Read", "Grep", "Glob"],
            maxTurns=4,
            background=True,
            permissionMode="dontAsk",
        ),
        "test-reviewer": AgentDefinition(
            description="Reviews release candidates only for verification and test-coverage risk.",
            prompt=(
                "You are the test specialist. Examine positive, negative, failure, rollback, "
                "concurrency, and security test coverage.\n\n"
                + output_contract
            ),
            tools=["Read", "Grep", "Glob"],
            maxTurns=4,
            background=True,
            permissionMode="dontAsk",
        ),
    }


def apply_policy(review: dict, completed_reviewers: set[str]) -> dict:
    reported_reviewers = {
        reviewer["name"] for reviewer in review.get("reviewers", [])
    }
    missing_reviewers = sorted(
        (REQUIRED_REVIEWERS - reported_reviewers)
        | (REQUIRED_REVIEWERS - set(completed_reviewers))
    )
    findings = [
        finding
        for reviewer in review.get("reviewers", [])
        for finding in reviewer.get("findings", [])
    ]
    severities = {finding.get("severity") for finding in findings}

    if missing_reviewers:
        decision = "BLOCK"
        reasons = [
            "Required reviewer coverage is incomplete: "
            + ", ".join(missing_reviewers)
        ]
    elif severities & {"critical", "high"}:
        decision = "BLOCK"
        reasons = ["At least one high or critical finding remains."]
    elif "medium" in severities:
        decision = "NEEDS_CHANGES"
        reasons = ["At least one medium finding must be addressed."]
    else:
        decision = "GO"
        reasons = ["All required reviewers completed with only low-severity findings."]

    return {
        "decision": decision,
        "reasons": reasons,
        "missing_reviewers": missing_reviewers,
    }


async def run_review(candidate: str) -> int:
    project_root = Path.cwd().resolve()
    candidate_path = Path(candidate).resolve()

    if not candidate_path.is_file():
        raise FileNotFoundError(f"Candidate file not found: {candidate_path}")

    try:
        relative_candidate = candidate_path.relative_to(project_root)
    except ValueError as error:
        raise ValueError("Candidate file must be inside the project folder") from error

    events: list[dict] = []
    completed_reviewers: set[str] = set()

    async def track_subagent(input_data, tool_use_id, context):
        event_name = input_data["hook_event_name"]
        agent_type = input_data["agent_type"]
        event = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "event": event_name,
            "agent_type": agent_type,
            "agent_id": input_data["agent_id"],
        }
        events.append(event)

        if event_name == "SubagentStop":
            completed_reviewers.add(agent_type)
            print(f"[STOP] {agent_type}")
        else:
            print(f"[START] {agent_type}")
        return {}

    options = ClaudeAgentOptions(
        tools=["Read", "Grep", "Glob", "Agent"],
        allowed_tools=["Read", "Grep", "Glob", "Agent"],
        disallowed_tools=["Bash", "Write", "Edit"],
        permission_mode="dontAsk",
        setting_sources=[],
        system_prompt=COORDINATOR_PROMPT,
        agents=build_agents(),
        output_format={"type": "json_schema", "schema": REVIEW_SCHEMA},
        hooks={
            "SubagentStart": [HookMatcher(hooks=[track_subagent])],
            "SubagentStop": [HookMatcher(hooks=[track_subagent])],
        },
        cwd=project_root,
        max_turns=12,
        max_budget_usd=2.0,
        env={
            "CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH": "1",
            "CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS": "3",
        },
    )

    prompt = (
        "Review the release candidate at "
        f"{relative_candidate.as_posix()}. "
        "Use all three named reviewers exactly once, launch them concurrently, "
        "wait for every report, and then synthesize their findings."
    )

    review = None
    last_result = None
    execution_error = None

    try:
        async for message in query(prompt=prompt, options=options):
            if isinstance(message, ResultMessage):
                last_result = message
                if message.total_cost_usd is not None:
                    print(f"Estimated API cost: ${message.total_cost_usd:.4f}")
                if message.subtype == "success" and message.structured_output:
                    review = message.structured_output
    except Exception as error:
        execution_error = str(error)
        print(f"Agent run ended with an error: {error}")

    Path("audit.json").write_text(
        json.dumps(events, indent=2),
        encoding="utf-8",
    )

    if review is None:
        review = {
            "summary": "No validated structured review was produced.",
            "reviewers": [],
        }

    policy = apply_policy(review, completed_reviewers)
    result_payload = {
        **policy,
        "completed_reviewers": sorted(completed_reviewers),
        "estimated_cost_usd": (
            last_result.total_cost_usd if last_result is not None else None
        ),
        "execution_error": execution_error,
        "review": review,
    }
    Path("review_result.json").write_text(
        json.dumps(result_payload, indent=2),
        encoding="utf-8",
    )

    print(f"Decision: {policy['decision']}")
    for reason in policy["reasons"]:
        print(f"Reason: {reason}")
    print("Wrote review_result.json and audit.json")

    return {"GO": 0, "NEEDS_CHANGES": 1, "BLOCK": 2}[policy["decision"]


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Run a bounded multi-agent release review."
    )
    parser.add_argument("candidate", help="Path to the release candidate Markdown file")
    args = parser.parse_args()
    raise SystemExit(asyncio.run(run_review(args.candidate)))


if __name__ == "__main__":
    main()

What Should Match?

This reference includes the schema plus the deterministic policy. It also includes the bounded options plus both artifact writes.

This Run Uses API Billing

The next command makes a real Claude API request. The $2.00 client-side guard limits estimated spend for this query tree. It is not an authoritative billing ceiling.

Before you run this, which decision do you expect the intentionally risky candidate to receive? Hold that prediction while the board runs.

  • Switch back to the activated PowerShell session from earlier.
  • Run the bounded review board with the release candidate by entering this command:
python app.py release_candidate.md

What Does This Command Prove?

The command sends one candidate through all three specialists. The coordinator must return validated structured output before Python applies the release policy.

You should see [START] and [STOP] lines for each named reviewer. You should then see a BLOCK decision plus confirmation that both JSON files were written.

  • Return to VS Code after the PowerShell prompt reappears.
  • Open review_result.json from the Explorer sidebar.
  • Confirm that decision contains BLOCK.
  • Confirm that completed_reviewers lists all three required reviewer names.
  • Confirm that estimated_cost_usd contains the SDK estimate when one was returned.
  • Open audit.json from the Explorer sidebar.
  • Confirm that the audit contains start and stop events for all three specialists.

That is the governance boundary working. The agents provide evidence while deterministic Python owns the release decision.

Missing a Result or Audit File?

Confirm that the PowerShell prompt still starts with (.venv). Confirm that the session still has the API key from the earlier setup.

If the run records an execution error, inspect execution_error in review_result.json. The decision should still fail closed.

Ask for help with the recorded failure: Help me diagnose why my bounded review run failed while preserving its fail-closed behavior.

Your review board now converts bounded agent work into an auditable release decision. Next, you will test that policy boundary locally without making more API calls.

Test the Decision Boundary

Your review board now produces structured findings. A deterministic policy turns those findings into the final decision.

That policy is the safety boundary for the whole board. In this step, you will use Python unit tests to lock down its behavior without making a Claude API call.

In this step, get ready to:
  • Build reusable synthetic reports for complete reviewer coverage.
  • Cover safe outcomes plus risky outcomes with deterministic policy cases.
  • Run the local suite to confirm the release boundary.
Build synthetic complete-review cases

A policy test needs predictable input. The helper below creates one synthetic report for every name in REQUIRED_REVIEWERS.

  • In the Visual Studio Code window from earlier, select the multi-agent-review-board folder in the left sidebar.
  • Select the new-file icon beside the folder name.
  • Name the file test_policy.py.
  • Add the imports plus the synthetic report helper by pasting this code:
import unittest

from app import REQUIRED_REVIEWERS, apply_policy


def complete_review(severity: str) -> dict:
    reviewers = []
    for index, name in enumerate(sorted(REQUIRED_REVIEWERS), start=1):
        reviewers.append(
            {
                "name": name,
                "summary": f"{name} completed its review.",
                "findings": [
                    {
                        "id": f"F-{index}",
                        "severity": severity,
                        "title": "Synthetic policy test finding",
                        "evidence": "Synthetic evidence",
                        "recommendation": "Synthetic recommendation",
                    }
                ],
            }
        )
    return {"summary": "Synthetic complete review", "reviewers": reviewers}

What does this helper do?

  • The unittest import supplies Python's local testing framework.
  • The imports from app.py reuse the production reviewer set plus the real policy function.
  • The complete_review() helper builds one report for every required reviewer.
  • The severity parameter lets each case change risk while keeping reviewer coverage complete.
  • Save test_policy.py.
  • Confirm test_policy.py now appears beside app.py in the left sidebar.

File in the wrong place?

  • Check that test_policy.py sits inside the same multi-agent-review-board folder as app.py.
  • Rename the file if its name includes an extra extension.

Help me place my Python test file beside app.py.

The first three cases hold reviewer coverage constant. Each case changes only the finding severity before checking the exact policy decision.

  • Add the three complete-review tests below complete_review() by pasting this code:
class PolicyTests(unittest.TestCase):
    def test_complete_low_severity_review_can_go(self):
        outcome = apply_policy(complete_review("low"), set(REQUIRED_REVIEWERS))
        self.assertEqual(outcome["decision"], "GO")

    def test_medium_finding_requires_changes(self):
        outcome = apply_policy(complete_review("medium"), set(REQUIRED_REVIEWERS))
        self.assertEqual(outcome["decision"], "NEEDS_CHANGES")

    def test_high_finding_blocks(self):
        outcome = apply_policy(complete_review("high"), set(REQUIRED_REVIEWERS))
        self.assertEqual(outcome["decision"], "BLOCK")

What do these cases prove?

  • The low-severity case expects GO because every required reviewer completed.
  • The medium-severity case expects NEEDS_CHANGES because its findings need remediation.
  • The high-severity case expects BLOCK because serious risk cannot pass the gate.
  • Every assertion calls the real apply_policy() function with an in-memory dictionary.
  • Save test_policy.py.
  • Check the first three policy boundaries by running this command:
python -m unittest -v test_policy.py

What does this first run prove?

  • The command loads the tests from test_policy.py.
  • The -v option lists each discovered test by name.
  • The run exercises only deterministic Python inputs.

You should see three named tests pass. The output ends with OK.

One of the first three tests failed?

  • Check that each expected decision matches the capitalization shown in the code.
  • Check that every test passes set(REQUIRED_REVIEWERS) into apply_policy().
  • Confirm the terminal is still inside the multi-agent-review-board folder.

Help me diagnose a failing policy test.

Add the missing-reviewer case

Severity coverage proves how the gate handles findings. A separate case must prove that incomplete worker coverage blocks an otherwise safe review.

  • Add the missing-reviewer test below test_high_finding_blocks() by pasting this code:
    def test_missing_reviewer_blocks(self):
        completed = set(REQUIRED_REVIEWERS) - {"test-reviewer"}
        outcome = apply_policy(complete_review("low"), completed)
        self.assertEqual(outcome["decision"], "BLOCK")
        self.assertIn("test-reviewer", outcome["missing_reviewers"])


if __name__ == "__main__":
    unittest.main()

How does this case simulate failure?

  • The synthetic review still reports low-severity findings from all three names.
  • The completed set removes test-reviewer from the observed lifecycle record.
  • The first assertion requires the decision to be BLOCK.
  • The second assertion requires missing_reviewers to identify the absent worker.
  • Save test_policy.py.
  • Confirm test_missing_reviewer_blocks() remains indented inside PolicyTests.
  • Confirm the final if __name__ == "__main__": line starts at the left edge.

Seeing an indentation problem?

  • Align def test_missing_reviewer_blocks with the other test methods.
  • Indent the five method-body lines one level beneath that definition.
  • Keep the final if block outside the class.

Help me fix the indentation in test_policy.py.

✔️ Awesome, I've got everything!

Your test_policy.py file now contains four deterministic policy cases.

ⓧ I'd like to double check the full code

The complete test_policy.py file is below for comparison.

import unittest

from app import REQUIRED_REVIEWERS, apply_policy


def complete_review(severity: str) -> dict:
    reviewers = []
    for index, name in enumerate(sorted(REQUIRED_REVIEWERS), start=1):
        reviewers.append(
            {
                "name": name,
                "summary": f"{name} completed its review.",
                "findings": [
                    {
                        "id": f"F-{index}",
                        "severity": severity,
                        "title": "Synthetic policy test finding",
                        "evidence": "Synthetic evidence",
                        "recommendation": "Synthetic recommendation",
                    }
                ],
            }
        )
    return {"summary": "Synthetic complete review", "reviewers": reviewers}


class PolicyTests(unittest.TestCase):
    def test_complete_low_severity_review_can_go(self):
        outcome = apply_policy(complete_review("low"), set(REQUIRED_REVIEWERS))
        self.assertEqual(outcome["decision"], "GO")

    def test_medium_finding_requires_changes(self):
        outcome = apply_policy(complete_review("medium"), set(REQUIRED_REVIEWERS))
        self.assertEqual(outcome["decision"], "NEEDS_CHANGES")

    def test_high_finding_blocks(self):
        outcome = apply_policy(complete_review("high"), set(REQUIRED_REVIEWERS))
        self.assertEqual(outcome["decision"], "BLOCK")

    def test_missing_reviewer_blocks(self):
        completed = set(REQUIRED_REVIEWERS) - {"test-reviewer"}
        outcome = apply_policy(complete_review("low"), completed)
        self.assertEqual(outcome["decision"], "BLOCK")
        self.assertIn("test-reviewer", outcome["missing_reviewers"])


if __name__ == "__main__":
    unittest.main()
Run the four-case policy suite

Importing app.py exposes the policy without starting main(). Each case calls only apply_policy() with synthetic data.

Before you run the full suite, consider which single case you expect to return GO.

  • Run all four local tests from the activated PowerShell session by running:
python -m unittest -v test_policy.py

What does this command prove?

  • The python -m unittest command runs the standard-library test framework.
  • The -v option exposes every policy case in the terminal.
  • The suite tests the release boundary without invoking query().
  • Only the complete low-severity case accepts the release.

You should see four named tests pass. The final line should show OK.

That closes the safety loop. Your release gate now has local proof for safe findings plus risky findings plus incomplete coverage.

The full suite did not pass?

  • Compare each expected decision with the strings returned by apply_policy().
  • Check that the missing-reviewer case removes only test-reviewer from completed.
  • Confirm the terminal remains inside the multi-agent-review-board folder.

Help me debug the four-case policy suite.

Secret mission

Run a Missing-Reviewer Failure Drill

Temporarily remove the test specialist from the board while keeping it in the required-reviewer contract. Run the degraded board to prove that observed worker coverage controls the final decision, then restore the healthy configuration.

Clean Up Your Resources

Clean Up Your Resources

Keeping the local files costs nothing because no service remains active after the review board exits. Future runs can add Claude API usage.

Resources you used:

  • The local multi-agent-review-board folder.
  • The Python virtual environment stored in .venv.
  • The project files app.py, test_policy.py, requirements.txt, and release_candidate.md.
  • The generated review_result.json and audit.json outputs.
  • The current PowerShell session holding ANTHROPIC_API_KEY.
  • A project-specific Claude Console API key, if you created one only for this project.

Keep everything running

No action is required. Choose this if you want to keep experimenting with the release review board.

  • Keep the multi-agent-review-board folder in its current location.
  • Store the project-specific API key in your normal secret-management workflow.
  • Run the board only when you intend to make another Claude API call.

The board has no background process. API charges occur only when the project sends another request.

Pause - I'll come back to this later

Pausing preserves the complete project while ending the current authenticated session.

  • Close the PowerShell session where .venv is activated.
  • Leave multi-agent-review-board in its current location.
  • Keep the project-specific API key in your normal secret-management workflow.

The process-scoped ANTHROPIC_API_KEY value disappears when the PowerShell session closes. Your files remain ready for a later review.

Delete - I don't want to use this again

Deletion removes the local review board. It also closes the project-specific API access you no longer need.

  • Close the PowerShell session where .venv is activated.
  • Use File Explorer to locate the existing multi-agent-review-board folder.
  • Select the multi-agent-review-board folder.
  • Press the Delete key to move the folder to the Recycle Bin.
  • Empty the Recycle Bin to remove the folder from your computer.

You should no longer find the multi-agent-review-board folder or its generated JSON outputs on your computer.

A project-specific API key remains active after the local files are deleted.

  • Go to the Claude Console API key management page in your browser.
  • Revoke the API key you created only for this project.
  • Open the Claude Console Usage page to check the authoritative billed usage for your runs.
  • Confirm the project-specific key no longer has active access.

That closes the loop. The deleted local board can no longer make authenticated API calls.

Nice Work!

Nice Work!

You did it! Your release review board now uses multi-agent orchestration to delegate one release candidate to three specialist reviewers. Deterministic Python turns their evidence into an auditable release decision.

You've learned how to:

  • Used the Claude Agent SDK to build bounded specialist delegation across security, reliability, and testing.
  • Captured auditable fan-out and fan-in through named lifecycle events in audit.json plus validated findings in review_result.json.
  • Protected the release boundary with a deterministic fail-closed policy whose four local tests cover low-risk, medium-risk, high-risk, and missing-reviewer outcomes.
  • Secret Mission: Removed one specialist during a failure drill. You proved that incomplete reviewer coverage produces BLOCK before restoring the healthy board.

Ready to quiz yourself?