Build a Claims Triage Copilot

Build an AI claims workbench with RAG, MCP status lookups, and local evals.

Introduction

30 Second Summary

Claims teams often receive long emails with vital details buried in prose. Adjusters must piece together the full picture across policy handbooks and claim systems.

In this project, you will build a browser-based Claims Triage Copilot for a fictional insurer using only synthetic data. It supports adjusters while keeping coverage and payment decisions with a human reviewer.

What You'll Build

Picture submitting a claims email and seeing validated intake data, a handbook answer with page citations, and current claim status together in one browser workbench.

By the end of this project, you'll have:

A measurable intake flow using structured outputs to convert a synthetic email into validated claim fields. A local eval harness reports whether the workflow passes its reliability checks.
A handbook-grounded policy answer powered by retrieval-augmented generation. Stable source IDs resolve into visible page citations.
A tested claim-status integration that displays the synthetic owner, status, and next action from a read-only MCP tool.
Secret Mission: An adversarial claims email that tries to approve a claim. Your new regression check proves the human-review gate still holds.

Are there any prerequisites?

You need basic Python familiarity with VS Code on Windows. You also need an OpenAI API key with billing enabled.

Model calls use usage-based billing, so exact cost depends on token usage.

Before We Start

Before any hands-on work, commit to a synthetic-data assistant that supports adjusters with intake summaries, handbook retrieval, and read-only claim status. Authorized human reviewers must retain coverage, liability, fraud, reserve, payment, denial, approval, and every other consequential decision.

Set Up the Windows Project

The copilot needs a reproducible local foundation before it can process a claims email. Windows PowerShell keeps every setup command in one visible place.

This step confirms a compatible Python version. It also isolates the pinned packages so package drift cannot derail the later workflow.

You will finish by launching a starter page with Streamlit. The browser result proves the interface works before any AI logic is added.

In this step, get ready to:

Confirm a supported Python version in an active virtual environment.
Install the pinned packages with a session-only API key.
Launch the starter Streamlit page from VS Code.

Prepare the Python environment

The project packages share support for Python 3.10 through 3.14. A virtual environment keeps those packages separate from other Python projects on your computer.

Press the Windows key to open the Windows search bar.
Type Windows PowerShell into the search bar.
Press Enter to open Windows PowerShell.
Check your installed Python version by running:

python --version

What Does This Check Prove?

This command prints the Python version that the python command currently uses. The result determines whether that interpreter can run every pinned package in this project.

✔️ I see a supported version

Your version is between Python 3.10 and 3.14. The same Python interpreter can create the project environment.

ⓧ I see an unsupported version

Your current Python version falls outside the supported range. Install a compatible Windows release before creating the environment.

Open the Python downloads for Windows page.
Install a Windows release between Python 3.10 and 3.14 by following the official installer instructions.
Close the current Windows PowerShell window after installation.
Press the Windows key to open the Windows search bar.
Type Windows PowerShell into the search bar.
Press Enter to open a new Windows PowerShell window.
Check the newly installed version by running:

python --version

What Should You See?

The result should now show Python 3.10 through 3.14. That confirms PowerShell resolves the python command to a supported interpreter.

ⓧ Command not found

PowerShell cannot currently find Python. Install a supported Windows release before continuing.

Open the Python downloads for Windows page.
Install a Windows release between Python 3.10 and 3.14 by following the official installer instructions.
Close the current Windows PowerShell window after installation.
Press the Windows key to open the Windows search bar.
Type Windows PowerShell into the search bar.
Press Enter to open a new Windows PowerShell window.
Confirm that PowerShell can now find Python by running:

python --version

What Should You See?

The command should now print a version from Python 3.10 through 3.14. That result confirms Python is available from the new PowerShell session.

Move to your Desktop by running this command:

cd ~/Desktop

What Does This Command Do?

This moves PowerShell to your Desktop. Creating the project there gives you a predictable place to find it later.

Create the project folder by running these commands:

mkdir claims-triage-copilot
cd claims-triage-copilot

What Do These Commands Do?

The first command creates the claims-triage-copilot folder on your Desktop.
The second command moves PowerShell into that folder. Every project file now lands in the correct location.

Your PowerShell prompt should now end with claims-triage-copilot. That visible path confirms the project folder is your current location.

Create the isolated project environment by running:

python -m venv .venv

What Does This Command Do?

Python creates the .venv folder inside claims-triage-copilot. This folder holds the interpreter and packages used only by this project.

Activate the virtual environment by running:

.venv\Scripts\Activate.ps1

What Does Activation Change?

Activation points the current PowerShell session at the Python tools inside .venv. Package installations now stay isolated inside the project.

Your environment is isolated and ready. You should see (.venv) at the start of the PowerShell prompt.

Virtual Environment Not Active?

Confirm that your prompt path ends with claims-triage-copilot before running the activation command.
Confirm that the activation path begins with .venv and uses the Activate.ps1 script.
Follow your organization's approved PowerShell policy guidance if script execution is restricted on your computer.

Still stuck? Help me activate the .venv environment in Windows PowerShell.

Install the pinned dependencies and set the key

Pinned versions make the project reproducible across computers. They also keep later code aligned with the APIs used throughout this walkthrough.

Install the three pinned project packages by running:

pip install openai-agents==0.22.3
pip install streamlit==1.64.0
pip install mcp==2.2.0

What Are These Packages?

The OpenAI Agents SDK package openai-agents at version 0.22.3 provides the agents and structured-output runner used later.
The streamlit package at version 1.64.0 provides the browser workbench.
The mcp package at version 2.2.0 provides the local MCP server used for read-only claim status.

The installation is complete when all three commands finish without an installation error. Your PowerShell prompt should return with (.venv) still visible.

Seeing an Installation Error?

Check that (.venv) appears at the start of the PowerShell prompt.
Check that your earlier Python version result was between 3.10 and 3.14.
Retry the failed package command after confirming your internet connection.

Need help? Help me diagnose my pinned Python package installation.

API Costs Start in Step 2

Local Streamlit and MCP processes are free. Each model call beginning in Step 2 is billed at the selected model's token rates.

Your OpenAI API key authorizes those billed requests. Keeping it in the current PowerShell session prevents it from being written into a project file.

Replace your-api-key-here with your real key inside the command below.
Set the key for the current PowerShell session by running:

$env:OPENAI_API_KEY = "your-api-key-here"

Where Does the Key Go?

PowerShell stores OPENAI_API_KEY in the environment of this session. Closing the window removes that session value.

The key stays out of app.py and every other project file. This keeps the credential separate from code you may share later.

The PowerShell prompt should return without printing your key. Your credential is now available to programs launched from this session.

Launch the visible starter

The first page has one job. It proves that VS Code can edit the project while the active environment serves the result in a browser.

Press the Windows key to open the Windows search bar.
Type VS Code into the search bar.
Press Enter to open VS Code.
Select File from the top menu.
Select Open Folder... from the menu.
Select the claims-triage-copilot folder on your Desktop.
Click Select Folder to open the project.
Select Yes, I trust the authors if the Workspace Trust dialog appears.

The Explorer sidebar should show claims-triage-copilot with its .venv folder. This confirms VS Code opened the same project location as PowerShell.

Select the New File... button in the Explorer sidebar.
Enter app.py as the file name.
Press Enter to create the file.
Build the starter page by copying this code into app.py:

import streamlit as st

st.markdown("# Claims Triage Copilot")

What Does This Code Do?

The import makes Streamlit available through the short name st.
The st.markdown() call renders the workbench title as a page heading. This gives you a visible browser result with only two lines of Python.

Save app.py by pressing Ctrl+S.
Switch back to the Windows PowerShell window from earlier.

Before you launch the app, predict whether the browser heading will match the text inside app.py.

Launch the starter page by running:

python -m streamlit run app.py

What Does This Command Do?

This launches Streamlit through the Python interpreter in the active virtual environment. Streamlit serves app.py as a local browser application.

The process keeps this PowerShell window busy while the app runs. Pressing Ctrl+C stops the local server when you need it later.

A browser tab should open with the heading “Claims Triage Copilot”. That is the first visible piece of your claims workbench running locally.

Leave the PowerShell window open while you capture the running browser page.

Browser Page Not Showing?

Check that the PowerShell prompt showed (.venv) before you launched Streamlit.
Check that app.py is saved inside the claims-triage-copilot folder.
Open the local address printed by Streamlit in the PowerShell output if the browser does not open automatically.

Still blocked? Help me troubleshoot why my Streamlit starter page is not opening.

✔️ Awesome, I've got everything!

Your saved app.py is running through the active environment. Keep the Streamlit process open while you complete the checkpoint.

ⓧ I'd like to double check the full code

Compare your complete app.py with this reference. The file should contain these lines in the same order.

import streamlit as st

st.markdown("# Claims Triage Copilot")

What Should Match?

The file contains one Streamlit import and one Markdown heading. Extra setup code does not belong in this starter file.

Your reproducible environment is ready. Next, you will turn a synthetic claims email into a typed intake object.

Build Structured Claims Intake

Your starter Streamlit page proves that the local workbench can render in a browser. Claims emails still arrive as free-form text that downstream software cannot safely treat as a contract.

A structured output turns each email into predictable fields. You will also enforce human review in application code so the model cannot remove that boundary.

In this step, get ready to:

Define a typed contract for claims intake.
Extract a synthetic claims email through an OpenAI agent.
Display the structured result in the browser workbench.

Define the intake contract

The intake contract controls which fields downstream code can receive. Consequential decisions stay outside this schema so an authorized reviewer keeps control.

In the VS Code Explorer sidebar, create workflow.py beside app.py.
Add the model identifier and typed schema by copying this code into workflow.py:

from dataclasses import dataclass

from agents import Agent, Runner

MODEL = "gpt-6-luna"


@dataclass
class ClaimExtraction:
    claim_id: str
    policy_number: str
    claimant_name: str
    incident_type: str
    incident_date: str
    location: str
    summary: str
    urgency: str
    missing_information: list[str]
    human_review_required: bool

What does this code do?

The ClaimExtraction dataclass defines the exact shape of every intake result.
The scalar fields hold facts extracted from the synthetic email.
The missing_information list records required details that the email does not provide.
The human_review_required field makes the review boundary visible in the structured result.

Why exclude decision fields?

The schema contains intake facts and review signals. It has no field for coverage, liability, fraud, payment, denial, or approval decisions.

That boundary prevents downstream code from treating model output as an authorized claims decision.

Save workflow.py.
Confirm that VS Code shows all ten fields inside ClaimExtraction.

Seeing a Python warning?

Check that every field inside ClaimExtraction is indented by four spaces.
Check that @dataclass sits directly above the class definition.
Compare the capitalization of ClaimExtraction with the code above.

Still stuck? Help me debug my ClaimExtraction dataclass.

Enforce the review gate

The OpenAI Agents SDK can return an instance of your dataclass through output_type=ClaimExtraction. Application code then resets the review flag to True before any caller receives the result.

Add the extraction function below ClaimExtraction in workflow.py by copying this code:

async def extract_claim(email_text: str) -> ClaimExtraction:
    agent = Agent(
        name="Claims Intake Extractor",
        model=MODEL,
        instructions=(
            "Extract only facts stated in the synthetic claims email. Use an empty string for "
            "an unknown scalar and list every missing required detail in missing_information. "
            "Use low, medium, or high for urgency. Do not decide coverage, liability, fraud, "
            "payment, or denial. Always set human_review_required to true."
        ),
        output_type=ClaimExtraction,
    )
    result = await Runner.run(agent, email_text)
    claim = result.final_output
    claim.human_review_required = True
    return claim

How does extraction stay bounded?

The agent instructions limit extraction to facts stated in the email.
The output_type setting requires the response to match ClaimExtraction.
The Runner.run call sends the synthetic email to the agent.
The final assignment forces human_review_required back to True even if the model returns another value.

Save workflow.py.
Check that VS Code shows no Python syntax errors in workflow.py.

Does the function show an error?

Check that extract_claim begins at the left edge of the file.
Check that the function body uses four spaces of indentation.
Confirm that both Agent and Runner appear in the import near the top of the file.

Need another pair of eyes? Help me find the syntax problem in extract_claim.

✔️ Awesome, I've got everything!

Your extraction module now defines a typed result and an application-enforced review gate. Save workflow.py before continuing.

ⓧ I'd like to double check the full code

from dataclasses import dataclass

from agents import Agent, Runner

MODEL = "gpt-6-luna"


@dataclass
class ClaimExtraction:
    claim_id: str
    policy_number: str
    claimant_name: str
    incident_type: str
    incident_date: str
    location: str
    summary: str
    urgency: str
    missing_information: list[str]
    human_review_required: bool


async def extract_claim(email_text: str) -> ClaimExtraction:
    agent = Agent(
        name="Claims Intake Extractor",
        model=MODEL,
        instructions=(
            "Extract only facts stated in the synthetic claims email. Use an empty string for "
            "an unknown scalar and list every missing required detail in missing_information. "
            "Use low, medium, or high for urgency. Do not decide coverage, liability, fraud, "
            "payment, or denial. Always set human_review_required to true."
        ),
        output_type=ClaimExtraction,
    )
    result = await Runner.run(agent, email_text)
    claim = result.final_output
    claim.human_review_required = True
    return claim

Build the browser workbench

The browser needs a safe synthetic email that always exercises the same intake fields. It also needs a visible reminder that every consequential decision belongs to a person.

In app.py, select the existing starter code.
Replace the starter code with this first workbench section:

import asyncio
import os

import streamlit as st

from workflow import extract_claim

SAMPLE_EMAIL = """Subject: Water damage at 18 Pine Street

Hello Claims Team,
I am Elena Ruiz. My policy number is POL-1042 and the claim ID is CLM-1002.
A kitchen pipe burst on September 18, 2026 at 18 Pine Street, Portland.
I shut off the water and hired an emergency drying company. The lower cabinets
and flooring are wet. I have photos, but I have not received the drying invoice yet.
Please tell me what happens next.
"""

st.markdown("# Claims Triage Copilot")
st.markdown(
    "Synthetic data only. This assistant supports intake and research. "
    "A human reviewer retains every coverage, liability, payment, and denial decision."
)

What does this section add?

The imports connect the browser app to asynchronous extraction and environment-variable checks.
The SAMPLE_EMAIL value supplies a complete synthetic water-damage scenario.
The safety statement keeps the human decision boundary visible above every result.

Save app.py.
Refresh the browser tab from earlier.

You should see the Claims Triage Copilot heading followed by the synthetic-data safety statement.

Did the starter page stop loading?

Confirm that workflow.py is beside app.py in the claims-triage-copilot folder.
Confirm that the import uses extract_claim with the same spelling as workflow.py.

If the browser still fails to load, help me debug the workflow import.

A form keeps the email and submit button together. The result appears only after the learner requests an analysis.

Add the form and result handling below the safety statement in app.py by copying this code:

with st.form("claim-workbench"):
    email_text = st.text_area(
        "Synthetic claims email",
        value=SAMPLE_EMAIL,
        height=260,
    )
    submitted = st.form_submit_button("Analyze claim")

if submitted:
    if not os.getenv("OPENAI_API_KEY"):
        st.markdown(
            "**Missing API key:** Set `OPENAI_API_KEY` in the PowerShell session that launches Streamlit."
        )
    else:
        try:
            claim = asyncio.run(extract_claim(email_text))
        except Exception as exc:
            st.markdown(f"**Run failed:** `{type(exc).__name__}`")
            st.write(str(exc))
        else:
            st.markdown("## Structured intake")
            st.json(vars(claim))
            st.markdown("Policy answer unavailable until the handbook is connected.")

How does the workbench respond?

The form sends the current text area value to extract_claim after you click Analyze claim.
The environment check prevents a model request when OPENAI_API_KEY is missing.
The vars call converts the dataclass instance into a dictionary that Streamlit can display as JSON.
The final message exposes the current evidence gap instead of inventing a policy answer.

Save app.py.
Switch back to the PowerShell terminal from earlier.
Stop the starter Streamlit process with Ctrl+C.

Before you relaunch the workbench, what controls do you expect to see beneath the safety statement?

Launch the updated workbench by running this command:

python -m streamlit run app.py

What does this command do?

Python starts the Streamlit server for app.py. The workbench opens in your browser so you can submit the synthetic email.

You should see the sample water-damage email inside a text area. You should also see an Analyze claim button.

The next click starts a billed API request. The provided synthetic email keeps this first test short and controlled.

Before you submit the sample, do you think this version can produce a policy answer from evidence it has never received?

Keep the synthetic sample email unchanged.
Click Analyze claim.

You should see a JSON object containing all ten ClaimExtraction fields. The human_review_required field should show true.

Below the JSON, you should see “Policy answer unavailable until the handbook is connected.” That limitation is intentional because this version has no handbook evidence.

That is the safety boundary working: the model structures intake while application code preserves human review.

Did the analysis fail?

Confirm that the active PowerShell prompt still shows (.venv).
Launch Streamlit from the same PowerShell session where OPENAI_API_KEY is configured.
Compare the field names in ClaimExtraction with the structured JSON section in the browser.

If the request still fails, help me troubleshoot my structured claims extraction.

✔️ Awesome, I've got everything!

Your workbench now turns a synthetic email into structured intake JSON. Confirm that both Python files are saved.

ⓧ I'd like to double check the full code

import asyncio
import os

import streamlit as st

from workflow import extract_claim

SAMPLE_EMAIL = """Subject: Water damage at 18 Pine Street

Hello Claims Team,
I am Elena Ruiz. My policy number is POL-1042 and the claim ID is CLM-1002.
A kitchen pipe burst on September 18, 2026 at 18 Pine Street, Portland.
I shut off the water and hired an emergency drying company. The lower cabinets
and flooring are wet. I have photos, but I have not received the drying invoice yet.
Please tell me what happens next.
"""

st.markdown("# Claims Triage Copilot")
st.markdown(
    "Synthetic data only. This assistant supports intake and research. "
    "A human reviewer retains every coverage, liability, payment, and denial decision."
)

with st.form("claim-workbench"):
    email_text = st.text_area(
        "Synthetic claims email",
        value=SAMPLE_EMAIL,
        height=260,
    )
    submitted = st.form_submit_button("Analyze claim")

if submitted:
    if not os.getenv("OPENAI_API_KEY"):
        st.markdown(
            "**Missing API key:** Set `OPENAI_API_KEY` in the PowerShell session that launches Streamlit."
        )
    else:
        try:
            claim = asyncio.run(extract_claim(email_text))
        except Exception as exc:
            st.markdown(f"**Run failed:** `{type(exc).__name__}`")
            st.write(str(exc))
        else:
            st.markdown("## Structured intake")
            st.json(vars(claim))
            st.markdown("Policy answer unavailable until the handbook is connected.")

Your workbench can now convert an unstructured claims email into a typed intake result. Next, you will connect handbook evidence so policy answers can carry validated page citations.

Ground Answers in Handbook Pages

The structured intake from the previous step turns a claims email into fields your application can use. Policy answers still lack evidence that an adjuster can inspect.

This step adds retrieval-augmented generation over a synthetic handbook. The application validates every returned source ID before it turns that source into a page citation.

In this step, get ready to:

Create a synthetic handbook with stable source IDs and page numbers.
Retrieve relevant handbook pages for each policy question.
Display grounded answers with application-validated citations.

Create page-labeled handbook context

Each handbook record carries a stable source ID plus a page number. The model cites the source ID while your application controls the page number shown to the adjuster.

Return to the claims-triage-copilot folder in the VS Code Explorer sidebar.
Click the New File button in the Explorer toolbar.
Name the new file handbook.py.
Add the first two synthetic handbook pages by copying this code into handbook.py:

PAGES = [
    {
        "id": "handbook-page-1",
        "page": 1,
        "title": "First Notice of Loss",
        "text": (
            "Every new claim must include a policy number, claimant name, incident date, "
            "incident location, and a short description. Missing information must be listed "
            "for an adjuster to review."
        ),
    },
    {
        "id": "handbook-page-2",
        "page": 2,
        "title": "Theft Claims",
        "text": (
            "Theft claims must be reported to law enforcement within 24 hours of discovery. "
            "The claimant should provide the police report number, an inventory of stolen "
            "items, and any available photographs or receipts."
        ),
    },
]

How is each page structured?

The id value gives the model a stable source identifier to return.
The page value gives the application a human-readable locator.
The title and text values supply searchable policy language.

Place the next two page records above the closing square bracket in handbook.py:

    {
        "id": "handbook-page-3",
        "page": 3,
        "title": "Emergency Water Mitigation",
        "text": (
            "Emergency water mitigation may begin before an adjuster inspection when needed "
            "to prevent further damage. Claimants should photograph affected areas and retain "
            "invoices. Permanent repairs require adjuster approval."
        ),
    },
    {
        "id": "handbook-page-4",
        "page": 4,
        "title": "Human Review Boundaries",
        "text": (
            "AI-assisted intake may summarize documents and flag missing information. Coverage, "
            "liability, fraud, reserve, payment, and denial decisions require an authorized "
            "human reviewer."
        ),
    },

Why keep this handbook synthetic?

The four pages provide enough policy variation to test retrieval without exposing customer records. Page 4 also preserves the boundary between AI assistance and decisions that require an authorized human reviewer.

Save handbook.py.
Confirm the VS Code Explorer lists handbook.py beside app.py and workflow.py.

Seeing a Python syntax warning?

Check that every page record ends with a comma.
Check that the four records remain inside the opening and closing square brackets for PAGES.
Check that each multi-line text value keeps its surrounding parentheses.

Still stuck? Help me check the structure of my four-page PAGES list in handbook.py.

✔️ Awesome, I've got everything!

Your synthetic handbook now has four page-labeled records. Save handbook.py before continuing.

ⓧ I'd like to double check the full code

PAGES = [
    {
        "id": "handbook-page-1",
        "page": 1,
        "title": "First Notice of Loss",
        "text": (
            "Every new claim must include a policy number, claimant name, incident date, "
            "incident location, and a short description. Missing information must be listed "
            "for an adjuster to review."
        ),
    },
    {
        "id": "handbook-page-2",
        "page": 2,
        "title": "Theft Claims",
        "text": (
            "Theft claims must be reported to law enforcement within 24 hours of discovery. "
            "The claimant should provide the police report number, an inventory of stolen "
            "items, and any available photographs or receipts."
        ),
    },
    {
        "id": "handbook-page-3",
        "page": 3,
        "title": "Emergency Water Mitigation",
        "text": (
            "Emergency water mitigation may begin before an adjuster inspection when needed "
            "to prevent further damage. Claimants should photograph affected areas and retain "
            "invoices. Permanent repairs require adjuster approval."
        ),
    },
    {
        "id": "handbook-page-4",
        "page": 4,
        "title": "Human Review Boundaries",
        "text": (
            "AI-assisted intake may summarize documents and flag missing information. Coverage, "
            "liability, fraud, reserve, payment, and denial decisions require an authorized "
            "human reviewer."
        ),
    },
]

A retriever narrows the handbook before the model receives any context. This project uses deterministic token overlap so the selected evidence stays inspectable.

In workflow.py, replace the import section at the top with this updated version:

import re
from dataclasses import dataclass

from agents import Agent, Runner

from handbook import PAGES

What do these imports add?

The re module breaks questions and handbook text into searchable tokens.
The PAGES import gives the workflow access to the synthetic handbook corpus.

Add the retrieval stop-word set directly below MODEL = "gpt-6-luna":

STOP_WORDS = {
    "a",
    "an",
    "and",
    "are",
    "can",
    "company",
    "cover",
    "covered",
    "do",
    "does",
    "for",
    "how",
    "in",
    "is",
    "of",
    "or",
    "policy",
    "the",
    "to",
    "what",
    "when",
}

Why remove stop words?

Common words such as the and what appear across unrelated questions. Removing them lets policy-specific words such as water, theft, or mitigation drive the match.

Add the token helper and retriever above extract_claim() in workflow.py:

def _tokens(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower())) - STOP_WORDS


def retrieve_pages(question: str, top_k: int = 2) -> list[dict]:
    question_tokens = _tokens(question)
    scored_pages = []

    for page in PAGES:
        page_tokens = _tokens(f"{page['title']} {page['text']}")
        score = len(question_tokens & page_tokens)
        if score > 0:
            scored_pages.append((score, page))

    scored_pages.sort(key=lambda item: (-item[0], item[1]["page"]))
    return [page for _, page in scored_pages[:top_k]]

How does retrieval work?

The _tokens() helper normalizes text into a unique set of lowercase words.
The retrieve_pages() function scores each page by the number of question tokens it shares.
The final sort favors higher overlap scores. Page number breaks a tied score consistently.
The top_k limit keeps the model context focused on at most two pages.

Save workflow.py.
Return to the PowerShell terminal containing the active .venv.
Confirm the existing workbench still imports cleanly by running:

python -m streamlit run app.py

What does this check prove?

The browser should still show the structured intake flow plus the handbook limitation message. A successful load confirms that workflow.py can import the new page corpus and retrieval code.

Does the workbench fail to load?

Check that handbook.py sits in the same claims-triage-copilot folder as workflow.py.
Check that the import uses the exact uppercase name PAGES.
Check the terminal output for the file and line number that Python could not parse.

Need another pair of eyes? Help me debug why workflow.py cannot import PAGES from handbook.py.

Generate and validate grounded answers

The policy agent returns a draft answer plus source IDs. A second application-owned type contains only page numbers that your validation logic has resolved from those IDs.

Stop the running Streamlit process with Ctrl+C if PowerShell is still occupied.
Add the policy result dataclasses below ClaimExtraction in workflow.py:

@dataclass
class PolicyDraft:
    answer: str
    source_ids: list[str]
    supported: bool


@dataclass
class PolicyAnswer:
    answer: str
    cited_pages: list[int]
    supported: bool

Why use two policy types?

The model produces PolicyDraft with source IDs from the retrieved blocks.
The application produces PolicyAnswer after checking those IDs and resolving their page numbers.
This separation prevents a model-generated page number from becoming a trusted citation.

Add the first part of answer_policy_question() below extract_claim():

async def answer_policy_question(question: str) -> PolicyAnswer:
    pages = retrieve_pages(question)
    if not pages:
        return PolicyAnswer(
            answer="I could not find that in the handbook.",
            cited_pages=[],
            supported=False,
        )

    context = "\n\n".join(
        (
            f'<BLOCK id="{page["id"]}" page="{page["page"]}">\n'
            f'{page["text"]}\n'
            "</BLOCK>"
        )
        for page in pages
    )

What happens before the model call?

The function retrieves relevant pages before creating any prompt context.
An empty retrieval result returns the safe fallback immediately.
Each retrieved page becomes a BLOCK with a stable ID and page attribute.
Only those retrieved blocks become available to the policy agent.

Continue the same function directly below the context block with the policy agent and model call:

    agent = Agent(
        name="Policy Grounding Agent",
        model=MODEL,
        instructions=(
            "Answer only from the supplied BLOCK elements. If the blocks do not support an "
            "answer, set supported to false, use no source IDs, and answer exactly: "
            "I could not find that in the handbook. If supported, cite only block IDs that "
            "directly support the answer. Never use outside knowledge or invent an ID."
        ),
        output_type=PolicyDraft,
    )
    result = await Runner.run(
        agent,
        f"Question: {question}\n\nSupporting context:\n{context}",
    )
    draft = result.final_output

How is the model constrained?

The agent receives the policy question plus the retrieved blocks.
The instructions require an unsupported answer when those blocks lack evidence.
The output_type contract requires the answer, support flag, and source IDs to match PolicyDraft.

Finish answer_policy_question() by adding the source validation logic directly below draft = result.final_output:

    page_lookup = {page["id"]: page["page"] for page in pages}
    valid_ids = set(page_lookup)
    returned_ids = set(draft.source_ids)

    if not draft.supported:
        return PolicyAnswer(
            answer="I could not find that in the handbook.",
            cited_pages=[],
            supported=False,
        )

    if not returned_ids or not returned_ids.issubset(valid_ids):
        return PolicyAnswer(
            answer="I could not find that in the handbook.",
            cited_pages=[],
            supported=False,
        )

    return PolicyAnswer(
        answer=draft.answer,
        cited_pages=sorted(page_lookup[source_id] for source_id in returned_ids),
        supported=True,
    )

Where does citation trust come from?

The page_lookup mapping contains IDs from the pages retrieved by application code.
An unsupported draft returns the fallback with an empty citation list.
An empty or unknown source ID also returns the fallback.
A valid source ID is resolved into its application-owned page number.

✔️ Awesome, I've got everything!

The retrieval and grounding workflow is assembled. Save workflow.py before connecting it to the browser.

ⓧ I'd like to double check the full code

import re
from dataclasses import dataclass

from agents import Agent, Runner

from handbook import PAGES

MODEL = "gpt-6-luna"
STOP_WORDS = {
    "a",
    "an",
    "and",
    "are",
    "can",
    "company",
    "cover",
    "covered",
    "do",
    "does",
    "for",
    "how",
    "in",
    "is",
    "of",
    "or",
    "policy",
    "the",
    "to",
    "what",
    "when",
}


@dataclass
class ClaimExtraction:
    claim_id: str
    policy_number: str
    claimant_name: str
    incident_type: str
    incident_date: str
    location: str
    summary: str
    urgency: str
    missing_information: list[str]
    human_review_required: bool


@dataclass
class PolicyDraft:
    answer: str
    source_ids: list[str]
    supported: bool


@dataclass
class PolicyAnswer:
    answer: str
    cited_pages: list[int]
    supported: bool


def _tokens(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower())) - STOP_WORDS


def retrieve_pages(question: str, top_k: int = 2) -> list[dict]:
    question_tokens = _tokens(question)
    scored_pages = []

    for page in PAGES:
        page_tokens = _tokens(f"{page['title']} {page['text']}")
        score = len(question_tokens & page_tokens)
        if score > 0:
            scored_pages.append((score, page))

    scored_pages.sort(key=lambda item: (-item[0], item[1]["page"]))
    return [page for _, page in scored_pages[:top_k]]


async def extract_claim(email_text: str) -> ClaimExtraction:
    agent = Agent(
        name="Claims Intake Extractor",
        model=MODEL,
        instructions=(
            "Extract only facts stated in the synthetic claims email. Use an empty string for "
            "an unknown scalar and list every missing required detail in missing_information. "
            "Use low, medium, or high for urgency. Do not decide coverage, liability, fraud, "
            "payment, or denial. Always set human_review_required to true."
        ),
        output_type=ClaimExtraction,
    )
    result = await Runner.run(agent, email_text)
    claim = result.final_output
    claim.human_review_required = True
    return claim


async def answer_policy_question(question: str) -> PolicyAnswer:
    pages = retrieve_pages(question)
    if not pages:
        return PolicyAnswer(
            answer="I could not find that in the handbook.",
            cited_pages=[],
            supported=False,
        )

    context = "\n\n".join(
        (
            f'<BLOCK id="{page["id"]}" page="{page["page"]}">\n'
            f'{page["text"]}\n'
            "</BLOCK>"
        )
        for page in pages
    )

    agent = Agent(
        name="Policy Grounding Agent",
        model=MODEL,
        instructions=(
            "Answer only from the supplied BLOCK elements. If the blocks do not support an "
            "answer, set supported to false, use no source IDs, and answer exactly: "
            "I could not find that in the handbook. If supported, cite only block IDs that "
            "directly support the answer. Never use outside knowledge or invent an ID."
        ),
        output_type=PolicyDraft,
    )
    result = await Runner.run(
        agent,
        f"Question: {question}\n\nSupporting context:\n{context}",
    )
    draft = result.final_output

    page_lookup = {page["id"]: page["page"] for page in pages}
    valid_ids = set(page_lookup)
    returned_ids = set(draft.source_ids)

    if not draft.supported:
        return PolicyAnswer(
            answer="I could not find that in the handbook.",
            cited_pages=[],
            supported=False,
        )

    if not returned_ids or not returned_ids.issubset(valid_ids):
        return PolicyAnswer(
            answer="I could not find that in the handbook.",
            cited_pages=[],
            supported=False,
        )

    return PolicyAnswer(
        answer=draft.answer,
        cited_pages=sorted(page_lookup[source_id] for source_id in returned_ids),
        supported=True,
    )

Show grounded evidence in the workbench

The browser needs controls for the policy question plus a result area for the validated answer. The displayed citation comes from PolicyAnswer after source validation.

In app.py, replace the existing workflow import with this updated import:

from workflow import answer_policy_question, extract_claim

What does this import enable?

The workbench keeps its existing typed extraction function. It can now call answer_policy_question() with the question entered in the form.

Add the policy question text area below the existing synthetic email text area inside the form:

    policy_question = st.text_area(
        "Policy question",
        value="Can emergency water mitigation start before inspection?",
        height=100,
    )

Why start with this question?

The default question shares specific terms with the emergency water mitigation record. A successful answer should therefore resolve to Page 3.

Save app.py.
Refresh the browser tab running the workbench.

You should see a new Policy question text area below the synthetic email. That visible field confirms the browser form now accepts both inputs.

Missing the policy question field?

Check that policy_question sits inside the existing st.form block.
Check that its indentation matches the existing email_text widget.
Save app.py before refreshing the browser.

Still missing? Help me place the policy question text area inside my existing Streamlit form.

Find the existing structured extraction call inside the try block.
Replace that part with the updated progress message and both workflow calls:

        st.markdown("Running structured extraction and handbook retrieval...")
        try:
            claim = asyncio.run(extract_claim(email_text))
            policy = asyncio.run(answer_policy_question(policy_question))

What runs when you submit?

The first call converts the email into the existing typed claim intake.
The second call retrieves handbook pages and validates the policy agent's source IDs.
Both calls use the values submitted through the same workbench form.

Replace the previous handbook limitation message inside the successful else block with this result display:

            st.markdown("## Handbook-grounded answer")
            st.write(policy.answer)
            citation_text = ", ".join(
                f"Page {page}" for page in policy.cited_pages
            )
            st.markdown(f"**Supported:** {policy.supported}")
            st.markdown(f"**Citations:** {citation_text or 'None'}")

            st.markdown("Claim status unavailable until MCP is connected.")

What does the workbench reveal?

The answer text shows the policy statement produced from retrieved evidence.
The support flag reveals whether the validated result contains acceptable evidence.
The citation line renders resolved page numbers or None.
The claim-status message keeps the next operational limitation visible.

✔️ Awesome, I've got everything!

The workbench now collects a policy question and renders the validated policy result. Save app.py before testing the full flow.

ⓧ I'd like to double check the full code

import asyncio
import os

import streamlit as st

from workflow import answer_policy_question, extract_claim

SAMPLE_EMAIL = """Subject: Water damage at 18 Pine Street

Hello Claims Team,
I am Elena Ruiz. My policy number is POL-1042 and the claim ID is CLM-1002.
A kitchen pipe burst on September 18, 2026 at 18 Pine Street, Portland.
I shut off the water and hired an emergency drying company. The lower cabinets
and flooring are wet. I have photos, but I have not received the drying invoice yet.
Please tell me what happens next.
"""

st.markdown("# Claims Triage Copilot")
st.markdown(
    "Synthetic data only. This assistant supports intake and research. "
    "A human reviewer retains every coverage, liability, payment, and denial decision."
)

with st.form("claim-workbench"):
    email_text = st.text_area(
        "Synthetic claims email",
        value=SAMPLE_EMAIL,
        height=260,
    )
    policy_question = st.text_area(
        "Policy question",
        value="Can emergency water mitigation start before inspection?",
        height=100,
    )
    submitted = st.form_submit_button("Analyze claim")

if submitted:
    if not os.getenv("OPENAI_API_KEY"):
        st.markdown(
            "**Missing API key:** Set `OPENAI_API_KEY` in the PowerShell session that launches Streamlit."
        )
    else:
        st.markdown("Running structured extraction and handbook retrieval...")
        try:
            claim = asyncio.run(extract_claim(email_text))
            policy = asyncio.run(answer_policy_question(policy_question))
        except Exception as exc:
            st.markdown(f"**Run failed:** `{type(exc).__name__}`")
            st.write(str(exc))
        else:
            st.markdown("## Structured intake")
            st.json(vars(claim))

            st.markdown("## Handbook-grounded answer")
            st.write(policy.answer)
            citation_text = ", ".join(
                f"Page {page}" for page in policy.cited_pages
            )
            st.markdown(f"**Supported:** {policy.supported}")
            st.markdown(f"**Citations:** {citation_text or 'None'}")

            st.markdown("Claim status unavailable until MCP is connected.")

Save app.py.
Save workflow.py.
Save handbook.py.
Return to the PowerShell terminal containing the active .venv.
Stop any existing Streamlit process with Ctrl+C.

Before you run the complete workbench, which handbook page do you expect the default water-mitigation question to cite?

Launch the updated workbench by running:

python -m streamlit run app.py

What does this launch do?

Streamlit runs app.py and opens the local workbench in your browser. Keep PowerShell open while the app is running.

Keep the sample water-damage email in the first text area.
Keep Can emergency water mitigation start before inspection? in the policy question text area.
Click Analyze claim.

You should see the structured intake plus a supported handbook answer citing Page 3. You should also see the message Claim status unavailable until MCP is connected.

That is the evidence chain working. The question selected a page-labeled block, the model returned its stable ID, and the application resolved the ID into the displayed page number.

Replace the policy question with Does the handbook cover dental cleanings for pets?.
Click Analyze claim again.

You should see I could not find that in the handbook. with Supported: False and Citations: None.

Missing Page 3 or seeing a failed run?

Check that the default question contains the words water and mitigation.
Check that handbook-page-3 contains the emergency mitigation policy text.
Check that the PowerShell session running Streamlit still has the API key configured.

Need help tracing the evidence? Help me debug why my policy question does not return a validated Page 3 citation.

Your policy answer now carries evidence that an adjuster can inspect. Next, you will connect read-only claim status through a local MCP server.

Connect Claim Status Through MCP

Your workbench already extracts typed claim data. It also supports policy answers with validated handbook citations. Its final panel still cannot retrieve operational claim status.

A read-only MCP boundary closes that gap. The model must call a local claim-system tool for status data. Consequential decisions remain with a human reviewer.

In this step, get ready to:

Build a local MCP server with one read-only claim-status tool.
Require the status agent to retrieve operational data through MCP.
Display structured intake data, policy evidence, and claim status in one browser submission.

Build the read-only MCP server

An MCP server exposes a controlled set of tools to an agent. This server contains synthetic claim records. Its only tool reads status information.

Why local stdio?

Local stdio lets the workflow launch the claim server as a subprocess. The integration stays on your computer without a hosted endpoint.

A single lookup tool also keeps the boundary easy to inspect. There is no tool for approvals, denials, payments, or claim modifications.

Create claims_mcp.py inside the claims-triage-copilot folder from the file sidebar in VS Code.
Add the server instance and synthetic claim records by copying this code into claims_mcp.py:

from mcp.server import MCPServer

mcp = MCPServer("Claims Core")

CLAIMS = {
    "CLM-1001": {
        "claim_id": "CLM-1001",
        "status": "Awaiting documents",
        "owner": "Jordan Lee",
        "next_action": "Request the police report number.",
    },
    "CLM-1002": {
        "claim_id": "CLM-1002",
        "status": "Under review",
        "owner": "Maya Chen",
        "next_action": "Confirm drying invoices and the moisture report.",
    },
}

What does this code do?

The module-level mcp object represents the local server that the workflow launches.
The CLAIMS dictionary acts as a synthetic claim system for this project.
Each record provides the exact fields that the browser displays after a lookup.

Save claims_mcp.py.
Confirm that claims_mcp.py appears in the file sidebar with records for CLM-1001 and CLM-1002.

Missing a claim record?

Check that both claim IDs are uppercase. Confirm that each record sits inside the outer CLAIMS dictionary.

Need another pair of eyes? Help me check the synthetic MCP claim dictionary.

The records now need a tool that can normalize a claim ID. The same tool supplies a safe fallback when no synthetic record matches.

Add the lookup tool below the CLAIMS dictionary in claims_mcp.py by copying this code:

@mcp.tool()
def lookup_claim_status(claim_id: str) -> dict[str, str]:
    """Return read-only status information for a synthetic claim ID."""
    normalized_id = claim_id.strip().upper()
    return CLAIMS.get(
        normalized_id,
        {
            "claim_id": normalized_id,
            "status": "Not found",
            "owner": "Unassigned",
            "next_action": "Verify the claim ID with a human reviewer.",
        },
    )


if __name__ == "__main__":
    mcp.run()

How does the lookup stay safe?

The @mcp.tool() decorator makes lookup_claim_status available to an MCP client.
The normalized ID accepts surrounding spaces or lowercase letters without changing the underlying records.
The fallback sends unknown IDs to human verification without inventing claim data.
The main guard starts stdio serving when the file runs as the server process.

Save claims_mcp.py.
Confirm that lookup_claim_status is the only function decorated with @mcp.tool().

Your MCP surface now exposes one read-only operation. Every consequential claim action remains outside the server.

Does the tool look incomplete?

Confirm that the decorator includes parentheses. Check that mcp.run() is indented beneath the main guard.

If the file shows a syntax warning, help me inspect the MCP tool definition.

✔️ Awesome, I've got everything!

Your claims_mcp.py file now contains two synthetic records and one read-only MCP tool.

ⓧ I'd like to double check the full code

Compare your complete claims_mcp.py file with this reference:

from mcp.server import MCPServer

mcp = MCPServer("Claims Core")

CLAIMS = {
    "CLM-1001": {
        "claim_id": "CLM-1001",
        "status": "Awaiting documents",
        "owner": "Jordan Lee",
        "next_action": "Request the police report number.",
    },
    "CLM-1002": {
        "claim_id": "CLM-1002",
        "status": "Under review",
        "owner": "Maya Chen",
        "next_action": "Confirm drying invoices and the moisture report.",
    },
}


@mcp.tool()
def lookup_claim_status(claim_id: str) -> dict[str, str]:
    """Return read-only status information for a synthetic claim ID."""
    normalized_id = claim_id.strip().upper()
    return CLAIMS.get(
        normalized_id,
        {
            "claim_id": normalized_id,
            "status": "Not found",
            "owner": "Unassigned",
            "next_action": "Verify the claim ID with a human reviewer.",
        },
    )


if __name__ == "__main__":
    mcp.run()

What should the complete file contain?

The complete server defines its synthetic data before registering the lookup tool. The main guard starts the stdio process only when the server file runs directly.

Attach the server to the workflow

The workflow needs a typed shape for status data. It also needs a client that starts claims_mcp.py with the active Python interpreter.

The status agent receives the local server through mcp_servers. Required tool choice prevents the agent from answering a status request from model memory.

Switch back to workflow.py.
Replace the import section at the top of workflow.py with this code:

import re
import sys
from dataclasses import dataclass
from pathlib import Path

from agents import Agent, ModelSettings, Runner
from agents.mcp import MCPServerStdio

from handbook import PAGES

Why are these imports needed?

The sys module provides the active virtual environment's Python executable.
The Path class locates claims_mcp.py beside workflow.py.
The MCPServerStdio client manages the local subprocess connection.
The ModelSettings class configures required tool use for the status agent.

Add the status result contract below PolicyAnswer in workflow.py by copying this code:

@dataclass
class ClaimStatus:
    claim_id: str
    status: str
    owner: str
    next_action: str

What does this contract protect?

The ClaimStatus dataclass defines the four fields expected from the tool-backed agent. The browser receives a predictable status object after every lookup.

Save workflow.py.
Check the new imports and status contract by running the existing Streamlit workbench from PowerShell:

python -m streamlit run app.py

What does this check prove?

The command imports workflow.py before rendering the workbench. A browser page showing Claims Triage Copilot confirms that the new imports and dataclass load successfully.

Confirm that the browser still shows Claims Triage Copilot.
Stop the current Streamlit process with Ctrl+C in PowerShell.

Does the workbench fail to load?

Confirm that your virtual environment remains active. Check the spelling of MCPServerStdio and ModelSettings against the import block.

Still blocked? Help me diagnose the workflow import failure.

The imports now load successfully. Next, the status function can launch the local server and force the agent to call its lookup tool.

Add the MCP-backed status function below answer_policy_question in workflow.py by copying this code:

async def lookup_claim_status(claim_id: str) -> ClaimStatus:
    server_path = Path(__file__).with_name("claims_mcp.py")

    async with MCPServerStdio(
        name="Claims Core",
        params={
            "command": sys.executable,
            "args": [str(server_path)],
        },
        cache_tools_list=True,
    ) as server:
        agent = Agent(
            name="Claim Status Agent",
            model=MODEL,
            instructions=(
                "Always call lookup_claim_status exactly once with the claim ID supplied by the "
                "user. Return the tool fields unchanged. Never invent claim status data."
            ),
            mcp_servers=[server],
            model_settings=ModelSettings(tool_choice="required"),
            output_type=ClaimStatus,
        )
        result = await Runner.run(agent, f"Look up claim {claim_id}.")
        return result.final_output

How does the MCP lookup work?

The server_path value locates the server file beside the workflow file.
The sys.executable value starts the subprocess with the interpreter from your active virtual environment.
The mcp_servers=[server] setting gives the agent access to the read-only lookup tool.
The ModelSettings(tool_choice="required") setting requires tool use before the agent can return ClaimStatus.

Add the workflow orchestrator directly below lookup_claim_status by copying this code:

async def run_workflow(email_text: str, policy_question: str) -> dict:
    claim = await extract_claim(email_text)
    policy = await answer_policy_question(policy_question)
    status = await lookup_claim_status(claim.claim_id)

    return {
        "claim": vars(claim),
        "policy": vars(policy),
        "status": vars(status),
    }

What does the orchestrator combine?

The extraction call converts the synthetic email into a typed claim object.
The policy call retrieves handbook evidence and validates source IDs.
The status call uses the extracted claim ID to request operational data through MCP.
The returned dictionary gives the browser one result containing claim, policy, and status sections.

Save workflow.py.
Verify that the completed workflow still imports by running the workbench again:

python -m streamlit run app.py

What does this second check prove?

The browser can render only after Python imports the completed workflow module. Seeing the existing workbench confirms that both new asynchronous functions compile and load.

Confirm that the browser opens the existing claims workbench.
Stop Streamlit with Ctrl+C in PowerShell.

Seeing a workflow error?

Check that lookup_claim_status sits outside answer_policy_question. Confirm that run_workflow sits outside the lookup function.

For targeted help, ask me to check the new workflow functions.

✔️ Awesome, I've got everything!

Your workflow.py file now joins extraction, grounded policy research, and MCP-backed status lookup.

ⓧ I'd like to double check the full code

Compare your complete workflow.py file with this reference:

import re
import sys
from dataclasses import dataclass
from pathlib import Path

from agents import Agent, ModelSettings, Runner
from agents.mcp import MCPServerStdio

from handbook import PAGES

MODEL = "gpt-6-luna"
STOP_WORDS = {
    "a",
    "an",
    "and",
    "are",
    "can",
    "company",
    "cover",
    "covered",
    "do",
    "does",
    "for",
    "how",
    "in",
    "is",
    "of",
    "or",
    "policy",
    "the",
    "to",
    "what",
    "when",
}


@dataclass
class ClaimExtraction:
    claim_id: str
    policy_number: str
    claimant_name: str
    incident_type: str
    incident_date: str
    location: str
    summary: str
    urgency: str
    missing_information: list[str]
    human_review_required: bool


@dataclass
class PolicyDraft:
    answer: str
    source_ids: list[str]
    supported: bool


@dataclass
class PolicyAnswer:
    answer: str
    cited_pages: list[int]
    supported: bool


@dataclass
class ClaimStatus:
    claim_id: str
    status: str
    owner: str
    next_action: str


def _tokens(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower())) - STOP_WORDS


def retrieve_pages(question: str, top_k: int = 2) -> list[dict]:
    question_tokens = _tokens(question)
    scored_pages = []

    for page in PAGES:
        page_tokens = _tokens(f"{page['title']} {page['text']}")
        score = len(question_tokens & page_tokens)
        if score > 0:
            scored_pages.append((score, page))

    scored_pages.sort(key=lambda item: (-item[0], item[1]["page"]))
    return [page for _, page in scored_pages[:top_k]]


async def extract_claim(email_text: str) -> ClaimExtraction:
    agent = Agent(
        name="Claims Intake Extractor",
        model=MODEL,
        instructions=(
            "Extract only facts stated in the synthetic claims email. Use an empty string for "
            "an unknown scalar and list every missing required detail in missing_information. "
            "Use low, medium, or high for urgency. Do not decide coverage, liability, fraud, "
            "payment, or denial. Always set human_review_required to true."
        ),
        output_type=ClaimExtraction,
    )
    result = await Runner.run(agent, email_text)
    claim = result.final_output
    claim.human_review_required = True
    return claim


async def answer_policy_question(question: str) -> PolicyAnswer:
    pages = retrieve_pages(question)
    if not pages:
        return PolicyAnswer(
            answer="I could not find that in the handbook.",
            cited_pages=[],
            supported=False,
        )

    context = "\n\n".join(
        (
            f'<BLOCK id="{page["id"]}" page="{page["page"]}">\n'
            f'{page["text"]}\n'
            "</BLOCK>"
        )
        for page in pages
    )

    agent = Agent(
        name="Policy Grounding Agent",
        model=MODEL,
        instructions=(
            "Answer only from the supplied BLOCK elements. If the blocks do not support an "
            "answer, set supported to false, use no source IDs, and answer exactly: "
            "I could not find that in the handbook. If supported, cite only block IDs that "
            "directly support the answer. Never use outside knowledge or invent an ID."
        ),
        output_type=PolicyDraft,
    )
    result = await Runner.run(
        agent,
        f"Question: {question}\n\nSupporting context:\n{context}",
    )
    draft = result.final_output

    page_lookup = {page["id"]: page["page"] for page in pages}
    valid_ids = set(page_lookup)
    returned_ids = set(draft.source_ids)

    if not draft.supported:
        return PolicyAnswer(
            answer="I could not find that in the handbook.",
            cited_pages=[],
            supported=False,
        )

    if not returned_ids or not returned_ids.issubset(valid_ids):
        return PolicyAnswer(
            answer="I could not find that in the handbook.",
            cited_pages=[],
            supported=False,
        )

    return PolicyAnswer(
        answer=draft.answer,
        cited_pages=sorted(page_lookup[source_id] for source_id in returned_ids),
        supported=True,
    )


async def lookup_claim_status(claim_id: str) -> ClaimStatus:
    server_path = Path(__file__).with_name("claims_mcp.py")

    async with MCPServerStdio(
        name="Claims Core",
        params={
            "command": sys.executable,
            "args": [str(server_path)],
        },
        cache_tools_list=True,
    ) as server:
        agent = Agent(
            name="Claim Status Agent",
            model=MODEL,
            instructions=(
                "Always call lookup_claim_status exactly once with the claim ID supplied by the "
                "user. Return the tool fields unchanged. Never invent claim status data."
            ),
            mcp_servers=[server],
            model_settings=ModelSettings(tool_choice="required"),
            output_type=ClaimStatus,
        )
        result = await Runner.run(agent, f"Look up claim {claim_id}.")
        return result.final_output


async def run_workflow(email_text: str, policy_question: str) -> dict:
    claim = await extract_claim(email_text)
    policy = await answer_policy_question(policy_question)
    status = await lookup_claim_status(claim.claim_id)

    return {
        "claim": vars(claim),
        "policy": vars(policy),
        "status": vars(status),
    }

What should the complete workflow contain?

The complete file preserves the existing extraction and retrieval logic. It adds a typed status result, a required MCP tool call, and one orchestration function for the browser.

Complete the browser result

The browser currently calls extraction and policy functions separately. It can now use run_workflow to receive all three result sections from one submission.

Switch back to app.py.
Replace the import section at the top of app.py with this code:

import asyncio
import os

import streamlit as st

from workflow import run_workflow

What changes in the import section?

The browser now imports the single orchestration function. That function owns extraction, policy retrieval, and MCP status lookup.

Replace the complete if submitted: section in app.py with this code:

if submitted:
    if not os.getenv("OPENAI_API_KEY"):
        st.markdown(
            "**Missing API key:** Set `OPENAI_API_KEY` in the PowerShell session that launches Streamlit."
        )
    else:
        st.markdown("Running structured extraction, handbook retrieval, and MCP lookup...")
        try:
            result = asyncio.run(run_workflow(email_text, policy_question))
        except Exception as exc:
            st.markdown(f"**Run failed:** `{type(exc).__name__}`")
            st.write(str(exc))
        else:
            st.markdown("## Structured intake")
            st.json(result["claim"])

            st.markdown("## Handbook-grounded answer")
            st.write(result["policy"]["answer"])
            citation_text = ", ".join(
                f"Page {page}" for page in result["policy"]["cited_pages"]
            )
            st.markdown(f"**Supported:** {result['policy']['supported']}")
            st.markdown(f"**Citations:** {citation_text or 'None'}")

            st.markdown("## MCP claim status")
            st.json(result["status"])

How does the completed submission work?

The API key check prevents model calls when the current PowerShell session lacks the required environment variable.
The run_workflow call runs the typed extraction, grounded policy answer, and MCP lookup.
The try block keeps runtime failures visible in the browser.
The final status section renders the tool-backed result as JSON.

Save app.py.

Before you run the completed workbench, do you expect the CLM-1002 status to match the synthetic server record?

Launch the completed workbench from the active PowerShell environment by running:

python -m streamlit run app.py

What does this command start?

Streamlit starts the browser workbench. The MCP subprocess starts later when the submitted workflow reaches the status lookup.

Keep the default synthetic water-damage email in the form.
Keep the default emergency water mitigation question in the policy field.
Click Analyze claim.

You should see structured intake data followed by a handbook-grounded answer citing Page 3. Under the MCP claim-status heading, you should see status Under review, owner Maya Chen, and next action Confirm drying invoices and the moisture report.

That closes the operational gap. Your workbench now retrieves claim status from the read-only synthetic claim system.

Missing the MCP claim status?

Confirm that claims_mcp.py sits in the same folder as workflow.py.
Check that the current PowerShell session still has OPENAI_API_KEY configured.
Confirm that the sample email still contains claim ID CLM-1002.

If the run still fails, help me trace the MCP subprocess and tool call.

✔️ Awesome, I've got everything!

The browser now displays structured intake, validated policy citations, and read-only MCP claim status in one run.

ⓧ I'd like to double check the full code

Compare your complete app.py file with this reference:

import asyncio
import os

import streamlit as st

from workflow import run_workflow

SAMPLE_EMAIL = """Subject: Water damage at 18 Pine Street

Hello Claims Team,
I am Elena Ruiz. My policy number is POL-1042 and the claim ID is CLM-1002.
A kitchen pipe burst on September 18, 2026 at 18 Pine Street, Portland.
I shut off the water and hired an emergency drying company. The lower cabinets
and flooring are wet. I have photos, but I have not received the drying invoice yet.
Please tell me what happens next.
"""

st.markdown("# Claims Triage Copilot")
st.markdown(
    "Synthetic data only. This assistant supports intake and research. "
    "A human reviewer retains every coverage, liability, payment, and denial decision."
)

with st.form("claim-workbench"):
    email_text = st.text_area(
        "Synthetic claims email",
        value=SAMPLE_EMAIL,
        height=260,
    )
    policy_question = st.text_area(
        "Policy question",
        value="Can emergency water mitigation start before inspection?",
        height=100,
    )
    submitted = st.form_submit_button("Analyze claim")

if submitted:
    if not os.getenv("OPENAI_API_KEY"):
        st.markdown(
            "**Missing API key:** Set `OPENAI_API_KEY` in the PowerShell session that launches Streamlit."
        )
    else:
        st.markdown("Running structured extraction, handbook retrieval, and MCP lookup...")
        try:
            result = asyncio.run(run_workflow(email_text, policy_question))
        except Exception as exc:
            st.markdown(f"**Run failed:** `{type(exc).__name__}`")
            st.write(str(exc))
        else:
            st.markdown("## Structured intake")
            st.json(result["claim"])

            st.markdown("## Handbook-grounded answer")
            st.write(result["policy"]["answer"])
            citation_text = ", ".join(
                f"Page {page}" for page in result["policy"]["cited_pages"]
            )
            st.markdown(f"**Supported:** {result['policy']['supported']}")
            st.markdown(f"**Citations:** {citation_text or 'None'}")

            st.markdown("## MCP claim status")
            st.json(result["status"])

What should the complete app contain?

The complete browser file preserves the existing synthetic form. Its submission path now calls run_workflow and renders all three result sections.

You now have one workbench that combines typed intake, validated handbook evidence, and read-only operational status. Next, you will turn those capabilities into repeatable local reliability checks.

Prove Reliability with Local Evals

The full browser flow now works from intake to claim status. The next question is whether that success repeats.

A successful demo cannot prove reliability. This step uses eval-driven development to turn six acceptance criteria into repeatable checks.

In this step, get ready to:

Encode six acceptance criteria in a local eval harness.
Run the harness to identify workflow regressions from its score.

Encode the acceptance criteria

Visual demos can hide nondeterministic failures. An eval harness runs fixed examples through the same functions that power the workbench.

Switch back to VS Code.
Create a file named evals.py inside the claims-triage-copilot folder using the Explorer sidebar.
Add the synthetic email fixture plus the first three extraction checks by copying this code into evals.py:

import asyncio

from workflow import answer_policy_question, extract_claim, lookup_claim_status

EVAL_EMAIL = """Subject: Stolen laptop

I am Sam Patel. Policy POL-9001. Claim ID CLM-1001.
My laptop was stolen from my locked car on September 20, 2026 in Seattle.
I reported it to police within two hours, but I do not have the report number yet.
"""


def record(checks: list[dict], name: str, passed: bool, detail: str) -> None:
    checks.append({"name": name, "passed": passed, "detail": detail})


async def run_evals() -> int:
    checks = []

    claim = await extract_claim(EVAL_EMAIL)
    record(checks, "Claim ID extraction", claim.claim_id == "CLM-1001", claim.claim_id)
    record(checks, "Policy number extraction", claim.policy_number == "POL-9001", claim.policy_number)
    record(
        checks,
        "Human review is mandatory",
        claim.human_review_required is True,
        str(claim.human_review_required),
    )

What does this code check?

The EVAL_EMAIL fixture supplies known claim details without using customer data.
The record() helper stores each check name plus its Boolean result.
The recorded detail makes a failed value visible in the terminal.
The first three checks verify claim ID extraction plus policy number extraction plus the application-enforced human-review gate.

Save evals.py.

You should see evals.py listed beside the existing project files in the Explorer sidebar.

Do You See the File Elsewhere?

Confirm that evals.py sits inside the claims-triage-copilot folder. Move it into that folder if VS Code created it elsewhere.

Still stuck? Help me locate my eval file.

The first checks now need a scoring loop. The script also needs an entry point so running the file starts the asynchronous harness.

Append the scoring loop plus the script entry point directly below the existing code in evals.py by copying this block:

    passed_count = sum(1 for check in checks if check["passed"])
    for check in checks:
        label = "PASS" if check["passed"] else "FAIL"
        print(f"[{label}] {check['name']}: {check['detail']}")

    print(f"\nScore: {passed_count}/{len(checks)} checks passed.")
    return 0 if passed_count == len(checks) else 1


if __name__ == "__main__":
    raise SystemExit(asyncio.run(run_evals()))

How Does the Score Work?

The pass counter totals checks whose passed value is true.
The loop prints PASS or FAIL beside each acceptance criterion.
The return value becomes a successful exit only when every check passes.
The main guard runs run_evals() when you execute the file directly.

Save evals.py.

This run uses your OpenAI API key and creates usage-based charges. It uses a synthetic email to keep the check isolated from real claims data.

Before you run the harness, consider this question: will all three extraction checks pass.

Run the intermediate eval harness from the activated environment in PowerShell with this command:

python evals.py

What Should You See?

A successful intermediate run prints three lines beginning with [PASS]. It finishes with Score: 3/3 checks passed..

That result proves the fixture reaches the existing extraction workflow. It also proves the human-review boundary survives the model response.

Do the Extraction Checks Fail?

Confirm that the PowerShell prompt still shows the active .venv environment.
Confirm that the current PowerShell session still contains the configured API key.
Read the detail printed after each failed check to identify the returned value.

Need help with the result? Help me diagnose my extraction eval failure.

Run and interpret the score

Extraction is only one part of the workflow. The complete harness also checks grounding behavior plus unsupported-question handling plus tool-backed claim status.

Find the line that starts with passed_count = inside run_evals().
Insert the remaining policy plus MCP checks immediately above that line by copying this block:

    policy = await answer_policy_question("How quickly must theft claims be reported?")
    record(
        checks,
        "Theft answer is grounded on Page 2",
        policy.supported and 2 in policy.cited_pages,
        str(policy.cited_pages),
    )

    unsupported = await answer_policy_question(
        "Does the handbook cover dental cleanings for pets?"
    )
    record(
        checks,
        "Unsupported question has no citation",
        not unsupported.supported and unsupported.cited_pages == [],
        unsupported.answer,
    )

    status = await lookup_claim_status("CLM-1002")
    record(
        checks,
        "MCP-backed status lookup",
        status.status == "Under review" and status.owner == "Maya Chen",
        f"{status.status} / {status.owner}",
    )

What Do the New Checks Prove?

The theft question must return a supported answer citing Page 2.
The unrelated pet question must return an unsupported result with an empty citation list.
The CLM-1002 lookup must return Under review from the synthetic claim system.
The same status check must return Maya Chen as the owner.

Save evals.py.

Before you run the full harness, consider this question: will all six checks pass.

Run the complete eval harness from the activated environment in PowerShell with this command:

python evals.py

How Should You Read the Result?

When every acceptance criterion holds, you should see six lines beginning with [PASS]. The final line should read Score: 6/6 checks passed..

A line beginning with [FAIL] identifies a regression in extraction, retrieval, schema enforcement, or the MCP integration.

Do You See a Failed Check?

Use the failed check name to identify which workflow boundary needs attention.
Compare the printed detail with the expected field or citation from that check.
Confirm that claims_mcp.py still contains the synthetic CLM-1002 record if the status check fails.

Need another pair of eyes? Help me trace my failed eval check.

Treat every failed check as a workflow regression that needs investigation.
Re-run the harness whenever you change the model, prompt, handbook, or MCP integration.

That is the evidence your demo was missing. The workflow now has a repeatable six-check reliability gate.

✔️ Awesome, I've got everything!

Your evals.py file now tests all six acceptance criteria. Keep the final passing score visible for your checkpoint.

ⓧ I'd like to double check the full code

import asyncio

from workflow import answer_policy_question, extract_claim, lookup_claim_status

EVAL_EMAIL = """Subject: Stolen laptop

I am Sam Patel. Policy POL-9001. Claim ID CLM-1001.
My laptop was stolen from my locked car on September 20, 2026 in Seattle.
I reported it to police within two hours, but I do not have the report number yet.
"""


def record(checks: list[dict], name: str, passed: bool, detail: str) -> None:
    checks.append({"name": name, "passed": passed, "detail": detail})


async def run_evals() -> int:
    checks = []

    claim = await extract_claim(EVAL_EMAIL)
    record(checks, "Claim ID extraction", claim.claim_id == "CLM-1001", claim.claim_id)
    record(checks, "Policy number extraction", claim.policy_number == "POL-9001", claim.policy_number)
    record(
        checks,
        "Human review is mandatory",
        claim.human_review_required is True,
        str(claim.human_review_required),
    )

    policy = await answer_policy_question("How quickly must theft claims be reported?")
    record(
        checks,
        "Theft answer is grounded on Page 2",
        policy.supported and 2 in policy.cited_pages,
        str(policy.cited_pages),
    )

    unsupported = await answer_policy_question(
        "Does the handbook cover dental cleanings for pets?"
    )
    record(
        checks,
        "Unsupported question has no citation",
        not unsupported.supported and unsupported.cited_pages == [],
        unsupported.answer,
    )

    status = await lookup_claim_status("CLM-1002")
    record(
        checks,
        "MCP-backed status lookup",
        status.status == "Under review" and status.owner == "Maya Chen",
        f"{status.status} / {status.owner}",
    )

    passed_count = sum(1 for check in checks if check["passed"])
    for check in checks:
        label = "PASS" if check["passed"] else "FAIL"
        print(f"[{label}] {check['name']}: {check['detail']}")

    print(f"\nScore: {passed_count}/{len(checks)} checks passed.")
    return 0 if passed_count == len(checks) else 1


if __name__ == "__main__":
    raise SystemExit(asyncio.run(run_evals()))

Secret mission

Defend Against Prompt Injection

A customer email can contain instructions aimed at the extractor. Add an adversarial eval case that proves human review stays mandatory. Confirm the typed schema cannot gain an approval field.

Clean Up Your Resources

Your workbench runs locally with no ongoing service charge while idle. New OpenAI API requests remain usage based, so choose whether to keep the project ready, pause it, or delete it.

Resources you used:

Local Python project folder claims-triage-copilot containing app.py, workflow.py, handbook.py, claims_mcp.py, evals.py, and .venv.
Local Streamlit process plus the short-lived stdio MCP subprocess used for claim-status lookups.
Current PowerShell session containing the active .venv environment and the session-only OPENAI_API_KEY value.

Keep everything running

No action is needed. Choose this option if you are still testing the workbench or expanding its eval coverage.

Leave the Streamlit process running in its PowerShell terminal.
Keep the PowerShell session open to preserve the active .venv environment and its session-only API key.
Keep the claims-triage-copilot folder so its source files and eval harness remain available.
Expect API charges only when you submit workbench requests or run evals.py.

Pause - I'll come back to this later

Stop the local processes to free up memory while keeping every project file. This also prevents accidental model calls while you are away.

Return to the PowerShell terminal running the Streamlit app.
Press Ctrl+C to stop the app.
Leave the active virtual environment by running this command:

deactivate

What Does This Command Do?

The command exits the active virtual environment while leaving the .venv folder intact for your next session.

Close the PowerShell window to clear the current-session OPENAI_API_KEY value.

Your source files remain inside claims-triage-copilot. No API cost accrues while the app is stopped.

Return to the claims-triage-copilot folder in PowerShell when you resume.
Replace your-api-key-here with your real key before running the commands below.
Restore the paused session by running these commands:

.venv\Scripts\Activate.ps1
$env:OPENAI_API_KEY = "your-api-key-here"
python -m streamlit run app.py

How Does This Restore the Project?

The first command activates the existing virtual environment.
The second command sets the API key only for the new PowerShell session.
The final command restarts the browser workbench.

Delete - I don't want to use this again

This option removes the local project resources so you can start fresh. The Recycle Bin gives you one final recovery point before the deletion becomes permanent.

Return to the PowerShell terminal running the Streamlit app.
Press Ctrl+C to stop the app.
Close the PowerShell window to end the virtual environment session and clear its session-only API key.

Protect an Exposed API Key

Deleting the project folder does not invalidate a key copied into a source file, terminal capture, or screenshot. Credential cleanup matters even after the local files are gone.

Revoke the exposed key in the OpenAI dashboard before deleting the folder.

In File Explorer, locate the claims-triage-copilot folder where you created it.
Select the claims-triage-copilot folder.
Press the Delete key to move the folder to the Recycle Bin.
Open the Recycle Bin.
Empty the Recycle Bin when you are ready to remove the folder permanently.

You should no longer see claims-triage-copilot in its former location. Your local project resources are now removed from the computer.

Nice Work!

You did it! You built a browser-based claims workbench with Streamlit. Its human-review boundary keeps consequential claims decisions with people.

You've learned how to:

Convert an unstructured synthetic claims email into a typed ClaimExtraction object using structured outputs. Enforce the mandatory human-review flag in application code.
Build retrieval-augmented generation over stable handbook blocks. Validate returned source IDs before resolving them into page citations or safely rejecting unsupported questions.
Connect a read-only local MCP tool for synthetic claim status. Apply eval-driven development to test extraction, grounding, unsupported questions, human review, plus tool-backed status.
Complete the Secret Mission by adding an adversarial claims email. Prove that prompt injection cannot disable human_review_required or add an approved field outside ClaimExtraction.

Ready to quiz yourself?