Build an AI Ticket Triage API

Introduction

30 Second Summary

A support inbox can hide an urgent customer crisis between routine billing questions and feature requests. If that ticket takes the ordinary route, the delay can affect every customer.

In this project, you will build a support-ticket triage API with FastAPI that turns unstructured tickets into typed decisions from Claude Sonnet 5. You will strengthen it with reproducible evaluation before packaging the tested service with Docker.

What You'll Build

You will demo a live API response that places Claude's recommendation beside the final guarded route so every safety override is visible immediately.

By the end of this project, you'll have:

• The ability to submit a support ticket and receive a typed decision containing category, priority, team, summary, and confidence.
• A measurable safety comparison that exposes 80% routing accuracy and 0% critical-incident recall before deterministic guardrails raise both metrics to 100%.
• A portable failure-safe service that runs in Docker and returns a controlled response when the AI provider is unavailable.
• Secret Mission: Add a human-review queue that captures low-confidence decisions and guardrail overrides for human escalation.

Are there any prerequisites?

You need Python 3.10 or later, VS Code, Docker Desktop, prior FastAPI experience, and an Anthropic API key with a small paid budget.

Live triage requests use paid Claude API tokens, while the fixture evaluation and automated tests make no API calls.

Before We Start

This is your chance to lock in the service you are building before any hands-on work begins. You are committing to a safety-aware AI support-ticket triage service where deterministic rules own critical security and production-outage routing.

Prepare the Project

Critical routing rules need a consistent Python environment. Pinned dependencies make the service reproducible when you build the API later.

Your Anthropic API key can authorize paid Claude requests. This step keeps that credential in an ignored local file before the API makes its first request.

In this step, get ready to:

• Confirm that your local runtime meets the project requirements.
• Create the project skeleton with pinned configuration.
• Prepare an isolated environment with protected credentials.

Verify your runtime tools

Python runs the service locally. Docker Desktop provides the container engine used in the final step.

• Check your installed Python version by running this command:

python --version

What does this command check?

The command reports the Python interpreter available in your current terminal. This project requires Python 3.10 or later.

✔️ I see version 3.10 or higher

Your interpreter meets the project requirement. That gives every pinned package a compatible runtime.

ⓧ I see an older version

The installed interpreter is below the project requirement. Update Python before creating the virtual environment.

• Visit the official Python downloads page.
• Install Python 3.10 or later.
• Start a fresh terminal session after the installation finishes.
• Repeat the version check by running this command:

python --version

What should you see?

You should see Python 3.10 or a newer release. That output confirms the updated interpreter is available.

ⓧ Command not found

Your terminal cannot find the Python command. Install the required interpreter before continuing.

• Visit the official Python downloads page.
• Install Python 3.10 or later.
• Start a fresh terminal session after the installation finishes.
• Confirm the installation by running this command:

python --version

What should you see?

You should see a Python version of 3.10 or later. The version output confirms your terminal can access the interpreter.

• Check the Docker Desktop client and local engine by running this command:

docker version

What does this command check?

The command checks whether the Docker client can communicate with the local container engine. A successful check includes information about both components.

✔️ I see client and server information

Your container engine is ready. You can use it to build the packaged version of the API later.

ⓧ I only see client information

Docker is installed on your machine. The missing server information means the local engine still needs to start.

• Press Cmd+Space on macOS or the Windows key on Windows to open system search.
• Type Docker Desktop into the search field.
• Press Enter to start Docker Desktop.
• Complete any first-run setup shown by Docker Desktop.
• Repeat the engine check by running this command:

docker version

What should you see?

You should see information about the client and server. The server information confirms that the local engine is running.

ⓧ Command not found

The Docker command is unavailable in your terminal. Install Docker Desktop before continuing.

• Visit the official Docker Desktop installation guide.
• Install Docker Desktop for your operating system.
• Start Docker Desktop through system search.
• Complete any first-run setup shown by Docker Desktop.
• Confirm the installation by running this command:

docker version

What should you see?

You should see information about the Docker client and server. Those details confirm that the command and local engine are available.

Create the project skeleton

The project separates API code from automated tests through two Python packages. VS Code gives you a file sidebar for creating that structure.

• Press Cmd+Space on macOS or the Windows key on Windows to open system search.
• Type VS Code into the search field.
• Press Enter to start VS Code.

You will see the VS Code welcome window. The next action gives the project a named home on your Desktop.

• Use the folder picker in VS Code to create a Desktop folder named ai-ticket-triage.
• Select the ai-ticket-triage folder as the folder to open.

You will see ai-ticket-triage at the top of the file sidebar. Every project artifact now has a concrete location.

• Create an app folder inside ai-ticket-triage using the sidebar's folder control.
• Create an empty app/__init__.py file using the sidebar's file control.

The sidebar now shows __init__.py inside app/. This empty marker makes the directory an importable Python package.

• Create a tests folder inside ai-ticket-triage using the sidebar's folder control.
• Create an empty tests/__init__.py file using the sidebar's file control.

The sidebar now shows a second importable package under tests/. Runtime code and automated checks have separate homes.

• Create requirements.txt inside ai-ticket-triage using the sidebar's file control.
• Create pyproject.toml inside ai-ticket-triage using the sidebar's file control.

The two configuration files are now visible beside app/ and tests/.

• Create .env.example inside ai-ticket-triage using the sidebar's file control.
• Create .gitignore inside ai-ticket-triage using the sidebar's file control.

The complete skeleton is now visible in the sidebar. Next, each configuration file gets its exact contents.

• Select requirements.txt in the file sidebar.
• Add the pinned dependency list by copying the code below:

fastapi[standard]==0.141.1
anthropic==1.3.0
pydantic==2.13.5
python-dotenv==1.2.3
httpx==0.28.1
pytest==9.1.1

What do these dependencies provide?

• The FastAPI package provides the web framework and its standard development tools.
• The Anthropic Python SDK connects the service to Claude.
• Pydantic validates the ticket and triage decision models.
• python-dotenv loads local environment values into the service.
• HTTPX supports API testing through the FastAPI test client.
• pytest runs the deterministic test suite.

• Save requirements.txt.
• Confirm that the saved file contains six pinned dependency lines.

Missing a dependency line?

Compare each package name and version with the reference block. A missing character can make the installation resolve a different package or fail.

Ask for help comparing the dependency file.

• Select pyproject.toml in the file sidebar.
• Add the FastAPI application entrypoint by copying the code below:

[tool.fastapi]
entrypoint = "app.main:app"

What does this configuration do?

The entrypoint tells the FastAPI command where the application object belongs. The value app.main:app points to the app object that you create in app/main.py during the next step.

• Save pyproject.toml.
• Confirm that app.main:app appears on the second line.

• Select .env.example in the file sidebar.
• Document the required environment variable by copying the code below:

ANTHROPIC_API_KEY=your-api-key-here

Why use an example file?

The ANTHROPIC_API_KEY name documents the environment variable expected by the SDK. The placeholder keeps the reusable example free of credentials.

• Save .env.example.
• Confirm that the file contains one variable assignment.

• Select .gitignore in the file sidebar.
• Add the local exclusion rules by copying the code below:

.env
.venv/
__pycache__/
*.py[cod]
.pytest_cache/
.DS_Store

What do these exclusions protect?

The first rule keeps the local credential file outside Git history. The remaining rules exclude the virtual environment and generated local files.

• Save .gitignore.
• Confirm that .env is the first line.

Is the secret file missing from the exclusions?

Make sure the first line contains exactly .env. A different spelling does not match the local credential file.

Ask for help checking the exclusion rules.

Use this comparison before installing anything. The empty package markers are represented as checks because a code block cannot display an empty file.

✔️ Awesome, I've got everything!

• Save every project file in VS Code.
• Confirm that app/__init__.py is completely empty.
• Confirm that tests/__init__.py is completely empty.

ⓧ I'd like to double check the full code

fastapi[standard]==0.141.1
anthropic==1.3.0
pydantic==2.13.5
python-dotenv==1.2.3
httpx==0.28.1
pytest==9.1.1

What belongs in this file?

This file contains the six exact dependency pins used by the project.

[tool.fastapi]
entrypoint = "app.main:app"

What belongs in this file?

This file contains the FastAPI application entrypoint.

ANTHROPIC_API_KEY=your-api-key-here

What belongs in this file?

This file documents the required environment variable with a safe placeholder.

.env
.venv/
__pycache__/
*.py[cod]
.pytest_cache/
.DS_Store

What belongs in this file?

This file excludes the local secret and generated artifacts from Git.

• Confirm that app/__init__.py contains zero lines.
• Confirm that tests/__init__.py contains zero lines.

Install dependencies and protect the key

A virtual environment isolates this project's packages from other Python work on your machine. Activating .venv also makes the final Python check use the project interpreter.

• Return to the terminal panel in VS Code.
• Prepare the isolated environment by running these commands:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

What do these commands do?

• The first command creates an isolated Python environment inside .venv.
• The second command activates that environment in the current terminal.
• The third command installs the exact versions from requirements.txt.

• Verify the installed pytest release by running this command:

pytest --version

What should you see?

You should see pytest version 9.1.1. That output proves the pinned dependencies are available inside the active environment.

That is the dependency setup complete. Your project can now reproduce the same package versions each time you activate .venv.

Did the dependency installation fail?

Confirm that the active Python version is 3.10 or later. Compare requirements.txt with the full-code reference if the installer reports a package or version problem.

Ask for help with the installation output.

Your API key authorizes billable Claude requests. Keeping it inside the ignored .env file reduces the risk of committing it with your source.

• Duplicate .env.example using the file sidebar.
• Rename the duplicate to .env.

You will see .env beside .env.example in the project directory. The example remains safe to share.

• Select .env in the file sidebar.
• Replace your-api-key-here with your Anthropic API key.
• Save .env.

Your real key now exists only in the local environment file. Keep the key out of screenshots and shared code.

• Return to .gitignore in the file sidebar.
• Confirm that .env remains the first exclusion rule.

Before you run the final checks, make a quick prediction about the three version signals you expect from the active environment.

• Check the active Python interpreter by running this command:

python --version

What should you see?

You should see Python 3.10 or later. The result comes from the active .venv environment.

• Check the installed pytest release by running this command:

pytest --version

What should you see?

You should see pytest version 9.1.1. This confirms that the pinned test dependency is installed.

• Check the Docker client and local engine by running this command:

docker version

What should you see?

You should see information about both the Docker client and server. This confirms that Docker Desktop is ready for the containerization step.

Your project foundation is ready. The active environment has the pinned test stack while the ignored .env file keeps your Anthropic key local.

Did one of the final checks fail?

Activate .venv again if Python or pytest reports the wrong environment. Start Docker Desktop if the Docker client cannot reach the local engine.

Ask for help interpreting the checks.

The reproducible environment is in place. Next, you will turn this skeleton into a live typed triage API.

Build the Live Triage API

Your reproducible project shell is ready. The API still has no live path that turns a support ticket into a routing decision.

Now you will use FastAPI to serve that path. Pydantic structured outputs keep each Claude Sonnet 5 response inside a typed contract.

This first live version returns Claude's typed decision unchanged. That direct path gives you a useful baseline before you introduce reliability controls.

In this step, get ready to:

• Define the typed contract for tickets and triage results.
• Connect the service to Claude through a replaceable provider.
• Serve a live triage endpoint through FastAPI.

Define the triage contract

A typed contract limits every routing choice to values your application understands. The enums define allowed routes before a ticket reaches the model.

• In the VS Code Explorer sidebar, create models.py inside the existing app/ folder.
• Add the routing enums to app/models.py by pasting this code:

from enum import Enum

from pydantic import BaseModel, Field


class Category(str, Enum):
    ACCOUNT = "account"
    BILLING = "billing"
    INCIDENT = "incident"
    PRODUCT = "product"
    SECURITY = "security"
    TECHNICAL = "technical"


class Priority(str, Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    CRITICAL = "critical"


class Team(str, Enum):
    ACCOUNT_SUPPORT = "account_support"
    BILLING = "billing"
    INCIDENT_RESPONSE = "incident_response"
    PRODUCT = "product"
    SECURITY = "security"
    TECHNICAL_SUPPORT = "technical_support"

What do these enums control?

• Category limits the type of support problem.
• Priority limits the operational urgency.
• Team limits the destination that receives the ticket.
• String-backed enums produce readable values in API responses.

• Save app/models.py.
• Confirm that the editor shows the three enum definitions without highlighting a syntax problem.

Seeing a syntax problem in the enums?

Check that every enum member stays indented beneath its class. Confirm that each quoted value has matching quotation marks.

Ask for help with the enum definitions.

The API also needs models for incoming tickets and outgoing results. These models preserve the original AI decision so later safety logic can show whether it changed the route.

• Add the ticket and decision models below the Team class by pasting this code:

class Ticket(BaseModel):
    ticket_id: str
    subject: str
    description: str


class AIDecision(BaseModel):
    category: Category
    priority: Priority
    team: Team
    summary: str
    confidence: float = Field(ge=0.0, le=1.0)


class TriageResult(BaseModel):
    ticket_id: str
    model_decision: AIDecision
    final_decision: AIDecision
    guardrail_applied: bool = False
    guardrail_reason: str | None = None

How does the contract protect the API?

• Ticket defines the three fields accepted from a caller.
• AIDecision requires a valid routing category.
• AIDecision requires a valid priority.
• AIDecision requires a valid destination team.
• Field keeps confidence between 0 and 1.
• TriageResult separates the model recommendation from the operational result.

• Save app/models.py.
• Confirm that models.py now contains all six class definitions.

Seeing a model definition warning?

Check that BaseModel and Field remain in the Pydantic import. Check that each model starts at the left edge of the file.

Ask for help with the model contract.

✔️ Awesome, I've got everything!

Your models.py file now defines the complete request and response contract.

ⓧ I'd like to double check the full code

from enum import Enum

from pydantic import BaseModel, Field


class Category(str, Enum):
    ACCOUNT = "account"
    BILLING = "billing"
    INCIDENT = "incident"
    PRODUCT = "product"
    SECURITY = "security"
    TECHNICAL = "technical"


class Priority(str, Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    CRITICAL = "critical"


class Team(str, Enum):
    ACCOUNT_SUPPORT = "account_support"
    BILLING = "billing"
    INCIDENT_RESPONSE = "incident_response"
    PRODUCT = "product"
    SECURITY = "security"
    TECHNICAL_SUPPORT = "technical_support"


class Ticket(BaseModel):
    ticket_id: str
    subject: str
    description: str


class AIDecision(BaseModel):
    category: Category
    priority: Priority
    team: Team
    summary: str
    confidence: float = Field(ge=0.0, le=1.0)


class TriageResult(BaseModel):
    ticket_id: str
    model_decision: AIDecision
    final_decision: AIDecision
    guardrail_applied: bool = False
    guardrail_reason: str | None = None

How to use this reference

This reference shows the complete saved state of app/models.py. Compare class names and field types carefully.

Connect the Claude provider

The provider owns the external AI call. A shared interface keeps the route independent from the live provider implementation.

• In the VS Code Explorer sidebar, create provider.py inside the existing app/ folder.
• Add the provider setup to app/provider.py by pasting this code:

from typing import Protocol

import anthropic
from dotenv import load_dotenv

from app.models import AIDecision, Ticket

load_dotenv()

TRIAGE_PROMPT = """
Classify the support ticket for operational routing.
Choose exactly one category: account, billing, incident, product, security, technical.
Choose exactly one priority: low, medium, high, critical.
Choose exactly one team: account_support, billing, incident_response, product, security, technical_support.
Return a concise summary and a confidence value from 0 to 1.
""".strip()


class Triager(Protocol):
    def triage(self, ticket: Ticket) -> AIDecision: ...


class TriageProviderError(RuntimeError):
    pass

What does the provider setup do?

• load_dotenv reads the local environment file before the client is created.
• TRIAGE_PROMPT defines the routing task for the model.
• Triager describes the method every provider must expose.
• The protocol creates a seam for dependency injection.
• TriageProviderError gives the API one stable provider failure type.

• Save app/provider.py.
• Confirm that the prompt and two support classes appear below the imports.

Seeing unresolved provider imports?

Confirm that VS Code is using the active virtual environment from the previous step. Check that AIDecision and Ticket match the names in app/models.py.

Ask for help with the provider setup.

ClaudeTriager turns a typed ticket into a prompt. The SDK then parses the model response directly into AIDecision.

• Add ClaudeTriager below TriageProviderError by pasting this code:

class ClaudeTriager:
    def __init__(self) -> None:
        self.client = anthropic.Anthropic()

    def triage(self, ticket: Ticket) -> AIDecision:
        prompt = (
            f"{TRIAGE_PROMPT}\n\n"
            f"Ticket ID: {ticket.ticket_id}\n"
            f"Subject: {ticket.subject}\n"
            f"Description: {ticket.description}"
        )
        try:
            response = self.client.messages.parse(
                model="claude-sonnet-5",
                max_tokens=256,
                messages=[{"role": "user", "content": prompt}],
                output_format=AIDecision,
            )
        except anthropic.APIError as exc:
            raise TriageProviderError("Claude triage request failed") from exc

        if response.parsed_output is None:
            raise TriageProviderError("Claude returned no parsed triage decision")
        return response.parsed_output

How does ClaudeTriager produce typed output?

• The prompt combines the routing instructions with the ticket fields.
• The parse call uses claude-sonnet-5.
• The output format points to AIDecision.
• The parsed result has already passed the model contract.
• Provider failures become TriageProviderError before they reach the route.

• Save app/provider.py.
• Confirm that ClaudeTriager contains both its client setup and triage method.

Seeing a problem in ClaudeTriager?

Check the indentation inside the class. Confirm that output_format points to AIDecision.

Ask for help with the provider implementation.

✔️ Awesome, I've got everything!

Your provider now converts a ticket into a schema-validated Claude decision.

ⓧ I'd like to double check the full code

from typing import Protocol

import anthropic
from dotenv import load_dotenv

from app.models import AIDecision, Ticket

load_dotenv()

TRIAGE_PROMPT = """
Classify the support ticket for operational routing.
Choose exactly one category: account, billing, incident, product, security, technical.
Choose exactly one priority: low, medium, high, critical.
Choose exactly one team: account_support, billing, incident_response, product, security, technical_support.
Return a concise summary and a confidence value from 0 to 1.
""".strip()


class Triager(Protocol):
    def triage(self, ticket: Ticket) -> AIDecision: ...


class TriageProviderError(RuntimeError):
    pass


class ClaudeTriager:
    def __init__(self) -> None:
        self.client = anthropic.Anthropic()

    def triage(self, ticket: Ticket) -> AIDecision:
        prompt = (
            f"{TRIAGE_PROMPT}\n\n"
            f"Ticket ID: {ticket.ticket_id}\n"
            f"Subject: {ticket.subject}\n"
            f"Description: {ticket.description}"
        )
        try:
            response = self.client.messages.parse(
                model="claude-sonnet-5",
                max_tokens=256,
                messages=[{"role": "user", "content": prompt}],
                output_format=AIDecision,
            )
        except anthropic.APIError as exc:
            raise TriageProviderError("Claude triage request failed") from exc

        if response.parsed_output is None:
            raise TriageProviderError("Claude returned no parsed triage decision")
        return response.parsed_output

How to use this reference

This reference shows the complete saved state of app/provider.py. Compare the prompt and provider method with your file.

Serve and test the live API

The application layer turns the typed provider into HTTP routes. The health route gives you a quick runtime check before the paid triage request.

• In the VS Code Explorer sidebar, create main.py inside the existing app/ folder.
• Add the application setup and health route to app/main.py by pasting this code:

from typing import Annotated

from fastapi import Depends, FastAPI, HTTPException

from app.models import Ticket, TriageResult
from app.provider import ClaudeTriager, Triager, TriageProviderError

app = FastAPI(title="AI Support Ticket Triage API", version="1.0.0")


def get_triager() -> Triager:
    return ClaudeTriager()


@app.get("/health")
def health() -> dict[str, str]:
    return {"status": "ok"}

What does the application setup provide?

• FastAPI creates the web application.
• get_triager supplies the live provider through a replaceable dependency.
• The health route proves that the local service can answer a request.
• The provider error import prepares the triage route for a controlled failure response.

• Save app/main.py.
• Start the development server in the terminal with the active virtual environment by running this command:

fastapi dev

What does this command do?

FastAPI reads the entry point from pyproject.toml. The development server reloads after saved Python changes.

Keep this process running while you test the routes. The terminal shows each request as it reaches the service.

• Visit http://127.0.0.1:8000/health in your browser.

You should see {"status":"ok"}. This proves the application imports successfully and can serve a response.

Health route not responding?

Check the terminal for an import or syntax problem. Confirm that the virtual environment remains active in the terminal running the server.

Ask for help with the health route.

The live server is now ready for its useful route. The triage route receives a typed ticket through FastAPI's dependency system.

• Return to app/main.py in VS Code.
• Add the triage route below the health() function by pasting this code:

@app.post("/triage", response_model=TriageResult)
def triage_ticket(
    ticket: Ticket,
    triager: Annotated[Triager, Depends(get_triager)],
) -> TriageResult:
    try:
        model_decision = triager.triage(ticket)
    except TriageProviderError as exc:
        raise HTTPException(
            status_code=503,
            detail="AI triage provider unavailable",
        ) from exc

    return TriageResult(
        ticket_id=ticket.ticket_id,
        model_decision=model_decision,
        final_decision=model_decision,
    )

How does the triage route work?

• FastAPI validates the request body as a Ticket.
• Depends supplies an object that follows the Triager protocol.
• The provider produces the typed model decision.
• A provider failure becomes a stable HTTP response.
• The initial final decision repeats the model decision unchanged.

• Save app/main.py.
• Visit http://127.0.0.1:8000/docs in your browser.
• Confirm that the documentation lists GET /health.
• Confirm that the documentation lists POST /triage.

Triage route missing from the documentation?

Confirm that the route decorator starts at the left edge of app/main.py. Check the server terminal for a reload failure after the save.

Ask for help with the route registration.

✔️ Awesome, I've got everything!

Your main.py file now serves both project routes.

ⓧ I'd like to double check the full code

from typing import Annotated

from fastapi import Depends, FastAPI, HTTPException

from app.models import Ticket, TriageResult
from app.provider import ClaudeTriager, Triager, TriageProviderError

app = FastAPI(title="AI Support Ticket Triage API", version="1.0.0")


def get_triager() -> Triager:
    return ClaudeTriager()


@app.get("/health")
def health() -> dict[str, str]:
    return {"status": "ok"}


@app.post("/triage", response_model=TriageResult)
def triage_ticket(
    ticket: Ticket,
    triager: Annotated[Triager, Depends(get_triager)],
) -> TriageResult:
    try:
        model_decision = triager.triage(ticket)
    except TriageProviderError as exc:
        raise HTTPException(
            status_code=503,
            detail="AI triage provider unavailable",
        ) from exc

    return TriageResult(
        ticket_id=ticket.ticket_id,
        model_decision=model_decision,
        final_decision=model_decision,
    )

How to use this reference

This reference shows the complete saved state of app/main.py. Compare the imports and both route functions with your file.

This request uses paid tokens

The next check sends one live request to Claude using your local API key. The included ticket is short so the verification stays focused.

• Return to the POST /triage section in the API documentation.
• Enable request editing for the route.
• Replace the generated request body by pasting this ticket:

{
  "ticket_id": "T-006",
  "subject": "Cannot open dashboard",
  "description": "The dashboard will not load for my account."
}

What does this request contain?

The ticket has the exact three fields required by Ticket. The identifier lets the response link its decision back to the request.

The subject gives the model a short signal. The description provides the routing context.

Before you submit the ticket, which route do you think Claude will choose for an account-specific dashboard problem?

• Submit the request through the interactive documentation.

You should see a successful typed response. Each decision contains category, priority, team, summary, and confidence.

The model_decision object contains Claude's recommendation. The final_decision object repeats those values in this first version.

You have the first live win: an unstructured support ticket now becomes a schema-validated routing decision through your own API.

Live triage request not succeeding?

Confirm that .env contains your Anthropic API key under the documented variable name. Keep the secret out of screenshots and chat messages.

Check the server terminal for a provider or validation problem. A request-body problem means one of the three ticket fields is missing or misspelled.

Ask for help without sharing your API key.

Your live endpoint now turns support text into a typed Claude decision. Next up, you will test whether a valid decision is also safe enough for real operational routing.

Evaluate the Unsafe Route

Your live FastAPI endpoint now turns support tickets into typed decisions. That gives you a working response path.

Schema-valid structured output guarantees shape. Operational safety still depends on whether the priority and team match the real incident.

In this step, get ready to:

• Define five labeled ticket cases with fixed AIDecision outputs.
• Build a fixture triager that measures routing accuracy.
• Measure critical-incident recall against the outage label.

Define the labeled fixture

A labeled evaluation pairs each ticket with its expected route. Fixed model decisions make the comparison reproducible without spending API credits.

• In VS Code's Explorer sidebar, select the app folder.
• Click the New File icon.
• Name the file evaluation.py.
• Add the evaluation case schema plus five fixed decisions by pasting this code into app/evaluation.py:

from dataclasses import dataclass
from typing import Iterable

from app.models import AIDecision, Category, Priority, Team, Ticket
from app.provider import Triager


@dataclass(frozen=True)
class EvaluationCase:
    ticket: Ticket
    expected_priority: Priority
    expected_team: Team
    fixture_decision: AIDecision


LABELED_CASES = (
    EvaluationCase(Ticket(ticket_id="T-001", subject="Charged twice", description="My card shows two charges for the same invoice."), Priority.MEDIUM, Team.BILLING, AIDecision(category=Category.BILLING, priority=Priority.MEDIUM, team=Team.BILLING, summary="Possible duplicate charge.", confidence=0.96)),
    EvaluationCase(Ticket(ticket_id="T-002", subject="Password reset loop", description="The reset link keeps returning me to the sign-in page."), Priority.MEDIUM, Team.ACCOUNT_SUPPORT, AIDecision(category=Category.ACCOUNT, priority=Priority.MEDIUM, team=Team.ACCOUNT_SUPPORT, summary="Customer cannot complete a password reset.", confidence=0.91)),
    EvaluationCase(Ticket(ticket_id="T-003", subject="Dark mode request", description="Please add a dark theme to the dashboard."), Priority.LOW, Team.PRODUCT, AIDecision(category=Category.PRODUCT, priority=Priority.LOW, team=Team.PRODUCT, summary="Product feature request for dark mode.", confidence=0.94)),
    EvaluationCase(Ticket(ticket_id="T-004", subject="Intermittent API errors", description="Every customer request returns 503 and production is down."), Priority.CRITICAL, Team.INCIDENT_RESPONSE, AIDecision(category=Category.TECHNICAL, priority=Priority.HIGH, team=Team.TECHNICAL_SUPPORT, summary="Customers are seeing API errors.", confidence=0.88)),
    EvaluationCase(Ticket(ticket_id="T-005", subject="CSV export formatting", description="Exported dates use the wrong display format."), Priority.LOW, Team.TECHNICAL_SUPPORT, AIDecision(category=Category.TECHNICAL, priority=Priority.LOW, team=Team.TECHNICAL_SUPPORT, summary="CSV date formatting issue.", confidence=0.89)),
)

What does this code do?

• The EvaluationCase class keeps each ticket beside its expected route and fixed decision.
• The LABELED_CASES collection provides five repeatable evaluation inputs.
• Each fixture decision is constructed as an AIDecision, so Pydantic validates the same typed contract used by the API.
• The fixture runs locally, so repeated evaluations make no Claude API calls.

• Save app/evaluation.py.
• Confirm evaluation.py is visible beneath the app folder in the Explorer sidebar.

Solid start: your five-ticket fixture is now saved locally without making an API request.

Seeing unresolved imports?

Confirm that evaluation.py sits inside the existing app folder. Check that the active virtual environment from earlier is still selected in VS Code.

Help me fix unresolved imports in app/evaluation.py.

Build the evaluator

The evaluator treats a route as correct only when both its priority and team match the label. The critical-ticket score applies that same standard to the cases labeled with critical priority.

• In app/evaluation.py, place your cursor after the closing parenthesis for LABELED_CASES.
• Build the fixture provider plus scoring logic by pasting this code:

class FixtureTriager:
    def __init__(self, cases: Iterable[EvaluationCase] = LABELED_CASES) -> None:
        self.decisions = {case.ticket.ticket_id: case.fixture_decision for case in cases}

    def triage(self, ticket: Ticket) -> AIDecision:
        return self.decisions[ticket.ticket_id]


def evaluate(triager: Triager, cases: Iterable[EvaluationCase] = LABELED_CASES) -> dict[str, float]:
    case_list = list(cases)
    correct_routes = sum(int(triager.triage(case.ticket).priority == case.expected_priority and triager.triage(case.ticket).team == case.expected_team) for case in case_list)
    critical_cases = [case for case in case_list if case.expected_priority == Priority.CRITICAL]
    critical_hits = sum(int(triager.triage(case.ticket).priority == case.expected_priority and triager.triage(case.ticket).team == case.expected_team) for case in critical_cases)
    return {"routing_accuracy": round(correct_routes / len(case_list), 2), "critical_incident_recall": round(critical_hits / len(critical_cases), 2)}


if __name__ == "__main__":
    baseline = evaluate(FixtureTriager())
    print(f"Baseline evaluation: routing_accuracy={baseline['routing_accuracy']:.0%}, critical_incident_recall={baseline['critical_incident_recall']:.0%}")

How does the evaluation work?

• The FixtureTriager satisfies the existing Triager protocol.
• Its triage() method returns the fixed decision associated with a ticket ID.
• The evaluate() function counts exact priority and team matches.
• The module entry point formats both metrics as percentages when you execute the file.

• Save app/evaluation.py.
• Confirm the final print(...) call appears at the bottom of the file.

The baseline evaluator now has every required piece.

✔️ Awesome, I've got everything!

Your saved app/evaluation.py file is ready to run.

ⓧ I'd like to double check the full code

from dataclasses import dataclass
from typing import Iterable

from app.models import AIDecision, Category, Priority, Team, Ticket
from app.provider import Triager


@dataclass(frozen=True)
class EvaluationCase:
    ticket: Ticket
    expected_priority: Priority
    expected_team: Team
    fixture_decision: AIDecision


LABELED_CASES = (
    EvaluationCase(Ticket(ticket_id="T-001", subject="Charged twice", description="My card shows two charges for the same invoice."), Priority.MEDIUM, Team.BILLING, AIDecision(category=Category.BILLING, priority=Priority.MEDIUM, team=Team.BILLING, summary="Possible duplicate charge.", confidence=0.96)),
    EvaluationCase(Ticket(ticket_id="T-002", subject="Password reset loop", description="The reset link keeps returning me to the sign-in page."), Priority.MEDIUM, Team.ACCOUNT_SUPPORT, AIDecision(category=Category.ACCOUNT, priority=Priority.MEDIUM, team=Team.ACCOUNT_SUPPORT, summary="Customer cannot complete a password reset.", confidence=0.91)),
    EvaluationCase(Ticket(ticket_id="T-003", subject="Dark mode request", description="Please add a dark theme to the dashboard."), Priority.LOW, Team.PRODUCT, AIDecision(category=Category.PRODUCT, priority=Priority.LOW, team=Team.PRODUCT, summary="Product feature request for dark mode.", confidence=0.94)),
    EvaluationCase(Ticket(ticket_id="T-004", subject="Intermittent API errors", description="Every customer request returns 503 and production is down."), Priority.CRITICAL, Team.INCIDENT_RESPONSE, AIDecision(category=Category.TECHNICAL, priority=Priority.HIGH, team=Team.TECHNICAL_SUPPORT, summary="Customers are seeing API errors.", confidence=0.88)),
    EvaluationCase(Ticket(ticket_id="T-005", subject="CSV export formatting", description="Exported dates use the wrong display format."), Priority.LOW, Team.TECHNICAL_SUPPORT, AIDecision(category=Category.TECHNICAL, priority=Priority.LOW, team=Team.TECHNICAL_SUPPORT, summary="CSV date formatting issue.", confidence=0.89)),
)


class FixtureTriager:
    def __init__(self, cases: Iterable[EvaluationCase] = LABELED_CASES) -> None:
        self.decisions = {case.ticket.ticket_id: case.fixture_decision for case in cases}

    def triage(self, ticket: Ticket) -> AIDecision:
        return self.decisions[ticket.ticket_id]


def evaluate(triager: Triager, cases: Iterable[EvaluationCase] = LABELED_CASES) -> dict[str, float]:
    case_list = list(cases)
    correct_routes = sum(int(triager.triage(case.ticket).priority == case.expected_priority and triager.triage(case.ticket).team == case.expected_team) for case in case_list)
    critical_cases = [case for case in case_list if case.expected_priority == Priority.CRITICAL]
    critical_hits = sum(int(triager.triage(case.ticket).priority == case.expected_priority and triager.triage(case.ticket).team == case.expected_team) for case in critical_cases)
    return {"routing_accuracy": round(correct_routes / len(case_list), 2), "critical_incident_recall": round(critical_hits / len(critical_cases), 2)}


if __name__ == "__main__":
    baseline = evaluate(FixtureTriager())
    print(f"Baseline evaluation: routing_accuracy={baseline['routing_accuracy']:.0%}, critical_incident_recall={baseline['critical_incident_recall']:.0%}")

How should I use this reference?

Compare this reference with your saved file from top to bottom. Matching the fixture values exactly keeps the evaluation deterministic.

Run the unsafe baseline

The fixed decisions now face their operational labels. This run reveals whether valid objects also produce safe routes.

Before you run this, do you expect five schema-valid decisions to produce perfect routing scores?

• Run the deterministic evaluation from the terminal from earlier by running this command:

python -m app.evaluation

What should you see?

You should see routing_accuracy=80% and critical_incident_recall=0% in the baseline output.

Every fixture decision was constructed as an AIDecision, so schema validation passed.

🙋‍♀️ Evaluation command not completing?

Confirm the terminal is in the directory that contains the app folder. Check that the virtual environment from earlier is still active.

Help me troubleshoot the evaluation command.

• Return to app/evaluation.py in VS Code.
• Find the EvaluationCase containing ticket_id="T-004".
• Compare Priority.CRITICAL with the fixture value Priority.HIGH.
• Compare Team.INCIDENT_RESPONSE with the fixture value Team.TECHNICAL_SUPPORT.

The T-004 label expects critical priority and the incident-response team. Its fixture decision returns high priority and technical support.

That mismatch is intentional. The object is structurally valid while its route misses the only critical incident.

Why do the metrics fall short?

Routing accuracy requires both priority and team to match for every ticket. Four of the five fixture routes meet that standard.

Critical-incident recall applies the same exact-match rule to critical labels. The only critical ticket misses both required values, so its recall is zero.

You have reproduced a schema-valid operational failure without relying on variable model behavior. Next, you will give critical routing to deterministic safety rules.

Add Safety Guardrails

Your fixture exposed the gap that matters. A schema-valid decision from Claude can still send a critical outage to the wrong team.

A deterministic guardrail gives critical signals an explicit route outside model judgment. You will apply the same rule layer to the evaluation harness and the live FastAPI service.

In this step, get ready to:

• Define deterministic security and production-outage rules.
• Apply the rules to both operational decision paths.
• Prove that critical routing now survives evaluation and live use.

Create deterministic routing guardrails

Critical security incidents and production outages have known operational destinations. Matching those signals in code makes the final route reproducible.

• In the VS Code Explorer sidebar, right-click the app folder.
• Select New File.
• Enter guardrails.py as the file name.
• Add the deterministic signal rules to app/guardrails.py by pasting this code:

from app.models import AIDecision, Category, Priority, Team, Ticket, TriageResult

SECURITY_SIGNALS = ("customer records exposed", "data exfiltration", "security breach")
OUTAGE_SIGNALS = ("production is down", "all customers", "returns 503")


def apply_guardrails(ticket: Ticket, decision: AIDecision) -> TriageResult:
    text = f"{ticket.subject} {ticket.description}".lower()
    if any(signal in text for signal in SECURITY_SIGNALS):
        final_decision = AIDecision(category=Category.SECURITY, priority=Priority.CRITICAL, team=Team.SECURITY, summary=decision.summary, confidence=decision.confidence)
        return TriageResult(ticket_id=ticket.ticket_id, model_decision=decision, final_decision=final_decision, guardrail_applied=True, guardrail_reason="Deterministic security-incident signal")
    if any(signal in text for signal in OUTAGE_SIGNALS):
        final_decision = AIDecision(category=Category.INCIDENT, priority=Priority.CRITICAL, team=Team.INCIDENT_RESPONSE, summary=decision.summary, confidence=decision.confidence)
        return TriageResult(ticket_id=ticket.ticket_id, model_decision=decision, final_decision=final_decision, guardrail_applied=True, guardrail_reason="Deterministic production-outage signal")
    return TriageResult(ticket_id=ticket.ticket_id, model_decision=decision, final_decision=decision)

What does this code do?

• The signal tuples hold phrases that identify urgent security events or production outages.
• The combined ticket text is converted to lowercase so matching stays consistent.
• A matching rule creates a critical final decision while preserving the original model decision.
• A routine ticket keeps the model decision unchanged.

• Save app/guardrails.py.
• Confirm that guardrails.py appears under the app folder in the Explorer sidebar.

Guardrails file missing or underlined?

Check that guardrails.py sits inside the existing app folder. Confirm that each imported model name matches app/models.py exactly.

Help me debug the guardrails module.

✔️ Awesome, I've got everything!

Your guardrail module now owns the critical security and outage routes.

ⓧ I'd like to double check the full code

from app.models import AIDecision, Category, Priority, Team, Ticket, TriageResult

SECURITY_SIGNALS = ("customer records exposed", "data exfiltration", "security breach")
OUTAGE_SIGNALS = ("production is down", "all customers", "returns 503")


def apply_guardrails(ticket: Ticket, decision: AIDecision) -> TriageResult:
    text = f"{ticket.subject} {ticket.description}".lower()
    if any(signal in text for signal in SECURITY_SIGNALS):
        final_decision = AIDecision(category=Category.SECURITY, priority=Priority.CRITICAL, team=Team.SECURITY, summary=decision.summary, confidence=decision.confidence)
        return TriageResult(ticket_id=ticket.ticket_id, model_decision=decision, final_decision=final_decision, guardrail_applied=True, guardrail_reason="Deterministic security-incident signal")
    if any(signal in text for signal in OUTAGE_SIGNALS):
        final_decision = AIDecision(category=Category.INCIDENT, priority=Priority.CRITICAL, team=Team.INCIDENT_RESPONSE, summary=decision.summary, confidence=decision.confidence)
        return TriageResult(ticket_id=ticket.ticket_id, model_decision=decision, final_decision=final_decision, guardrail_applied=True, guardrail_reason="Deterministic production-outage signal")
    return TriageResult(ticket_id=ticket.ticket_id, model_decision=decision, final_decision=decision)

What should match?

Compare the signal phrases and every branch of apply_guardrails(). Security signals must route to the security team while outage signals must route to incident response.

Apply guardrails to the evaluation harness

The evaluation needs to measure the model-only route and the guarded route against the same five labels. That shared fixture makes the improvement attributable to the rule layer.

• In app/evaluation.py, find the import from app.models.
• Add this guardrail import directly above it:

from app.guardrails import apply_guardrails

Why import the rule layer?

The evaluation calls the production guardrail function directly. This keeps the measured behavior aligned with the route used by the API.

• Select the existing evaluate() function from its definition through its return statement.
• Replace the first part of that function with this chunk:

def evaluate(
    triager: Triager,
    cases: Iterable[EvaluationCase] = LABELED_CASES,
    use_guardrails: bool = False,
) -> dict[str, float]:
    case_list = list(cases)
    correct_routes = 0
    critical_total = 0
    critical_hits = 0

    for case in case_list:
        model_decision = triager.triage(case.ticket)
        final_decision = (
            apply_guardrails(case.ticket, model_decision).final_decision
            if use_guardrails
            else model_decision
        )

How does the comparison work?

The use_guardrails switch selects the decision under evaluation. The fixture produces the same model output on every run.

• Complete evaluate() by adding this chunk directly below the closing parenthesis:

        route_is_correct = (
            final_decision.priority == case.expected_priority
            and final_decision.team == case.expected_team
        )
        correct_routes += int(route_is_correct)

        if case.expected_priority == Priority.CRITICAL:
            critical_total += 1
            critical_hits += int(route_is_correct)

    return {
        "routing_accuracy": round(correct_routes / len(case_list), 2),
        "critical_incident_recall": round(critical_hits / critical_total, 2),
    }

What do the metrics count?

A route counts as correct only when its priority and team both match the label. Critical-incident recall applies that same exact-match rule to labeled critical tickets.

• Select the current execution block at the bottom of app/evaluation.py.
• Replace it with this evaluation entry point:

def main() -> None:
    triager = FixtureTriager()
    baseline = evaluate(triager, use_guardrails=False)
    guarded = evaluate(triager, use_guardrails=True)
    print(
        "Baseline evaluation: "
        f"routing_accuracy={baseline['routing_accuracy']:.0%}, "
        f"critical_incident_recall={baseline['critical_incident_recall']:.0%}"
    )
    print(
        "Guarded evaluation: "
        f"routing_accuracy={guarded['routing_accuracy']:.0%}, "
        f"critical_incident_recall={guarded['critical_incident_recall']:.0%}"
    )


if __name__ == "__main__":
    main()

Why run both evaluations?

Both measurements use the same tickets and fixture decisions. The only changed condition is whether apply_guardrails() controls the final route.

• Save app/evaluation.py.

Before you run the evaluation, predict whether one explicit outage rule can recover the missed critical ticket.

• Compare the baseline with the guarded result by running:

python -m app.evaluation

What should you see?

The baseline still reports routing_accuracy=80% and critical_incident_recall=0%. The guarded evaluation reports routing_accuracy=100% and critical_incident_recall=100%.

That change proves the schema-valid fixture stayed fixed while operational routing improved.

🙋‍♀️ Metrics unchanged or evaluation failing?

Confirm that the guarded call passes use_guardrails=True. Check that T-004 still contains the phrase production is down.

Help me debug the guarded evaluation.

✔️ Awesome, I've got everything!

Your fixed fixture now measures both the unsafe baseline and the guarded result.

ⓧ I'd like to double check the full code

from dataclasses import dataclass
from typing import Iterable

from app.guardrails import apply_guardrails
from app.models import AIDecision, Category, Priority, Team, Ticket
from app.provider import Triager


@dataclass(frozen=True)
class EvaluationCase:
    ticket: Ticket
    expected_priority: Priority
    expected_team: Team
    fixture_decision: AIDecision


LABELED_CASES = (
    EvaluationCase(
        ticket=Ticket(
            ticket_id="T-001",
            subject="Charged twice",
            description="My card shows two charges for the same invoice.",
        ),
        expected_priority=Priority.MEDIUM,
        expected_team=Team.BILLING,
        fixture_decision=AIDecision(
            category=Category.BILLING,
            priority=Priority.MEDIUM,
            team=Team.BILLING,
            summary="Possible duplicate charge.",
            confidence=0.96,
        ),
    ),
    EvaluationCase(
        ticket=Ticket(
            ticket_id="T-002",
            subject="Password reset loop",
            description="The reset link keeps returning me to the sign-in page.",
        ),
        expected_priority=Priority.MEDIUM,
        expected_team=Team.ACCOUNT_SUPPORT,
        fixture_decision=AIDecision(
            category=Category.ACCOUNT,
            priority=Priority.MEDIUM,
            team=Team.ACCOUNT_SUPPORT,
            summary="Customer cannot complete a password reset.",
            confidence=0.91,
        ),
    ),
    EvaluationCase(
        ticket=Ticket(
            ticket_id="T-003",
            subject="Dark mode request",
            description="Please add a dark theme to the dashboard.",
        ),
        expected_priority=Priority.LOW,
        expected_team=Team.PRODUCT,
        fixture_decision=AIDecision(
            category=Category.PRODUCT,
            priority=Priority.LOW,
            team=Team.PRODUCT,
            summary="Product feature request for dark mode.",
            confidence=0.94,
        ),
    ),
    EvaluationCase(
        ticket=Ticket(
            ticket_id="T-004",
            subject="Intermittent API errors",
            description="Every customer request returns 503 and production is down.",
        ),
        expected_priority=Priority.CRITICAL,
        expected_team=Team.INCIDENT_RESPONSE,
        fixture_decision=AIDecision(
            category=Category.TECHNICAL,
            priority=Priority.HIGH,
            team=Team.TECHNICAL_SUPPORT,
            summary="Customers are seeing API errors.",
            confidence=0.88,
        ),
    ),
    EvaluationCase(
        ticket=Ticket(
            ticket_id="T-005",
            subject="CSV export formatting",
            description="Exported dates use the wrong display format.",
        ),
        expected_priority=Priority.LOW,
        expected_team=Team.TECHNICAL_SUPPORT,
        fixture_decision=AIDecision(
            category=Category.TECHNICAL,
            priority=Priority.LOW,
            team=Team.TECHNICAL_SUPPORT,
            summary="CSV date formatting issue.",
            confidence=0.89,
        ),
    ),
)


class FixtureTriager:
    def __init__(self, cases: Iterable[EvaluationCase] = LABELED_CASES) -> None:
        self.decisions = {
            case.ticket.ticket_id: case.fixture_decision for case in cases
        }

    def triage(self, ticket: Ticket) -> AIDecision:
        return self.decisions[ticket.ticket_id]


def evaluate(
    triager: Triager,
    cases: Iterable[EvaluationCase] = LABELED_CASES,
    use_guardrails: bool = False,
) -> dict[str, float]:
    case_list = list(cases)
    correct_routes = 0
    critical_total = 0
    critical_hits = 0

    for case in case_list:
        model_decision = triager.triage(case.ticket)
        final_decision = (
            apply_guardrails(case.ticket, model_decision).final_decision
            if use_guardrails
            else model_decision
        )

        route_is_correct = (
            final_decision.priority == case.expected_priority
            and final_decision.team == case.expected_team
        )
        correct_routes += int(route_is_correct)

        if case.expected_priority == Priority.CRITICAL:
            critical_total += 1
            critical_hits += int(route_is_correct)

    return {
        "routing_accuracy": round(correct_routes / len(case_list), 2),
        "critical_incident_recall": round(critical_hits / critical_total, 2),
    }


def main() -> None:
    triager = FixtureTriager()
    baseline = evaluate(triager, use_guardrails=False)
    guarded = evaluate(triager, use_guardrails=True)
    print(
        "Baseline evaluation: "
        f"routing_accuracy={baseline['routing_accuracy']:.0%}, "
        f"critical_incident_recall={baseline['critical_incident_recall']:.0%}"
    )
    print(
        "Guarded evaluation: "
        f"routing_accuracy={guarded['routing_accuracy']:.0%}, "
        f"critical_incident_recall={guarded['critical_incident_recall']:.0%}"
    )


if __name__ == "__main__":
    main()

What should match?

Compare the imports and the complete evaluate() function. The entry point must print the baseline before the guarded result.

Guard the live API route

The evaluation now proves the rule layer works on fixed data. The live route needs the same final-decision path while preserving its existing provider-failure response.

• In app/main.py, find the imports below the FastAPI import.
• Add this guardrail import above the models import:

from app.guardrails import apply_guardrails

Why share one guardrail function?

The API and evaluation now call the same implementation. A passing evaluation therefore measures the rule layer used by live requests.

• Select the existing POST /triage route from its decorator through its final return statement.
• Replace that route with this guarded version:

@app.post("/triage", response_model=TriageResult)
def triage_ticket(ticket: Ticket, triager: Annotated[Triager, Depends(get_triager)]) -> TriageResult:
    try:
        model_decision = triager.triage(ticket)
    except TriageProviderError as exc:
        raise HTTPException(status_code=503, detail="AI triage provider unavailable") from exc
    return apply_guardrails(ticket, model_decision)

What happens on each path?

A successful provider response passes through apply_guardrails() before the API returns it. The response still exposes the original model decision beside the final decision.

The existing domain error still becomes HTTP 503 with the stable detail AI triage provider unavailable.

• Save app/main.py.
• Return to the terminal panel from earlier.
• Stop the earlier development server with Control-C if it is still running.
• Start the updated API by running:

fastapi dev

What does this command do?

FastAPI loads the package entry point from pyproject.toml and starts the development server. Keep this terminal running while you submit the ticket.

🙋‍♀️ Development server not starting?

Confirm that your virtual environment is active. Check the traceback for a file name that points to a missed import or indentation problem.

Help me debug the updated FastAPI server.

This next request uses paid Claude tokens. One short ticket keeps the call focused while proving the live path.

Before you submit it, predict which decision fields the deterministic rule will control.

• Open http://127.0.0.1:8000/docs in your browser.
• Expand POST /triage.
• Enable the request editor in the route panel.
• Replace the request body with this outage ticket:

            {
                "ticket_id": "T-004",
                "subject": "Intermittent API errors",
                "description": "Every customer request returns 503 and production is down."
            }

Why use this ticket?

The ticket contains the same production-outage signal as the deterministic fixture. Its wording gives the live guardrail a reproducible critical condition to detect.

• Submit the request from the route panel.

You should see model_decision and final_decision in the response. The final priority is critical while the final team is incident_response.

The response also shows guardrail_applied as true with the production-outage reason. That is the live proof that operational safety now owns the final route.

Live response missing the override?

Confirm that the ticket description contains production is down exactly. Check that the route returns apply_guardrails(ticket, model_decision).

Help me trace the missing live override.

✔️ Awesome, I've got everything!

Your live route now preserves Claude's recommendation while enforcing the final operational decision.

ⓧ I'd like to double check the full code

from typing import Annotated

from fastapi import Depends, FastAPI, HTTPException

from app.guardrails import apply_guardrails
from app.models import Ticket, TriageResult
from app.provider import ClaudeTriager, Triager, TriageProviderError

app = FastAPI(title="AI Support Ticket Triage API", version="1.0.0")


def get_triager() -> Triager:
    return ClaudeTriager()


@app.get("/health")
def health() -> dict[str, str]:
    return {"status": "ok"}


@app.post("/triage", response_model=TriageResult)
def triage_ticket(ticket: Ticket, triager: Annotated[Triager, Depends(get_triager)]) -> TriageResult:
    try:
        model_decision = triager.triage(ticket)
    except TriageProviderError as exc:
        raise HTTPException(status_code=503, detail="AI triage provider unavailable") from exc
    return apply_guardrails(ticket, model_decision)

What should match?

Confirm that app/main.py imports apply_guardrails. The triage route must preserve the existing provider error translation before returning the guarded result.

Your service now separates model judgment from operational authority. Next up, you will lock that behavior into automated tests and package the API in Docker.

Test and Containerize the API

Your guarded FastAPI service now reroutes critical incidents deterministically. It also returns a controlled response when Claude is unavailable.

Those paths still need repeatable proof that avoids paid API calls. Dependency-injected pytest tests provide that proof with fixed decisions.

A Docker image packages the same API into a portable runtime. You can verify the service locally without changing its response contract.

In this step, get ready to:

• Test the evaluation metrics and API behavior without making Anthropic API calls.
• Keep secrets and local development files outside the Docker image.
• Build the tagged image and verify the containerized health route.

Test guarded behavior without API calls

FastAPI dependency overrides replace the live ClaudeTriager with local test doubles. Each test controls the provider response while exercising the real route and guardrail code.

The first double returns a schema-valid outage decision with the wrong route. The test proves that apply_guardrails() changes that decision before the API responds.

• In VS Code, select the tests/ folder in the Explorer sidebar.
• Use the file creation control to create tests/test_api.py.
• Add the test client and successful provider stub by pasting this code into tests/test_api.py:

from fastapi.testclient import TestClient

from app.main import app, get_triager
from app.models import AIDecision, Category, Priority, Team, Ticket
from app.provider import TriageProviderError

client = TestClient(app)


class StubTriager:
    def triage(self, ticket: Ticket) -> AIDecision:
        return AIDecision(category=Category.TECHNICAL, priority=Priority.HIGH, team=Team.TECHNICAL_SUPPORT, summary="Customers are seeing API errors.", confidence=0.88)

What Does This Test Setup Do?

• TestClient sends requests through the FastAPI application without starting a separate server.
• StubTriager always returns the same high-priority technical decision.
• The fixed response makes the guardrail test reproducible and free of Anthropic API calls.

• Save tests/test_api.py.
• Confirm that test_api.py appears inside the tests/ folder in the Explorer sidebar.

Test File in the Wrong Folder?

Check that test_api.py sits beside tests/__init__.py. A file created beside the tests/ folder has the wrong path.

Help me place the API test file correctly.

• Add the outage override test below StubTriager by pasting this code:

def test_critical_outage_is_overridden() -> None:
    app.dependency_overrides[get_triager] = StubTriager
    try:
        response = client.post("/triage", json={"ticket_id":"T-004","subject":"Intermittent API errors","description":"Every customer request returns 503 and production is down."})
    finally:
        app.dependency_overrides = {}
    assert response.status_code == 200
    body = response.json()
    assert body["model_decision"]["priority"] == "high"
    assert body["final_decision"]["priority"] == "critical"
    assert body["final_decision"]["team"] == "incident_response"
    assert body["guardrail_applied"] is True

How Does the Override Test Work?

• The dependency override makes POST /triage use StubTriager for this request.
• The finally block clears the override even if an assertion fails.
• The assertions compare the model's high-priority route with the final critical incident-response route.

• Save tests/test_api.py.

Before you run the first test, which decision do you expect to carry the critical incident route?

• Run the current test suite from the project terminal with this command:

pytest -q

What Does This Check?

The -q option keeps pytest output concise. The test still executes the complete API route through TestClient.

You should see the outage override test pass. This proves the final response carries critical priority and the incident-response team.

Outage Override Test Failing?

Check that the ticket description contains production is down. Confirm that the test resets app.dependency_overrides inside the finally block.

Help me debug the outage override test.

The second test double raises the same domain error that ClaudeTriager uses for provider failures. This isolates the API's failure response from live provider availability.

• Add the failing provider stub below StubTriager by pasting this code:

class FailingTriager:
    def triage(self, ticket: Ticket) -> AIDecision:
        raise TriageProviderError("simulated provider failure")

What Does the Failing Stub Do?

FailingTriager raises TriageProviderError every time the API requests a decision. The fixed failure gives the HTTP error-handling path a deterministic input.

• Save tests/test_api.py.
• Confirm that FailingTriager appears below StubTriager.

Failing Stub Showing a Syntax Error?

Keep raise TriageProviderError("simulated provider failure") indented inside triage(). Keep the method indented inside FailingTriager.

Help me fix the FailingTriager class.

• Add the provider failure test below test_critical_outage_is_overridden() by pasting this code:

def test_provider_failure_returns_503() -> None:
    app.dependency_overrides[get_triager] = FailingTriager
    try:
        response = client.post("/triage", json={"ticket_id":"T-006","subject":"Cannot open dashboard","description":"The dashboard will not load for my account."})
    finally:
        app.dependency_overrides = {}
    assert response.status_code == 503
    assert response.json() == {"detail": "AI triage provider unavailable"}

How Does the Failure Test Work?

• The dependency override forces the route through FailingTriager.
• The API catches TriageProviderError and returns the stable client-facing response.
• The assertions lock down both the HTTP status and response detail.

• Save tests/test_api.py.

Before you rerun the tests, do you expect the simulated provider failure to escape as an exception or return a controlled API response?

• Run both API tests with this command:

pytest -q

What Does This Run Prove?

Pytest exercises the override path and the provider-failure path. Both tests use local doubles instead of the Anthropic API.

You should see both API tests pass. The simulated failure returns HTTP 503 with AI triage provider unavailable.

Provider Failure Test Failing?

Confirm that FailingTriager raises TriageProviderError. Check that the expected detail matches the response in app/main.py exactly.

Help me debug the provider failure test.

✔️ Awesome, I've got everything!

Great. Save tests/test_api.py before moving to the evaluation tests.

ⓧ I'd like to double check the full code

Compare your complete tests/test_api.py file with this reference.

from fastapi.testclient import TestClient

from app.main import app, get_triager
from app.models import AIDecision, Category, Priority, Team, Ticket
from app.provider import TriageProviderError

client = TestClient(app)


class StubTriager:
    def triage(self, ticket: Ticket) -> AIDecision:
        return AIDecision(category=Category.TECHNICAL, priority=Priority.HIGH, team=Team.TECHNICAL_SUPPORT, summary="Customers are seeing API errors.", confidence=0.88)


class FailingTriager:
    def triage(self, ticket: Ticket) -> AIDecision:
        raise TriageProviderError("simulated provider failure")


def test_critical_outage_is_overridden() -> None:
    app.dependency_overrides[get_triager] = StubTriager
    try:
        response = client.post("/triage", json={"ticket_id":"T-004","subject":"Intermittent API errors","description":"Every customer request returns 503 and production is down."})
    finally:
        app.dependency_overrides = {}
    assert response.status_code == 200
    body = response.json()
    assert body["model_decision"]["priority"] == "high"
    assert body["final_decision"]["priority"] == "critical"
    assert body["final_decision"]["team"] == "incident_response"
    assert body["guardrail_applied"] is True


def test_provider_failure_returns_503() -> None:
    app.dependency_overrides[get_triager] = FailingTriager
    try:
        response = client.post("/triage", json={"ticket_id":"T-006","subject":"Cannot open dashboard","description":"The dashboard will not load for my account."})
    finally:
        app.dependency_overrides = {}
    assert response.status_code == 503
    assert response.json() == {"detail": "AI triage provider unavailable"}

What Should Match?

Your file should contain two test doubles and two API tests. Each test must clear the dependency override after its request.

The API tests cover request behavior. The evaluation tests preserve the project's central lesson by proving that valid model output can still route a critical incident incorrectly.

• In VS Code, select the tests/ folder in the Explorer sidebar.
• Use the file creation control to create tests/test_evaluation.py.
• Add the schema-valid operational failure test by pasting this code into tests/test_evaluation.py:

from app.evaluation import FixtureTriager, LABELED_CASES, evaluate
from app.models import AIDecision, Priority, Team


def test_fixture_is_schema_valid_but_operationally_wrong() -> None:
    outage_case = next(case for case in LABELED_CASES if case.ticket.ticket_id == "T-004")
    decision = FixtureTriager().triage(outage_case.ticket)
    assert isinstance(decision, AIDecision)
    assert decision.priority == Priority.HIGH
    assert decision.team == Team.TECHNICAL_SUPPORT
    assert decision.priority != outage_case.expected_priority
    assert decision.team != outage_case.expected_team

What Does the Baseline Test Prove?

• The fixture decision passes AIDecision validation.
• Its high priority disagrees with the labeled critical priority.
• Its technical-support team disagrees with the labeled incident-response team.

• Save tests/test_evaluation.py.
• Run the expanded test suite with this command:

pytest -q

What Does This Check Add?

The new test locks the designed operational failure into the suite. Schema validity now has a separate assertion from routing correctness.

You should see the API tests and the schema-valid failure test pass.

Fixture Test Failing?

Confirm that the test selects ticket T-004. Check that its fixture decision remains high priority and routes to technical support.

Help me debug the evaluation fixture test.

• Add the metric improvement test below test_fixture_is_schema_valid_but_operationally_wrong() by pasting this code:

def test_guardrails_improve_evaluation_metrics() -> None:
    triager = FixtureTriager()
    assert evaluate(triager, use_guardrails=False) == {"routing_accuracy": 0.8, "critical_incident_recall": 0.0}
    assert evaluate(triager, use_guardrails=True) == {"routing_accuracy": 1.0, "critical_incident_recall": 1.0}

What Does the Metrics Test Prove?

The same fixture produces the baseline and guarded metrics. The only changed input is whether evaluate() applies deterministic guardrails.

• Save tests/test_evaluation.py.

Before you run the complete suite, do you expect any test to contact Claude or depend on a variable model response?

• Run every test with this command:

pytest -q

What Does the Complete Suite Cover?

• The fixture test preserves the schema-valid routing failure.
• The metrics test proves the deterministic guardrails improve both measurements.
• The API tests cover an override response and a controlled provider failure.

You should see every test pass. The suite completes without making an Anthropic API call.

Test Suite Not Passing?

Read the first failing assertion before changing code. Confirm that your test files match the fixed ticket IDs and expected metric dictionaries.

Help me debug the complete pytest suite.

✔️ Awesome, I've got everything!

Your deterministic tests are complete. Keep the passing terminal output visible for the checkpoint.

ⓧ I'd like to double check the full code

Compare your complete tests/test_evaluation.py file with this reference.

from app.evaluation import FixtureTriager, LABELED_CASES, evaluate
from app.models import AIDecision, Priority, Team


def test_fixture_is_schema_valid_but_operationally_wrong() -> None:
    outage_case = next(case for case in LABELED_CASES if case.ticket.ticket_id == "T-004")
    decision = FixtureTriager().triage(outage_case.ticket)
    assert isinstance(decision, AIDecision)
    assert decision.priority == Priority.HIGH
    assert decision.team == Team.TECHNICAL_SUPPORT
    assert decision.priority != outage_case.expected_priority
    assert decision.team != outage_case.expected_team


def test_guardrails_improve_evaluation_metrics() -> None:
    triager = FixtureTriager()
    assert evaluate(triager, use_guardrails=False) == {"routing_accuracy": 0.8, "critical_incident_recall": 0.0}
    assert evaluate(triager, use_guardrails=True) == {"routing_accuracy": 1.0, "critical_incident_recall": 1.0}

What Should Match?

Your file should contain the reproducible failure test and the before-and-after metrics test. Both tests use FixtureTriager.

Define the container boundary

A container build context can include files that the running API never needs. The ignore file keeps your secret, virtual environment, caches, and repository metadata outside the image.

• In VS Code, select the top-level project folder in the Explorer sidebar.
• Use the file creation control to create .dockerignore inside that folder.
• Define the excluded files by pasting this content into .dockerignore:

.env
.venv
__pycache__
.pytest_cache
.git
.gitignore
.DS_Store

What Does This File Exclude?

• The .env entry keeps the Anthropic API key out of the image.
• The .venv entry keeps the local virtual environment out of the build context.
• The remaining entries exclude Python caches, pytest state, Git metadata, and the local OS file.

• Save .dockerignore.
• Confirm that .dockerignore appears beside requirements.txt in the Explorer sidebar.

Ignore File Missing?

Check that the filename starts with a single dot. Remove any automatic .txt suffix if your editor added one.

Help me verify my Docker ignore file.

✔️ Awesome, I've got everything!

Your container boundary now excludes local secrets and development state.

ⓧ I'd like to double check the full code

Compare your complete .dockerignore file with this reference.

.env
.venv
__pycache__
.pytest_cache
.git
.gitignore
.DS_Store

What Should Match?

Each excluded path should appear on its own line. The file should include .env and .venv.

The Dockerfile turns the source into a runnable image. It installs the pinned dependencies before copying app/ into the image.

• In VS Code, select the top-level project folder in the Explorer sidebar.
• Use the file creation control to create Dockerfile inside that folder.
• Define the container build by pasting this content into Dockerfile:

FROM python:3.14.7-slim-trixie

WORKDIR /app

COPY requirements.txt pyproject.toml ./
RUN pip install --no-cache-dir -r requirements.txt

COPY app ./app

EXPOSE 8000

CMD ["fastapi", "run"]

How Does the Image Run the API?

• The python:3.14.7-slim-trixie base image supplies Python inside the container.
• The dependency files are copied before app/ so Docker can reuse the installation layer when only application code changes.
• The image exposes port 8000 and starts the production FastAPI command.

• Save Dockerfile.
• Confirm that Dockerfile appears beside .dockerignore in the Explorer sidebar.

Dockerfile Showing as Plain Text?

Confirm that the filename is exactly Dockerfile with no extension. Keep it in the same folder as requirements.txt.

Help me check my Dockerfile location and name.

✔️ Awesome, I've got everything!

Your image recipe is ready. Save the file before starting the build.

ⓧ I'd like to double check the full code

Compare your complete Dockerfile with this reference.

FROM python:3.14.7-slim-trixie

WORKDIR /app

COPY requirements.txt pyproject.toml ./
RUN pip install --no-cache-dir -r requirements.txt

COPY app ./app

EXPOSE 8000

CMD ["fastapi", "run"]

What Should Match?

The image should install from requirements.txt and copy only app/ after the dependency layer.

Build and run the container

The saved Dockerfile is the build recipe. Building it creates the reusable image tagged ai-ticket-triage:latest.

The first build can take a few minutes while Docker downloads the base image and installs the dependencies. A quiet pause during those downloads is expected.

• Build the tagged image from the project folder by running this command:

docker build -t ai-ticket-triage:latest .

What Does This Build Command Do?

• The -t option assigns the image name and tag.
• The final dot sends the current project folder as the build context.
• The .dockerignore rules remove the secret and local development files from that context.

You should see Docker finish building ai-ticket-triage:latest successfully.

Image Build Failing?

Confirm that Docker Desktop is running. Check that your terminal is in the folder containing Dockerfile and requirements.txt.

Help me debug the Docker image build.

The container reads your existing .env file only at runtime. Your API key stays outside the image layers.

Before you start the container, what do you expect the health route to return once the packaged API is ready?

• Start the container with its API key supplied at runtime by running this command:

docker run --rm --name ai-ticket-triage -p 127.0.0.1:8000:8000 --env-file .env ai-ticket-triage:latest

How Does the Container Start?

• The --name option gives the running container a predictable name.
• The -p option publishes container port 8000 on your computer's loopback address.
• The --env-file option supplies the environment values from .env at runtime.
• The --rm option removes the stopped container while preserving the image.

The terminal stays attached to the running FastAPI server.

• Leave the container terminal running.
• Enter http://127.0.0.1:8000/health in your browser's address bar.

You should see {"status":"ok"} from the containerized API.

• Return to the Swagger UI from Step 2.

You should see the /health and /triage routes listed by the containerized service.

That is the portable runtime working. Your guarded API now behaves consistently in tests and inside its Docker image.

Container Health Route Not Loading?

Check the running terminal for a startup failure. Confirm that port 8000 is free and that .env exists beside Dockerfile.

Help me debug the containerized health route.

Your service now has deterministic coverage for its risky paths and a reproducible container runtime. The live AI call remains available for demos while repeated verification stays free of model requests.

Secret mission

Add a Human-Review Queue

Your guarded API can catch critical incidents. This extension also sends uncertain decisions to a process-local human-review queue so a person can inspect them.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the project available, pause its running processes, or delete its local resources entirely. Only live requests to the Claude API create new usage charges.

Cost warning

Live POST /triage requests consume paid Claude API tokens. Claude Sonnet 5 costs $2 per million input tokens. Output tokens cost $10 per million.

The deterministic fixture evaluation makes no API calls. Your automated tests also make no API calls.

Resources you used:

• Project source folders app/ and tests/.
• Project configuration files including requirements.txt, pyproject.toml, .env.example, .gitignore, .dockerignore, and Dockerfile.
• Virtual environment stored in .venv/.
• Secret file stored in .env with your Anthropic API key.
• Docker image tagged ai-ticket-triage:latest.
• FastAPI development server started with fastapi dev if it is still running.
• Docker container named ai-ticket-triage if it is still running.
• Process-local review queue held in memory as review_queue while the API process runs.

Keep everything running

Choose this option if you plan to keep demonstrating the guarded triage API. Your local files and Docker image can remain idle without creating ongoing charges.

• Retain app/, tests/, and the project configuration files for future demonstrations.
• Keep .venv/ so the pinned Python dependencies remain available.
• Keep .env on your machine.
• Keep the ai-ticket-triage:latest image for future container runs.
• Use the fixture provider for repeated evaluations without paid model calls.

The review_queue exists only inside the current API process. Restarting that process clears its queued results.

Pause - I'll come back to this later

Pause the project by stopping its running processes. Your source files, virtual environment, secret file, and Docker image stay ready for your return.

• Stop the development server by pressing Control-C in the terminal running fastapi dev.
• Stop the container by pressing Control-C in the terminal running ai-ticket-triage.
• Leave the project source files in place.
• Leave .venv/ in place.
• Leave .env in place.
• Leave the ai-ticket-triage:latest image in Docker.

The original container command uses --rm, so Docker removes the stopped container automatically. Stopping the API also clears its process-local review queue.

Delete - I don't want to use this again

Deleting these resources permanently removes your code, local environment, and local API key copy. Save any work you want to keep before continuing.

• Stop the development server by pressing Control-C in its terminal if it is running.
• Stop the container by pressing Control-C in its terminal if it is running.
• Remove the tagged Docker image by running this command:

docker image rm ai-ticket-triage:latest

What does this command remove?

This removes or untags the local Docker image named ai-ticket-triage:latest.

Your source files remain available until you remove them separately.

You should see Docker confirm that the tag or image was removed.

Docker cannot remove the image?

• Stop any container that still uses ai-ticket-triage:latest.
• Retry the image removal command after the container stops.

Help me remove the local Docker image..

• Delete the local project resources from the folder containing app/ and tests/ by running this command:

rm -rf app tests .venv .env .pytest_cache __pycache__ requirements.txt pyproject.toml .env.example .gitignore .dockerignore Dockerfile

What does this command remove?

This removes the application code, tests, virtual environment, secret file, generated caches, and project configuration files.

The unnamed outer project folder remains because it is not part of the command.

A successful removal returns you to the shell prompt without an error. The folder no longer contains app/, tests/, .venv/, or .env.

Still see project files?

• Confirm your terminal is inside the folder that contains app/ and tests/.
• Check whether your account has permission to remove the remaining files.

Help me clean up the remaining local files..

Nice Work!

Nice Work!

You did it! Your FastAPI ticket triage API now turns model output into guarded operational decisions.

What you learned:

• Built a live Claude-powered endpoint that turns unstructured tickets into a typed triage contract. Used structured outputs with Pydantic validation to enforce fields plus enum values.
• Exposed a reproducible operational safety failure with a fixed five-ticket fixture. Watched the baseline score 80% routing accuracy with 0% critical-incident recall. Confirmed deterministic guardrails raised both metrics to 100%.
• Used dependency injection to test guarded routing without paid model calls. Confirmed provider failures return a controlled HTTP 503 response. Packaged the guarded service as a Docker image for repeatable local demos.
• Secret Mission: Added a human-review queue that captures low-confidence results plus guardrail overrides for escalation. Documented its process-local limit so the demo points toward durable shared storage.

Ready to quiz yourself?