Deploy a B2B Support Router

Introduction

30 Second Summary

A support queue can mix urgent outages with billing questions and risky account changes. Treating every request the same slows customers down and creates avoidable risk.

In this project, you will act as a Forward Deployed Engineer to build a support control plane for a fictional B2B software company. The API uses deterministic controls and replayed Jev decisions to route each ticket through a confidence gate with an auditable result.

What You'll Build

You run one demo that sends a known outage to service status, an ambiguous invoice to billing, and a security-sensitive request to human review with a trace for every route.

By the end of this project, you'll have:

• A live routing demo where validated tickets move through mock CRM, service-status, knowledge, specialist-model, or human-review handlers. You can see the action performed by every route.
• A bounded decision path where known incidents stay in ordinary code, unresolved requests use typed Jev decisions, and security-sensitive changes stop before automation.
• Audit and evaluation evidence that compares rules-only routing with the hybrid control plane across seven labeled tickets. The report shows route accuracy, automation coverage, unsafe automation, and downstream LLM demand.
• Secret Mission: Write an evidence-based shadow-mode rollout recommendation with acceptance gates, three rollback triggers, named owners, and a go or no-go decision.

Are there any prerequisites?

You should be comfortable reading Python and working with API-based AI applications. The required path uses replay data, so it needs no paid OpenRouter usage.

Before We Start

This is your moment to commit to a support-routing control plane that handles mixed B2B tickets safely. Before any hands-on work, you'll define the bottleneck plus the evidence that will prove it works.

Set Up the Deployment Workspace

The support router eventually makes decisions that affect customer access, billing questions, and security requests. A mismatched runtime or dependency can undermine those decisions before the routing logic even runs.

You'll verify the Python runtime first. Then you'll prepare a Visual Studio Code workspace with pinned FastAPI, Pydantic, and Uvicorn dependencies.

In this step, get ready to:

• Confirm that Python 3.10 or newer is available.
• Prepare the project structure with pinned dependencies.
• Launch the starter API on your local machine.

Verify the Python runtime

The project dependencies require Python 3.10 or newer. Checking the interpreter now prevents confusing installation failures later.

• Press Command-Space on macOS to open Spotlight.
• Type Visual Studio Code and press Return to open it.
• Select View from the menu bar.
• Select Terminal to show the integrated terminal.
• Print the installed Python version by running this command:

python --version

What does this check show?

The version command prints the interpreter used when you enter python. The major and minor numbers determine whether the pinned dependencies can run.

✔️ I see version 3.10 or higher

Your Python runtime meets the project requirement. You can use python for the remaining commands.

ⓧ I see an older version

The installed interpreter is below the minimum required by FastAPI and Uvicorn. Install the current stable Python release before continuing.

• Open the Python.org macOS downloads page.
• Select the macOS installer for Python 3.14.7.
• Complete the installer using its default options.
• Return to Visual Studio Code.
• Select Terminal from the menu bar.
• Select New Terminal.
• Check the active Python version again by running:

python --version

Why open a new terminal?

A new terminal reloads the command paths available to your shell. The version check confirms that the updated interpreter is active.

Continue when the output reports Python 3.10 or newer.

ⓧ Command not found

Your terminal cannot currently find a Python interpreter through the python command. Install the current stable macOS package before continuing.

• Open the Python.org macOS downloads page.
• Select the macOS installer for Python 3.14.7.
• Complete the installer using its default options.
• Return to Visual Studio Code.
• Select Terminal from the menu bar.
• Select New Terminal.
• Check whether the terminal can find Python by running:

python --version

What confirms the installation?

A version number confirms that the terminal can now invoke Python. Continue when that version is 3.10 or newer.

Create the project structure and install dependencies

A workspace keeps the API code, test fixtures, evaluation script, and rollout notes inside one named directory. The empty placeholders also show where each later part of the control plane belongs.

• Select File from the Visual Studio Code menu bar.
• Select Open Folder....
• Select New Folder on your Desktop.
• Enter supportops-control-plane as the folder name.
• Select Open to use the folder as your workspace.

You should see supportops-control-plane at the top of the Explorer sidebar.

• Click New Folder in the Explorer sidebar.
• Enter data as the folder name.
• Click New File in the Explorer sidebar.
• Enter requirements.txt as the file name.
• Repeat New File for main.py, router.py, handlers.py, decision_provider.py, demo.py, evaluate.py, and rollout.md.

Your Explorer now shows the complete project skeleton. The next steps fill these placeholders with routing, decision, evaluation, and rollout logic.

• Select requirements.txt in the Explorer sidebar.
• Add the three dependency pins by pasting this content:

fastapi==0.141.1
pydantic==2.13.5
uvicorn==0.53.0

Why pin these dependencies?

• The fastapi==0.141.1 pin supplies the API framework and request-routing features.
• The pydantic==2.13.5 pin supplies the validated data models used for incoming tickets.
• The uvicorn==0.53.0 pin supplies the local server that runs the application.

• Save requirements.txt.
• Install the pinned project dependencies by running:

python -m pip install -r requirements.txt

What does this command do?

Python runs pip through the verified interpreter. Pip reads requirements.txt and installs each exact version listed there.

The exact pins keep this deployment workspace consistent across repeated customer demos.

Once the command finishes, the terminal should report that the requirements were installed or already satisfied.

Dependency installation failed?

• Confirm that the terminal is rooted in the supportops-control-plane workspace shown in the Explorer.
• Check that requirements.txt contains exactly three lines with no extra punctuation.
• Use the terminal output when asking for help: Help me diagnose this dependency installation failure.

Launch the starter health endpoint

A health endpoint gives operators a small request that proves the API process is reachable. It becomes the first visible signal that the deployment shell works.

• Select main.py in the Explorer sidebar.
• Add the starter FastAPI application by pasting this code:

from fastapi import FastAPI

app = FastAPI(
    title="SupportOps Control Plane",
    version="1.0.0",
    description="Confidence-gated routing for a B2B SaaS support workflow.",
)


@app.get("/health")
def health():
    return {"status": "ok"}

What does this code do?

• The FastAPI import provides the application class.
• The app variable identifies the application that Uvicorn serves.
• The @app.get("/health") decorator connects a GET request to the health() function.
• The response provides a small healthy status that you can check without any ticket-routing logic.

• Save main.py.

Before you start the server, do you think Uvicorn can locate the application from the file name and the app variable?

• Start the local API by running this command:

uvicorn main:app

What does this command do?

The main portion points Uvicorn to main.py. The app portion selects the FastAPI application stored in that variable.

You should see the terminal report that Uvicorn is running on http://127.0.0.1:8000.

• Paste http://127.0.0.1:8000/health into your browser address bar.
• Press Return to send the health request.

You should see {"status":"ok"} in the browser. Your local API now accepts requests through its first endpoint.

Can't reach the health endpoint?

• Keep the terminal running while you load the health URL. Closing the process stops the local server.
• Confirm that the command targets main:app. These names must match main.py and the app variable.
• Check that the route in main.py is spelled /health.
• Share the terminal error for focused help: Help me fix my local FastAPI startup error.

✔️ Awesome, I've got everything!

Your dependencies are pinned. Your health endpoint is responding through the running local server.

ⓧ I'd like to double check the full code

Compare the two populated files with these complete versions. Keep the other project files as empty placeholders for the later steps.

fastapi==0.141.1
pydantic==2.13.5
uvicorn==0.53.0

from fastapi import FastAPI

app = FastAPI(
    title="SupportOps Control Plane",
    version="1.0.0",
    description="Confidence-gated routing for a B2B SaaS support workflow.",
)


@app.get("/health")
def health():
    return {"status": "ok"}

Your deployment workspace now has a validated runtime, reproducible dependencies, and a reachable API shell. Next up, you'll turn that shell into a rules-first support gateway.

Build the Rules-First Gateway

Your FastAPI shell can prove that the service is alive. A support gateway also needs to understand each ticket before it can choose a safe destination.

Known incidents have dependable routes. Security-sensitive admin changes need human approval. Ambiguous requests expose the limit of fixed rules because the gateway has no reliable rule to interpret their meaning.

In this step, get ready to:

• Define the validated support ticket contract.
• Add mock customer records plus visible support handlers.
• Run the rules-only gateway against a known incident plus an ambiguous billing ticket.

Model validated support tickets

A Pydantic model defines the fields that every incoming ticket must contain. It also gives the gateway explicit flags for known incidents plus security-sensitive admin changes.

• Return to main.py in VS Code.
• Replace the import area above app = FastAPI( with this code:

from fastapi import FastAPI
from pydantic import BaseModel


class Ticket(BaseModel):
    ticket_id: str
    account_id: str
    customer_tier: str
    message: str
    known_incident: bool = False
    admin_change_requested: bool = False

What Does the Ticket Model Control?

• The four required text fields identify the ticket plus its customer context.
• The known_incident flag lets trusted incident data bypass semantic classification.
• The admin_change_requested flag marks a request that needs a protected route.
• Both flags default to False when the upstream system does not set them.

• Save main.py.
• Stop the running server by pressing Control+C in its terminal.
• Restart the service by running this command:

uvicorn main:app

What Does This Check Prove?

Uvicorn imports the updated main.py before it starts the service. A successful startup confirms that the ticket model has valid Python syntax.

You should see Uvicorn report that the service is running on http://127.0.0.1:8000. Your API now has a typed ticket contract.

Does the Server Stop During Startup?

• Check that every field inside Ticket is indented by four spaces.
• Confirm that BaseModel uses the same capitalization in the import plus the class definition.
• Ask for help with the exact traceback: Help me debug the Ticket model in main.py from this Uvicorn traceback.

Add mock customer systems

The router needs realistic boundaries around it. Local JSON fixtures represent the customer records plus support knowledge that enterprise integrations would normally provide.

The labeled tickets also give you repeatable customer scenarios. Each fixture carries the route that a safe deployment should eventually select.

• Switch to data/accounts.json from the project structure you created earlier.
• Replace its placeholder content with these mock CRM records:

{
  "acc-100": {
    "company": "Northstar Analytics",
    "plan": "enterprise",
    "status": "active",
    "sso_enabled": true
  },
  "acc-200": {
    "company": "Harbor Labs",
    "plan": "pro",
    "status": "locked",
    "sso_enabled": false
  },
  "acc-300": {
    "company": "Cedar Systems",
    "plan": "enterprise",
    "status": "active",
    "sso_enabled": true
  }
}

What Do the Account Records Represent?

These records act as a mock CRM. An account handler can retrieve the company plan plus account state without connecting to a real customer system.

• Save data/accounts.json.
• Confirm that VS Code shows three top-level account identifiers in the file.

• Switch to data/knowledge.json.
• Replace its placeholder content with these support articles:

[
  {
    "article_id": "kb-api-429",
    "title": "API rate-limit responses",
    "body": "A 429 response indicates the client should reduce request rate and retry later."
  },
  {
    "article_id": "kb-sso-login",
    "title": "Troubleshoot SSO login",
    "body": "Confirm domain mapping, identity-provider status, and workspace membership."
  },
  {
    "article_id": "kb-webhook",
    "title": "Webhook delivery troubleshooting",
    "body": "Inspect delivery attempts, endpoint status, and signature verification failures."
  }
]

How Will Knowledge Retrieval Work?

The knowledge handler compares words from the ticket with words from each article. It returns the article with the largest overlap so the route produces a visible retrieval result.

• Save data/knowledge.json.
• Confirm that VS Code shows three article objects inside the array.

• Switch to data/tickets.json.
• Replace its placeholder content with the seven labeled customer tickets:

[
  {
    "ticket_id": "t-001",
    "account_id": "acc-100",
    "customer_tier": "enterprise",
    "message": "Every dashboard request is timing out. Is there an outage?",
    "known_incident": true,
    "admin_change_requested": false,
    "expected_route": "status_handler"
  },
  {
    "ticket_id": "t-002",
    "account_id": "acc-200",
    "customer_tier": "pro",
    "message": "I reset my password but still cannot access the workspace.",
    "known_incident": false,
    "admin_change_requested": false,
    "expected_route": "account_lookup"
  },
  {
    "ticket_id": "t-003",
    "account_id": "acc-100",
    "customer_tier": "enterprise",
    "message": "Our invoice lists two Pro seats that we do not recognize. Can you explain the charge?",
    "known_incident": false,
    "admin_change_requested": false,
    "expected_route": "billing_llm"
  },
  {
    "ticket_id": "t-004",
    "account_id": "acc-300",
    "customer_tier": "enterprise",
    "message": "Can the product export scheduled reports to our data warehouse?",
    "known_incident": false,
    "admin_change_requested": false,
    "expected_route": "product_llm"
  },
  {
    "ticket_id": "t-005",
    "account_id": "acc-300",
    "customer_tier": "enterprise",
    "message": "Transfer workspace ownership and disable MFA for the current owner.",
    "known_incident": false,
    "admin_change_requested": true,
    "expected_route": "human_security"
  },
  {
    "ticket_id": "t-006",
    "account_id": "acc-100",
    "customer_tier": "enterprise",
    "message": "Things are not working and our renewal is tomorrow. Someone needs to fix this now.",
    "known_incident": false,
    "admin_change_requested": false,
    "expected_route": "human_review"
  },
  {
    "ticket_id": "t-007",
    "account_id": "acc-300",
    "customer_tier": "enterprise",
    "message": "Our API requests return 429 responses during the nightly sync. What should we check?",
    "known_incident": false,
    "admin_change_requested": false,
    "expected_route": "knowledge_rag"
  }
]

Why Label Every Ticket?

The expected_route value records the intended outcome for each scenario. This label becomes evaluation evidence later without entering the API request itself.

• Save data/tickets.json.
• Confirm that the first ticket identifier is t-001.
• Confirm that the final ticket identifier is t-007.

Does VS Code Flag Invalid JSON?

• Check for a missing comma between adjacent objects.
• Confirm that every property name uses double quotes.
• Ask for help without sharing customer data from another project: Help me find the JSON syntax error in these local mock fixtures.

Routes become useful when they perform an observable action. The handler layer turns each route into a mock status update, account lookup, knowledge retrieval, specialist queue, or human handoff.

• Switch to handlers.py from the existing project structure.
• Replace its placeholder content with the complete handler module below:

import json
import re
from pathlib import Path

BASE_DIR = Path(__file__).resolve().parent
DATA_DIR = BASE_DIR / "data"


def _read_json(name):
    with (DATA_DIR / name).open(encoding="utf-8") as file:
        return json.load(file)


def _words(text):
    return set(re.findall(r"[a-z0-9]+", text.lower()))


def _knowledge_search(message):
    query_words = _words(message)
    articles = _read_json("knowledge.json")
    ranked = sorted(
        articles,
        key=lambda article: len(query_words & _words(article["title"] + " " + article["body"])),
        reverse=True,
    )
    best = ranked[0]
    return {
        "status": "retrieved",
        "article_id": best["article_id"],
        "title": best["title"],
    }


def execute_handler(route, ticket):
    if route == "status_handler":
        return {
            "status": "resolved",
            "message": "Linked the ticket to the active service incident.",
            "downstream_llm_required": False,
        }

    if route == "account_lookup":
        account = _read_json("accounts.json").get(ticket["account_id"])
        return {
            "status": "resolved" if account else "not_found",
            "account": account,
            "downstream_llm_required": False,
        }

    if route == "knowledge_rag":
        return {
            **_knowledge_search(ticket["message"]),
            "downstream_llm_required": True,
        }

    if route in {"billing_llm", "product_llm"}:
        return {
            "status": "queued",
            "queue": route,
            "execution": "simulated specialist model",
            "downstream_llm_required": True,
        }

    if route in {"human_review", "human_security"}:
        return {
            "status": "escalated",
            "queue": "security" if route == "human_security" else "support-operations",
            "handoff_context": {
                "ticket_id": ticket["ticket_id"],
                "account_id": ticket["account_id"],
                "customer_tier": ticket["customer_tier"],
                "message": ticket["message"],
            },
            "downstream_llm_required": False,
        }

    raise ValueError(f"Unknown route: {route}")

What Does the Handler Layer Do?

• The status path links a ticket to an active incident without requiring another model.
• The account path reads customer state from accounts.json.
• The knowledge path retrieves the closest local article.
• The specialist plus human paths return visible queue details with downstream model demand recorded separately.

• Save handlers.py.
• Confirm that VS Code recognizes execute_handler() as the final function in the file.

Does the Handler File Show a Syntax Error?

• Check the closing braces inside the handoff_context dictionary.
• Confirm that execute_handler() starts at the left edge of the file.
• Ask for targeted help: Help me debug the execute_handler function without changing its route names.

Route tickets with deterministic rules

The first routing layer owns facts plus permission boundaries. A known incident takes the status path immediately. An admin change takes the protected human path immediately.

Every other ticket falls through to manual review. Running that version lets you see the operational cost of rules that cannot interpret ambiguous language.

• Switch to router.py.
• Replace its placeholder content with the rules-only function below:

def route_rules_only(ticket):
    if ticket["known_incident"]:
        return {
            "matched": True,
            "route": "status_handler",
            "reason": "known_incident",
        }
    if ticket["admin_change_requested"]:
        return {
            "matched": True,
            "route": "human_security",
            "reason": "admin_change_requires_human",
        }
    return {
        "matched": False,
        "route": "human_review",
        "reason": "rules_exhausted",
    }

How Do the Rules Protect the Boundary?

• The incident check runs first because upstream incident state is already known.
• The admin-change check forces a security handoff before any automated action.
• The fallback records rules_exhausted so operators can distinguish ambiguity from a protected escalation.

• Save router.py.
• Stop Uvicorn by pressing Control+C in its terminal.
• Check that the new router module imports successfully by restarting the service:

uvicorn main:app

What Does This Restart Check?

The service startup confirms that the new Python module does not introduce an import-time failure. Uvicorn remains available for the API wiring that comes next.

You should see the local server start again at http://127.0.0.1:8000.

• Return to main.py.
• Add these imports below the existing Pydantic import:

from handlers import execute_handler
from router import route_rules_only

Why Import Both Layers?

The router chooses the destination. The handler turns that destination into a visible mock outcome.

• Add the temporary rules-only route below the existing health endpoint:

@app.post("/tickets/route")
def route(ticket: Ticket):
    payload = ticket.model_dump()
    result = route_rules_only(payload)
    handler_result = execute_handler(result["route"], payload)
    return {
        **result,
        "decision_mode": "deterministic",
        "handler_result": handler_result,
    }

What Happens When a Ticket Arrives?

• FastAPI validates the request against Ticket.
• The model_dump() call converts the validated model into a dictionary.
• The rules choose a route before execute_handler() performs the corresponding mock action.
• The response preserves the route reason plus the handler outcome for auditability.

• Save main.py.
• Stop Uvicorn by pressing Control+C in its terminal.
• Load the new POST /tickets/route endpoint by restarting the service:

uvicorn main:app

What Should the Server Report?

Uvicorn should report that the application startup is complete. Keep this terminal running because the demo sends requests to the local API.

Does the Endpoint Import Fail?

• Confirm that handlers.py plus router.py are in the same directory as main.py.
• Check that the imported names match execute_handler plus route_rules_only exactly.
• Ask for help with the traceback: Help me fix the local module import error in my FastAPI support router.

The demo sends one known outage plus one ambiguous billing request through the same endpoint. This side-by-side test reveals where fixed rules succeed plus where they stop.

• Switch to demo.py from the existing project structure.
• Replace its placeholder content with this two-ticket client:

import json
from pathlib import Path
from urllib.error import URLError
from urllib.request import Request, urlopen

BASE_DIR = Path(__file__).resolve().parent
API_URL = "http://127.0.0.1:8000/tickets/route"

with (BASE_DIR / "data" / "tickets.json").open(encoding="utf-8") as file:
    tickets = {ticket["ticket_id"]: ticket for ticket in json.load(file)}

for ticket_id in ["t-001", "t-003"]:
    ticket = {key: value for key, value in tickets[ticket_id].items() if key != "expected_route"}
    request = Request(
        API_URL,
        data=json.dumps(ticket).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urlopen(request, timeout=5) as response:
            result = json.load(response)
    except URLError as exc:
        raise SystemExit("Start the API with `uvicorn main:app`, then run this demo again.") from exc

    print(
        f"{ticket_id}: route={result['route']} "
        f"mode={result['decision_mode']} reason={result['reason']}"
    )

How Does the Demo Exercise the Gateway?

• The client loads the labeled fixtures from data/tickets.json.
• It removes expected_route before sending each ticket because that label belongs to evaluation.
• It posts t-001 plus t-003 to the local gateway.
• It prints the selected route plus the reason that produced it.

• Save demo.py.
• Create a second terminal in VS Code so the Uvicorn terminal can keep serving requests.

Before you run the demo, do you think the billing ticket has enough explicit flags for a deterministic rule to choose its specialist route?

• Run the two-ticket demo from the second terminal:

python demo.py

What Should the Demo Reveal?

You should see t-001 route to status_handler with the reason known_incident.

You should see t-003 route to human_review with the reason rules_exhausted. The ambiguous billing request has no deterministic flag that identifies its meaning.

That shortfall is intentional. Your gateway protects known facts plus sensitive actions, while semantic ambiguity still consumes human-review capacity.

Does the Demo Fail to Reach the API?

• Confirm that the first terminal still shows Uvicorn running at http://127.0.0.1:8000.
• Check that API_URL ends with /tickets/route.
• Ask for help using the terminal output: Help me diagnose why demo.py cannot reach my local FastAPI endpoint.

✔️ I see both routes

Your rules-first gateway is working. The known incident takes the deterministic status path while the ambiguous billing ticket exposes the manual-review gap.

ⓧ My routes differ

• Confirm that t-001 has known_incident set to true.
• Confirm that t-003 has both routing flags set to false.
• Check that route_rules_only() returns human_review after both conditional checks.
• Ask for a comparison of your result: Help me trace why my rules-only gateway chose a different route.

You now have a validated support API with deterministic safety boundaries plus visible enterprise-style handlers. Next, you can address the ambiguity that forced t-003 into manual review.

Add the Jev Decision Layer

The rules-first gateway already protects known incidents and security-sensitive admin changes. However, an ambiguous billing ticket still falls straight into manual review because the rules cannot interpret its meaning.

This step adds Jev as a bounded decision layer for unresolved tickets. Deterministic code keeps control of routing thresholds and handler execution.

In this step, get ready to:

• Create a replay-first provider for typed team, urgency, and human-review decisions.
• Apply confidence gates before any unresolved ticket reaches a handler.
• Verify the deterministic outage path and replay-based billing path.

Create the replay-first decision provider

Replay mode supplies fixed typed decisions for the labeled tickets. This keeps the required path free while preserving the same contract used by the optional live adapter.

Why replay before live calls?

The replay fixtures are synthetic project data. They let you test routing policy and failure handling without presenting the results as live inference.

The optional adapter uses OpenRouter only when the configuration explicitly selects live mode. This separation keeps paid traffic outside the required path.

• In the VS Code Explorer sidebar, create config.json inside the existing data folder.
• Open the second tab below to copy the complete contents into data/config.json.
• Save data/config.json.

✔️ My configuration matches

Your configuration keeps the decision provider in replay mode. The three policy thresholds are ready for the router.

ⓧ I'd like to double check the full code

{
  "decision_mode": "replay",
  "min_team_confidence": 0.5,
  "human_probability_threshold": 0.7,
  "urgent_score_threshold": 1.5
}

What does this configuration control?

• The decision_mode value selects the local replay adapter.
• The min_team_confidence value sends uncertain team choices to human review.
• The human_probability_threshold value protects requests that need judgment or approval.
• The urgent_score_threshold value escalates critical requests.

• Validate the configuration file by running this command in the VS Code terminal:

python -m json.tool data/config.json

What should I see?

Python prints the formatted JSON and returns to the prompt. That output confirms the file contains valid JSON.

Seeing a JSON error?

• Check that every property except the last one ends with a comma.
• Confirm that replay remains inside double quotes.
• Ask for help with the exact line reported by Python: Help me fix the JSON syntax in data/config.json.

• In the VS Code Explorer sidebar, create jev_replay.json inside the data folder.
• Open the second tab below to copy the complete fixture set into data/jev_replay.json.
• Save data/jev_replay.json.

✔️ My replay fixtures match

Your replay cache contains typed decisions for every unresolved fixture used by the project.

ⓧ I'd like to double check the full code

{
  "t-002": {
    "model": "typesafe/jev-1.13",
    "provider": "replay-fixture",
    "answers": {
      "team": {
        "type": "choice",
        "choice": "account",
        "confidence": 0.9,
        "probabilities": {
          "account": 0.94,
          "billing": 0.01,
          "technical": 0.04,
          "product": 0.01
        }
      },
      "urgency": {
        "type": "score",
        "score": 0.4,
        "confidence": 0.8,
        "probabilities": {"0": 0.6, "1": 0.4, "2": 0.0},
        "legend": {
          "0": "Routine request that can wait for normal support hours.",
          "1": "Important request affecting work but with a workaround.",
          "2": "Critical request blocking production work or revenue."
        }
      },
      "needs_human": {"type": "noul", "noul": 0.1}
    },
    "usage": {"input_tokens": 0, "output_tokens": 0, "cost": 0}
  },
  "t-003": {
    "model": "typesafe/jev-1.13",
    "provider": "replay-fixture",
    "answers": {
      "team": {
        "type": "choice",
        "choice": "billing",
        "confidence": 0.88,
        "probabilities": {
          "account": 0.03,
          "billing": 0.91,
          "technical": 0.02,
          "product": 0.04
        }
      },
      "urgency": {
        "type": "score",
        "score": 0.8,
        "confidence": 0.72,
        "probabilities": {"0": 0.25, "1": 0.7, "2": 0.05},
        "legend": {
          "0": "Routine request that can wait for normal support hours.",
          "1": "Important request affecting work but with a workaround.",
          "2": "Critical request blocking production work or revenue."
        }
      },
      "needs_human": {"type": "noul", "noul": 0.2}
    },
    "usage": {"input_tokens": 0, "output_tokens": 0, "cost": 0}
  },
  "t-004": {
    "model": "typesafe/jev-1.13",
    "provider": "replay-fixture",
    "answers": {
      "team": {
        "type": "choice",
        "choice": "product",
        "confidence": 0.85,
        "probabilities": {
          "account": 0.01,
          "billing": 0.01,
          "technical": 0.08,
          "product": 0.9
        }
      },
      "urgency": {
        "type": "score",
        "score": 0.2,
        "confidence": 0.9,
        "probabilities": {"0": 0.85, "1": 0.15, "2": 0.0},
        "legend": {
          "0": "Routine request that can wait for normal support hours.",
          "1": "Important request affecting work but with a workaround.",
          "2": "Critical request blocking production work or revenue."
        }
      },
      "needs_human": {"type": "noul", "noul": 0.1}
    },
    "usage": {"input_tokens": 0, "output_tokens": 0, "cost": 0}
  },
  "t-006": {
    "model": "typesafe/jev-1.13",
    "provider": "replay-fixture",
    "answers": {
      "team": {
        "type": "choice",
        "choice": "technical",
        "confidence": 0.42,
        "probabilities": {
          "account": 0.14,
          "billing": 0.18,
          "technical": 0.43,
          "product": 0.25
        }
      },
      "urgency": {
        "type": "score",
        "score": 1.8,
        "confidence": 0.86,
        "probabilities": {"0": 0.0, "1": 0.2, "2": 0.8},
        "legend": {
          "0": "Routine request that can wait for normal support hours.",
          "1": "Important request affecting work but with a workaround.",
          "2": "Critical request blocking production work or revenue."
        }
      },
      "needs_human": {"type": "noul", "noul": 0.82}
    },
    "usage": {"input_tokens": 0, "output_tokens": 0, "cost": 0}
  },
  "t-007": {
    "model": "typesafe/jev-1.13",
    "provider": "replay-fixture",
    "answers": {
      "team": {
        "type": "choice",
        "choice": "technical",
        "confidence": 0.94,
        "probabilities": {
          "account": 0.01,
          "billing": 0.0,
          "technical": 0.97,
          "product": 0.02
        }
      },
      "urgency": {
        "type": "score",
        "score": 0.5,
        "confidence": 0.75,
        "probabilities": {"0": 0.5, "1": 0.5, "2": 0.0},
        "legend": {
          "0": "Routine request that can wait for normal support hours.",
          "1": "Important request affecting work but with a workaround.",
          "2": "Critical request blocking production work or revenue."
        }
      },
      "needs_human": {"type": "noul", "noul": 0.08}
    },
    "usage": {"input_tokens": 0, "output_tokens": 0, "cost": 0}
  }
}

How are the replay decisions structured?

• Each choice answer selects a support team.
• Each team answer includes a confidence value and option probabilities.
• Each score answer represents ticket urgency.
• Each noul answer represents the probability that human judgment is required.
• Each usage object records zero cost because these responses come from local fixtures.

• Validate the replay cache by running this command in the VS Code terminal:

python -m json.tool data/jev_replay.json

What should I see?

Python prints fixtures for t-002 through t-007 where a semantic decision is required. The missing fixture numbers belong to tickets resolved by deterministic controls.

Is the replay cache invalid?

• Check the line number reported by the JSON validator.
• Confirm that every fixture except t-007 ends with a comma.
• Ask for help locating the malformed fixture: Help me fix the JSON syntax in data/jev_replay.json.

• In the VS Code Explorer sidebar, create decision_provider.py beside main.py.
• Open the second tab below to copy the complete provider into decision_provider.py.
• Save decision_provider.py.

✔️ My provider matches

Your provider reads local decisions by default. It also contains the optional live adapter without activating it.

ⓧ I'd like to double check the full code

import json
import os
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

BASE_DIR = Path(__file__).resolve().parent
DATA_DIR = BASE_DIR / "data"
OPENROUTER_URL = "https://openrouter.ai/api/alpha/decisions"
MODEL_ID = "typesafe/jev-1.13"

QUESTIONS = {
    "team": {
        "type": "choice",
        "instructions": "Which support team should own this ticket?",
        "criteria": {
            "account": "Login, access, identity, workspace membership, or profile issues.",
            "billing": "Invoices, subscription charges, plan changes, or payment questions.",
            "technical": "Broken product behavior, API failures, outages, or integration errors.",
            "product": "How-to questions, feature questions, and workflow guidance.",
        },
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgent is this support ticket?",
        "criteria": [
            "Routine request that can wait for normal support hours.",
            "Important request affecting work but with a workaround.",
            "Critical request blocking production work or revenue.",
        ],
    },
    "needs_human": {
        "type": "noul",
        "instructions": "Does this ticket require human judgment or approval before action?",
        "criteria": {
            "true": "The request is ambiguous, sensitive, disputed, or requires approval.",
            "false": "The request can be handled safely by a defined automated path.",
        },
    },
}


class DecisionUnavailable(RuntimeError):
    pass


def _read_json(name):
    with (DATA_DIR / name).open(encoding="utf-8") as file:
        return json.load(file)


def current_mode():
    return _read_json("config.json")["decision_mode"]


def _replay_decision(ticket):
    fixtures = _read_json("jev_replay.json")
    try:
        result = fixtures[ticket["ticket_id"]]
    except KeyError as exc:
        raise DecisionUnavailable(
            f"No replay decision exists for {ticket['ticket_id']}."
        ) from exc
    return {**result, "decision_mode": "replay"}


def _live_decision(ticket):
    api_key = os.environ.get("OPENROUTER_API_KEY")
    if not api_key:
        raise DecisionUnavailable("OPENROUTER_API_KEY is required for live mode.")

    payload = {
        "model": MODEL_ID,
        "state": {
            "ticket": ticket["message"],
            "customer_tier": ticket["customer_tier"],
        },
        "questions": QUESTIONS,
    }
    request = Request(
        OPENROUTER_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        },
        method="POST",
    )

    try:
        with urlopen(request, timeout=20) as response:
            result = json.load(response)
    except HTTPError as exc:
        detail = exc.read().decode("utf-8", errors="replace")
        raise DecisionUnavailable(
            f"OpenRouter returned HTTP {exc.code}: {detail}"
        ) from exc
    except URLError as exc:
        raise DecisionUnavailable(f"OpenRouter could not be reached: {exc.reason}") from exc

    return {**result, "decision_mode": "live"}


def get_decision(ticket):
    mode = current_mode()
    if mode == "replay":
        return _replay_decision(ticket)
    if mode == "live":
        return _live_decision(ticket)
    raise DecisionUnavailable(f"Unsupported decision mode: {mode}")

What does this provider do?

• The QUESTIONS contract limits the decision to team ownership, urgency, and human-review probability.
• The _replay_decision() function loads a fixture by ticket identifier.
• The _live_decision() function builds the optional authenticated request.
• The get_decision() function selects the configured adapter.
• The DecisionUnavailable exception gives the router one safe failure type to catch.

The live adapter reads OPENROUTER_API_KEY only when live mode is selected. Keep decision_mode set to replay so this project does not send a paid request.

• Check the provider for Python syntax errors by running this command:

python -m py_compile decision_provider.py

What should I see?

The command returns to the prompt without printing an error. That silent success confirms Python can compile the provider.

Seeing a provider syntax error?

• Use the reported line number to find the first incomplete bracket or quote.
• Confirm that the QUESTIONS dictionary closes before the DecisionUnavailable class.
• Ask for help comparing the file with the source of truth: Help me debug decision_provider.py without changing its decision contract.

Gate unresolved tickets with typed decisions

The provider can now describe an unresolved ticket. The router still decides whether that description is safe enough to automate.

Why keep execution in the router?

Typed decisions describe intent and risk. Ordinary Python applies customer policy before selecting a handler.

This boundary keeps known facts and high-risk permissions under deterministic control. It also creates one place for safe fallback behavior.

• In the VS Code Explorer sidebar, return to router.py.
• Open the second tab below to replace router.py with the complete confidence-gated router.
• Save router.py.

✔️ My router matches

Your existing deterministic rules remain first in the routing order. Only unresolved tickets reach the decision provider.

ⓧ I'd like to double check the full code

import json
from datetime import datetime, timezone
from pathlib import Path
from time import perf_counter

from decision_provider import DecisionUnavailable, get_decision
from handlers import execute_handler

BASE_DIR = Path(__file__).resolve().parent
DATA_DIR = BASE_DIR / "data"
AUDIT_PATH = DATA_DIR / "audit.jsonl"

TEAM_ROUTES = {
    "account": "account_lookup",
    "billing": "billing_llm",
    "technical": "knowledge_rag",
    "product": "product_llm",
}


def _config():
    with (DATA_DIR / "config.json").open(encoding="utf-8") as file:
        return json.load(file)


def route_rules_only(ticket):
    if ticket["known_incident"]:
        return {
            "matched": True,
            "route": "status_handler",
            "reason": "known_incident",
        }
    if ticket["admin_change_requested"]:
        return {
            "matched": True,
            "route": "human_security",
            "reason": "admin_change_requires_human",
        }
    return {
        "matched": False,
        "route": "human_review",
        "reason": "rules_exhausted",
    }


def _append_audit(record):
    DATA_DIR.mkdir(exist_ok=True)
    with AUDIT_PATH.open("a", encoding="utf-8") as file:
        file.write(json.dumps(record, sort_keys=True) + "\n")


def route_ticket(ticket, write_audit=True):
    started = perf_counter()
    rules_result = route_rules_only(ticket)
    decision = None
    team = None
    team_confidence = None
    urgency = None
    needs_human_probability = None

    if rules_result["matched"]:
        route = rules_result["route"]
        reason = rules_result["reason"]
        mode = "deterministic"
    else:
        try:
            decision = get_decision(ticket)
            mode = decision["decision_mode"]
            answers = decision["answers"]
            team = answers["team"]["choice"]
            team_confidence = answers["team"]["confidence"]
            urgency = answers["urgency"]["score"]
            needs_human_probability = answers["needs_human"]["noul"]
            policy = _config()

            if team_confidence < policy["min_team_confidence"]:
                route = "human_review"
                reason = "low_team_confidence"
            elif needs_human_probability >= policy["human_probability_threshold"]:
                route = "human_review"
                reason = "human_judgment_required"
            elif urgency >= policy["urgent_score_threshold"]:
                route = "human_review"
                reason = "critical_urgency"
            else:
                route = TEAM_ROUTES[team]
                reason = "confidence_gate_passed"
        except (DecisionUnavailable, KeyError, ValueError) as exc:
            route = "human_review"
            reason = f"decision_provider_failed: {exc}"
            mode = "unavailable"

    handler_result = execute_handler(route, ticket)
    usage_cost = 0
    if decision:
        usage_cost = decision.get("usage", {}).get("cost", 0)

    result = {
        "ticket_id": ticket["ticket_id"],
        "route": route,
        "reason": reason,
        "decision_mode": mode,
        "team": team,
        "team_confidence": team_confidence,
        "urgency_score": urgency,
        "needs_human_probability": needs_human_probability,
        "latency_ms": round((perf_counter() - started) * 1000, 2),
        "decision_cost_usd": usage_cost,
        "downstream_llm_required": handler_result["downstream_llm_required"],
        "handler_result": handler_result,
    }

    if write_audit:
        _append_audit(
            {
                "timestamp": datetime.now(timezone.utc).isoformat(),
                **result,
            }
        )
    return result


def read_metrics():
    if not AUDIT_PATH.exists():
        return {
            "tickets": 0,
            "automation_rate": 0,
            "human_review_rate": 0,
            "downstream_llm_calls": 0,
            "decision_cost_usd": 0,
        }

    records = [
        json.loads(line)
        for line in AUDIT_PATH.read_text(encoding="utf-8").splitlines()
        if line.strip()
    ]
    total = len(records)
    human = sum(record["route"].startswith("human_") for record in records)
    downstream = sum(record["downstream_llm_required"] for record in records)
    cost = sum(record["decision_cost_usd"] for record in records)
    return {
        "tickets": total,
        "automation_rate": round((total - human) / total, 3),
        "human_review_rate": round(human / total, 3),
        "downstream_llm_calls": downstream,
        "decision_cost_usd": round(cost, 8),
    }

How does the confidence gate work?

• The route_rules_only() function keeps known incidents and admin changes on deterministic paths.
• The TEAM_ROUTES mapping converts bounded team choices into existing handler names.
• Low team confidence routes to human_review.
• High human-review probability routes to human_review.
• Critical urgency routes to human_review.
• Missing fixtures and malformed decisions also route to human_review.
• The audit record preserves the decision fields and visible handler result.

• In the VS Code Explorer sidebar, return to main.py.
• Open the second tab below to replace main.py with the final API wiring.
• Save main.py.

✔️ My API wiring matches

Your ticket endpoint now calls route_ticket(). The health response also reports the current decision mode.

ⓧ I'd like to double check the full code

from fastapi import FastAPI
from pydantic import BaseModel

from decision_provider import current_mode
from router import read_metrics, route_ticket


class Ticket(BaseModel):
    ticket_id: str
    account_id: str
    customer_tier: str
    message: str
    known_incident: bool = False
    admin_change_requested: bool = False


app = FastAPI(
    title="SupportOps Control Plane",
    version="1.0.0",
    description="Confidence-gated routing for a B2B SaaS support workflow.",
)


@app.get("/health")
def health():
    return {"status": "ok", "decision_mode": current_mode()}


@app.post("/tickets/route")
def route(ticket: Ticket):
    return route_ticket(ticket.model_dump())


@app.get("/metrics")
def metrics():
    return read_metrics()

What changed in the API?

• The health endpoint reads the configured decision mode through current_mode().
• The ticket endpoint converts the validated model into a dictionary through model_dump().
• The ticket endpoint passes that dictionary to route_ticket().
• The metrics endpoint exposes the counters derived from routed tickets.

• Return to the VS Code terminal where Uvicorn is running.
• Stop the current process by pressing Control-C.
• Start the updated API by running this command:

uvicorn main:app

What should I see?

Uvicorn reports that the application is running on http://127.0.0.1:8000. The updated imports have loaded successfully.

Does the API fail during startup?

• Check that decision_provider.py sits beside main.py.
• Confirm that both JSON files sit inside the existing data folder.
• Ask for help with the startup traceback: Help me debug the Uvicorn startup error after adding the Jev decision layer.

Verify deterministic and replay routing

The same demo now exercises two control-plane paths. One ticket should bypass the decision provider while the ambiguous ticket should use its typed replay decision.

• In VS Code, select Terminal from the top menu.
• Select New Terminal to keep Uvicorn running in its current terminal.

Before you run the demo, do you expect both tickets to use replay mode?

• Run the two-ticket demo in the new terminal with this command:

python demo.py

What should I see?

The terminal prints t-001: route=status_handler mode=deterministic reason=known_incident for the outage ticket.

It also prints t-003: route=billing_llm mode=replay reason=confidence_gate_passed for the ambiguous billing ticket.

That is the control boundary working. Known facts stay deterministic while bounded semantic decisions unlock safe automation.

• In the VS Code Explorer sidebar, select the new data/audit.jsonl file.
• Find the record whose ticket_id is t-003.

You should see team billing with confidence 0.88. The urgency score is 0.8. The human-review probability is 0.2.

Does the billing ticket still reach human review?

• Confirm that decision_mode remains set to replay in data/config.json.
• Check that the t-003 fixture contains team choice billing with confidence 0.88.
• Ask for help tracing the route: Help me find why t-003 still routes to human_review.

Your support gateway now combines deterministic boundaries with typed semantic decisions. Next, you will measure whether that hybrid design improves routing without creating unsafe automation.

Evaluate Safety and Value

Your confidence-gated router now handles known incidents deterministically. It also routes unresolved tickets through bounded replay decisions.

A customer still needs labeled evidence before approving a rollout. This step measures routing accuracy, automation coverage, unsafe automation, and downstream model demand.

In this step, get ready to:

• Generate append-only audit evidence from routed tickets.
• Inspect operational metrics derived from the audit trail.
• Compare rules-only routing with the hybrid control plane.

Inspect the append-only audit trail

An audit trail preserves the evidence behind each routing outcome. The service stores one JSON Lines record for every ticket routed with auditing enabled.

• Append fresh audit records by running the existing two-ticket demo:

python demo.py

What did this run prove?

• Ticket t-001 follows the deterministic status_handler route.
• Ticket t-003 follows the replay-mode billing_llm route.
• Each request reaches route_ticket() with write_audit=True by default.
• The router appends each completed result to data/audit.jsonl.

• Switch back to Visual Studio Code.
• Select data/audit.jsonl from the file sidebar.

You should see one line per routed ticket. Each line includes the route, its reason, the decision mode, confidence signals, latency, cost, downstream model demand, and handler result.

You now have traceable evidence for both routing paths. Every decision carries enough context for an operational review.

Can't find the audit file?

Confirm the demo reached the running local API. A failed request cannot create an audit record.

Check that data is beside router.py in the SupportOps project folder.

Ask for help with the audit path.

Read the operational metrics

The FastAPI metrics endpoint turns individual audit records into rollout signals. Operators can use these totals to track automation pressure without reading every line manually.

• Enter http://127.0.0.1:8000 followed immediately by /metrics in your browser's address bar.
• Press Enter to load the current metrics.

You'll see a JSON object derived from every record currently stored in data/audit.jsonl.

How are these metrics calculated?

• The tickets field counts the audit records.
• The automation_rate field measures routes that do not start with human_.
• The human_review_rate field measures routes that start with human_.
• The downstream_llm_calls field counts handlers that require a downstream model.
• The decision_cost_usd field adds the decision cost recorded for every ticket.

That gives you a live operational view of the control plane. Replay decisions keep the accumulated decision cost at zero.

Metrics endpoint unavailable?

Confirm the local API process from earlier is still running. The metrics endpoint depends on that process.

Check that the address ends with the project endpoint /metrics.

Help me diagnose the local metrics endpoint.

Build the labeled evaluator

A labeled evaluation compares each predicted route with a customer-approved expected route. It also flags any automated outcome that should have remained with a human.

• Create evaluate.py beside router.py using Visual Studio Code's new-file control.
• Paste this first section into evaluate.py to build one comparison row per ticket:

import json
from pathlib import Path

from router import route_rules_only, route_ticket

BASE_DIR = Path(__file__).resolve().parent
HUMAN_ROUTES = {"human_review", "human_security"}

with (BASE_DIR / "data" / "tickets.json").open(encoding="utf-8") as file:
    TICKETS = json.load(file)


def evaluate(name, policy):
    rows = []
    for ticket in TICKETS:
        expected = ticket["expected_route"]
        payload = {key: value for key, value in ticket.items() if key != "expected_route"}
        result = policy(payload)
        route = result["route"]
        rows.append(
            {
                "ticket_id": ticket["ticket_id"],
                "expected": expected,
                "actual": route,
                "correct": route == expected,
                "unsafe": expected in HUMAN_ROUTES and route not in HUMAN_ROUTES,
                "downstream_llm": result.get("downstream_llm_required", False),
            }
        )

What does this section measure?

• The HUMAN_ROUTES set defines the two safe escalation outcomes.
• The evaluator removes expected_route before sending each fixture into a routing policy.
• The correct field records whether the actual route matches the labeled route.
• The unsafe field catches automation applied to a ticket labeled for human handling.
• The downstream_llm field tracks demand created after routing.

• Save evaluate.py.
• Confirm the editor shows no syntax error markers in this section.

Seeing syntax markers?

Check the indentation inside evaluate(). Every field inside the row dictionary must stay nested under rows.append().

Help me fix the evaluator structure.

The row list holds the raw comparison evidence. The remaining logic aggregates those rows into rollout metrics.

• Place your cursor on a blank line after the closing ) inside evaluate().
• Paste this section to calculate the summary and run both policies:

    total = len(rows)
    correct = sum(row["correct"] for row in rows)
    automated = sum(row["actual"] not in HUMAN_ROUTES for row in rows)
    unsafe = sum(row["unsafe"] for row in rows)
    downstream = sum(row["downstream_llm"] for row in rows)

    print(f"\n{name}")
    for row in rows:
        marker = "PASS" if row["correct"] else "MISS"
        print(
            f"  {marker} {row['ticket_id']}: "
            f"expected={row['expected']} actual={row['actual']}"
        )
    print(f"  route_accuracy: {correct}/{total}")
    print(f"  automation_coverage: {automated}/{total}")
    print(f"  unsafe_automations: {unsafe}")
    print(f"  downstream_llm_calls: {downstream}")


if __name__ == "__main__":
    evaluate("RULES ONLY", route_rules_only)
    evaluate("HYBRID CONTROL PLANE", lambda ticket: route_ticket(ticket, write_audit=False))

What does this section report?

• The summary counts correct routes across all seven labeled fixtures.
• Automation coverage counts routes outside HUMAN_ROUTES.
• Unsafe automation counts tickets that bypassed required human handling.
• Downstream model demand counts handlers that require another model call.
• The hybrid evaluation uses write_audit=False to keep test traffic out of operational metrics.

• Save evaluate.py.
• Add another terminal using the plus control beside the running API terminal.

Before you run the evaluator, make a quick mental prediction about hybrid accuracy and unsafe automation.

• Run the labeled policy comparison in the new terminal by entering this command:

python evaluate.py

What should you see?

• The RULES ONLY report shows route accuracy of 3/7.
• The HYBRID CONTROL PLANE report shows route accuracy of 7/7.
• Hybrid automation coverage reaches 5/7.
• Hybrid unsafe automation remains at 0.
• Hybrid downstream model demand reaches 3 calls.

Strong result. You have measured evidence that the hybrid policy improves route accuracy while preserving every required human escalation.

Evaluator not reaching the expected totals?

Confirm data/config.json still uses replay mode. The labeled totals depend on the supplied replay fixtures.

Check that the hybrid call passes write_audit=False into route_ticket().

Help me compare my evaluation output.

✔️ Awesome, I've got everything!

Your saved evaluate.py now compares both policies without adding evaluation traffic to the audit log.

ⓧ I'd like to double check the full code

Your complete evaluate.py should match this file.

import json
from pathlib import Path

from router import route_rules_only, route_ticket

BASE_DIR = Path(__file__).resolve().parent
HUMAN_ROUTES = {"human_review", "human_security"}

with (BASE_DIR / "data" / "tickets.json").open(encoding="utf-8") as file:
    TICKETS = json.load(file)


def evaluate(name, policy):
    rows = []
    for ticket in TICKETS:
        expected = ticket["expected_route"]
        payload = {key: value for key, value in ticket.items() if key != "expected_route"}
        result = policy(payload)
        route = result["route"]
        rows.append(
            {
                "ticket_id": ticket["ticket_id"],
                "expected": expected,
                "actual": route,
                "correct": route == expected,
                "unsafe": expected in HUMAN_ROUTES and route not in HUMAN_ROUTES,
                "downstream_llm": result.get("downstream_llm_required", False),
            }
        )

    total = len(rows)
    correct = sum(row["correct"] for row in rows)
    automated = sum(row["actual"] not in HUMAN_ROUTES for row in rows)
    unsafe = sum(row["unsafe"] for row in rows)
    downstream = sum(row["downstream_llm"] for row in rows)

    print(f"\n{name}")
    for row in rows:
        marker = "PASS" if row["correct"] else "MISS"
        print(
            f"  {marker} {row['ticket_id']}: "
            f"expected={row['expected']} actual={row['actual']}"
        )
    print(f"  route_accuracy: {correct}/{total}")
    print(f"  automation_coverage: {automated}/{total}")
    print(f"  unsafe_automations: {unsafe}")
    print(f"  downstream_llm_calls: {downstream}")


if __name__ == "__main__":
    evaluate("RULES ONLY", route_rules_only)
    evaluate("HYBRID CONTROL PLANE", lambda ticket: route_ticket(ticket, write_audit=False))

Secret mission

Write the Shadow-Rollout Recommendation

Turn your evaluation results into a customer-facing deployment decision. You will define a shadow-mode phase with measurable gates, named owners, rollback triggers, and a recommendation grounded in replay evidence.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the local project available, pause its Uvicorn server, or delete its files. Replay mode made no paid OpenRouter requests or ongoing service charges.

Resources you used:

• Local server process running through Uvicorn.
• Application files (decision_provider.py, demo.py, evaluate.py, handlers.py, main.py, and router.py).
• Dependency manifest (requirements.txt).
• Mock data and replay evidence in data/ (accounts.json, audit.jsonl, config.json, jev_replay.json, knowledge.json, and tickets.json).
• Customer-facing shadow-rollout recommendation (rollout.md).

Keep everything running

No action is needed. Choose this if you plan to refine the routing policy or reuse the deployment evidence.

• The local server remains available for ticket-routing demonstrations.
• The replay configuration keeps future demonstrations free from paid decision requests.
• The audit log remains available as evidence of routed tickets and escalation decisions.
• The evaluation report and rollout.md remain available for customer rollout discussions.
• The Python packages from requirements.txt remain installed for future local work.

Pause - I'll come back to this later

Stop the running server to free local resources while preserving every project artifact.

• Return to the Visual Studio Code terminal panel from earlier.
• Stop the local server by pressing Control-C.

The command prompt returns when the local server stops. Your files stay in place for the next session.

Delete - I don't want to use this again

Deletion permanently removes the local project artifacts listed above. Your OpenRouter account remains unchanged.

• Return to the Visual Studio Code terminal panel from earlier.
• Stop the local server by pressing Control-C.

The command prompt returns when Uvicorn stops.

• Remove every recorded project artifact by running these commands:

rm -rf data
rm -f decision_provider.py demo.py evaluate.py handlers.py main.py requirements.txt rollout.md router.py

What Does This Cleanup Remove?

• The first command removes the mock customer records, replay fixtures, labeled tickets, configuration, and audit log.
• The second command removes the application code, demo scripts, evaluator, dependency manifest, and rollout recommendation.
• The containing folder remains available as an empty location.

• Check the project file list in Visual Studio Code.

You should no longer see data/, decision_provider.py, demo.py, evaluate.py, handlers.py, main.py, requirements.txt, rollout.md, or router.py. Your local SupportOps artifacts are now cleared.

The Python packages from requirements.txt remain installed. This avoids removing shared packages from your Python environment.

Still See Project Files?

• Check that you used the same terminal panel where the local server was running.
• Repeat the cleanup commands from that terminal panel.
• Help me verify why the recorded project files still appear after cleanup.

Nice Work!

Nice Work!

Congratulations! Your customer-facing SupportOps Control Plane now routes B2B support tickets behind explicit safety gates. Its replay-mode evidence supports a measured rollout decision.

You've learned how to:

• Build a validated FastAPI support API with deterministic controls for known incidents. Force security-sensitive admin changes into human review.
• Route ambiguous tickets through a replay-first Jev contract that returns typed decisions behind confidence gates. Send each approved route to the matching mock CRM workflow. Support status handling, knowledge retrieval, specialist models, plus human escalation.
• Prove the hybrid router reached 7/7 route accuracy with 5/7 automation coverage. Confirm zero unsafe automations in replay mode. Inspect append-only JSON Lines audit records plus operational metrics.
• Secret Mission: Write a customer-facing shadow-mode rollout recommendation with measurable acceptance gates. Define at least three rollback triggers. Name the responsible owners. Separate reusable control-plane practices from customer-specific routing policy. Tie the go or no-go decision to the observed replay evidence.

Ready to quiz yourself?