Build an Observable Support Agent

Build a tool-using support agent with guardrails, traces, and evaluations.

Introduction

30 Second Summary

A support reply can sound polished while quietly guessing the customer’s account status or the state of a service. That confidence becomes risky when a real customer acts on it.

In this project, you will build an observable support-triage AI agent for a fictional SaaS product called AcmeSync. It gathers synthetic operational evidence before a deterministic safety policy decides whether to reply or escalate.

What You'll Build

You will watch a synthetic support ticket become an evidence-backed decision with every tool call visible in the terminal.

By the end of this project, you'll have:

  • A baseline comparison where baseline.py gives a polished guess before agent_core.py exposes customer evidence plus incident evidence.
  • A deterministic safety gate that turns unknown customers, security-sensitive requests, dependency failures, or incomplete runs into visible human escalations.
  • Repeatable proof through clear PASS or FAIL evaluations, inspectable JSONL traces, plus a DESIGN.md architecture brief.
  • Secret Mission: Inject an incident-service outage to prove the agent records the failure in a safe human escalation.

Are there any prerequisites?

You need a Google account for Google AI Studio.

Step 1 verifies that Python 3.10 or newer is ready.

The Gemini Developer API Free Tier keeps this synthetic-data project free.

Before We Start

Before any setup begins, commit to the support decision your AcmeSync agent must make. Your answer will connect evidence-backed triage with the production judgment a Forward Deployed Engineering interviewer wants to see.

Prepare the Free macOS Workspace

Every later result from your support-triage agent depends on a predictable Python runtime. A reproducible workspace removes setup ambiguity before the first model call.

An isolated environment keeps the project on one dependency set. A session-only API key keeps the credential out of your project files.

In this step, get ready to:
  • Open a named project folder as a Visual Studio Code workspace.
  • Prepare a Python 3.10 or newer virtual environment.
  • Install the pinned SDK with Free Tier authorization.
Create the project workspace

The project folder gives every later script a predictable home. Opening it as a VS Code workspace makes the integrated terminal start in the same location.

  • Select Desktop in the Finder sidebar.
  • Press Shift+Cmd+N to create a folder.

Finder highlights the new folder name so you can replace it.

  • Type fde-triage-agent as the folder name.
  • Press Return to save the name.

You should see fde-triage-agent on your Desktop.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.

VS Code opens to its starting screen.

  • Click File in the VS Code menu bar.
  • Select Open Folder....

You should see a folder picker where you can choose the workspace location.

  • Select the fde-triage-agent folder on your Desktop.
  • Click Open.

The Explorer sidebar should now show fde-triage-agent as the open workspace.

  • Click View in the VS Code menu bar.
  • Select Terminal.

You should see the integrated terminal beneath the editor. That first setup hurdle is complete because the terminal now starts inside your named workspace.

Did the Wrong Folder Open?

Check whether fde-triage-agent appears at the top of the Explorer sidebar. Return to Open Folder... if another folder appears there.

Close the existing terminal after correcting the workspace. Create a new terminal from the View menu.

Help me verify that the correct folder is open in VS Code.

Verify Python and isolate the environment

The Google Gen AI SDK requires Python 3.10 or newer. Checking the runtime first prevents an older installation from blocking the pinned package.

  • Check the Python version available on your Mac by running this command:
python3 --version

What Does This Command Check?

The command asks your shell which Python 3 runtime it can access. The reported version determines whether the runtime meets the SDK requirement.

✔️ I see version 3.10 or higher

Your Python runtime meets the project requirement. Continue with this installation.

ⓧ I see an older version

The available runtime is below the SDK requirement. Install the current stable macOS release before creating the project environment.

  • Open the Python.org macOS releases page in your browser.
  • Download the current stable macOS installer.
  • Run the downloaded installer with its default options.
  • Quit VS Code after the installer finishes.
  • Reopen VS Code through your search bar.
  • Return to the integrated terminal in fde-triage-agent.
  • Repeat the version check from above.

Continue when the terminal reports Python 3.10 or newer.

ⓧ Command not found

Your shell cannot find a Python 3 runtime. Install the current stable macOS release from Python.org.

  • Open the Python.org macOS releases page in your browser.
  • Download the current stable macOS installer.
  • Run the downloaded installer with its default options.
  • Quit VS Code after the installer finishes.
  • Reopen VS Code through your search bar.
  • Return to the integrated terminal in fde-triage-agent.
  • Repeat the version check from above.

Continue when the terminal reports Python 3.10 or newer.

A virtual environment gives this project its own interpreter context. Packages installed there stay separate from your other Python projects.

  • Create the isolated project environment by running this command:
python3 -m venv .venv

What Does This Command Create?

Python creates a .venv folder inside fde-triage-agent. That folder holds the project-specific interpreter context.

  • Locate .venv in the VS Code Explorer sidebar.

You should see .venv inside fde-triage-agent.

Did the Environment Fail to Appear?

Confirm that the Explorer sidebar shows fde-triage-agent as the workspace. Confirm that the version check reports Python 3.10 or newer.

Help me diagnose why the virtual environment was not created.

  • Activate the virtual environment by running this command:
source .venv/bin/activate

What Does Activation Change?

Activation points python at the interpreter inside .venv. Package installation now stays inside this workspace.

You should see the virtual environment name at the start of the terminal prompt. That prefix confirms the isolated environment is active.

Is the Environment Name Missing?

Confirm that .venv exists in the Explorer sidebar. Repeat the activation command from the integrated terminal.

Help me activate the virtual environment in VS Code.

Pin the SDK and protect the key

A pinned dependency makes the workspace reproducible. This project uses Google Gen AI SDK version 2.33.0.

  • Select the new-file icon at the top of the VS Code Explorer sidebar.
  • Enter requirements.txt as the file name.

The empty requirements.txt file should open in the editor.

  • Add the pinned dependency by pasting this line:
google-genai==2.33.0

What Does This Dependency Pin Do?

The double equals sign selects one exact release of google-genai. Every installation from this file uses the SDK behavior expected by the project.

  • Press Cmd+S to save requirements.txt.

You should see requirements.txt in the Explorer sidebar with one pinned dependency.

Is the Requirements File Missing?

Confirm that the file sits directly inside fde-triage-agent. Check that its name ends with .txt only once.

Help me verify the dependency file.

✔️ Awesome, I've got everything!

Your dependency file is ready. Confirm that it is saved before continuing.

ⓧ I'd like to double check the full code

google-genai==2.33.0

What Should This File Contain?

The complete file contains one exact dependency pin. Compare the spelling plus the version before continuing.

The workspace also needs a boundary around local packages plus generated data. A .gitignore file records those exclusions before any agent code exists.

  • Select the new-file icon at the top of the VS Code Explorer sidebar.
  • Enter .gitignore as the file name.

The empty .gitignore file should open in the editor.

  • Add the project exclusions by pasting this content:
.venv/
__pycache__/
*.pyc
traces.jsonl
.env

What Does This File Protect?

  • The .venv/ entry excludes the local environment.
  • The Python cache entries exclude generated bytecode.
  • The traces.jsonl entry excludes local execution traces.
  • The .env entry excludes environment files.
  • Press Cmd+S to save .gitignore.

You should see .gitignore in the Explorer sidebar. Its five entries now cover the local environment plus generated project data.

Is the File Hard to Find?

Files beginning with a period can look muted in an editor. Check the Explorer sidebar inside fde-triage-agent before creating another copy.

Help me verify the project exclusions.

✔️ Awesome, I've got everything!

Your exclusion file is ready. Confirm that it is saved before installing the dependency.

ⓧ I'd like to double check the full code

.venv/
__pycache__/
*.pyc
traces.jsonl
.env

What Should This File Contain?

The complete file contains five exclusions. Compare every line with your saved file.

  • Install the pinned dependency into the active environment by running this command:
python -m pip install -r requirements.txt

What Does This Installation Command Do?

The active Python interpreter runs pip inside the virtual environment. Pip reads requirements.txt before installing the pinned SDK.

The terminal returns to its prompt after the installation completes. Your isolated environment now contains the project dependency.

Did the Dependency Installation Fail?

Confirm that the terminal prompt shows the active environment name. Confirm that requirements.txt is saved inside fde-triage-agent.

Help me diagnose the SDK installation failure.

The Gemini Developer API uses an API key to authorize requests. This project keeps the credential in the current terminal session.

Keep the Project Free and Synthetic

The Free Tier lists input plus output for gemini-3.8-flash as free of charge. Free Tier content may be used to improve Google products.

  • Keep the account on the Free Tier by leaving Set up billing untouched.
  • Use only the synthetic AcmeSync data supplied in this project.

Creating a key can feel risky. You will keep it out of screenshots plus project files.

You should see the API Keys page with the available key controls.

  • Click Create API key.
  • Keep the automatically created project selected if Google AI Studio asks for a project.

You should see a new authorization key in the key creation flow.

  • Copy the new API key to your clipboard.
  • Leave Set up billing untouched.

That sensitive setup is complete. Your Free Tier key is ready for the current terminal session.

  • Return to the integrated terminal in VS Code.
  • Export the key for the current terminal by replacing YOUR_API_KEY in this command:
export GEMINI_API_KEY="YOUR_API_KEY"

How Does the Session-Only Key Work?

The shell stores GEMINI_API_KEY for processes started from this terminal. A new terminal requires another export.

The SDK reads this environment variable when it creates a client. The terminal returns to its prompt without displaying the key.

Does a Later Request Lack Authorization?

Return to the terminal where the virtual environment is active. Repeat the export command with your key after opening any new terminal session.

Keep the key out of source files plus troubleshooting messages. Revoke it in Google AI Studio if it becomes exposed.

Help me verify the session configuration without revealing the key.

Before you run the final check, what two version details do you expect to see? Hold your answer in mind.

  • Verify the active Python runtime plus the installed SDK by running these commands:
python --version
python -m pip show google-genai

What Does the Final Check Prove?

The first command checks the Python interpreter selected by the active environment. The second command displays the installed google-genai package details.

You should see Python 3.10 or newer. The package details should include Version: 2.33.0.

You have cleared the setup hurdle. The workspace now has an isolated runtime plus a pinned SDK.

Your free macOS workspace is ready. Next, you will run a polished baseline assistant and expose the operational evidence it cannot gather.

Compare the Naive Baseline

Your workspace is ready to send its first request to Gemini. Now you need a baseline that shows how the model behaves when it receives a support ticket by itself.

A polished support answer can hide whether the assistant checked customer or incident facts. This step lets the naive version speak first, so you can judge its claims against the evidence it actually received.

In this step, get ready to:
  • Create a baseline script for the synthetic C-102 delayed-sync ticket.
  • Run the baseline to inspect every operational claim.
  • Identify the gap between fluent language and verified customer-system evidence.
Create the baseline assistant

The baseline.py script gives Gemini one synthetic ticket plus a broad instruction. This creates a clean comparison point for later versions of the system.

  • In the VS Code Explorer view, select New File....

You'll see a file-name field inside the Explorer view.

  • Enter baseline.py in the file-name field.
  • Press Enter.

The new baseline.py file opens in the editor.

  • Replace the empty contents of baseline.py with this code:
from google import genai

MODEL = "gemini-3.8-flash"
TICKET = "Customer C-102 says AcmeSync jobs have been delayed for 45 minutes. What should support tell them?"

client = genai.Client()
try:
    interaction = client.interactions.create(
        model=MODEL,
        input=(
            "You are a helpful SaaS support assistant. Reply to this ticket with a "
            f"confident recommendation.\n\nTicket: {TICKET}"
        ),
    )
    print(interaction.output_text)
finally:
    client.close()

What Does This Code Do?

  • The MODEL value selects gemini-3.8-flash for the request.
  • The TICKET value contains the synthetic delayed-sync question from customer C-102.
  • The genai.Client() client uses the GEMINI_API_KEY available in your current terminal session.
  • The client.interactions.create() call sends the instruction plus the ticket to the model.
  • The interaction.output_text value contains the model's final reply.
  • The finally block closes the client even if the request fails.
  • Press Cmd+S (macOS) or Ctrl+S (Windows) to save baseline.py.

The baseline.py tab remains open with the complete script. The file is listed directly inside fde-triage-agent in Explorer.

Can't Find baseline.py?

Confirm that baseline.py sits directly inside fde-triage-agent in the Explorer view.

If the file appears under another folder, move it into fde-triage-agent before continuing.

Help me place baseline.py in the correct VS Code folder.

✔️ Awesome, I've got everything!

Your baseline.py file is saved. It is ready to run.

ⓧ I'd like to double check the full code

The complete file should match this reference.

from google import genai

MODEL = "gemini-3.8-flash"
TICKET = "Customer C-102 says AcmeSync jobs have been delayed for 45 minutes. What should support tell them?"

client = genai.Client()
try:
    interaction = client.interactions.create(
        model=MODEL,
        input=(
            "You are a helpful SaaS support assistant. Reply to this ticket with a "
            f"confident recommendation.\n\nTicket: {TICKET}"
        ),
    )
    print(interaction.output_text)
finally:
    client.close()
Run the baseline and inspect its claims

The baseline has one source of information: the ticket text stored in TICKET. Running it tests whether fluent language alone can support an operational decision.

Before you run it, which claims do you think Gemini can verify from the ticket alone? Take a moment to predict.

  • Test the baseline by running this command:
python baseline.py

What Does This Command Check?

  • Python runs baseline.py inside your active virtual environment.
  • The script sends the synthetic ticket to Gemini.
  • The script prints the returned text in your terminal.

You'll see a fluent support reply in the terminal. The exact wording can vary between runs.

The run contains no verified customer or incident record. It shows no tool call or trace.

This missing evidence is the intended result of the baseline. It exposes the risk hidden behind a confident response.

Didn't Get a Reply?

Return to the VS Code terminal from setup. Confirm that the active environment and session-only GEMINI_API_KEY are still available there.

If you replaced that terminal session, reactivate .venv before exporting the key again.

Help me troubleshoot why baseline.py does not print a Gemini reply.

  • Read the terminal reply one claim at a time.
  • Mark every operational claim on paper.
  • Compare each marked claim with the exact text stored in TICKET.
  • Circle every claim that TICKET does not support.

You've now isolated the baseline's gap. Production triage needs verified system evidence behind every operational claim.

You found the exact weakness: the baseline can sound convincing without proving its claims. Next, you'll give the agent bounded tools for customer and incident evidence.

Build the Bounded Tool Agent

Your baseline proved that fluent language can hide missing operational evidence. A support decision needs verified customer facts before it reaches the customer.

You will now build a bounded agent loop that gathers evidence through small tools. Required checks stay under Python's control. Gemini decides whether an optional knowledge-base search adds value.

In this step, get ready to:
  • Define synthetic support records with three bounded tools.
  • Build a five-turn loop that forces the required customer check.
  • Run the delayed-sync ticket with visible customer evidence.
Build the bounded tool layer

A bounded tool returns a small record for one clear question. This keeps the model focused on the evidence needed for a support decision.

  • Create a new file named tools.py from the Explorer sidebar.
  • Add the synthetic records with their lookup functions by pasting this code:
CUSTOMERS = {
    "C-101": {"found": True, "customer_id": "C-101", "plan": "Starter", "region": "us-west", "account_status": "active"},
    "C-102": {"found": True, "customer_id": "C-102", "plan": "Enterprise", "region": "us-east", "account_status": "active"},
}

INCIDENTS = {
    "sync": {"service": "sync", "status": "degraded", "incident_id": "INC-42", "customer_message": "We are investigating delayed sync jobs in us-east."},
    "billing": {"service": "billing", "status": "operational", "incident_id": None, "customer_message": "No active billing incident is reported."},
    "login": {"service": "login", "status": "operational", "incident_id": None, "customer_message": "No active login incident is reported."},
}

KNOWLEDGE_BASE = [
    {"article_id": "KB-101", "keywords": ["sync", "delay", "delayed"], "title": "Responding to delayed sync jobs", "guidance": "Acknowledge the delay, share a verified incident ID, and avoid promising a recovery time."},
    {"article_id": "KB-201", "keywords": ["billing", "duplicate", "charge"], "title": "Investigating duplicate charges", "guidance": "Collect the invoice ID and route refund decisions to a billing specialist."},
]


def get_customer_context(customer_id: str) -> dict:
    return CUSTOMERS.get(customer_id, {"found": False, "customer_id": customer_id, "message": "No matching synthetic customer record."})


def get_incident_status(service: str) -> dict:
    return INCIDENTS.get(service.lower(), {"service": service, "status": "unknown", "incident_id": None, "customer_message": "This service is not present in the synthetic status system."})


def search_knowledge_base(query: str) -> dict:
    normalized = query.lower()
    matches = [article for article in KNOWLEDGE_BASE if any(keyword in normalized for keyword in article["keywords"])]
    return {"query": query, "matches": matches[:2]}

What does this code do?

  • The CUSTOMERS records provide account status without exposing a raw customer database.
  • The INCIDENTS records provide a verified status for each known service.
  • The KNOWLEDGE_BASE records provide concise guidance for matching support symptoms.
  • Each function accepts one focused input. Each function returns a small dictionary that the agent can inspect.
  • Save tools.py.
  • Locate tools.py in the Explorer sidebar.

You should see tools.py beside baseline.py. The workspace now contains its synthetic source of operational truth.

Is the tool file missing?

Check that you created tools.py inside the same fde-triage-agent folder as baseline.py.

Confirm that the filename ends with .py.

Help me check where my tools.py file was created.

The Python functions can execute a lookup. The model also needs tool contracts that describe each available function and its accepted arguments.

  • Place your cursor below the search_knowledge_base function.
  • Add the tool declarations with their function registry by pasting this code:
TOOLS = [
    {"type": "function", "name": "support_get_customer_context", "description": "Look up one synthetic AcmeSync customer before making a support decision.", "parameters": {"type": "object", "properties": {"customer_id": {"type": "string", "description": "Customer identifier from the ticket, for example C-102."}}, "required": ["customer_id"]}},
    {"type": "function", "name": "support_get_incident_status", "description": "Check the synthetic operational status for one named AcmeSync service.", "parameters": {"type": "object", "properties": {"service": {"type": "string", "description": "Service name from the ticket, such as sync, billing, or login."}}, "required": ["service"]}},
    {"type": "function", "name": "support_search_knowledge_base", "description": "Search concise support guidance after customer and service facts are known.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "A short problem description using the verified service and symptom."}}, "required": ["query"]}},
]

AVAILABLE_FUNCTIONS = {"support_get_customer_context": get_customer_context, "support_get_incident_status": get_incident_status, "support_search_knowledge_base": search_knowledge_base}

How do the tool contracts work?

  • Each JSON Schema declaration gives Gemini a function name. It also explains the function's purpose.
  • The parameters object defines the accepted arguments. The required list identifies the argument that must be supplied.
  • The AVAILABLE_FUNCTIONS dictionary connects each declared name to the Python function that executes it.
  • Save tools.py.
  • Scroll to the bottom of tools.py.

You should see AVAILABLE_FUNCTIONS as the final assignment. All three declared tool names should map to their matching Python functions.

Do the declarations look incomplete?

Check that the TOOLS list closes before AVAILABLE_FUNCTIONS begins.

Compare the three registry keys with the three name values. Every name must match exactly.

Help me find a mismatch in my Python tool declarations.

✔️ Awesome, I've got everything!

Great. Double-check that you saved tools.py before building the agent loop.

ⓧ I'd like to double check the full code

CUSTOMERS = {
    "C-101": {"found": True, "customer_id": "C-101", "plan": "Starter", "region": "us-west", "account_status": "active"},
    "C-102": {"found": True, "customer_id": "C-102", "plan": "Enterprise", "region": "us-east", "account_status": "active"},
}

INCIDENTS = {
    "sync": {"service": "sync", "status": "degraded", "incident_id": "INC-42", "customer_message": "We are investigating delayed sync jobs in us-east."},
    "billing": {"service": "billing", "status": "operational", "incident_id": None, "customer_message": "No active billing incident is reported."},
    "login": {"service": "login", "status": "operational", "incident_id": None, "customer_message": "No active login incident is reported."},
}

KNOWLEDGE_BASE = [
    {"article_id": "KB-101", "keywords": ["sync", "delay", "delayed"], "title": "Responding to delayed sync jobs", "guidance": "Acknowledge the delay, share a verified incident ID, and avoid promising a recovery time."},
    {"article_id": "KB-201", "keywords": ["billing", "duplicate", "charge"], "title": "Investigating duplicate charges", "guidance": "Collect the invoice ID and route refund decisions to a billing specialist."},
]


def get_customer_context(customer_id: str) -> dict:
    return CUSTOMERS.get(customer_id, {"found": False, "customer_id": customer_id, "message": "No matching synthetic customer record."})


def get_incident_status(service: str) -> dict:
    return INCIDENTS.get(service.lower(), {"service": service, "status": "unknown", "incident_id": None, "customer_message": "This service is not present in the synthetic status system."})


def search_knowledge_base(query: str) -> dict:
    normalized = query.lower()
    matches = [article for article in KNOWLEDGE_BASE if any(keyword in normalized for keyword in article["keywords"])]
    return {"query": query, "matches": matches[:2]}


TOOLS = [
    {"type": "function", "name": "support_get_customer_context", "description": "Look up one synthetic AcmeSync customer before making a support decision.", "parameters": {"type": "object", "properties": {"customer_id": {"type": "string", "description": "Customer identifier from the ticket, for example C-102."}}, "required": ["customer_id"]}},
    {"type": "function", "name": "support_get_incident_status", "description": "Check the synthetic operational status for one named AcmeSync service.", "parameters": {"type": "object", "properties": {"service": {"type": "string", "description": "Service name from the ticket, such as sync, billing, or login."}}, "required": ["service"]}},
    {"type": "function", "name": "support_search_knowledge_base", "description": "Search concise support guidance after customer and service facts are known.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "A short problem description using the verified service and symptom."}}, "required": ["query"]}},
]

AVAILABLE_FUNCTIONS = {"support_get_customer_context": get_customer_context, "support_get_incident_status": get_incident_status, "support_search_knowledge_base": search_knowledge_base}
Wire the five-turn agent loop

The application owns the required workflow. It forces the customer lookup first. A detected service adds a required incident check before Gemini can choose an optional knowledge-base search.

  • Create a new file named agent_core.py from the Explorer sidebar.
  • Add the imports with the service detector and prompt builder by pasting this code:
import json
import time

from google import genai

from tools import AVAILABLE_FUNCTIONS, TOOLS

MODEL = "gemini-3.8-flash"
KNOWN_SERVICES = ("sync", "billing", "login")


def detect_service(ticket: str):
    normalized = ticket.lower()
    return next((service for service in KNOWN_SERVICES if service in normalized), None)


def build_prompt(ticket: str) -> str:
    return f"""You are the AcmeSync support triage agent.
Investigate the ticket with the supplied tools. Never invent customer, incident, or policy data.
The application will require the customer lookup first and a service check when a known service is named.
After those checks, use the knowledge-base tool only if it adds useful support guidance.
Finish with exactly these fields:
DECISION: RESPOND or ESCALATE
EVIDENCE: concise facts from tool results
RESPONSE: a customer-safe draft or a human-handoff note

Ticket: {ticket}
"""

What does this code do?

  • The imports bring in the SDK client. They also bring in the tool declarations and executable registry.
  • The detect_service function checks for sync, billing, or login in the ticket.
  • The build_prompt function defines the response format. It also tells the model to rely on supplied evidence.
  • Save agent_core.py.
  • Locate agent_core.py in the Explorer sidebar.

You should see both new Python files beside baseline.py. The agent now has a prompt boundary and a service detector.

Is an import marked as unresolved?

Check that tools.py sits in the same folder as agent_core.py.

Confirm that the import uses AVAILABLE_FUNCTIONS and TOOLS with the same capitalization as tools.py.

Help me diagnose the imports in agent_core.py.

A maximum turn count creates a failure boundary around the model. The loop also starts with a fallback answer so an incomplete run never produces an empty decision.

  • Place your cursor below the closing quotes of build_prompt.
  • Add the outer run_agent loop by pasting this code:
def run_agent(ticket: str, available_functions=None, max_turns: int = 5) -> dict:
    functions = available_functions or AVAILABLE_FUNCTIONS
    service = detect_service(ticket)
    forced_sequence = [["support_get_customer_context"]]
    if service:
        forced_sequence.append(["support_get_incident_status"])
    current_input = build_prompt(ticket)
    previous_id = None
    events = []
    answer = "DECISION: ESCALATE\nEVIDENCE: The agent did not complete.\nRESPONSE: A human should review this ticket."
    started = time.perf_counter()
    client = genai.Client()
    try:
        for turn in range(max_turns):
            request = {"model": MODEL, "input": current_input, "tools": TOOLS}
            if previous_id:
                request["previous_interaction_id"] = previous_id
            if turn < len(forced_sequence):
                request["generation_config"] = {"tool_choice": {"allowed_tools": {"mode": "any", "tools": forced_sequence[turn]}}}
            interaction = client.interactions.create(**request)
            function_results = []
    finally:
        client.close()
    latency_ms = round((time.perf_counter() - started) * 1000)
    return {"answer": answer, "events": events, "called_tools": [event["tool"] for event in events if event["type"] == "tool_call"], "latency_ms": latency_ms}

How is the loop bounded?

  • The max_turns value limits the loop to five model interactions by default.
  • The forced_sequence requires the customer lookup first. It adds an incident lookup when detect_service finds a known service.
  • The request sends the tool declarations with every interaction. The any mode requires the selected function call during each forced turn.
  • The finally block closes the client even when a request fails.
  • Save agent_core.py.
  • Find the return statement at the end of run_agent.

The returned dictionary should expose the answer. It should also expose recorded events, called tool names, and elapsed latency.

Does the loop indentation look uneven?

Check that for turn in range(max_turns): sits inside try.

Check that client.close() sits inside finally.

Help me repair the indentation in run_agent.

The model selects a function and supplies its arguments. Python executes the selected function. The result then returns to the same interaction as evidence.

  • Find the function_results = [] line inside the turn loop.
  • Insert the function-call handling directly below that line by pasting this code:
            for step in interaction.steps or []:
                if step.type != "function_call":
                    continue
                arguments = dict(step.arguments or {})
                events.append({"type": "tool_call", "tool": step.name, "arguments": arguments})
                print(f"Called {step.name} with {arguments}")
                try:
                    function = functions[step.name]
                    result = function(**arguments)
                except Exception as exc:
                    result = {"error": f"{type(exc).__name__}: {exc}"}
                events.append({"type": "tool_result", "tool": step.name, "result": result})
                function_results.append({"type": "function_result", "name": step.name, "call_id": step.id, "result": [{"type": "text", "text": json.dumps(result)}]})
            if not function_results:
                if interaction.output_text:
                    answer = interaction.output_text
                break
            current_input = function_results
            previous_id = interaction.id

How does a tool call become evidence?

  • The loop filters the interaction steps for function_call entries.
  • The selected tool name and arguments become a tool_call event. The same details print in the terminal.
  • Python executes the matching function from AVAILABLE_FUNCTIONS.
  • The returned record becomes a function_result for the next model turn. Exceptions become structured error results.
  • A turn without another function call captures the final text and ends the loop.
  • Save agent_core.py.
  • Confirm that current_input = function_results appears before the finally block.

The completed loop now feeds every tool result back into the active interaction. Each turn can build on evidence gathered during the previous turn.

Are tool events outside the loop?

Check that the inserted for step in interaction.steps or []: line aligns with function_results = [].

Check that the two events.append calls remain inside the inner step loop.

Help me place the function-call handling inside run_agent.

The final block supplies the same delayed-sync ticket used by the baseline. This makes the comparison depend on the agent design instead of a different customer scenario.

  • Place your cursor below the run_agent function.
  • Add the runnable sample by pasting this code:
if __name__ == "__main__":
    sample_ticket = "Customer C-102 says AcmeSync sync jobs have been delayed for 45 minutes. What should support tell them?"
    result = run_agent(sample_ticket)
    print("\nFinal answer:\n" + result["answer"])
    print("\nCalled tools:", result["called_tools"])

What does the sample run show?

The sample passes the synthetic C-102 sync ticket into run_agent. It prints the final answer after printing each executed tool call.

The last line prints the collected tool names. That list makes required tool use visible outside the model's prose.

  • Save agent_core.py.

✔️ Awesome, I've got everything!

Great. Double-check that you saved agent_core.py before running the grounded ticket.

ⓧ I'd like to double check the full code

import json
import time

from google import genai

from tools import AVAILABLE_FUNCTIONS, TOOLS

MODEL = "gemini-3.8-flash"
KNOWN_SERVICES = ("sync", "billing", "login")


def detect_service(ticket: str):
    normalized = ticket.lower()
    return next((service for service in KNOWN_SERVICES if service in normalized), None)


def build_prompt(ticket: str) -> str:
    return f"""You are the AcmeSync support triage agent.
Investigate the ticket with the supplied tools. Never invent customer, incident, or policy data.
The application will require the customer lookup first and a service check when a known service is named.
After those checks, use the knowledge-base tool only if it adds useful support guidance.
Finish with exactly these fields:
DECISION: RESPOND or ESCALATE
EVIDENCE: concise facts from tool results
RESPONSE: a customer-safe draft or a human-handoff note

Ticket: {ticket}
"""


def run_agent(ticket: str, available_functions=None, max_turns: int = 5) -> dict:
    functions = available_functions or AVAILABLE_FUNCTIONS
    service = detect_service(ticket)
    forced_sequence = [["support_get_customer_context"]]
    if service:
        forced_sequence.append(["support_get_incident_status"])
    current_input = build_prompt(ticket)
    previous_id = None
    events = []
    answer = "DECISION: ESCALATE\nEVIDENCE: The agent did not complete.\nRESPONSE: A human should review this ticket."
    started = time.perf_counter()
    client = genai.Client()
    try:
        for turn in range(max_turns):
            request = {"model": MODEL, "input": current_input, "tools": TOOLS}
            if previous_id:
                request["previous_interaction_id"] = previous_id
            if turn < len(forced_sequence):
                request["generation_config"] = {"tool_choice": {"allowed_tools": {"mode": "any", "tools": forced_sequence[turn]}}}
            interaction = client.interactions.create(**request)
            function_results = []
            for step in interaction.steps or []:
                if step.type != "function_call":
                    continue
                arguments = dict(step.arguments or {})
                events.append({"type": "tool_call", "tool": step.name, "arguments": arguments})
                print(f"Called {step.name} with {arguments}")
                try:
                    function = functions[step.name]
                    result = function(**arguments)
                except Exception as exc:
                    result = {"error": f"{type(exc).__name__}: {exc}"}
                events.append({"type": "tool_result", "tool": step.name, "result": result})
                function_results.append({"type": "function_result", "name": step.name, "call_id": step.id, "result": [{"type": "text", "text": json.dumps(result)}]})
            if not function_results:
                if interaction.output_text:
                    answer = interaction.output_text
                break
            current_input = function_results
            previous_id = interaction.id
    finally:
        client.close()
    latency_ms = round((time.perf_counter() - started) * 1000)
    return {"answer": answer, "events": events, "called_tools": [event["tool"] for event in events if event["type"] == "tool_call"], "latency_ms": latency_ms}


if __name__ == "__main__":
    sample_ticket = "Customer C-102 says AcmeSync sync jobs have been delayed for 45 minutes. What should support tell them?"
    result = run_agent(sample_ticket)
    print("\nFinal answer:\n" + result["answer"])
    print("\nCalled tools:", result["called_tools"])
Run the grounded comparison

The baseline answered before checking any operational system. This run should expose the evidence-gathering path before it prints the final support response.

Before you run it, which required tool do you expect to appear first?

  • Run the bounded agent from the active terminal with this command:
python agent_core.py

What should I see?

You should first see a call to support_get_customer_context for customer C-102. You should then see a call to support_get_incident_status for the sync service.

The final response should use the verified Enterprise account and degraded sync incident. The called-tools list should include both required tool names.

That is the evidence boundary working. Your support answer now follows verified customer and incident checks instead of relying on fluent guesswork.

  • Compare the grounded answer with the earlier output from baseline.py.
  • Identify the customer fact used by the grounded answer.
  • Identify the incident fact used by the grounded answer.

Did the grounded run fail?

Confirm that your terminal still shows the active .venv environment. Confirm that the current session still has access to GEMINI_API_KEY.

If the script reports a Python syntax problem, compare both files with the full-code tabs. Focus on indentation inside the nested loops.

Help me debug my bounded tool agent run.

Your agent now gathers bounded operational evidence before it answers. Next, you will place deterministic escalation rules after the model and trace every decision.

Add the Deterministic Safety Gate

Your agent loop now gathers customer evidence before drafting a support response. Those checks make the response more grounded.

Grounding does not guarantee safe behavior when identity or dependencies remain uncertain. A deterministic policy gate will own those high-risk decisions. A local JSONL trace will preserve the evidence behind every override.

In this step, get ready to:
  • Record timestamped events in an append-only trace.
  • Enforce deterministic escalation after the model loop.
  • Test an unknown customer against the completed safety gate.
Record trace events

Each trace record captures one event as a separate line. This makes the execution history readable even when a later event fails.

  • Create safe_agent.py beside agent_core.py from the VS Code file sidebar.
  • Add the trace foundation by copying this code into safe_agent.py:
import json
from datetime import datetime, timezone
from pathlib import Path

from agent_core import run_agent

TRACE_PATH = Path("traces.jsonl")
SECURITY_TERMS = ("password", "credential", "breach", "data leak")


def write_trace(event: dict) -> None:
    record = {"timestamp": datetime.now(timezone.utc).isoformat(), **event}
    with TRACE_PATH.open("a", encoding="utf-8") as trace_file:
        trace_file.write(json.dumps(record) + "\n")

What does this code do?

  • The TRACE_PATH constant keeps every event in traces.jsonl.
  • The SECURITY_TERMS tuple defines wording that the policy gate treats as sensitive.
  • The write_trace() function adds a UTC timestamp before appending one JSON record.
  • Save safe_agent.py.
  • Confirm safe_agent.py appears beside agent_core.py in the file sidebar.

Your new file now has a reusable trace writer plus the policy terms needed by the next function.

File missing from the sidebar?

  • Check that you created safe_agent.py inside the same fde-triage-agent folder as agent_core.py.
  • Check that the filename ends with .py.

Ask for help with the file location or trace writer.

Enforce policy after the model

The policy gate examines recorded tool results after the flexible model loop finishes. Deterministic application code can then override the draft when a known safety boundary is crossed.

  • Place your cursor below write_trace() in safe_agent.py.
  • Add the policy function by copying this code:
def apply_policy(ticket: str, result: dict):
    tool_results = [event for event in result["events"] if event["type"] == "tool_result"]
    failures = [event for event in tool_results if isinstance(event["result"], dict) and event["result"].get("error")]
    customer_result = next((event["result"] for event in tool_results if event["tool"] == "support_get_customer_context"), {"found": False})
    if failures:
        return ("DECISION: ESCALATE\nEVIDENCE: A required support dependency failed.\nRESPONSE: A human support engineer should verify system status before replying.", "Tool dependency failed.")
    if not customer_result.get("found"):
        return ("DECISION: ESCALATE\nEVIDENCE: The customer identity could not be verified.\nRESPONSE: A human support engineer should verify the account before replying.", "Unknown customer.")
    if any(term in ticket.lower() for term in SECURITY_TERMS):
        return ("DECISION: ESCALATE\nEVIDENCE: The ticket contains a security-sensitive term.\nRESPONSE: Route this ticket to the security response queue without requesting secrets.", "Security-sensitive ticket.")
    if result["answer"].startswith("DECISION: ESCALATE\nEVIDENCE: The agent did not complete"):
        return result["answer"], "Agent loop incomplete."
    return result["answer"], None

How does the gate decide?

  • A tool error produces a dependency-failure escalation.
  • A missing customer record produces an identity-verification escalation.
  • A security term produces a security-queue escalation.
  • An unfinished agent loop preserves its safe fallback response.
  • Save safe_agent.py.
  • Review the four if blocks inside apply_policy().

The function ends by returning the model answer unchanged only when no policy condition applies.

Policy function showing an error?

  • Check that every dictionary key retains its matching quotation marks.
  • Check that each escalation branch returns both the final answer and its policy reason.
  • Check that SECURITY_TERMS matches the constant near the top of the file.

Ask for help finding the mismatch in the policy function.

The wrapper now needs to connect the existing agent loop to the trace writer and policy gate. It records the original tool evidence before writing the final policy outcome.

  • Place your cursor below apply_policy().
  • Complete safe_agent.py by copying this code:
def run_triage(ticket: str, available_functions=None) -> dict:
    write_trace({"type": "run_started", "ticket": ticket})
    result = run_agent(ticket, available_functions=available_functions)
    for event in result["events"]:
        write_trace(event)
    final_answer, policy_override = apply_policy(ticket, result)
    final_result = {**result, "answer": final_answer, "policy_override": policy_override}
    write_trace({"type": "run_completed", "latency_ms": result["latency_ms"], "called_tools": result["called_tools"], "policy_override": policy_override, "answer": final_answer})
    return final_result


if __name__ == "__main__":
    sample_ticket = "Customer C-999 says login is unavailable and asks for an immediate fix."
    result = run_triage(sample_ticket)
    print("\nFinal answer:\n" + result["answer"])
    print("\nPolicy override:", result["policy_override"])
    print("Trace file:", TRACE_PATH)

What does the wrapper control?

  • The run_started event records the incoming synthetic ticket.
  • The loop writes every tool call and tool result produced by run_agent().
  • The run_completed event records latency plus the final policy outcome.
  • The built-in sample uses unknown customer C-999 to exercise the policy boundary.
  • Save safe_agent.py.
  • Scroll to the bottom of safe_agent.py.

You should see the C-999 sample ticket followed by three print calls.

Wrapper code in the wrong place?

  • Check that run_triage() begins after apply_policy() finishes.
  • Check that the sample-ticket block starts at the left edge of the file.
  • Check that the statements inside run_triage() use consistent indentation.

Ask for help correcting the function order or indentation.

✔️ Awesome, I've got everything!

  • Save safe_agent.py before running the safety check.

ⓧ I'd like to double check the full code

import json
from datetime import datetime, timezone
from pathlib import Path

from agent_core import run_agent

TRACE_PATH = Path("traces.jsonl")
SECURITY_TERMS = ("password", "credential", "breach", "data leak")


def write_trace(event: dict) -> None:
    record = {"timestamp": datetime.now(timezone.utc).isoformat(), **event}
    with TRACE_PATH.open("a", encoding="utf-8") as trace_file:
        trace_file.write(json.dumps(record) + "\n")


def apply_policy(ticket: str, result: dict):
    tool_results = [event for event in result["events"] if event["type"] == "tool_result"]
    failures = [event for event in tool_results if isinstance(event["result"], dict) and event["result"].get("error")]
    customer_result = next((event["result"] for event in tool_results if event["tool"] == "support_get_customer_context"), {"found": False})
    if failures:
        return ("DECISION: ESCALATE\nEVIDENCE: A required support dependency failed.\nRESPONSE: A human support engineer should verify system status before replying.", "Tool dependency failed.")
    if not customer_result.get("found"):
        return ("DECISION: ESCALATE\nEVIDENCE: The customer identity could not be verified.\nRESPONSE: A human support engineer should verify the account before replying.", "Unknown customer.")
    if any(term in ticket.lower() for term in SECURITY_TERMS):
        return ("DECISION: ESCALATE\nEVIDENCE: The ticket contains a security-sensitive term.\nRESPONSE: Route this ticket to the security response queue without requesting secrets.", "Security-sensitive ticket.")
    if result["answer"].startswith("DECISION: ESCALATE\nEVIDENCE: The agent did not complete"):
        return result["answer"], "Agent loop incomplete."
    return result["answer"], None


def run_triage(ticket: str, available_functions=None) -> dict:
    write_trace({"type": "run_started", "ticket": ticket})
    result = run_agent(ticket, available_functions=available_functions)
    for event in result["events"]:
        write_trace(event)
    final_answer, policy_override = apply_policy(ticket, result)
    final_result = {**result, "answer": final_answer, "policy_override": policy_override}
    write_trace({"type": "run_completed", "latency_ms": result["latency_ms"], "called_tools": result["called_tools"], "policy_override": policy_override, "answer": final_answer})
    return final_result


if __name__ == "__main__":
    sample_ticket = "Customer C-999 says login is unavailable and asks for an immediate fix."
    result = run_triage(sample_ticket)
    print("\nFinal answer:\n" + result["answer"])
    print("\nPolicy override:", result["policy_override"])
    print("Trace file:", TRACE_PATH)
Verify the override and trace

Before you run this check, consider whether the model or the policy gate will own the final decision.

  • Run the unknown-customer ticket from your active terminal with this command:
python safe_agent.py

What should you see?

The agent still calls support_get_customer_context plus support_get_incident_status to gather evidence.

The final output contains DECISION: ESCALATE. It also reports Policy override: Unknown customer. plus the traces.jsonl path.

Run failed before the override?

  • Check that your terminal still uses the active virtual environment from setup.
  • Check that the current shell still has access to your Gemini API key without displaying its value.
  • Check that safe_agent.py imports run_agent from the existing agent_core.py file.

Ask for help diagnosing the failed safety-gate run.

  • Return to the VS Code file sidebar.
  • Select traces.jsonl.
  • Scroll to the latest records at the bottom of the file.

What proves the run is observable?

  • The latest sequence begins with a run_started record containing the synthetic ticket.
  • The middle records preserve each tool_call plus its corresponding tool_result.
  • The final run_completed record contains latency and called tools.
  • The final record stores Unknown customer. as the policy override.

That is the safety boundary working. Your agent now gathers evidence while deterministic code controls unsafe outcomes.

Next, you will turn this behavior into repeatable evaluations and document the architecture choices behind it.

Evaluate and Document the Design

Your agent now places a deterministic safety gate after the model. Its JSONL traces expose every tool result plus policy decision.

A demo can look convincing by chance. This step turns the demo into repeatable evidence through evaluations plus a written architecture record.

In this step, get ready to:
  • Create a behavioral evaluation suite covering mandatory evidence plus security escalation.
  • Verify both cases through terminal output plus trace evidence.
  • Document the architecture boundaries, failure modes, success measures, plus core trade-off.
Build the behavioral evaluations

A behavioral evaluation checks observable system contracts. Each case verifies required tool use plus the policy outcome without depending on exact model wording.

  • Return to the Explorer view in the Activity Bar of VS Code.
  • Select New File... in the Explorer toolbar.

The Explorer opens a field for the new file name.

  • Enter evals.py inside the fde-triage-agent folder.
  • Press Enter to create the file.

The empty evals.py file opens in the editor.

  • Add the evaluation cases plus runner by pasting this code into evals.py:
from safe_agent import run_triage

CASES = [
    {"name": "known customer uses mandatory evidence", "ticket": "Customer C-102 reports delayed sync jobs in AcmeSync.", "required_tools": {"support_get_customer_context", "support_get_incident_status"}, "expected_override": None},
    {"name": "security-sensitive ticket escalates", "ticket": "Customer C-101 says a password may have leaked during login.", "required_tools": {"support_get_customer_context", "support_get_incident_status"}, "expected_override": "Security-sensitive ticket."},
]


def run_evaluations() -> None:
    passed = 0
    for case in CASES:
        result = run_triage(case["ticket"])
        tools_ok = case["required_tools"].issubset(set(result["called_tools"]))
        policy_ok = result["policy_override"] == case["expected_override"]
        case_passed = tools_ok and policy_ok
        status = "PASS" if case_passed else "FAIL"
        print(f"{status}: {case['name']} | tools={result['called_tools']} | policy={result['policy_override']} | latency_ms={result['latency_ms']}")
        passed += int(case_passed)
    print(f"\n{passed}/{len(CASES)} evaluations passed")


if __name__ == "__main__":
    run_evaluations()

What does this evaluation test?

  • The first case requires the customer lookup plus incident check for a known sync ticket.
  • The second case requires the same evidence before checking for the Security-sensitive ticket. override.
  • The required_tools set checks the observable tool contract for each ticket.
  • The run_evaluations() function counts a case only when its tool contract plus policy contract pass.
  • Save evals.py.
  • Confirm evals.py now appears beside safe_agent.py in the Explorer.

Does the evaluation file look incomplete?

Check that both case dictionaries sit inside CASES. Confirm the final two lines call run_evaluations() only when the file runs directly.

Use the full-code comparison below to catch missing quotes or brackets.

Help me compare my evaluation file with the expected structure.

✔️ Awesome, I've got everything!

Great. Save evals.py before running the suite.

ⓧ I'd like to double check the full code

from safe_agent import run_triage

CASES = [
    {"name": "known customer uses mandatory evidence", "ticket": "Customer C-102 reports delayed sync jobs in AcmeSync.", "required_tools": {"support_get_customer_context", "support_get_incident_status"}, "expected_override": None},
    {"name": "security-sensitive ticket escalates", "ticket": "Customer C-101 says a password may have leaked during login.", "required_tools": {"support_get_customer_context", "support_get_incident_status"}, "expected_override": "Security-sensitive ticket."},
]


def run_evaluations() -> None:
    passed = 0
    for case in CASES:
        result = run_triage(case["ticket"])
        tools_ok = case["required_tools"].issubset(set(result["called_tools"]))
        policy_ok = result["policy_override"] == case["expected_override"]
        case_passed = tools_ok and policy_ok
        status = "PASS" if case_passed else "FAIL"
        print(f"{status}: {case['name']} | tools={result['called_tools']} | policy={result['policy_override']} | latency_ms={result['latency_ms']}")
        passed += int(case_passed)
    print(f"\n{passed}/{len(CASES)} evaluations passed")


if __name__ == "__main__":
    run_evaluations()

Before you run the suite, do you expect both cases to satisfy their tool contract plus policy contract?

  • Run the behavioral evaluations in the terminal with this command:
python evals.py

What should I see?

You'll see one PASS line for the known customer case. You'll see another PASS line for the security-sensitive case.

The final summary reads 2/2 evaluations passed. Each line also exposes called tools, policy outcome, plus latency.

Seeing a failed evaluation?

Use the trace before changing any prompt. A FAIL line already shows whether the missing contract involves tools or policy.

Open traces.jsonl to inspect the matching tool results plus final override.

Help me diagnose a failed behavioral evaluation from its output and JSONL trace.

  • Return to the Explorer view.
  • Select traces.jsonl.

The trace now contains records from both evaluation cases.

  • Scroll to the latest two run_completed records.

Each completed run records its called tools, latency, policy override, plus final answer. That evidence connects every evaluation result to an inspectable execution path.

Document the architecture decisions

An architecture brief preserves the reasoning behind the implementation. It gives an interviewer a compact record of system boundaries, safe failure behavior, measurable contracts, plus the model-autonomy trade-off.

  • Return to the Explorer view in the Activity Bar.
  • Select New File... in the Explorer toolbar.

The Explorer opens another field for a file name.

  • Enter DESIGN.md inside the fde-triage-agent folder.
  • Press Enter to create the file.

The empty DESIGN.md file opens in the editor.

  • Start the brief with the customer problem by pasting this content into DESIGN.md:
# AcmeSync Support Triage Agent

## Customer problem

Support engineers need a fast draft without allowing a language model to invent account state, incident status, or security decisions.

Why start with the customer problem?

The opening frames the design around a support outcome. Every later component exists to prevent invented operational claims while preserving a fast draft.

  • Save DESIGN.md.
  • Press Cmd+Shift+V (macOS) or Ctrl+Shift+V (Windows) to open the Markdown preview.

The preview shows the AcmeSync title above a Customer problem section.

Missing the title in the preview?

Confirm the file is named DESIGN.md. Check that the first line begins with one # character.

Help me fix the opening of my architecture brief.

The architecture diagram shows where flexible model behavior ends. It also shows where deterministic policy ownership begins.

  • Select the DESIGN.md editor tab to return to the source.
  • Append the architecture section by pasting this content below the customer problem:
## Architecture

```text
Synthetic ticket
      |
      v
Hybrid agent loop
  | required customer check
  | required service check when applicable
  | optional model-selected KB search
      |
      v
Deterministic policy gate
  | unknown customer
  | security-sensitive request
  | tool dependency failure
  | incomplete loop
      |
      +----> Human escalation
      |
      +----> Customer-safe draft

Every run and tool event ----> traces.jsonl
Evaluation cases -----------> acceptance report
```

What does the flow show?

The hybrid loop gathers evidence before drafting. Required checks stay under application control.

The policy gate owns escalation after the loop. Traces plus evaluations expose whether both layers met their contracts.

  • Save DESIGN.md.
  • Press Cmd+Shift+V (macOS) or Ctrl+Shift+V (Windows) to refresh the preview.

The preview now shows a vertical path from the synthetic ticket to either human escalation or a customer-safe draft.

Does the diagram lose its spacing?

Confirm the diagram begins with ```text and ends with a closing code fence. Those fences preserve the diagram's fixed-width alignment.

Help me repair the architecture diagram in my Markdown file.

The remaining sections turn the diagram into an engineering decision record. They name ownership, expected failures, measurable success, plus the cost of limiting model autonomy.

  • Select the DESIGN.md editor tab to return to the source.
  • Complete the brief by pasting this content below the architecture diagram:
## Component boundaries

- Gemini selects optional information-gathering actions and drafts natural language.
- Python controls required checks, maximum turns, tool execution, errors, and escalation.
- Tools return small, relevant records instead of exposing raw databases.
- The trace is append-only local JSONL for this prototype.

## Designed failure modes

| Failure | Safe behavior | Evidence |
| --- | --- | --- |
| Unknown customer | Escalate | Customer tool returns `found: false` |
| Security-sensitive wording | Escalate | Deterministic keyword policy |
| Tool exception | Escalate | Error stored in tool-result trace |
| Agent exceeds bounded loop | Escalate | Incomplete-loop fallback |
| Model skips a required check | Prevented | Application forces required tool choice |

## Success measures

- Mandatory customer checks occur on every run.
- Named services receive an incident-status check.
- Unknown customers, security-sensitive tickets, and dependency failures always escalate.
- Every run records tool calls, outcomes, latency, and policy overrides.

## Trade-off

The design gives up some model autonomy to gain predictable checks and safer failure behavior. The model still adds value where flexible language and optional knowledge retrieval are useful. In production, the synthetic dictionaries would become authenticated service adapters, JSONL would become centralized telemetry, and the evaluation set would expand with real but properly governed cases.

How does the decision record help?

  • The component boundaries assign optional retrieval plus drafting to Gemini.
  • The failure table links each risk to a safe response plus observable evidence.
  • The success measures translate the architecture into acceptance criteria.
  • The trade-off explains why predictable checks justify reduced model autonomy.
  • Save DESIGN.md.
  • Press Cmd+Shift+V (macOS) or Ctrl+Shift+V (Windows) to refresh the preview.

The preview now shows six sections. The designed failure modes appear as a table.

Is the failure table misaligned?

Check that every table row starts plus ends with a pipe character. Confirm the separator row contains the three dashed columns.

Help me fix the failure-mode table in my architecture brief.

✔️ Awesome, I've got everything!

Your architecture brief now captures the complete system-design argument.

ⓧ I'd like to double check the full code

# AcmeSync Support Triage Agent

## Customer problem

Support engineers need a fast draft without allowing a language model to invent account state, incident status, or security decisions.

## Architecture

```text
Synthetic ticket
      |
      v
Hybrid agent loop
  | required customer check
  | required service check when applicable
  | optional model-selected KB search
      |
      v
Deterministic policy gate
  | unknown customer
  | security-sensitive request
  | tool dependency failure
  | incomplete loop
      |
      +----> Human escalation
      |
      +----> Customer-safe draft

Every run and tool event ----> traces.jsonl
Evaluation cases -----------> acceptance report
```

## Component boundaries

- Gemini selects optional information-gathering actions and drafts natural language.
- Python controls required checks, maximum turns, tool execution, errors, and escalation.
- Tools return small, relevant records instead of exposing raw databases.
- The trace is append-only local JSONL for this prototype.

## Designed failure modes

| Failure | Safe behavior | Evidence |
| --- | --- | --- |
| Unknown customer | Escalate | Customer tool returns `found: false` |
| Security-sensitive wording | Escalate | Deterministic keyword policy |
| Tool exception | Escalate | Error stored in tool-result trace |
| Agent exceeds bounded loop | Escalate | Incomplete-loop fallback |
| Model skips a required check | Prevented | Application forces required tool choice |

## Success measures

- Mandatory customer checks occur on every run.
- Named services receive an incident-status check.
- Unknown customers, security-sensitive tickets, and dependency failures always escalate.
- Every run records tool calls, outcomes, latency, and policy overrides.

## Trade-off

The design gives up some model autonomy to gain predictable checks and safer failure behavior. The model still adds value where flexible language and optional knowledge retrieval are useful. In production, the synthetic dictionaries would become authenticated service adapters, JSONL would become centralized telemetry, and the evaluation set would expand with real but properly governed cases.

Before the final run, do you expect the same two contracts to pass after documenting the design?

  • Run the final evaluation check with this command:
python evals.py

What should the final check prove?

You'll see two lines beginning with PASS. The summary reads 2/2 evaluations passed.

The known-customer line shows both mandatory tools with no policy override. The security-sensitive line shows both mandatory tools plus the expected override.

Did the final check stop passing?

Compare evals.py with the full-code tab. Confirm the active terminal still has access to your API key.

Inspect the latest tool_result plus run_completed records before editing any prompt.

Help me trace why my final evaluation check failed.

  • Select traces.jsonl in the Explorer.
  • Scroll to the final two run_completed records.

Those records prove that the final evaluation created a trace for each case. Your terminal output proves that both behavioral contracts passed.

That completes the evidence package. You now have passing evaluations, inspectable traces, plus an architecture brief that explains the design choices behind the system.

Secret mission

Survive an Incident-Service Failure

A healthy demo only proves the happy path. Inject an incident-service outage to prove that your agent preserves the error in its trace and hands the ticket to a human.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep your project available, pause your work, or remove its resources. Keeping the local workspace and your Gemini Developer API Free Tier access has no ongoing charge.

Resources you used:

  • The local fde-triage-agent folder containing your source files, .venv, and traces.jsonl.
  • The Gemini API authorization key in Google AI Studio.

Keep everything running

No action is needed. Choose this option if you want to test more synthetic tickets or expand the behavioral evaluations.

  • Keep the fde-triage-agent folder available for continued practice.
  • Preserve traces.jsonl so you can review earlier tool calls, failures, latency, and policy overrides.
  • Leave the Gemini API authorization key active for future project runs.
  • Continue sending synthetic project data only through the Free Tier.

Pause - I'll come back to this later

Pause your terminal work while keeping the project ready for another session. Your files and Gemini API authorization key remain available.

  • Close the current terminal session in Visual Studio Code.
  • Keep the fde-triage-agent folder in its current location.
  • Leave the Gemini API authorization key active in Google AI Studio.

What Happens to the Environment Variable?

Closing the terminal ends the session that contains GEMINI_API_KEY. The authorization key itself stays active in Google AI Studio.

A new terminal session needs the key exported again before the scripts can call Gemini.

Delete - I don't want to use this again

This option disables the project credential and removes the complete local workspace. It gives you a clean reset when you no longer need the project.

Deletion Is Permanent

Revoking the Gemini API authorization key prevents future requests from using that credential. Deleting fde-triage-agent erases the source code and trace evidence stored on your Mac.

Save any files you want to keep before continuing.

  • Return to the Google AI Studio API Keys page from earlier.
  • Identify the Gemini API authorization key created for this project.
  • Revoke the project key.
  • Confirm the API Keys page no longer shows the key as active.

That credential can no longer authorize Gemini requests for this project.

  • Close Visual Studio Code before removing the local workspace.
  • Return to Finder from earlier.
  • Locate the fde-triage-agent folder.
  • Control-click the fde-triage-agent folder.
  • Choose the folder-removal option from the context menu.
  • Confirm Finder no longer lists fde-triage-agent.

Your Mac no longer holds the project source files, .venv, or traces.jsonl.

Still See the Project Folder?

Check that you selected the complete fde-triage-agent folder. Removing only .venv or traces.jsonl leaves the source files behind.

Ask for help before repeating the deletion if you are unsure which folder contains the project. Help me verify that I found the correct fde-triage-agent folder before deleting it.

Nice Work!

Nice Work!

You did it! You've built an observable support triage agent for AcmeSync that uses bounded tool calling to gather evidence before answering. Gemini drafts each response while deterministic policies protect risky cases.

You've learned how to:

  • Compare an ungrounded baseline with an evidence-backed agent. You can now spot unsupported operational claims before they reach a customer.
  • Implement a bounded agent loop that forces customer checks plus relevant service checks. You used small tool contracts for focused evidence. You kept risky escalation decisions inside a deterministic policy gate.
  • Record each run in traces.jsonl for inspection. You ran behavioral evaluations that report clear outcomes. You documented the system boundaries plus production trade-offs in DESIGN.md.
  • Secret Mission: Inject an incident-service outage through failure_drill.py. The run proves safe escalation. The latest trace preserves the dependency error.

Ready to quiz yourself?