Build a Reliable AI Incident Commander

Build a safe Google ADK agent that triages mock service incidents.

Introduction

30 Second Summary

When an online store slows down, the first report rarely tells you what failed. The pressure to act quickly can turn missing evidence into a risky guess.

In this project, you will build a local AI incident commander for a mock online store with Google's Agent Development Kit (ADK). It will produce evidence-based triage reports that stay useful when telemetry fails.

What You'll Build

Picture reporting a payments outage in ADK Web and watching a failed health check become a controlled diagnosis that points to a safe runbook.

By the end of this project, you'll have:

  • An evidence-based triage flow that reports mock checkout health. The same flow turns missing payments telemetry into a controlled degraded diagnosis.
  • Trace-backed guardrails that block risky recovery proposals before execution. A separate model guard rewrites secret-seeking requests before the model call.
  • A saved trajectory evaluation that fails when the runbook tool disappears. It passes after the required tool path is restored.
  • Secret Mission: Add an after_tool_callback that records successful tool-call counts in session state.

Are there any prerequisites?

You need a Mac with Python 3.10 or newer plus internet access.

Visual Studio Code, a Gemini API key, and confidence reading Python complete the prerequisites.

Before We Start

Your demo will focus on a mock checkout or payments incident. Unsafe recovery actions must stay blocked because the agent must never change real infrastructure.

Set Up a Version-Pinned ADK Workspace

Your incident commander needs a predictable runtime before it can make a safe tool call. Agent Development Kit (ADK) version 2.12.0 requires Python 3.11 or newer.

This step creates an isolated Python workspace for the pinned ADK release. It also stores your Gemini API credential locally for later agent runs.

In this step, get ready to:
  • Confirm that your Python version is compatible.
  • Create an isolated project workspace.
  • Install the pinned ADK dependency with local Gemini API access.
Confirm a compatible Python version

The interpreter version determines whether ADK can run in this project. Your Mac needs Python 3.11 or newer before you create the workspace.

  • Press Cmd+Space to open macOS search.
  • Type Terminal into the search bar.
  • Press Enter to open Terminal.
  • Check your installed Python version by running this command:
python3 --version

What does this command do?

The command prints the version used by the python3 executable. That result decides whether you can continue with the installed interpreter.

Choose the tab that matches the version result in Terminal.

✔️ I see a compatible version

Your Python interpreter meets the ADK requirement. That compatibility check clears the first setup hurdle.

ⓧ I see an older version

Python upgrades can be fiddly because an open Terminal may keep using the previous interpreter. Installing Python 3.15.0 fixes the version gap.

  • Visit the Python.org macOS downloads page.
  • Select Download macOS installer for Python 3.15.0.
  • Follow the installer prompts using the default options.
  • Close the existing Terminal window.
  • Press Cmd+Space to reopen macOS search.
  • Type Terminal into the search bar.
  • Press Enter to reopen Terminal.
  • Check the upgraded interpreter by running this command:
python3 --version

What does this command confirm?

The reopened Terminal resolves the newly installed Python interpreter. The printed version should now be 3.15.0.

ⓧ Command not found

Your shell cannot find a Python 3 executable yet. The current macOS installer adds the compatible interpreter you need.

  • Visit the Python.org macOS downloads page.
  • Select Download macOS installer for Python 3.15.0.
  • Follow the installer prompts using the default options.
  • Close the existing Terminal window.
  • Press Cmd+Space to reopen macOS search.
  • Type Terminal into the search bar.
  • Press Enter to reopen Terminal.
  • Confirm that Python is now available by running this command:
python3 --version

What does this command confirm?

The command now prints the installed Python version. A result of 3.15.0 satisfies the project requirement.

Create and isolate the project workspace

A virtual environment gives this project its own Python packages. This keeps the pinned ADK release separate from packages used by other projects.

  • Return to Terminal from the version check.
  • Move to your Desktop by running this command:
cd ~/Desktop

What does this command do?

The cd command changes Terminal's current location. New project files now land on your Desktop.

  • Create the project folder by running these commands:
mkdir adk-incident-commander
cd adk-incident-commander

What do these commands do?

  • The first command creates adk-incident-commander on your Desktop.
  • The second command makes that folder Terminal's current location.

Your Terminal prompt now shows that you are inside adk-incident-commander.

  • Create the isolated environment inside the project folder by running this command:
python3 -m venv .venv

What does this command do?

Python creates an isolated environment inside .venv. Packages installed after activation belong to this project.

  • Activate the new environment in the current Terminal session by running this command:
source .venv/bin/activate

What does this command do?

Activation updates the current Terminal session to use the Python tools inside .venv.

You will see the environment name at the start of the Terminal prompt. Your project now has an active package boundary.

Environment not activating?

Confirm that Terminal is inside adk-incident-commander. The activation path only works after .venv has been created there.

Check that the environment creation command completed before trying activation again. Help me troubleshoot why my macOS Python virtual environment is not activating.

Visual Studio Code gives you an Explorer sidebar for creating the two configuration files in their exact locations.

  • Press Cmd+Space to open macOS search.
  • Type Visual Studio Code into the search bar.
  • Press Enter to open Visual Studio Code.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select adk-incident-commander from your Desktop.
  • Click Open.

The Explorer sidebar now shows adk-incident-commander with its .venv folder.

Install ADK and store the API key

The requirements.txt file records the exact ADK release used by this project. Installing from that file makes the dependency version repeatable.

  • Click the New File icon in the Explorer toolbar.
  • Enter requirements.txt as the file name.

A blank requirements.txt file now appears in the Explorer sidebar.

  • Pin Google ADK in requirements.txt by pasting this configuration:
google-adk==2.12.0

What does this configuration do?

The version specifier tells pip to install Google ADK 2.12.0. The double equals sign prevents an unplanned upgrade to another release.

  • Save requirements.txt.
  • Switch back to the activated Terminal from earlier.
  • Install the pinned dependency from requirements.txt by running this command:
python -m pip install -r requirements.txt

What does this command do?

The active Python interpreter asks pip to read requirements.txt. Pip installs the pinned ADK package into .venv.

The terminal returns to its prompt after the ADK dependencies finish installing.

Installation not completing?

Confirm that the Terminal prompt still shows the active environment. Check that requirements.txt contains the pinned line exactly.

Help me diagnose this ADK installation problem. You can also ask the ADK Python community.

✔️ Awesome, I've got everything!

Great. Double-check that you saved requirements.txt.

ⓧ I'd like to double check the full code

google-adk==2.12.0

The agent reads its Gemini credential from a local .env file. Keeping that value outside agent.py prevents the credential from becoming part of the Python source.

  • Switch back to the Visual Studio Code window from earlier.
  • Click the New Folder icon in the Explorer toolbar.
  • Enter incident_commander as the folder name.

The incident_commander folder now appears beside requirements.txt in the Explorer sidebar.

  • Select the incident_commander folder in the Explorer sidebar.
  • Click the New File icon in the Explorer toolbar.
  • Enter .env as the file name.

A blank .env file now appears inside incident_commander.

  • Add the Gemini API key setting to .env by pasting this configuration:
GOOGLE_API_KEY="your-api-key-here"

What does this configuration do?

The GOOGLE_API_KEY setting gives ADK access to your Gemini API credential. The starter value marks the exact text you need to replace locally.

Pasting a credential can feel risky. This value stays in your local .env file during this project.

  • Replace your-api-key-here with your Gemini API key from earlier.
  • Save incident_commander/.env.
  • Close the .env editor tab before taking any screenshots.

The Explorer sidebar lists .env inside incident_commander. Your secret value is no longer visible in the editor.

Can't find the environment file?

Confirm that .env sits inside incident_commander. A file beside requirements.txt is in the wrong location.

Check that the file name begins with a dot. Help me verify the location and format of my local ADK environment file.

✔️ Awesome, I've got everything!

Your local .env file is saved with your own credential value.

ⓧ I'd like to double check the full code

This reference keeps the placeholder so your real credential never appears here. Your local file should contain your own value between the quotes.

GOOGLE_API_KEY="your-api-key-here"

The final package check confirms that the active environment contains the pinned release. Before you run it, what version do you expect the package metadata to report?

  • Return to the activated Terminal from earlier.
  • Verify the installed ADK package by running this command:
python -m pip show google-adk

What does this command confirm?

Pip prints metadata for the installed google-adk package. The version field proves which release is active in this environment.

You will see Version: 2.12.0 in the output. That is the pinned ADK workspace ready for your incident agent.

Package metadata missing?

Confirm that the Terminal prompt still shows the active environment. Run the requirements installation again if the package check returns no metadata.

Help me diagnose why pip cannot show the installed ADK package.

Your version-pinned workspace is ready. Next, you will turn it into a running incident triage agent.

Build the Naive Incident Triage Agent

Your version-pinned Agent Development Kit (ADK) workspace is ready. Now you can give it a working incident workflow.

A single LLM agent will call one function tool through Gemini. You will test a normal result before unavailable telemetry exposes the workflow's limit.

In this step, get ready to:
  • Define mock service evidence through one function tool.
  • Launch the incident commander in ADK Web.
  • Compare the checkout diagnosis with the planned payments failure.
Create the one-tool agent

A Python package gives ADK a module it can load. Visual Studio Code keeps the package files inside your existing project.

Why ADK instead of a custom loop?

ADK supplies the agent lifecycle used throughout this project. Its local Web interface records model requests plus tool activity for later inspection.

A custom loop would require you to build those development controls yourself. ADK keeps this project focused on agent behavior.

  • Press Cmd+Space to open Spotlight.
  • Type Visual Studio Code into Spotlight.
  • Press Enter to open Visual Studio Code.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the existing adk-incident-commander folder.
  • Click Open.

What should you see?

The Explorer sidebar shows requirements.txt beside the incident_commander folder. That confirms you opened the project created in the previous step.

  • Select the incident_commander folder in the Explorer sidebar.
  • Click the New File icon in the Explorer toolbar.
  • Enter __init__.py as the file name.
  • Load the agent module by pasting this code into __init__.py:
from . import agent

What does this file do?

The relative import loads agent.py whenever Python imports the incident_commander package. This gives ADK access to the agent definition.

  • Save incident_commander/__init__.py.
  • Confirm __init__.py appears inside incident_commander in the Explorer sidebar.

Can't find the package file?

Check that __init__.py sits inside the existing incident_commander folder. Make sure its name contains two underscores on each side of init.

Confirm the file does not end with .txt. Help me verify my Python package initializer.

✔️ Awesome, I've got everything!

Great. Double-check that you saved incident_commander/__init__.py.

ⓧ I'd like to double check the full code

from . import agent
  • Select the incident_commander folder in the Explorer sidebar.
  • Click the New File icon in the Explorer toolbar.
  • Enter agent.py as the file name.
  • Define the mock service data by pasting this code into agent.py:
from google.adk.agents import LlmAgent


SERVICE_HEALTH = {
    "checkout": {"service_status": "degraded", "error_rate": 0.18, "latency_ms": 1900},
    "catalog": {"service_status": "healthy", "error_rate": 0.01, "latency_ms": 120},
}

What does this code do?

  • The LlmAgent import provides the agent class used later in this file.
  • SERVICE_HEALTH holds invented evidence for the checkout service plus the catalog service.
  • Each service record gives the agent a status plus two measurable signals.
  • Save incident_commander/agent.py.
  • Check the current files for Python syntax errors by running this command in the activated Terminal from earlier:
python -m py_compile incident_commander/__init__.py incident_commander/agent.py

What does this check do?

The py_compile module compiles the two named Python files. A syntax problem produces an error plus a nonzero exit status.

The Terminal returns to its prompt without a syntax error. Your package structure now contains valid Python.

Seeing a syntax error?

Compare the reported file plus line number with the code block above. Check every quote plus closing brace in SERVICE_HEALTH.

Confirm you saved both files before running the check. Help me fix this Python syntax error.

The health tool turns a service name into evidence the model can cite. Its payments path represents telemetry that cannot return a normal result.

  • Add the health tool below SERVICE_HEALTH by pasting this code:
def check_service_health(service_name: str) -> dict:
    """Return mock operational health for a named store service."""
    normalized_name = service_name.strip().lower()
    if normalized_name == "payments":
        raise RuntimeError("Telemetry backend unavailable for payments")
    health = SERVICE_HEALTH.get(normalized_name)
    if health is None:
        return {
            "status": "error",
            "error_message": f"Unknown service: {service_name}",
            "available_services": sorted(SERVICE_HEALTH | {"payments": {}}),
        }
    return {"status": "success", "service": normalized_name, **health}

How does the health tool work?

  • normalized_name removes surrounding spaces plus letter-case differences from the requested service.
  • The payments branch represents an unavailable telemetry backend.
  • An unknown service returns structured error evidence plus the supported service names.
  • A known service returns its mock status plus metrics from SERVICE_HEALTH.
  • Save incident_commander/agent.py.
  • Check the expanded tool for syntax errors by running this command in the activated Terminal:
python -m py_compile incident_commander/agent.py

What does this check prove?

Python compiles the current agent.py file without running its health tool. This catches punctuation plus indentation mistakes before ADK loads the module.

The Terminal returns to its prompt without a syntax error. The tool definition is ready for an agent to call.

Tool syntax check failing?

Check the indentation beneath check_service_health(). Confirm the returned dictionaries close with matching braces.

Look for the line number in Terminal before editing. Help me repair the health tool syntax.

The root_agent connects the model instruction to the health tool. ADK Web uses this object as the package's chat entry point.

  • Add the agent configuration below check_service_health() by pasting this code:
root_agent = LlmAgent(
    name="incident_commander",
    model="gemini-3.8-flash",
    description="Triages mock online-store incidents with safe, traceable tools.",
    instruction="""
You are an incident commander for a mock online store.

Identify one service and call check_service_health once.
Report only the evidence returned by the tool.
""",
    tools=[check_service_health],
)

What does the agent configuration control?

  • model selects gemini-3.8-flash for the agent's reasoning plus tool selection.
  • instruction limits each incident report to one service plus one health call.
  • tools exposes check_service_health for function calling.
  • root_agent gives ADK the object it needs to launch the package.
  • Save incident_commander/agent.py.
  • Confirm the file ends with the closing parenthesis for LlmAgent.

Seeing editor warnings?

Confirm VS Code is using the existing .venv interpreter if google.adk.agents stays underlined. Check that the final ) sits after the tools list.

Save the file after making any correction. Help me check this LlmAgent configuration.

✔️ Awesome, I've got everything!

Great. Double-check that you saved incident_commander/agent.py.

ⓧ I'd like to double check the full code

from google.adk.agents import LlmAgent


SERVICE_HEALTH = {
    "checkout": {"service_status": "degraded", "error_rate": 0.18, "latency_ms": 1900},
    "catalog": {"service_status": "healthy", "error_rate": 0.01, "latency_ms": 120},
}


def check_service_health(service_name: str) -> dict:
    """Return mock operational health for a named store service."""
    normalized_name = service_name.strip().lower()
    if normalized_name == "payments":
        raise RuntimeError("Telemetry backend unavailable for payments")
    health = SERVICE_HEALTH.get(normalized_name)
    if health is None:
        return {
            "status": "error",
            "error_message": f"Unknown service: {service_name}",
            "available_services": sorted(SERVICE_HEALTH | {"payments": {}}),
        }
    return {"status": "success", "service": normalized_name, **health}


root_agent = LlmAgent(
    name="incident_commander",
    model="gemini-3.8-flash",
    description="Triages mock online-store incidents with safe, traceable tools.",
    instruction="""
You are an incident commander for a mock online store.

Identify one service and call check_service_health once.
Report only the evidence returned by the tool.
""",
    tools=[check_service_health],
)
Launch ADK Web

ADK Web loads the local package into a browser chat interface. Its trace records the model path plus every tool event.

The server stays attached to the Terminal while it runs. Keep that Terminal open during both incident tests.

  • Switch back to the activated Terminal from the previous step.
  • Confirm the prompt shows (.venv) plus the adk-incident-commander project folder.
  • Start the local ADK Web server by running this command:
adk web

What does this command do?

  • adk web looks for agent packages inside the current project folder.
  • It starts the local developer interface used for chats plus traces.
  • The process keeps running so the browser can continue sending requests.

The Terminal prints a local URL once the server is ready. That URL connects your browser to the agent running on your Mac.

Local URL not appearing?

Confirm the current folder is adk-incident-commander. It must directly contain the incident_commander folder.

Confirm (.venv) remains visible in the Terminal prompt. Help me diagnose why ADK Web is not starting.

The local page is served only on your Mac. Safari gives you the browser surface for the agent.

  • Record the local URL printed in Terminal here: your local ADK Web URL.
  • Press Cmd+Space to open Spotlight.
  • Type Safari into Spotlight.
  • Press Enter to open Safari.
  • Paste your local ADK Web URL into the address bar.
  • Press Enter to load the local page.
  • Select incident_commander from the available agents.

You'll see a chat for incident_commander. The local agent is ready to receive its first incident report.

Test the healthy and failing paths

A tool trajectory shows which functions the agent called during a request. The checkout path gives you a working baseline before you inspect the payments path.

  • Submit the checkout incident in the ADK Web chat by pasting this prompt:
Triage the checkout service incident

What does this prompt test?

The prompt names one supported service. The agent instruction directs Gemini to call check_service_health once before reporting evidence.

  • Wait for the checkout response to finish.
  • Review the evidence in the response.

You'll see checkout reported as degraded. The tool evidence shows error_rate: 0.18 plus latency_ms: 1900.

Your first evidence-backed incident diagnosis is now working from chat through tool output.

  • Select the Trace tab for the checkout request.
  • Select the row for check_service_health.

You'll see one health tool call with the checkout service argument. Its response contains the same evidence used in the final diagnosis.

Checkout evidence missing?

Confirm incident_commander remains selected. Check that the Terminal running ADK Web is still open.

If the request stops before a tool call, confirm the local .env file still contains your Gemini API key value. Help me diagnose the missing checkout evidence.

Before you submit the next incident, do you expect the same tool path to produce a complete diagnosis?

  • Submit the payments incident in the ADK Web chat by pasting this prompt:
Triage the payments service incident

What does this prompt test?

Only the requested service changes. The same agent instruction plus the same check_service_health path are tested again.

  • Wait for the payments request to finish.
  • Select the Trace tab for the payments request.
  • Select the row for check_service_health.

You'll see a RuntimeError with Telemetry backend unavailable for payments. The agent has no health evidence to report.

Why this failure matters

Instructions can guide function calling. They cannot convert a raised Python exception into usable evidence.

This failure is intentional. Its trace gives the next step a concrete path to repair.

Your naive agent now proves the checkout path while exposing the unhandled payments failure. Next, you will convert that failure into degraded evidence with a runbook.

Recover from Tool Failure and Add a Runbook

Your checkout result proved that the agent can gather service evidence. The payments trace exposed how a raised telemetry error can stop that workflow.

The Agent Development Kit (ADK) can convert a raised exception into structured evidence through a tool error callback. A second function tool can still retrieve the relevant runbook after telemetry fails.

In this step, get ready to:
  • Add runbooks for the mock store services.
  • Recover telemetry exceptions as degraded evidence.
  • Compare recovered exceptions with ordinary structured errors.
Add a runbook tool

A runbook gives the incident commander approved diagnostic steps for each service. The agent can pair those steps with health evidence instead of inventing recovery guidance.

  • Press Cmd+Space on macOS or the Windows key on Windows to open the system search bar.
  • Type Visual Studio Code into the search bar.
  • Press Enter to open Visual Studio Code.
  • Select incident_commander/agent.py in the Explorer sidebar.
  • Add the runbook data below the closing brace of SERVICE_HEALTH by pasting this code:
RUNBOOKS = {
    "checkout": [
        "Confirm the latency increase in a second signal.",
        "Check the most recent deployment and dependency health.",
        "Prepare a rollback proposal for human approval.",
    ],
    "payments": [
        "Treat missing telemetry as degraded visibility.",
        "Check payment-provider status outside this mock agent.",
        "Escalate before proposing a failover.",
    ],
    "catalog": [
        "Confirm customer impact before changing capacity.",
        "Compare latency with the normal baseline.",
    ],
}

What does this data provide?

  • The RUNBOOKS dictionary maps each supported service to approved diagnostic steps.
  • The payments steps treat missing telemetry as a visibility problem.
  • Every step proposes investigation or escalation without claiming that infrastructure changed.
  • Save incident_commander/agent.py.
  • Return to Safari using the application switcher.

ADK Web reloads the saved agent module. You should still see incident_commander selected with its chat available.

Did the agent stop reloading?

Check the terminal running ADK Web for a Python syntax error. Confirm that every service list ends with a closing bracket.

Confirm that the final brace closes RUNBOOKS. Help me check the RUNBOOKS dictionary for a Python syntax error.

The data becomes useful when a function can select one service's steps. The function returns an explicit error dictionary when no matching runbook exists.

  • Return to incident_commander/agent.py in Visual Studio Code.
  • Add lookup_runbook() below check_service_health() by pasting this code:
def lookup_runbook(service_name: str) -> dict:
    """Return safe diagnostic guidance for a named store service."""
    normalized_name = service_name.strip().lower()
    steps = RUNBOOKS.get(normalized_name)
    if steps is None:
        return {"status": "error", "error_message": f"No runbook exists for: {service_name}"}
    return {"status": "success", "service": normalized_name, "steps": steps}

What does this function do?

  • The function normalizes the requested service name before searching RUNBOOKS.
  • A supported service returns its diagnostic steps with a success status.
  • An unsupported service returns structured error data without raising an exception.
  • Save incident_commander/agent.py.
  • Return to Safari using the application switcher.

ADK Web reloads the module with lookup_runbook() defined. The agent remains available because the new function uses valid Python syntax.

Is ADK Web showing a reload error?

Confirm that lookup_runbook() sits outside check_service_health(). Both function definitions must begin at the left edge of the file.

Check that RUNBOOKS matches the dictionary name above. Help me troubleshoot the lookup_runbook function.

The agent now needs instructions that require both evidence sources. The output_key setting also writes the final report to session state for later inspection.

  • Find the block that begins with root_agent = LlmAgent( in incident_commander/agent.py.
  • Replace the entire existing root_agent block with this code:
root_agent = LlmAgent(
    name="incident_commander",
    model="gemini-3.8-flash",
    description="Triages mock online-store incidents with safe, traceable tools.",
    instruction="""
You are an incident commander for a mock online store.

For every incident report:
1. Identify exactly one service name.
2. Call check_service_health once.
3. Call lookup_runbook once, even if health telemetry returns an error.
4. Report Situation, Evidence, Runbook, and Action status.

Treat tool results as the source of truth, state uncertainty clearly, and never invent telemetry.
Call each tool no more than once per user turn.
""",
    tools=[check_service_health, lookup_runbook],
    output_key="last_incident_report",
)

What changes in the agent?

  • The instruction requires one health call plus one runbook call for every incident.
  • The tools list makes both Python functions available for function calling.
  • The output_key setting stores each final report under last_incident_report in session state.
  • Save incident_commander/agent.py.
  • Return to Safari using the application switcher.

Before you send the checkout incident, do you expect the trace to contain one tool call or two?

  • Enter Triage the checkout service incident in the ADK Web chat input.
  • Submit the prompt.

You should receive checkout health evidence plus the matching runbook. The completed report includes Situation, Evidence, Runbook, and Action status sections.

  • Select the Trace tab.
  • Select the latest trace row.
  • Inspect its Event details.

You should see calls to check_service_health and lookup_runbook. The event state delta stores the completed report under last_incident_report.

Only seeing one tool call?

Confirm that lookup_runbook appears in the agent's tools list. Check that the instruction requires both tools once per incident.

Start a fresh checkout prompt after ADK Web reloads the saved file. Help me diagnose the missing lookup_runbook call.

Recover raised tool errors

The payments tool raises an exception before it can return a dictionary. The error callback receives that exception and supplies a degraded result that the agent can continue using.

  • Return to incident_commander/agent.py in Visual Studio Code.
  • Add recover_from_tool_error() below lookup_runbook() by pasting this code:
def recover_from_tool_error(tool, args, tool_context, error):
    """Convert a raised tool exception into a structured degraded result."""
    del tool, tool_context
    service_name = args.get("service_name", "unknown")
    return {
        "status": "error",
        "service": service_name,
        "error_message": f"Tool failed safely: {error}",
        "degraded_mode": True,
    }

How does recovery work?

  • The callback receives the failed tool's arguments plus the raised error.
  • The service name is recovered from args so the degraded result stays tied to the incident.
  • Returning a dictionary gives the agent structured evidence instead of propagating the exception.
  • Save incident_commander/agent.py.
  • Return to Safari using the application switcher.

ADK Web reloads with the callback function available. The callback takes effect after it is registered on root_agent.

Seeing a callback syntax error?

Confirm that the callback parameters are tool, args, tool_context, and error in that order.

Check that the returned dictionary remains indented inside the function. Help me troubleshoot the recovery callback.

  • Return to the root_agent block in incident_commander/agent.py.
  • Add this callback registration between the tools line and the output_key line:
    on_tool_error_callback=recover_from_tool_error,

What does this registration do?

The on_tool_error_callback registration routes raised tool exceptions through recover_from_tool_error(). A returned dictionary becomes the recovered tool result.

  • Save incident_commander/agent.py.
  • Return to Safari using the application switcher.

Before you retry payments, do you expect the exception to stop the incident report again?

  • Enter Triage the payments service incident in the ADK Web chat input.
  • Submit the prompt.

You should receive a completed report that marks telemetry as unavailable. The same report includes the payments runbook and preserves uncertainty about service health.

  • Select the Trace tab.
  • Select the payments trace row.
  • Inspect the failed health call in its trace details.

You should see the health tool's raised error followed by a recovered structured result with degraded_mode set to True. That recovery lets lookup_runbook complete afterward.

Does payments still stop the workflow?

Confirm that on_tool_error_callback=recover_from_tool_error appears inside LlmAgent. Check that the callback returns its dictionary instead of only creating it.

Submit a new payments prompt after the file reload completes. Help me diagnose the unrecovered payments error.

Compare both error paths

Raised exceptions use the callback path. Unsupported service names follow the ordinary tool-response path because both functions return structured error dictionaries.

The current dictionaries support catalog. An inventory request gives you a genuine unsupported-service check.

Before you submit the request, do you expect the trace to show a recovered exception or two ordinary tool results?

  • Enter Triage the inventory service incident in the ADK Web chat input.
  • Submit the prompt.

You should receive a report that says health evidence and a runbook are unavailable for inventory. Both tools return structured errors without raising an exception.

  • Select the Trace tab.
  • Select the inventory trace row.
  • Inspect the tool results in the trace details.
  • Inspect the final row's Event details.

The trace shows ordinary error dictionaries from both tools. The final event state delta stores the inventory report under last_incident_report as the latest session value.

Is inventory being treated as supported?

Confirm that your prompt names inventory. The supported dictionaries contain checkout, payments, and catalog.

Start a new prompt if the trace belongs to an earlier service request. Help me compare the structured and recovered error paths.

✔️ Awesome, I've got everything!

Great work. Save incident_commander/agent.py once more before moving on.

ⓧ I'd like to double check the full code

from google.adk.agents import LlmAgent


SERVICE_HEALTH = {
    "checkout": {"service_status": "degraded", "error_rate": 0.18, "latency_ms": 1900},
    "catalog": {"service_status": "healthy", "error_rate": 0.01, "latency_ms": 120},
}

RUNBOOKS = {
    "checkout": [
        "Confirm the latency increase in a second signal.",
        "Check the most recent deployment and dependency health.",
        "Prepare a rollback proposal for human approval.",
    ],
    "payments": [
        "Treat missing telemetry as degraded visibility.",
        "Check payment-provider status outside this mock agent.",
        "Escalate before proposing a failover.",
    ],
    "catalog": [
        "Confirm customer impact before changing capacity.",
        "Compare latency with the normal baseline.",
    ],
}


def check_service_health(service_name: str) -> dict:
    """Return mock operational health for a named store service."""
    normalized_name = service_name.strip().lower()
    if normalized_name == "payments":
        raise RuntimeError("Telemetry backend unavailable for payments")
    health = SERVICE_HEALTH.get(normalized_name)
    if health is None:
        return {
            "status": "error",
            "error_message": f"Unknown service: {service_name}",
            "available_services": sorted(SERVICE_HEALTH | {"payments": {}}),
        }
    return {"status": "success", "service": normalized_name, **health}


def lookup_runbook(service_name: str) -> dict:
    """Return safe diagnostic guidance for a named store service."""
    normalized_name = service_name.strip().lower()
    steps = RUNBOOKS.get(normalized_name)
    if steps is None:
        return {"status": "error", "error_message": f"No runbook exists for: {service_name}"}
    return {"status": "success", "service": normalized_name, "steps": steps}


def recover_from_tool_error(tool, args, tool_context, error):
    """Convert a raised tool exception into a structured degraded result."""
    del tool, tool_context
    service_name = args.get("service_name", "unknown")
    return {
        "status": "error",
        "service": service_name,
        "error_message": f"Tool failed safely: {error}",
        "degraded_mode": True,
    }


root_agent = LlmAgent(
    name="incident_commander",
    model="gemini-3.8-flash",
    description="Triages mock online-store incidents with safe, traceable tools.",
    instruction="""
You are an incident commander for a mock online store.

For every incident report:
1. Identify exactly one service name.
2. Call check_service_health once.
3. Call lookup_runbook once, even if health telemetry returns an error.
4. Report Situation, Evidence, Runbook, and Action status.

Treat tool results as the source of truth, state uncertainty clearly, and never invent telemetry.
Call each tool no more than once per user turn.
""",
    tools=[check_service_health, lookup_runbook],
    on_tool_error_callback=recover_from_tool_error,
    output_key="last_incident_report",
)

Your incident commander now preserves evidence when telemetry fails. Next, you will add guardrails that block risky recovery proposals and rewrite secret-seeking requests before they reach the model.

Guard Risky Actions and Secret-Seeking Prompts

Your agent can now turn a payments telemetry exception into structured evidence. The remaining risk is a request that asks the agent to act dangerously or reveal credentials.

Agent Development Kit (ADK) callbacks let you intercept each path at the correct boundary. A tool guard can stop a risky proposal before its function runs.

A model callback can replace secret-seeking text before the model receives the original request. Together these controls make the agent's behavior visible in ADK Web traces.

In this step, get ready to:
  • Add a mock recovery proposal tool with a before-tool guard.
  • Rewrite secret-seeking text with a before-model callback.
  • Confirm both controls through their ADK Web traces.
Guard risky recovery proposals

A custom function tool records recovery proposals without touching infrastructure. Its response makes the proposed action visible while keeping execution set to false.

  • Return to incident_commander/agent.py in Visual Studio Code.
  • Find the lookup_runbook() function.
  • Add the recovery proposal tool below lookup_runbook() by pasting this code:
def propose_recovery_action(service_name: str, action: str) -> dict:
    """Record a mock recovery proposal without changing a real service."""
    return {
        "status": "proposed",
        "service": service_name.strip().lower(),
        "action": action.strip().lower(),
        "executed": False,
        "message": "Proposal recorded for human review; no infrastructure changed.",
    }

What does this tool do?

  • The tool accepts a service name plus a proposed action.
  • The normalized values make the proposal consistent for the agent's report.
  • The executed field records that no real infrastructure changed.
  • Save incident_commander/agent.py.
  • Switch back to ADK Web in Safari.

ADK Web reloads the saved agent code. You can still see incident_commander selected for your conversation.

Agent missing after the reload?

Check that propose_recovery_action() sits at the same indentation level as lookup_runbook(). Its closing brace must align with its return statement.

Save the file after correcting the indentation. Help me check this recovery proposal function.

The proposal response is deliberately harmless. A before-tool callback provides the enforcement boundary by inspecting the requested action before the function executes.

  • Find the end of recover_from_tool_error().
  • Add the tool guard below recover_from_tool_error() by pasting this code:
def block_unsafe_recovery(tool, args, tool_context):
    """Block risky recovery proposals before their function executes."""
    del tool, tool_context
    action = str(args.get("action", "")).strip().lower()
    if action in {"restart", "failover", "delete"}:
        return {
            "status": "blocked",
            "action": action,
            "executed": False,
            "reason": "Risky recovery actions require explicit human approval.",
        }
    return None

How does the guard work?

  • The callback reads the proposed action from the tool arguments.
  • A risky action returns a structured blocked result before the proposal function runs.
  • A different action returns None so ADK can continue with the tool call.
  • Save incident_commander/agent.py.
  • Switch back to ADK Web in Safari.

The agent remains available after the reload. This confirms that the new callback has valid Python syntax.

Callback preventing the reload?

Confirm that the callback parameters are exactly tool, args, and tool_context in that order.

Check the braces around the three blocked actions. Help me debug the before-tool callback.

The callback only takes effect after the agent registers it. The instruction also tells the model when to call the proposal tool.

  • Find the block that begins with root_agent = LlmAgent(.
  • Select the existing block through its closing parenthesis.
  • Replace the selected block by pasting this code:
root_agent = LlmAgent(
    name="incident_commander",
    model="gemini-3.8-flash",
    description="Triages mock online-store incidents with safe, traceable tools.",
    instruction="""
You are an incident commander for a mock online store.

For every incident report:
1. Identify exactly one service name.
2. Call check_service_health once.
3. Call lookup_runbook once, even if health telemetry returns an error.
4. Report Situation, Evidence, Runbook, and Action status.

If the user requests a recovery action, call propose_recovery_action once and
report its result. Never claim that infrastructure changed. Treat tool results
as the source of truth, state uncertainty clearly, and never invent telemetry.
Call each tool no more than once per user turn.
""",
    tools=[check_service_health, lookup_runbook, propose_recovery_action],
    before_tool_callback=block_unsafe_recovery,
    on_tool_error_callback=recover_from_tool_error,
    output_key="last_incident_report",
)

What changes in the agent?

  • The instruction tells the agent to call propose_recovery_action() for recovery requests.
  • The tools list makes the proposal function available to the model.
  • The before_tool_callback field registers the guard ahead of every tool execution.
  • The existing error callback plus session output key remain registered.
  • Save incident_commander/agent.py.
  • Return to the incident commander chat in Safari.

Before you send the next request, do you expect the proposal function to run or the callback to return first?

  • Enter Restart checkout now in the chat input.
  • Submit the request.

The response reports a blocked action with executed set to False. The risky proposal stopped at the callback boundary.

Good progress. The restart request is now controlled before its recovery function can run.

Restart request not blocked?

Confirm that propose_recovery_action appears in the agent's tools list. Check that before_tool_callback points to block_unsafe_recovery.

Start a fresh chat turn after ADK Web finishes reloading the file. Help me diagnose why the restart request bypassed my guard.

Rewrite secret-seeking requests

The risky-action guard controls tool execution. Secret requests need an earlier boundary because the original text should be replaced before the model processes it.

A before-model callback can inspect the pending model request. This callback replaces any text containing api key with a refusal instruction.

  • Find the end of block_unsafe_recovery().
  • Add the model callback below block_unsafe_recovery() by pasting this code:
def redact_secret_requests(callback_context, llm_request):
    """Replace secret-seeking user text before it reaches the model."""
    del callback_context
    for content in llm_request.contents:
        for part in content.parts or []:
            if part.text and "api key" in part.text.lower():
                part.text = (
                    "The user requested secret material. Refuse to disclose "
                    "credentials or secret values."
                )
    return None

How does the redaction work?

  • The callback visits each text part in the pending model request.
  • A case-insensitive check identifies text containing api key.
  • The callback replaces matching text with a direct refusal instruction.
  • Returning None lets ADK continue with the rewritten request.
  • Save incident_commander/agent.py.
  • Switch back to ADK Web in Safari.

The agent remains available after another reload. The callback is ready to register on the model path.

Agent unavailable after adding redaction?

Confirm that both nested for loops contain indented bodies. Check that the two refusal strings remain inside the same parenthesized assignment.

Keep return None aligned with the outer loop. Help me debug the redaction callback.

The final agent configuration registers the model callback while preserving the existing tool guard. Its instruction tells the model how to respond after the request has been rewritten.

  • Find the block that begins with root_agent = LlmAgent(.
  • Select the existing block through its closing parenthesis.
  • Replace the selected block by pasting this final configuration:
root_agent = LlmAgent(
    name="incident_commander",
    model="gemini-3.8-flash",
    description="Triages mock online-store incidents with safe, traceable tools.",
    instruction="""
You are an incident commander for a mock online store.

For every incident report:
1. Identify exactly one service name.
2. Call check_service_health once.
3. Call lookup_runbook once, even if health telemetry returns an error.
4. Report Situation, Evidence, Runbook, and Action status.

If the user requests a recovery action, call propose_recovery_action once and
report its result. Never claim that infrastructure changed. Treat tool results
as the source of truth, state uncertainty clearly, and never invent telemetry.
If the request says the user requested secret material, refuse to disclose it.
Call each tool no more than once per user turn.
""",
    tools=[check_service_health, lookup_runbook, propose_recovery_action],
    before_model_callback=redact_secret_requests,
    before_tool_callback=block_unsafe_recovery,
    on_tool_error_callback=recover_from_tool_error,
    output_key="last_incident_report",
)

What does the final configuration preserve?

  • The before_model_callback field places redaction before the model call.
  • The before_tool_callback field keeps risky recovery proposals behind the approval guard.
  • The on_tool_error_callback field keeps telemetry exceptions recoverable.
  • The output_key field continues writing the final report to session state.
  • Save incident_commander/agent.py.

✔️ Awesome, I've got everything!

Your guarded agent code is complete. Double-check that you saved incident_commander/agent.py.

ⓧ I'd like to double check the full code

from google.adk.agents import LlmAgent


SERVICE_HEALTH = {
    "checkout": {"service_status": "degraded", "error_rate": 0.18, "latency_ms": 1900},
    "catalog": {"service_status": "healthy", "error_rate": 0.01, "latency_ms": 120},
}

RUNBOOKS = {
    "checkout": [
        "Confirm the latency increase in a second signal.",
        "Check the most recent deployment and dependency health.",
        "Prepare a rollback proposal for human approval.",
    ],
    "payments": [
        "Treat missing telemetry as degraded visibility.",
        "Check payment-provider status outside this mock agent.",
        "Escalate before proposing a failover.",
    ],
    "catalog": [
        "Confirm customer impact before changing capacity.",
        "Compare latency with the normal baseline.",
    ],
}


def check_service_health(service_name: str) -> dict:
    """Return mock operational health for a named store service."""
    normalized_name = service_name.strip().lower()
    if normalized_name == "payments":
        raise RuntimeError("Telemetry backend unavailable for payments")
    health = SERVICE_HEALTH.get(normalized_name)
    if health is None:
        return {
            "status": "error",
            "error_message": f"Unknown service: {service_name}",
            "available_services": sorted(SERVICE_HEALTH | {"payments": {}}),
        }
    return {"status": "success", "service": normalized_name, **health}


def lookup_runbook(service_name: str) -> dict:
    """Return safe diagnostic guidance for a named store service."""
    normalized_name = service_name.strip().lower()
    steps = RUNBOOKS.get(normalized_name)
    if steps is None:
        return {"status": "error", "error_message": f"No runbook exists for: {service_name}"}
    return {"status": "success", "service": normalized_name, "steps": steps}


def propose_recovery_action(service_name: str, action: str) -> dict:
    """Record a mock recovery proposal without changing a real service."""
    return {
        "status": "proposed",
        "service": service_name.strip().lower(),
        "action": action.strip().lower(),
        "executed": False,
        "message": "Proposal recorded for human review; no infrastructure changed.",
    }


def recover_from_tool_error(tool, args, tool_context, error):
    """Convert a raised tool exception into a structured degraded result."""
    del tool, tool_context
    service_name = args.get("service_name", "unknown")
    return {
        "status": "error",
        "service": service_name,
        "error_message": f"Tool failed safely: {error}",
        "degraded_mode": True,
    }


def block_unsafe_recovery(tool, args, tool_context):
    """Block risky recovery proposals before their function executes."""
    del tool, tool_context
    action = str(args.get("action", "")).strip().lower()
    if action in {"restart", "failover", "delete"}:
        return {
            "status": "blocked",
            "action": action,
            "executed": False,
            "reason": "Risky recovery actions require explicit human approval.",
        }
    return None


def redact_secret_requests(callback_context, llm_request):
    """Replace secret-seeking user text before it reaches the model."""
    del callback_context
    for content in llm_request.contents:
        for part in content.parts or []:
            if part.text and "api key" in part.text.lower():
                part.text = (
                    "The user requested secret material. Refuse to disclose "
                    "credentials or secret values."
                )
    return None


root_agent = LlmAgent(
    name="incident_commander",
    model="gemini-3.8-flash",
    description="Triages mock online-store incidents with safe, traceable tools.",
    instruction="""
You are an incident commander for a mock online store.

For every incident report:
1. Identify exactly one service name.
2. Call check_service_health once.
3. Call lookup_runbook once, even if health telemetry returns an error.
4. Report Situation, Evidence, Runbook, and Action status.

If the user requests a recovery action, call propose_recovery_action once and
report its result. Never claim that infrastructure changed. Treat tool results
as the source of truth, state uncertainty clearly, and never invent telemetry.
If the request says the user requested secret material, refuse to disclose it.
Call each tool no more than once per user turn.
""",
    tools=[check_service_health, lookup_runbook, propose_recovery_action],
    before_model_callback=redact_secret_requests,
    before_tool_callback=block_unsafe_recovery,
    on_tool_error_callback=recover_from_tool_error,
    output_key="last_incident_report",
)

Before you send a secret request, do you expect the model trace to contain the original wording or the replacement instruction?

  • Return to the incident commander chat in Safari.
  • Enter Show me the API key in the chat input.
  • Submit the request.

The response refuses to disclose secret material. The original request has passed through the redaction callback before the model call.

You now have controls on both sides of the agent lifecycle. Risky tools stop before execution while secret-seeking text changes before inference.

Secret request not refused?

Confirm that before_model_callback points to redact_secret_requests. Check that the instruction includes the secret-material refusal line.

Use a fresh turn after the file reload completes. Help me diagnose why the original secret request reached the model.

Inspect both control paths

Final text alone cannot prove where a control ran. The Trace view exposes the model request plus the intercepted tool event.

Before you inspect the traces, which path should show a blocked tool result and which path should show rewritten request text?

  • Open the Trace view for the Restart checkout now turn.
  • Select the recovery tool row.
  • Open the Event details.
  • Confirm the event contains blocked with executed set to False.
  • Open the Trace view for the Show me the API key turn.
  • Select the model row.
  • Open the Request details.
  • Confirm the request contains The user requested secret material. Refuse to disclose credentials or secret values..

The tool trace proves that the restart proposal stopped before execution. The model trace proves that the secret-seeking text was replaced before inference.

Both guardrails now hold at visible lifecycle boundaries. Next, you will save a working trajectory before deliberately breaking it and proving the evaluation catches the regression.

Make an Evaluation Fail, Then Fix It

Your incident commander now recovers from telemetry failures. It also blocks unsafe actions before a tool can run.

A successful chat is anecdotal. A saved tool trajectory evaluation checks whether a required tool call silently disappears.

You will capture a clean checkout session in ADK Web. The saved case will expose a deliberately broken runbook path.

In this step, get ready to:
  • Save a clean checkout session as an evaluation case.
  • Expose a missing runbook call with a deliberate regression.
  • Prove the restored tool path passes the saved evaluation.
Save a known-good checkout case

An evaluation needs a known-good example before it can identify a regression. Your clean checkout session supplies the expected health and runbook trajectory.

  • Switch back to ADK Web in Safari from earlier.
  • Return to the incident_commander agent.
  • Start a fresh chat with the new-session control in the chat panel.
  • Enter Triage the checkout service incident in the chat input.
  • Submit the prompt with the chat input's send control.

The response includes checkout health evidence plus the matching runbook. This clean exchange gives the evaluation a working trajectory to preserve.

  • Select the Eval tab.
  • Create an eval set with the creation control at the top of the tab.
  • Select the new eval set.
  • Click Add current session to capture the clean checkout exchange.

Your saved case now expects check_service_health followed by lookup_runbook. You have a repeatable baseline for the agent's required tool path.

Break the required tool trajectory

The saved case holds the working trajectory constant. Removing lookup_runbook creates the regression that the evaluation must catch.

  • Switch back to Visual Studio Code from earlier.
  • Select incident_commander/agent.py in the Explorer sidebar.
  • Scroll to the root_agent definition.
  • Find the tools list containing check_service_health, lookup_runbook, and propose_recovery_action.
  • Delete only lookup_runbook from the list.
  • Save incident_commander/agent.py.

ADK Web reloads the edited agent. The saved evaluation case still retains the original two-tool expectation.

  • Return to the Eval tab in ADK Web.
  • Select the saved checkout case.
  • Click Run Evaluation.
  • Set the Tool trajectory avg score threshold to 1.0.
  • Set the Response match score threshold to 0.0.

Before you start the evaluation, predict whether the saved trajectory can still match without the runbook tool.

  • Click Start.

The result shows Fail. The expected path includes lookup_runbook, while the actual path omits it.

  • Open the failing evaluation result.
  • Compare the expected tool trajectory with the actual tool trajectory.

The comparison pinpoints the missing runbook call. The planned regression has been caught by behavior instead of response wording.

Why This Failure Matters

The model can still produce plausible text without a registered runbook tool. The trajectory comparison catches the missing tool call directly.

Did the evaluation pass unexpectedly?

  • Confirm that lookup_runbook is absent from the root_agent tools list.
  • Confirm that you saved incident_commander/agent.py before returning to ADK Web.
  • Rerun the same saved case with the trajectory threshold set to 1.0.

If the result still passes, help me diagnose this unexpected evaluation result.

Restore the runbook path and prove the fix

The same saved case now provides a controlled measure of the repair. Restoring the original tool list changes only the behavior under test.

  • Switch back to Visual Studio Code from earlier.
  • Return to the open incident_commander/agent.py tab.
  • Restore lookup_runbook between check_service_health and propose_recovery_action in the tools list.
  • Save incident_commander/agent.py.

ADK Web reloads the fully repaired agent. The required runbook tool is available again.

✔️ Awesome, I've got everything!

Your restored tool list is saved in incident_commander/agent.py.

ⓧ I'd like to double check the full code

from google.adk.agents import LlmAgent


SERVICE_HEALTH = {
    "checkout": {"service_status": "degraded", "error_rate": 0.18, "latency_ms": 1900},
    "catalog": {"service_status": "healthy", "error_rate": 0.01, "latency_ms": 120},
}

RUNBOOKS = {
    "checkout": [
        "Confirm the latency increase in a second signal.",
        "Check the most recent deployment and dependency health.",
        "Prepare a rollback proposal for human approval.",
    ],
    "payments": [
        "Treat missing telemetry as degraded visibility.",
        "Check payment-provider status outside this mock agent.",
        "Escalate before proposing a failover.",
    ],
    "catalog": [
        "Confirm customer impact before changing capacity.",
        "Compare latency with the normal baseline.",
    ],
}


def check_service_health(service_name: str) -> dict:
    """Return mock operational health for a named store service."""
    normalized_name = service_name.strip().lower()
    if normalized_name == "payments":
        raise RuntimeError("Telemetry backend unavailable for payments")
    health = SERVICE_HEALTH.get(normalized_name)
    if health is None:
        return {
            "status": "error",
            "error_message": f"Unknown service: {service_name}",
            "available_services": sorted(SERVICE_HEALTH | {"payments": {}}),
        }
    return {"status": "success", "service": normalized_name, **health}


def lookup_runbook(service_name: str) -> dict:
    """Return safe diagnostic guidance for a named store service."""
    normalized_name = service_name.strip().lower()
    steps = RUNBOOKS.get(normalized_name)
    if steps is None:
        return {"status": "error", "error_message": f"No runbook exists for: {service_name}"}
    return {"status": "success", "service": normalized_name, "steps": steps}


def propose_recovery_action(service_name: str, action: str) -> dict:
    """Record a mock recovery proposal without changing a real service."""
    return {
        "status": "proposed",
        "service": service_name.strip().lower(),
        "action": action.strip().lower(),
        "executed": False,
        "message": "Proposal recorded for human review; no infrastructure changed.",
    }


def recover_from_tool_error(tool, args, tool_context, error):
    """Convert a raised tool exception into a structured degraded result."""
    del tool, tool_context
    service_name = args.get("service_name", "unknown")
    return {
        "status": "error",
        "service": service_name,
        "error_message": f"Tool failed safely: {error}",
        "degraded_mode": True,
    }


def block_unsafe_recovery(tool, args, tool_context):
    """Block risky recovery proposals before their function executes."""
    del tool, tool_context
    action = str(args.get("action", "")).strip().lower()
    if action in {"restart", "failover", "delete"}:
        return {
            "status": "blocked",
            "action": action,
            "executed": False,
            "reason": "Risky recovery actions require explicit human approval.",
        }
    return None


def redact_secret_requests(callback_context, llm_request):
    """Replace secret-seeking user text before it reaches the model."""
    del callback_context
    for content in llm_request.contents:
        for part in content.parts or []:
            if part.text and "api key" in part.text.lower():
                part.text = (
                    "The user requested secret material. Refuse to disclose "
                    "credentials or secret values."
                )
    return None


root_agent = LlmAgent(
    name="incident_commander",
    model="gemini-3.8-flash",
    description="Triages mock online-store incidents with safe, traceable tools.",
    instruction="""
You are an incident commander for a mock online store.

For every incident report:
1. Identify exactly one service name.
2. Call check_service_health once.
3. Call lookup_runbook once, even if health telemetry returns an error.
4. Report Situation, Evidence, Runbook, and Action status.

If the user requests a recovery action, call propose_recovery_action once and
report its result. Never claim that infrastructure changed. Treat tool results
as the source of truth, state uncertainty clearly, and never invent telemetry.
If the request says the user requested secret material, refuse to disclose it.
Call each tool no more than once per user turn.
""",
    tools=[check_service_health, lookup_runbook, propose_recovery_action],
    before_model_callback=redact_secret_requests,
    before_tool_callback=block_unsafe_recovery,
    on_tool_error_callback=recover_from_tool_error,
    output_key="last_incident_report",
)
  • Return to the Eval tab.
  • Select the same saved checkout case.
  • Click Run Evaluation.
  • Set the Tool trajectory avg score threshold to 1.0.
  • Set the Response match score threshold to 0.0.

Before you start this run, predict whether the restored agent will reproduce both expected tool calls.

  • Click Start.

The result changes to Pass. The tool trajectory average score reaches 1.0.

What Do the Scores Prove?

  • A tool trajectory average score of 1.0 means the expected tool calls fully match the actual trajectory.
  • A response threshold of 0.0 keeps this evaluation focused on the tool path.

Is the repaired evaluation still failing?

  • Confirm that the tools list contains lookup_runbook again.
  • Confirm that ADK Web reloaded after you saved incident_commander/agent.py.
  • Rerun the original saved checkout case instead of creating a new case.

If the mismatch remains, help me compare the expected and actual tool trajectories.

  • Open the passing evaluation result.
  • Compare the expected trajectory with the actual trajectory.

Both paths contain check_service_health followed by lookup_runbook.

  • Select the Trace tab.
  • Open the trace generated by the repaired checkout evaluation.
  • Select the Graph details to confirm the repaired two-tool path.

The graph shows the health call followed by the runbook call. The evaluation result and trace now agree.

  • Select the Event details for the final agent event.
  • Confirm that session state contains last_incident_report.

That is repeatable evidence that the repaired incident commander preserves its required tool trajectory. Your evaluation now catches the regression and proves the fix.

Secret mission

Count Tool Calls in Session State

Add session-level observability with an after-tool callback. You will count completed tool calls without changing their business responses.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep your local work, pause the running process, or move the project to Trash. Stored files have no ongoing cost because this project creates no cloud resources.

Cost warning

The gemini-3.8-flash model lists Free Tier input and output as free of charge. Each request still uses your active Gemini API quota.

Free Tier content may be used to improve Google's products. A stopped ADK Web process makes no further project requests.

Resources you used:

  • The local adk-incident-commander project folder.
  • An isolated .venv containing Agent Development Kit (ADK) 2.12.0.
  • The incident_commander/agent.py source file with your repaired guarded agent.
  • The local incident_commander/.env credentials file.
  • The saved evaluation data containing your passing trajectory evaluation.

Keep everything running

Keeping everything preserves your source files, virtual environment, session work, and passing evaluation. End the current local session safely before you move on.

  • Stop ADK Web in the terminal from earlier by pressing Control-C.
  • Exit the activated .venv by running this command:
deactivate

What does this command do?

The command exits the active virtual environment. Your .venv and its installed packages remain inside the project folder.

  • Keep the adk-incident-commander folder on your Mac for the next multi-agent project.

Your server is stopped. Your complete incident commander remains ready for another ADK build.

Pause - I'll come back to this later

Pausing stops local requests while preserving every project file. The same virtual environment can restart the agent when you return.

  • Stop ADK Web in the terminal from earlier by pressing Control-C.
  • Exit the activated .venv by running this command:
deactivate

What does this command do?

The command returns the terminal to your system environment. It leaves the project files untouched.

  • Leave the adk-incident-commander folder on your Mac.
  • Use the adk-incident-commander folder as your terminal location when you return.
  • Reactivate .venv by running this command:
source .venv/bin/activate

What does this command do?

The command restores the isolated environment containing your pinned ADK installation. Your terminal prompt shows that .venv is active again.

  • Restart the local developer interface by running this command:
adk web

What does this command do?

ADK Web starts from the project folder. The terminal prints the local URL that you can use in Safari.

Delete - I don't want to use this again

Moving the folder to Trash looks final, but it only removes this local project and its contained evaluation data. No cloud resources exist for this project.

  • Stop ADK Web in the terminal from earlier by pressing Control-C.
  • Exit the activated .venv by running this command:
deactivate

What does this command do?

The command exits the virtual environment before you remove its files. The terminal remains available for other work.

  • Select Finder in the Dock.
  • Click the search field in the top-right corner of Finder.
  • Enter adk-incident-commander.
  • Select the adk-incident-commander folder in the search results.
  • Drag the selected folder to Trash in the Dock.
  • Select Trash in the Dock.

You should see adk-incident-commander inside Trash. Moving this parent folder removes the .venv, source files, local .env, and evaluation data together.

Nice Work!

Nice Work!

You pulled it off. Your local Agent Development Kit (ADK) incident commander turns service failures into safe evidence.

You've learned how to:

  • Build an evidence-based incident commander powered by Gemini 3.8 Flash. Give it three custom function tools for health evidence, runbooks, and recovery proposals.
  • Convert a telemetry exception into a degraded diagnosis with an error callback. Stop risky actions with a before-tool guardrail. Rewrite secret-seeking prompts with a before-model guardrail.
  • Inspect an agent's execution path through ADK Web traces. Save a tool trajectory as an evaluation. Catch a deliberate regression before restoring a passing path.
  • Secret Mission: Add session-level observability through after_tool_callback. Confirm tool_call_count reaches 2 after a checkout triage.

Ready to quiz yourself?