Build an AI Agent Evaluation Platform
Build a scoring engine that measures AI agent tool-calling accuracy.
Introduction
30 Second Summary
Everyone trusts AI agents to handle important tasks, but how do you know they actually work correctly? The same way engineering teams do: you build a system that tests them automatically and scores their performance.
In this project, you will build an AI Agent Evaluation Platform that measures how well an AI agent performs across multiple dimensions. You will score everything from tool selection accuracy to whether the agent fabricates information.
What You'll Build
A production-architecture evaluation platform with a domain-agnostic scoring engine, an IT Help Desk reference agent powered by Claude, and a structured test dataset that exercises the agent across multiple failure modes.
By the end of this project, you'll have:
- A domain-agnostic evaluation engine with three pluggable scoring metrics that can evaluate any tool-calling agent.
- An IT Help Desk agent that uses Claude's tool calling to resolve support tickets against 5 simulated enterprise tools.
- A CLI runner that executes the full evaluation suite and reports per-metric scores across 8 test cases covering happy paths and edge cases.
- Secret Mission: Add a hallucination detection metric that catches the agent fabricating information not grounded in tool outputs.
Are there any prerequisites?
You need Python installed (3.10 or higher), a Claude API key (ANTHROPIC_API_KEY), and familiarity with Python classes and JSON.
Before We Start
AI agents that call tools are only as reliable as your ability to measure them. Without structured evaluation, you have no idea whether your agent picked the right tool, passed valid arguments, or actually solved the user's problem.
In this step, you will install the Anthropic Python SDK, create the project directory structure for the evaluation platform, and verify everything works with a test API call to Claude.
In this step, get ready to:
- Install the Anthropic Python SDK inside a virtual environment.
- Create the project directory structure separating evaluation, agents, tools, and datasets.
- Verify your setup with a test API call to Claude.
Install the Anthropic Python SDK
The Anthropic Python SDK is the official library for communicating with the Claude API. You will install it inside a virtual environment to keep your project dependencies isolated.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type VS Code and press Enter to open it.
- Open the integrated terminal by pressing Ctrl+` (backtick key, same shortcut on both macOS and Windows).
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir agent-eval-platform
cd agent-eval-platform- Create a Python virtual environment by running:
python3 -m venv venvWhat is a virtual environment?
A virtual environment is an isolated copy of Python for your project. Packages you install inside it stay separate from your system Python, preventing version conflicts between projects.
- Activate the virtual environment:
macOS
source venv/bin/activateWindows
venv\Scripts\Activate.ps1Seeing an execution policy error?
If PowerShell blocks the activation script, run this command first to allow it:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUserThen try the activation command again.
You should see (venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.
- Create the requirements.txt file that pins your project dependencies by running:
echo "anthropic==0.116.0" > requirements.txt- Install the Anthropic SDK by running:
pip install -r requirements.txtYou should see output ending with a line like Successfully installed anthropic-0.116.0 along with its dependencies.
Install failing or hanging?
Make sure you see (venv) in your prompt. If it is missing, the virtual environment is not active. Re-run the activation command above.
If you see a permissions error, confirm you are inside the agent-eval-platform folder by running pwd (macOS) or cd (Windows).
Help me troubleshoot my pip install
- Set your Anthropic API key as an environment variable so the SDK can find it:
macOS
export ANTHROPIC_API_KEY=your-api-key-hereReplace your-api-key-here with your actual API key starting with sk-ant-.
Windows
$env:ANTHROPIC_API_KEY="your-api-key-here"Replace your-api-key-here with your actual API key starting with sk-ant-.
Where does the SDK look for your key?
The Anthropic SDK automatically reads the ANTHROPIC_API_KEY environment variable when you create a client. You never need to paste the key directly into your code.
Create the project directory structure
The evaluation platform separates concerns into distinct modules. The evaluation engine, agents, tools, and test datasets each live in their own directory.
- Create all the directories and initialization files by running these commands:
mkdir -p src/tools src/agents src/evaluation datasets
touch src/__init__.py src/tools/__init__.py src/agents/__init__.py src/evaluation/__init__.pyWhat does this structure do?
- src/evaluation/ holds the scoring engine and metrics that measure agent performance.
- src/agents/ holds the agents being evaluated (starting with the IT Help Desk agent).
- src/tools/ holds the tool definitions and simulated implementations the agent can call.
- datasets/ holds the structured test cases you run your agent against.
- The __init__.py files mark each directory as a Python package so modules can import from each other.
- Confirm the files exist by running:
ls src/ src/tools/ src/agents/ src/evaluation/ datasets/You should see __init__.py listed under each src/ subdirectory, and the datasets/ directory should be empty.
Seeing 'No such file or directory'?
Make sure you are in the agent-eval-platform folder. Run pwd to check your current location. If you are somewhere else, run cd ~/Desktop/agent-eval-platform to get back.
Help me fix my directory structure
Verify the setup with a test API call
Before building anything further, confirm that your SDK installation and API key are working together. You will send a simple message to Claude and check for a response.
- Run this test command in your terminal:
python3 -c "
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model='claude-sonnet-4-5-20250929',
max_tokens=50,
messages=[{'role': 'user', 'content': 'Say hello in exactly 5 words.'}]
)
print(response.content[0].text)
"You should see a short five-word greeting printed in your terminal. The exact words will vary each time.
What does this test do?
This creates an Anthropic client (which automatically reads your ANTHROPIC_API_KEY), sends a message to the claude-sonnet-4-5-20250929 model, and prints Claude's response. If it prints text without errors, your SDK and API key are both working.
Seeing an authentication error?
Double check that you set ANTHROPIC_API_KEY in the same terminal session where you are running the test. If you opened a new terminal tab, you need to set the variable again.
Make sure your key starts with sk-ant- and has no extra spaces or quotes around it.
Help me fix my Claude API authentication
✔️ Awesome, I've got everything!
Great. Double check you have saved requirements.txt in your agent-eval-platform folder.
ⓧ I'd like to double check the full code
Here is the complete requirements.txt file:
anthropic==0.116.0Your evaluation platform foundation is ready. Next up, you will build the IT Help Desk tools that give your agent something to call.
Set Up the Project Structure
Your SDK is installed and Claude is responding to API calls. Now it's time to give the project a proper architecture.
A production evaluation platform separates concerns into distinct modules so each piece can evolve independently. You'll create directories that keep the evaluation engine, agents, tools, and test datasets in their own spaces.
In this step, get ready to:
- Create the project directory structure with separate modules for evaluation, agents, and tools.
- Add Python package files so each module is importable.
Create the directories and package files
The platform follows a modular layout with four top-level concerns. Each Python module gets its own directory with an __init__.py file that marks it as an importable package.
- Create the full directory structure by running the commands in the tab that matches your operating system:
macOS
mkdir -p src/tools src/agents src/evaluation datasets
touch src/__init__.py src/tools/__init__.py src/agents/__init__.py src/evaluation/__init__.pyWindows
New-Item -ItemType Directory -Path src/tools, src/agents, src/evaluation, datasets -Force
New-Item -ItemType File -Path src/__init__.py, src/tools/__init__.py, src/agents/__init__.py, src/evaluation/__init__.pyWhat does this structure do?
- src/tools/ holds the simulated enterprise tools that agents will call.
- src/agents/ holds the AI agents that get evaluated.
- src/evaluation/ holds the scoring engine and metrics.
- datasets/ holds the structured test cases the engine runs agents against.
This separation means you can swap in a different agent or add new metrics without touching unrelated code.
- Confirm the directories and files were created by running:
lsYou should see datasets, requirements.txt, and src in the output.
- Verify the contents of the src/ directory by running:
ls src/You should see __init__.py, agents, evaluation, and tools listed in the output.
Not seeing the expected directories?
- Make sure you ran the commands from your project root directory (the folder containing requirements.txt).
- On Windows, confirm you are using PowerShell, not Command Prompt. The New-Item command does not work in CMD.
- If you accidentally created the directories in the wrong location, delete them and re-run the commands after running cd back into your project root.
Still stuck? Help me figure out why my project directories weren't created correctly.
Your project skeleton is in place. Next up, you'll fill the src/tools/ module with the five simulated IT help desk tools that your agent will call.
Build the IT Help Desk Tools
Before an AI agent can resolve support tickets, it needs tools to work with. In a production environment, these tools would connect to real systems like Active Directory or Jira. For our evaluation platform, we need simulated tools that return predictable, deterministic responses so we can score the agent's behavior reliably.
In this step, you'll define 5 IT help desk tools as JSON schemas that Claude can understand, build simulated data to power them, and wire up a dispatcher function that executes the right tool for each call.
In this step, get ready to:
- Define 5 tool schemas that tell Claude what tools are available and what arguments they accept.
- Create simulated enterprise data for users, servers, and documentation.
- Implement a tool execution dispatcher that routes tool calls to the correct simulated implementation.
Define the tool schemas
Claude's tool calling works by receiving JSON schemas that describe each tool's name, purpose, and required arguments. When Claude decides to use a tool, it returns a structured request matching the schema you defined. The quality of these schemas directly affects how accurately the agent selects tools.
- Create a new file called helpdesk.py inside the src/tools/ folder.
- Add the imports and tool schema definitions by pasting this code:
import json
from typing import Any
HELPDESK_TOOLS = [
{
"name": "reset_password",
"description": "Reset a user's password and send them a temporary password via email. Requires the username of the account to reset.",
"input_schema": {
"type": "object",
"properties": {
"username": {
"type": "string",
"description": "The username of the account to reset the password for"
}
},
"required": ["username"]
}
},
{
"name": "unlock_account",
"description": "Unlock a user account that has been locked due to too many failed login attempts. Requires the username.",
"input_schema": {
"type": "object",
"properties": {
"username": {
"type": "string",
"description": "The username of the account to unlock"
}
},
"required": ["username"]
}
},
{
"name": "check_server_status",
"description": "Check the current status of a server including uptime, CPU usage, and whether it is responding to requests. Requires the server name.",
"input_schema": {
"type": "object",
"properties": {
"server_name": {
"type": "string",
"description": "The name of the server to check (e.g., 'prod-api-01', 'staging-db-02')"
}
},
"required": ["server_name"]
}
},
{
"name": "search_documentation",
"description": "Search the internal IT documentation knowledge base for articles matching a query. Returns relevant article titles and summaries.",
"input_schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The search query to find relevant documentation"
}
},
"required": ["query"]
}
},
{
"name": "create_ticket",
"description": "Create a new support ticket in the ticketing system for issues that require further investigation or escalation.",
"input_schema": {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "Brief title describing the issue"
},
"description": {
"type": "string",
"description": "Detailed description of the issue and steps already taken"
},
"priority": {
"type": "string",
"enum": ["low", "medium", "high", "critical"],
"description": "Priority level of the ticket"
}
},
"required": ["title", "description", "priority"]
}
}
]What do these schemas do?
Each tool schema has three parts that Claude reads when deciding which tool to call:
- name tells Claude the identifier to use when calling the tool.
- description helps Claude decide WHEN to use this tool based on the user's request.
- input_schema defines what arguments the tool expects, so Claude can extract the right values from the conversation.
- Save helpdesk.py.
- Verify Python can parse the file by running this command in your terminal:
python -c "from src.tools.helpdesk import HELPDESK_TOOLS; print(len(HELPDESK_TOOLS))"You should see 5 printed in the terminal, confirming all 5 tool schemas loaded correctly.
Seeing a SyntaxError or IndentationError?
JSON-like Python dicts are sensitive to missing commas between items and mismatched brackets. Check that every { has a matching } and that commas separate each tool dict in the list.
help me debug a SyntaxError in my HELPDESK_TOOLS list
Create the simulated enterprise data
Real IT tools query databases and APIs. For evaluation, you need predictable responses so you can verify the agent passes the correct arguments. A simulated data layer gives you deterministic outputs for every input, which is essential for repeatable scoring.
- In src/tools/helpdesk.py, add the simulated data dictionary below the HELPDESK_TOOLS list by pasting this code:
SIMULATED_DATA = {
"users": {
"jsmith": {"name": "John Smith", "email": "jsmith@company.com", "locked": True},
"adoe": {"name": "Alice Doe", "email": "adoe@company.com", "locked": False},
"bwilson": {"name": "Bob Wilson", "email": "bwilson@company.com", "locked": True},
},
"servers": {
"prod-api-01": {"status": "healthy", "uptime": "45 days", "cpu": "23%"},
"staging-db-02": {"status": "degraded", "uptime": "2 days", "cpu": "89%"},
"prod-web-03": {"status": "down", "uptime": "0 days", "cpu": "0%"},
},
"docs": {
"vpn": {"title": "VPN Setup Guide", "summary": "Steps to configure VPN access for remote workers. Requires submitting a VPN access request form to IT."},
"password": {"title": "Password Policy", "summary": "Passwords must be 12+ characters with uppercase, lowercase, number, and special character. Expires every 90 days."},
"mfa": {"title": "MFA Enrollment", "summary": "Multi-factor authentication setup using the company authenticator app. Required for all employees."},
}
}Why include edge cases in the data?
Notice that adoe has "locked": False while others are locked, and prod-web-03 is down while others are healthy or degraded. These edge cases let the evaluation engine test whether the agent handles non-standard responses correctly.
- Save helpdesk.py.
You won't see a visible change yet. The next chunk uses this data to return realistic tool responses.
Implement the tool execution dispatcher
When Claude calls a tool, your code receives the tool name and arguments. You need a dispatcher function that routes each call to the correct logic and returns a JSON response. This is the function the evaluation engine will trace to score argument correctness.
- Below the SIMULATED_DATA dictionary, add the execute_tool function by pasting this code:
def execute_tool(tool_name: str, arguments: dict[str, Any]) -> str:
if tool_name == "reset_password":
username = arguments.get("username", "")
user = SIMULATED_DATA["users"].get(username)
if user is None:
return json.dumps({"success": False, "error": f"User '{username}' not found in the system"})
return json.dumps({"success": True, "message": f"Password reset for {user['name']}. Temporary password sent to {user['email']}"})
elif tool_name == "unlock_account":
username = arguments.get("username", "")
user = SIMULATED_DATA["users"].get(username)
if user is None:
return json.dumps({"success": False, "error": f"User '{username}' not found in the system"})
if not user["locked"]:
return json.dumps({"success": True, "message": f"Account '{username}' is not locked. No action needed."})
return json.dumps({"success": True, "message": f"Account '{username}' has been unlocked successfully"})
elif tool_name == "check_server_status":
server_name = arguments.get("server_name", "")
server = SIMULATED_DATA["servers"].get(server_name)
if server is None:
return json.dumps({"success": False, "error": f"Server '{server_name}' not found in inventory"})
return json.dumps({"success": True, "server": server_name, "status": server["status"], "uptime": server["uptime"], "cpu_usage": server["cpu"]})
elif tool_name == "search_documentation":
query = arguments.get("query", "").lower()
results = []
for key, doc in SIMULATED_DATA["docs"].items():
if key in query or any(word in query for word in doc["title"].lower().split()):
results.append(doc)
if not results:
return json.dumps({"success": True, "results": [], "message": "No documentation found matching your query"})
return json.dumps({"success": True, "results": results})
elif tool_name == "create_ticket":
title = arguments.get("title", "Untitled")
priority = arguments.get("priority", "medium")
return json.dumps({"success": True, "ticket_id": "TKT-2024-0891", "title": title, "priority": priority, "message": "Ticket created and assigned to IT support queue"})
else:
return json.dumps({"success": False, "error": f"Unknown tool: {tool_name}"})What does this function do?
The execute_tool function acts as a dispatcher. It receives the tool name and arguments, routes to the correct branch, looks up data from SIMULATED_DATA, and returns a JSON string.
- Every branch handles a "not found" edge case (unknown user, unknown server) so the agent must deal with error responses.
- The unlock_account branch also checks whether the account is actually locked, testing if the agent handles "no action needed" gracefully.
- The final else clause catches unknown tools, which protects against hallucinated tool names.
- Save helpdesk.py.
- Test the dispatcher end-to-end by running this command:
python -c "from src.tools.helpdesk import execute_tool; print(execute_tool('unlock_account', {'username': 'jsmith'}))"You should see JSON output containing "success": true and a message confirming the account was unlocked. This proves the full chain works: tool name routes to the correct branch, looks up the user in simulated data, and returns a structured response.
Seeing a NameError or ImportError?
Make sure you're running the command from your project's root directory (the folder containing the src/ folder). Python needs to find the src package from the current directory.
Also confirm the src/__init__.py and src/tools/__init__.py files exist. Python requires these to treat the folders as importable packages.
help me debug an ImportError when importing execute_tool from src.tools.helpdesk
✔️ Awesome, I've got everything!
Great. Double check you've saved src/tools/helpdesk.py and that your test command returned a successful JSON response.
ⓧ I'd like to double check the full code
Here's the complete src/tools/helpdesk.py file. Make sure yours matches exactly:
import json
from typing import Any
HELPDESK_TOOLS = [
{
"name": "reset_password",
"description": "Reset a user's password and send them a temporary password via email. Requires the username of the account to reset.",
"input_schema": {
"type": "object",
"properties": {
"username": {
"type": "string",
"description": "The username of the account to reset the password for"
}
},
"required": ["username"]
}
},
{
"name": "unlock_account",
"description": "Unlock a user account that has been locked due to too many failed login attempts. Requires the username.",
"input_schema": {
"type": "object",
"properties": {
"username": {
"type": "string",
"description": "The username of the account to unlock"
}
},
"required": ["username"]
}
},
{
"name": "check_server_status",
"description": "Check the current status of a server including uptime, CPU usage, and whether it is responding to requests. Requires the server name.",
"input_schema": {
"type": "object",
"properties": {
"server_name": {
"type": "string",
"description": "The name of the server to check (e.g., 'prod-api-01', 'staging-db-02')"
}
},
"required": ["server_name"]
}
},
{
"name": "search_documentation",
"description": "Search the internal IT documentation knowledge base for articles matching a query. Returns relevant article titles and summaries.",
"input_schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The search query to find relevant documentation"
}
},
"required": ["query"]
}
},
{
"name": "create_ticket",
"description": "Create a new support ticket in the ticketing system for issues that require further investigation or escalation.",
"input_schema": {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "Brief title describing the issue"
},
"description": {
"type": "string",
"description": "Detailed description of the issue and steps already taken"
},
"priority": {
"type": "string",
"enum": ["low", "medium", "high", "critical"],
"description": "Priority level of the ticket"
}
},
"required": ["title", "description", "priority"]
}
}
]
SIMULATED_DATA = {
"users": {
"jsmith": {"name": "John Smith", "email": "jsmith@company.com", "locked": True},
"adoe": {"name": "Alice Doe", "email": "adoe@company.com", "locked": False},
"bwilson": {"name": "Bob Wilson", "email": "bwilson@company.com", "locked": True},
},
"servers": {
"prod-api-01": {"status": "healthy", "uptime": "45 days", "cpu": "23%"},
"staging-db-02": {"status": "degraded", "uptime": "2 days", "cpu": "89%"},
"prod-web-03": {"status": "down", "uptime": "0 days", "cpu": "0%"},
},
"docs": {
"vpn": {"title": "VPN Setup Guide", "summary": "Steps to configure VPN access for remote workers. Requires submitting a VPN access request form to IT."},
"password": {"title": "Password Policy", "summary": "Passwords must be 12+ characters with uppercase, lowercase, number, and special character. Expires every 90 days."},
"mfa": {"title": "MFA Enrollment", "summary": "Multi-factor authentication setup using the company authenticator app. Required for all employees."},
}
}
def execute_tool(tool_name: str, arguments: dict[str, Any]) -> str:
if tool_name == "reset_password":
username = arguments.get("username", "")
user = SIMULATED_DATA["users"].get(username)
if user is None:
return json.dumps({"success": False, "error": f"User '{username}' not found in the system"})
return json.dumps({"success": True, "message": f"Password reset for {user['name']}. Temporary password sent to {user['email']}"})
elif tool_name == "unlock_account":
username = arguments.get("username", "")
user = SIMULATED_DATA["users"].get(username)
if user is None:
return json.dumps({"success": False, "error": f"User '{username}' not found in the system"})
if not user["locked"]:
return json.dumps({"success": True, "message": f"Account '{username}' is not locked. No action needed."})
return json.dumps({"success": True, "message": f"Account '{username}' has been unlocked successfully"})
elif tool_name == "check_server_status":
server_name = arguments.get("server_name", "")
server = SIMULATED_DATA["servers"].get(server_name)
if server is None:
return json.dumps({"success": False, "error": f"Server '{server_name}' not found in inventory"})
return json.dumps({"success": True, "server": server_name, "status": server["status"], "uptime": server["uptime"], "cpu_usage": server["cpu"]})
elif tool_name == "search_documentation":
query = arguments.get("query", "").lower()
results = []
for key, doc in SIMULATED_DATA["docs"].items():
if key in query or any(word in query for word in doc["title"].lower().split()):
results.append(doc)
if not results:
return json.dumps({"success": True, "results": [], "message": "No documentation found matching your query"})
return json.dumps({"success": True, "results": results})
elif tool_name == "create_ticket":
title = arguments.get("title", "Untitled")
priority = arguments.get("priority", "medium")
return json.dumps({"success": True, "ticket_id": "TKT-2024-0891", "title": title, "priority": priority, "message": "Ticket created and assigned to IT support queue"})
else:
return json.dumps({"success": False, "error": f"Unknown tool: {tool_name}"})Your IT help desk tools are ready. You have 5 tool schemas Claude can understand, simulated data that covers happy paths and edge cases, and a dispatcher that routes every tool call to the right logic. Next up, you'll build the agent that actually uses these tools to resolve support tickets.
Build the IT Help Desk Agent
Your tools are built and ready to execute. Now you need an agent that can decide which tool to call, when to call it, and how to use the result to respond to the user.
The agent is an agentic loop: it sends the user's message to Claude along with the available tool definitions, Claude decides whether to call a tool or respond directly, and the loop repeats until Claude produces a final answer. Every tool call along the way gets recorded in a trace object so the evaluation engine can score it later.
In this step, get ready to:
- Implement the agentic loop with Claude's tool calling API.
- Add trace capture to record every tool call for evaluation.
- Test the agent end-to-end with a sample IT support query.
Define the trace data structures
Before writing the loop itself, you need data structures to record what the agent does. The evaluation engine will inspect these traces to score the agent's behavior.
- In VS Code, open src/agents/helpdesk_agent.py (the empty file inside your src/agents/ folder).
- Add the imports and trace dataclasses by pasting this code:
import anthropic
from dataclasses import dataclass, field
from src.tools.helpdesk import HELPDESK_TOOLS, execute_tool
@dataclass
class ToolCall:
tool_name: str
arguments: dict
result: str
@dataclass
class AgentTrace:
user_query: str
tool_calls: list[ToolCall] = field(default_factory=list)
final_response: str = ""What does this code do?
- ToolCall stores a single tool invocation: which tool was called, what arguments were passed, and what the tool returned.
- AgentTrace stores the full execution record: the original user query, the list of all tool calls made, and the agent's final text response.
- The evaluation engine will receive an AgentTrace object and score it against expected behavior. This is why capturing tool calls matters.
Implement the agentic loop
The agentic loop is the core pattern for tool-calling agents. It sends a message to Claude, checks whether Claude wants to use a tool, executes that tool, feeds the result back, and repeats until Claude is ready to respond directly.
- Below the AgentTrace dataclass in src/agents/helpdesk_agent.py, add the run_helpdesk_agent function:
def run_helpdesk_agent(user_query: str, max_turns: int = 10) -> AgentTrace:
client = anthropic.Anthropic()
trace = AgentTrace(user_query=user_query)
system_prompt = (
"You are an IT Help Desk agent. You help employees with technical issues "
"by using the available tools. Always use tools to look up information or "
"take actions rather than guessing. If you cannot resolve an issue with "
"the available tools, create a support ticket for escalation. "
"Be concise and professional in your responses."
)
messages = [{"role": "user", "content": user_query}]
for _ in range(max_turns):
response = client.messages.create(
model="claude-sonnet-4-5-20250929",
max_tokens=1024,
system=system_prompt,
tools=HELPDESK_TOOLS,
messages=messages,
)
if response.stop_reason == "end_turn":
for block in response.content:
if block.type == "text":
trace.final_response = block.text
break
tool_use_blocks = [block for block in response.content if block.type == "tool_use"]
if not tool_use_blocks:
for block in response.content:
if block.type == "text":
trace.final_response = block.text
break
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for tool_block in tool_use_blocks:
result = execute_tool(tool_block.name, tool_block.input)
trace.tool_calls.append(
ToolCall(
tool_name=tool_block.name,
arguments=tool_block.input,
result=result,
)
)
tool_results.append({
"type": "tool_result",
"tool_use_id": tool_block.id,
"content": result,
})
messages.append({"role": "user", "content": tool_results})
return traceWhat does this code do?
This function implements the full agentic loop pattern:
- client.messages.create() sends the conversation to Claude along with the 5 tool definitions from HELPDESK_TOOLS.
- If response.stop_reason == "end_turn", Claude is done and has produced a final text response. The loop breaks.
- If the response contains tool_use blocks, the loop executes each tool using your execute_tool() function and records the call in the trace.
- The tool results are sent back as tool_result messages so Claude can decide what to do next.
- max_turns = 10 prevents infinite loops if the agent keeps calling tools without converging on an answer.
Why build the loop manually instead of using a helper?
The Anthropic SDK includes helper utilities that automate the agentic loop. However, those helpers do not expose intermediate tool calls for inspection.
Since the evaluation engine needs to score every tool call the agent makes, you need direct access to each tool_use block. Building the loop manually gives you full control over trace capture.
- Save src/agents/helpdesk_agent.py.
✔️ Awesome, I've got everything!
Great. Double check you have saved your file before moving on.
ⓧ I'd like to double check the full code
Here is the complete src/agents/helpdesk_agent.py file:
import anthropic
from dataclasses import dataclass, field
from src.tools.helpdesk import HELPDESK_TOOLS, execute_tool
@dataclass
class ToolCall:
tool_name: str
arguments: dict
result: str
@dataclass
class AgentTrace:
user_query: str
tool_calls: list[ToolCall] = field(default_factory=list)
final_response: str = ""
def run_helpdesk_agent(user_query: str, max_turns: int = 10) -> AgentTrace:
client = anthropic.Anthropic()
trace = AgentTrace(user_query=user_query)
system_prompt = (
"You are an IT Help Desk agent. You help employees with technical issues "
"by using the available tools. Always use tools to look up information or "
"take actions rather than guessing. If you cannot resolve an issue with "
"the available tools, create a support ticket for escalation. "
"Be concise and professional in your responses."
)
messages = [{"role": "user", "content": user_query}]
for _ in range(max_turns):
response = client.messages.create(
model="claude-sonnet-4-5-20250929",
max_tokens=1024,
system=system_prompt,
tools=HELPDESK_TOOLS,
messages=messages,
)
if response.stop_reason == "end_turn":
for block in response.content:
if block.type == "text":
trace.final_response = block.text
break
tool_use_blocks = [block for block in response.content if block.type == "tool_use"]
if not tool_use_blocks:
for block in response.content:
if block.type == "text":
trace.final_response = block.text
break
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for tool_block in tool_use_blocks:
result = execute_tool(tool_block.name, tool_block.input)
trace.tool_calls.append(
ToolCall(
tool_name=tool_block.name,
arguments=tool_block.input,
result=result,
)
)
tool_results.append({
"type": "tool_result",
"tool_use_id": tool_block.id,
"content": result,
})
messages.append({"role": "user", "content": tool_results})
return traceTest the agent end-to-end
Time to verify the agent can receive a user query, call the right tool, and capture the full execution trace.
- Run the agent with a sample IT support query by running this command in your terminal:
python -c "from src.agents.helpdesk_agent import run_helpdesk_agent; trace = run_helpdesk_agent('My account is locked, username is jsmith'); print(f'Tools called: {[tc.tool_name for tc in trace.tool_calls]}'); print(f'Response: {trace.final_response[:100]}')"What should I see?
You should see output similar to this:
- Tools called: ['unlock_account'] confirms Claude identified the correct tool from the user's request.
- The response text should mention that the account has been unlocked for jsmith.
The exact response wording will vary because Claude generates it dynamically. The key verification is that unlock_account appears in the tools list.
Seeing an authentication error?
- Make sure your ANTHROPIC_API_KEY environment variable is set in the same terminal session you are running the command from.
- If you see a ModuleNotFoundError, confirm your virtual environment is active and that you installed anthropic with pip install anthropic.
- Make sure you are running the command from your project root folder (the one containing the src/ directory).
Help me debug my helpdesk agent test
Your agent is working. It receives a user query, identifies the right tool, executes it against your simulated data, and produces a helpful response. Next up, you will build the evaluation engine that scores this agent's behavior automatically across an entire test dataset.
Build the Evaluation Engine
Your IT Help Desk agent can resolve tickets, but you have no way to measure how well it performs. Does it pick the right tools? Does it pass the correct arguments? Does its final response actually address the user's problem?
In this step, you will build a domain-agnostic evaluation engine with pluggable metrics. The engine separates scoring logic from agent logic, so you can swap in different agents or add new metrics without changing the core architecture.
In this step, get ready to:
- Create a Metric base class that defines the interface for all scoring metrics.
- Implement three concrete metrics for tool selection, argument correctness, and task completion.
- Build the EvaluationEngine class that runs any agent against a dataset and produces scored reports.
Create the Metric base class
Every evaluation metric needs a consistent interface. By defining a base class, you ensure that new metrics can be added later without changing the engine code.
- Create a new file called metrics.py inside the src/evaluation/ folder.
- Add the following code to define the MetricResult dataclass and the Metric base class:
import json
from dataclasses import dataclass
from src.agents.helpdesk_agent import AgentTrace
@dataclass
class MetricResult:
metric_name: str
score: float
details: str
class Metric:
name: str = "base_metric"
def score(self, trace: AgentTrace, test_case: dict) -> MetricResult:
raise NotImplementedErrorWhat does this code do?
The MetricResult dataclass stores three pieces of information for every scoring run: which metric produced the score, the numeric score itself (0.0 to 1.0), and a human-readable string explaining why.
The Metric base class defines a single method, score(), that accepts an AgentTrace and a test case dictionary. Every concrete metric must override this method.
- Save metrics.py.
Why separate metrics from the engine?
Keeping each metric as its own class means you can add or remove scoring dimensions without touching the evaluation loop. When you add hallucination detection in the Secret Mission, you will drop in a new class and register it. The engine does not need to change.
Implement the three scoring metrics
Each metric scores one dimension of agent behavior. Measuring dimensions separately reveals which layer is breaking. A composite score would hide whether the agent picks the wrong tool or passes the wrong arguments.
- In src/evaluation/metrics.py, add the ToolSelectionAccuracy class below the Metric base class:
class ToolSelectionAccuracy(Metric):
name = "tool_selection_accuracy"
def score(self, trace: AgentTrace, test_case: dict) -> MetricResult:
expected_tools = test_case.get("expected_tools", [])
actual_tools = [tc.tool_name for tc in trace.tool_calls]
if not expected_tools:
score = 1.0 if not actual_tools else 0.0
return MetricResult(
metric_name=self.name,
score=score,
details=f"Expected no tool calls. Agent called: {actual_tools}"
)
expected_set = set(expected_tools)
actual_set = set(actual_tools)
correct = expected_set.intersection(actual_set)
missing = expected_set - actual_set
extra = actual_set - expected_set
precision = len(correct) / len(actual_set) if actual_set else 0.0
recall = len(correct) / len(expected_set) if expected_set else 0.0
if precision + recall == 0:
f1 = 0.0
else:
f1 = 2 * (precision * recall) / (precision + recall)
details = f"Expected: {expected_tools}, Actual: {actual_tools}"
if missing:
details += f", Missing: {list(missing)}"
if extra:
details += f", Extra: {list(extra)}"
return MetricResult(metric_name=self.name, score=f1, details=details)What does this metric measure?
ToolSelectionAccuracy compares the set of tools the agent actually called against the set of tools the test case expected. It uses an F1 score (combining precision and recall) so that both calling extra tools and missing expected tools reduce the score.
For example, if the test expects unlock_account and reset_password, but the agent only calls unlock_account, the recall drops to 0.5.
- Save metrics.py.
- Below ToolSelectionAccuracy, add the ArgumentCorrectness class:
class ArgumentCorrectness(Metric):
name = "argument_correctness"
def score(self, trace: AgentTrace, test_case: dict) -> MetricResult:
expected_args = test_case.get("expected_args", {})
if not expected_args:
return MetricResult(
metric_name=self.name,
score=1.0,
details="No expected arguments specified"
)
total_checks = 0
correct_checks = 0
issues = []
for tool_name, expected_params in expected_args.items():
matching_calls = [tc for tc in trace.tool_calls if tc.tool_name == tool_name]
if not matching_calls:
total_checks += len(expected_params)
issues.append(f"Tool '{tool_name}' was never called")
continue
actual_args = matching_calls[0].arguments
for param_name, expected_value in expected_params.items():
total_checks += 1
actual_value = actual_args.get(param_name)
if actual_value is not None and str(actual_value).lower() == str(expected_value).lower():
correct_checks += 1
else:
issues.append(f"{tool_name}.{param_name}: expected '{expected_value}', got '{actual_value}'")
score = correct_checks / total_checks if total_checks > 0 else 1.0
details = f"Correct: {correct_checks}/{total_checks}"
if issues:
details += f". Issues: {'; '.join(issues)}"
return MetricResult(metric_name=self.name, score=score, details=details)What does this metric measure?
ArgumentCorrectness checks whether the agent passed the right values to each tool. It iterates over every expected parameter in the test case and compares it against what the agent actually sent.
This catches a common failure mode: the agent picks the right tool but extracts the wrong value from the user's message (e.g., calling unlock_account with username: "john" instead of username: "jsmith").
- Save metrics.py.
- Below ArgumentCorrectness, add the TaskCompletion class:
class TaskCompletion(Metric):
name = "task_completion"
def score(self, trace: AgentTrace, test_case: dict) -> MetricResult:
expected_outcome_keywords = test_case.get("expected_outcome_keywords", [])
if not expected_outcome_keywords:
score = 1.0 if trace.final_response else 0.0
return MetricResult(
metric_name=self.name,
score=score,
details="No specific outcome keywords to check. Agent responded: " + ("yes" if trace.final_response else "no")
)
response_lower = trace.final_response.lower()
found = []
missing = []
for keyword in expected_outcome_keywords:
if keyword.lower() in response_lower:
found.append(keyword)
else:
missing.append(keyword)
score = len(found) / len(expected_outcome_keywords)
details = f"Found {len(found)}/{len(expected_outcome_keywords)} keywords"
if missing:
details += f". Missing: {missing}"
return MetricResult(metric_name=self.name, score=score, details=details)What does this metric measure?
TaskCompletion checks whether the agent's final text response actually addresses the user's problem. It looks for expected keywords in the response (e.g., "unlocked" and "jsmith" for an account unlock request).
Keyword matching is a deterministic proxy. It catches cases where the agent calls the right tool but fails to communicate the result clearly to the user.
- Save metrics.py.
Seeing a red underline on the import?
If VS Code shows an import error on from src.agents.helpdesk_agent import AgentTrace, make sure you are opening the project root folder (the one containing the src/ directory) in VS Code. The Python path resolver needs to see the full package structure.
help me fix this import error in metrics.py
✔️ Awesome, I've got everything!
Great. Make sure you have saved src/evaluation/metrics.py with all four classes (MetricResult, Metric, ToolSelectionAccuracy, ArgumentCorrectness, TaskCompletion).
ⓧ I'd like to double check the full code
import json
from dataclasses import dataclass
from src.agents.helpdesk_agent import AgentTrace
@dataclass
class MetricResult:
metric_name: str
score: float
details: str
class Metric:
name: str = "base_metric"
def score(self, trace: AgentTrace, test_case: dict) -> MetricResult:
raise NotImplementedError
class ToolSelectionAccuracy(Metric):
name = "tool_selection_accuracy"
def score(self, trace: AgentTrace, test_case: dict) -> MetricResult:
expected_tools = test_case.get("expected_tools", [])
actual_tools = [tc.tool_name for tc in trace.tool_calls]
if not expected_tools:
score = 1.0 if not actual_tools else 0.0
return MetricResult(
metric_name=self.name,
score=score,
details=f"Expected no tool calls. Agent called: {actual_tools}"
)
expected_set = set(expected_tools)
actual_set = set(actual_tools)
correct = expected_set.intersection(actual_set)
missing = expected_set - actual_set
extra = actual_set - expected_set
precision = len(correct) / len(actual_set) if actual_set else 0.0
recall = len(correct) / len(expected_set) if expected_set else 0.0
if precision + recall == 0:
f1 = 0.0
else:
f1 = 2 * (precision * recall) / (precision + recall)
details = f"Expected: {expected_tools}, Actual: {actual_tools}"
if missing:
details += f", Missing: {list(missing)}"
if extra:
details += f", Extra: {list(extra)}"
return MetricResult(metric_name=self.name, score=f1, details=details)
class ArgumentCorrectness(Metric):
name = "argument_correctness"
def score(self, trace: AgentTrace, test_case: dict) -> MetricResult:
expected_args = test_case.get("expected_args", {})
if not expected_args:
return MetricResult(
metric_name=self.name,
score=1.0,
details="No expected arguments specified"
)
total_checks = 0
correct_checks = 0
issues = []
for tool_name, expected_params in expected_args.items():
matching_calls = [tc for tc in trace.tool_calls if tc.tool_name == tool_name]
if not matching_calls:
total_checks += len(expected_params)
issues.append(f"Tool '{tool_name}' was never called")
continue
actual_args = matching_calls[0].arguments
for param_name, expected_value in expected_params.items():
total_checks += 1
actual_value = actual_args.get(param_name)
if actual_value is not None and str(actual_value).lower() == str(expected_value).lower():
correct_checks += 1
else:
issues.append(f"{tool_name}.{param_name}: expected '{expected_value}', got '{actual_value}'")
score = correct_checks / total_checks if total_checks > 0 else 1.0
details = f"Correct: {correct_checks}/{total_checks}"
if issues:
details += f". Issues: {'; '.join(issues)}"
return MetricResult(metric_name=self.name, score=score, details=details)
class TaskCompletion(Metric):
name = "task_completion"
def score(self, trace: AgentTrace, test_case: dict) -> MetricResult:
expected_outcome_keywords = test_case.get("expected_outcome_keywords", [])
if not expected_outcome_keywords:
score = 1.0 if trace.final_response else 0.0
return MetricResult(
metric_name=self.name,
score=score,
details="No specific outcome keywords to check. Agent responded: " + ("yes" if trace.final_response else "no")
)
response_lower = trace.final_response.lower()
found = []
missing = []
for keyword in expected_outcome_keywords:
if keyword.lower() in response_lower:
found.append(keyword)
else:
missing.append(keyword)
score = len(found) / len(expected_outcome_keywords)
details = f"Found {len(found)}/{len(expected_outcome_keywords)} keywords"
if missing:
details += f". Missing: {missing}"
return MetricResult(metric_name=self.name, score=score, details=details)Build the EvaluationEngine class
The engine is the orchestrator. It loads test cases from a JSON dataset, runs the agent on each case, applies every registered metric to the resulting trace, and aggregates the scores into a report.
- Create a new file called engine.py inside the src/evaluation/ folder.
- Add the imports and data classes that define the shape of evaluation results:
import json
import time
from dataclasses import dataclass, field
from pathlib import Path
from src.agents.helpdesk_agent import AgentTrace, run_helpdesk_agent
from src.evaluation.metrics import Metric, MetricResult
@dataclass
class TestCaseResult:
test_case_id: str
user_query: str
metric_results: list[MetricResult] = field(default_factory=list)
execution_time: float = 0.0
@dataclass
class EvaluationReport:
total_cases: int
results: list[TestCaseResult] = field(default_factory=list)
aggregate_scores: dict[str, float] = field(default_factory=dict)What do these data classes hold?
A TestCaseResult stores the outcome of running one test case: which case it was, all metric scores for that case, and how long the agent took to respond.
An EvaluationReport collects all case results together with aggregate scores (the average of each metric across all cases). This is what gets printed at the end of an evaluation run.
- Save engine.py.
- Below the data classes in engine.py, add the EvaluationEngine class:
class EvaluationEngine:
def __init__(self, metrics: list[Metric]):
self.metrics = metrics
def load_dataset(self, dataset_path: str) -> list[dict]:
path = Path(dataset_path)
with open(path, "r") as f:
return json.load(f)
def evaluate_trace(self, trace: AgentTrace, test_case: dict) -> list[MetricResult]:
results = []
for metric in self.metrics:
result = metric.score(trace, test_case)
results.append(result)
return results
def run_evaluation(self, dataset_path: str) -> EvaluationReport:
dataset = self.load_dataset(dataset_path)
report = EvaluationReport(total_cases=len(dataset))
for test_case in dataset:
start_time = time.time()
trace = run_helpdesk_agent(test_case["user_query"])
metric_results = self.evaluate_trace(trace, test_case)
execution_time = time.time() - start_time
case_result = TestCaseResult(
test_case_id=test_case["id"],
user_query=test_case["user_query"],
metric_results=metric_results,
execution_time=execution_time,
)
report.results.append(case_result)
print(f" [{test_case['id']}] completed in {execution_time:.1f}s")
score_sums: dict[str, float] = {}
score_counts: dict[str, int] = {}
for case_result in report.results:
for metric_result in case_result.metric_results:
name = metric_result.metric_name
score_sums[name] = score_sums.get(name, 0.0) + metric_result.score
score_counts[name] = score_counts.get(name, 0) + 1
for name in score_sums:
report.aggregate_scores[name] = score_sums[name] / score_counts[name]
return reportWhat does the engine do?
- load_dataset() reads a JSON file containing structured test cases.
- evaluate_trace() runs every registered metric against a single agent trace and returns the list of scores.
- run_evaluation() orchestrates the full loop: load the dataset, run the agent on each case, score it, time it, and compute aggregate averages across all cases.
- Save engine.py.
Seeing import errors between metrics and engine?
Both files live in src/evaluation/ and import from src.agents. Make sure every __init__.py file exists (they can be empty). You should have one in src/, src/agents/, src/tools/, and src/evaluation/.
help me fix circular imports or missing __init__.py issues in my evaluation platform
✔️ Awesome, I've got everything!
Great. Make sure you have saved src/evaluation/engine.py with both the data classes and the EvaluationEngine class.
ⓧ I'd like to double check the full code
import json
import time
from dataclasses import dataclass, field
from pathlib import Path
from src.agents.helpdesk_agent import AgentTrace, run_helpdesk_agent
from src.evaluation.metrics import Metric, MetricResult
@dataclass
class TestCaseResult:
test_case_id: str
user_query: str
metric_results: list[MetricResult] = field(default_factory=list)
execution_time: float = 0.0
@dataclass
class EvaluationReport:
total_cases: int
results: list[TestCaseResult] = field(default_factory=list)
aggregate_scores: dict[str, float] = field(default_factory=dict)
class EvaluationEngine:
def __init__(self, metrics: list[Metric]):
self.metrics = metrics
def load_dataset(self, dataset_path: str) -> list[dict]:
path = Path(dataset_path)
with open(path, "r") as f:
return json.load(f)
def evaluate_trace(self, trace: AgentTrace, test_case: dict) -> list[MetricResult]:
results = []
for metric in self.metrics:
result = metric.score(trace, test_case)
results.append(result)
return results
def run_evaluation(self, dataset_path: str) -> EvaluationReport:
dataset = self.load_dataset(dataset_path)
report = EvaluationReport(total_cases=len(dataset))
for test_case in dataset:
start_time = time.time()
trace = run_helpdesk_agent(test_case["user_query"])
metric_results = self.evaluate_trace(trace, test_case)
execution_time = time.time() - start_time
case_result = TestCaseResult(
test_case_id=test_case["id"],
user_query=test_case["user_query"],
metric_results=metric_results,
execution_time=execution_time,
)
report.results.append(case_result)
print(f" [{test_case['id']}] completed in {execution_time:.1f}s")
score_sums: dict[str, float] = {}
score_counts: dict[str, int] = {}
for case_result in report.results:
for metric_result in case_result.metric_results:
name = metric_result.metric_name
score_sums[name] = score_sums.get(name, 0.0) + metric_result.score
score_counts[name] = score_counts.get(name, 0) + 1
for name in score_sums:
report.aggregate_scores[name] = score_sums[name] / score_counts[name]
return reportYour evaluation engine is ready to score any agent against any dataset. In the next step, you will create the test dataset and run the full evaluation suite to see exactly where your Help Desk agent succeeds and fails.
Create the Test Dataset and Run Evaluations
Your evaluation engine is built with three pluggable metrics ready to score any agent trace. The missing piece is a structured test dataset that exercises your agent across different scenarios and a runner that ties everything together.
An evaluation is only as good as its test cases. If you only test happy paths, you will never catch the failures that matter in production. In this step, you will design test cases that cover single-tool calls, multi-tool sequences, error handling, and edge cases. Then you will build a CLI runner and execute the full suite to see exactly where your agent succeeds and fails.
In this step, get ready to:
- Create 8 structured test cases covering single-tool, multi-tool, error, and edge-case scenarios.
- Build a CLI runner that orchestrates the agent, applies metrics, and prints a formatted report.
- Run the full evaluation suite and analyze per-metric results.
Create the evaluation test dataset
A good evaluation dataset exercises multiple dimensions of agent behavior. You need cases where the agent should call a single tool, cases requiring multi-tool sequences, cases where the tool returns an error, and cases where the agent encounters unexpected input.
- Create a datasets directory in your project root by running this command:
mkdir datasets- Create a new file called helpdesk_eval.json inside the datasets folder and paste the following test cases:
[
{
"id": "tc_001",
"user_query": "My account is locked. My username is jsmith.",
"expected_tools": ["unlock_account"],
"expected_args": {"unlock_account": {"username": "jsmith"}},
"expected_outcome_keywords": ["unlocked", "jsmith"],
"tags": ["account_management", "single_tool"]
},
{
"id": "tc_002",
"user_query": "I forgot my password and I think my account is locked too. Username is bwilson.",
"expected_tools": ["unlock_account", "reset_password"],
"expected_args": {"unlock_account": {"username": "bwilson"}, "reset_password": {"username": "bwilson"}},
"expected_outcome_keywords": ["unlocked", "password", "reset"],
"tags": ["account_management", "multi_tool"]
},
{
"id": "tc_003",
"user_query": "Can you check if prod-api-01 is running okay?",
"expected_tools": ["check_server_status"],
"expected_args": {"check_server_status": {"server_name": "prod-api-01"}},
"expected_outcome_keywords": ["healthy", "prod-api-01"],
"tags": ["infrastructure", "single_tool"]
},
{
"id": "tc_004",
"user_query": "The staging database server staging-db-02 seems slow. What's going on?",
"expected_tools": ["check_server_status"],
"expected_args": {"check_server_status": {"server_name": "staging-db-02"}},
"expected_outcome_keywords": ["degraded", "cpu"],
"tags": ["infrastructure", "single_tool"]
},
{
"id": "tc_005",
"user_query": "How do I set up VPN access?",
"expected_tools": ["search_documentation"],
"expected_args": {"search_documentation": {"query": "vpn"}},
"expected_outcome_keywords": ["vpn", "access"],
"tags": ["documentation", "single_tool"]
},
{
"id": "tc_006",
"user_query": "prod-web-03 is down and it's affecting customers. This is urgent!",
"expected_tools": ["check_server_status", "create_ticket"],
"expected_args": {"check_server_status": {"server_name": "prod-web-03"}},
"expected_outcome_keywords": ["down", "ticket"],
"tags": ["infrastructure", "multi_tool", "escalation"]
},
{
"id": "tc_007",
"user_query": "Reset the password for user xyz_nonexistent please.",
"expected_tools": ["reset_password"],
"expected_args": {"reset_password": {"username": "xyz_nonexistent"}},
"expected_outcome_keywords": ["not found"],
"tags": ["account_management", "error_handling"]
},
{
"id": "tc_008",
"user_query": "Unlock the account for adoe please.",
"expected_tools": ["unlock_account"],
"expected_args": {"unlock_account": {"username": "adoe"}},
"expected_outcome_keywords": ["not locked"],
"tags": ["account_management", "edge_case"]
}
]What does this dataset cover?
Each test case defines four evaluation dimensions:
- expected_tools tells the ToolSelectionAccuracy metric which tools the agent should call.
- expected_args tells ArgumentCorrectness what parameter values to check.
- expected_outcome_keywords tells TaskCompletion what words should appear in the agent's final response.
- tags categorize each case for filtering and analysis later.
Cases tc_007 and tc_008 are designed to test edge cases. The agent must gracefully handle a nonexistent user and an account that is already unlocked.
- Save datasets/helpdesk_eval.json.
- Confirm the file exists by running:
ls datasets/You should see helpdesk_eval.json listed in the output.
Don't see the file?
- Make sure you created the datasets directory first with mkdir datasets.
- Verify you saved the file inside the datasets/ folder, not in the project root.
Help me find where my dataset file ended up.
Build the CLI runner
The CLI runner is the entry point that wires everything together. It instantiates the metrics, creates the evaluation engine, runs the agent against every test case, and prints a human-readable report.
- Create a new file called run_eval.py in your project root (the same level as requirements.txt).
- Add the imports and report formatting function by pasting this code:
from src.evaluation.engine import EvaluationEngine
from src.evaluation.metrics import ToolSelectionAccuracy, ArgumentCorrectness, TaskCompletion
def print_report(report):
# Print the evaluation report header
print("\n" + "=" * 70)
print(" AI AGENT EVALUATION REPORT")
print("=" * 70)
# Summary stats
print(f"\n Total test cases: {report.total_cases}")
print(f" Metrics evaluated: {len(report.aggregate_scores)}")
# Aggregate scores with visual bar chart
print("\n AGGREGATE SCORES")
print(" " + "-" * 50)
for metric_name, score in report.aggregate_scores.items():
bar_length = int(score * 30)
bar = "#" * bar_length + "." * (30 - bar_length)
print(f" {metric_name:<25} [{bar}] {score:.2f}")
# Per-case breakdown showing pass/fail per metric
print("\n PER-CASE RESULTS")
print(" " + "-" * 50)
for case_result in report.results:
print(f"\n [{case_result.test_case_id}] {case_result.user_query[:50]}...")
print(f" Time: {case_result.execution_time:.1f}s")
for metric_result in case_result.metric_results:
status = "PASS" if metric_result.score >= 0.8 else "FAIL"
print(f" {metric_result.metric_name:<25} {metric_result.score:.2f} {status}")
if metric_result.score < 0.8:
print(f" -> {metric_result.details}")
print("\n" + "=" * 70)What does print_report do?
- It prints aggregate scores as a visual bar chart so you can see overall performance at a glance.
- For each test case, it shows PASS (score >= 0.8) or FAIL with detailed diagnostics for any failing metric.
- The details line only prints for failures, making it easy to focus on what went wrong.
- Add the main() function below print_report by pasting this code at the bottom of the file:
def main():
print("Starting AI Agent Evaluation Platform...")
print("Running IT Help Desk agent evaluation suite\n")
# Register the three scoring metrics
metrics = [
ToolSelectionAccuracy(),
ArgumentCorrectness(),
TaskCompletion(),
]
# Create the evaluation engine with registered metrics
engine = EvaluationEngine(metrics=metrics)
# Run the agent against every test case and score results
report = engine.run_evaluation("datasets/helpdesk_eval.json")
# Print the formatted report
print_report(report)
if __name__ == "__main__":
main()What does main do?
- It instantiates all three metrics (ToolSelectionAccuracy, ArgumentCorrectness, TaskCompletion) and passes them to the EvaluationEngine.
- run_evaluation loads the dataset, runs the agent on each case, scores with all metrics, and returns an EvaluationReport.
- The if __name__ == "__main__" guard ensures main() only runs when you execute the file directly.
- Save run_eval.py.
Seeing an import error?
- Make sure all __init__.py files exist in src/, src/evaluation/, src/agents/, and src/tools/. These files can be empty but they must exist for Python to treat those directories as packages.
- Confirm you are running from the project root directory (the folder containing run_eval.py and the src/ folder).
Help me fix my import errors in run_eval.py.
✔️ Awesome, I've got everything!
Great. Make sure you have saved run_eval.py before moving on.
ⓧ I'd like to double check the full code
from src.evaluation.engine import EvaluationEngine
from src.evaluation.metrics import ToolSelectionAccuracy, ArgumentCorrectness, TaskCompletion
def print_report(report):
print("\n" + "=" * 70)
print(" AI AGENT EVALUATION REPORT")
print("=" * 70)
print(f"\n Total test cases: {report.total_cases}")
print(f" Metrics evaluated: {len(report.aggregate_scores)}")
print("\n AGGREGATE SCORES")
print(" " + "-" * 50)
for metric_name, score in report.aggregate_scores.items():
bar_length = int(score * 30)
bar = "#" * bar_length + "." * (30 - bar_length)
print(f" {metric_name:<25} [{bar}] {score:.2f}")
print("\n PER-CASE RESULTS")
print(" " + "-" * 50)
for case_result in report.results:
print(f"\n [{case_result.test_case_id}] {case_result.user_query[:50]}...")
print(f" Time: {case_result.execution_time:.1f}s")
for metric_result in case_result.metric_results:
status = "PASS" if metric_result.score >= 0.8 else "FAIL"
print(f" {metric_result.metric_name:<25} {metric_result.score:.2f} {status}")
if metric_result.score < 0.8:
print(f" -> {metric_result.details}")
print("\n" + "=" * 70)
def main():
print("Starting AI Agent Evaluation Platform...")
print("Running IT Help Desk agent evaluation suite\n")
metrics = [
ToolSelectionAccuracy(),
ArgumentCorrectness(),
TaskCompletion(),
]
engine = EvaluationEngine(metrics=metrics)
report = engine.run_evaluation("datasets/helpdesk_eval.json")
print_report(report)
if __name__ == "__main__":
main()Run the full evaluation suite
This is the moment where everything comes together. The runner will send 8 queries to your IT Help Desk agent, capture full execution traces, and score each trace against all three metrics.
How long does this take?
Each test case makes at least one API call to Claude, and some multi-tool cases require multiple round trips. Expect the full suite to take 30-90 seconds depending on API response times. You will see progress printed for each case as it completes.
- Run the evaluation suite by executing this command in your terminal:
python run_eval.pyYou should see each test case completing one by one, followed by a formatted evaluation report with aggregate scores and per-case results.
What should I look for in the results?
The aggregate scores section shows overall performance across all 8 cases. A perfect score is 1.00. Look for which metric scores lowest. That reveals the agent's weakest dimension.
In the per-case results, FAIL lines include diagnostics explaining exactly what went wrong. Common patterns include the agent calling extra tools beyond what was expected, or the final response missing expected keywords. These diagnostics are what make evaluation engineering valuable. Instead of guessing why an agent behaves poorly, you get precise measurements.
Seeing an authentication error?
- Confirm your ANTHROPIC_API_KEY environment variable is still set in your current terminal session. If you opened a new terminal, you need to set it again.
- Check that your API key has not expired and has available credits.
- Make sure your virtual environment is active (you should see the environment name in your terminal prompt).
Help me debug my evaluation run failing with an API error.
Why separate metrics instead of one overall score?
A single aggregate "pass/fail" hides which layer broke. If your agent picks the right tool but passes the wrong username, a composite score tells you nothing actionable. Per-metric scoring pinpoints the exact failure layer so you know whether to fix the system prompt, adjust the tool descriptions, or refine the agent's response generation.
Secret mission
Add Hallucination Detection
Your current metrics measure whether the agent calls the right tools with the right arguments. But what about when the agent confidently makes things up? Add a fourth metric that uses Claude as a judge to catch hallucinated claims not grounded in actual tool outputs.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing costs. You only incur Claude API charges when you actively run the evaluation suite.
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Local project folder containing src/, datasets/, and run_eval.py.
- Python virtual environment.
- ANTHROPIC_API_KEY environment variable.
- Anthropic API account (free to maintain, billed per API call only).
Keep everything running
No action needed. Choose this if you plan to keep iterating on the evaluation platform, adding new metrics, or expanding the test dataset.
- Your project files stay on your Desktop.
- Your virtual environment stays ready to activate.
- No costs are incurred until you run run_eval.py again.
Pause - I'll come back to this later
Deactivate your virtual environment to free up your terminal session. Your files and environment stay on disk for when you return.
- In your terminal, deactivate the virtual environment by running:
deactivate- When you come back, reactivate it with source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows).
Delete - I don't want to use this again
Remove all project files and clean up your environment completely.
- Close VS Code.
- Deactivate your virtual environment by running:
deactivate- Delete the project folder from your Desktop using Finder (macOS) or File Explorer (Windows). Right-click the folder and select Move to Trash or Delete.
- Unset your API key environment variable. On macOS/Linux, run:
unset ANTHROPIC_API_KEY- On Windows PowerShell, run:
Remove-Item Env:ANTHROPIC_API_KEYWhat about my Anthropic account?
Your Anthropic account is free to keep. You are only charged when you make API calls. There is no subscription or idle cost to worry about.
Nice Work!
Nice Work!
Nice work! You just built a production-architecture AI Agent Evaluation Platform that separates the evaluation engine from the agents it tests.
You've learned how to:
- Build a domain-agnostic evaluation engine with a pluggable Metric base class that can score any tool-calling agent.
- Implement an agentic loop with full trace capture, recording every tool call, argument, and result for later evaluation.
- Create three production-grade evaluation metrics covering tool selection accuracy, argument correctness, and task completion across a structured test dataset.
- Secret Mission: Added hallucination detection via the LLM-as-judge pattern, proving the agent's outputs are grounded in actual tool results.
Ready to quiz yourself?