Build an AI Agent Evaluation Platform

Build a LangGraph agent eval API with OTel tracing, deployed on Render.

Introduction

30 Second Summary

When you hire someone new, you don't just trust they'll do a great job. You test them, watch how they work, and measure their performance. AI agents need the same scrutiny.

In this project, you will build an evaluation platform that runs AI agents against test tasks and scores their performance across multiple dimensions: correctness, cost, latency, and tool usage.

What You'll Build

You'll send a task to your live API endpoint and get back a detailed scorecard showing exactly how your agent performed, what tools it called, and whether it got the answer right.

By the end of this project, you'll have:

  • A LangGraph agent with custom tools that you can point at any question-answering task and watch it reason through tool calls to reach an answer.
  • An evaluation pipeline that instruments the agent with OpenTelemetry, captures every LLM call and tool invocation, and scores the results using an LLM-as-judge.
  • A live REST API deployed on Render where you can create tasks, trigger evaluations, and retrieve scored results from anywhere.
  • Secret Mission: Add a hallucination detection scorer that cross-references the agent's final answer against its tool outputs.

Are there any prerequisites?

This is an advanced project. You should be comfortable with Python (async functions, type hints, and Pydantic models), and have Docker Desktop, PostgreSQL, and a GitHub account ready to go.

Before We Start

To evaluate LLM agents, you need more than a simple pass/fail test. You need to measure correctness, tool usage, cost, and latency across multiple tasks.

In this step, you'll set up a local development environment with all dependencies installed and a running Postgres database to store evaluation data.

In this step, get ready to:
  • Install Python dependencies and start a Postgres database with Docker.
  • Configure your Groq API key and project environment.
  • Define the database schema for evaluation tasks and results.
Create the project and start Postgres

Every terminal command in this project runs from inside a single project folder. You'll create it on your Desktop so it's easy to find.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows) and typing Terminal.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir agent-eval-platform
cd agent-eval-platform

Your Docker Compose file defines the Postgres database that stores evaluation tasks and results.

  • Create a file called docker-compose.yml in your agent-eval-platform folder with the following content:
services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: evaluser
      POSTGRES_PASSWORD: evalpass
      POSTGRES_DB: agent_eval
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

What does this file do?

This tells Docker to run a PostgreSQL 16 container with a user called evaluser, a password evalpass, and a database called agent_eval.

The volumes section ensures your data persists even if you stop the container.

  • Start the Postgres container by running:
docker compose up -d

You should see Docker pull the postgres:16-alpine image and start the container. The -d flag runs it in the background.

Seeing a connection error?

Make sure Docker Desktop is running. If you see a message about the Docker daemon, open Docker Desktop first, wait for it to fully start, then retry the command.

help me debug Docker Compose not starting

Now set up your Python virtual environment to isolate this project's dependencies.

  • Create and activate a virtual environment by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

You should see (venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

  • Create a file called requirements.txt in your agent-eval-platform folder with these dependencies:
langgraph==1.2.9
langchain-groq==1.1.3
langchain-core>=1.4.0
fastapi==0.139.0
uvicorn[standard]==0.34.0
sqlalchemy[asyncio]==2.0.41
asyncpg==0.30.0
opentelemetry-sdk==1.33.0
opentelemetry-instrumentation-langchain==0.62.1
opentelemetry-exporter-otlp==1.33.0
python-dotenv==1.1.0
pydantic==2.11.4

What are these packages?

  • langgraph is the agent framework that models agents as state machines.
  • langchain-groq connects LangGraph to Groq's fast inference API.
  • opentelemetry-instrumentation-langchain auto-captures LLM call traces including token counts and tool names.
  • sqlalchemy and asyncpg handle async database operations with Postgres.
  • Install all dependencies by running:
pip install -r requirements.txt

This takes about a minute. You should see packages downloading and installing with no errors at the end.

  • Create the app package directory by running:
mkdir app
touch app/__init__.py
  • Confirm everything is in place by running:
ls

You should see app, docker-compose.yml, requirements.txt, and venv in the output.

Configure your Groq API key and environment

Groq provides free LLM inference. You'll use it to power both the agent under test and the LLM-as-judge scorer.

  • Open https://console.groq.com in your browser.
  • Sign up with your email address or GitHub account. No credit card is required.
  • Once signed in, navigate to the API Keys section in the left sidebar.
  • Click Create API Key, give it a name like agent-eval, and copy the key.

Store your key somewhere safe

Groq only shows the full API key once at creation. Paste it into a text file or password manager before continuing.

  • While you're setting up accounts, also sign up for a free Render account at https://render.com. You'll use it for deployment later. Signing up is free and takes seconds.

Now store your credentials in a .env file so your code can access them without hardcoding secrets.

  • Create a file called .env in your agent-eval-platform folder with this content:
GROQ_API_KEY=your-api-key-here
DATABASE_URL=postgresql+asyncpg://evaluser:evalpass@localhost:5432/agent_eval
  • Replace your-api-key-here with the Groq API key you just copied.

What is the DATABASE_URL?

This connection string tells SQLAlchemy how to reach the Postgres container you started with Docker Compose. The username, password, and database name match what you configured in docker-compose.yml.

Verify that your Groq key works by running a quick test.

  • Test your Groq connection by running:
python -c "from langchain_groq import ChatGroq; import os; from dotenv import load_dotenv; load_dotenv(); llm = ChatGroq(model='llama-3.1-8b-instant', api_key=os.getenv('GROQ_API_KEY')); print(llm.invoke('Say hello in one word').content)"

You should see a single word response like Hello printed. This confirms your API key is valid and the Groq connection works.

Seeing an authentication error?

Double-check that you pasted the full API key into your .env file with no leading or trailing spaces. The key typically starts with gsk_.

help me fix my Groq API key authentication

Define database models and create tables

Your evaluation platform needs two database tables: one to store tasks (the questions the agent will answer) and one to store results (scores, latency, token counts).

  • Create a file called app/models.py with the EvalTask model and its base class:
from sqlalchemy import Column, Integer, String, Float, JSON, DateTime, func
from sqlalchemy.orm import DeclarativeBase


class Base(DeclarativeBase):
    pass


class EvalTask(Base):
    __tablename__ = "eval_tasks"

    id = Column(Integer, primary_key=True)
    question = Column(String, nullable=False)
    expected_answer = Column(String, nullable=False)
    required_tools = Column(JSON, default=list)
    created_at = Column(DateTime, server_default=func.now())

What does this model represent?

Each EvalTask is a test case for the agent. It has a question to answer, the expected correct answer, and a list of tools the agent should use.

The required_tools field lets you verify whether the agent actually called the right tools during evaluation.

  • Add the EvalResult model below EvalTask in the same file:
class EvalResult(Base):
    __tablename__ = "eval_results"

    id = Column(Integer, primary_key=True)
    task_id = Column(Integer, nullable=False)
    agent_output = Column(String)
    scores = Column(JSON)
    latency_ms = Column(Integer)
    token_count = Column(Integer)
    spans_data = Column(JSON)
    created_at = Column(DateTime, server_default=func.now())

What does EvalResult store?

Each result captures the agent's response to one task, plus the full scorecard (scores as JSON), how long it took (latency_ms), how many tokens it used, and raw OpenTelemetry span data for debugging.

  • Save app/models.py.

Now create the database connection module that your app will use to talk to Postgres.

  • Create a file called app/database.py with this content:
import os

from dotenv import load_dotenv
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker

load_dotenv()

DATABASE_URL = os.getenv("DATABASE_URL")

engine = create_async_engine(DATABASE_URL, echo=False)
async_session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)


async def get_session():
    async with async_session() as session:
        yield session


async def init_db():
    from app.models import Base
    async with engine.begin() as conn:
        await conn.run_sync(Base.metadata.create_all)

What does this module do?

  • create_async_engine opens an async connection pool to Postgres using your DATABASE_URL.
  • get_session() is a dependency that FastAPI endpoints will use to get a database session.
  • init_db() creates all tables defined by your models when the app starts.
  • Save app/database.py.
  • Create the tables in your running Postgres instance by running:
python -c "import asyncio; from app.database import init_db; asyncio.run(init_db()); print('Tables created successfully')"

You should see Tables created successfully printed. This confirms your models connected to the Docker Postgres and created the eval_tasks and eval_results tables.

Seeing a connection refused error?

Run docker compose ps to check if your Postgres container is running. If it shows as exited, run docker compose up -d again. It can take a few seconds for Postgres to accept connections after starting.

help me fix my database connection

Before you move on, confirm that both the database and your Python environment are fully working.

  • What do you think will happen when you import LangGraph? Will it succeed or fail?
  • Run the final verification:
python -c "from langgraph.graph import StateGraph; print('OK')"

You should see OK printed. This confirms LangGraph installed correctly and your environment is ready to build the agent.

✔️ I see OK printed

Your environment is fully set up. Postgres is running, dependencies are installed, and your database schema is in place.

ⓧ I see an import error

Make sure your virtual environment is active. You should see (venv) in your terminal prompt.

  • Reactivate it with source venv/bin/activate (macOS) or venv\Scripts\activate (Windows).
  • Then reinstall with pip install -r requirements.txt.
  • Run the import check again.

✔️ Awesome, I've got everything!

Great. Make sure you've saved both app/models.py and app/database.py.

ⓧ I'd like to double check the full code

Here is app/models.py in full:

from sqlalchemy import Column, Integer, String, Float, JSON, DateTime, func
from sqlalchemy.orm import DeclarativeBase


class Base(DeclarativeBase):
    pass


class EvalTask(Base):
    __tablename__ = "eval_tasks"

    id = Column(Integer, primary_key=True)
    question = Column(String, nullable=False)
    expected_answer = Column(String, nullable=False)
    required_tools = Column(JSON, default=list)
    created_at = Column(DateTime, server_default=func.now())


class EvalResult(Base):
    __tablename__ = "eval_results"

    id = Column(Integer, primary_key=True)
    task_id = Column(Integer, nullable=False)
    agent_output = Column(String)
    scores = Column(JSON)
    latency_ms = Column(Integer)
    token_count = Column(Integer)
    spans_data = Column(JSON)
    created_at = Column(DateTime, server_default=func.now())

Here is app/database.py in full:

import os

from dotenv import load_dotenv
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker

load_dotenv()

DATABASE_URL = os.getenv("DATABASE_URL")

engine = create_async_engine(DATABASE_URL, echo=False)
async_session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)


async def get_session():
    async with async_session() as session:
        yield session


async def init_db():
    from app.models import Base
    async with engine.begin() as conn:
        await conn.run_sync(Base.metadata.create_all)

Your local environment is fully operational. You have a Postgres database running in Docker, all Python packages installed, and a schema ready to store evaluation data. Next up, you'll build the LangGraph agent that this platform will evaluate.

Build the Tool-Calling Agent

Your database is ready and your models are defined. Before you can evaluate an agent, you need one to evaluate.

In this step, you'll build a LangGraph tool-calling agent with two custom tools and prove it works by running it on a single question.

In this step, get ready to:
  • Define two mock tools the agent can call (data lookup and calculator).
  • Build a LangGraph state graph with conditional routing for tool calls.
  • Run the agent on a test question and see it use a tool to answer.
Define the agent's tools

An agent is only as useful as the tools it can reach. You'll create two mock tools that simulate real capabilities: one for looking up facts, and one for doing math.

  • Create a new file called app/tools.py inside your app/ folder and add the following code:
from langchain_core.tools import tool


@tool
def lookup_data(query: str) -> str:
    """Look up factual data for a given query. Returns relevant information."""
    # Simulated data store with canned responses
    data_store = {
        "population of france": "The population of France is approximately 68.4 million as of 2024.",
        "capital of japan": "The capital of Japan is Tokyo.",
        "speed of light": "The speed of light in vacuum is 299,792,458 meters per second.",
        "python creator": "Python was created by Guido van Rossum and first released in 1991.",
    }
    # Normalize the query and check for partial matches
    key = query.lower().strip()
    for k, v in data_store.items():
        if k in key or key in k:
            return v
    return f"No data found for: {query}"

What does this code do?

The @tool decorator from LangChain turns a plain Python function into something an LLM can discover and call. The docstring becomes the tool's description that the model reads when deciding which tool to use.

The lookup_data tool uses a dictionary of canned responses. In production, this could hit a real database or API. The partial matching logic lets the LLM's query be slightly different from the exact key.

  • Now add the second tool below lookup_data in the same file:
@tool
def calculate(expression: str) -> str:
    """Evaluate a mathematical expression and return the result."""
    try:
        # Only allow safe characters for evaluation
        allowed_chars = set("0123456789+-*/.() ")
        if not all(c in allowed_chars for c in expression):
            return "Error: Invalid characters in expression"
        result = eval(expression)
        return f"Result: {result}"
    except Exception as e:
        return f"Calculation error: {e}"

What does calculate do?

The calculate tool evaluates math expressions. The allowed_chars check acts as a basic safety guard, preventing arbitrary code from running through eval().

  • Save app/tools.py.
  • Verify the tools import cleanly by running:
python -c "from app.tools import lookup_data, calculate; print('Tools loaded:', lookup_data.name, calculate.name)"

You should see Tools loaded: lookup_data calculate printed.

Seeing a ModuleNotFoundError?

Make sure you're running the command from your project root directory (the folder containing the app/ folder). Also confirm your virtual environment is activated.

help me fix a ModuleNotFoundError when importing from my app package

Build the LangGraph state graph

With tools defined, you need an agent that can decide when to call them. A StateGraph in LangGraph defines nodes (functions) and edges (transitions) that control the agent's reasoning loop.

  • Create a new file called app/agent.py and add the imports, tool list, and state definition:
import os
import time
from typing import Annotated, TypedDict

from dotenv import load_dotenv
from langchain_core.messages import BaseMessage, HumanMessage
from langchain_groq import ChatGroq
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode

from app.tools import lookup_data, calculate

load_dotenv()

# Register both tools so the agent can discover them
TOOLS = [lookup_data, calculate]


# Typed state that flows through the graph
class AgentState(TypedDict):
    messages: Annotated[list[BaseMessage], add_messages]

What is AgentState?

LangGraph passes a typed state dictionary between every node in the graph. AgentState holds the conversation's message history. The add_messages annotation tells LangGraph to append new messages rather than overwrite the list each time.

  • Below the AgentState class, add the create_agent() function that builds the full graph:
def create_agent():
    # Initialize Groq LLM with tool bindings
    llm = ChatGroq(
        model="llama-3.3-70b-versatile",
        api_key=os.getenv("GROQ_API_KEY"),
        temperature=0,
    ).bind_tools(TOOLS)

    def agent_node(state: AgentState) -> AgentState:
        # LLM decides: respond directly or call a tool
        response = llm.invoke(state["messages"])
        return {"messages": [response]}

    def should_continue(state: AgentState) -> str:
        # Route to tools if the LLM requested a tool call
        last_message = state["messages"][-1]
        if last_message.tool_calls:
            return "tools"
        return END

    tool_node = ToolNode(TOOLS)

    # Wire up the graph: agent -> conditional -> tools -> agent
    graph = StateGraph(AgentState)
    graph.add_node("agent", agent_node)
    graph.add_node("tools", tool_node)
    graph.add_edge(START, "agent")
    graph.add_conditional_edges("agent", should_continue, {"tools": "tools", END: END})
    graph.add_edge("tools", "agent")

    return graph.compile()

How does the graph route decisions?

The should_continue function inspects the LLM's latest response. If the response contains tool_calls, the graph routes to the tools node, which executes the requested tool. After execution, it loops back to the agent node so the LLM can incorporate the tool result.

If there are no tool calls, the graph routes to END and returns the final answer. This creates a reasoning loop: the agent can call multiple tools before settling on a response.

  • Save app/agent.py.
  • Verify the graph compiles by running:
python -c "from app.agent import create_agent; agent = create_agent(); print('Agent compiled:', type(agent).__name__)"

You should see Agent compiled: CompiledStateGraph printed. This confirms the LLM connected to your tools and the graph structure is valid.

Seeing an authentication error?

If you see a message about an invalid API key, double check that your .env file contains a valid GROQ_API_KEY and that load_dotenv() is at the top of agent.py.

help me debug a Groq API authentication error when creating my LangGraph agent

Run the agent on a test question

The graph compiles, but you haven't actually seen it think yet. You need a wrapper function that sends a question and returns the result with timing data.

  • Add the run_agent() function at the bottom of app/agent.py:
def run_agent(question: str) -> dict:
    agent = create_agent()
    start_time = time.time()
    # Invoke the graph with the user's question
    result = agent.invoke({"messages": [HumanMessage(content=question)]})
    elapsed_ms = int((time.time() - start_time) * 1000)

    final_message = result["messages"][-1]
    return {
        "output": final_message.content,
        "messages": result["messages"],
        "latency_ms": elapsed_ms,
    }

What does run_agent return?

The function compiles a fresh agent, times the full invocation, and returns three things: the final text output, the complete message history (including tool calls and tool results), and how long the whole run took in milliseconds. The message history is what you'll later feed into scoring.

  • Save app/agent.py.

Before you run the agent, think about what should happen. You're about to ask it "What is the population of France?" and it has a lookup_data tool with that exact answer in its data store.

Do you think the agent will call the tool, or try to answer from its own training data?

  • Test the agent by running:
python -c "
from app.agent import run_agent
result = run_agent('What is the population of France?')
print('Output:', result['output'])
print('Latency:', result['latency_ms'], 'ms')
print('Messages:', len(result['messages']))
for msg in result['messages']:
    print(f'  [{type(msg).__name__}] {msg.content[:80] if msg.content else msg.tool_calls}')
"

You should see the agent call the lookup_data tool, receive the canned response about 68.4 million, and then produce a final answer incorporating that data. The message history shows four messages: your question, the tool call request, the tool result, and the final answer.

Agent answering without calling tools?

If the agent responds with general knowledge instead of calling lookup_data, check that .bind_tools(TOOLS) is chained onto the ChatGroq initialization. Without tool bindings, the LLM doesn't know tools exist.

help me figure out why my LangGraph agent isn't calling its tools

✔️ Awesome, I've got everything!

Great. Double check you've saved both app/tools.py and app/agent.py.

ⓧ I'd like to double check the full code

Here is the full app/tools.py:

from langchain_core.tools import tool


@tool
def lookup_data(query: str) -> str:
    """Look up factual data for a given query. Returns relevant information."""
    data_store = {
        "population of france": "The population of France is approximately 68.4 million as of 2024.",
        "capital of japan": "The capital of Japan is Tokyo.",
        "speed of light": "The speed of light in vacuum is 299,792,458 meters per second.",
        "python creator": "Python was created by Guido van Rossum and first released in 1991.",
    }
    key = query.lower().strip()
    for k, v in data_store.items():
        if k in key or key in k:
            return v
    return f"No data found for: {query}"


@tool
def calculate(expression: str) -> str:
    """Evaluate a mathematical expression and return the result."""
    try:
        allowed_chars = set("0123456789+-*/.() ")
        if not all(c in allowed_chars for c in expression):
            return "Error: Invalid characters in expression"
        result = eval(expression)
        return f"Result: {result}"
    except Exception as e:
        return f"Calculation error: {e}"

For app/agent.py, confirm you have these three sections in order:

  • Imports, TOOLS list, and AgentState class at the top.
  • The create_agent() function with LLM initialization, agent_node, should_continue, and graph wiring.
  • The run_agent() function at the bottom.

Your agent works. It receives a question, decides which tool to call, executes the tool, and incorporates the result into a final answer. Next up, you'll instrument this agent with OpenTelemetry to capture exactly what happens at each step and run it against a batch of evaluation tasks.

Instrument and Run Evaluation Tasks

In the previous step, you proved your agent can handle a single question. It called a tool, got the result, and composed a final answer.

One successful answer tells you nothing about reliability across different task types. You need to run the agent against a batch of tasks and capture exactly what happened at each step. In this step you will add OpenTelemetry instrumentation to record every LLM call and tool invocation, then run a batch evaluation. The gap will become obvious: you get outputs and telemetry but no scores.

In this step, get ready to:
  • Enable OpenTelemetry tracing to capture per-step telemetry from agent runs.
  • Create an evaluation runner that processes tasks from Postgres and stores raw results.
  • Seed evaluation tasks and run the pipeline to see outputs without scores.
Enable OpenTelemetry instrumentation

OpenTelemetry gives you a standard way to capture traces from your agent runs. By hooking into LangChain's callback system, every LLM call and tool invocation automatically produces a span with attributes like token counts and tool names.

  • Create a new file called app/instrumentation.py and add the OTel setup code:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory import InMemorySpanExporter
from opentelemetry.instrumentation.langchain import LangchainInstrumentor

# Create an in-memory exporter so we can read spans programmatically
_exporter = InMemorySpanExporter()
_provider = TracerProvider()
_provider.add_span_processor(SimpleSpanProcessor(_exporter))
trace.set_tracer_provider(_provider)

# Instrument all LangChain/LangGraph calls automatically
LangchainInstrumentor().instrument()

What does this code do?

Instead of sending traces to an external collector, you use an InMemorySpanExporter that stores spans in a Python list. After each agent run you can programmatically read every span and extract attributes like token counts and tool names.

The LangchainInstrumentor().instrument() call patches LangChain's callback system so every LLM invocation and tool call produces a trace span automatically.

  • Verify the instrumentation loads without errors by running:
python -c "import app.instrumentation; print('OpenTelemetry instrumentation active')"

You should see OpenTelemetry instrumentation active printed with no import errors.

Seeing an ImportError?

Make sure your virtual environment is activated and that you installed all dependencies from requirements.txt in Step 1. The package opentelemetry-instrumentation-langchain must be installed.

help me fix this OpenTelemetry import error

  • Below the LangchainInstrumentor().instrument() line in app/instrumentation.py, add the span collection helper functions:
def get_exporter() -> InMemorySpanExporter:
    return _exporter


def collect_spans() -> list[dict]:
    """Harvest finished spans and clear the exporter for the next run."""
    spans = _exporter.get_finished_spans()
    collected = []
    for span in spans:
        attrs = dict(span.attributes) if span.attributes else {}
        collected.append({
            "name": span.name,
            "duration_ms": int((span.end_time - span.start_time) / 1_000_000),
            "attributes": attrs,
        })
    _exporter.clear()
    return collected

What does collect_spans do?

After each agent run, collect_spans() reads all finished spans from the in-memory exporter, extracts their name, duration, and attributes into plain dictionaries, then clears the exporter so the next run starts fresh.

  • Below the collect_spans() function, add the metrics extraction function:
def extract_metrics(spans: list[dict]) -> dict:
    """Compute token counts, cost, and tool usage from collected spans."""
    total_input_tokens = 0
    total_output_tokens = 0
    tools_called = []

    for span in spans:
        attrs = span.get("attributes", {})
        input_tokens = attrs.get("gen_ai.usage.input_tokens", 0)
        output_tokens = attrs.get("gen_ai.usage.output_tokens", 0)
        total_input_tokens += input_tokens
        total_output_tokens += output_tokens

        tool_name = attrs.get("gen_ai.tool.name")
        if tool_name:
            tools_called.append(tool_name)

    total_tokens = total_input_tokens + total_output_tokens

    # Groq pricing for llama-3.3-70b-versatile: $0.59/1M input, $0.79/1M output
    cost_usd = (total_input_tokens * 0.59 / 1_000_000) + (
        total_output_tokens * 0.79 / 1_000_000
    )

    return {
        "total_tokens": total_tokens,
        "input_tokens": total_input_tokens,
        "output_tokens": total_output_tokens,
        "cost_usd": round(cost_usd, 6),
        "tools_called": tools_called,
    }

What does extract_metrics do?

This function reads the standard OpenTelemetry GenAI semantic convention attributes (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.tool.name) from each span.

It sums token counts across all spans, collects tool names, and calculates cost using Groq's published pricing: $0.59 per million input tokens and $0.79 per million output tokens for llama-3.3-70b-versatile.

  • Save app/instrumentation.py.
  • Test that the full instrumentation pipeline works end-to-end by running:
python -c "
import app.instrumentation
from app.instrumentation import collect_spans, extract_metrics
from app.agent import run_agent

result = run_agent('What is the capital of Japan?')
spans = collect_spans()
metrics = extract_metrics(spans)
print('Agent output:', result['output'][:80])
print('Tokens used:', metrics['total_tokens'])
print('Tools called:', metrics['tools_called'])
print('Cost USD:', metrics['cost_usd'])
"

You should see the agent's answer along with token counts, the lookup_data tool name, and a tiny cost value.

Seeing 0 tokens or empty tools_called?

Make sure import app.instrumentation appears BEFORE from app.agent import run_agent. The instrumentation must be active before LangChain is used for the first time.

help me debug why OpenTelemetry spans are empty

✔️ Awesome, I've got everything!

Great. Double check you have saved app/instrumentation.py.

ⓧ I'd like to double check the full code

Your app/instrumentation.py should contain these sections in order. Scroll up and confirm each is present:

  • The import block (5 imports from opentelemetry and the LangchainInstrumentor).
  • Module-level setup: _exporter, _provider, span processor, set_tracer_provider, and LangchainInstrumentor().instrument().
  • The get_exporter() and collect_spans() functions.
  • The extract_metrics() function with token counting and cost calculation.
Create the evaluation runner

Now you need a function that loads all tasks from the database, runs the agent on each one, collects the telemetry, and stores raw results. This is the evaluation pipeline without scoring.

  • Create a new file called app/evaluator.py and add the imports:
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession

from app.agent import run_agent
from app.instrumentation import collect_spans, extract_metrics
from app.models import EvalTask, EvalResult
  • Below the imports, add the run_evaluation() function:
async def run_evaluation(session: AsyncSession) -> list[dict]:
    """Run the agent against all tasks and store raw results."""
    result_set = await session.execute(select(EvalTask))
    tasks = result_set.scalars().all()

    results = []
    for task in tasks:
        agent_result = run_agent(task.question)
        spans = collect_spans()
        metrics = extract_metrics(spans)

        eval_result = EvalResult(
            task_id=task.id,
            agent_output=agent_result["output"],
            scores=None,
            latency_ms=agent_result["latency_ms"],
            token_count=metrics["total_tokens"],
            spans_data=[s for s in spans[:10]],
        )
        session.add(eval_result)
        results.append({
            "task_id": task.id,
            "question": task.question,
            "agent_output": agent_result["output"],
            "metrics": metrics,
        })

    await session.commit()
    return results

What does run_evaluation do?

For each task in the database, it runs your LangGraph agent, harvests the OTel spans, extracts metrics (tokens, tools, cost), and writes an EvalResult row with the raw output. Notice scores=None. There is no scoring logic yet.

The function returns a list of results with the agent output and metrics so you can inspect them immediately.

  • Save app/evaluator.py.

Seeing a circular import error?

If Python complains about circular imports between app.agent and app.instrumentation, make sure app/instrumentation.py does not import anything from app/agent.py. The dependency flows one way: evaluator imports both agent and instrumentation.

help me fix this circular import

✔️ Awesome, I've got everything!

Great. Make sure app/evaluator.py is saved.

ⓧ I'd like to double check the full code

Your app/evaluator.py should contain these sections in order:

  • The import block (sqlalchemy select, AsyncSession, run_agent, collect_spans, extract_metrics, EvalTask, EvalResult).
  • The run_evaluation() async function that loops through tasks, runs the agent, collects spans, stores results with scores=None, and returns the results list.
Seed tasks and run the evaluation

You need real tasks in the database to evaluate against. You will seed three tasks that test different agent capabilities: one requiring a tool lookup, one requiring calculation, and one trick question that needs no tools.

  • Create a new file called seed_tasks.py in your project root (next to docker-compose.yml) with this seeding script:
import asyncio

from app.database import async_session, init_db
from app.models import EvalTask

async def seed():
    await init_db()
    async with async_session() as session:
        tasks = [
            EvalTask(
                question="What is the population of France?",
                expected_answer="The population of France is approximately 68.4 million.",
                required_tools=["lookup_data"],
            ),
            EvalTask(
                question="What is 245 * 18?",
                expected_answer="4410",
                required_tools=["calculate"],
            ),
            EvalTask(
                question="What is the meaning of life according to Douglas Adams?",
                expected_answer="42",
                required_tools=[],
            ),
        ]
        session.add_all(tasks)
        await session.commit()
        print(f"Seeded {len(tasks)} evaluation tasks.")

if __name__ == "__main__":
    asyncio.run(seed())

What do these three tasks test?

Task 1 requires the agent to call lookup_data to find France's population. Task 2 requires calculate for arithmetic. Task 3 is a trick question. The answer is common knowledge, so the agent should respond directly without calling any tools.

Each task stores its expected_answer and required_tools so you can later score correctness and tool usage.

  • Save seed_tasks.py and run it to seed the database:
python seed_tasks.py

You should see Seeded 3 evaluation tasks. printed.

Connection refused error?

Make sure your Docker Compose Postgres container is still running. Check with docker compose ps and restart with docker compose up -d if needed.

help me fix this database connection error

Now for the moment of truth. Before you run the evaluation, think about this: what will the output look like if there is no scoring logic?

  • Run the full evaluation pipeline:
python -c "
import asyncio, json
from app.database import async_session
from app.evaluator import run_evaluation

async def main():
    async with async_session() as session:
        results = await run_evaluation(session)
        for r in results:
            print('---')
            print(f\"Task {r['task_id']}: {r['question']}\")
            print(f\"Output: {r['agent_output'][:100]}\")
            print(f\"Tokens: {r['metrics']['total_tokens']}\")
            print(f\"Tools called: {r['metrics']['tools_called']}\")
            print(f\"Cost: \${r['metrics']['cost_usd']}\")
            print(f\"Scores: None\")

asyncio.run(main())
"

You should see three task results printed. Each shows the agent's output, token counts, tool names called, and cost. But every result says Scores: None.

Why are there no scores?

You have raw telemetry: you know what the agent said, which tools it called, how many tokens it used, and how long it took. But you have no idea if the answers are actually correct.

Did the agent get France's population right? Did it calculate 245 * 18 correctly? You cannot tell just from the output. You need a scoring pipeline that judges correctness and checks tool usage. That is exactly what the next step builds.

You have a working evaluation pipeline that captures exactly what happens inside each agent run. Next up, you will build the scoring logic that turns these raw outputs into quantified metrics.

Build the Scoring Pipeline

In the previous step, you ran your agent against three evaluation tasks and captured raw telemetry. You have outputs and metrics, but no way to tell whether the agent actually answered correctly.

Raw outputs are useless without scores. This step adds the evaluation logic that turns "the agent said X" into quantified metrics: correctness, tool-use success, cost, and latency.

In this step, get ready to:
  • Build an LLM-as-judge scorer that rates correctness on a 0.0 to 1.0 scale.
  • Build a tool-use checker that verifies the agent called the required tools.
  • Wire both scorers into the evaluation pipeline and print a full scorecard.
Create the scoring module

The LLM-as-a-Judge pattern uses a second LLM to evaluate the first one's output. You send the question, the expected answer, and the agent's answer to a judge model, and it returns a numeric score.

  • Create a new file app/scoring.py inside your app folder and add these imports:
import os

from dotenv import load_dotenv
from langchain_groq import ChatGroq
from langchain_core.messages import HumanMessage, SystemMessage

load_dotenv()

What does this code do?

These imports bring in the Groq LLM client and LangChain message types you need to call the judge model. load_dotenv() loads your GROQ_API_KEY from the .env file.

  • Below the imports in app/scoring.py, add the score_correctness() function:
def score_correctness(question: str, expected_answer: str, agent_output: str) -> dict:
    judge_llm = ChatGroq(
        model="llama-3.1-8b-instant",
        api_key=os.getenv("GROQ_API_KEY"),
        temperature=0,
    )

    prompt = f"""You are an evaluation judge. Score how correct the agent's answer is compared to the expected answer.

Question: {question}
Expected Answer: {expected_answer}
Agent's Answer: {agent_output}

Respond with ONLY a JSON object (no markdown, no explanation):
{{"score": <float 0.0 to 1.0>, "rationale": "<one sentence>"}}"""

    response = judge_llm.invoke([
        SystemMessage(content="You are a strict evaluation judge. Respond only with valid JSON."),
        HumanMessage(content=prompt),
    ])

    try:
        import json
        result = json.loads(response.content)
        return {
            "correctness": float(result.get("score", 0.0)),
            "rationale": result.get("rationale", ""),
        }
    except (json.JSONDecodeError, ValueError):
        return {"correctness": 0.0, "rationale": "Judge failed to produce valid JSON"}

What does this code do?

  • It creates a separate LLM instance using llama-3.1-8b-instant as the judge. This is a cheaper and faster model than the agent's llama-3.3-70b-versatile, which avoids rate-limit contention.
  • The prompt gives the judge the question, expected answer, and agent's answer, then asks for a JSON score between 0.0 and 1.0.
  • The try/except block handles the case where the judge fails to produce valid JSON, returning a 0.0 score as a safe default.

Why use a different model for the judge?

Using the same model to judge itself creates a bias problem. A separate, smaller model is faster and avoids consuming your rate limit on the primary agent model. llama-3.1-8b-instant costs $0.05/1M input tokens vs $0.59/1M for the 70B model.

  • Save app/scoring.py.
Add the tool-use scorer

Correctness alone does not tell you whether the agent actually used its tools. A tool-use scorer checks whether the agent called the tools the task required.

  • Below score_correctness() in app/scoring.py, add the score_tool_use() function:
def score_tool_use(required_tools: list[str], tools_called: list[str]) -> dict:
    if not required_tools:
        return {"tool_use_success": True, "missing_tools": []}

    missing = [t for t in required_tools if t not in tools_called]
    return {
        "tool_use_success": len(missing) == 0,
        "missing_tools": missing,
    }

What does this code do?

  • It compares the required_tools list from the task metadata against the tools_called list extracted from the OTel spans.
  • If the task has no required tools (like the Douglas Adams trick question), it returns True automatically.
  • If any required tool is missing from the agent's actual calls, it reports failure and lists which tools were skipped.

What about cost scoring?

Cost calculation already lives in extract_metrics() inside app/instrumentation.py. It computes cost_usd from token counts using Groq's pricing. You do not need a separate scorer for it.

  • Save app/scoring.py.
  • Verify the file has no syntax errors by running:
python -c "from app.scoring import score_correctness, score_tool_use; print('OK')"

You should see OK printed with no import errors.

Seeing an ImportError?

  • Make sure your virtual environment is activated. The langchain-groq and langchain-core packages should be installed there.
  • Check that app/scoring.py is saved inside the app/ directory alongside __init__.py.

Still stuck? Help me debug this scoring module import error.

✔️ Awesome, I've got everything!

Great. Double check you have saved app/scoring.py with both functions.

ⓧ I'd like to double check the full code

Your app/scoring.py should contain these sections in order:

  • The import group (os, load_dotenv, ChatGroq, HumanMessage, SystemMessage) followed by load_dotenv().
  • The score_correctness() function that calls llama-3.1-8b-instant and returns a dict with correctness and rationale keys.
  • The score_tool_use() function that returns a dict with tool_use_success and missing_tools keys.

Scroll up and confirm each chunk is present and matches what you pasted.

Wire scorers into the evaluation pipeline

Now you need to connect these scorers to the evaluation loop so every task gets a full scorecard. You will replace the contents of app/evaluator.py with a version that calls both scorers after each agent run.

  • Open app/evaluator.py and replace its entire contents with this first section (imports through tool scoring):
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession

from app.agent import run_agent
from app.instrumentation import collect_spans, extract_metrics
from app.scoring import score_correctness, score_tool_use
from app.models import EvalTask, EvalResult


async def run_evaluation(session: AsyncSession) -> list[dict]:
    result_set = await session.execute(select(EvalTask))
    tasks = result_set.scalars().all()

    results = []
    for task in tasks:
        agent_result = run_agent(task.question)
        spans = collect_spans()
        metrics = extract_metrics(spans)

        correctness = score_correctness(
            task.question, task.expected_answer, agent_result["output"]
        )

        # Collect spans again after judge call and discard them
        collect_spans()

        tool_score = score_tool_use(
            task.required_tools or [], metrics["tools_called"]
        )

What's happening here?

  • The new import from app.scoring import score_correctness, score_tool_use brings in both scorers.
  • After each agent run, score_correctness() calls the judge LLM. Since the judge call itself generates OTel spans, the second collect_spans() discards those so they do not pollute the next task's metrics.
  • score_tool_use() compares the task's required tools against what the agent actually called.
  • Immediately below the code you just pasted (still inside the for task in tasks: loop), add the scores assembly and database storage:
        scores = {
            "correctness": correctness["correctness"],
            "correctness_rationale": correctness["rationale"],
            "tool_use_success": tool_score["tool_use_success"],
            "missing_tools": tool_score["missing_tools"],
            "cost_usd": metrics["cost_usd"],
            "latency_ms": agent_result["latency_ms"],
            "total_tokens": metrics["total_tokens"],
        }

        eval_result = EvalResult(
            task_id=task.id,
            agent_output=agent_result["output"],
            scores=scores,
            latency_ms=agent_result["latency_ms"],
            token_count=metrics["total_tokens"],
            spans_data=[s for s in spans[:10]],
        )
        session.add(eval_result)
        results.append({
            "task_id": task.id,
            "question": task.question,
            "agent_output": agent_result["output"],
            "scores": scores,
        })

    await session.commit()
    return results

What does this code do?

  • The scores dict combines all dimensions into one JSON-serializable object: correctness, tool-use, cost, latency, and token count.
  • EvalResult now stores the full scores JSON (previously it was None).
  • The results list includes scores so you can print a scorecard after the run completes.
  • Save app/evaluator.py.
  • Verify the updated evaluator imports cleanly by running:
python -c "from app.evaluator import run_evaluation; print('OK')"

You should see OK with no errors.

Seeing an IndentationError?

  • Make sure the scores = { block is indented with 8 spaces (two levels: one for for loop, one for async def).
  • Check that await session.commit() and return results are at 4 spaces (inside the function but outside the loop).

Still stuck? Help me fix this indentation error in my evaluator.

✔️ Awesome, I've got everything!

Great. Double check you have saved app/evaluator.py with both code sections combined.

ⓧ I'd like to double check the full code

Your app/evaluator.py should contain these sections in order:

  • Imports: select, AsyncSession, run_agent, collect_spans, extract_metrics, score_correctness, score_tool_use, EvalTask, EvalResult.
  • The run_evaluation() function: loads tasks, runs agent, collects spans, calls score_correctness(), discards judge spans, calls score_tool_use(), builds the scores dict, creates EvalResult, commits, and returns results.

Scroll up and confirm each chunk is present and matches what you pasted.

Now it is time to run the full scoring pipeline against your three seed tasks. Before you run it, ask yourself: do you think the "meaning of life" trick question (which has no required tools) will score the same correctness as the factual questions?

  • Run the evaluation with scoring by executing:
python -c "
import asyncio
from app.database import async_session
from app.evaluator import run_evaluation

async def main():
    async with async_session() as session:
        results = await run_evaluation(session)
        for r in results:
            s = r['scores']
            print(f\"Task {r['task_id']}: {r['question']}\")
            print(f\"  Correctness: {s['correctness']} - {s['correctness_rationale']}\")
            print(f\"  Tool Use: {s['tool_use_success']} | Missing: {s['missing_tools']}\")
            print(f\"  Cost: \${s['cost_usd']} | Latency: {s['latency_ms']}ms\")
            print(f\"  Tokens: {s['total_tokens']}\")
            print()

asyncio.run(main())
"

You should see a formatted scorecard for each of the three tasks. Each scorecard displays correctness (a float between 0.0 and 1.0), tool_use_success (True or False), cost_usd, and latency_ms.

What should I see?

  • The population question should score high on correctness (close to 1.0) and show tool_use_success: True.
  • The math question (245 * 18) should also score high with tool_use_success: True.
  • The Douglas Adams trick question may score lower on correctness. It has no required tools, so tool_use_success is True by default.

Exact scores vary per run because the judge LLM's evaluation can fluctuate slightly. The key signal is that your pipeline produces structured, multi-dimensional scores for every task.

Getting rate limit errors?

  • Groq's free tier limits llama-3.3-70b-versatile to 30 requests per minute. Running 3 tasks means at least 3 agent calls plus 3 judge calls. If you hit a 429 error, wait 60 seconds and try again.
  • If errors persist, check your GROQ_API_KEY in .env is correct and has no extra whitespace.

Still stuck? Help me debug rate limiting or scoring errors in my evaluation pipeline.

Your evaluation platform now produces real, quantified scores instead of raw text outputs. Next up, you will wrap this entire pipeline in a FastAPI service and deploy it live so anyone can trigger evaluations via a REST API.

Deploy the FastAPI Service to Render

Your evaluation pipeline works locally, but it only exists on your machine. Nobody else can trigger evaluations or see results without accessing your laptop directly.

In this step, you will wrap the pipeline in a FastAPI REST API, containerize it with Docker, and deploy the whole thing to Render so anyone can POST a task and get back a scorecard from a public URL.

In this step, get ready to:
  • Create a FastAPI app with endpoints for task creation, evaluation, and results retrieval.
  • Containerize the app with a Dockerfile and test it locally.
  • Deploy the service live on Render with a free Postgres database.
Create the FastAPI application

Your API needs four endpoints: one to create tasks, one to run evaluations, one to retrieve results, and a health check for Render to confirm the service is alive.

  • Create a new file called app/main.py and add the app setup with imports and lifespan by pasting the following code:
from contextlib import asynccontextmanager

from fastapi import FastAPI, Depends
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession

import app.instrumentation  # noqa: F401 - triggers OTel setup on import
from app.database import get_session, init_db
from app.models import EvalTask, EvalResult
from app.evaluator import run_evaluation


@asynccontextmanager
async def lifespan(app: FastAPI):
    await init_db()
    yield


app = FastAPI(title="Agent Eval Platform", lifespan=lifespan)

What does this code do?

The lifespan context manager runs init_db() once when the server starts. This creates your database tables automatically on boot.

The import app.instrumentation line triggers OpenTelemetry setup as a side effect. Importing the module is enough to activate tracing for all subsequent LangChain calls.

  • Below the app instance, add the Pydantic request and response models:
class TaskCreate(BaseModel):
    question: str
    expected_answer: str
    required_tools: list[str] = []


class TaskResponse(BaseModel):
    id: int
    question: str
    expected_answer: str
    required_tools: list[str]

What are these models for?

Pydantic models define the shape of incoming requests and outgoing responses. TaskCreate validates what the caller sends when creating a task. TaskResponse controls what gets returned.

  • Save app/main.py.
  • Start the development server with hot-reloading by running this command:
uvicorn app.main:app --reload

You should see output including Application startup complete in the terminal. The server is now running and will auto-reload as you add endpoints.

Seeing a connection error on startup?

The app connects to Postgres on startup via the lifespan function. Make sure your Docker Compose Postgres container is still running. Run docker compose up -d to restart it if needed.

help me debug my FastAPI app failing to connect to Postgres on startup

  • Open a new terminal (keep uvicorn running in the first one).
  • In app/main.py, below the Pydantic models, add the POST /tasks and POST /evaluations/run endpoints:
@app.post("/tasks", response_model=TaskResponse)
async def create_task(task: TaskCreate, session: AsyncSession = Depends(get_session)):
    db_task = EvalTask(
        question=task.question,
        expected_answer=task.expected_answer,
        required_tools=task.required_tools,
    )
    session.add(db_task)
    await session.commit()
    await session.refresh(db_task)
    return TaskResponse(
        id=db_task.id,
        question=db_task.question,
        expected_answer=db_task.expected_answer,
        required_tools=db_task.required_tools or [],
    )


@app.post("/evaluations/run")
async def trigger_evaluation(session: AsyncSession = Depends(get_session)):
    results = await run_evaluation(session)
    return {"status": "completed", "results": results}

What does this code do?

The POST /tasks endpoint accepts a JSON body with a question, expected answer, and required tools. It saves the task to Postgres and returns the created record.

The POST /evaluations/run endpoint triggers the full evaluation pipeline on all tasks in the database and returns the scored results as JSON.

  • Save app/main.py. The server auto-reloads in your other terminal.
  • Test creating a task by running this in your new terminal:
curl -X POST http://localhost:8000/tasks \
  -H "Content-Type: application/json" \
  -d '{"question": "What is the capital of Japan?", "expected_answer": "Tokyo", "required_tools": ["lookup_data"]}'

You should see a JSON response with an id, the question, expected answer, and required tools returned.

  • Add the final two endpoints at the bottom of app/main.py:
@app.get("/evaluations/{result_id}")
async def get_result(result_id: int, session: AsyncSession = Depends(get_session)):
    result = await session.execute(
        select(EvalResult).where(EvalResult.id == result_id)
    )
    eval_result = result.scalar_one_or_none()
    if not eval_result:
        return {"error": "Result not found"}
    return {
        "id": eval_result.id,
        "task_id": eval_result.task_id,
        "agent_output": eval_result.agent_output,
        "scores": eval_result.scores,
        "latency_ms": eval_result.latency_ms,
        "token_count": eval_result.token_count,
    }


@app.get("/health")
async def health():
    return {"status": "ok"}

What does this code do?

The GET /evaluations/{result_id} endpoint retrieves a single scored result by its database ID. The GET /health endpoint returns a simple JSON response that Render uses to confirm the service is alive.

  • Save app/main.py.
  • Test the health endpoint by running:
curl http://localhost:8000/health

You should see {"status":"ok"} printed in the terminal.

  • Stop the development server with Ctrl+C in the uvicorn terminal.

✔️ Awesome, I've got everything!

Great. Make sure you have saved app/main.py before moving on.

ⓧ I'd like to double check the full code

Your app/main.py file combines four code chunks in order. Scroll up and confirm each is present:

  • Chunk 1: Imports, lifespan, and the app instance creation.
  • Chunk 2: TaskCreate and TaskResponse Pydantic models.
  • Chunk 3: POST /tasks and POST /evaluations/run endpoints.
  • Chunk 4: GET /evaluations/{result_id} and GET /health endpoints.
Containerize and test locally with Docker

Render deploys your app from a Docker image. You need a Dockerfile that installs dependencies, copies your code, and starts uvicorn on the port Render expects.

  • Create a file called Dockerfile in your project root (the same folder as requirements.txt) and paste the following:
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 10000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "10000"]

What does this Dockerfile do?

It starts from the official python:3.12-slim image. It copies and installs requirements.txt first so dependency layers are cached. Then it copies the rest of your code and starts uvicorn on port 10000.

Render's default port for web services is 10000. Binding to 0.0.0.0 makes the server accessible from outside the container.

  • Save Dockerfile.
  • Build the Docker image by running:
docker build -t agent-eval .

You should see a series of build steps ending with a message confirming the image was tagged as agent-eval.

  • Run the container locally to confirm it starts by running:
docker run --rm -p 10000:10000 \
  -e GROQ_API_KEY=your-api-key-here \
  -e DATABASE_URL=postgresql+asyncpg://evaluser:evalpass@host.docker.internal:5432/agent_eval \
  agent-eval

Why host.docker.internal?

Inside a Docker container, localhost refers to the container itself, not your machine. The hostname host.docker.internal is a special DNS name that Docker Desktop provides to reach services running on your host machine.

  • In a new terminal, test the containerized health endpoint:
curl http://localhost:10000/health

You should see {"status":"ok"} confirming the container is running correctly on port 10000.

  • Stop the container with Ctrl+C in the terminal where it is running.

Container crashes on startup?

If you see a database connection error, confirm Docker Compose is still running your Postgres container with docker compose ps. On Linux, you may need to add --add-host=host.docker.internal:host-gateway to the docker run command.

help me debug Docker container failing to connect to my local Postgres

✔️ Awesome, I've got everything!

Great. Confirm Dockerfile is saved in your project root.

ⓧ I'd like to double check the full code

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 10000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "10000"]
Push to GitHub and deploy to Render

Your code runs locally in Docker. Now you need it running on a public URL so anyone can trigger evaluations. Render can build and deploy directly from a GitHub repository.

  • Create a .gitignore file in your project root to keep secrets and generated files out of the repo:
echo ".env
venv/
__pycache__/
*.pyc" > .gitignore
  • Initialize a Git repository and make your first commit by running these commands:
git init
git add .
git commit -m "Initial commit: agent eval platform"
  • Go to github.com/new in your browser.
  • Name the repository agent-eval-platform (or any name you prefer).
  • Leave it set to Public and do not add a README or .gitignore.
  • Click Create repository.
  • Copy the remote URL GitHub shows you and push your code:
git remote add origin [[REPO_URL="https://github.com/your-username/agent-eval-platform.git"]]
git branch -M main
git push -u origin main

Your code is now on GitHub. Next, set up the Render database and web service.

  • Open dashboard.render.com and sign in to your Render account.
  • Click + New in the top-right corner and select PostgreSQL.
  • Enter agent-eval-db as the name.
  • Select the Free instance type.
  • Click Create Database.
  • On the database page that appears, open the Connect menu in the top-right corner.
  • Copy the Internal Database URL. It will look like postgresql://user:password@host:5432/dbname.

Why the internal URL?

Render services in the same region communicate over a private network. The internal URL keeps traffic off the public internet, which reduces latency and avoids exposing your database credentials externally.

Your app uses the asyncpg driver, which requires postgresql+asyncpg:// as the URL scheme. You will modify the internal URL when you set the environment variable in the next steps.

  • Back in the Render Dashboard, click + New and select Web Service.
  • Connect your GitHub account if prompted, then select your agent-eval-platform repository and click Connect.
  • Set the Name to agent-eval (this becomes part of your public URL).
  • For Language, select Docker. Render will detect your Dockerfile automatically.
  • Select the Free instance type.
  • Scroll down to Environment Variables and add two variables:
  • Key: GROQ_API_KEY with your Groq API key as the value.
  • Key: DATABASE_URL with your Render internal database URL, modified to use the asyncpg driver. Replace postgresql:// at the start with postgresql+asyncpg://.

Modifying the DATABASE_URL

If your internal URL is postgresql://user:pass@host:5432/db, change it to postgresql+asyncpg://user:pass@host:5432/db. Only the scheme prefix changes. Everything after :// stays the same.

  • Click Create Web Service.

Render will pull your code from GitHub, build the Docker image, and start the container. This takes 2-5 minutes. You can watch progress on the service's Events page.

What about the free tier spin-down?

Render free web services spin down after 15 minutes of inactivity. The first request after idle takes about 1 minute while the container restarts. This is normal for the free tier.

Before you test, what do you expect to see when you hit the /health endpoint on your live Render URL?

  • Once the deploy shows Live in the Render dashboard, test your live service by running:
curl https://[[RENDER_URL="your-app-name"]].onrender.com/health

You should see {"status":"ok"} returned from your live Render URL.

Deploy failed or no response?

Check the Logs tab in your Render service dashboard. Common issues include a missing environment variable (the app crashes because DATABASE_URL is not set) or the wrong database URL scheme (must start with postgresql+asyncpg://).

If the first request times out, wait 60 seconds and try again. Free services take up to a minute to wake from idle.

help me debug my Render deployment that is failing to start

Now run the real end-to-end test. Do you expect the evaluation to succeed on the first try, or might something go wrong with a fresh database?

  • Seed a task and trigger an evaluation on your live service by running:
curl -X POST https://[[RENDER_URL="your-app-name"]].onrender.com/tasks \
  -H "Content-Type: application/json" \
  -d '{"question": "What is the population of France?", "expected_answer": "The population of France is approximately 68.4 million.", "required_tools": ["lookup_data"]}'
  • Now trigger the evaluation:
curl -X POST https://[[RENDER_URL="your-app-name"]].onrender.com/evaluations/run

You should see a JSON response with "status": "completed" and a results array containing the agent output and scored metrics including correctness, tool use success, cost, and latency.

Your evaluation platform is live on the internet. Anyone with the URL can create tasks and trigger scored evaluations. Next up, a secret mission to catch when the agent makes things up.

Secret mission

Hallucination Detection Scorer

Your evaluation platform scores correctness and tool use. But what about fabrication? An agent can score high on correctness while still inventing details that never came from its tools. Build a hallucination detector that cross-references the agent's claims against what the tools actually returned.

Clean Up Your Resources

Clean Up Your Resources

This project uses Render's free tier for both the web service and Postgres database, so there are no ongoing costs. The free web service spins down after 15 minutes of inactivity. The free Postgres database expires automatically 30 days after creation.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Render free web service (your Agent Eval Platform API).
  • Render free Postgres database (evaluation task and result storage).
  • Local Docker Postgres container (via docker-compose.yml).
  • Local project directory on your Desktop.
  • GitHub repository.

Keep everything running

No action needed. Choose this if you plan to keep iterating on your evaluation platform.

  • Your Render web service costs nothing. It auto-sleeps after 15 minutes of inactivity and wakes on the next request.
  • Your Render Postgres database will expire 30 days after creation. You can recreate it when needed.
  • Your local Docker Postgres container continues to run and uses system memory. Stop it anytime with docker compose down from your project directory.
  • Your project files and virtual environment remain on your Desktop.

Pause - I'll come back to this later

Free up local memory while keeping your code and cloud resources intact.

  • Stop the local Docker Postgres container by running this from your project directory:
  • Your Render web service auto-sleeps after 15 minutes of inactivity. No action needed.
  • Your project files stay on your Desktop, ready to pick back up.
docker compose down

When you return, run docker compose up -d from your project directory to restart Postgres.

Delete - I don't want to use this again

Remove all project resources and start fresh.

  • Stop and remove the local Docker Postgres container and its data by running this from your project directory:
docker compose down -v
  • Delete your local project directory. On macOS or Linux:
cd ~/Desktop
rm -rf agent-eval-platform

On Windows:

cd ~\Desktop
Remove-Item -Recurse -Force agent-eval-platform
  • Delete your Render web service. Open the Render Dashboard, click your web service, then open its Settings page. Scroll to the bottom and click Delete Web Service. Type the service name to confirm.
  • Delete your Render Postgres database. In the Render Dashboard, click your database, open its Info page, scroll to the bottom, and click Delete Database.
  • Delete your GitHub repository. Go to your repository page on GitHub, click Settings, scroll to the bottom under Danger Zone, and click Delete this repository.

That's a Wrap!

That's a Wrap!

Nice work. You just built a production-grade AI agent evaluation platform from scratch and deployed it live on the internet.

You've learned how to:

  • Build a LangGraph tool-calling agent with custom tools and instrument it with OpenTelemetry to capture per-step cost, latency, and tool-use telemetry.
  • Create a scoring pipeline that uses an LLM-as-judge for correctness evaluation and extracts quantified metrics from trace data.
  • Deploy the entire system as a FastAPI REST API live on Render using Docker, accessible via a public URL.
  • Secret Mission: Added hallucination detection that cross-references agent claims against tool outputs.

Ready to quiz yourself?