Build a ReAct Agent with Tool Calling
Build a ReAct agent with tool calling and full Langfuse tracing.
Introduction
30 Second Summary
When you ask a smart assistant a complex question, it doesn't just blurt out the first thing that comes to mind. It figures out what information it needs, goes and gets it, then puts together a proper answer.
In this project, you will build a ReAct-style AI agent that reasons about which tools to use, calls them iteratively, and synthesizes answers from the results. Every reasoning step will be traced in a monitoring dashboard so you can see exactly how your agent thinks.
What You'll Build
A locally-running agentic RAG system served via FastAPI that decides how to answer questions using multiple tools, with full observability through Langfuse tracing.
By the end of this project, you'll have:
- A ReAct agent loop that reasons about tool selection and calls tools iteratively using Ollama's native tool-calling API with llama3.1.
- Three callable tools (knowledge base search, calculator, datetime) registered with a clean schema pattern and backed by ChromaDB.
- Full Langfuse tracing with nested agent, generation, and tool observations capturing inputs and outputs at every step.
- An evaluation pipeline that scores whether the agent selected the correct tools for each query type.
- Secret Mission: add a fourth tool that calls an external API and implement parallel tool calling.
Are there any prerequisites?
This project builds on Projects 1-3 of the LLMOps series. You should already have a working RAG pipeline with Ollama, ChromaDB, Langfuse, and FastAPI deployed locally.
You will also need Claude Code installed and a Windows 11 machine with at least 32 GB of RAM to run llama3.1:8b comfortably.
Before We Start
Before We Start...
Set Up SDKs and Verify Tool Calling
Your existing RAG pipeline sends every query straight to ChromaDB.
An agent needs more flexibility. It needs to reason about a question, decide which tool fits best, call that tool, and observe the result before answering. For that to work, your SDKs need to be at versions that support structured tool calling and the new tracing model.
In this step, get ready to:
- Upgrade the ollama and langfuse Python SDKs to tool-calling-ready versions.
- Verify Ollama returns structured tool_calls from llama3.1.
- Confirm Langfuse v4 tracing creates visible observations in your dashboard.
Upgrade the ollama SDK and verify tool calling
Ollama's Python SDK at version 0.6.2 supports the native tools= parameter in chat requests. This lets you pass tool definitions as structured schemas and receive tool_calls back from the model.
- Open your terminal and navigate to the folder where your RAG pipeline code from the previous projects lives.
- Upgrade the ollama SDK by running this command:
pip install ollama==0.6.2- Confirm the install succeeded by checking the version:
pip show ollamaYou should see Version: 0.6.2 in the output.
Seeing a permission error?
If pip reports a permissions issue, try running with the --user flag: pip install --user ollama==0.6.2. If you are using a virtual environment, make sure it is activated first.
Help me fix my pip install error
Now you need to verify that tool calling actually works. You will create a quick test script that sends a dummy tool definition to llama3.1 and checks that the response contains structured tool calls.
- Create a file called test_tool_calling.py in your project folder and paste the following code:
from ollama import chat
# Define a dummy tool schema to test tool calling
test_tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"required": ["location"],
"properties": {
"location": {
"type": "string",
"description": "The city name",
}
},
},
},
}
]
# Send a message that should trigger tool calling
messages = [{"role": "user", "content": "What is the weather in London?"}]
response = chat(model="llama3.1", messages=messages, tools=test_tools)
# Print the tool_calls to confirm it works
print("Tool calls:", response.message.tool_calls)What does this code do?
This script defines a fake get_weather tool with the exact schema format Ollama expects. It sends a question that should trigger that tool, then prints the response's tool_calls field.
If tool calling works, you will see a list of ToolCall objects instead of None. Each ToolCall has a function.name and function.arguments dict.
- Run the test script:
python test_tool_calling.pyYou should see output showing a tool_calls list with at least one entry containing get_weather as the function name and London in the arguments.
Seeing None or an empty list?
Make sure Ollama is running. Start it with ollama serve in a separate terminal if needed.
If the model returns a text response instead of tool calls, you may need to pull the latest version of llama3.1. You will do this in substep 3.
Why is my tool calling test returning None instead of tool_calls?
Upgrade Langfuse SDK and verify connectivity
Langfuse v4 introduced a unified observation model. Instead of separate methods for spans and generations, everything uses start_as_current_observation with an as_type parameter.
- Upgrade the Langfuse Python SDK by running:
pip install langfuse==4.12.0- Verify the install:
pip show langfuseYou should see Version: 4.12.0 in the output.
Now confirm that your Langfuse environment variables are still set and the SDK can send traces to your dashboard. You will create a test observation using the new v4 API.
- Create a file called test_langfuse.py and paste the following code:
from langfuse import get_client
langfuse = get_client()
# Create a test observation to verify v4 connectivity
with langfuse.start_as_current_observation(
as_type="span",
name="test",
input={"message": "Testing Langfuse v4 connection"},
) as obs:
obs.update(output={"status": "connected"})
langfuse.flush()
print("Test trace sent to Langfuse. Check your dashboard.")What does this code do?
get_client() returns the Langfuse client using your existing environment variables (LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST).
start_as_current_observation(as_type="span", name="test") creates a new observation of type span and automatically nests any child observations beneath it. The flush() call ensures the trace is sent before the script exits.
- Run the test:
python test_langfuse.py- Open your Langfuse Cloud dashboard in your browser.
- Click Traces in the left sidebar.
- You should see a new trace with a span named test near the top of the list.
No trace appearing in the dashboard?
Confirm your Langfuse environment variables are set. Run echo %LANGFUSE_PUBLIC_KEY% in your terminal. If it returns empty, re-export the keys from your previous project setup.
Also check that LANGFUSE_HOST points to your Langfuse Cloud URL. The default is https://cloud.langfuse.com.
Help me debug my Langfuse connectivity issue
Pull the latest llama3.1 and confirm tool calling via API
Tool calling requires a recent version of the llama3.1 model weights. Pulling the model ensures you have the latest release with full tool support.
- Pull the latest llama3.1 model by running:
ollama pull llama3.1This may take a minute if an update is available. If you already have the latest, it will confirm immediately.
Now verify tool calling at the raw HTTP API level. This confirms Ollama's server is serving tool calls correctly, independent of the Python SDK.
- Save the following JSON to a file called test_request.json in your project folder:
{
"model": "llama3.1",
"messages": [{"role": "user", "content": "What is 2+2?"}],
"tools": [{
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate a math expression",
"parameters": {
"type": "object",
"required": ["expression"],
"properties": {
"expression": {
"type": "string",
"description": "The math expression to evaluate"
}
}
}
}
}],
"stream": false
}- Send the request to Ollama's API by running:
curl http://localhost:11434/api/chat -d @test_request.jsonWhy test with curl separately?
Testing at the HTTP level proves that Ollama's server is returning tool_calls in the JSON response. If this works but the Python SDK does not, you know the issue is in the SDK layer. If both work, your full stack is ready.
In the JSON response, look for a "tool_calls" array inside the "message" object. It should contain a call to calculate with an expression argument.
Using -d @filename avoids quoting headaches
On Windows, passing JSON directly to curl requires awkward quote escaping. Loading from a file with -d @test_request.json works identically across all platforms.
Getting a connection refused error?
Make sure Ollama is running. Open another terminal and run ollama serve. Then retry the curl command.
If the response has a text message in content but no tool_calls array, ensure you did not include a "format": "json" field in your request. Using format: "json" with tools breaks tool calling in Ollama.
Help me troubleshoot my Ollama curl tool-calling test
Your SDKs are upgraded and both Ollama tool calling and Langfuse tracing are confirmed working. Next up, you will build the actual tools your agent will call.
Build the Tool Registry and Implementations
Your SDKs are upgraded and tool calling is confirmed working with Ollama. Now you need to give the agent something to actually call.
In this step, you will build three tools that the agent can invoke: a knowledge base search (backed by your existing ChromaDB collection), a calculator, and a datetime lookup. You will also create a registry that maps tool names to their implementations and defines the JSON schemas Ollama needs to understand each tool's interface.
In this step, get ready to:
- Create three tool functions in a dedicated module.
- Build a tool registry with Ollama-compatible JSON schemas.
- Test each tool independently to confirm correct output.
Create the tool functions
A ReAct agent decides which tools to use based on the user's question. Each tool is a plain Python function that accepts arguments, does work, and returns a string result.
- Create a new file called tools.py in your project directory.
- Add the imports and the first tool function, search_knowledge_base, by pasting this code:
import chromadb
from datetime import datetime
def search_knowledge_base(query: str) -> str:
"""Search the vector store for relevant documents."""
# Connect to the existing ChromaDB persistent store
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_collection("documents")
# Query for the top 3 most relevant results
results = collection.query(query_texts=[query], n_results=3)
if results["documents"] and results["documents"][0]:
docs = results["documents"][0]
return "\n\n".join(
f"[Result {i+1}]: {doc}" for i, doc in enumerate(docs)
)
return "No relevant documents found."What does this code do?
This function connects to the same ChromaDB collection you built in Projects 1-3. It takes a search query, finds the 3 most relevant document chunks, and formats them into a numbered list the agent can read.
If no documents match, it returns a clear message so the agent knows the knowledge base had nothing useful.
You won't see a change yet. The next chunk completes the file with the remaining two tools.
- Add the calculate and get_current_datetime functions below the existing code in tools.py:
def calculate(expression: str) -> str:
"""Safely evaluate a mathematical expression."""
# Only allow safe characters to prevent code injection
allowed_chars = set("0123456789+-*/().% ")
if not all(c in allowed_chars for c in expression):
return f"Error: Invalid characters in expression: {expression}"
try:
result = eval(expression)
return str(result)
except Exception as e:
return f"Error evaluating expression: {e}"
def get_current_datetime() -> str:
"""Get the current date and time."""
now = datetime.now()
return now.strftime("%Y-%m-%d %H:%M:%S (%A)")What does this code do?
- calculate uses a character allowlist to block anything that is not a digit or arithmetic operator. This prevents injection attacks when the agent runs math expressions through Python's eval().
- get_current_datetime gives the agent temporal grounding. Without it, the model has no idea what day or time it is.
Why use eval() with a character allowlist?
In production, you would use a dedicated math parser library. For this project, the allowlist approach teaches the concept of input sanitization for agent tools. The agent might pass arbitrary strings from user input, so your tools must defend themselves.
- Save tools.py.
Seeing a syntax error?
- Make sure there is a blank line between the end of search_knowledge_base and the start of calculate. Python requires blank lines between top-level function definitions.
- Check that the f-string quotes are consistent. Each f-string uses double quotes on the outside and single quotes or different delimiters inside.
Help me fix a syntax error in my tools.py file
✔️ Awesome, I've got everything!
Great. Make sure you have saved the file before moving on.
ⓧ I'd like to double check the full code
import chromadb
from datetime import datetime
def search_knowledge_base(query: str) -> str:
"""Search the vector store for relevant documents."""
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_collection("documents")
results = collection.query(query_texts=[query], n_results=3)
if results["documents"] and results["documents"][0]:
docs = results["documents"][0]
return "\n\n".join(
f"[Result {i+1}]: {doc}" for i, doc in enumerate(docs)
)
return "No relevant documents found."
def calculate(expression: str) -> str:
"""Safely evaluate a mathematical expression."""
allowed_chars = set("0123456789+-*/().% ")
if not all(c in allowed_chars for c in expression):
return f"Error: Invalid characters in expression: {expression}"
try:
result = eval(expression)
return str(result)
except Exception as e:
return f"Error evaluating expression: {e}"
def get_current_datetime() -> str:
"""Get the current date and time."""
now = datetime.now()
return now.strftime("%Y-%m-%d %H:%M:%S (%A)")Create the tool registry
Ollama needs a JSON schema for each tool so it knows what arguments to pass. The registry maps tool names to both their Python implementations and their schema definitions.
- Create a new file called tool_registry.py in your project directory.
- Add the imports and the implementation map by pasting this code:
from tools import search_knowledge_base, calculate, get_current_datetime
# Map tool names (as strings) to their Python functions
TOOL_IMPLEMENTATIONS = {
"search_knowledge_base": search_knowledge_base,
"calculate": calculate,
"get_current_datetime": get_current_datetime,
}What does this code do?
TOOL_IMPLEMENTATIONS is a dictionary that lets the agent look up a function by name. When Ollama returns a tool call like "search_knowledge_base", the agent loop uses this dict to find and execute the matching Python function.
You won't see a change yet. The next chunk adds the JSON schemas that Ollama reads to understand each tool.
- Add the tool schemas below the existing code in tool_registry.py:
# JSON schemas that tell Ollama what each tool does and what arguments it accepts
TOOL_SCHEMAS = [
{
"type": "function",
"function": {
"name": "search_knowledge_base",
"description": "Search the knowledge base for relevant documents. Use this when the user asks about topics that might be covered in the stored documents.",
"parameters": {
"type": "object",
"required": ["query"],
"properties": {
"query": {
"type": "string",
"description": "The search query to find relevant documents",
}
},
},
},
},
{
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate a mathematical expression. Use this for any calculation, percentage, or arithmetic question.",
"parameters": {
"type": "object",
"required": ["expression"],
"properties": {
"expression": {
"type": "string",
"description": "The mathematical expression to evaluate, e.g. '15/100 * 340'",
}
},
},
},
},
{
"type": "function",
"function": {
"name": "get_current_datetime",
"description": "Get the current date and time. Use this when the user asks about today's date, the current time, or needs temporal context.",
"parameters": {
"type": "object",
"properties": {},
},
},
},
]What does this code do?
TOOL_SCHEMAS is the list you pass to Ollama's tools= parameter. Each schema tells the model what the tool does (via the description) and what arguments it needs (via parameters).
The model reads these descriptions to decide which tool fits the user's question. Clear, specific descriptions lead to better tool selection.
- Save tool_registry.py.
- Confirm the file imports correctly by running:
python -c "from tool_registry import TOOL_SCHEMAS; print(len(TOOL_SCHEMAS), 'tools registered')"You should see 3 tools registered printed in the terminal.
Seeing an ImportError?
- Make sure both tools.py and tool_registry.py are in the same directory.
- Check that the function names in your tools.py match exactly: search_knowledge_base, calculate, get_current_datetime.
Help me debug my tool_registry.py import error
✔️ Awesome, I've got everything!
Great. Make sure you have saved the file before moving on.
ⓧ I'd like to double check the full code
from tools import search_knowledge_base, calculate, get_current_datetime
TOOL_IMPLEMENTATIONS = {
"search_knowledge_base": search_knowledge_base,
"calculate": calculate,
"get_current_datetime": get_current_datetime,
}
TOOL_SCHEMAS = [
{
"type": "function",
"function": {
"name": "search_knowledge_base",
"description": "Search the knowledge base for relevant documents. Use this when the user asks about topics that might be covered in the stored documents.",
"parameters": {
"type": "object",
"required": ["query"],
"properties": {
"query": {
"type": "string",
"description": "The search query to find relevant documents",
}
},
},
},
},
{
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate a mathematical expression. Use this for any calculation, percentage, or arithmetic question.",
"parameters": {
"type": "object",
"required": ["expression"],
"properties": {
"expression": {
"type": "string",
"description": "The mathematical expression to evaluate, e.g. '15/100 * 340'",
}
},
},
},
},
{
"type": "function",
"function": {
"name": "get_current_datetime",
"description": "Get the current date and time. Use this when the user asks about today's date, the current time, or needs temporal context.",
"parameters": {
"type": "object",
"properties": {},
},
},
},
]Test each tool independently
Before wiring these tools into the agent loop, confirm each one returns valid output on its own.
- Test the calculate tool by running:
python -c "from tools import calculate; print(calculate('15/100 * 340'))"You should see 51.0 printed.
- Test the get_current_datetime tool by running:
python -c "from tools import get_current_datetime; print(get_current_datetime())"You should see today's date, time, and day of the week in the format 2026-07-06 14:30:22 (Monday) (your values will differ).
- Test the search_knowledge_base tool by running:
python -c "from tools import search_knowledge_base; print(search_knowledge_base('machine learning'))"You should see numbered results from your ChromaDB collection, or No relevant documents found. if your collection is empty. Both confirm the function works correctly.
Getting a collection not found error?
- The search_knowledge_base function expects a ChromaDB collection named documents at the path ./chroma_db. Make sure you are running from the same project directory where you set up ChromaDB in the previous projects.
- If you moved directories, run your terminal commands from the folder containing the chroma_db directory.
Help me fix my ChromaDB connection in search_knowledge_base
- Verify the tool schemas are valid by running:
python -c "from tool_registry import TOOL_SCHEMAS; [print(t['function']['name']) for t in TOOL_SCHEMAS]"You should see the three tool names printed on separate lines: search_knowledge_base, calculate, and get_current_datetime.
Your tools are built, registered, and tested. Next up, you will wire them into a ReAct agent loop that reasons about which tool to call and iterates until it has a complete answer.
Implement the ReAct Agent Loop
You now have a set of tools and a registry that maps their names to implementations and schemas. The missing piece is a decision-maker that looks at a user's question and figures out which tool to call.
That decision-maker is a ReAct agent. It sends the user's query to the LLM along with the available tool schemas. The model reasons about which tool(s) to use, the agent executes those tools, feeds the results back, and loops until the model produces a final text answer.
In this step, get ready to:
- Create the agent module with a system prompt that explains the available tools.
- Build the iterative agent loop that calls tools and accumulates results.
- Test the agent with three different query types to confirm correct tool routing.
Create the agent module with the system prompt
The agent needs a system prompt that tells the LLM it is in a loop and can make multiple tool calls before producing a final answer. This prompt also lists the available tools so the model knows what it can reach for.
- Create a new file called agent.py in your project directory.
- Add the imports, Langfuse client initialization, and system prompt by pasting this code:
from ollama import chat
from langfuse import observe, get_client, propagate_attributes
from tool_registry import TOOL_IMPLEMENTATIONS, TOOL_SCHEMAS
langfuse = get_client()
AGENT_SYSTEM_PROMPT = """You are a helpful assistant with access to tools. You are in an agent loop and can make multiple tool calls to gather information before answering.
Available tools:
- search_knowledge_base: Search stored documents for relevant information
- calculate: Evaluate mathematical expressions
- get_current_datetime: Get the current date and time
Guidelines:
- Use tools when you need specific information to answer accurately.
- You may call multiple tools if needed.
- Once you have enough information, provide a clear final answer.
- If no tool is needed, answer directly from your knowledge."""What does this code do?
The imports pull in the Ollama chat function for LLM calls, Langfuse utilities for tracing (which you will wire up in the next step), and your tool registry from the previous step.
The AGENT_SYSTEM_PROMPT tells the LLM exactly what tools exist and when to use them. It also explicitly states that the model is in a loop and can call multiple tools before answering. This framing is critical for reliable tool selection.
Why import Langfuse now?
You will add full tracing in the next step. Importing observe, get_client, and propagate_attributes now means you only need to add decorators later without restructuring your imports.
Build the ReAct loop function
The core of the agent is a loop. Each iteration sends the full message history to the LLM, checks whether it wants to call a tool, and either executes that tool or returns the final answer.
- In agent.py, add the run_agent function below the system prompt by pasting this code:
def run_agent(query: str, max_iterations: int = 5) -> dict:
"""Run the ReAct agent loop."""
messages = [
{"role": "system", "content": AGENT_SYSTEM_PROMPT},
{"role": "user", "content": query},
]
tools_used = []
iteration = 0
for iteration in range(max_iterations):
response = chat(
model="llama3.1",
messages=messages,
tools=TOOL_SCHEMAS,
stream=False,
)
if not response.message.tool_calls:
return {
"answer": response.message.content,
"tools_used": tools_used,
"iterations": iteration + 1,
}
messages.append(response.message)
for tool_call in response.message.tool_calls:
tool_name = tool_call.function.name
tool_args = tool_call.function.arguments
if tool_name in TOOL_IMPLEMENTATIONS:
tool_fn = TOOL_IMPLEMENTATIONS[tool_name]
if tool_args:
result = tool_fn(**tool_args)
else:
result = tool_fn()
else:
result = f"Error: Unknown tool '{tool_name}'"
tools_used.append(tool_name)
messages.append({
"role": "tool",
"content": str(result),
"tool_name": tool_name,
})
return {
"answer": response.message.content or "Max iterations reached without final answer.",
"tools_used": tools_used,
"iterations": max_iterations,
}What does this loop do?
The function initializes a message history with the system prompt and user query, then enters the loop.
- Each iteration calls chat() with the full message history and the tool schemas. Ollama decides whether to call a tool or produce a final answer.
- If response.message.tool_calls is empty, the model has produced its final text answer. The function returns immediately.
- If tool calls are present, the function executes each one via TOOL_IMPLEMENTATIONS and appends the results as role: "tool" messages so the model can see them on the next iteration.
- The max_iterations guard prevents infinite loops if the model keeps requesting tools without converging.
Why not use format: "json" here?
Ollama's format: "json" option is incompatible with tool calling. If you set it, the model embeds tool calls in the content string instead of the tool_calls array. Always omit it when using the tools= parameter.
- Save agent.py.
Seeing an import error?
If Python reports it cannot find tool_registry, make sure tool_registry.py is in the same directory as agent.py. Check that the filename is exactly tool_registry.py with an underscore, not a hyphen.
Help me debug an import error in my agent.py file
✔️ Awesome, I've got everything!
Great. Double check you have saved agent.py before moving on.
ⓧ I'd like to double check the full code
from ollama import chat
from langfuse import observe, get_client, propagate_attributes
from tool_registry import TOOL_IMPLEMENTATIONS, TOOL_SCHEMAS
langfuse = get_client()
AGENT_SYSTEM_PROMPT = """You are a helpful assistant with access to tools. You are in an agent loop and can make multiple tool calls to gather information before answering.
Available tools:
- search_knowledge_base: Search stored documents for relevant information
- calculate: Evaluate mathematical expressions
- get_current_datetime: Get the current date and time
Guidelines:
- Use tools when you need specific information to answer accurately.
- You may call multiple tools if needed.
- Once you have enough information, provide a clear final answer.
- If no tool is needed, answer directly from your knowledge."""
def run_agent(query: str, max_iterations: int = 5) -> dict:
"""Run the ReAct agent loop."""
messages = [
{"role": "system", "content": AGENT_SYSTEM_PROMPT},
{"role": "user", "content": query},
]
tools_used = []
iteration = 0
for iteration in range(max_iterations):
response = chat(
model="llama3.1",
messages=messages,
tools=TOOL_SCHEMAS,
stream=False,
)
if not response.message.tool_calls:
return {
"answer": response.message.content,
"tools_used": tools_used,
"iterations": iteration + 1,
}
messages.append(response.message)
for tool_call in response.message.tool_calls:
tool_name = tool_call.function.name
tool_args = tool_call.function.arguments
if tool_name in TOOL_IMPLEMENTATIONS:
tool_fn = TOOL_IMPLEMENTATIONS[tool_name]
if tool_args:
result = tool_fn(**tool_args)
else:
result = tool_fn()
else:
result = f"Error: Unknown tool '{tool_name}'"
tools_used.append(tool_name)
messages.append({
"role": "tool",
"content": str(result),
"tool_name": tool_name,
})
return {
"answer": response.message.content or "Max iterations reached without final answer.",
"tools_used": tools_used,
"iterations": max_iterations,
}Test the agent with three query types
A good test covers each tool independently. You will send a math question, a date question, and a factual question to confirm the agent routes each to the correct tool.
- Test the calculate tool by running this command in your terminal:
python -c "from agent import run_agent; result = run_agent('What is 15 percent of 340?'); print(result)"- You should see a dictionary with "tools_used": ["calculate"] and an answer containing 51.
- Test the get_current_datetime tool by running:
python -c "from agent import run_agent; result = run_agent('What day of the week is it today?'); print(result)"- You should see "tools_used": ["get_current_datetime"] and an answer that mentions today's day of the week.
- Test the search_knowledge_base tool by running:
python -c "from agent import run_agent; result = run_agent('What do our documents say about machine learning?'); print(result)"- You should see "tools_used": ["search_knowledge_base"] and an answer synthesized from your ChromaDB documents.
What if the agent picks a different tool?
LLMs are non-deterministic. Occasionally llama3.1 might answer a math question from memory instead of calling calculate. If the tool selection looks wrong, try rephrasing the query to be more explicit (e.g. "Use a calculator to compute 15% of 340").
Agent hangs or times out?
Make sure Ollama is running. If the response is very slow, confirm llama3.1 is pulled and loaded by checking ollama list in a separate terminal.
If the agent loops without stopping, it may be hitting max_iterations. Check that your tool functions return valid strings. A tool that returns None can confuse the model into calling it again.
Help me debug my agent loop that seems to hang or hit max iterations
Your agent is making decisions and calling tools on its own. Next up, you will instrument every reasoning step with Langfuse tracing so you can see the full chain of thought in your dashboard.
Add Langfuse Tracing to the Agent
Your agent loop works. It reasons about queries, picks tools, and synthesizes answers. But right now you have no visibility into what happens inside the loop. If the agent picks the wrong tool or loops too many times, you would never know unless you manually print every step.
Langfuse solves this by recording every decision as a nested observation. In this step, you will instrument agent.py so that each agent run produces a full trace showing every LLM call and tool execution with their inputs and outputs.
In this step, get ready to:
- Add the @observe decorator and propagate_attributes wrapper to the agent function.
- Wrap each LLM call and tool execution with observation context managers that capture inputs and outputs.
- Verify nested traces appear correctly in your Langfuse dashboard.
Wrap the agent function with Langfuse decorators
Langfuse v4 uses a layered approach to tracing. The @observe decorator creates the root observation for the entire function. Inside, propagate_attributes attaches metadata to all child observations. Then start_as_current_observation context managers wrap individual operations as nested children.
- In agent.py, find the run_agent function (starting at def run_agent(query: str, max_iterations: int = 5) -> dict:).
- Replace the entire run_agent function (from the def line to the final return) with the following instrumented version:
@observe(as_type="agent")
def run_agent(query: str, max_iterations: int = 5) -> dict:
"""Run the ReAct agent loop."""
with propagate_attributes(
trace_name="agent-run",
metadata={"max_iterations": max_iterations},
):
messages = [
{"role": "system", "content": AGENT_SYSTEM_PROMPT},
{"role": "user", "content": query},
]
tools_used = []
iteration = 0
for iteration in range(max_iterations):
with langfuse.start_as_current_observation(
as_type="generation",
name=f"llm-reasoning-step-{iteration}",
input={"messages": messages},
) as gen_obs:
response = chat(
model="llama3.1",
messages=messages,
tools=TOOL_SCHEMAS,
stream=False,
)
gen_obs.update(output={"response": response.message.content, "tool_calls": bool(response.message.tool_calls)})
if not response.message.tool_calls:
return {
"answer": response.message.content,
"tools_used": tools_used,
"iterations": iteration + 1,
}
messages.append(response.message)
for tool_call in response.message.tool_calls:
tool_name = tool_call.function.name
tool_args = tool_call.function.arguments
with langfuse.start_as_current_observation(
as_type="tool",
name=tool_name,
input=tool_args,
) as tool_obs:
if tool_name in TOOL_IMPLEMENTATIONS:
tool_fn = TOOL_IMPLEMENTATIONS[tool_name]
if tool_args:
result = tool_fn(**tool_args)
else:
result = tool_fn()
else:
result = f"Error: Unknown tool '{tool_name}'"
tool_obs.update(output=result)
tools_used.append(tool_name)
messages.append({
"role": "tool",
"content": str(result),
"tool_name": tool_name,
})
return {
"answer": response.message.content or "Max iterations reached without final answer.",
"tools_used": tools_used,
"iterations": max_iterations,
}What does this code do?
This adds three layers of Langfuse instrumentation to the agent loop:
- @observe(as_type="agent") creates a root observation of type "agent" for the entire function call. This is the top-level node you will see in the Langfuse trace view.
- propagate_attributes(trace_name="agent-run", metadata={...}) names the trace and attaches metadata that flows automatically to all child observations created inside its scope.
- langfuse.start_as_current_observation(as_type="generation", ...) wraps each Ollama call. It records the messages array as input and the model's response as output. These appear as "generation" children nested under the agent.
- langfuse.start_as_current_observation(as_type="tool", ...) wraps each tool execution. It records the tool arguments as input and the tool result as output. These appear as "tool" children nested under the same parent.
- Save agent.py.
Seeing an indentation error?
The entire function body is now indented one extra level inside the with propagate_attributes(...) block. Make sure every line from messages = [ onward is indented 8 spaces (two levels inside the function).
help me fix indentation errors in my agent.py file after adding the propagate_attributes context manager
Understand the input and output capture
Each observation records what goes in and what comes out. This is what makes your Langfuse traces useful for debugging.
How does the I/O capture work?
For generation observations (LLM calls):
- input= receives the full messages array sent to Ollama. This lets you see exactly what context the model had at each reasoning step.
- gen_obs.update(output=...) records the model's text response and whether it decided to call tools. This tells you if the model is reasoning correctly.
For tool observations:
- input= receives tool_args (the arguments the model passed to the tool). This reveals whether the model extracted the right parameters from the query.
- tool_obs.update(output=result) records what the tool returned. This helps you verify that tools produce the data the model needs.
Run test queries and verify traces in Langfuse
Now you will run 5 diverse queries to generate traces. Each query exercises a different tool so you can verify the full observation hierarchy in your dashboard.
- Create a test script by running this command:
touch test_tracing.py- Add the following test code to test_tracing.py:
from agent import run_agent
from langfuse import get_client
langfuse = get_client()
# Five diverse queries to exercise different tools
test_queries = [
"What is 25% of 480?",
"What day of the week is it today?",
"Search the knowledge base for information about Python.",
"Calculate 2 to the power of 8.",
"What is the current date and time?",
]
for i, query in enumerate(test_queries):
print(f"\n[{i+1}/5] Query: {query}")
result = run_agent(query)
print(f" Answer: {result['answer'][:100]}")
print(f" Tools used: {result['tools_used']}")
print(f" Iterations: {result['iterations']}")
# Flush to ensure all observations are sent to Langfuse
langfuse.flush()
print("\nAll traces sent to Langfuse. Check your dashboard.")What does this script do?
It calls run_agent five times with queries targeting different tools: two math questions for calculate, two datetime questions for get_current_datetime, and one knowledge base question for search_knowledge_base.
The langfuse.flush() call at the end forces all buffered observations to be sent to the Langfuse server before the script exits.
- Run the test script:
python test_tracing.pyYou should see output showing each query, the answer, and which tools the agent used.
- Switch back to your Langfuse Cloud dashboard in your browser.
- Click Traces in the left sidebar.
- You should see 5 new traces, each named agent-run.
- Click on any trace to expand it. You should see a nested tree: the root "agent" observation containing "generation" children (one per LLM call) and "tool" children (one per tool execution).
- Click on a generation observation to verify it shows the full messages array as input and the model response as output.
- Click on a tool observation to verify it shows the tool arguments as input and the tool result as output.
Traces not appearing in Langfuse?
Make sure your LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST environment variables are set correctly in your terminal session.
If you see traces but they appear empty (no nested observations), check that your propagate_attributes block wraps all the code that creates child observations. Every start_as_current_observation call must be inside the with propagate_attributes(...) block.
help me debug why my Langfuse traces are empty or not showing up
✔️ Awesome, I've got everything!
Great. Double check you have saved agent.py with all the instrumentation in place.
ⓧ I'd like to double check the full code
from ollama import chat
from langfuse import observe, get_client, propagate_attributes
from tool_registry import TOOL_IMPLEMENTATIONS, TOOL_SCHEMAS
langfuse = get_client()
AGENT_SYSTEM_PROMPT = """You are a helpful assistant with access to tools. You are in an agent loop and can make multiple tool calls to gather information before answering.
Available tools:
- search_knowledge_base: Search stored documents for relevant information
- calculate: Evaluate mathematical expressions
- get_current_datetime: Get the current date and time
Guidelines:
- Use tools when you need specific information to answer accurately.
- You may call multiple tools if needed.
- Once you have enough information, provide a clear final answer.
- If no tool is needed, answer directly from your knowledge."""
@observe(as_type="agent")
def run_agent(query: str, max_iterations: int = 5) -> dict:
"""Run the ReAct agent loop."""
with propagate_attributes(
trace_name="agent-run",
metadata={"max_iterations": max_iterations},
):
messages = [
{"role": "system", "content": AGENT_SYSTEM_PROMPT},
{"role": "user", "content": query},
]
tools_used = []
iteration = 0
for iteration in range(max_iterations):
with langfuse.start_as_current_observation(
as_type="generation",
name=f"llm-reasoning-step-{iteration}",
input={"messages": messages},
) as gen_obs:
response = chat(
model="llama3.1",
messages=messages,
tools=TOOL_SCHEMAS,
stream=False,
)
gen_obs.update(output={"response": response.message.content, "tool_calls": bool(response.message.tool_calls)})
if not response.message.tool_calls:
return {
"answer": response.message.content,
"tools_used": tools_used,
"iterations": iteration + 1,
}
messages.append(response.message)
for tool_call in response.message.tool_calls:
tool_name = tool_call.function.name
tool_args = tool_call.function.arguments
with langfuse.start_as_current_observation(
as_type="tool",
name=tool_name,
input=tool_args,
) as tool_obs:
if tool_name in TOOL_IMPLEMENTATIONS:
tool_fn = TOOL_IMPLEMENTATIONS[tool_name]
if tool_args:
result = tool_fn(**tool_args)
else:
result = tool_fn()
else:
result = f"Error: Unknown tool '{tool_name}'"
tool_obs.update(output=result)
tools_used.append(tool_name)
messages.append({
"role": "tool",
"content": str(result),
"tool_name": tool_name,
})
return {
"answer": response.message.content or "Max iterations reached without final answer.",
"tools_used": tools_used,
"iterations": max_iterations,
}Your agent is now fully observable. Every reasoning step, tool call, and final answer is recorded in Langfuse with its inputs and outputs. Next up, you will expose the agent via a FastAPI endpoint and build an evaluation pipeline to score its tool-selection accuracy.
Expose via FastAPI and Evaluate
Your agent reasons about queries, calls the right tools, and produces fully traced executions in Langfuse. Now it is time to make it accessible as a proper API endpoint and measure how well it performs.
In this step, you'll add an /agent endpoint to your FastAPI app and build an evaluation pipeline that scores tool-selection accuracy across a structured test dataset.
In this step, get ready to:
- Add a /agent POST endpoint to your FastAPI app.
- Create an evaluation script with a 10-query test dataset.
- Run the evaluation and review accuracy results in Langfuse.
Add the /agent endpoint to FastAPI
Your existing main.py served the /query RAG endpoint from previous projects. You'll now update it to expose your agent through a dedicated /agent endpoint that returns structured responses.
- Open main.py in your editor and replace its contents with the following imports, data models, and app setup:
from fastapi import FastAPI
from pydantic import BaseModel
from agent import run_agent
from langfuse import get_client
app = FastAPI(title="RAG Agent API")
langfuse = get_client()
# Define the request shape
class AgentRequest(BaseModel):
query: str
max_iterations: int = 5
# Define the response shape
class AgentResponse(BaseModel):
answer: str
tools_used: list[str]
iterations: int
trace_id: strWhat does this code do?
AgentRequest defines what the caller sends: a query string and an optional max iteration limit (defaulting to 5).
AgentResponse defines what comes back: the final answer, which tools were used, how many iterations the loop took, and the Langfuse trace ID for observability.
- Below the AgentResponse class, add the endpoint function:
@app.post("/agent", response_model=AgentResponse)
def agent_endpoint(request: AgentRequest):
result = run_agent(
query=request.query,
max_iterations=request.max_iterations,
)
trace_id = langfuse.get_current_trace_id() or "unknown"
langfuse.flush()
return AgentResponse(
answer=result["answer"],
tools_used=result["tools_used"],
iterations=result["iterations"],
trace_id=trace_id,
)What does this endpoint do?
The endpoint calls run_agent() with the user's query. After execution, it grabs the current trace ID from Langfuse so the caller can look up the full trace.
langfuse.flush() ensures all trace data is sent to Langfuse before the response is returned. Without this, observations might be lost if the process shuts down.
- Save main.py.
Import errors?
Make sure agent.py is in the same directory as main.py. The from agent import run_agent line expects to find agent.py as a sibling file.
help me debug an import error in my FastAPI agent app
✔️ Awesome, I've got everything!
Great. Make sure you have saved main.py before moving on.
ⓧ I'd like to double check the full code
from fastapi import FastAPI
from pydantic import BaseModel
from agent import run_agent
from langfuse import get_client
app = FastAPI(title="RAG Agent API")
langfuse = get_client()
class AgentRequest(BaseModel):
query: str
max_iterations: int = 5
class AgentResponse(BaseModel):
answer: str
tools_used: list[str]
iterations: int
trace_id: str
@app.post("/agent", response_model=AgentResponse)
def agent_endpoint(request: AgentRequest):
result = run_agent(
query=request.query,
max_iterations=request.max_iterations,
)
trace_id = langfuse.get_current_trace_id() or "unknown"
langfuse.flush()
return AgentResponse(
answer=result["answer"],
tools_used=result["tools_used"],
iterations=result["iterations"],
trace_id=trace_id,
)Create the evaluation script
An agent that seems to work on a few manual tests might still make poor tool choices on edge cases. An evaluation dataset gives you a repeatable way to measure tool-selection accuracy across many query types.
- Create a new file called evaluate_agent.py in your project directory and add the imports and evaluation dataset:
from agent import run_agent
from langfuse import get_client
langfuse = get_client()
# 10 test queries with the tool each should trigger
EVAL_DATASET = [
{"query": "What is 15% of 340?", "expected_tools": ["calculate"]},
{"query": "What is the current date?", "expected_tools": ["get_current_datetime"]},
{"query": "What do our documents say about machine learning?", "expected_tools": ["search_knowledge_base"]},
{"query": "Calculate 2^10", "expected_tools": ["calculate"]},
{"query": "What day of the week is it today?", "expected_tools": ["get_current_datetime"]},
{"query": "Find information about neural networks in the knowledge base", "expected_tools": ["search_knowledge_base"]},
{"query": "What is 1024 / 16?", "expected_tools": ["calculate"]},
{"query": "Search our documents for deployment best practices", "expected_tools": ["search_knowledge_base"]},
{"query": "What time is it right now?", "expected_tools": ["get_current_datetime"]},
{"query": "Compute 3.14 * 25", "expected_tools": ["calculate"]},
]What is this dataset?
Each entry pairs a query with the tool name the agent should select. The dataset covers all three tools evenly: calculate, get_current_datetime, and search_knowledge_base.
After the agent runs each query, the script checks whether the expected tool appears anywhere in the tools the agent actually used. A match counts as a pass.
- Below the dataset, add the evaluation function and entry point:
def evaluate():
correct = 0
total = len(EVAL_DATASET)
for i, test_case in enumerate(EVAL_DATASET):
print(f"\n[{i+1}/{total}] Query: {test_case['query']}")
result = run_agent(test_case["query"])
actual_tools = result["tools_used"]
expected_tools = test_case["expected_tools"]
tool_match = any(t in actual_tools for t in expected_tools)
if tool_match:
correct += 1
print(f" PASS - Used: {actual_tools}, Expected: {expected_tools}")
else:
print(f" FAIL - Used: {actual_tools}, Expected: {expected_tools}")
accuracy = correct / total
print(f"\n{'='*50}")
print(f"Tool Selection Accuracy: {correct}/{total} ({accuracy:.0%})")
print(f"\n{'='*50}")
langfuse.flush()
return accuracy
if __name__ == "__main__":
evaluate()What does the evaluate function do?
It loops through all 10 queries, runs the full agent on each (including Langfuse tracing), and checks if the expected tool was among the tools actually used.
At the end, it prints an overall accuracy score like 8/10 (80%). The langfuse.flush() call ensures all 10 traced runs are sent to Langfuse before the script exits.
- Save evaluate_agent.py.
- Confirm the file exists by running:
ls evaluate_agent.pyYou should see evaluate_agent.py listed in the output.
✔️ Awesome, I've got everything!
Great. Make sure evaluate_agent.py is saved before continuing.
ⓧ I'd like to double check the full code
from agent import run_agent
from langfuse import get_client
langfuse = get_client()
EVAL_DATASET = [
{"query": "What is 15% of 340?", "expected_tools": ["calculate"]},
{"query": "What is the current date?", "expected_tools": ["get_current_datetime"]},
{"query": "What do our documents say about machine learning?", "expected_tools": ["search_knowledge_base"]},
{"query": "Calculate 2^10", "expected_tools": ["calculate"]},
{"query": "What day of the week is it today?", "expected_tools": ["get_current_datetime"]},
{"query": "Find information about neural networks in the knowledge base", "expected_tools": ["search_knowledge_base"]},
{"query": "What is 1024 / 16?", "expected_tools": ["calculate"]},
{"query": "Search our documents for deployment best practices", "expected_tools": ["search_knowledge_base"]},
{"query": "What time is it right now?", "expected_tools": ["get_current_datetime"]},
{"query": "Compute 3.14 * 25", "expected_tools": ["calculate"]},
]
def evaluate():
correct = 0
total = len(EVAL_DATASET)
for i, test_case in enumerate(EVAL_DATASET):
print(f"\n[{i+1}/{total}] Query: {test_case['query']}")
result = run_agent(test_case["query"])
actual_tools = result["tools_used"]
expected_tools = test_case["expected_tools"]
tool_match = any(t in actual_tools for t in expected_tools)
if tool_match:
correct += 1
print(f" PASS - Used: {actual_tools}, Expected: {expected_tools}")
else:
print(f" FAIL - Used: {actual_tools}, Expected: {expected_tools}")
accuracy = correct / total
print(f"\n{'='*50}")
print(f"Tool Selection Accuracy: {correct}/{total} ({accuracy:.0%})")
print(f"\n{'='*50}")
langfuse.flush()
return accuracy
if __name__ == "__main__":
evaluate()Run the evaluation and review results in Langfuse
With both files ready, you'll start the FastAPI server, test the endpoint manually, then run the full evaluation suite.
- Start the FastAPI server by running:
uvicorn main:app --reload- Test the /agent endpoint manually by running this in a separate terminal:
curl -X POST http://localhost:8000/agent -H "Content-Type: application/json" -d "{\"query\": \"What is 25% of 200?\"}"You should see a JSON response containing an answer, tools_used, iterations, and trace_id field.
Seeing a connection error?
Make sure Ollama is running in the background. The agent needs Ollama's API to process queries. Also confirm you're running the curl command in a separate terminal from the one running uvicorn.
help me debug a connection error when calling my /agent FastAPI endpoint
- Stop the FastAPI server with Ctrl+C.
- Run the evaluation script by running:
python evaluate_agent.pyThe script sends all 10 queries through your agent. Each query triggers the full ReAct loop with Langfuse tracing. This takes a few minutes since each query involves one or more LLM calls.
What should I see?
Your terminal prints PASS or FAIL for each query, then a final accuracy line like Tool Selection Accuracy: 8/10 (80%). Your exact score depends on how consistently llama3.1 selects the right tool for each query type.
- Switch back to your Langfuse dashboard in the browser.
- Click Traces in the left sidebar.
- You should see 10 new traces (one per evaluation query), each with the nested agent, generation, and tool observations from your earlier tracing work.
Why are these traces useful?
Each failed evaluation query now has a full trace you can inspect. If the agent picked the wrong tool, you can open that trace and see exactly what the LLM's reasoning step looked like before it made the tool call.
This is how production teams debug agent behavior: run an eval suite, find the failures, then drill into individual traces to understand why.
Not seeing new traces?
Make sure the LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY environment variables are still set in your terminal session. The langfuse.flush() call at the end of the script forces all data to be sent, but it needs valid credentials.
help me figure out why my evaluation traces are not appearing in Langfuse
Secret mission
Parallel Tool Calling with an External API
Your agent can reason and call tools, but it's limited to local data. What if a question requires both your knowledge base AND fresh information from the web? In this mission, you'll add a fourth tool and push the agent into parallel tool calling territory.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally except for Langfuse Cloud traces, which live on the free tier with no ongoing cost. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Local Python files: tools.py, tool_registry.py, agent.py, evaluate_agent.py, main.py.
- Running processes: FastAPI server, Ollama.
- Langfuse Cloud traces and evaluation scores (free tier, no cost).
Keep everything running
No action needed. Choose this if you are still actively building on this agent or continuing with the next project in the series.
- Your FastAPI server continues serving the /agent endpoint locally.
- Ollama keeps the llama3.1 model loaded and ready for tool-calling requests.
- Langfuse Cloud retains your traces indefinitely on the free tier at no cost.
Pause - I'll come back to this later
Shut down running processes to free up memory and GPU resources. Your files and Langfuse traces stay intact.
- Stop the FastAPI server by pressing Ctrl+C in the terminal where it is running.
- Stop Ollama to free GPU and RAM by running:
ollama stop llama3.1- Your Python files remain on disk. Your Langfuse traces remain in the cloud dashboard.
- When you return, restart Ollama and the FastAPI server to pick up where you left off.
Delete - I don't want to use this again
Remove all project resources and start fresh. This is irreversible for Langfuse traces.
- Stop the FastAPI server by pressing Ctrl+C in the terminal where it is running.
- Stop Ollama by running:
ollama stop llama3.1- Delete the local Python files by running:
rm tools.py tool_registry.py agent.py evaluate_agent.py main.py- Delete your Langfuse traces: open the Langfuse dashboard, navigate to the Traces view in your project, select the traces from this project, then choose Delete in the Actions dropdown.
Nice Work!
Nice Work!
Nice work! You just built a production-grade ReAct agent that reasons about which tools to use, calls them iteratively, and traces every decision in an observability platform.
You've learned how to:
- Build a ReAct agent loop that reasons about tool selection and calls tools iteratively using Ollama's native tool-calling API with llama3.1.
- Implement a tool registry pattern with three tools (knowledge base search, calculator, datetime) and wire them into the agent with structured JSON schemas.
- Instrument every agent step with Langfuse v4 tracing using nested agent, generation, and tool observations with full input/output capture, and build an evaluation pipeline that scores tool-selection accuracy.
- Secret Mission: Added a fourth web search tool calling an external API and implemented parallel tool calling with sibling tool observations in Langfuse.
Ready to quiz yourself?