Build a Code Verification Agent
Build an AI agent that verifies Python functions using property-based testing.
Introduction
30 Second Summary
Every developer has shipped code that "seemed to work" only to discover a hidden bug in production. What if an AI agent could automatically prove whether your code is correct before it ever leaves your machine?
In this project, you will build VerifyBot, a Python CLI agent that takes any function, reasons about its mathematical properties, generates tests, and reports whether the code is correct or broken.
What You'll Build
VerifyBot is a multi-tool AI agent powered by Groq that uses property-based testing to autonomously verify Python functions.
By the end of this project, you'll have:
- A multi-tool AI agent with a full ReAct agentic loop (THINK > ACT > OBSERVE).
- Three custom tools that analyze functions, generate Hypothesis property tests, and execute them.
- An orchestration engine that chains tool calls until verification is complete.
- Secret Mission: Add a "Fix Mode" where the agent automatically proposes and verifies corrected versions of buggy functions.
Are there any prerequisites?
You need basic Python knowledge and a free Groq API key. If you don't have a Groq account yet, you can sign up at console.groq.com and generate a key in seconds.
Set Up Your Environment
Before your agent can reason about code, it needs a few building blocks. The Groq SDK connects to the LLM that powers your agent's reasoning. Hypothesis generates property-based tests. And python-dotenv keeps your API key out of your source code.
In this step, you'll create a project folder, install those three packages, and confirm everything works with a quick API call to Groq.
In this step, get ready to:
- Install the groq, hypothesis, and python-dotenv packages.
- Create a .env file to store your API key.
- Verify your Groq API connection works.
Create your project folder and install packages
You need a dedicated folder for your project files and a terminal to run commands.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Visual Studio Code and press Enter to open it.
- Open the integrated terminal by pressing Ctrl+` (backtick key, both macOS and Windows).
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir verifybot
cd verifybot- Open this folder in VS Code by clicking File in the top menu bar.
- Click Open Folder.
- Navigate to your Desktop, select the verifybot folder, and click Open.
Why open the folder?
Opening the folder in VS Code makes it your workspace root. All file paths and terminal commands will start from verifybot automatically.
- Open a fresh terminal in the new workspace by pressing Ctrl+`.
- Install the three packages your project needs by running:
pip install groq hypothesis python-dotenv- You should see output ending with Successfully installed followed by the package names.
Seeing a permission error or command not found?
- If pip is not found, try pip3 instead. Some systems use pip3 for Python 3.
- If you see a permissions error on macOS, try adding --user to the end of the command.
Still stuck? Help me fix my pip install error for groq, hypothesis, and python-dotenv.
Create your .env file
Your agent needs an API key to talk to Groq's servers. A .env file stores the key locally so it never ends up in your code.
- Create a new file in VS Code by clicking File > New File.
- When prompted for a file name, type .env and press Enter.
- Paste the following line into the file:
GROQ_API_KEY=your-api-key-here- Replace your-api-key-here with your actual Groq API key.
- Save the file.
Where do I find my Groq API key?
Go to console.groq.com, sign in, and navigate to API Keys in the left sidebar. Create a new key if you don't already have one.
Keep your key safe
Never share your API key or commit your .env file to version control. The python-dotenv library loads this file into your environment at runtime so your code can access the key without hardcoding it.
Verify your Groq API connection
Time to confirm everything is wired up correctly. You'll write a tiny script that sends a message to Groq and prints the response.
- Create a new file called agent.py by clicking File > New File.
- Type agent.py as the file name and press Enter.
- Add the following code to agent.py:
import os
from dotenv import load_dotenv
from groq import Groq
# Load environment variables from .env
load_dotenv()
# Create a Groq client (reads GROQ_API_KEY automatically)
client = Groq()
# Send a test message and print the response
print(client.chat.completions.create(model='llama-3.3-70b-versatile', messages=[{'role': 'user', 'content': 'Say hello!'}]).choices[0].message.content)What does this code do?
- load_dotenv() reads your .env file and loads GROQ_API_KEY into the environment.
- Groq() creates a client that automatically picks up the key from the environment.
- The chat.completions.create call sends your message to the llama-3.3-70b-versatile model and returns a response.
- Save agent.py.
- Run the script in your terminal:
python agent.pyYou should see a friendly greeting from the LLM printed in your terminal.
Seeing an authentication error?
- Double-check that your .env file is in the same folder as agent.py (both inside verifybot).
- Make sure there are no extra spaces around the = sign in your .env file.
- Confirm your API key is valid by signing into console.groq.com and checking the API Keys section.
Help me debug my Groq API authentication error in Python.
✔️ Awesome, I've got everything!
Great. Make sure you've saved both files before moving on.
ⓧ I'd like to double check the full code
Here is what each file should contain at this point.
.env:
GROQ_API_KEY=your-api-key-hereagent.py:
import os
from dotenv import load_dotenv
from groq import Groq
load_dotenv()
client = Groq()
print(client.chat.completions.create(model='llama-3.3-70b-versatile', messages=[{'role': 'user', 'content': 'Say hello!'}]).choices[0].message.content)Your environment is ready and your agent can talk to Groq. Next up, you'll build the three tools that give your agent the power to analyze, test, and verify code.
Build the Tool Implementations
Your agent needs hands. Right now it can think (via the LLM) but it cannot do anything. Tools are the Python functions an agent calls to interact with the world.
In this step, you will build three tools that give VerifyBot its verification superpowers: one to analyze code, one to generate tests, and one to execute them.
In this step, get ready to:
- Create the analyze_function tool that inspects a function's signature and docstring.
- Create the generate_property_test tool that stores a Hypothesis test as a string.
- Create the run_test tool that dynamically executes tests and captures results.
Create the analyze_function tool
Before your agent can verify a function, it needs to understand what that function does. The analyze_function tool parses a Python function's code to extract its signature, docstring, and structure.
First, you need to replace the test code in agent.py with a proper foundation for your agent.
- Select all the existing code in agent.py by pressing Cmd+A (macOS) or Ctrl+A (Windows).
- Delete it and replace it with the following imports and setup code:
import json
import os
import subprocess
import sys
import tempfile
import textwrap
from dotenv import load_dotenv
from groq import Groq
load_dotenv()
client = Groq()
MODEL = "llama-3.3-70b-versatile"
# =============================================================================
# Tool Implementations
# =============================================================================What does this code do?
The imports bring in everything your tools will need. json serializes tool results. subprocess and tempfile let the run_test tool execute generated tests safely in a separate process.
The MODEL constant stores the model ID in one place so you can swap models later without hunting through your code.
- Save agent.py.
- Confirm there are no red underlines in VS Code. If you see any, check that all six standard library imports are spelled correctly.
Seeing a red underline under groq or dotenv?
This means VS Code is using a different Python interpreter than the one you installed your packages into. Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Python: Select Interpreter, and choose the Python 3.13 environment where you ran pip install.
Help me fix import errors in VS Code
Now add the first tool. This function will let the agent inspect any Python function it receives.
- Add the following function below the # Tool Implementations comment section in agent.py:
def analyze_function(function_code: str) -> str:
"""Analyze a Python function and return its signature, docstring, and structure."""
try:
# Extract basic info by parsing
lines = function_code.strip().split("\n")
signature = lines[0] if lines else "Unknown"
# Check for docstring
docstring = ""
if len(lines) > 1:
for line in lines[1:]:
stripped = line.strip()
if stripped.startswith('"""') or stripped.startswith("'''"):
docstring = stripped.strip("\"'")
break
elif stripped.startswith('#'):
docstring = stripped.lstrip('# ')
break
return json.dumps({
"signature": signature,
"docstring": docstring,
"line_count": len(lines),
"code": function_code
})
except Exception as e:
return json.dumps({"error": str(e)})What does this code do?
- It splits the function code into lines and grabs the first line as the signature (e.g., def my_sort(lst):).
- It scans subsequent lines for a docstring or comment to understand the function's purpose.
- It returns a JSON string because all tool outputs must be strings that the LLM can read.
- Save agent.py.
You will not see output yet. This function is called by the agent loop, which you will build in the next step.
Create the generate_property_test tool
Once the agent understands a function, it needs a way to produce tests. This tool stores a Hypothesis property-based test that the agent generates. The test code itself is a string that the run_test tool will execute later.
- Add the following function below analyze_function in agent.py:
def generate_property_test(
function_code: str,
test_code: str,
test_description: str
) -> str:
"""Store a generated property-based test for later execution."""
return json.dumps({
"test_description": test_description,
"test_code": test_code,
"function_code": function_code,
"status": "generated"
})What does this code do?
This tool is deliberately simple. It packages the test code, description, and target function into a JSON string. The real intelligence comes from the LLM, which decides WHAT test to write. This tool just stores the result so the agent can pass it to run_test later.
Why is this a separate tool from run_test?
Separating generation from execution gives the agent a chance to reason between steps. It can generate a test, inspect the result, then decide whether to run it or generate a different one. This mirrors how the ReAct pattern works: THINK, then ACT, then OBSERVE.
- Save agent.py.
You will not see output yet. The next tool is the one that makes things happen.
Create the run_test tool
This is the most powerful tool in your agent's toolkit. It takes the function code and a Hypothesis test, writes them to a temporary file, executes that file in a subprocess, and reports whether the test passed or found a counterexample.
- Add the following function below generate_property_test in agent.py:
def run_test(function_code: str, test_code: str) -> str:
"""Execute a Hypothesis property-based test and return the results."""
full_code = textwrap.dedent(f"""
import json
import sys
from hypothesis import given, strategies as st, settings
{function_code}
{test_code}
if __name__ == "__main__":
try:
test_func = [v for k, v in globals().items() if k.startswith("test_")][0]
test_func()
print(json.dumps({{"status": "passed", "message": "All property tests passed!"}}))
except AssertionError as e:
print(json.dumps({{"status": "failed", "message": str(e), "type": "assertion_error"}}))
except Exception as e:
print(json.dumps({{"status": "error", "message": str(e), "type": type(e).__name__}}))
""")
try:
with tempfile.NamedTemporaryFile(
mode='w', suffix='.py', delete=False
) as f:
f.write(full_code)
temp_path = f.name
result = subprocess.run(
[sys.executable, temp_path],
capture_output=True,
text=True,
timeout=30
)
os.unlink(temp_path)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip()
elif result.stderr:
# Parse Hypothesis output for counterexamples
stderr = result.stderr
if "Falsifying example" in stderr:
return json.dumps({
"status": "failed",
"message": "Property violation found",
"details": stderr[-500:]
})
return json.dumps({
"status": "error",
"message": stderr[-500:]
})
else:
return json.dumps({
"status": "error",
"message": "No output from test execution"
})
except subprocess.TimeoutExpired:
os.unlink(temp_path)
return json.dumps({
"status": "error",
"message": "Test execution timed out (30s limit)"
})
except Exception as e:
return json.dumps({"status": "error", "message": str(e)})What does this code do?
- It builds a complete Python script from the function code and test code, including all necessary Hypothesis imports.
- It writes that script to a temporary file and runs it as a subprocess using sys.executable (your current Python interpreter).
- The 30-second timeout prevents infinite loops from hanging your agent.
- It checks for "Falsifying example" in stderr, which is how Hypothesis reports counterexamples that break a property.
Why run tests in a subprocess?
Running LLM-generated code in your main process would be dangerous. A bug in the generated test could crash your entire agent. The subprocess acts as a sandbox. If the test crashes, your agent just reads the error and moves on.
- Save agent.py.
Seeing a syntax error on the f-string?
The double braces {{ and }} inside the f-string are how you write literal curly braces. Make sure every { that is part of a JSON structure is doubled. The only single braces should be around {function_code} and {test_code} (the template variables).
Help me fix a syntax error in my f-string
Now you need a registry so the agent loop can look up the right function by name when the LLM asks to call a tool.
- Add the following code at the bottom of agent.py, below the run_test function:
# =============================================================================
# Tool Registry
# =============================================================================
available_functions = {
"analyze_function": analyze_function,
"generate_property_test": generate_property_test,
"run_test": run_test,
}What does this code do?
The available_functions dictionary maps tool names (as strings) to their Python function objects. When the LLM says "call run_test", your agent loop will use this dict to find and execute the actual run_test function.
- Save agent.py.
- Verify the file has no errors by running it in your terminal:
python agent.pyYou should see no output and no errors. The file defines functions and a dictionary but does not call anything yet.
Seeing an error when you run the file?
If you see IndentationError, make sure your editor is using spaces, not tabs. If you see NameError, check that the functions are defined in order (analyze_function, then generate_property_test, then run_test) before the available_functions dictionary.
Help me debug my agent.py file
✔️ Awesome, I've got everything!
Great. Double check you have saved agent.py and that running it produces no output and no errors.
ⓧ I'd like to double check the full code
Here is the complete agent.py at this point:
import json
import os
import subprocess
import sys
import tempfile
import textwrap
from dotenv import load_dotenv
from groq import Groq
load_dotenv()
client = Groq()
MODEL = "llama-3.3-70b-versatile"
# =============================================================================
# Tool Implementations
# =============================================================================
def analyze_function(function_code: str) -> str:
"""Analyze a Python function and return its signature, docstring, and structure."""
try:
# Extract basic info by parsing
lines = function_code.strip().split("\n")
signature = lines[0] if lines else "Unknown"
# Check for docstring
docstring = ""
if len(lines) > 1:
for line in lines[1:]:
stripped = line.strip()
if stripped.startswith('"""') or stripped.startswith("'''"):
docstring = stripped.strip("\"'")
break
elif stripped.startswith('#'):
docstring = stripped.lstrip('# ')
break
return json.dumps({
"signature": signature,
"docstring": docstring,
"line_count": len(lines),
"code": function_code
})
except Exception as e:
return json.dumps({"error": str(e)})
def generate_property_test(
function_code: str,
test_code: str,
test_description: str
) -> str:
"""Store a generated property-based test for later execution."""
return json.dumps({
"test_description": test_description,
"test_code": test_code,
"function_code": function_code,
"status": "generated"
})
def run_test(function_code: str, test_code: str) -> str:
"""Execute a Hypothesis property-based test and return the results."""
full_code = textwrap.dedent(f"""
import json
import sys
from hypothesis import given, strategies as st, settings
{function_code}
{test_code}
if __name__ == "__main__":
try:
test_func = [v for k, v in globals().items() if k.startswith("test_")][0]
test_func()
print(json.dumps({{"status": "passed", "message": "All property tests passed!"}}))
except AssertionError as e:
print(json.dumps({{"status": "failed", "message": str(e), "type": "assertion_error"}}))
except Exception as e:
print(json.dumps({{"status": "error", "message": str(e), "type": type(e).__name__}}))
""")
try:
with tempfile.NamedTemporaryFile(
mode='w', suffix='.py', delete=False
) as f:
f.write(full_code)
temp_path = f.name
result = subprocess.run(
[sys.executable, temp_path],
capture_output=True,
text=True,
timeout=30
)
os.unlink(temp_path)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip()
elif result.stderr:
# Parse Hypothesis output for counterexamples
stderr = result.stderr
if "Falsifying example" in stderr:
return json.dumps({
"status": "failed",
"message": "Property violation found",
"details": stderr[-500:]
})
return json.dumps({
"status": "error",
"message": stderr[-500:]
})
else:
return json.dumps({
"status": "error",
"message": "No output from test execution"
})
except subprocess.TimeoutExpired:
os.unlink(temp_path)
return json.dumps({
"status": "error",
"message": "Test execution timed out (30s limit)"
})
except Exception as e:
return json.dumps({"status": "error", "message": str(e)})
# =============================================================================
# Tool Registry
# =============================================================================
available_functions = {
"analyze_function": analyze_function,
"generate_property_test": generate_property_test,
"run_test": run_test,
}Your agent now has three working tools. Next up, you will define the tool schemas that tell the LLM what these tools do and wire up the agentic loop that chains them together.
Define Tool Schemas and Wire Up the Agent
You have three tool functions ready to go, but your LLM has no idea they exist. For an AI agent to call a function, it needs a structured description of what that function does, what parameters it accepts, and what each parameter means.
In this step, you will define JSON tool schemas that describe your tools to the model, write a system prompt that instructs the agent how to reason, and build the orchestration loop that ties everything together into a working agent.
In this step, get ready to:
- Define JSON tool schemas so the model knows how to call each tool.
- Write a system prompt that guides VerifyBot's verification workflow.
- Implement the agentic loop that chains tool calls until verification is complete.
Define the tool schemas
Tool schemas are JSON objects that tell the model exactly what functions are available and how to call them. Each schema includes a name, a description of what the tool does, and a parameters object listing each argument with its type.
- In agent.py, find the available_functions dictionary at the bottom of your file.
- Add the following tool schemas below it:
# =============================================================================
# Tool Schemas
# =============================================================================
tools = [
{
"type": "function",
"function": {
"name": "analyze_function",
"description": "Analyze a Python function to understand its signature, purpose, and structure. Use this first to understand what the function does before generating tests.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code to analyze"
}
},
"required": ["function_code"]
}
}
},
{
"type": "function",
"function": {
"name": "generate_property_test",
"description": "Generate a Hypothesis property-based test for a function. The test should use @given decorators with appropriate strategies. Return both the function code and the test code.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code being tested"
},
"test_code": {
"type": "string",
"description": "The complete Hypothesis test code using @given and strategies"
},
"test_description": {
"type": "string",
"description": "A description of what property this test verifies"
}
},
"required": ["function_code", "test_code", "test_description"]
}
}
},
{
"type": "function",
"function": {
"name": "run_test",
"description": "Execute a property-based test against a function. Returns whether the test passed or failed, including any counterexamples found.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code to test"
},
"test_code": {
"type": "string",
"description": "The Hypothesis test code to execute"
}
},
"required": ["function_code", "test_code"]
}
}
}
]What does this code do?
Each tool schema follows the OpenAI-compatible format that Groq uses for tool calling. The model reads these schemas to decide which tool to invoke and what arguments to pass.
- "name" matches the key in available_functions so the loop can look up the right Python function.
- "description" tells the model when and why to use the tool.
- "parameters" defines each argument's type and purpose so the model generates valid JSON arguments.
- Save agent.py.
Seeing a syntax error?
The tools list uses nested dictionaries and lists. Double-check that every opening { has a matching } and every opening [ has a matching ]. Make sure the entire block sits below the available_functions dictionary, not inside it.
Help me fix a syntax error in my tools schema list.
Write the system prompt
The system prompt is the instruction set that shapes how your agent thinks and acts. It defines the verification workflow and tells the model which properties to look for when testing a function.
- In agent.py, add the following system prompt below the tools list:
# =============================================================================
# System Prompt
# =============================================================================
SYSTEM_PROMPT = """You are VerifyBot, an AI agent that verifies Python functions using property-based testing.
Your workflow:
1. ANALYZE: First use analyze_function to understand what the function does
2. REASON: Think about what mathematical/logical properties the function should satisfy
3. GENERATE: Use generate_property_test to create Hypothesis tests for those properties
4. TEST: Use run_test to execute each test
5. REPORT: Summarize findings - did the function pass all property tests, or were bugs found?
Properties to consider:
- Idempotency: f(f(x)) == f(x) where applicable
- Invariants: what should always be true about the output?
- Round-trip: if there's an inverse, does f_inv(f(x)) == x?
- Edge cases: empty inputs, zero, negative numbers
- Ordering: if the function sorts, is the output always sorted?
- Size: does the output have expected length/size relative to input?
Generate 2-3 property tests per function. Be thorough but focused.
When reporting results, clearly state which properties were verified and any violations found."""What does this prompt do?
This prompt establishes the ReAct pattern: the model should THINK about properties, ACT by calling tools, then OBSERVE results and decide what to do next.
The numbered workflow gives the model a clear sequence to follow. The properties list gives it concrete ideas for what to test. Without these hints, the model might generate trivial or unfocused tests.
- Save agent.py.
Implement the agentic loop
The agentic loop is the engine that makes your agent autonomous. It sends a message to the model, checks if the model wants to call a tool, executes that tool, feeds the result back, and repeats until the model delivers a final answer.
- In agent.py, add the following run_agent function below the SYSTEM_PROMPT:
# =============================================================================
# Agentic Loop
# =============================================================================
def run_agent(function_code: str) -> None:
"""Run the verification agent on a given function."""
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{
"role": "user",
"content": f"Please verify the following Python function using property-based testing:\n\n```python\n{function_code}\n```"
}
]
max_iterations = 10
iteration = 0
print("\n" + "=" * 60)
print("VERIFYBOT - Code Verification Agent")
print("=" * 60)
print(f"\nVerifying function:\n{function_code}\n")
print("-" * 60)
while iteration < max_iterations:
iteration += 1
response = client.chat.completions.create(
model=MODEL,
messages=messages,
tools=tools,
tool_choice="auto",
temperature=0.2
)
response_message = response.choices[0].message
# If no tool calls, we have the final answer
if not response_message.tool_calls:
print(f"\n{'=' * 60}")
print("VERIFICATION REPORT")
print("=" * 60)
print(response_message.content)
return
# Process tool calls
messages.append(response_message)
for tool_call in response_message.tool_calls:
function_name = tool_call.function.name
function_args = json.loads(tool_call.function.arguments)
print(f"\n[Step {iteration}] ACT: {function_name}")
print(f" Args: {json.dumps(function_args, indent=2)[:200]}...")
# Execute the tool
function_to_call = available_functions[function_name]
function_response = function_to_call(**function_args)
# Parse and display result
try:
result = json.loads(function_response)
status = result.get("status", "completed")
print(f" OBSERVE: {status}")
if status == "failed":
print(f" Details: {result.get('message', '')[:200]}")
except json.JSONDecodeError:
print(f" OBSERVE: {function_response[:100]}")
# Add tool result to conversation
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"name": function_name,
"content": function_response
})
print("\nMax iterations reached. Agent stopped.")What does this code do?
This is the complete ReAct loop. Here is how each piece works:
- messages starts with the system prompt and the user's function. Every tool call and result gets appended, building up multi-turn context.
- client.chat.completions.create sends the conversation to Groq with tool_choice="auto" so the model can decide whether to call a tool or respond with text.
- When response_message.tool_calls is empty, the model is done reasoning and has produced a final verification report.
- When tool calls exist, the loop executes each one via available_functions and sends the result back as a role: "tool" message.
- max_iterations = 10 prevents infinite loops if the model keeps calling tools without converging on an answer.
Why low temperature?
The temperature=0.2 setting keeps the model's tool calls predictable. A higher temperature would make it more creative with test ideas, but also more likely to produce malformed JSON arguments or call the wrong tool.
- Save agent.py.
Seeing an indentation error?
Make sure the entire run_agent function is at the top-level of the file (no extra indentation). The while loop should be indented one level inside the function, and the for tool_call loop should be indented two levels.
Help me fix an indentation error in my run_agent function.
✔️ Awesome, I've got everything!
Great. Double check you have saved agent.py with all three new sections: the tool schemas, system prompt, and agentic loop.
ⓧ I'd like to double check the full code
import json
import os
import subprocess
import sys
import tempfile
import textwrap
from dotenv import load_dotenv
from groq import Groq
load_dotenv()
client = Groq()
MODEL = "llama-3.3-70b-versatile"
# =============================================================================
# Tool Implementations
# =============================================================================
def analyze_function(function_code: str) -> str:
"""Analyze a Python function and return its signature, docstring, and structure."""
try:
# Extract basic info by parsing
lines = function_code.strip().split("\n")
signature = lines[0] if lines else "Unknown"
# Check for docstring
docstring = ""
if len(lines) > 1:
for line in lines[1:]:
stripped = line.strip()
if stripped.startswith('"""') or stripped.startswith("'''"):
docstring = stripped.strip("\"'")
break
elif stripped.startswith('#'):
docstring = stripped.lstrip('# ')
break
return json.dumps({
"signature": signature,
"docstring": docstring,
"line_count": len(lines),
"code": function_code
})
except Exception as e:
return json.dumps({"error": str(e)})
def generate_property_test(
function_code: str,
test_code: str,
test_description: str
) -> str:
"""Store a generated property-based test for later execution."""
return json.dumps({
"test_description": test_description,
"test_code": test_code,
"function_code": function_code,
"status": "generated"
})
def run_test(function_code: str, test_code: str) -> str:
"""Execute a Hypothesis property-based test and return the results."""
full_code = textwrap.dedent(f"""
import json
import sys
from hypothesis import given, strategies as st, settings
{function_code}
{test_code}
if __name__ == "__main__":
try:
test_func = [v for k, v in globals().items() if k.startswith("test_")][0]
test_func()
print(json.dumps({{"status": "passed", "message": "All property tests passed!"}}))
except AssertionError as e:
print(json.dumps({{"status": "failed", "message": str(e), "type": "assertion_error"}}))
except Exception as e:
print(json.dumps({{"status": "error", "message": str(e), "type": type(e).__name__}}))
""")
try:
with tempfile.NamedTemporaryFile(
mode='w', suffix='.py', delete=False
) as f:
f.write(full_code)
temp_path = f.name
result = subprocess.run(
[sys.executable, temp_path],
capture_output=True,
text=True,
timeout=30
)
os.unlink(temp_path)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip()
elif result.stderr:
# Parse Hypothesis output for counterexamples
stderr = result.stderr
if "Falsifying example" in stderr:
return json.dumps({
"status": "failed",
"message": "Property violation found",
"details": stderr[-500:]
})
return json.dumps({
"status": "error",
"message": stderr[-500:]
})
else:
return json.dumps({
"status": "error",
"message": "No output from test execution"
})
except subprocess.TimeoutExpired:
os.unlink(temp_path)
return json.dumps({
"status": "error",
"message": "Test execution timed out (30s limit)"
})
except Exception as e:
return json.dumps({"status": "error", "message": str(e)})
# =============================================================================
# Tool Registry
# =============================================================================
available_functions = {
"analyze_function": analyze_function,
"generate_property_test": generate_property_test,
"run_test": run_test,
}
# =============================================================================
# Tool Schemas
# =============================================================================
tools = [
{
"type": "function",
"function": {
"name": "analyze_function",
"description": "Analyze a Python function to understand its signature, purpose, and structure. Use this first to understand what the function does before generating tests.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code to analyze"
}
},
"required": ["function_code"]
}
}
},
{
"type": "function",
"function": {
"name": "generate_property_test",
"description": "Generate a Hypothesis property-based test for a function. The test should use @given decorators with appropriate strategies. Return both the function code and the test code.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code being tested"
},
"test_code": {
"type": "string",
"description": "The complete Hypothesis test code using @given and strategies"
},
"test_description": {
"type": "string",
"description": "A description of what property this test verifies"
}
},
"required": ["function_code", "test_code", "test_description"]
}
}
},
{
"type": "function",
"function": {
"name": "run_test",
"description": "Execute a property-based test against a function. Returns whether the test passed or failed, including any counterexamples found.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code to test"
},
"test_code": {
"type": "string",
"description": "The Hypothesis test code to execute"
}
},
"required": ["function_code", "test_code"]
}
}
}
]
# =============================================================================
# System Prompt
# =============================================================================
SYSTEM_PROMPT = """You are VerifyBot, an AI agent that verifies Python functions using property-based testing.
Your workflow:
1. ANALYZE: First use analyze_function to understand what the function does
2. REASON: Think about what mathematical/logical properties the function should satisfy
3. GENERATE: Use generate_property_test to create Hypothesis tests for those properties
4. TEST: Use run_test to execute each test
5. REPORT: Summarize findings - did the function pass all property tests, or were bugs found?
Properties to consider:
- Idempotency: f(f(x)) == f(x) where applicable
- Invariants: what should always be true about the output?
- Round-trip: if there's an inverse, does f_inv(f(x)) == x?
- Edge cases: empty inputs, zero, negative numbers
- Ordering: if the function sorts, is the output always sorted?
- Size: does the output have expected length/size relative to input?
Generate 2-3 property tests per function. Be thorough but focused.
When reporting results, clearly state which properties were verified and any violations found."""
# =============================================================================
# Agentic Loop
# =============================================================================
def run_agent(function_code: str) -> None:
"""Run the verification agent on a given function."""
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{
"role": "user",
"content": f"Please verify the following Python function using property-based testing:\n\n```python\n{function_code}\n```"
}
]
max_iterations = 10
iteration = 0
print("\n" + "=" * 60)
print("VERIFYBOT - Code Verification Agent")
print("=" * 60)
print(f"\nVerifying function:\n{function_code}\n")
print("-" * 60)
while iteration < max_iterations:
iteration += 1
response = client.chat.completions.create(
model=MODEL,
messages=messages,
tools=tools,
tool_choice="auto",
temperature=0.2
)
response_message = response.choices[0].message
# If no tool calls, we have the final answer
if not response_message.tool_calls:
print(f"\n{'=' * 60}")
print("VERIFICATION REPORT")
print("=" * 60)
print(response_message.content)
return
# Process tool calls
messages.append(response_message)
for tool_call in response_message.tool_calls:
function_name = tool_call.function.name
function_args = json.loads(tool_call.function.arguments)
print(f"\n[Step {iteration}] ACT: {function_name}")
print(f" Args: {json.dumps(function_args, indent=2)[:200]}...")
# Execute the tool
function_to_call = available_functions[function_name]
function_response = function_to_call(**function_args)
# Parse and display result
try:
result = json.loads(function_response)
status = result.get("status", "completed")
print(f" OBSERVE: {status}")
if status == "failed":
print(f" Details: {result.get('message', '')[:200]}")
except json.JSONDecodeError:
print(f" OBSERVE: {function_response[:100]}")
# Add tool result to conversation
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"name": function_name,
"content": function_response
})
print("\nMax iterations reached. Agent stopped.")Your agent now has everything it needs to reason autonomously. Next up, you will build the CLI interface and watch VerifyBot verify real Python functions.
Build the CLI and Test VerifyBot
Your agent has tools, schemas, and an orchestration loop. Everything is wired up, but there's no way to actually trigger it.
In this step, you'll add a CLI interface so you can feed Python functions into VerifyBot from your terminal. Then you'll watch the agent autonomously verify a correct function and catch a bug in a broken one.
In this step, get ready to:
- Create a CLI that reads function code from a file or stdin.
- Test with a correct sort function and observe the agent verify it.
- Test with a buggy function and observe the agent find a counterexample.
Add the CLI interface
A CLI (command-line interface) is the entry point that lets you run your agent from the terminal. Your CLI needs to handle three scenarios: reading a function from a file path, reading from interactive input, or running a built-in demo.
- In agent.py, scroll to the very bottom of the file (after the run_agent function).
- Add the CLI interface by pasting this code at the bottom:
# =============================================================================
# CLI Interface
# =============================================================================
if __name__ == "__main__":
print("\nVerifyBot - AI Code Verification Agent")
print("Enter a Python function to verify (end with an empty line):\n")
if len(sys.argv) > 1:
# Read from file
with open(sys.argv[1], 'r') as f:
function_code = f.read()
else:
# Read from stdin
lines = []
while True:
try:
line = input()
if line == "" and lines and lines[-1] == "":
lines.pop()
break
lines.append(line)
except EOFError:
break
function_code = "\n".join(lines)
if function_code.strip():
run_agent(function_code.strip())
else:
# Demo mode with example functions
print("No input provided. Running demo...\n")
# Example: A correct sort function
correct_sort = '''def my_sort(lst):
"""Sort a list of integers in ascending order."""
return sorted(lst)'''
# Example: A buggy absolute value
buggy_abs = '''def my_abs(x):
"""Return the absolute value of x."""
if x < 0:
return x # Bug: should be -x
return x'''
print("=" * 60)
print("TEST 1: Correct sort function")
print("=" * 60)
run_agent(correct_sort)
print("\n\n")
print("=" * 60)
print("TEST 2: Buggy absolute value function")
print("=" * 60)
run_agent(buggy_abs)What does this code do?
The if __name__ == "__main__": guard ensures this code only runs when you execute the file directly (not when importing it as a module).
- sys.argv[1] checks if you passed a file path as an argument (e.g. python agent.py my_function.py).
- The else branch reads multi-line input from the terminal. Pressing Enter twice (empty line after empty line) signals you're done typing.
- If no input is provided at all, demo mode kicks in. It tests two built-in example functions so you can see the agent work immediately.
- Save agent.py.
Seeing a syntax error when you save?
- Make sure the triple-quoted strings (the ''' blocks for correct_sort and buggy_abs) are properly closed. Each one needs opening and closing ''' marks.
- Check that the indentation inside the if __name__ block uses 4 spaces consistently. Mixing tabs and spaces causes Python to reject the file.
Help me fix a syntax error in my agent.py CLI interface code.
Run the agent on a correct function
Time to see VerifyBot in action. Running the script without any arguments triggers demo mode, which tests a correct sort function first.
- Run the agent in demo mode by executing this command in your terminal:
python agent.pyThe agent starts by testing the my_sort function. Watch the terminal output carefully. You'll see the full THINK > ACT > OBSERVE loop in action.
What should I see for the correct function?
For the my_sort function, the agent will:
- Call analyze_function to understand the function's signature and purpose.
- Call generate_property_test to create tests for properties like "output is sorted" and "output length equals input length."
- Call run_test for each test. You should see OBSERVE: passed for all of them.
- Print a VERIFICATION REPORT confirming all properties held.
What is property-based testing?
Instead of testing one specific input (like "does sort([3,1,2]) equal [1,2,3]?"), property-based testing checks rules that should hold for ALL inputs. Hypothesis generates hundreds of random inputs to try to break those rules.
For a sort function, "the output list is always in ascending order" is a property. If Hypothesis can find ANY list where that fails, it reports the counterexample.
Observe the agent catch a bug
After the sort test completes, the agent automatically moves to the second demo function: a buggy absolute value implementation. This is where things get interesting.
What's wrong with the buggy function?
The my_abs function has a subtle bug: when x < 0, it returns x (the negative number itself) instead of -x (which would flip it to positive). The agent should detect this by testing the property "absolute value is always non-negative."
- Look at the output from TEST 2 in your terminal.
- You should see OBSERVE: failed for at least one test, along with details about the property violation.
- The final VERIFICATION REPORT will describe which property was violated and a counterexample (any negative number).
Why is this powerful?
You didn't write any tests yourself. The agent reasoned about what properties absolute value should satisfy, generated Hypothesis tests to check them, executed those tests, and reported the exact failure. This is the ReAct pattern at work: the LLM thinks, acts via tools, observes results, and repeats until it reaches a conclusion.
Agent seems stuck or times out?
- If you see "Max iterations reached," the agent hit the 10-iteration cap. This can happen if the model generates tests that error out repeatedly. Run the command again. LLM outputs are non-deterministic, so a second run often succeeds.
- If you see a rate limit error, wait 60 seconds and try again. The free tier allows 30 requests per minute.
- If you see an authentication error, double-check that your .env file contains a valid GROQ_API_KEY value.
Help me debug my VerifyBot agent run. It's not completing successfully.
✔️ Awesome, I've got everything!
Great. Double check you've saved agent.py and that you saw both the passing verification and the bug detection in your terminal output.
ⓧ I'd like to double check the full code
Here's the complete agent.py file as it should look now:
import json
import os
import subprocess
import sys
import tempfile
import textwrap
from dotenv import load_dotenv
from groq import Groq
load_dotenv()
client = Groq()
MODEL = "llama-3.3-70b-versatile"
# =============================================================================
# Tool Implementations
# =============================================================================
def analyze_function(function_code: str) -> str:
"""Analyze a Python function and return its signature, docstring, and structure."""
try:
# Extract basic info by parsing
lines = function_code.strip().split("\n")
signature = lines[0] if lines else "Unknown"
# Check for docstring
docstring = ""
if len(lines) > 1:
for line in lines[1:]:
stripped = line.strip()
if stripped.startswith('"""') or stripped.startswith("'''"):
docstring = stripped.strip("\"'")
break
elif stripped.startswith('#'):
docstring = stripped.lstrip('# ')
break
return json.dumps({
"signature": signature,
"docstring": docstring,
"line_count": len(lines),
"code": function_code
})
except Exception as e:
return json.dumps({"error": str(e)})
def generate_property_test(
function_code: str,
test_code: str,
test_description: str
) -> str:
"""Store a generated property-based test for later execution."""
return json.dumps({
"test_description": test_description,
"test_code": test_code,
"function_code": function_code,
"status": "generated"
})
def run_test(function_code: str, test_code: str) -> str:
"""Execute a Hypothesis property-based test and return the results."""
full_code = textwrap.dedent(f"""
import json
import sys
from hypothesis import given, strategies as st, settings
{function_code}
{test_code}
if __name__ == "__main__":
try:
# Run the test function
test_func = [v for k, v in globals().items() if k.startswith("test_")][0]
test_func()
print(json.dumps({{"status": "passed", "message": "All property tests passed!"}}))
except AssertionError as e:
print(json.dumps({{"status": "failed", "message": str(e), "type": "assertion_error"}}))
except Exception as e:
print(json.dumps({{"status": "error", "message": str(e), "type": type(e).__name__}}))
""")
try:
with tempfile.NamedTemporaryFile(
mode='w', suffix='.py', delete=False
) as f:
f.write(full_code)
temp_path = f.name
result = subprocess.run(
[sys.executable, temp_path],
capture_output=True,
text=True,
timeout=30
)
os.unlink(temp_path)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip()
elif result.stderr:
# Parse Hypothesis output for counterexamples
stderr = result.stderr
if "Falsifying example" in stderr:
return json.dumps({
"status": "failed",
"message": "Property violation found",
"details": stderr[-500:]
})
return json.dumps({
"status": "error",
"message": stderr[-500:]
})
else:
return json.dumps({
"status": "error",
"message": "No output from test execution"
})
except subprocess.TimeoutExpired:
os.unlink(temp_path)
return json.dumps({
"status": "error",
"message": "Test execution timed out (30s limit)"
})
except Exception as e:
return json.dumps({"status": "error", "message": str(e)})
# =============================================================================
# Tool Registry
# =============================================================================
available_functions = {
"analyze_function": analyze_function,
"generate_property_test": generate_property_test,
"run_test": run_test,
}
# =============================================================================
# Tool Schemas
# =============================================================================
tools = [
{
"type": "function",
"function": {
"name": "analyze_function",
"description": "Analyze a Python function to understand its signature, purpose, and structure. Use this first to understand what the function does before generating tests.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code to analyze"
}
},
"required": ["function_code"]
}
}
},
{
"type": "function",
"function": {
"name": "generate_property_test",
"description": "Generate a Hypothesis property-based test for a function. The test should use @given decorators with appropriate strategies. Return both the function code and the test code.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code being tested"
},
"test_code": {
"type": "string",
"description": "The complete Hypothesis test code using @given and strategies"
},
"test_description": {
"type": "string",
"description": "A description of what property this test verifies"
}
},
"required": ["function_code", "test_code", "test_description"]
}
}
},
{
"type": "function",
"function": {
"name": "run_test",
"description": "Execute a property-based test against a function. Returns whether the test passed or failed, including any counterexamples found.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code to test"
},
"test_code": {
"type": "string",
"description": "The Hypothesis test code to execute"
}
},
"required": ["function_code", "test_code"]
}
}
}
]
# =============================================================================
# System Prompt
# =============================================================================
SYSTEM_PROMPT = """You are VerifyBot, an AI agent that verifies Python functions using property-based testing.
Your workflow:
1. ANALYZE: First use analyze_function to understand what the function does
2. REASON: Think about what mathematical/logical properties the function should satisfy
3. GENERATE: Use generate_property_test to create Hypothesis tests for those properties
4. TEST: Use run_test to execute each test
5. REPORT: Summarize findings - did the function pass all property tests, or were bugs found?
Properties to consider:
- Idempotency: f(f(x)) == f(x) where applicable
- Invariants: what should always be true about the output?
- Round-trip: if there's an inverse, does f_inv(f(x)) == x?
- Edge cases: empty inputs, zero, negative numbers
- Ordering: if the function sorts, is the output always sorted?
- Size: does the output have expected length/size relative to input?
Generate 2-3 property tests per function. Be thorough but focused.
When reporting results, clearly state which properties were verified and any violations found."""
# =============================================================================
# Agentic Loop
# =============================================================================
def run_agent(function_code: str) -> None:
"""Run the verification agent on a given function."""
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{
"role": "user",
"content": f"Please verify the following Python function using property-based testing:\n\n```python\n{function_code}\n```"
}
]
max_iterations = 10
iteration = 0
print("\n" + "=" * 60)
print("VERIFYBOT - Code Verification Agent")
print("=" * 60)
print(f"\nVerifying function:\n{function_code}\n")
print("-" * 60)
while iteration < max_iterations:
iteration += 1
response = client.chat.completions.create(
model=MODEL,
messages=messages,
tools=tools,
tool_choice="auto",
temperature=0.2
)
response_message = response.choices[0].message
# If no tool calls, we have the final answer
if not response_message.tool_calls:
print(f"\n{'=' * 60}")
print("VERIFICATION REPORT")
print("=" * 60)
print(response_message.content)
return
# Process tool calls
messages.append(response_message)
for tool_call in response_message.tool_calls:
function_name = tool_call.function.name
function_args = json.loads(tool_call.function.arguments)
print(f"\n[Step {iteration}] ACT: {function_name}")
print(f" Args: {json.dumps(function_args, indent=2)[:200]}...")
# Execute the tool
function_to_call = available_functions[function_name]
function_response = function_to_call(**function_args)
# Parse and display result
try:
result = json.loads(function_response)
status = result.get("status", "completed")
print(f" OBSERVE: {status}")
if status == "failed":
print(f" Details: {result.get('message', '')[:200]}")
except json.JSONDecodeError:
print(f" OBSERVE: {function_response[:100]}")
# Add tool result to conversation
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"name": function_name,
"content": function_response
})
print("\nMax iterations reached. Agent stopped.")
# =============================================================================
# CLI Interface
# =============================================================================
if __name__ == "__main__":
print("\nVerifyBot - AI Code Verification Agent")
print("Enter a Python function to verify (end with an empty line):\n")
if len(sys.argv) > 1:
# Read from file
with open(sys.argv[1], 'r') as f:
function_code = f.read()
else:
# Read from stdin
lines = []
while True:
try:
line = input()
if line == "" and lines and lines[-1] == "":
lines.pop()
break
lines.append(line)
except EOFError:
break
function_code = "\n".join(lines)
if function_code.strip():
run_agent(function_code.strip())
else:
# Demo mode with example functions
print("No input provided. Running demo...\n")
# Example: A correct sort function
correct_sort = '''def my_sort(lst):
"""Sort a list of integers in ascending order."""
return sorted(lst)'''
# Example: A buggy absolute value
buggy_abs = '''def my_abs(x):
"""Return the absolute value of x."""
if x < 0:
return x # Bug: should be -x
return x'''
print("=" * 60)
print("TEST 1: Correct sort function")
print("=" * 60)
run_agent(correct_sort)
print("\n\n")
print("=" * 60)
print("TEST 2: Buggy absolute value function")
print("=" * 60)
run_agent(buggy_abs)Your agent is fully functional. It can autonomously verify Python code, catch bugs, and explain its findings. Next up, you'll enhance the agent to generate multiple property tests and show a clean trace of its reasoning process.
Add Multi-Step Reasoning and Iteration
In the previous step, you built the CLI and tested the agent with real functions. You saw it analyze, generate tests, run them, and report back.
Now it's time to understand the multi-step reasoning that makes your agent powerful. The key pieces are already in your code: the SYSTEM_PROMPT instructs the model to generate 2-3 property tests, and the while loop allows up to 10 iterations of tool calling. In this step, you will trace through how these features produce the THINK > ACT > OBSERVE pattern that defines real AI agent systems.
In this step, get ready to:
- Understand how the system prompt drives multi-test generation.
- Trace the iteration logic that lets the agent refine tests based on results.
- Run the agent and observe the full THINK > ACT > OBSERVE trace.
Understand how the system prompt drives multiple property tests
A single test can miss bugs that only appear under specific conditions. Generating multiple property tests from different angles gives broader coverage.
- In VS Code, open agent.py and scroll to the SYSTEM_PROMPT section.
- Find this line near the end of the prompt:
Generate 2-3 property tests per function. Be thorough but focused.Why 2-3 tests instead of 1?
Each property test checks a different mathematical or logical invariant. For a sort function, one test might verify the output is ordered. Another might verify the output has the same length as the input. A single test would only catch one category of bug.
The instruction "Be thorough but focused" keeps the LLM from generating dozens of redundant tests that burn through API calls without adding coverage.
- Look at the list of properties the prompt tells the model to consider:
Properties to consider:
- Idempotency: f(f(x)) == f(x) where applicable
- Invariants: what should always be true about the output?
- Round-trip: if there's an inverse, does f_inv(f(x)) == x?
- Edge cases: empty inputs, zero, negative numbers
- Ordering: if the function sorts, is the output always sorted?
- Size: does the output have expected length/size relative to input?What does this properties list do?
This gives the LLM a menu of verification strategies to choose from. Instead of guessing what to test, the model picks properties that are relevant to the specific function it received. A sort function triggers "Ordering" and "Size" checks. An absolute value function triggers "Invariants" and "Edge cases."
Trace the iteration logic that enables test refinement
A single API call would only let the model do one thing. The power of an agent comes from looping: call the model, execute its tool requests, feed results back, and repeat until the model is done reasoning.
- Scroll down to the run_agent function and find this section:
max_iterations = 10
iteration = 0
print("\n" + "=" * 60)
print("VERIFYBOT - Code Verification Agent")
print("=" * 60)
print(f"\nVerifying function:\n{function_code}\n")
print("-" * 60)
while iteration < max_iterations:
iteration += 1What does the iteration loop do?
The while iteration < max_iterations loop is the beating heart of the ReAct loop. Each iteration is one round of THINK > ACT > OBSERVE. The model might use 3 iterations to analyze, generate, and run one test. Then 2 more for a second test. Then 1 final iteration to deliver the report.
The max_iterations = 10 safeguard prevents runaway loops where the model keeps calling tools indefinitely. This is essential for any production agent system.
- Now find the termination condition inside the loop:
response_message = response.choices[0].message
# If no tool calls, we have the final answer
if not response_message.tool_calls:
print(f"\n{'=' * 60}")
print("VERIFICATION REPORT")
print("=" * 60)
print(response_message.content)
returnHow does the agent know when to stop?
The loop continues as long as the model returns tool_calls in its response. When the model decides it has enough information to deliver a final verdict, it returns a plain text response with no tool calls. That triggers the VERIFICATION REPORT output and the function returns.
This is what makes it an agent, not just a script. The model itself decides how many steps to take.
Run the agent and observe the THINK > ACT > OBSERVE trace
The formatted output in your loop prints each tool call as a step with ACT (what the agent does) and OBSERVE (what it learns from the result). This makes the agent's reasoning visible.
- Find the formatted output section inside the tool-call processing loop:
for tool_call in response_message.tool_calls:
function_name = tool_call.function.name
function_args = json.loads(tool_call.function.arguments)
print(f"\n[Step {iteration}] ACT: {function_name}")
print(f" Args: {json.dumps(function_args, indent=2)[:200]}...")
# Execute the tool
function_to_call = available_functions[function_name]
function_response = function_to_call(**function_args)
# Parse and display result
try:
result = json.loads(function_response)
status = result.get("status", "completed")
print(f" OBSERVE: {status}")
if status == "failed":
print(f" Details: {result.get('message', '')[:200]}")
except json.JSONDecodeError:
print(f" OBSERVE: {function_response[:100]}")
# Add tool result to conversation
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"name": function_name,
"content": function_response
})What does this code do?
For each tool call the model requests, this block prints the step number, the tool name (ACT), and a preview of the arguments. After executing the tool, it prints OBSERVE with the result status.
The [:200] slicing keeps the terminal output readable. Without it, full function code and test code would flood the screen.
- Run the agent in demo mode to see the full multi-step trace:
python agent.pyWatch the terminal output. You should see multiple numbered steps as the agent works through both test functions.
What should I see?
For each function, you should see something like this pattern in your terminal:
- [Step 1] ACT: analyze_function followed by OBSERVE showing "completed".
- [Step 2] ACT: generate_property_test with OBSERVE showing "generated".
- More generate and run steps as the agent creates and executes 2-3 property tests.
- A final VERIFICATION REPORT summarizing which properties passed or failed.
For the buggy my_abs function, you should see at least one OBSERVE with status "failed" and details about the property violation.
Agent stops after just one or two steps?
If the agent seems to skip steps or produce a short report, it might be hitting rate limits on the free tier. Wait 60 seconds and try again. The free tier allows 30 requests per minute.
If you see "Max iterations reached. Agent stopped." that means the agent used all 10 loop iterations without finishing. This is rare but can happen if the model gets stuck in a generate-test-fail-regenerate cycle.
help me debug why my VerifyBot agent isn't completing all its reasoning steps
✔️ Awesome, I've got everything!
Great. Double check you have saved agent.py and your agent is producing the full multi-step trace.
ⓧ I'd like to double check the full code
Here is the complete agent.py file. Compare it against yours to spot any differences.
import json
import os
import subprocess
import sys
import tempfile
import textwrap
from dotenv import load_dotenv
from groq import Groq
load_dotenv()
client = Groq()
MODEL = "llama-3.3-70b-versatile"
# =============================================================================
# Tool Implementations
# =============================================================================
def analyze_function(function_code: str) -> str:
"""Analyze a Python function and return its signature, docstring, and structure."""
try:
# Extract basic info by parsing
lines = function_code.strip().split("\n")
signature = lines[0] if lines else "Unknown"
# Check for docstring
docstring = ""
if len(lines) > 1:
for line in lines[1:]:
stripped = line.strip()
if stripped.startswith('"""') or stripped.startswith("'''"):
docstring = stripped.strip("\"'")
break
elif stripped.startswith('#'):
docstring = stripped.lstrip('# ')
break
return json.dumps({
"signature": signature,
"docstring": docstring,
"line_count": len(lines),
"code": function_code
})
except Exception as e:
return json.dumps({"error": str(e)})
def generate_property_test(
function_code: str,
test_code: str,
test_description: str
) -> str:
"""Store a generated property-based test for later execution."""
return json.dumps({
"test_description": test_description,
"test_code": test_code,
"function_code": function_code,
"status": "generated"
})
def run_test(function_code: str, test_code: str) -> str:
"""Execute a Hypothesis property-based test and return the results."""
full_code = textwrap.dedent(f"""
import json
import sys
from hypothesis import given, strategies as st, settings
{function_code}
{test_code}
if __name__ == "__main__":
try:
# Run the test function
test_func = [v for k, v in globals().items() if k.startswith("test_")][0]
test_func()
print(json.dumps({{"status": "passed", "message": "All property tests passed!"}}))
except AssertionError as e:
print(json.dumps({{"status": "failed", "message": str(e), "type": "assertion_error"}}))
except Exception as e:
print(json.dumps({{"status": "error", "message": str(e), "type": type(e).__name__}}))
""")
try:
with tempfile.NamedTemporaryFile(
mode='w', suffix='.py', delete=False
) as f:
f.write(full_code)
temp_path = f.name
result = subprocess.run(
[sys.executable, temp_path],
capture_output=True,
text=True,
timeout=30
)
os.unlink(temp_path)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip()
elif result.stderr:
# Parse Hypothesis output for counterexamples
stderr = result.stderr
if "Falsifying example" in stderr:
return json.dumps({
"status": "failed",
"message": "Property violation found",
"details": stderr[-500:]
})
return json.dumps({
"status": "error",
"message": stderr[-500:]
})
else:
return json.dumps({
"status": "error",
"message": "No output from test execution"
})
except subprocess.TimeoutExpired:
os.unlink(temp_path)
return json.dumps({
"status": "error",
"message": "Test execution timed out (30s limit)"
})
except Exception as e:
return json.dumps({"status": "error", "message": str(e)})
# =============================================================================
# Tool Registry
# =============================================================================
available_functions = {
"analyze_function": analyze_function,
"generate_property_test": generate_property_test,
"run_test": run_test,
}
# =============================================================================
# Tool Schemas
# =============================================================================
tools = [
{
"type": "function",
"function": {
"name": "analyze_function",
"description": "Analyze a Python function to understand its signature, purpose, and structure. Use this first to understand what the function does before generating tests.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code to analyze"
}
},
"required": ["function_code"]
}
}
},
{
"type": "function",
"function": {
"name": "generate_property_test",
"description": "Generate a Hypothesis property-based test for a function. The test should use @given decorators with appropriate strategies. Return both the function code and the test code.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code being tested"
},
"test_code": {
"type": "string",
"description": "The complete Hypothesis test code using @given and strategies"
},
"test_description": {
"type": "string",
"description": "A description of what property this test verifies"
}
},
"required": ["function_code", "test_code", "test_description"]
}
}
},
{
"type": "function",
"function": {
"name": "run_test",
"description": "Execute a property-based test against a function. Returns whether the test passed or failed, including any counterexamples found.",
"parameters": {
"type": "object",
"properties": {
"function_code": {
"type": "string",
"description": "The complete Python function code to test"
},
"test_code": {
"type": "string",
"description": "The Hypothesis test code to execute"
}
},
"required": ["function_code", "test_code"]
}
}
}
]
# =============================================================================
# System Prompt
# =============================================================================
SYSTEM_PROMPT = """You are VerifyBot, an AI agent that verifies Python functions using property-based testing.
Your workflow:
1. ANALYZE: First use analyze_function to understand what the function does
2. REASON: Think about what mathematical/logical properties the function should satisfy
3. GENERATE: Use generate_property_test to create Hypothesis tests for those properties
4. TEST: Use run_test to execute each test
5. REPORT: Summarize findings - did the function pass all property tests, or were bugs found?
Properties to consider:
- Idempotency: f(f(x)) == f(x) where applicable
- Invariants: what should always be true about the output?
- Round-trip: if there's an inverse, does f_inv(f(x)) == x?
- Edge cases: empty inputs, zero, negative numbers
- Ordering: if the function sorts, is the output always sorted?
- Size: does the output have expected length/size relative to input?
Generate 2-3 property tests per function. Be thorough but focused.
When reporting results, clearly state which properties were verified and any violations found."""
# =============================================================================
# Agentic Loop
# =============================================================================
def run_agent(function_code: str) -> None:
"""Run the verification agent on a given function."""
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{
"role": "user",
"content": f"Please verify the following Python function using property-based testing:\n\n```python\n{function_code}\n```"
}
]
max_iterations = 10
iteration = 0
print("\n" + "=" * 60)
print("VERIFYBOT - Code Verification Agent")
print("=" * 60)
print(f"\nVerifying function:\n{function_code}\n")
print("-" * 60)
while iteration < max_iterations:
iteration += 1
response = client.chat.completions.create(
model=MODEL,
messages=messages,
tools=tools,
tool_choice="auto",
temperature=0.2
)
response_message = response.choices[0].message
# If no tool calls, we have the final answer
if not response_message.tool_calls:
print(f"\n{'=' * 60}")
print("VERIFICATION REPORT")
print("=" * 60)
print(response_message.content)
return
# Process tool calls
messages.append(response_message)
for tool_call in response_message.tool_calls:
function_name = tool_call.function.name
function_args = json.loads(tool_call.function.arguments)
print(f"\n[Step {iteration}] ACT: {function_name}")
print(f" Args: {json.dumps(function_args, indent=2)[:200]}...")
# Execute the tool
function_to_call = available_functions[function_name]
function_response = function_to_call(**function_args)
# Parse and display result
try:
result = json.loads(function_response)
status = result.get("status", "completed")
print(f" OBSERVE: {status}")
if status == "failed":
print(f" Details: {result.get('message', '')[:200]}")
except json.JSONDecodeError:
print(f" OBSERVE: {function_response[:100]}")
# Add tool result to conversation
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"name": function_name,
"content": function_response
})
print("\nMax iterations reached. Agent stopped.")
# =============================================================================
# CLI Interface
# =============================================================================
if __name__ == "__main__":
print("\nVerifyBot - AI Code Verification Agent")
print("Enter a Python function to verify (end with an empty line):\n")
if len(sys.argv) > 1:
# Read from file
with open(sys.argv[1], 'r') as f:
function_code = f.read()
else:
# Read from stdin
lines = []
while True:
try:
line = input()
if line == "" and lines and lines[-1] == "":
lines.pop()
break
lines.append(line)
except EOFError:
break
function_code = "\n".join(lines)
if function_code.strip():
run_agent(function_code.strip())
else:
# Demo mode with example functions
print("No input provided. Running demo...\n")
# Example: A correct sort function
correct_sort = '''def my_sort(lst):
"""Sort a list of integers in ascending order."""
return sorted(lst)'''
# Example: A buggy absolute value
buggy_abs = '''def my_abs(x):
"""Return the absolute value of x."""
if x < 0:
return x # Bug: should be -x
return x'''
print("=" * 60)
print("TEST 1: Correct sort function")
print("=" * 60)
run_agent(correct_sort)
print("\n\n")
print("=" * 60)
print("TEST 2: Buggy absolute value function")
print("=" * 60)
run_agent(buggy_abs)Your agent is now complete. You have a working AI system that reasons through multiple steps, generates multiple tests, and delivers a detailed verification report. Next up, you will tackle the Secret Mission and give your agent the ability to fix the bugs it finds.
Secret mission
Fix Mode
Your agent can find bugs, but what if it could also fix them? In this extension, you'll add a propose_fix tool that lets VerifyBot automatically generate corrected code and verify its own fixes using the same property-based testing pipeline.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing costs. Your Groq API usage stays within the free tier as long as you are not running the agent continuously.
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Local project files (agent.py and .env).
- Installed pip packages (groq, hypothesis, python-dotenv).
- Groq API key stored in your .env file.
Keep everything running
No action needed. Choose this if you are still actively building or want to keep experimenting with VerifyBot.
- Your agent.py and .env files stay on your machine.
- Your installed pip packages remain available for future Python projects.
- Your Groq API key remains active. The free tier renews daily, so there is no charge for keeping it.
Pause - I'll come back to this later
Nothing to pause for this project. Since everything runs locally and Groq's free tier has no ongoing charges, your resources are effectively paused whenever you are not running the script.
- Close VS Code to free up memory.
- Your files and packages remain intact for when you return.
Delete - I don't want to use this again
Remove all project resources and start fresh.
- Delete your project files by running this command from the folder that contains your project directory:
macOS
rm -rf ~/Desktop/verifybotWindows
rmdir /s /q %USERPROFILE%\Desktop\verifybot- Uninstall the pip packages by running:
pip uninstall groq hypothesis python-dotenv -y- Revoke your Groq API key by visiting the Groq Console API Keys page. Find the key you created for this project and click the delete icon next to it.
Nice Work!
Nice Work!
You just built a code verification agent from scratch. VerifyBot can autonomously reason about Python functions, generate property-based tests, find bugs, and even fix them.
You've learned how to:
- Built a multi-tool AI agent using the Groq API with local tool calling.
- Implemented the full ReAct pattern (THINK > ACT > OBSERVE) as a manual orchestration loop.
- Used property-based testing with Hypothesis to verify code correctness.
- Added autonomous bug-fixing capabilities to your agent with Fix Mode.
Ready to quiz yourself?