Evaluate an AI Agent with ThinkingBox

Run a database-backed benchmark to verify an AI agent completed a task.

Introduction

30 Second Summary

A digital assistant can confidently say it placed an order or updated a record. The real question is whether the work actually exists after the conversation ends.

In this project, you will run Microsoft's ThinkingBox benchmark on your laptop with one retail task. You will compare the AI agent's claim with database evidence across repeated runs.

What You'll Build

You'll watch a retail task unfold before seeing a pass or fail verdict based on the database state left behind.

By the end of this project, you'll have:

  • A local benchmark you can start on your laptop. You can confirm its Typesense database is running inside Docker.
  • A single model endpoint you can test before it serves all three benchmark roles. The endpoint serves the agent role. It also serves the simulated-user role. It also serves the judge role.
  • An evaluation report that lets you compare the agent's claim with the database evidence behind its verdict. A repeated run turns those verdicts into a consistency score.
  • Secret Mission: An optional challenge to push your skills further.

Before We Start

The published ThinkingBox setup targets Linux or WSL. Its local environment depends on Docker for containers.

You will install uv for Python dependency management. You will then prepare the ThinkingBox environment from OpenEnv.

In this step, get ready to:
  • Prepare a Linux or WSL system with Docker.
  • Install uv for dependency management.
  • Prepare the local ThinkingBox environment from OpenEnv.
Verify Docker and uv

Docker provides the container runtime that ThinkingBox needs for its database services. The version check also confirms that your terminal can reach the Docker engine.

  • Check your Docker installation by running this command:
docker version

What does this command check?

The command asks the Docker CLI for version details. A working setup shows both a Client section and a Server section.

✔️ I see Client and Server details

That is one dependency ready. Your terminal can communicate with the running Docker engine.

  • Continue to the uv check below.

ⓧ I need Docker on Linux

  • Open the official Docker Engine installation guide.
  • Choose the guide for your Linux distribution.
  • Complete the installation steps for your distribution.
  • Complete the official Linux post-installation steps so your user can run Docker commands.
  • Return to your Linux terminal.
  • Confirm that the Docker engine is available by running this command:
docker version

What confirms Docker is ready?

The output should include details for the Docker client and server. The server details prove that the engine is running.

Docker is now ready to run the database container used later in the project.

ⓧ I need Docker on Windows with WSL

  • Download Docker Desktop from the official Windows installation guide.
  • Run the downloaded installer.
  • Keep Use WSL 2 based engine selected during setup.
  • Start Docker Desktop from the Windows Start menu.
  • Return to your WSL terminal.
  • Confirm that WSL can reach the Docker engine by running this command:
docker version

What confirms WSL integration?

The output should include details for the Docker client and server. The server section confirms that your WSL terminal can use Docker Desktop.

Your Windows host and WSL environment can now work with the same Docker engine.

Docker engine unavailable?

If you only see client information, make sure the Docker service or Docker Desktop is running. Windows learners should also confirm that WSL integration is enabled for their Linux distribution.

If Linux requires elevated permissions, complete Docker's post-installation steps before trying again.

Help me troubleshoot Docker on Linux or WSL.

uv reads a project's dependency configuration and creates the matching Python environment. This keeps the ThinkingBox packages together in one managed setup.

  • Check whether uv is available by running this command:
uv --version

What does this command check?

The command asks uv to print its installed version. A version number confirms that your shell can find the tool.

✔️ I see a uv version

Good, uv is ready to build the project environment.

  • Continue to the OpenEnv clone below.

ⓧ Command not found

  • Install uv in your Linux or WSL terminal by running this command:
curl -LsSf https://astral.sh/uv/install.sh | sh

What does this installer do?

The command downloads uv's official installation script over HTTPS. The script installs uv for your current user.

  • Load uv into your current shell by running this command:
source "$HOME/.local/bin/env"

Why load this file?

The installer adds uv to a user-level executable directory. Loading this environment file makes the command available in your current terminal session.

  • Confirm that uv is now available by running this command:
uv --version

What should you see?

You should see a uv version number. That output confirms that the installation succeeded.

The dependency manager is now ready for the ThinkingBox environment.

Still cannot run uv?

Open a new Linux or WSL terminal so it reloads your shell configuration. Run the version check again from that new terminal.

Help me fix my uv installation.

Clone the OpenEnv repository

A Git repository holds the OpenEnv source files. Cloning it gives you the packaged ThinkingBox environment on your laptop.

  • Move to your Desktop by running this command:
cd ~/Desktop

What does this command do?

The ~ symbol points to your Linux home directory. This command places the repository inside its Desktop folder.

  • Clone OpenEnv to your Desktop by running this command:
git clone https://github.com/huggingface/OpenEnv

What does this command do?

Git downloads the OpenEnv repository into a new OpenEnv folder. That folder includes the ThinkingBox environment definition.

The clone may take a minute while Git downloads the repository history. You will see download progress before the terminal prompt returns.

Cloned locally with the ThinkingBox environment available. You now have the source files that the remaining setup uses.

Clone did not complete?

If the terminal cannot find Git, install Git through your Linux distribution's package manager. If the download stops, check your internet connection before rerunning the clone command.

If an OpenEnv folder already exists on your Desktop, remove the incomplete folder or choose a clean location before trying again.

Help me troubleshoot the OpenEnv clone.

Sync the ThinkingBox environment

The repository includes a locked dependency plan for ThinkingBox. uv uses that plan to create a reproducible local environment.

  • Enter the cloned OpenEnv folder by running this command:
cd OpenEnv

Why enter this folder?

The next command refers to the envs/thinkingbox_env path inside OpenEnv. Entering the repository makes that path available.

Before you run this check, do you expect uv to finish from the repository's locked dependency plan?

  • Sync the ThinkingBox environment by running this command:
uv sync --project envs/thinkingbox_env --frozen

What does this command do?

  • The --project option points uv at the ThinkingBox environment inside OpenEnv.
  • The --frozen option tells uv to use the repository's existing lock file without changing it.
  • uv downloads the required packages into a managed Python environment.

Dependencies synced successfully with uv. You now have the local environment that every evaluation step builds on.

Dependency sync failed?

Confirm that your terminal is inside the cloned OpenEnv folder. A path mistake prevents uv from finding the ThinkingBox project.

A download failure can also come from a temporary network interruption. Restore the connection before running the same sync command again.

Help me fix the ThinkingBox dependency sync.

✔️ Awesome, I've got everything!

Your local tools are ready. Keep the OpenEnv folder on your Desktop for the remaining steps.

ⓧ I'd like to double check the full code

Cloned locally with the ThinkingBox environment available.

Dependencies synced successfully with uv.

Your local ThinkingBox environment is ready. Next up, you will start its Typesense database container.

Start the Typesense Database

The benchmark needs a running database before it can inspect the records an agent leaves behind. Without that evidence store, ThinkingBox cannot judge whether the requested work actually happened.

Your ThinkingBox dependencies are already synced. In this step, you'll use Docker to start Typesense 30.1 in a detached container.

In this step, get ready to:
  • Launch the Typesense container with the setup guide configuration.
  • Confirm the container appears in Docker's running-container list.
Start Typesense from the setup guide

Typesense needs a local directory for its database files. The setup guide mounts .typesense-data into the container so those files live on your laptop.

  • Use a second terminal session.
  • Run the setup guide sequence below from that terminal session:
mkdir -p .typesense-data
docker run --rm -d --name thinkingbox-typesense \
  -p 8108:8108 \
  -v "$PWD/.typesense-data:/data" \
  typesense/typesense:30.1 \
  --data-dir /data --api-key=Fake --enable-cors
until curl -fsS http://127.0.0.1:8108/health; do sleep 1; done

What does this setup do?

  • The .typesense-data directory stores the database files on your laptop.
  • The -d flag keeps the database running in the background.
  • The -p 8108:8108 setting makes the Typesense service available through port 8108.
  • The final loop waits until the health endpoint responds successfully.
  • Wait for the command to return to the terminal prompt.

The successful health response confirms that Typesense can answer requests. The container is ready for an independent running-state check.

Health check still waiting?

  • Confirm Docker is running if the health check continues waiting.
  • Check whether an earlier container named thinkingbox-typesense is already running before changing the setup.
  • Review the startup output for the first message that mentions the image or container.
  • Help me troubleshoot the Typesense container startup.
Verify the container is running

A detached container keeps running after its startup command finishes. Docker's running-container list gives you direct evidence that the database process is still active.

Before the check, predict whether the Typesense row is present.

  • List the running Docker containers by running this command:
docker ps

What does this check prove?

  • The output includes only containers that are currently running.
  • The name thinkingbox-typesense identifies the benchmark database container.
  • The image typesense/typesense:30.1 confirms that the expected Typesense version is running.
  • The port value 8108 confirms that the database service is exposed locally.

You'll see a row for thinkingbox-typesense using the typesense/typesense:30.1 image. That row proves the database container is running.

Typesense missing from the list?

  • Return to the second terminal session if the Typesense row is missing.
  • Review the container startup output for the first reported problem.
  • Run the setup guide sequence again after resolving that problem.
  • Help me find why the container stopped.

You've got the data layer online. ThinkingBox now has a running Typesense service for the state its judge evaluates.

Your database is running. Next, you'll connect the three benchmark roles to one model endpoint.

Configure the Model Endpoint

Your Typesense container now gives ThinkingBox a live database to inspect. The benchmark still needs a model connection before it can attempt a task.

ThinkingBox separates model work into an agent, a simulated user, and a judge. A shared LLM endpoint keeps your first evaluation simple.

In this step, get ready to:
  • Collect the URL and model name for one OpenAI-compatible endpoint.
  • Configure the agent, simulated user, and judge to share that endpoint.
  • Send one authenticated request and confirm the model responds.
Choose one endpoint for every role

An OpenAI-compatible Chat Completions endpoint gives ThinkingBox one consistent request shape. The same endpoint can serve every role.

  • Find the full Chat Completions URL in your model provider's endpoint details.
  • Record the endpoint URL here: your chat-completions endpoint URL.
  • Record the served model name here: your served model name.
  • Keep the API key for that endpoint ready.

Why use one endpoint?

The agent performs the business task. The simulated user supplies private details when the agent asks for them.

The judge handles checks that need language understanding. Routing all three roles through one endpoint removes extra provider setup from your first run.

Create the ThinkingBox configuration

The OpenEnv repository reads model settings from a YAML file. You will build thinkingbox.yaml in three small sections so each role is easy to check.

  • Switch back to the terminal that is already in your cloned OpenEnv repository.
  • Create thinkingbox.yaml with the proxy and agent configuration by running:
cat > thinkingbox.yaml <<'YAML'
# Connect ThinkingBox to the Session Proxy configured earlier.
mcp_proxy:
  endpoint_url: "http://127.0.0.1:7111"
  timeout: 300.0

orchestrator:
  type: thinkingbox
  agent_model:
    # Send evaluated-agent requests to your chosen model.
    type: aoai
    credential: &shared_credential
      type: api-key
      api_key: "your-api-key-here"
    deployment: "[[MODEL_NAME="your served model name"]]"
    endpoint_url: "[[MODEL_ENDPOINT_URL="your chat-completions endpoint URL"]]"
    is_reasoning: true
    temperature: 1.0
    max_completion_tokens: 4096
    timeout: 600.0
YAML
cat thinkingbox.yaml

What does this configuration do?

  • The mcp_proxy section connects ThinkingBox to the MCP Session Proxy used by its tools.
  • The agent_model section tells the evaluated agent which endpoint and model to call.
  • The shared_credential anchor lets the other roles reuse the same authentication without duplicating your key.
  • The final cat command prints the saved section for inspection.
  • Compare the printed agent_model section with the block above.

You'll see your endpoint URL and model name in the printed configuration. The API key remains a harmless placeholder for now.

Did the terminal keep waiting?

Make sure the closing YAML starts at the far left of its line. Leading spaces prevent the shell from recognizing the end of the file content.

Delete the incomplete thinkingbox.yaml file before running the block again.

Help me fix the unfinished YAML command.

  • Append the judge configuration by running:
cat >> thinkingbox.yaml <<'YAML'

# Use the same endpoint for narrow language judgments.
judge_model:
  type: aoai
  credential: *shared_credential
  deployment: "[[MODEL_NAME="your served model name"]]"
  endpoint_url: "[[MODEL_ENDPOINT_URL="your chat-completions endpoint URL"]]"
  is_reasoning: false
  temperature: 0.0
  seed: 42
  max_completion_tokens: 128
  timeout: 600.0

judge_type: "legacy"
YAML
tail -n 15 thinkingbox.yaml

How is the judge configured?

  • The judge_model section reuses the agent's authentication through *shared_credential.
  • The endpoint URL and deployment match the values assigned to the agent.
  • The lower completion limit fits the judge's focused evaluation questions.
  • The tail command prints the newly appended lines.
  • Confirm the printed section contains judge_model and judge_type.

You'll see the judge pointing to the same endpoint URL and model name as the agent.

Is the judge section missing?

Check that you ran the append command from the OpenEnv repository containing thinkingbox.yaml.

Check the closing YAML line if the terminal is still waiting for input.

Help me repair the judge configuration.

  • Append the simulated-user configuration by running:
cat >> thinkingbox.yaml <<'YAML'

# Let the simulated user answer through the shared endpoint.
user_model:
  type: aoai
  credential: *shared_credential
  deployment: "[[MODEL_NAME="your served model name"]]"
  endpoint_url: "[[MODEL_ENDPOINT_URL="your chat-completions endpoint URL"]]"
  is_reasoning: false
  temperature: 0.0
  max_completion_tokens: 4096
  timeout: 600.0
YAML
tail -n 13 thinkingbox.yaml

How is the simulated user configured?

  • The user_model section supplies the model that answers the agent's follow-up questions.
  • The shared credential keeps authentication consistent across all three roles.
  • The repeated deployment and endpoint values route every role to the same model service.
  • The final tail command prints the completed user section.
  • Confirm the printed section starts with user_model.

You'll see the simulated user pointing to the same endpoint URL and model name as the other roles.

Does the user section look misplaced?

Check that user_model starts at the far left. Extra indentation can place it inside judge_model.

Help me fix the user model indentation.

Protect your model key

Your API key is sensitive. The next edit places it in a local configuration file.

  • Keep thinkingbox.yaml out of version control.
  • Keep the editor off-screen during recordings or screenshots.
  • Open thinkingbox.yaml in Nano by running this command:
nano thinkingbox.yaml

What happens in Nano?

Nano opens the configuration inside your terminal. Its save and exit shortcuts appear along the bottom of the screen.

  • Find your-api-key-here in the agent_model credential section.
  • Replace the placeholder with your model API key.
  • Press Ctrl+O to save the file.
  • Press Enter to confirm the file name.
  • Press Ctrl+X to close Nano.

Nano not available?

Use another local text editor to change the single API key placeholder. Preserve the quotation marks around the key value.

Help me update the credential safely.

✔️ Awesome, I've got everything!

Your agent, simulated user, and judge now share one endpoint URL, one model name, and one authentication credential.

ⓧ I'd like to double check the full code

Compare your thinkingbox.yaml with this reference. The credential stays redacted here, while your local file should contain your actual API key.

# Connect ThinkingBox to the Session Proxy configured earlier.
mcp_proxy:
  endpoint_url: "http://127.0.0.1:7111"
  timeout: 300.0

orchestrator:
  type: thinkingbox
  agent_model:
    # Send evaluated-agent requests to your chosen model.
    type: aoai
    credential: &shared_credential
      type: api-key
      api_key: "your-api-key-here"
    deployment: "[[MODEL_NAME="your served model name"]]"
    endpoint_url: "[[MODEL_ENDPOINT_URL="your chat-completions endpoint URL"]]"
    is_reasoning: true
    temperature: 1.0
    max_completion_tokens: 4096
    timeout: 600.0

# Use the same endpoint for narrow language judgments.
judge_model:
  type: aoai
  credential: *shared_credential
  deployment: "[[MODEL_NAME="your served model name"]]"
  endpoint_url: "[[MODEL_ENDPOINT_URL="your chat-completions endpoint URL"]]"
  is_reasoning: false
  temperature: 0.0
  seed: 42
  max_completion_tokens: 128
  timeout: 600.0

judge_type: "legacy"

# Let the simulated user answer through the shared endpoint.
user_model:
  type: aoai
  credential: *shared_credential
  deployment: "[[MODEL_NAME="your served model name"]]"
  endpoint_url: "[[MODEL_ENDPOINT_URL="your chat-completions endpoint URL"]]"
  is_reasoning: false
  temperature: 0.0
  max_completion_tokens: 4096
  timeout: 600.0
Test the endpoint

A correct configuration still depends on a reachable endpoint and valid authentication. One direct request checks the network path, the model name, and the API key before the evaluation begins.

  • Send one authenticated request to your configured endpoint by running the block below:
  • Paste your API key when the terminal waits after the first line.
  • Press Enter after pasting the key.

Before you send the request, do you expect the endpoint to accept the same model name stored in thinkingbox.yaml?

read -s MODEL_API_KEY
echo
curl -sS "[[MODEL_ENDPOINT_URL="your chat-completions endpoint URL"]]" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${MODEL_API_KEY}" \
  -d '{"model":"[[MODEL_NAME="your served model name"]]","messages":[{"role":"user","content":"Reply with exactly READY."}]}'
unset MODEL_API_KEY

What does this request prove?

  • The read -s command accepts your key without displaying the characters you type.
  • The curl request sends your recorded model name to the recorded Chat Completions URL.
  • The authorization header proves that the endpoint accepts your credential.
  • The unset command removes the temporary key from the current shell.

You'll see a JSON response containing the assistant's reply with READY in its content.

That connection is live for every ThinkingBox model role.

Did the endpoint reject the request?

If the response mentions authentication, confirm that you pasted the active API key for this endpoint. Confirm that your provider accepts bearer authentication.

If the response mentions the model, compare your recorded model name with the provider's served model name. Capitalization and punctuation must match.

If the connection fails, confirm that your URL includes the full Chat Completions path.

Help me diagnose the endpoint response.

Your model connection is configured and tested. Next up, you will define one retail task and run its first ThinkingBox evaluation.

Run One Retail Evaluation

Your three ThinkingBox roles can now reach the same tested model endpoint. Typesense is ready to hold the task state.

Now you can give the agent one retail task. A confident final reply cannot prove that the records changed correctly.

ThinkingBox captures the trajectory. Its judge returns a verdict from the resulting database state.

In this step, get ready to:
  • Create a one-line task list for a retail benchmark case.
  • Run one evaluated attempt.
  • Confirm the trajectory and pass-or-fail verdict.
Create the retail task file

ThinkingBox reads task references from a YAML list. A leading hyphen selects a single benchmark case for this run.

  • Return to the terminal you used for setup in the cloned OpenEnv repository.
  • Create one_task.yaml with the retail task reference by running this command:
echo "- sandbox_external_retail_group1.py:test_case_ST002_001" > one_task.yaml

What does this task file do?

  • The hyphen makes the line a one-item YAML list.
  • The sandbox_external_retail_group1.py:test_case_ST002_001 reference selects one retail benchmark task.
  • The redirect writes that selection into one_task.yaml.
  • Confirm one_task.yaml exists by running this command:
ls one_task.yaml

What does this check prove?

This asks the shell to list that specific file. Seeing its name confirms that the task selector exists in your current repository folder.

You'll see one_task.yaml in the terminal output.

Task file missing?

  • Check that your terminal is still inside the cloned OpenEnv repository.
  • Run the file creation command again if the terminal reports that one_task.yaml is missing.

Help me find my ThinkingBox task file.

✔️ Awesome, I've got everything!

Your one-line retail task selector is saved in the OpenEnv repository.

ⓧ I'd like to double check the full code

Your complete one_task.yaml file should contain this single line.

- sandbox_external_retail_group1.py:test_case_ST002_001

What does this line select?

This one-item list points the evaluator at the chosen retail business task. Only that task runs during this evaluation.

Test the agent's database work

The evaluator starts one complete episode. It records the agent's trajectory before the judge checks the final database state.

This run sends live requests to your configured model endpoint, so its normal usage charges can apply. The --repeat 1 setting limits this step to one attempt.

A complete episode can pause while the three roles exchange messages. The timeout allows each message up to 30 minutes.

Before you run it, do you think the agent's final claim will match the database state the judge checks?

  • Start the single retail evaluation by running this command:
uv run --project envs/thinkingbox_env thinkingbox-eval \
  one_task.yaml \
  --config "$PWD/thinkingbox.yaml" \
  --output results.jsonl \
  --errors-output errors.jsonl \
  --repeat 1 --message-timeout 1800

What does this command do?

  • The --project envs/thinkingbox_env option uses the environment you synced earlier.
  • The one_task.yaml argument selects your retail task.
  • The --config "$PWD/thinkingbox.yaml" option loads the three model-role settings from earlier.
  • The --output results.jsonl option stores the completed evaluation result.
  • The --errors-output errors.jsonl option separates operational failures from model outcomes.
  • The --repeat 1 option runs one attempt.
  • The --message-timeout 1800 option sets the maximum wait for each message.
  • Let the evaluator finish the single attempt.
  • Scroll through the terminal output to locate the agent trajectory.
  • Find the pass-or-fail verdict at the end.

You'll see the agent's messages and tool-use sequence. A pass-or-fail verdict follows the completed trajectory.

That's your first evidence-backed evaluation: the agent's work has been judged against the database state it left behind.

No verdict at the end?

  • Confirm the Typesense container from earlier is still running if the evaluator cannot reach the task database.
  • Check the tested endpoint configuration from earlier if the episode stops during a model turn.
  • Review errors.jsonl for an operational failure that prevented a verdict.

Help me troubleshoot my incomplete evaluation.

Your first retail evaluation is complete. Next, you'll open the report and compare the agent's claim with the database evidence.

Verify the Judge Evidence

Your first ThinkingBox run produced an agent trajectory plus a verdict. Now comes the core question: does the agent's final claim match the state stored in Typesense?

The evaluation report forms an evidence chain from the agent's words to the judge's database-state checks. You will use that chain to decide whether the verdict is supported.

In this step, get ready to:
  • Open the single-run evaluation report.
  • Identify the agent's claimed action plus the judge's database-state evidence.
  • Determine whether the claim matches the evidence.
Open the single-run report

The report is the durable record of the attempt. It keeps the trajectory, verdict, agent claim, and judge evidence together.

What does the report prove?

The trajectory shows what the agent said. It also shows the tools used during the attempt.

The judge's evidence captures the final backend state plus the side effects left after the agent stops. These persistent changes decide whether the work happened.

A tool call proves that the agent attempted an action. The stored result proves whether the action actually happened.

  • Return to the terminal session from the previous step.
  • Scroll to the earlier thinkingbox-eval command in your terminal history.
  • Locate the report path supplied after --output.
  • Open that output file in a text editor.
  • Find the record whose task reference matches the retail task in one_task.yaml.
  • Keep that record visible for the comparison.

Can't find the report?

  • Check the earlier evaluation command for the value after --output.

A relative path starts from the OpenEnv repository folder where you ran the evaluation.

Help me locate my ThinkingBox evaluation report.

Compare the claim with the database state

The final response is the agent's claim about the job. The database-state evidence shows the records left behind after the conversation ended.

  • Find the agent's final response in the report.
  • Identify every completed action that the final response claims.
  • Find the judge section that describes the final database state.
  • Match each claimed action to the corresponding record change.
  • Check the evidence for any missing required effect.
  • Check the evidence for any unintended extra effect.
  • Check whether the task's requested final state appears in the judge evidence.

How do I read a mismatch?

  • A wrong value leaves the target record in an incorrect state.
  • A missing effect means the required change is absent.
  • An extra effect means the agent changed more than the task requested.
Confirm the verdict

One passing attempt is credible only when the claim, database evidence, and verdict align.

Before you make the final comparison, do you expect the agent's claim to survive the judge's database-state check?

  • Read the pass-or-fail verdict again.
  • Compare that verdict with the evidence chain you traced.

You should now see one of two outcomes: the evidence supports the claim, or it exposes a mismatch.

✔️ The claim matches the evidence

The database evidence supports the agent's claimed outcome. A pass verdict is consistent with that evidence.

ⓧ The claim conflicts with the evidence

The database evidence shows a wrong value, a missing effect, or an extra effect. A fail verdict is consistent with that mismatch.

That is the key check complete: you can now explain the verdict using persistent database evidence.

Your single run now has an evidence-based interpretation. Next, you'll repeat the same retail task to measure how consistently the model reaches the correct state.

Measure Run Consistency

Your first ThinkingBox run showed whether the agent's claim matched the database evidence once. A single verdict cannot show whether that result is dependable.

Now you will repeat the exact retail task five times. The pass total becomes the model's consistency score.

In this step, get ready to:
  • Run the same retail task five times.
  • Read the number of passing runs out of the total.
Rerun the retail task five times

Repeated trials expose outcomes that a single run can hide. Keeping one_task.yaml unchanged isolates consistency because the task stays fixed.

Five attempts create more model traffic than one. The evaluation may stay active longer while each attempt reaches a verdict.

  • Predict whether all five attempts will share the same verdict.
  • Generate a five-run report by running this command:
uv run --project envs/thinkingbox_env thinkingbox-eval \
  one_task.yaml \
  --config "$PWD/thinkingbox.yaml" \
  --output repeated_results.jsonl \
  --errors-output repeated_errors.jsonl \
  --repeat 5 --message-timeout 1800

What Does This Command Do?

  • The one_task.yaml argument keeps the retail task fixed across every attempt.
  • The --repeat 5 option tells the evaluator to execute five trials.
  • The --output repeated_results.jsonl option stores the trial records in a separate JSONL report.
  • The --errors-output repeated_errors.jsonl option stores operational failures separately.
  • Wait for the evaluator to finish all five attempts.
  • Confirm the terminal reaches an outcome for each attempt.

You'll see five attempt outcomes in the terminal. Their verdicts may match or differ.

Missing One or More Outcomes?

  • Check repeated_errors.jsonl for an operational failure linked to the missing attempt.
  • Confirm the Typesense container from earlier is still running.
  • Resolve any endpoint or container failure before generating a fresh repeated report.

Help me diagnose an incomplete ThinkingBox repeat run.

Read the consistency score

A consistency score is the number of passing attempts out of the total attempts. This score measures repeated success on this exact retail task.

  • Predict whether the repeated score will support the single-run verdict.
  • Return to the evaluation report view from the previous step.
  • Load repeated_results.jsonl as the report source.
  • Count the pass verdicts across the five attempts.
  • Record your consistency score here: your passes out of 5.

The repeated report shows a pass total out of five. A mixed set of verdicts exposes inconsistency that the single run could not reveal.

✔️ Awesome, I've got everything!

Your repeated report covers the same one_task.yaml retail task across five attempts. It shows the number of passing runs out of the total.

ⓧ I'd like to double check the full code

Compare your repeated evaluation command with this complete reference.

uv run --project envs/thinkingbox_env thinkingbox-eval \
  one_task.yaml \
  --config "$PWD/thinkingbox.yaml" \
  --output repeated_results.jsonl \
  --errors-output repeated_errors.jsonl \
  --repeat 5 --message-timeout 1800

Why These Values Matter

This command keeps the original task file unchanged. It writes five attempt records to a separate repeated evaluation report.

You have closed the reliability gap. Your repeated report now measures the model across five attempts.

Secret mission

Audit the Model Before Trusting It

Turn your single-run verdict and repeat score into an evidence-based recommendation. Decide whether the model is trustworthy for the retail task by comparing its claim with the Typesense state.

Clean Up Your Resources

Clean Up Your Resources

Your local Typesense container does not create ongoing cloud charges. Future ThinkingBox evaluations can still consume model endpoint credits.

Resources you used:

  • A local Typesense container running in Docker. It holds the database state that ThinkingBox evaluates.

Keep Your Evaluation Evidence

Your one_task.yaml lets you rerun the same retail task on another model. The single-run report preserves the comparison between the agent's claim and the database evidence.

The repeated evaluation report preserves the model's consistency score. These files make future model comparisons repeatable.

Keep everything running

No action is needed. Choose this option when you plan to run another evaluation soon.

  • Leave the Typesense container running for your next evaluation.
  • Keep one_task.yaml for another model check.
  • Keep the single-run evaluation report for its claim-to-evidence comparison.
  • Keep the repeated evaluation report for its consistency score.

Pause - I'll come back to this later

This option stops the local container to free your laptop's resources. Your evaluation artifacts remain ready for another session.

  • Stop the Typesense container using Docker's container controls.
  • Check the running containers by running this command in the terminal from earlier:
docker ps

What Does This Check Show?

The command lists containers that are currently running. You should not see the Typesense container after stopping it.

Still See Typesense Running?

Confirm that you stopped the same Typesense container you verified earlier.

Help me troubleshoot why the Typesense container is still running.

  • Leave one_task.yaml in place for your next model check.
  • Leave the single-run evaluation report in place for future comparison.
  • Leave the repeated evaluation report in place for future comparison.

Delete - I don't want to use this again

Removing the container permanently deletes its local database state. The task file plus both evaluation reports remain available for future comparisons.

  • Stop the Typesense container using Docker's container controls if it is still running.
  • Remove the stopped Typesense container using Docker's container controls.
  • Confirm the Typesense container no longer appears in Docker's container list.
  • Leave one_task.yaml in place for future model checks.
  • Leave the single-run evaluation report in place as evidence from the first attempt.
  • Leave the repeated evaluation report in place as evidence of the consistency score.

Container Still Listed?

  • Stop the Typesense container before removing it.
  • Retry the remove action after Docker shows the container as stopped.

Help me remove the remaining Typesense container.

Nice Work!

Nice Work!

Strong finish! You used ThinkingBox to test whether an AI agent's claim matched the database evidence. You also turned repeated runs into a consistency score that you can compare across models.

You've learned how to:

  • Run a single-task evaluation to capture the agent's trajectory. Read its pass-or-fail verdict.
  • Compare the agent's claim with the Typesense database evidence checked by the judge.
  • Repeat the same task to measure a model's consistency score as passing runs out of the total.
  • Take on an optional Secret Mission to push your evidence-based evaluation skills further.

Ready to quiz yourself?