Compare AI Browser Agent Models

Benchmark Browser Use Cloud models for speed, cost, and accuracy.

Introduction

30 Second Summary

A browser agent can finish a web task quickly, but a model name does not tell you whether that speed is worth the price. A fair comparison comes from giving both models the same work.

In this project, you will send real web tasks to Browser Use Cloud with curl. You will compare bu-ultrafast with bu-fast using time, cost, and accuracy recorded in a CSV.

What You'll Build

Your benchmark finishes with a chart that reveals which model gives your tasks the best balance of speed, cost, and accuracy.

By the end of this project, you'll have:

  • A live agent answer returned after you send a real web task with curl.
  • A playable recording that lets you inspect how the agent completes a multi-page task.
  • A repeatable model benchmark with one CSV row per run. Its chart makes the speed, cost, and accuracy gap visible.
  • Secret Mission: Add one of your own web tasks to the benchmark and test whether your model choice still holds.

Are there any prerequisites?

You need a laptop with a terminal that can run curl. You also need a Google, GitHub, or Microsoft account to create a Browser Use Cloud account without a card.

Before We Start

Your benchmark needs cloud credit before it can run real web tasks. It also needs a protected credential so those tasks can reach Browser Use Cloud.

In this step, you will create your account without a payment card. You will also prepare the local tools that collect your timing results.

In this step, get ready to:
  • Create a Browser Use Cloud account with signup credit.
  • Create an API key and store it outside your repository.
  • Prepare a local repository with curl and Python charting support.
Create your Browser Use Cloud account

Browser Use Cloud provides a one-time $1 signup credit for eligible Google, GitHub, or Microsoft signups. You can claim it without entering payment card details.

  • Visit the Browser Use Cloud signup page.
  • Choose Google, GitHub, or Microsoft as your sign-in method.
  • Complete the sign-in process with your chosen provider.
  • Approve the sign-in request if your provider asks for permission.
  • Check the credit balance shown in your Browser Use Cloud dashboard.

You should see $1 of signup credit in the dashboard. Your account is ready to fund the short benchmark runs later in the project.

How does the signup credit work?

The signup credit is a one-time grant. Every cloud run draws from the balance associated with your selected Browser Use Cloud project.

No subscription or payment card is required to claim the credit. Staying within this balance keeps the project free.

Create and store your API key

An API key identifies your requests to Browser Use Cloud. Anyone holding the key can spend your project credit.

  • Select Settings in the Browser Use Cloud dashboard.
  • Select API keys.
  • Create an API key from the API keys page.
  • Keep Full access enabled for this learning project.
  • Copy the key when Browser Use Cloud displays it.

Treat the copied value like a password. The next command keeps it in your current terminal session instead of placing it in a project file.

  • Press Cmd+Space on macOS or the Windows key on Windows to open your search bar.
  • Type Terminal on macOS or PowerShell on Windows.
  • Press Enter to open the terminal application.
  • Store the key in BROWSER_USE_API_KEY by using the tab for your operating system.

macOS

  • Replace your_key with the API key you copied.
  • Store the key in your current terminal session by running this command:
export BROWSER_USE_API_KEY=your_key

What does this command do?

  • The export command makes the key available to programs launched from this terminal.
  • The BROWSER_USE_API_KEY name matches the environment variable used by Browser Use Cloud.
  • The value remains outside the browser-use-benchmark repository.

Windows

  • Replace your_key with the API key you copied.
  • Store the key in your current PowerShell session by running this command:
$Env:BROWSER_USE_API_KEY = "your_key"

What does this command do?

  • The Env: provider stores environment variables for your current PowerShell session.
  • The BROWSER_USE_API_KEY name matches the environment variable used by Browser Use Cloud.
  • The value remains outside the browser-use-benchmark repository.

Having trouble storing the key?

Check that you replaced the entire your_key placeholder. Keep the quotation marks in the PowerShell command.

Leave this terminal open after setting the variable. Closing it removes the value from the current session.

Help me store my Browser Use Cloud API key as BROWSER_USE_API_KEY without exposing it in my repository.

Prepare your benchmark repository

The benchmark script uses Python to write CSV data. It uses Matplotlib to turn that data into a chart.

Current Matplotlib releases require Python 3.11 or newer. Check your installed version before creating the repository.

  • Return to the terminal from your API key setup.
  • Check your Python version by using the tab for your operating system.

macOS

  • Print your installed Python version by running this command:
python3 --version

What does this command show?

The version output identifies the Python interpreter available through python3. You need version 3.11 or higher for Matplotlib.

Windows

  • Print your installed Python version by running this command:
py --version

What does this command show?

The version output identifies the Python interpreter selected by the Windows launcher. You need version 3.11 or higher for Matplotlib.

✔️ I see a supported Python version

Good. Python 3.11 or higher is ready for your benchmark environment.

ⓧ I see an older Python version

Your installed Python cannot use the current Matplotlib release. Install a supported version before continuing.

  • Download the latest stable installer from the official Python downloads page.
  • Run the installer for your operating system.
  • Allow the installer to add Python to your command path.
  • Return to the version check above after the installation finishes.

ⓧ Python is not installed

Install Python before creating the benchmark environment. The official installers include the package tools used later in this step.

  • Download a supported installer from the official Python downloads page.
  • Run the installer for your operating system.
  • Allow the installer to add Python to your command path.
  • Return to the version check above after the installation finishes.

You also need curl for API requests. Git keeps the benchmark files under version control.

  • Confirm curl and Git are available by running these commands:
curl --version
git --version

What do these checks prove?

  • The first command prints the installed curl version plus its supported protocols.
  • The second command prints the installed Git version.
  • Version output from both commands confirms the local tools are reachable from this terminal.

✔️ I see both version numbers

Your terminal can reach both curl and Git. The repository setup can continue.

ⓧ One command is missing

  • Install curl from the official curl downloads page if the first command fails.
  • Install Git from the official Git downloads page if the second command fails.
  • Close the terminal after the installation completes.
  • Reopen the terminal through your operating system search.
  • Set BROWSER_USE_API_KEY again using the earlier platform tab.
  • Rerun both version checks.

A dedicated repository keeps the benchmark script beside its CSV results. It also gives every later file a predictable home.

  • Create the browser-use-benchmark repository on your Desktop by using the tab for your operating system.

macOS

cd ~/Desktop
mkdir browser-use-benchmark
cd browser-use-benchmark
git init

What do these commands do?

  • The first command moves your terminal to the Desktop.
  • The next two commands create browser-use-benchmark and enter it.
  • The final command creates the hidden .git directory used by Git.

Windows

$DesktopPath = [Environment]::GetFolderPath('Desktop')
Set-Location -Path $DesktopPath
New-Item -ItemType Directory -Path browser-use-benchmark
Set-Location -Path browser-use-benchmark
git init

What do these commands do?

  • The first two commands find your Windows Desktop and move the terminal there.
  • The next two commands create browser-use-benchmark and enter it.
  • The final command creates the hidden .git directory used by Git.

You should see a message confirming that Git initialized an empty repository. Your local benchmark now has a version-controlled home.

Repository already exists?

A directory creation error usually means browser-use-benchmark already exists on your Desktop. Enter that existing directory before running git init.

Help me initialize my existing browser-use-benchmark folder as a Git repository.

A virtual environment isolates Matplotlib from packages used by your other projects. The package download can take a minute while Python installs Matplotlib and its dependencies.

  • Create the virtual environment and install Matplotlib by using the tab for your operating system.

macOS

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U matplotlib

What do these commands do?

  • The first command creates an isolated Python environment in .venv.
  • The second command activates that environment in your current terminal.
  • The final command installs Matplotlib inside the isolated environment.
  • Python already includes the csv module used to write benchmark results.

Windows

py -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -U matplotlib

What do these commands do?

  • The first command creates an isolated Python environment in .venv.
  • The second command activates that environment in your current PowerShell session.
  • The final command installs Matplotlib inside the isolated environment.
  • Python already includes the csv module used to write benchmark results.

Matplotlib installation failed?

Confirm your terminal prompt begins with (.venv). This shows that the virtual environment is active.

On Windows, PowerShell may block the activation script. Follow the execution-policy guidance in the official Python virtual environment documentation.

Help me diagnose why Matplotlib will not install in my .venv environment.

Before you run the final check, which three parts of the setup do you expect it to confirm?

  • Verify the API key variable and Python libraries by running these commands:
python -c "print(('API key set', bool(__import__('os').environ.get('BROWSER_USE_API_KEY')), 'CSV ready', bool(__import__('csv')), 'Matplotlib ready', bool(__import__('matplotlib'))))"
git status

What does this final check prove?

  • The Python check confirms that BROWSER_USE_API_KEY exists without printing its secret value.
  • The library imports confirm that CSV support and Matplotlib are available.
  • The Git check confirms that your terminal is inside an initialized repository.

You should see True beside all three readiness checks. Git should report the current branch with no commits yet.

You now have the complete benchmark workspace ready for its first Browser Use Cloud request.

Seeing False in the readiness check?

A false API key result means the current terminal does not contain BROWSER_USE_API_KEY. Repeat the environment-variable command for your operating system.

A failed Matplotlib import means the package was installed outside the active .venv environment. Activate the environment before repeating the installation.

Help me fix the failed readiness check for my Browser Use benchmark setup.

✔️ I have everything

Your account, protected API key, repository, curl installation, and Python environment are ready.

ⓧ I'd like to double check my setup

  • Browser Use Cloud account: Created without a payment card; signup credit is visible in the dashboard.
  • BROWSER_USE_API_KEY: Created in Browser Use Cloud and stored locally as an environment variable; the key value is not committed to the repository.
  • Project repository: Local repository initialized for the Browser Use Cloud model benchmark.

Your workspace can now authenticate without exposing its key. Next, you will send a real task to the hosted browser agent with curl.

Run Your First Browser Task

Your Browser Use Cloud account is ready. The API key from the previous step is available in your terminal.

Now you will send a live web task with curl through the REST API. You will track the run through completion before inspecting the agent's answer.

In this step, get ready to:
  • Submit a web task with the bu-ultrafast model.
  • Track the run until it completes.
  • Inspect the agent's answer in the completed response.
Submit your first task

A run is one task sent to the hosted browser agent. The request uses the task field for the instruction.

The model field selects how the agent handles that task. This request uses bu-ultrafast for a short web lookup.

Before you send the request, predict whether the API returns a finished answer or a run record.

  • Switch back to the terminal from the previous step.
  • Submit the Hacker News task by running this command:
curl https://api.browser-use.com/api/v4/runs \
  -H "X-Browser-Use-API-Key: $BROWSER_USE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task":"Find the top Hacker News story","model":"bu-ultrafast"}'

What Does This Request Do?

  • The task field tells the browser agent what to find.
  • The model field selects bu-ultrafast for the run.
  • The X-Browser-Use-API-Key header authenticates the request with your existing environment variable.
  • The Content-Type header identifies the request body as JSON.

You'll receive a JSON response with a new run id. Its status value shows the run's current stage.

Request Did Not Create a Run?

Confirm that the API key variable is available in this same terminal. A new terminal may not have inherited the value you stored earlier.

Check that the request body keeps its single quotes around the JSON. Curly quotes copied from another app can break the request.

Help me troubleshoot my first Browser Use Cloud curl request.

  • Record the new id value here: your run ID.
  • Store your recorded run ID in this terminal session by running:
export RUN_ID="[[RUN_ID="your run ID"]]"

Why Store the Run ID?

The RUN_ID variable gives the next requests a direct reference to this run. Its value stays in your current terminal session.

Track the run to completion

The status endpoint gives you a lightweight progress check. It avoids fetching the full run record while the browser agent is still working.

  • Check the current run status by running this command:
curl https://api.browser-use.com/api/v4/runs/$RUN_ID/status \
  -H "X-Browser-Use-API-Key: $BROWSER_USE_API_KEY"

How Do Run Statuses Work?

  • The /status endpoint returns the run's current state.
  • An unfinished run can show queued, dispatching, or running.
  • A finished run shows completed, failed, or cancelled.

A short wait is normal here. Browser Use reports a median task time of 22 seconds for bu-ultrafast.

  • Wait a few seconds if you see queued, dispatching, or running.
  • Repeat the status command until you see completed.

The completed status confirms that the browser agent has finished the task.

Run Did Not Complete?

Give the run another short wait if it remains active. Repeated requests made too quickly do not make the browser finish sooner.

If the run ends without completing, read the response for its error details before submitting another task.

Help me understand why my Browser Use Cloud run did not complete.

Inspect the completed result

The full run endpoint returns the completed record. This record includes the agent's final answer.

Before you fetch the full run, predict which field holds the agent's answer.

  • Fetch the completed run response by running this command:
curl https://api.browser-use.com/api/v4/runs/$RUN_ID \
  -H "X-Browser-Use-API-Key: $BROWSER_USE_API_KEY"

What Does the Full Response Show?

  • The status field confirms whether the run completed.
  • The model field confirms that the run used bu-ultrafast.
  • The result field contains the agent's final text answer.

You'll see status set to completed. The result field contains the answer to the Hacker News task.

That's your first live task done: your terminal now retrieves a finished Browser Use Cloud answer.

Result Field Is Empty?

Confirm that the full response reports completed. Repeat the status check if the run is still active.

Check that RUN_ID still refers to the run you created in this step. A copied space can point the request at an invalid path.

Help me find the result in my completed Browser Use Cloud response.

Your first API run works. Next, you will record a multi-page task so you can watch the browser agent at work.

Record a Multi-Page Run

Your first Browser Use Cloud run returned an answer in its result field. That field only shows the finish.

A session recording reveals how the agent navigates between pages. In this step, you'll record a task that crosses two pages. You'll watch the saved video from start to finish.

In this step, get ready to:
  • Enable recording for a Browser Use Cloud run.
  • Send a task that moves between two web pages.
  • Watch the saved recording in Browser Use Cloud.
Enable recording for a multi-page run

API runs have recording disabled by default. The browserSettings.record option opts this request into a saved video.

  • Submit the two-page task with recording enabled by running this command:
curl https://api.browser-use.com/api/v4/runs \
  -H "X-Browser-Use-API-Key: $BROWSER_USE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task":"Visit https://example.com. Follow the Learn more link. Return the title of the destination page.","model":"bu-ultrafast","browserSettings":{"record":true}}'

What does this request do?

  • The task field gives the agent a two-page route from https://example.com to the linked IANA page.
  • The model field keeps this run on bu-ultrafast.
  • The browserSettings.record option enables recording for the browser session.
  • The $BROWSER_USE_API_KEY reference reads the key from your terminal environment.
  • The command avoids printing the secret value.
  • Confirm the terminal returns a JSON object for the new run.

Your recorded run is now underway. The response confirms that Browser Use Cloud accepted the task.

Was the run rejected?

If the terminal reports an authentication problem, confirm that the API key environment variable is still available in this terminal. Avoid printing or pasting the key itself.

If the request is rejected before creating a run, compare the quotation marks and braces against the complete request below.

Help me troubleshoot this recorded Browser Use Cloud request.

Wait for the recording

Browser Use Cloud finalizes the saved video after the browser stops. A completed run can keep its browser open for about two minutes.

You are still on track if the recording player takes a few minutes to appear. Upload processing starts after the browser stops.

  • Switch back to the signed-in Browser Use Cloud dashboard from earlier.
  • Locate the newest run by matching the task that begins with Visit https://example.com.
  • Select the matching task to open its run details.
  • Wait until the run status is completed.
  • Refresh the run details page after the browser has stopped.
  • Confirm the run details show a recording player.

Recording player still missing?

If the run is complete but the player is missing, allow more time for the browser to stop. Refresh the run details page after processing finishes.

Compare the submitted JSON with the request above. The browserSettings object must contain "record":true.

Help me find my Browser Use Cloud recording.

Watch the saved run

Before you start playback, which page change do you expect the video to capture?

  • Start playback in the recording player.
  • Watch the recording from start to finish.

You should see the agent begin on https://example.com. The video then shows it follow the named link to the destination page.

The moving playback proves that the recording was saved. The page change proves that the task crossed multiple pages.

You now have a replayable record of the agent's work. That evidence lets you inspect the journey behind its final answer.

✔️ Awesome, I've got everything!

Great. Your completed multi-page run has a saved recording that plays in Browser Use Cloud.

ⓧ I'd like to double check the full code

Compare your recorded request with this complete command. Run it again if your original request omitted the recording option.

curl https://api.browser-use.com/api/v4/runs \
  -H "X-Browser-Use-API-Key: $BROWSER_USE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task":"Visit https://example.com. Follow the Learn more link. Return the title of the destination page.","model":"bu-ultrafast","browserSettings":{"record":true}}'

Your multi-page evidence is saved. Next, you'll turn isolated runs into a repeatable benchmark across both models.

Benchmark Both Models

Your recorded multi-page run proved that Browser Use Cloud can finish a real web task. A single successful run still cannot tell you which model fits your work.

This step turns that question into a repeatable Python benchmark. The same three tasks run twice on each model. This gives you 12 results to compare against each other.

In this step, get ready to:
  • Define a fixed matrix of three web tasks.
  • Automate each run through both models.
  • Generate a CSV file with one row per run.
Define the benchmark matrix

A fair benchmark gives both models the same work. Repeating each task reveals whether one fast result was a lucky run.

  • Return to the local project repository from earlier in your code editor.
  • Create benchmark_browser_use.py at the top level of that repository with your editor's new-file control.
  • Paste the benchmark configuration into benchmark_browser_use.py using the code below:
import csv
import json
import os
import time
from urllib import request

# Use the Cloud API contract from the runs you tested earlier.
API_URL = "https://api.browser-use.com/api/v4/runs"
API_KEY = os.environ["BROWSER_USE_API_KEY"]
HEADERS = {
    "X-Browser-Use-API-Key": API_KEY,
    "Content-Type": "application/json",
}

# Keep every model on the same task matrix.
TASKS = [
    "Visit example.com and return the exact page heading.",
    "Visit Hacker News and return the top story title and points.",
    "Visit the Browser Use pricing page and return the cloud browser hourly price.",
]
MODELS = ["bu-ultrafast", "bu-fast"]
REPETITIONS = 2
OUTPUT_FILE = "benchmark_results.csv"
TERMINAL_STATUSES = {"completed", "failed", "cancelled"}

print(f"Configured {len(TASKS)} tasks across {len(MODELS)} models.")

What does this setup define?

  • The HEADERS dictionary reads your API key from the existing BROWSER_USE_API_KEY environment variable.
  • The TASKS list covers a fixed page. It also covers a changing news page. The final task checks a specific fact.
  • The MODELS list gives the same workload to bu-ultrafast and bu-fast.
  • Two repetitions produce 12 runs in total.
  • Save benchmark_browser_use.py.
  • Confirm the benchmark matrix loads by running this command:
python benchmark_browser_use.py

What should you see?

The terminal prints Configured 3 tasks across 2 models. This confirms the script can read its configuration without exposing your API key.

Does the script stop immediately?

Check that the terminal still has access to the environment variable you created earlier. The script stops before making a request if that variable is unavailable.

Check the indentation inside HEADERS if Python reports a syntax problem.

Ask for help with the configuration error.

Automate one Browser Use run

Each run progresses until it reaches a terminal status. The next function submits one task-model pair. It polls the lightweight status endpoint before fetching the final answer.

  • In benchmark_browser_use.py, find the line that prints the configured task count.
  • Insert the following function directly above that print line:
def run_once(task, model):
    # Start one timed run for this task-model pair.
    payload = json.dumps({"task": task, "model": model}).encode("utf-8")
    started_at = time.perf_counter()
    create_request = request.Request(API_URL, data=payload, headers=HEADERS, method="POST")
    with request.urlopen(create_request) as response:
        create_data = json.load(response)

    # Poll the compact status response until the run stops.
    run_id = create_data["id"]
    status = create_data["status"]
    print(f"{run_id} is {status}")
    while status not in TERMINAL_STATUSES:
        time.sleep(2)
        status_request = request.Request(f"{API_URL}/{run_id}/status", headers=HEADERS)
        with request.urlopen(status_request) as response:
            status = json.load(response)["status"]
        print(f"{run_id} is {status}")

    # Fetch the answer and model cost once the run is terminal.
    result_request = request.Request(f"{API_URL}/{run_id}", headers=HEADERS)
    with request.urlopen(result_request) as response:
        final_data = json.load(response)
    elapsed_seconds = round(time.perf_counter() - started_at, 2)
    answer = final_data.get("result") or final_data.get("error") or ""
    cost_usd = final_data.get("totalCostUsd") or "0"
    return {"task": task, "model": model, "answer": answer, "elapsed_seconds": elapsed_seconds, "cost_usd": cost_usd}

How does one run work?

  • The first request creates a run with the selected task and model.
  • The loop checks the run every two seconds until it is completed, failed, or cancelled.
  • The final request reads the answer from result.
  • The timer measures the full wait from submission to the final response.

Which cost does the script record?

The CSV stores totalCostUsd as cost_usd. This is the run's model and web-search charge.

Browser time and proxy traffic belong to session-level costs. Your dashboard credit can therefore fall by more than the sum of this column.

  • Save benchmark_browser_use.py.
  • Confirm the new function loads without a syntax error by running:
python benchmark_browser_use.py

What does this check prove?

You should see the same configuration message. Python has now loaded the request function successfully.

Seeing a Python error?

Confirm that run_once() starts at the far left of the file. Every line inside the function must use the same four-space indentation.

Check that the function appears above the existing print line.

Ask for help with the function error.

Run the full benchmark

A CSV keeps every attempt as a separate row. Flushing each row immediately also preserves completed results if a later cloud request fails.

  • Move to the end of benchmark_browser_use.py.
  • Append the loop that writes every run using the code below:
# Write each terminal run immediately so partial progress survives.
FIELDNAMES = ["task", "model", "answer", "elapsed_seconds", "cost_usd"]
with open(OUTPUT_FILE, "w", newline="", encoding="utf-8") as csv_file:
    writer = csv.DictWriter(csv_file, fieldnames=FIELDNAMES)
    writer.writeheader()
    total_runs = len(TASKS) * len(MODELS) * REPETITIONS
    run_number = 0
    for task in TASKS:
        for model in MODELS:
            for repetition in range(1, REPETITIONS + 1):
                run_number += 1
                print(f"Starting {run_number}/{total_runs}: {model}, repetition {repetition}")
                writer.writerow(run_once(task, model))
                csv_file.flush()

# Report the file that now holds the complete benchmark.
print(f"Saved {total_runs} rows to {OUTPUT_FILE}.")

What does the benchmark loop do?

  • The nested loops cover every task-model-repetition combination.
  • The DictWriter keeps every row in the same five-column format.
  • The flush after each row pushes the latest result into benchmark_results.csv immediately.
  • The final message reports how many rows the script saved.
  • Save benchmark_browser_use.py.

This starts 12 cloud runs that consume part of your existing signup credit. Your account has no payment card attached, so the benchmark can only use the credit already available.

Expect the full pass to take several minutes. The terminal can sit between updates while a browser agent works through its current task.

Before you run the benchmark, which model do you expect to finish the 12-run matrix faster?

  • Start the complete benchmark by running this command:
python benchmark_browser_use.py

What should you see?

The terminal prints a starting line for each run. Status updates continue until every run reaches a terminal state.

The final line reads Saved 12 rows to benchmark_results.csv. This confirms that the full matrix reached the results file.

Did the benchmark stop early?

Keep the terminal open if a run remains queued or running. Those messages show that polling is still active.

If a request fails, confirm that your signup credit remains available in the Browser Use Cloud dashboard. Rerunning the script starts a fresh CSV file.

Ask for help with the failed benchmark request.

  • Open benchmark_results.csv from your editor's file sidebar.
  • Count the data rows beneath the header.
  • Confirm that both model names appear in the model column.

You should see 12 data rows beneath the header. Each row includes task, model, answer, elapsed_seconds, and cost_usd.

That is the benchmark captured. You now have repeatable evidence from both models on the same workload.

✔️ Awesome, I've got everything!

Great. Double check that benchmark_browser_use.py and benchmark_results.csv are saved in your local project repository.

ⓧ I'd like to double check the full code

Compare your complete benchmark_browser_use.py file against this reference.

import csv
import json
import os
import time
from urllib import request

# Use the Cloud API contract from the runs you tested earlier.
API_URL = "https://api.browser-use.com/api/v4/runs"
API_KEY = os.environ["BROWSER_USE_API_KEY"]
HEADERS = {
    "X-Browser-Use-API-Key": API_KEY,
    "Content-Type": "application/json",
}

# Keep every model on the same task matrix.
TASKS = [
    "Visit example.com and return the exact page heading.",
    "Visit Hacker News and return the top story title and points.",
    "Visit the Browser Use pricing page and return the cloud browser hourly price.",
]
MODELS = ["bu-ultrafast", "bu-fast"]
REPETITIONS = 2
OUTPUT_FILE = "benchmark_results.csv"
TERMINAL_STATUSES = {"completed", "failed", "cancelled"}


def run_once(task, model):
    # Start one timed run for this task-model pair.
    payload = json.dumps({"task": task, "model": model}).encode("utf-8")
    started_at = time.perf_counter()
    create_request = request.Request(API_URL, data=payload, headers=HEADERS, method="POST")
    with request.urlopen(create_request) as response:
        create_data = json.load(response)

    # Poll the compact status response until the run stops.
    run_id = create_data["id"]
    status = create_data["status"]
    print(f"{run_id} is {status}")
    while status not in TERMINAL_STATUSES:
        time.sleep(2)
        status_request = request.Request(f"{API_URL}/{run_id}/status", headers=HEADERS)
        with request.urlopen(status_request) as response:
            status = json.load(response)["status"]
        print(f"{run_id} is {status}")

    # Fetch the answer and model cost once the run is terminal.
    result_request = request.Request(f"{API_URL}/{run_id}", headers=HEADERS)
    with request.urlopen(result_request) as response:
        final_data = json.load(response)
    elapsed_seconds = round(time.perf_counter() - started_at, 2)
    answer = final_data.get("result") or final_data.get("error") or ""
    cost_usd = final_data.get("totalCostUsd") or "0"
    return {"task": task, "model": model, "answer": answer, "elapsed_seconds": elapsed_seconds, "cost_usd": cost_usd}


print(f"Configured {len(TASKS)} tasks across {len(MODELS)} models.")

# Write each terminal run immediately so partial progress survives.
FIELDNAMES = ["task", "model", "answer", "elapsed_seconds", "cost_usd"]
with open(OUTPUT_FILE, "w", newline="", encoding="utf-8") as csv_file:
    writer = csv.DictWriter(csv_file, fieldnames=FIELDNAMES)
    writer.writeheader()
    total_runs = len(TASKS) * len(MODELS) * REPETITIONS
    run_number = 0
    for task in TASKS:
        for model in MODELS:
            for repetition in range(1, REPETITIONS + 1):
                run_number += 1
                print(f"Starting {run_number}/{total_runs}: {model}, repetition {repetition}")
                writer.writerow(run_once(task, model))
                csv_file.flush()

# Report the file that now holds the complete benchmark.
print(f"Saved {total_runs} rows to {OUTPUT_FILE}.")

Your benchmark data is ready. Next, you will chart the speed and cost differences. You will also mark which answers were correct.

Chart the Benchmark Results

Your Browser Use Cloud benchmark now produces one row of evidence for every run. Those rows still hide the tradeoff between speed, cost, and answer quality.

In this step, you will grade every answer against an expected result. You will turn the marked CSV data into a visual model comparison.

In this step, get ready to:
  • Review each answer against its expected result.
  • Mark every run as correct or incorrect.
  • Use a saved chart to choose a model.
Review and mark each answer

Accuracy needs a consistent rule. A run is correct when its answer satisfies the original task requirement.

Use the same expected answer for every repeat of a task. This keeps the comparison between bu-ultrafast and bu-fast fair.

  • Open your local project repository in a CSV-capable editor.
  • Open benchmark_results.csv from the repository.

You should see one row for every completed run. Each row includes the task, model, answer, elapsed time, and cost.

  • Add a column named expected answer after the cost column.
  • Add a column named accuracy after the expected answer column.

Your table should now include two empty grading columns beside the original benchmark data.

  • Enter the expected result for every run based on the original task requirement.
  • Compare each answer cell with its expected answer cell.

Each comparison now has a clear reference point. Apply the same standard across both models.

  • Enter correct in the accuracy column when the answer satisfies the task.
  • Enter incorrect in the accuracy column when the answer misses any required part.

Every run should now have a visible accuracy mark. Incorrect answers remain easy to find during charting.

  • Save benchmark_results.csv in your local project repository.
  • Scan the two new columns for blank cells.

You should see an expected answer plus a correct or incorrect mark on every row.

How should you score close answers?

Use the original task requirement as your scoring rule. Mark a partial answer as incorrect when it omits a required detail.

Consistency matters more than generosity here. One scoring rule gives both models the same test.

Build the model comparison chart

Elapsed time, cost, and accuracy use different units. Separate panels keep each measurement readable while preserving one model comparison.

Why use three panels?

An elapsed-time panel reveals the faster model. A cost panel reveals the cheaper model.

An accuracy panel shows whether either advantage came with more wrong answers. The three views expose the complete tradeoff.

  • Create a new chart from benchmark_results.csv with the Python charting support available in your project environment.
  • Group the CSV rows by the model column.

Your chart preview should now separate the runs for bu-ultrafast from the runs for bu-fast.

  • Set the first panel to show mean elapsed time for each model.
  • Set the second panel to show mean cost for each model.

The first two panels should now place one bar for each model beside the corresponding measurement scale.

  • Set the third panel to show accuracy as a 100 percent stacked bar for each model.
  • Use correct and incorrect as the stack categories.

The accuracy panel should now expose the share of successful runs. Its incorrect segment clearly marks the answers that failed your grading rule.

  • Label the model groups bu-ultrafast and bu-fast.
  • Save the chart in your local project repository as benchmark_chart.png.

You should see benchmark_chart.png beside your benchmark script and CSV file.

Chart missing data?

If a measurement panel is empty, confirm its CSV column contains numbers in every row. Remove any units or currency symbols stored inside those cells.

If both models appear as one group, confirm the chart groups rows by the model column.

If the accuracy panel is incomplete, check that every mark uses exactly correct or incorrect.

Ask for help with fixing my Browser Use benchmark chart.

Choose a model for future tasks

A useful benchmark ends with a decision. Your preferred model depends on how much speed, cost, and wrong answers matter for the tasks you plan to run.

Before you open the saved image, make a mental prediction about which model your measurements will favor.

  • Open benchmark_chart.png from your local project repository.

You should see three panels comparing both models. The panels show mean elapsed time, mean cost, and the percentage of correct versus incorrect runs.

  • Identify the model with the lower mean elapsed time.
  • Identify the model with the lower mean cost.
  • Inspect which model has the larger correct segment.

Match the Model to the Risk

Prioritize accuracy when a wrong answer creates rework or risk. Speed and cost can break a tie when both models meet your accuracy needs.

  • Choose the model whose tradeoff best fits your future web tasks.

That is the benchmark finished. Your model choice now comes from your own measured runs.

✔️ My benchmark is complete

Your marked CSV and saved comparison chart are ready to reuse with future tasks.

ⓧ I'd like to double check the results

  • Confirm benchmark_results.csv still contains one row per benchmark run.
  • Confirm every row includes the task, model, answer, elapsed time, cost, expected answer, and accuracy mark.
  • Confirm every accuracy mark is either correct or incorrect.
  • Confirm benchmark_chart.png compares bu-ultrafast with bu-fast across elapsed time, cost, and accuracy.
  • Confirm the accuracy panel clearly displays incorrect runs.

Secret mission

Create a Model Selection Policy

Your chart shows how the two models performed in this benchmark. In this mission, you will turn that evidence into a default-model policy with a measurable fallback trigger.

Clean Up Your Resources

Clean Up Your Resources

The benchmark files on your computer create no ongoing costs. Choose whether to keep the setup, pause Browser Use Cloud activity, or delete the setup entirely.

Cost warning

An active Browser Use Cloud session can consume your signup credit through browser time. Proxy traffic can consume additional credit.

The local project repository does not create cloud charges.

Resources you used:

  • Any Browser Use Cloud browser session that still shows as running.
  • The Browser Use Cloud API key stored locally as BROWSER_USE_API_KEY.
  • The local project repository containing benchmark_browser_use.py, benchmark_results.csv, plus benchmark_chart.png.

Keep everything running

No action is needed. Choose this option if you plan to rerun the benchmark or test new tasks.

  • Keep the local project repository so you can rerun the benchmark.
  • Keep the API key active for future Browser Use Cloud requests.
  • Review your remaining signup credit before starting another benchmark.
  • Stop any browser session that you no longer need.

Pause - I'll come back to this later

Pausing preserves your API access plus your local benchmark artifacts. It leaves no browser session running.

  • Return to the Browser Use Cloud dashboard from earlier.
  • Open the run details page for any browser session that still shows as running.
  • Use the run's stop control.
  • Confirm the session no longer shows as running.
  • Keep BROWSER_USE_API_KEY available for a later run.

That gives you a clean stopping point. Your benchmark artifacts remain ready for the next session.

Delete - I don't want to use this again

Deleting the API key blocks future requests that use it. Your Browser Use Cloud account plus its completed run history remain in place.

  • Return to the Browser Use Cloud dashboard from earlier.
  • Open the run details page for any browser session that still shows as running.
  • Use the run's stop control.
  • Confirm the session no longer shows as running.
  • Return to the key management page that you used earlier.
  • Select the API key that supplies BROWSER_USE_API_KEY.
  • Use the page's delete control.
  • Confirm the API key no longer appears on the page.

That is the risky part finished. Browser Use Cloud can no longer accept requests with the deleted key.

  • Remove BROWSER_USE_API_KEY from the same local place where you stored it earlier.
  • Close the terminal session to clear any remaining in-memory value.

Deleting the repository removes the local benchmark artifacts permanently after you empty deleted items. Your Browser Use Cloud account remains unaffected.

macOS

  • Use Finder to locate the local project repository that contains benchmark_browser_use.py.
  • Move the entire repository folder to Trash.
  • Empty Trash.
  • Confirm benchmark_browser_use.py no longer appears in its former location.

Windows

  • Use File Explorer to locate the local project repository that contains benchmark_browser_use.py.
  • Delete the entire repository folder.
  • Empty Recycle Bin.
  • Confirm benchmark_browser_use.py no longer appears in its former location.

Cleanup is complete. The reusable access plus local benchmark files are gone.

Nice Work!

Nice Work!

Outstanding work! You built a repeatable Browser Use Cloud benchmark that gives you measured evidence for choosing between two models on real web tasks.

You learned how to:

  • Submitted hosted web tasks with curl using bu-ultrafast. Confirmed each completed run by inspecting its result field.
  • Enabled a recorded multi-page run. Replayed the saved video to follow the browser agent from start to finish.
  • Built a repeatable model benchmark in benchmark_browser_use.py. Saved one row per run in benchmark_results.csv. Used benchmark_chart.png to compare model speed. The same chart compares cost. It also keeps accuracy visible through marked wrong answers.
  • Secret Mission: Extended the benchmark through the optional challenge. Applied the same evidence-first method to evaluate its result.

Ready to quiz yourself?