Reduce Agent Tool Token Costs

Measure how tool search reduces input tokens in an AI agent.

Introduction

30 Second Summary

A helpful assistant often receives a long instruction manual before every message. As that manual grows, each reply carries more reading before the real work even starts.

In this project, you will build a script that logs input tokens for the same three-turn conversation before and after using OpenRouter tool search. The final comparison shows the saving per turn while confirming that the AI agent still chooses the right tool.

What You'll Build

You will see the same three-turn test produce a smaller token log with every correct tool choice preserved.

By the end of this project, you'll have:

  • A repeatable 40-tool request that exposes the input token count returned for one turn.
  • Clear three-turn logs that show how a large tool library affects each request as the conversation continues.
  • A before-and-after savings table that shows the token reduction per turn alongside the tool selected.
  • Secret Mission: An optional challenge to push your skills further.

Are there any prerequisites?

You need an OpenRouter account with credits. You also need Python or curl to send the requests.

Before We Start

Every result in this project comes from repeated requests through OpenRouter. A missing account would stop the experiment before you can compare token counts.

Your computer also needs a local runtime for the measurement script. You will prove that Node.js 18 or newer can run from your terminal.

In this step, get ready to:
  • Create an OpenRouter account without creating an API key.
  • Confirm Node.js 18 or newer runs from your terminal.
  • Plan to store your future key in an environment variable outside source control.
Create an OpenRouter account

Your account provides access to the service that runs the experiment. No API key is created in this step.

The sign-up flow may ask you to verify your email or approve access through an identity provider.

  • Visit the OpenRouter sign-up page in your browser.
  • Choose one of the sign-up methods shown on the page.
  • Complete the account prompts shown by your chosen method.
  • Finish the verification request if one appears.
  • Stop when OpenRouter shows your signed-in account controls.

Good start. Your OpenRouter account is ready without exposing a credential.

Prepare Node.js and your terminal

Node.js runs JavaScript files on your computer. The terminal gives you a visible way to confirm that the runtime is available.

  • Open a terminal by following the instructions for your operating system.

macOS

  • Press Cmd+Space to open Spotlight.
  • Type Terminal into Spotlight.
  • Press Enter to open Terminal.

Windows

  • Press the Windows key to open search.
  • Type PowerShell into search.
  • Press Enter to open PowerShell.
  • Check the installed Node.js version by running this command:
node --version

What does this command do?

The command asks Node.js to print its installed version. This reveals whether your runtime meets the project's minimum.

✔️ I see a supported version

The first number in your output is 18 or greater. Your runtime is ready for the project.

ⓧ I see an older version

Your installed runtime is below Node.js 18. Installing the current LTS release gives you a supported version.

  • Visit the official Node.js download page.
  • Download the LTS installer for your operating system.
  • Run the downloaded installer with its default options.
  • Close your current terminal.
  • Open a fresh terminal using the instructions above.
  • Confirm the updated version by running this command:
node --version

What does this command confirm?

The command reads the version from the newly installed Node.js runtime. A fresh terminal allows your system to find the updated installation.

You should see a version whose first number is 18 or greater.

Still seeing the old version?

Close every terminal window before opening a new one. An existing terminal can continue using the older runtime path.

If the old version remains, help me find why my terminal still uses an older Node.js version.

ⓧ Command not found

Node.js is not installed yet. The official LTS installer provides a version newer than the project's 18 minimum.

  • Visit the official Node.js download page.
  • Download the LTS installer for your operating system.
  • Run the downloaded installer with its default options.
  • Close your current terminal.
  • Open a fresh terminal using the instructions above.
  • Confirm the installation by running this command:
node --version

What does this command confirm?

The command proves that your terminal can locate the Node.js runtime. It also prints the installed version for comparison with the project minimum.

You should see a version number whose first number is 18 or greater.

Node.js still unavailable?

Confirm that the installer completed before reopening your terminal. Restart your computer if a fresh terminal still cannot locate Node.js.

If the command remains unavailable, help me make Node.js available in my terminal.

Plan safe API key storage

An API key authorizes requests against your account balance. You are safe to plan its storage now because no secret exists yet.

Why use an environment variable?

An environment variable keeps the key outside your script file. Your script reads the value when it runs.

OpenRouter documents OPENROUTER_API_KEY as the variable name. Keeping the key out of source control prevents it from entering a shared code history.

  • Use OPENROUTER_API_KEY as the environment variable name when a later step creates your key.
  • Keep the future key value out of every project file.
  • Keep the future key value out of source control.

Before you run the final check, what version do you expect your terminal to report?

  • Confirm your runtime is still available by running this command:
node --version

What does this final check prove?

The command proves that your terminal can reach the runtime after setup. This is the environment your measurement script can use.

You should see a version number whose first number is 18 or greater.

Did the version check stop working?

Open a fresh terminal before trying again. A terminal left open during installation may not know where Node.js is located.

If the command still fails, help me restore access to Node.js from my terminal.

  • Switch back to the OpenRouter page from earlier.
  • Confirm your signed-in account controls remain visible in the page header.

You have both foundations in place. Your account is available while your local runtime is ready to execute the experiment.

Your OpenRouter account and Node.js runtime are ready. Next, you will create a protected API key for the first measured request.

Measure One Large Tool Request

Every definition in a large tool library becomes part of the model's input when all tools are loaded. That repeated input creates a token cost before the model has answered anything.

In this step, you will connect to OpenRouter with an API key. You will send one request with 40 standard definitions for tool calling to establish your baseline.

In this step, get ready to:
  • Create an OpenRouter API key without storing it in source control.
  • Build a Python script containing 40 distinct tool definitions.
  • Send one request to record its input-token count.
Create and protect your API key

An API key authorizes the script to use your OpenRouter account. The environment variable keeps that credential outside the Python file.

This key can authorize requests that spend account credits. Keep it inside your terminal session so it never enters the script or a repository.

  • Return to your OpenRouter account's API keys page.
  • Create a key named tool-token-baseline with the key creation form.
  • Copy the generated key when it is shown.
  • Switch back to the terminal from your local development environment.

macOS

  • Set OPENROUTER_API_KEY for the current terminal session by replacing the placeholder in this command:
export OPENROUTER_API_KEY="sk-or-v1-YOUR-API-KEY"

What does this command do?

The command stores your key in OPENROUTER_API_KEY for this terminal session. The value stays outside test_openrouter.py.

Windows

  • Set OPENROUTER_API_KEY for the current PowerShell session by replacing the placeholder in this command:
$env:OPENROUTER_API_KEY="sk-or-v1-YOUR-API-KEY"

What does this command do?

The command stores your key in OPENROUTER_API_KEY for this PowerShell session. The value stays outside test_openrouter.py.

Linux

  • Set OPENROUTER_API_KEY for the current terminal session by replacing the placeholder in this command:
export OPENROUTER_API_KEY="sk-or-v1-YOUR-API-KEY"

What does this command do?

The command stores your key in OPENROUTER_API_KEY for this terminal session. The value stays outside test_openrouter.py.

Having trouble setting the key?

Check that you replaced the placeholder with the complete key. Keep the quotation marks around the value.

Use the same terminal session when you run the script. A new terminal does not inherit a session-only environment variable.

Ask for help without including the key itself: help me set an OpenRouter API key as a session-only environment variable.

Build the 40-tool baseline

The baseline needs every tool definition loaded into one request. Each definition follows the same function schema while keeping a distinct name.

  • Create test_openrouter.py on your Desktop with your code editor's new-file control.
  • Paste this first code chunk into test_openrouter.py.
import json, os, urllib.error, urllib.request

# Keep the model and API endpoint consistent across every measurement.
MODEL = "google/gemini-3-flash-preview"
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"

# Forty names make every generated function definition distinct.
TOOL_NAMES = (
    "get_weather", "get_forecast", "search_flights", "book_flight", "search_hotels", "book_hotel", "get_exchange_rate", "convert_currency", "get_stock_price", "get_market_news",
    "search_products", "get_product_details", "check_inventory", "create_order", "cancel_order", "track_order", "calculate_shipping", "validate_address", "find_restaurants", "book_restaurant",
    "search_events", "buy_event_ticket", "get_movie_times", "buy_movie_ticket", "search_books", "get_book_details", "search_music", "create_playlist", "send_email", "read_email",
    "create_calendar_event", "list_calendar_events", "create_task", "complete_task", "search_contacts", "create_contact", "get_map_route", "find_nearby_places", "translate_text", "summarize_text",
)

# Build the standard function schema that the model receives for every tool.
tools = [
    {
        "type": "function",
        "function": {
            "name": name,
            "description": f"Use the {name.replace('_', ' ')} capability.",
            "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]},
        },
    }
    for name in TOOL_NAMES
]

print(f"Loaded tools: {len(tools)}")

What does this code do?

  • MODEL fixes the model used for this measurement.
  • TOOL_NAMES contains 40 distinct function names.
  • tools converts every name into a standard JSON function definition.
  • No definition contains defer_loading. This keeps the baseline fully loaded.
  • Save test_openrouter.py.
  • Run test_openrouter.py with your editor's Python run control.

You'll see Loaded tools: 40 in the terminal. That output confirms the script generated one definition for every tool name.

Seeing a different tool count?

Compare each entry in TOOL_NAMES with the code block. A missing name lowers the count.

Check that the closing parenthesis remains below summarize_text.

Need another set of eyes? help me find why this Python tuple does not generate exactly 40 tool definitions.

Send the request and record usage

OpenRouter returns a usage object with every non-streaming response. Its prompt_tokens value counts the request input across the user message plus the loaded tool schemas.

  • Append this code below the Loaded tools print line in test_openrouter.py.
# Read the credential without placing its value in source code.
api_key = os.getenv("OPENROUTER_API_KEY")
if not api_key:
    raise RuntimeError("Set OPENROUTER_API_KEY before running this script.")

# Send all 40 schemas with one weather question.
payload = {
    "model": MODEL,
    "messages": [{"role": "user", "content": "What's the weather in Tokyo?"}],
    "tools": tools,
    "tool_choice": "auto",
}
request = urllib.request.Request(
    ENDPOINT,
    data=json.dumps(payload).encode("utf-8"),
    headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
    method="POST",
)

try:
    with urllib.request.urlopen(request) as response:
        data = json.load(response)
except urllib.error.HTTPError as error:
    print(error.read().decode("utf-8"))
    raise

# Print the complete usage object before isolating the baseline input count.
usage = data["usage"]
print(json.dumps(usage, indent=2))
print(f"Input tokens: {usage['prompt_tokens']}")

What does this code do?

  • os.getenv() reads the credential from your terminal environment.
  • payload combines the selected model with one user message.
  • tools places all 40 function definitions in the same request.
  • usage['prompt_tokens'] isolates the input-token count you need for the baseline.
  • Save test_openrouter.py.

Before you run the request, do you expect the input count to resemble a short chat message or a message carrying 40 schemas?

  • Run test_openrouter.py with your editor's Python run control.

You'll see the complete usage object. You'll also see an Input tokens: line containing your baseline count.

  • Record the displayed count here: your baseline input-token count.

Request failed or usage is missing?

An authentication error usually means the current terminal cannot read your key. Set the environment variable again in the same session that runs the script.

A model error can mean the selected model is temporarily unavailable. Confirm that your script still uses the model value shown in the full code below.

Share the error without sharing your credential: help me debug this OpenRouter Python request while keeping my API key private.

✔️ Awesome, I've got everything!

Your script contains 40 fully loaded tool definitions. Your baseline input-token count is recorded.

ⓧ I'd like to double check the full code

import json, os, urllib.error, urllib.request

# Keep the model and API endpoint consistent across every measurement.
MODEL = "google/gemini-3-flash-preview"
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"

# Forty names make every generated function definition distinct.
TOOL_NAMES = (
    "get_weather", "get_forecast", "search_flights", "book_flight", "search_hotels", "book_hotel", "get_exchange_rate", "convert_currency", "get_stock_price", "get_market_news",
    "search_products", "get_product_details", "check_inventory", "create_order", "cancel_order", "track_order", "calculate_shipping", "validate_address", "find_restaurants", "book_restaurant",
    "search_events", "buy_event_ticket", "get_movie_times", "buy_movie_ticket", "search_books", "get_book_details", "search_music", "create_playlist", "send_email", "read_email",
    "create_calendar_event", "list_calendar_events", "create_task", "complete_task", "search_contacts", "create_contact", "get_map_route", "find_nearby_places", "translate_text", "summarize_text",
)

# Build the standard function schema that the model receives for every tool.
tools = [
    {
        "type": "function",
        "function": {
            "name": name,
            "description": f"Use the {name.replace('_', ' ')} capability.",
            "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]},
        },
    }
    for name in TOOL_NAMES
]

print(f"Loaded tools: {len(tools)}")

# Read the credential without placing its value in source code.
api_key = os.getenv("OPENROUTER_API_KEY")
if not api_key:
    raise RuntimeError("Set OPENROUTER_API_KEY before running this script.")

# Send all 40 schemas with one weather question.
payload = {
    "model": MODEL,
    "messages": [{"role": "user", "content": "What's the weather in Tokyo?"}],
    "tools": tools,
    "tool_choice": "auto",
}
request = urllib.request.Request(
    ENDPOINT,
    data=json.dumps(payload).encode("utf-8"),
    headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
    method="POST",
)

try:
    with urllib.request.urlopen(request) as response:
        data = json.load(response)
except urllib.error.HTTPError as error:
    print(error.read().decode("utf-8"))
    raise

# Print the complete usage object before isolating the baseline input count.
usage = data["usage"]
print(json.dumps(usage, indent=2))
print(f"Input tokens: {usage['prompt_tokens']}")

Your 40-tool baseline is now measured without exposing your credential. Next, you will extend this request into a three-turn conversation.

Log the Baseline Conversation

Your single request gave you one baseline count for input tokens. A conversation creates a stronger test because each later request includes the earlier messages.

This step turns your OpenRouter test into a repeatable three-turn baseline. You will keep all 40 tools fully loaded while the script records one input-token count per turn.

In this step, get ready to:
  • Define three fixed requests for the baseline conversation.
  • Preserve the message history across all three turns.
  • Record a labelled input-token count for every turn.
Define the three turns

A fair comparison uses identical requests in both runs. Fixing the order now gives you a baseline that you can reproduce later.

  • Switch back to the token-measurement script from earlier.
  • Choose three distinct tools from the existing 40-tool library.
  • Write one unambiguous user request for each selected tool.
  • Place the three requests in a fixed ordered collection inside the script.
  • Keep the wording of every request unchanged for the later comparison.

Why Freeze the Wording?

Prompt wording affects how many tokens enter a request. Fixed wording keeps the tool-loading strategy as the main difference between your two runs.

Preserve the conversation history

A multi-turn conversation needs one persistent message collection. Each request receives everything said before that turn.

How Does the History Grow?

  • The loop appends the current user request before each API call.
  • The request sends the complete accumulated message collection.
  • The loop appends the assistant response before moving to the next turn.
  • Replace the single-request flow with a three-iteration conversation loop that follows the sequence above.
  • Save the token-measurement script.
  • Run the script with the same terminal command you used for the single-request test.

You should see three request cycles complete instead of one. This confirms that every prompt reaches OpenRouter in the expected order.

Only Seeing One Request Cycle?

  • Check that the request call sits inside the three-turn loop.
  • Confirm that the loop reads the entire ordered collection of three requests.
  • Check that the assistant response is stored before the loop advances.

Get help with the conversation loop.

Run and record the baseline

The final baseline needs a turn label beside every value from the usage field. Those labels keep the two experiment runs aligned.

  • Update the existing usage readout so every input-token count includes its turn number.
  • Save the token-measurement script.

Before you run the baseline, do you expect the input-token count to stay flat as the saved conversation grows?

  • Run the script with the same terminal command you used earlier.

You should see three labelled input-token counts for turns 1 through 3. Each label should have a numeric value from its response usage field.

Missing a Turn or Token Count?

  • Confirm that the turn label is generated inside the three-turn loop.
  • Check that each iteration reads the input-token count from its own response usage field.
  • Confirm that the script prints the count before advancing to the next request.

Get help tracing the missing baseline value.

  • Record the first value here: turn 1 input-token count.
  • Record the second value here: turn 2 input-token count.
  • Record the third value here: turn 3 input-token count.

Solid work. Your script now carries one conversation through three turns while measuring every request.

Your three labelled baseline counts are ready for a direct comparison with the next run.

Defer Tool Loading

Your baseline three-turn OpenRouter run is complete. You now have an input-token count for each turn with all 40 tool definitions loaded.

That baseline gives you a control for the experiment. This step changes only how the tool library reaches the model.

In this step, get ready to:
  • Mark all 40 existing tool definitions for deferred loading.
  • Rerun the same three-turn conversation.
  • Record the new input-token count for each turn.
Mark every tool for deferred loading

Every fully loaded tool definition adds text to the model input. The defer_loading field with the value true keeps a definition hidden until tool search reveals it.

Why change only one variable?

A controlled comparison changes one variable. Here, deferred loading is that variable.

Everything else stays fixed. This makes any change in token usage traceable to the loading method.

  • Return to the token-measurement script from the previous step.
  • Find the collection that contains the 40 standard function tool definitions.
  • Add the boolean field defer_loading with the value true inside every existing function tool definition.
  • Leave every other tool field unchanged.
  • Save the token-measurement script.
  • Use your editor's search feature to find every occurrence of defer_loading.
  • Inspect the matching definition for each of the 40 standard tools.

You should find 40 matches for defer_loading. Each match should have the value true.

Fewer than 40 tools deferred?

Move through the full tool collection from its first definition to its last. Count only the 40 standard function definitions.

Check that each defer_loading field sits inside a function tool definition.

Review the missing definitions with help me check why fewer than 40 tool definitions contain defer_loading.

Rerun the controlled conversation

The rerun uses the same experiment conditions as the baseline. This keeps the comparison focused on deferred loading.

Keep the comparison controlled

The OpenRouter model stays the same. The three conversation prompts stay the same.

The turn order stays the same. The message-history logic stays the same.

Each existing tool schema also stays unchanged except for defer_loading.

  • Return to the terminal from the previous run.

Before you rerun the script, do you expect the three input-token counts to match your baseline?

  • Rerun the token-measurement script with the same terminal command from the previous step.

You should see a fresh input-token count for turn 1. The script should also log separate counts for turns 2 and 3.

  • Record the turn 1 value here: your turn 1 deferred input-token count.
  • Record the turn 2 value here: your turn 2 deferred input-token count.
  • Record the turn 3 value here: your turn 3 deferred input-token count.

You have secured the second dataset for the same three-turn conversation. Your deferred-loading log now contains one labeled input-token count for every turn.

Run stopped before three counts?

Confirm that OPENROUTER_API_KEY is still available in the terminal session. Keep its value hidden.

Check the tool definitions for misplaced commas or brackets near the new fields. Restore the original schema structure before rerunning the script.

Compare your existing search configuration with the OpenRouter tool-search guide if the request rejects deferred tools.

Investigate the failed turn with help me troubleshoot why my deferred-loading OpenRouter run stopped before logging all three input-token counts.

Your deferred-loading token log is complete. Next, you'll confirm that each turn still selects the right tool.

Verify the Tool Choices

Your deferred-loading run now has input-token counts for all three turns. Those counts are useful only if the model kept choosing tools that fit each request.

OpenRouter Tool Search surfaces relevant definitions before the model makes a tool call. In this step, you'll trace each request through its search output to the selected function.

In this step, get ready to:
  • Review the tool-search output for each deferred-loading turn.
  • Match each selected tool to the request that triggered it.
  • Confirm appropriate tool selection across all three turns.
Trace the first turn

The tool-search output shows the definitions surfaced for a request. The tool-call output identifies the function the model selected.

  • Switch back to the terminal from the previous step.
  • Rerun the same deferred-loading conversation if the complete responses are no longer visible.
  • Keep the complete response for every turn visible during this review.
  • Locate the first request in the deferred-loading output.
  • Read the tool-search output for the first turn.
  • Identify the selected function in the tool-call output for the first turn.
  • Compare the selected function's description with the first request.

You should see a selected function whose stated purpose fits the first request.

Missing the complete response output?

A usage-only log cannot prove which tool the model selected. Keep the full deferred-loading responses available beside their turn numbers.

  • Return to the token-measurement script from earlier.
  • Retain the search output for every deferred-loading turn.
  • Retain the selected tool call for every deferred-loading turn.
  • Rerun the same three-turn conversation with the same deferred-loading settings.

Help me retain and inspect the tool-search output plus selected tool calls from my three-turn OpenRouter script.

Trace the second turn

Conversation history gives the model context for the second request. The selected function should serve the action requested in that preserved context.

  • Locate the second request in the same deferred-loading output.
  • Read the tool-search output for the second turn.
  • Identify the selected function in the tool-call output for the second turn.
  • Compare the selected function's description with the second request.

You should see a second request-to-tool match that fits the action requested in that turn.

Confirm the final turn

The third turn has the longest preserved history. Its evidence completes the selection check across the conversation.

  • Locate the third request in the same deferred-loading output.
  • Read the tool-search output for the third turn.
  • Identify the selected function in the tool-call output for the third turn.

Before you make the final check, do you expect the third selected function to fit its request?

  • Compare the third selected function's description with the third request.
  • Review the confirmed match from the first turn.
  • Review the confirmed match from the second turn.
  • Confirm that all three requests have an appropriate tool selection.

You should now have three confirmed request-to-tool matches. The tool-search output plus tool-call output should support each decision.

The selection evidence does not require any credentials. A safe screenshot shows the requests plus their selected tool names.

That closes the quality gap. Your deferred-loading run still selected an appropriate tool for every turn.

Next, you'll place both token logs in a table. You'll calculate the input-token saving for each turn.

Calculate the Token Savings

Your deferred-loading OpenRouter run still selected the appropriate tool on all three turns. The agent kept its tool-selection accuracy while loading tools on demand.

The two input-token logs show each run separately. A turn-by-turn comparison turns those raw counts into evidence of the savings.

In this step, get ready to:
  • Pair the baseline count with the deferred-loading count for each turn.
  • Calculate the input-token savings for all three turns.
  • Retain the script plus its results as a reusable testing template.
Build the comparison table

A useful benchmark keeps matching turns on the same row. This isolates the effect of marking the tool definitions defer_loading.

  • Return to the baseline three-turn log from earlier.
  • Return to the deferred-loading three-turn log from earlier.
  • Create a four-column table with the headers Turn, Baseline input tokens, Deferred-loading input tokens, and Token savings.
  • Add three rows labeled Turn 1, Turn 2, and Turn 3.

You should now see an empty row for each matching turn from the two runs.

  • Enter the fully loaded input-token count in each matching baseline cell.
  • Enter the deferred-loading input-token count in each matching deferred cell.

Each row should now place the two input-token counts side by side.

Why Match by Turn?

Each turn includes a different amount of preserved message history. Comparing different turns would mix two different conversation contexts.

Matching the same turn from both runs keeps the model request comparable. The remaining difference shows the effect of tool loading more clearly.

Calculate each turn's saving

Token savings is the difference between the matched input-token counts. Keeping that difference per turn shows how the benefit behaves as conversation history grows.

  • Calculate every row using baseline input-token count - deferred-loading input-token count.
  • Enter each result in the matching Token savings cell.
  • Check each subtraction against the two counts shown on its row.

You should now see three complete rows. Each row shows the baseline count, the deferred-loading count, and the calculated saving.

Preserve the reusable template

A reusable benchmark records the conditions that make two runs comparable. These notes let you repeat the test with another agent without rebuilding the method from memory.

  • Add a Reuse checklist below your comparison table.
  • Record that both runs use the same OpenRouter model.
  • Record that both runs use the same defined three-turn conversation.
  • Record that both runs preserve message history between turns.

What Makes This Reusable?

Fixed comparison conditions make the result meaningful. Changing the model or conversation could change the input-token count before tool loading has any effect.

The checklist turns this experiment into a repeatable benchmark for future agents.

  • Record that the baseline mode uses all 40 fully loaded tool definitions.
  • Record that the deferred mode marks all 40 tool definitions defer_loading.
  • Record that each run retains its usage counts plus its tool-search output plus its tool-call output.
  • Keep the token-measurement script with the completed comparison table in the existing project location.

Before the final check, which turn do you expect to show the largest saving?

  • Verify that the table contains one complete row for each of the three turns.
  • Verify that every token-savings value equals its baseline count minus its deferred-loading count.
  • Confirm that the reuse checklist describes the fully loaded mode.
  • Confirm that the reuse checklist describes the deferred-loading mode.
  • Confirm that the retained results include the tool-selection output for all three turns.

You should see three matched comparisons with a calculated saving on every row. You should also have the script plus its testing conditions ready for another tool-library benchmark.

Do the Savings Look Inconsistent?

  • Check that every row pairs the same turn from both logs.
  • Confirm that both runs used the same OpenRouter model.
  • Confirm that both runs preserved the same message history.

Help me audit my token-savings comparison for mismatched turns or testing conditions.

Strong finish. Your table now proves the input-token savings per turn while your retained script provides a repeatable template for future agents.

Secret mission

Turn Token Savings Into a Loading Rule

Extend your comparison table with percentage savings for each turn. Use the overall reduction and your tool-selection checks to write a loading recommendation for future agents.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to preserve this reusable test setup, put it aside for later, or remove it completely. Nothing keeps running after the script finishes, so the idle project has no ongoing usage cost. Future runs can consume OpenRouter credits.

Resources you used:

  • OpenRouter API key created for this project.
  • Local OPENROUTER_API_KEY environment variable.
  • Local token measurement script configured with 40 tool definitions.
  • Saved baseline logs, deferred-loading logs, token-savings comparison table, and tool-selection verification.

Keep everything running

No action is needed. Choose this option if you want to reuse the script for future agents or larger tool libraries.

  • Keep the OpenRouter API key active for future tests.
  • Keep OPENROUTER_API_KEY in your local environment while continuing to keep it out of source control.
  • Keep the token measurement script as your reusable testing template.
  • Use the saved comparison table as a baseline for future tool-loading experiments.

Pause - I'll come back to this later

Close your terminal session to clear any session-only configuration. Your script and measurements stay available for your next test.

  • Close any terminal session that currently holds the OPENROUTER_API_KEY environment variable.
  • Keep the token measurement script in its current local location.
  • Keep the baseline logs, deferred-loading logs, comparison table, and tool-selection verification with the script.
  • Restore OPENROUTER_API_KEY in a fresh terminal session when you resume testing.

Delete - I don't want to use this again

Remove the local project resources if you no longer need the testing template. Revoking the key permanently stops it from authorizing future requests.

Protect Your OpenRouter Key

Your OpenRouter account remains available after you revoke this project key. You can create a different key if you run another experiment later.

Revoke the OpenRouter API key:

  • Return to the OpenRouter account you used earlier.
  • Go to the API key management area where you created the project key.
  • Select the API key created for this project.
  • Use the available key-management control to revoke or delete it.
  • Confirm that the key no longer shows as active.

Remove the local environment variable:

  • Close the terminal session that contains OPENROUTER_API_KEY.
  • Remove any persistent OPENROUTER_API_KEY entry from the same shell profile or system environment settings where you saved it.
  • Confirm that the persistent entry is absent from that location.

Delete the local project files:

  • Use your computer's file manager to locate the local project files containing the token measurement script and its saved results.
  • Move those project files to your system trash.
  • Empty your system trash to remove the files permanently.
  • Confirm that the original location no longer contains the token measurement script or its saved results.

Nice Work!

Nice Work!

You completed a controlled OpenRouter experiment that measures the input-token overhead of a 40-tool library. Your reusable script now compares fully loaded tools with deferred loading across the same three-turn conversation.

You've learned how to:

  • Measure input-token usage for a single request. You also captured baseline counts across three turns with message history preserved.
  • Test deferred tool loading by marking all 40 tools defer_loading. Your second run used the same model plus the same conversation.
  • Verify tool-selection accuracy for every deferred-loading turn. Your comparison table shows the token savings for each turn.
  • Secret Mission: Complete an optional challenge that extends your token-cost testing skills.

You now have a repeatable benchmark for deciding whether deferred tool loading can reduce the cost of future agents.

Ready to quiz yourself?