Compare OpenRouter Model Providers

Compare model providers, measure responses, and pin your preferred route.

Introduction

30 Second Summary

When an AI answer changes from one request to the next, the model name alone does not explain why. A different company may have handled each request behind the scenes.

In this project, you will use OpenRouter to uncover which company serves your AI requests. You will compare two providers with the same prompt before pinning your app to the provider you choose.

What You'll Build

Run your finished script to reveal the providers behind one model before proving that your chosen provider handles the final request.

By the end of this project, you'll have:

  • Print a provider table showing the price, context length, and quantization available for one model ID.
  • Send an identical prompt to two named providers. Save both replies so you can compare them side by side.
  • Measure each provider's latency, token use, and cost. Lock future requests to your chosen provider and confirm that the pin is enforced.
  • Secret Mission: Take on an optional challenge to push your provider comparison skills further.

Are there any prerequisites?

You need a laptop plus an OpenRouter account with an API key. A terminal or Python is enough, with no GPU or local model required.

Before We Start

A model name can hide which company actually handles your request. You need a clean starting point before you can expose that routing through OpenRouter.

In this step, you prepare your account access. You also confirm that your command-line tools are ready for the provider comparisons ahead.

In this step, get ready to:
  • Create or sign in to an OpenRouter account.
  • Confirm Python major version 3, curl, and jq are available.
  • Verify that your dashboard and terminal are ready without creating an API key.
Create your OpenRouter account

Your OpenRouter account provides access to the dashboard used throughout this project. No API key is created in this step.

  • Visit the OpenRouter homepage in your browser.
  • Choose the account path that matches your current situation below.

✔️ I already have an account

  • Click Sign In in the page navigation.
  • Complete the guided sign-in flow.

You should arrive in your signed-in OpenRouter workspace.

ⓧ I need an account

  • Click Sign Up in the page navigation.
  • Complete the guided account creation flow.

You should arrive in your new signed-in OpenRouter workspace.

You're in. Your OpenRouter workspace now gives you a safe starting point for creating credentials in the next step.

What is OpenRouter doing here?

OpenRouter routes one model ID across several backend inference providers. Your dashboard gives you the account controls needed to inspect that routing later.

Prepare your terminal tools

The project uses command-line tools to send requests and inspect JSON responses. Checking each tool now prevents setup problems from interrupting your first request.

macOS

  • Use your computer's app search to open a terminal window.
  • Check your Python installation by running this command:
python3 --version

What does this check prove?

The command prints the Python version available to your terminal. Confirm that the version line starts with major version 3.

  • Check your HTTP command-line client by running this command:
curl --version

Why check curl?

The command confirms that your terminal can call the HTTP client used to send API requests. Its output includes a curl version line.

  • Check your JSON processor by running this command:
jq --version

Why check jq?

The command confirms that your terminal can call the processor used to format API responses. Its output includes a jq version line.

  • Choose the outcome below that matches your three checks.

✔️ All three tools returned versions

Good work. Your terminal can now run Python scripts, send API requests, and format their responses.

  • Keep this terminal window open for the final readiness check.

ⓧ One or more tools are missing

Tool installation can feel fiddly because each download uses a different installer. Use the official source for every missing tool.

You should now see a Python major version of 3, a curl version line, and a jq version line.

Windows

  • Use your computer's app search to open a command-line window.
  • Check your Python installation by running this command:
py --version

What does this check prove?

The command prints the Python version managed by the Windows launcher. Confirm that the version line starts with major version 3.

  • Check your HTTP command-line client by running this command:
curl --version

Why check curl?

The command confirms that your command-line window can call the client used to send API requests. Its output includes a curl version line.

  • Check your JSON processor by running this command:
jq --version

Why check jq?

The command confirms that your command-line window can call the processor used to format API responses. Its output includes a jq version line.

  • Choose the outcome below that matches your three checks.

✔️ All three tools returned versions

That's your local toolkit ready. Your command-line window can run Python scripts, send API requests, and format their responses.

  • Keep this command-line window open for the final readiness check.

ⓧ One or more tools are missing

Tool installation can feel fiddly because each download uses a different installer. Use the official source for every missing tool.

  • Install Python using the official Windows instructions if the Python check failed.
  • Install curl from the official curl for Windows page if the curl check failed.
  • Install jq using an option on the official jq downloads page if the jq check failed.
  • Close the existing command-line window after the installations finish.
  • Use your computer's app search to open a new command-line window.
  • Repeat each version check shown in this tab.

You should now see a Python major version of 3, a curl version line, and a jq version line.

Still missing a command?

  • Restart your command-line window after an installation so it can load the updated command paths.
  • Return to the official download page for the missing tool if its version check still fails.

Ask for help with a command that is still unavailable.

Verify your starting point

Your dashboard proves that the account is available. One final terminal check confirms that the last tool in the response-processing chain is still accessible.

  • Return to the browser tab containing your OpenRouter workspace.
  • Confirm that your signed-in workspace is still visible.
  • Switch back to the terminal window from the previous substep.

macOS

Before you run this final check, do you expect jq to print its version line?

  • Run the final readiness check with this command:
jq --version

How to read the result

A jq version line confirms that the command remains available in your open terminal. Your earlier Python and curl results complete the toolkit check.

You should see the installed jq version in the terminal.

Windows

Before you run this final check, do you expect jq to print its version line?

  • Run the final readiness check with this command:
jq --version

How to read the result

A jq version line confirms that the command remains available in your command-line window. Your earlier Python and curl results complete the toolkit check.

You should see the installed jq version in your command-line window.

Did the final check stop working?

Open a fresh command-line window if the command worked earlier but now fails. This reloads the command paths changed by an installer.

Ask for help with restoring access to jq.

Your OpenRouter account and command-line tools are ready. Next, you'll create an API key and send your first request so the provider name appears in the response.

Send Your First OpenRouter Request

OpenRouter can route one model ID to several backend providers. The provider stays invisible until the response exposes the routing decision.

In this step, you create a private API key. You then send a baseline request to openai/gpt-4o-mini and save the returned JSON for inspection.

In this step, get ready to:
  • Create an OpenRouter API key for your terminal session.
  • Save a successful chat-completions response in baseline_response.json.
  • Identify the backend provider that served your request.
Create and load your API key

An API key authorizes requests against your OpenRouter account. Treat this credential like a password.

Working with a credential can feel sensitive. You keep this key in the current shell session without writing it into a project file.

  • Return to the OpenRouter dashboard from earlier.
  • Visit the OpenRouter key settings page in the same browser session.
  • Use the page's key creation control.
  • Enter NextWork provider check as the key name.
  • Create the key.
  • Copy the key value shown after creation.

An environment variable gives commands access to the key while this terminal remains open. The confirmation check verifies that a value exists without displaying the credential.

  • Switch back to the terminal from earlier.
  • Replace sk-or-v1-... with the copied key in the command below.
  • Set OPENROUTER_API_KEY for the current terminal session by running these commands:
export OPENROUTER_API_KEY=sk-or-v1-...
test -n "$OPENROUTER_API_KEY" && echo "OPENROUTER_API_KEY is set"

How does this protect the key?

  • The export command makes the key available to commands launched from this terminal.
  • The value stays out of baseline_response.json and other project files.
  • The test command checks that the variable contains a value without printing that value.

You should see OPENROUTER_API_KEY is set in the terminal.

No confirmation message?

  • Check that you replaced the placeholder with the complete key before running the commands.
  • Keep the key on the same line as OPENROUTER_API_KEY=.
  • Run the block again in the terminal session you plan to use for the request.

Help me set the OpenRouter API key safely in my current shell without saving it to a project file.

Send the baseline request

A chat-completions request sends a list of messages to one model ID. The metadata header asks OpenRouter to include the routing decision in the response.

Before you run this, which backend company do you think OpenRouter will choose for openai/gpt-4o-mini?

  • Save the returned response as baseline_response.json by running this command:
curl https://openrouter.ai/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "X-OpenRouter-Metadata: enabled" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [
      {"role": "user", "content": "In one sentence, explain why identical AI model names can produce different results across providers."}
    ]
  }' > baseline_response.json
jq '.' baseline_response.json

What does this request do?

  • The Authorization header reads the key from OPENROUTER_API_KEY.
  • The request sends one user message to openai/gpt-4o-mini.
  • The X-OpenRouter-Metadata: enabled header asks OpenRouter to return routing details.
  • The redirect saves the complete response in baseline_response.json.
  • The final jq command formats the saved response for inspection.

You should see a formatted response containing an assistant message. The terminal prompt returns after the file has been saved.

Did the request return an error?

  • Confirm that the key check printed its success message in this same terminal session.
  • Check that the model ID is exactly openai/gpt-4o-mini.
  • Confirm that your OpenRouter account can make a request with the selected model.

Help me troubleshoot my first OpenRouter chat-completions request without asking me to reveal my API key.

Inspect the provider response

The saved response contains the assistant reply under choices. Its routing metadata identifies the endpoint that OpenRouter selected.

Before you inspect the file, which value do you expect to change if another company serves a later request?

  • Extract the selected provider with the model ID and response details by running this command:
jq '{
  provider: (.openrouter_metadata.endpoints.available[] | select(.selected == true).provider),
  model: .model,
  message: .choices[0].message.content,
  usage: .usage
}' baseline_response.json

What should I see?

  • The provider field shows the company behind the selected endpoint.
  • The model field shows the model ID used for the completion.
  • The message field contains the assistant's one-sentence reply.
  • The usage field contains the request's token data.

Provider value missing?

  • Confirm that the request includes the X-OpenRouter-Metadata: enabled header.
  • Rerun the baseline request after restoring the metadata header.
  • Check that baseline_response.json contains a successful response instead of an error object.

Help me find the selected provider in my saved OpenRouter response.

That is your first routing result captured. Your saved response now connects one model ID to the company that actually served it.

Your baseline response now shows which company served the model. Next, you will list every provider available behind that model ID.

List Available Model Providers

Your baseline response proved that OpenRouter chose one backend provider for openai/gpt-4o-mini. One response cannot show the other endpoints that could serve your next request.

A reusable Python script can reveal the full inventory. The inventory exposes each endpoint's pricing. It also shows context length. It records quantization too.

In this step, get ready to:
  • Retrieve every available endpoint for the model.
  • Write the endpoint details to a readable table.
  • Select two provider identifiers for comparison.
Retrieve the endpoint inventory

The list-endpoints API returns the current endpoints for one model. Your script uses the key already held by this shell to authenticate.

  • Create provider_probe.py beside baseline_response.json by running this command:
cat > provider_probe.py <<'PY'
import csv
import json
import os
from urllib.request import Request, urlopen

# Keep the model and output file stable across every run
MODEL_ID = 'openai/gpt-4o-mini'
ENDPOINTS_URL = f'https://openrouter.ai/api/v1/models/{MODEL_ID}/endpoints'
OUTPUT_FILE = 'provider_endpoints.tsv'

# Retrieve every endpoint available for the selected model
request = Request(
    ENDPOINTS_URL,
    headers={'Authorization': f'Bearer {os.environ["OPENROUTER_API_KEY"]}'},
)
with urlopen(request) as response:
    endpoints = json.load(response)['data']['endpoints']

print(f'Retrieved {len(endpoints)} provider endpoints for {MODEL_ID}.')
PY

What does this code do?

  • MODEL_ID keeps every API call focused on openai/gpt-4o-mini.
  • ENDPOINTS_URL points to the model's endpoint inventory.
  • Request reads OPENROUTER_API_KEY from the current shell without saving its value.
  • endpoints holds every endpoint returned in the response.
  • Run the new script from the current terminal by using this command:
python3 provider_probe.py

You should see a positive endpoint count for openai/gpt-4o-mini. That count proves the API returned the inventory to your script.

Can't retrieve the endpoints?

  • Confirm that OPENROUTER_API_KEY is still set in this terminal session.
  • Check that MODEL_ID matches openai/gpt-4o-mini exactly.
  • Confirm that your terminal has an active internet connection.

Help me diagnose why the OpenRouter endpoints request in provider_probe.py is failing.

Write the provider table

Raw endpoint JSON is complete. A tab-separated table makes its repeated fields easier to scan.

  • Append the table writer to provider_probe.py by running this command:
cat >> provider_probe.py <<'PY'

# Write comparable routing details for every available endpoint
columns = ['provider', 'input_price', 'output_price', 'context_length', 'quantization']
with open(OUTPUT_FILE, 'w', newline='', encoding='utf-8') as table:
    writer = csv.writer(table, delimiter='\t')
    writer.writerow(columns)
    for endpoint in endpoints:
        writer.writerow(
            [
                endpoint['tag'],
                endpoint['pricing']['prompt'],
                endpoint['pricing']['completion'],
                endpoint['context_length'],
                endpoint['quantization'],
            ]
        )

print(f'Wrote {len(endpoints)} rows to {OUTPUT_FILE}.')
with open(OUTPUT_FILE, encoding='utf-8') as table:
    print(table.read())
PY

What does the table show?

  • provider contains the endpoint's exact routing identifier from tag.
  • input_price comes from the endpoint's prompt pricing value.
  • output_price comes from the endpoint's completion pricing value.
  • context_length shows the endpoint's context capacity.
  • quantization records the model precision used by that endpoint.

Before you run the expanded script, how many table rows do you expect compared with the endpoint count you just retrieved?

  • Generate provider_endpoints.tsv by running the expanded script:
python3 provider_probe.py

You should see a header row followed by one row for every returned endpoint. The written row count should match the retrieved endpoint count.

Table missing rows or columns?

  • Check that the table-writing code sits below the first endpoint count in provider_probe.py.
  • Compare the indentation beneath for endpoint in endpoints: with the full-file reference below.
  • Rerun the script after confirming that every field name matches the reference.

Help me fix an incomplete provider_endpoints.tsv file.

✔️ Awesome, I've got everything!

Your saved provider_probe.py now retrieves every endpoint. It writes the inventory to provider_endpoints.tsv.

ⓧ I'd like to double check the full code

import csv
import json
import os
from urllib.request import Request, urlopen

# Keep the model and output file stable across every run
MODEL_ID = 'openai/gpt-4o-mini'
ENDPOINTS_URL = f'https://openrouter.ai/api/v1/models/{MODEL_ID}/endpoints'
OUTPUT_FILE = 'provider_endpoints.tsv'

# Retrieve every endpoint available for the selected model
request = Request(
    ENDPOINTS_URL,
    headers={'Authorization': f'Bearer {os.environ["OPENROUTER_API_KEY"]}'},
)
with urlopen(request) as response:
    endpoints = json.load(response)['data']['endpoints']

print(f'Retrieved {len(endpoints)} provider endpoints for {MODEL_ID}.')

# Write comparable routing details for every available endpoint
columns = ['provider', 'input_price', 'output_price', 'context_length', 'quantization']
with open(OUTPUT_FILE, 'w', newline='', encoding='utf-8') as table:
    writer = csv.writer(table, delimiter='\t')
    writer.writerow(columns)
    for endpoint in endpoints:
        writer.writerow(
            [
                endpoint['tag'],
                endpoint['pricing']['prompt'],
                endpoint['pricing']['completion'],
                endpoint['context_length'],
                endpoint['quantization'],
            ]
        )

print(f'Wrote {len(endpoints)} rows to {OUTPUT_FILE}.')
with open(OUTPUT_FILE, encoding='utf-8') as table:
    print(table.read())
Choose two providers

The tag value is the provider slug that OpenRouter accepts for routing. Two distinct slugs give the next comparison two real candidates.

  • Read the provider column in the terminal table.
  • Choose the first provider identifier for your comparison.
  • Record the first selection here: first provider identifier.
  • Choose a different provider identifier for your comparison.
  • Record the second selection here: second provider identifier.
  • Store both selections in the current shell by running these commands:
export PROVIDER_A="[[PROVIDER_A="first provider identifier"]]"
export PROVIDER_B="[[PROVIDER_B="second provider identifier"]]"

What do these exports do?

These exports keep both selections in the current shell. Later requests can read them by name.

The values disappear when this terminal session ends. No project file receives them.

Before you check, do you expect both lines to show different provider identifiers?

  • Print the two stored selections by running this command:
printf 'PROVIDER_A=%s\nPROVIDER_B=%s\n' "$PROVIDER_A" "$PROVIDER_B"

You should see two different identifiers that match rows in provider_endpoints.tsv. That gives your next comparison a fixed pair of available providers.

Selections empty or identical?

  • Replace an empty value with an identifier copied from the provider column.
  • Choose a different table row if both variables contain the same identifier.
  • Run the export commands again after correcting either selection.

Help me store two distinct OpenRouter provider identifiers in PROVIDER_A and PROVIDER_B.

Your provider inventory now shows every available route behind the model ID. Next up, you will send the same prompt through PROVIDER_A and PROVIDER_B to compare their replies.

Compare Two Provider Replies

Your OpenRouter endpoint table shows which companies can serve the same model. You can now test two of those providers under matching conditions.

A shared model ID can still produce different replies through provider routing. You will keep the model plus prompt fixed while changing only the selected provider.

In this step, get ready to:
  • Define one comparison prompt for both providers.
  • Restrict each request with provider.only.
  • Save each raw response plus its assistant reply for a side-by-side comparison.
Fix the comparison inputs

A fair comparison changes one variable at a time. The model plus prompt stay fixed while PROVIDER_A and PROVIDER_B determine where each request goes.

  • Return to provider_probe.py in the editor you used for the endpoint table.
  • Replace the import plus configuration section above the first function with this expanded section:
import csv
import json
import os
import textwrap
import urllib.request
from itertools import zip_longest

# Keep the model and prompt identical across both provider requests.
MODEL_ID = 'openai/gpt-4o-mini'
ENDPOINTS_URL = 'https://openrouter.ai/api/v1/models/{author}/{slug}/endpoints'
CHAT_URL = 'https://openrouter.ai/api/v1/chat/completions'
PROVIDER_A = os.environ['PROVIDER_A']
PROVIDER_B = os.environ['PROVIDER_B']
PROMPT = 'Explain why the sky appears blue in exactly three sentences for a curious 12-year-old.'

Why Keep the Prompt Fixed?

  • The shared PROMPT gives both providers the same task.
  • The two provider variables reuse the identifiers you selected from provider_endpoints.tsv.
  • The API key remains in your shell environment instead of being written into provider_probe.py.
  • Save provider_probe.py.
  • Confirm the endpoint inventory still runs with the expanded configuration by running this command:
python provider_probe.py

What Does This Check?

Python reads both provider identifiers from the current terminal session. The existing endpoint logic should still refresh provider_endpoints.tsv without exposing your API key.

You should see confirmation that provider_endpoints.tsv was saved. This proves the existing inventory logic still works with the comparison settings in place.

Provider Variables Missing?

  • Confirm that PROVIDER_A remains set in the current terminal session.
  • Confirm that PROVIDER_B remains set in the same terminal session.
  • Copy each identifier exactly from provider_endpoints.tsv if either value needs to be set again.

Help me check why my provider environment variables are unavailable to provider_probe.py.

Send the same prompt twice

The only list limits each request to one eligible provider. Saving the complete JSON response preserves the routing evidence while a text file keeps the assistant reply easy to read.

  • Scroll to the bottom of provider_probe.py.
  • Add the paired provider requests plus terminal comparison with this code:
def probe_provider(provider, label):
    # Restrict each request to one provider and preserve its evidence.
    headers = {'Authorization': f"Bearer {os.environ['OPENROUTER_API_KEY']}", 'Content-Type': 'application/json'}
    payload = {'model': MODEL_ID, 'messages': [{'role': 'user', 'content': PROMPT}], 'provider': {'only': [provider]}}
    request = urllib.request.Request(CHAT_URL, data=json.dumps(payload).encode('utf-8'), headers=headers, method='POST')
    with urllib.request.urlopen(request) as response:
        data = json.load(response)
    raw_filename = f'{label}_response.json'
    reply = data['choices'][0]['message']['content']
    reply_filename = f'{label}_reply.txt'

    # Save the full response for evidence and the reply for reading.
    with open(raw_filename, 'w', encoding='utf-8') as raw_file:
        json.dump(data, raw_file, indent=2)
    with open(reply_filename, 'w', encoding='utf-8') as reply_file:
        reply_file.write(reply)
    print(f"Saved {raw_filename} and {reply_filename} from {data['provider']}.")
    return reply

reply_a = probe_provider(PROVIDER_A, 'provider_a')
reply_b = probe_provider(PROVIDER_B, 'provider_b')

# Wrap both replies into readable terminal columns.
width = 54
print(f"\n{PROVIDER_A:<{width}} | {PROVIDER_B}")
for left, right in zip_longest(textwrap.wrap(reply_a, width), textwrap.wrap(reply_b, width), fillvalue=''):
    print(f"{left:<{width}} | {right}")

What Does This Code Do?

  • The request body keeps MODEL_ID plus PROMPT identical for both calls.
  • The only list changes from PROVIDER_A to PROVIDER_B.
  • Each complete response is saved before the assistant text is extracted into its own reply file.
  • The final loop wraps both replies into adjacent terminal columns for comparison.
  • Save provider_probe.py.

This run sends two short paid requests. Your earlier table shows the published token rates for both selected providers.

Before you run the comparison, do you expect two providers serving the same model to return identical wording?

  • Create both response pairs by running this command:
python provider_probe.py

What Should This Run Create?

The first request creates provider_a_response.json plus provider_a_reply.txt. The second creates the matching provider_b_response.json plus provider_b_reply.txt files.

The terminal then places both extracted replies in adjacent columns. That view makes differences in wording plus structure easier to spot.

You should see two save confirmations followed by columns headed with your selected provider identifiers. Each column contains that provider's answer to the shared prompt.

One Provider Request Failed?

  • Check that the failing provider identifier matches its entry in provider_endpoints.tsv exactly.
  • Choose another currently available identifier from the table if the selected provider cannot accept the request.
  • Run provider_probe.py again from the terminal session that contains your API key plus provider variables.

Help me diagnose why one provider-only OpenRouter request failed while the other succeeded.

✔️ Awesome, I've got everything!

Great work. Save provider_probe.py so your provider comparison remains reproducible.

ⓧ I'd like to double check the full code

import csv
import json
import os
import textwrap
import urllib.request
from itertools import zip_longest

# Keep the model and prompt identical across both provider requests.
MODEL_ID = 'openai/gpt-4o-mini'
ENDPOINTS_URL = 'https://openrouter.ai/api/v1/models/{author}/{slug}/endpoints'
CHAT_URL = 'https://openrouter.ai/api/v1/chat/completions'
PROVIDER_A = os.environ['PROVIDER_A']
PROVIDER_B = os.environ['PROVIDER_B']
PROMPT = 'Explain why the sky appears blue in exactly three sentences for a curious 12-year-old.'

def api_request(url, payload=None):
    # Use the shell key for authenticated GET and POST requests.
    headers = {'Authorization': f"Bearer {os.environ['OPENROUTER_API_KEY']}"}
    body = None
    method = 'GET'
    if payload is not None:
        headers['Content-Type'] = 'application/json'
        body = json.dumps(payload).encode('utf-8')
        method = 'POST'
    request = urllib.request.Request(url, data=body, headers=headers, method=method)
    with urllib.request.urlopen(request) as response:
        return json.load(response)

def write_provider_table():
    # Refresh the endpoint inventory used to choose provider identifiers.
    author, slug = MODEL_ID.split('/', 1)
    endpoint_url = ENDPOINTS_URL.format(author=author, slug=slug)
    endpoint_data = api_request(endpoint_url)
    columns = ['provider_identifier', 'input_price', 'output_price', 'context_length', 'quantization']
    with open('provider_endpoints.tsv', 'w', encoding='utf-8', newline='') as table_file:
        writer = csv.writer(table_file, delimiter='\t')
        writer.writerow(columns)
        for endpoint in endpoint_data['data']['endpoints']:
            pricing = endpoint['pricing']
            writer.writerow([
                endpoint.get('tag', ''),
                pricing.get('prompt', ''),
                pricing.get('completion', ''),
                endpoint.get('context_length', ''),
                endpoint.get('quantization', ''),
            ])
    print('Saved provider_endpoints.tsv.')

write_provider_table()

def probe_provider(provider, label):
    # Restrict each request to one provider and preserve its evidence.
    headers = {'Authorization': f"Bearer {os.environ['OPENROUTER_API_KEY']}", 'Content-Type': 'application/json'}
    payload = {'model': MODEL_ID, 'messages': [{'role': 'user', 'content': PROMPT}], 'provider': {'only': [provider]}}
    request = urllib.request.Request(CHAT_URL, data=json.dumps(payload).encode('utf-8'), headers=headers, method='POST')
    with urllib.request.urlopen(request) as response:
        data = json.load(response)
    raw_filename = f'{label}_response.json'
    reply = data['choices'][0]['message']['content']
    reply_filename = f'{label}_reply.txt'

    # Save the full response for evidence and the reply for reading.
    with open(raw_filename, 'w', encoding='utf-8') as raw_file:
        json.dump(data, raw_file, indent=2)
    with open(reply_filename, 'w', encoding='utf-8') as reply_file:
        reply_file.write(reply)
    print(f"Saved {raw_filename} and {reply_filename} from {data['provider']}.")
    return reply

reply_a = probe_provider(PROVIDER_A, 'provider_a')
reply_b = probe_provider(PROVIDER_B, 'provider_b')

# Wrap both replies into readable terminal columns.
width = 54
print(f"\n{PROVIDER_A:<{width}} | {PROVIDER_B}")
for left, right in zip_longest(textwrap.wrap(reply_a, width), textwrap.wrap(reply_b, width), fillvalue=''):
    print(f"{left:<{width}} | {right}")
Compare the saved replies

Provider differences become useful evidence when you can inspect the exact outputs. The separate reply files let you compare them without digging through response metadata.

Before you look closely, do you expect the two providers to use the same examples plus sentence structure?

  • Return to the editor you used earlier.
  • Open provider_a_reply.txt from the file sidebar.
  • Place provider_b_reply.txt beside the first reply with your editor's split-view control.
  • Compare the wording in both replies.
  • Compare how closely each reply follows the three-sentence constraint.

You should see two answers to the same prompt in separate files. The terminal headings plus raw response files show which selected provider produced each answer.

You now have direct evidence of how two providers handle the same model plus prompt. Next, you will measure the latency plus token usage behind each reply.

Measure Latency and Token Cost

Your two provider-only requests now give you replies you can compare side by side. Text quality is only part of the routing decision.

In this step, you'll time both OpenRouter requests. You'll combine returned token counts with endpoint prices to choose one provider using evidence from your own calls.

In this step, get ready to:
  • Measure the elapsed wall-clock latency for both provider-only requests.
  • Calculate an estimated request cost from token usage plus endpoint pricing.
  • Select one provider identifier as CHOSEN_PROVIDER.
Add timing and usage measurements

Wall-clock latency measures the full wait from sending a request to receiving its complete response. Both calls must keep the same model plus the same prompt for the comparison to remain meaningful.

How the measurements work

Start the timer immediately before each provider-only request. Stop it after the complete response returns.

The response's usage object supplies usage.prompt_tokens, usage.completion_tokens, plus usage.total_tokens.

  • In provider_probe.py, locate the code path that sends each provider-only request.
  • Update that request path with a result record containing the four measurements described above.
  • Save provider_probe.py.

Before you rerun the script, which request do you expect to finish faster?

  • Use the terminal's command history to rerun the command that executes provider_probe.py.

You should see two successful provider-only calls. Each provider should have a positive elapsed latency plus values for prompt, completion, total tokens.

Missing latency or token values?

  • Confirm the timer starts immediately before each request.
  • Confirm the timer stops after the complete response returns.
  • Read the usage values from each provider-specific response.

Help me debug the missing measurements in my provider comparison.

Calculate each request cost

The endpoint inventory already contains the prompt price plus completion price for each provider. Matching those prices to the measured token counts produces an estimated cost for each request.

How the estimate works

Each endpoint's prompt price represents USD per input token. Multiply that value by the request's prompt token count.

The completion price represents USD per output token. Add the completion price multiplied by the completion token count to get the estimated request cost.

Each report row identifies the returned provider value. It also records elapsed latency plus all three token counts.

The same row carries both endpoint price inputs. It ends with the estimated request cost.

  • Add comparison-report output to provider_probe.py so it creates the complete provider_comparison.tsv report described above.
  • Save provider_probe.py.

Before you run the updated script, do you expect the provider with lower latency to have the lower estimated cost?

  • Use the terminal's command history to rerun the command that executes provider_probe.py.

You should see both provider-only requests complete successfully. The new provider_comparison.tsv file should contain one measured row for each provider.

Report missing a provider row?

  • Match the returned provider value to the corresponding row in provider_endpoints.tsv.
  • Confirm both provider-specific responses completed before writing provider_comparison.tsv.
  • Keep numeric prices as numbers during the estimated-cost calculation.

Help me find why provider_comparison.tsv is missing data.

Choose your provider

Your measurements turn provider selection into an explicit policy. The strongest choice is the provider whose observed trade-offs fit your app.

  • Use the same file viewer from earlier to open provider_comparison.tsv.
  • Review the two rows for the trade-off that matters most to your app.
  • Record the exact provider identifier you selected here: your selected provider identifier.
  • In provider_probe.py, locate the constants for PROVIDER_A plus PROVIDER_B.
  • Set CHOSEN_PROVIDER to your selected provider identifier.
  • Save provider_probe.py.
  • Use your editor's search feature to confirm CHOSEN_PROVIDER contains the exact selected provider identifier.

✔️ Awesome, I've got everything!

Your script now measures both providers. Your report captures the evidence behind your selected provider.

ⓧ I'd like to double check the full code

  • Confirm provider_probe.py still uses openai/gpt-4o-mini with the same shared prompt.
  • Confirm both provider-only requests record elapsed wall-clock latency.
  • Confirm both responses provide prompt, completion, total token counts.
  • Confirm provider_comparison.tsv contains both provider identifiers plus their measured results.
  • Confirm each report row includes both endpoint price inputs plus calculated estimated request cost.
  • Confirm CHOSEN_PROVIDER equals the exact provider identifier you selected.

Before you rerun the script, which provider do you expect to have lower latency this time?

  • Return to the same terminal session.
  • Use the terminal's command history to rerun the command that executes provider_probe.py.

You should see two successful provider-only measurements. The refreshed provider_comparison.tsv should show provider identity, elapsed latency, three token counts, both endpoint prices, estimated request cost for each row.

That is a real routing decision. Your provider choice now rests on measurements from your own requests.

Your comparison is ready. Next up, you'll turn CHOSEN_PROVIDER into an enforced routing rule.

Pin and Test Your Provider

Your comparison report has turned two provider choices into measured evidence. You have selected CHOSEN_PROVIDER from that evidence.

A measurement only changes application behaviour when the request carries a routing rule. In this step, you will pin OpenRouter to that provider.

In this step, get ready to:
  • Add a pinned request for CHOSEN_PROVIDER.
  • Confirm OpenRouter returns the selected provider.
  • Capture the response from a request with no eligible endpoint.
Send the pinned request

The provider.only option is an allowlist inside the request's provider object. A list containing only CHOSEN_PROVIDER makes your measured choice part of the application call.

  • Use your existing code editor to edit provider_probe.py.
  • Reuse the existing provider-only request path for a dedicated pinned request.
  • Set provider.only to a list containing CHOSEN_PROVIDER.
  • Keep MODEL_ID set to openai/gpt-4o-mini for the pinned request.
  • Reuse the existing shared comparison prompt for the pinned request.
  • Write the raw successful response to pinned_response.json.
  • Print a comparison line showing CHOSEN_PROVIDER beside the returned provider value.
  • Save provider_probe.py.

Before you rerun the script, do you expect the returned provider to match CHOSEN_PROVIDER?

  • Rerun provider_probe.py with the same Python command from the previous step.

You will see the chosen provider match the returned provider. The script also saves the raw response in pinned_response.json.

Solid work. Your application request now lands on the provider you selected from your own measurements.

Returned provider does not match?

  • Check that provider.only receives a list containing only CHOSEN_PROVIDER.
  • Confirm CHOSEN_PROVIDER exactly matches its identifier in provider_endpoints.tsv.

Help me diagnose why my pinned OpenRouter request returned a different provider.

Check the stored evidence

Terminal output confirms the result during the run. The saved response preserves that proof after the terminal scrolls.

  • Use the same code editor to inspect pinned_response.json.
  • Find the top-level provider field.
  • Compare its value with CHOSEN_PROVIDER.
  • Keep pinned_response.json in its current location.

The matching value proves the allowlist was honoured for this request.

Why Save the Raw Response?

A raw JSON response preserves the returned provider identity with the model output. It gives you evidence that does not depend on a summary printed by your script.

Together with provider_comparison.tsv, this file connects your measured decision to the route your application used.

Exclude every available provider

The negative test uses provider.ignore to turn your complete endpoint inventory into a routing boundary. This checks how OpenRouter responds when the request rules leave the router with zero permitted choices.

  • Use your existing code editor to edit provider_probe.py.
  • Build an exclusion list from every provider identifier already loaded by the script.
  • Add a negative-test request by reusing the existing chat request path.
  • Set provider.ignore to the complete exclusion list.
  • Capture the non-success response body before the script reads success-only fields.
  • Write the captured response to excluded_providers_error.json.
  • Print an outcome line confirming that the failure response was saved.
  • Save provider_probe.py.

Before you rerun the script, do you think OpenRouter will ignore the exclusion list or refuse the request?

  • Rerun provider_probe.py with the same Python command from the previous step.

The pinned request succeeds first. The negative request then returns an OpenRouter error response.

Your script stores that response in excluded_providers_error.json. This is the intended result of the negative test.

Why This Failure Matters

Pinning trades some availability for predictable behaviour. A strict provider rule gives OpenRouter fewer fallback routes.

The saved error proves the router respected your restrictions when no endpoint qualified. Your application can now handle that condition deliberately.

Negative test did not save the response?

  • Check that the exclusion list contains every provider identifier represented in provider_endpoints.tsv.
  • Capture the response body before accessing assistant message content or usage fields.
  • Confirm the script writes the error body to excluded_providers_error.json.

Help me capture an expected OpenRouter error response without stopping my Python script.

Your final evidence set connects the available routes to the provider you measured. It also records both sides of the routing policy.

  • Confirm provider_endpoints.tsv remains listed in your code editor.
  • Confirm provider_comparison.tsv remains listed in your code editor.
  • Confirm pinned_response.json contains the successful pinned response.
  • Confirm excluded_providers_error.json contains the captured OpenRouter error response.

That closes the loop. Your chosen provider now succeeds under a strict pin.

The captured failure proves OpenRouter cannot honour a request when every available endpoint is excluded.

Secret mission

Create a Provider Routing Policy

Turn your measured comparison into a provider routing policy that explains why CHOSEN_PROVIDER stays pinned. Define the evidence that triggers a review. Decide how the application should respond when the pin cannot be honoured.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the comparison evidence, pause your work, or remove the local artifacts. Keeping the files does not send more requests to OpenRouter, so they create no ongoing API usage.

Resources you used:

  • The provider_probe.py comparison script.
  • The endpoint inventory in provider_endpoints.tsv.
  • The measured comparison report in provider_comparison.tsv.
  • The baseline evidence in baseline_response.json.
  • The provider comparison evidence in provider_a_response.json, provider_a_reply.txt, provider_b_response.json, and provider_b_reply.txt.
  • The pinning evidence in pinned_response.json and excluded_providers_error.json.
  • The session-only OPENROUTER_API_KEY environment value in the current terminal.

Keep everything running

No deletion is needed for the local evidence. Choose this if you want to rerun the comparison or preserve the measurements behind your provider choice.

  • Keep provider_probe.py with its retained evidence files.
  • Review your shell history for any entry containing the full API key.
  • Delete every history entry that contains the full API key.
  • Rotate your OpenRouter API key in the dashboard if you pasted it anywhere shared.
  • Close the current terminal window to clear OPENROUTER_API_KEY from that session.

Pause - I'll come back to this later

Pausing ends the active terminal session while preserving your script and measurements. You can return to the retained evidence when you are ready to compare providers again.

  • Review your shell history for any entry containing the full API key.
  • Delete every history entry that contains the full API key.
  • Rotate your OpenRouter API key in the dashboard if you pasted it anywhere shared.
  • Close the current terminal window to clear OPENROUTER_API_KEY from that session.
  • Keep provider_probe.py with all ten evidence files for your next session.

Delete - I don't want to use this again

The deletion only removes the listed local files. Your OpenRouter account remains available after those files are gone.

  • Review your shell history for any entry containing the full API key.
  • Delete every history entry that contains the full API key.
  • Rotate your OpenRouter API key in the dashboard if you pasted it anywhere shared.
  • Close the current terminal window to clear OPENROUTER_API_KEY from that session.
  • Open the folder containing provider_probe.py in Finder on macOS or File Explorer on Windows.
  • Select provider_probe.py, baseline_response.json, excluded_providers_error.json, pinned_response.json, provider_a_reply.txt, provider_a_response.json, provider_b_reply.txt, provider_b_response.json, provider_comparison.tsv, and provider_endpoints.tsv.
  • Delete the selected files.

You should no longer see the script or evidence files in that folder. The API key is no longer present in the closed terminal session.

Nice Work!

Nice Work!

You did it! You built a provider probe for OpenRouter that reveals which company serves each model request. Your saved evidence now explains why your application pins its chosen provider.

What you learned:

  • Created a provider endpoint inventory in provider_endpoints.tsv. It shows input pricing, output pricing, context length, and quantization for openai/gpt-4o-mini.
  • Ran a provider comparison with one shared prompt. You saved both replies in separate files. You measured latency, token usage, and estimated request cost in provider_comparison.tsv.
  • Enforced a provider pin with provider.only restricted to CHOSEN_PROVIDER. You confirmed the returned provider. You captured the failure produced when no permitted provider was available.
  • Secret Mission: Took on an optional challenge to push your provider-routing skills further.

Ready to quiz yourself?