Compare Web Search Engines

Use one AI model to compare web search answers, citations, and costs.

Introduction

30 Second Summary

A search result can look convincing while hiding very different sources behind the answer. Without a side-by-side test, it is hard to know which search engine gives you the strongest evidence for the money.

In this project, you will use OpenRouter to send the same questions through Exa, Parallel, and Perplexity while keeping one AI model fixed. Your script will save every answer with its citations and cost in a CSV comparison.

What You'll Build

Picture opening your finished CSV to see how three search engines answered the same five questions through one AI model, complete with each engine's citations and cost.

By the end of this project, you'll have:

A repeatable engine comparison that runs the same question through Exa, Parallel, and Perplexity while one AI model stays fixed.
Side-by-side answers with citations that let you compare which sources each engine chose.
A five-question cost comparison CSV that helps you judge source usefulness against the price of each call.
An optional Secret Mission to push your skills further.

Are there any prerequisites?

You need an OpenRouter account with a small credit balance. Basic Python or command-line familiarity helps, but every project action is guided.

Before We Start

A fair search-engine comparison starts with a controlled environment. Every engine needs the same local tools.

In this step, you will create an account with OpenRouter.

You will also prepare Python plus Visual Studio Code in a local project folder.

In this step, get ready to:

Get signed in to your OpenRouter account.
Verify Python version 3 plus Visual Studio Code.
Create the local project folder inside Visual Studio Code.

Create your OpenRouter account

OpenRouter provides the account that later records your model requests. For now, the goal is simply to reach the website while signed in.

I need a new account

Visit the official OpenRouter sign-up page.
Complete the registration flow using one of the methods shown.
Return to the OpenRouter website after registration.

I already have an account

Visit the official OpenRouter sign-in page.
Complete the sign-in flow using your existing account.
Return to the OpenRouter website after signing in.

Confirm that your signed-in account controls are visible in the page header.

Your account access is ready. No API request has been sent.

Can't Complete Sign-In?

Retry the sign-in flow in a private browser window if the page keeps returning you to the account screen.

Ask for help with the exact stage where the flow stops: Help me troubleshoot my OpenRouter sign-in flow.

Install Python and Visual Studio Code

Python runs the comparison script later in the project. A version check tells you whether the runtime is already available.

Press Cmd+Space on macOS or the Windows key on Windows to open system search.
Type Terminal on macOS or PowerShell on Windows.
Press Enter to open the terminal.

Before you run the check, do you expect Python to be available already?

macOS

Check the installed Python version by running this command:

python3 --version

What Does This Command Do?

The command asks the Python interpreter to print its version. It exits without running a script.

A working installation prints the word Python followed by a version beginning with 3.

Windows

Check the installed Python version by running this command:

python --version

What Does This Command Do?

The command asks the default Python interpreter to print its version. It exits without running a script.

A working installation prints the word Python followed by a version beginning with 3.

✔️ I see Python 3

Good, the runtime can now execute the scripts you build in this project.

ⓧ The Command Is Not Found

Install Python using the path for your computer.

macOS

Visit the official Python downloads page for macOS.
Download the current supported Python installer.
Open the downloaded .pkg file.
Complete the default installation flow.
Approve the macOS permission request.
Open the Python folder inside Applications.
Run Install Certificates.command to finish the installation.
Repeat the macOS version check above.

You should now see Python followed by a version beginning with 3.

Windows

Visit the official Python downloads page for Windows.
Download Python Install Manager.
Open the downloaded installer.
Select Install.
Approve the installation prompt if it appears.
Return to PowerShell from earlier.
Repeat the Windows version check above.

You should now see Python followed by a version beginning with 3.

Visual Studio Code gives your project folder one workspace for scripts plus generated comparison files.

Press Cmd+Space on macOS or the Windows key on Windows to open system search.
Type Visual Studio Code.
Press Enter.

✔️ Visual Studio Code Opens

Your editor is ready for the project folder.

ⓧ Visual Studio Code Is Not Installed

Install Visual Studio Code using the path for your computer.

macOS

Visit the official Visual Studio Code download page.
Download the macOS .dmg installer.
Open the downloaded .dmg file.
Drag Visual Studio Code.app to the Applications folder.
Repeat the Visual Studio Code search above.

Visual Studio Code should now open.

Windows

Visit the official Visual Studio Code download page.
Download the User Setup .exe installer.
Open the downloaded installer.
Accept the default options on each installer screen.
Repeat the Visual Studio Code search above.

Visual Studio Code should now open.

Still Missing a Tool?

Close the terminal before repeating the Python check after installation. Restart your computer if the newly installed application still does not appear in system search.

Share your operating system plus the result you see: Help me finish my Python or Visual Studio Code installation.

Create the local project folder

A dedicated folder keeps every answer plus the final CSV together. Opening that folder as a workspace also gives later terminal commands the correct location.

Select File in Visual Studio Code.
Select Open Folder....
Select your Desktop in the folder dialog.
Select New Folder.
Enter openrouter-search-engine-comparison as the folder name.
Select the new openrouter-search-engine-comparison folder.
Click Open on macOS or Select Folder on Windows.
Select Yes, I trust the authors if the Workspace Trust dialog appears.

Before you check, where do you expect the project name to appear in Visual Studio Code?

Check the Explorer sidebar in Visual Studio Code.

You should see openrouter-search-engine-comparison as the open folder. The workspace is empty because no project files exist yet.

Don't See the Folder Name?

Repeat the folder-opening flow if Visual Studio Code still shows its welcome screen. Make sure you select the folder itself before confirming the dialog.

Ask for help if a different folder opens: Help me open the correct project folder in Visual Studio Code.

Your account plus local workspace are ready for a controlled comparison. Next, you will create an API key and prove it works with one plain request.

Create and Test Your API Key

Your OpenRouter account is ready, but the comparison script still has no safe way to authenticate an API request.

First, you will add a small credit balance to cover the requests in this project. You will create a spending-limited key after the balance is available.

You will then store the key outside your code. A plain request through openai/gpt-4o-mini will prove that the key works.

In this step, get ready to:

Add a small credit balance to OpenRouter.
Store a newly created key in OPENROUTER_API_KEY.
Send a plain request through the fixed model.

Add a small credit balance

Credits are deposits that OpenRouter deducts from as you use models. The comparison requests need a positive balance before they can run.

Keep the balance small

This purchase uses a real payment method. The API usage for this project is expected to stay below $1.

OpenRouter charges a 5.5% fee when you purchase credits. The minimum purchase fee is $0.80.

Return to the OpenRouter Credits page from your signed-in session.
Click Add Credits.
Enter the smallest credit amount the checkout accepts.
Complete the payment form.
Confirm that the Credits page shows a positive balance.

Your account can now pay for the model calls in this project. That small balance is the fuel for every comparison you run.

Create and store your API key

An API key proves that each request belongs to your account. A credit limit caps how much that individual key can spend.

The one-time key reveal is easy to rush. OpenRouter cannot display the plaintext key again after you leave it.

Switch to the OpenRouter API Keys page.
Click Create Key.
Enter search-engine-comparison in the Name field.
Enter 1 in the Credit limit field.
Confirm the key creation in the dialog.
Copy the revealed key immediately.
Use your editor's terminal menu to open an integrated terminal for the openrouter-search-engine-comparison folder.

Choose your operating system below to place the copied secret in OPENROUTER_API_KEY.

macOS

Replace sk-or-v1-YOUR-API-KEY in the command with the key you copied.
Store the key for this terminal session by running this command:

export OPENROUTER_API_KEY="sk-or-v1-YOUR-API-KEY"

What does this command do?

The command stores the secret in the shell's current process environment. Your Python script can read it without placing the key in compare_search_engines.py.

Keep this terminal session open for the request test later in this step.

You should return to the terminal prompt without seeing the secret printed.

Key not stored?

Check that the terminal uses a macOS shell.
Replace the full placeholder with the copied key before running the command again.

Need another pair of eyes? Help me store my OpenRouter key in a macOS environment variable.

Windows

Use a PowerShell terminal in your editor.
Replace sk-or-v1-YOUR-API-KEY in the command with the key you copied.
Store the key for this terminal session by running this command:

$env:OPENROUTER_API_KEY="sk-or-v1-YOUR-API-KEY"

What does this command do?

The command stores the secret in the current PowerShell process environment. Your Python script can read it without placing the key in compare_search_engines.py.

Keep this PowerShell session open for the request test later in this step.

You should return to the PowerShell prompt without seeing the secret printed.

Key not stored?

Check that the editor terminal is using PowerShell.
Replace the full placeholder with the copied key before running the command again.

Need another pair of eyes? Help me store my OpenRouter key in a PowerShell environment variable.

Return to the one-time key view in OpenRouter.
Close the one-time key view.
Confirm that the API Keys page shows search-engine-comparison with a masked key value.

Build and run the plain request

Python's standard library can send the first request without another package. The script will build a plain message before sending it to OpenRouter's chat completions endpoint.

Create compare_search_engines.py inside the openrouter-search-engine-comparison folder by using your editor's file sidebar.
Add the plain request by pasting this code into compare_search_engines.py:

import json
import os
import urllib.request

# Read the secret from the terminal environment.
api_key = os.environ["OPENROUTER_API_KEY"]

# Build the plain message for the fixed model.
payload = {
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "Say 'Hello from OpenRouter!' and nothing else."}],
}

# Authenticate the request without exposing the key in source code.
request = urllib.request.Request(
    "https://openrouter.ai/api/v1/chat/completions",
    data=json.dumps(payload).encode("utf-8"),
    headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
)

# Send the request and decode the JSON response.
with urllib.request.urlopen(request) as response:
    data = json.load(response)

# Print the assistant's answer.
print(data["choices"][0]["message"]["content"])

What does this code do?

The api_key variable reads the secret from OPENROUTER_API_KEY.
The payload variable holds the fixed model plus the plain user message.
The JSON conversion prepares the payload for the endpoint.
The Authorization header authenticates the request with the key.
The final line extracts the assistant's answer from the response.

Save compare_search_engines.py.

✔️ Awesome, I've got everything!

Your saved file contains the complete plain-request script.

ⓧ I'd like to double check the full code

import json
import os
import urllib.request

# Read the secret from the terminal environment.
api_key = os.environ["OPENROUTER_API_KEY"]

# Build the plain message for the fixed model.
payload = {
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "Say 'Hello from OpenRouter!' and nothing else."}],
}

# Authenticate the request without exposing the key in source code.
request = urllib.request.Request(
    "https://openrouter.ai/api/v1/chat/completions",
    data=json.dumps(payload).encode("utf-8"),
    headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
)

# Send the request and decode the JSON response.
with urllib.request.urlopen(request) as response:
    data = json.load(response)

# Print the assistant's answer.
print(data["choices"][0]["message"]["content"])

Before you run the script, do you expect the model to return the requested sentence or an authentication failure?

Send the plain request by running this command in the same terminal that holds your key:

python compare_search_engines.py

You should see Hello from OpenRouter! in the terminal.

That is your first authenticated OpenRouter response. Your fixed model can now answer requests from the comparison script.

Request did not return the answer?

Confirm that you ran the script in the same terminal where you stored OPENROUTER_API_KEY.
Confirm that the OpenRouter Credits page still shows a positive balance.
Compare compare_search_engines.py with the full-code tab above.

Need help tracing the failure? Help me debug my plain OpenRouter request.

Your fixed-model baseline is working. Next, you will turn on web search and inspect the citations returned with an answer.

Test Default Web Search

Your plain request proved that your OpenRouter API key can reach the fixed model openai/gpt-4o-mini. That response gives you a baseline without live sources.

Technology news changes throughout the week. Web search gives the model current evidence. It also returns links you can inspect.

In this step, get ready to:

Enable web search without pinning an engine.
Send a weekly technology news question through the fixed model.
Display the returned answer with its source links.

Enable default web search

A default-engine request leaves the engine setting out of the request. OpenRouter chooses the search route while your model stays fixed.

In compare_search_engines.py from earlier, keep the imports plus the API key block at the top.
Replace the existing question plus request block with this code:

# Keep the model fixed while asking a current-events question
question = "What are the biggest technology news stories this week?"

# Let OpenRouter choose the default search engine
response = requests.post(
    "https://openrouter.ai/api/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    },
    json={
        "model": "openai/gpt-4o-mini",
        "messages": [{"role": "user", "content": question}],
        "tools": [{"type": "openrouter:web_search"}],
    },
)
response.raise_for_status()

What does this code do?

The question variable holds the exact current-events prompt used throughout the comparison.
The tools array enables openrouter:web_search for this request.
The request omits an engine setting. This creates the default-engine baseline.
The response.raise_for_status() call stops the script when the API does not accept the request.

Save compare_search_engines.py.

Before you rerun the script, do you expect the answer to rely only on the model or include links to current reporting?

Rerun the script with the same terminal command from the previous step.

You should see an answer about this week's technology news. The answer should contain source links or citation markers.

Good progress. The same fixed model can now ground its answer in current web results.

Request stops before printing an answer?

Confirm that your OpenRouter account still has credits available.
Check that OPENROUTER_API_KEY is available through the same private method that worked earlier.
Compare the tools array with the code block above. The tool type must be openrouter:web_search.

Help me diagnose my default OpenRouter web search request.

Print the returned sources

The answer text shows what the model concluded. Citation annotations give you the source titles plus URLs needed for the engine comparison.

Replace the response-handling lines after response.raise_for_status() with this code:

# Extract the assistant message and its citation annotations
data = response.json()
message = data["choices"][0]["message"]
answer = message["content"]
annotations = message.get("annotations", [])

# Print the answer first so it stays readable
print("\nAnswer:\n")
print(answer)

# Print each returned source as a separate line
print("\nSources:")
if annotations:
    for annotation in annotations:
        citation = annotation.get("url_citation", annotation)
        url = citation.get("url")
        title = citation.get("title") or "Source"
        if url:
            print(f"- {title}: {url}")
else:
    print("- Check the answer above for inline source links.")

How are the sources extracted?

The message variable holds the assistant's answer plus its metadata.
The annotations list holds citation records returned with the message.
Each url_citation record can provide a source title plus its URL.
The fallback message points you to inline links when separate annotations are unavailable.

Save compare_search_engines.py.

Before you run the final check, how many separate source links do you expect beneath the answer?

Rerun the script with the same terminal command from the previous step.

You should see an Answer: section containing the current-events response. Below it, the Sources: section should show returned URLs or direct you to inline links in the answer.

Sources section looks empty?

Confirm that the question still asks about this week's technology news.
Check that the request still contains the openrouter:web_search tool.
Inspect the answer for clickable Markdown links when no separate annotations appear.

Help me inspect the citations returned by OpenRouter.

✔️ Awesome, I've got everything!

Your script now sends the weekly technology news question through default web search. It prints the answer plus the returned source links.

ⓧ I'd like to double check the full code

# Import the libraries used for configuration and API requests
import os
import requests
from dotenv import load_dotenv

# Load private values from a local .env file when present
load_dotenv()

# Read the OpenRouter key without placing it in the script
api_key = os.getenv("OPENROUTER_API_KEY")
if not api_key:
    raise ValueError("OPENROUTER_API_KEY is not set.")

# Keep the model fixed while asking a current-events question
question = "What are the biggest technology news stories this week?"

# Let OpenRouter choose the default search engine
response = requests.post(
    "https://openrouter.ai/api/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    },
    json={
        "model": "openai/gpt-4o-mini",
        "messages": [{"role": "user", "content": question}],
        "tools": [{"type": "openrouter:web_search"}],
    },
)
response.raise_for_status()

# Extract the assistant message and its citation annotations
data = response.json()
message = data["choices"][0]["message"]
answer = message["content"]
annotations = message.get("annotations", [])

# Print the answer first so it stays readable
print("\nAnswer:\n")
print(answer)

# Print each returned source as a separate line
print("\nSources:")
if annotations:
    for annotation in annotations:
        citation = annotation.get("url_citation", annotation)
        url = citation.get("url")
        title = citation.get("title") or "Source"
        if url:
            print(f"- {title}: {url}")
else:
    print("- Check the answer above for inline source links.")

Your unpinned search baseline is ready. Next, you will pin individual engines so their answers can be compared against it.

Compare the Three Search Engines

Your default OpenRouter web search proved that the script can retrieve current results. Its answer plus source links give you a baseline for this comparison.

Now you will hold the model plus question fixed. You will pin Exa, Parallel, then Perplexity to isolate the effect of the search engine. Each response becomes a structured record containing its answer text plus citation URLs.

In this step, get ready to:

Prepare a fixed comparison configuration for all three search engines.
Run the identical news question through Exa, Parallel, then Perplexity.
Store each answer with its citation URLs in a structured result record.

Prepare a fair comparison

A fair experiment changes one input at a time. This setup preserves the fixed model openai/gpt-4o-mini plus the fixed question from your default search.

In compare_search_engines.py, replace the current contents with this reusable comparison setup:

import json
import os
import urllib.request
from pathlib import Path

# Load a private .env file when the key is not already in the environment.
if "OPENROUTER_API_KEY" not in os.environ and Path(".env").exists():
    for line in Path(".env").read_text(encoding="utf-8").splitlines():
        if line.startswith("OPENROUTER_API_KEY="):
            value = line.split("=", 1)[1].strip().strip("\"'")
            os.environ["OPENROUTER_API_KEY"] = value

# Keep every comparison input fixed except the search engine.
API_URL = "https://openrouter.ai/api/v1/chat/completions"
API_KEY = os.environ["OPENROUTER_API_KEY"]
MODEL = "openai/gpt-4o-mini"
QUESTION = "What are the biggest technology news stories this week?"
ENGINES = ["exa", "parallel", "perplexity"]
results = []

print(f"Ready to compare: {', '.join(ENGINES)}")

What does this setup do?

The key-loading section keeps using your existing OPENROUTER_API_KEY environment variable. It can also read that value from your private .env file.
The MODEL plus QUESTION constants keep the comparison inputs consistent.
The ENGINES list fixes the comparison order. The results list collects the structured records.

Save compare_search_engines.py.
Use your code editor's Python run control to run compare_search_engines.py.

You should see Ready to compare: exa, parallel, perplexity in the output.

Your experiment now has one fixed model plus one fixed question. The engine is the only comparison input that changes.

Do you see a key-loading error?

Confirm that the environment variable is named exactly OPENROUTER_API_KEY. If you use .env, keep that file inside openrouter-search-engine-comparison.

Check that the line starts with OPENROUTER_API_KEY= without extra spaces before the name.

Help me diagnose why my Python script cannot load OPENROUTER_API_KEY from the environment or a private .env file.

Pin the first search to Exa

The web search server tool accepts an engine parameter. Setting it to exa makes the first record an explicit Exa result.

Add this code below the existing print line in compare_search_engines.py to send one pinned request:

# Send one grounded request and normalize its answer plus citation URLs.
def run_search(engine):
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": QUESTION}],
        "tools": [{
            "type": "openrouter:web_search",
            "parameters": {"engine": engine, "max_uses": 1},
        }],
    }
    request = urllib.request.Request(
        API_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request) as response:
        data = json.load(response)
    message = data["choices"][0]["message"]
    citations = [
        item["url_citation"]["url"]
        for item in message.get("annotations", [])
        if item.get("type") == "url_citation"
    ]
    return {"engine": engine, "answer": message["content"], "citations": citations}

# Capture Exa first so its record can be inspected before the full comparison.
results.append(run_search(ENGINES[0]))
print(json.dumps(results[0], indent=2))

How does the pinned request work?

The run_search() function builds the same request for every engine.
The openrouter:web_search tool receives the current engine value. The max_uses value caps the request at one search.
The citation extraction reads every returned url_citation annotation. It stores each source URL in the record's citations list.
The returned dictionary gives every engine the same record shape. That consistency makes the final comparison easier to inspect.

Save compare_search_engines.py.

Before you run it, do you expect the Exa citation list to match your earlier default-engine list exactly?

Use your code editor's Python run control to run compare_search_engines.py.

You should see one JSON record with engine set to exa. The record also contains answer text plus a citation URL list.

That first pinned result proves the engine choice is reaching OpenRouter. You now have a consistent record format for the full comparison.

Missing the Exa record or citations?

Confirm that your OpenRouter credit balance still covers another request. Check that your API key remains active without printing its value.

Compare the tools object with the code above. The tool type must be openrouter:web_search.

Help me troubleshoot why my pinned Exa request does not return an answer with url_citation annotations.

Run Parallel and Perplexity side by side

The Exa record establishes the comparison format. The final loop reuses that exact request for the remaining engines before printing all three records together.

Add this code below the Exa result output in compare_search_engines.py to complete the comparison:

# Run the identical request through the two remaining engines.
for engine in ENGINES[1:]:
    result = run_search(engine)
    results.append(result)
    print(json.dumps(result, indent=2))

# Print one combined structure for direct side-by-side inspection.
print("\n=== SIDE-BY-SIDE RESULTS ===")
print(json.dumps(results, indent=2))

What does the comparison loop preserve?

The slice ENGINES[1:] selects Parallel plus Perplexity. Exa already ran immediately above it.
Every loop iteration calls run_search() with the same model plus question.
Each returned record enters the same results list. The final output places all three engine records in one structure.

Save compare_search_engines.py.

Before you run the final comparison, do you expect all three engines to return the same answer sources?

Use your code editor's Python run control to run compare_search_engines.py.

You should see individual records for exa, parallel, then perplexity. The final array should contain three records with an answer plus citation URLs for each engine.

You have now isolated the search engine as the changing input. The three structured records make differences in answers plus sources visible.

Did one engine fail to produce a record?

Check the engine values against exa, parallel, plus perplexity. A spelling change prevents OpenRouter from selecting the intended provider.

Run the script again after confirming your OpenRouter key remains active. A stopped request prevents the later records from entering results.

Help me diagnose why one pinned OpenRouter search engine fails while the other engine records complete.

✔️ Awesome, I've got everything!

Your script now runs one fixed question through all three pinned engines. Double-check that you saved compare_search_engines.py.

ⓧ I'd like to double check the full code

import json
import os
import urllib.request
from pathlib import Path

# Load a private .env file when the key is not already in the environment.
if "OPENROUTER_API_KEY" not in os.environ and Path(".env").exists():
    for line in Path(".env").read_text(encoding="utf-8").splitlines():
        if line.startswith("OPENROUTER_API_KEY="):
            value = line.split("=", 1)[1].strip().strip("\"'")
            os.environ["OPENROUTER_API_KEY"] = value

# Keep every comparison input fixed except the search engine.
API_URL = "https://openrouter.ai/api/v1/chat/completions"
API_KEY = os.environ["OPENROUTER_API_KEY"]
MODEL = "openai/gpt-4o-mini"
QUESTION = "What are the biggest technology news stories this week?"
ENGINES = ["exa", "parallel", "perplexity"]
results = []

print(f"Ready to compare: {', '.join(ENGINES)}")

# Send one grounded request and normalize its answer plus citation URLs.
def run_search(engine):
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": QUESTION}],
        "tools": [{
            "type": "openrouter:web_search",
            "parameters": {"engine": engine, "max_uses": 1},
        }],
    }
    request = urllib.request.Request(
        API_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request) as response:
        data = json.load(response)
    message = data["choices"][0]["message"]
    citations = [
        item["url_citation"]["url"]
        for item in message.get("annotations", [])
        if item.get("type") == "url_citation"
    ]
    return {"engine": engine, "answer": message["content"], "citations": citations}

# Capture Exa first so its record can be inspected before the full comparison.
results.append(run_search(ENGINES[0]))
print(json.dumps(results[0], indent=2))

# Run the identical request through the two remaining engines.
for engine in ENGINES[1:]:
    result = run_search(engine)
    results.append(result)
    print(json.dumps(result, indent=2))

# Print one combined structure for direct side-by-side inspection.
print("\n=== SIDE-BY-SIDE RESULTS ===")
print(json.dumps(results, indent=2))

Your engine comparison is ready. Next up, you will record each call's cost so answer quality plus search price can be compared together.

Capture Search Costs

Your three pinned searches already reveal how engine choice changes an answer. Their source lists make those differences visible.

Cost is the missing part of the comparison. The fixed model keeps model choice constant.

In this step, you will preserve the usage data reported by OpenRouter. You will compare any search-related amount with the published range.

In this step, get ready to:

Capture the response usage for each pinned engine.
Compare each search-related cost with OpenRouter's published range.

Capture usage for each engine

The API response's usage data provides the evidence for the cost side of your comparison. A complete usage record preserves every cost detail returned for that call.

Switch back to compare_search_engines.py in the code editor from the previous step.
Find the existing loop that sends one request for each pinned engine.
Find the structured result record that stores the current engine's answer.
Read the response's usage data from the same response object used for the answer.
Add that complete usage data to the current engine's structured result record.
Save compare_search_engines.py.

Why save the complete usage object?

The API can report several usage details for one call. Preserving the full object keeps the original evidence available for later comparisons.

The existing answer stays in each record. The existing source URLs stay with it.

Compare costs with the published range

OpenRouter publishes an approximate $0.001 to $0.007 per search range for its web search tool. This gives you a reference for the search-related part of each call.

Search Cost Versus Total Call Cost

The published range describes the search component. A full call can also include model tokens.

A separately reported search amount supports a direct range comparison. An overall total captures the complete amount charged for the call.

This next run spends credit on only the three live searches already planned for your small test.

Predict whether the fixed question forces all three engines to report the same usage or cost.
Run compare_search_engines.py again with the same editor run control from the previous step.

Your output should contain three refreshed result records. Each record should now carry its own usage or cost data.

That closes the missing evidence gap. Every pinned engine now has its own API-reported usage record.

Missing Usage From a Record?

Confirm that you read usage inside the engine loop. Confirm that you add it to the same structured record as the current answer.

Help me find why one engine record is missing its usage data.

Record OpenRouter's published approximate range as $0.001 to $0.007 per search beside your saved results.
Inspect the saved usage breakdown for Exa.
Compare Exa's separately reported search amount with the published range.
Inspect the saved usage breakdown for Parallel.
Compare Parallel's separately reported search amount with the published range.
Inspect the saved usage breakdown for Perplexity.
Compare Perplexity's separately reported search amount with the published range.
Label any overall call total clearly when the response does not provide a separate search amount.

Predict whether every recorded overall call total belongs inside the search-only reference range.
Review all three structured result records against the published range.

Your finished comparison should show one usage or cost record for each engine. It should also show the published approximate range beside the three saved results.

A separately reported search amount supports the range comparison. A labelled overall total preserves the complete call cost when no separate search amount appears.

That completes the financial side of your first comparison. You can now discuss engine differences with recorded evidence.

Cost Looks Higher Than the Range?

The published range covers the search portion of the call. A full response total can sit above it because model tokens add to the cost.

Help me interpret the cost data in my OpenRouter response.

Your one-question comparison now includes cost evidence beside every engine result. Next, you will scale this comparison into a five-question CSV.

Generate Your Comparison CSV

Your earlier OpenRouter comparison produced a useful three-engine snapshot. Now you will turn that snapshot into a repeatable benchmark with enough rows to reveal patterns.

One news question can favor an engine by chance. Five questions create a broader sample that you can inspect in a CSV while the model stays fixed.

In this step, get ready to:

Expand the benchmark from one question to five questions.
Export 15 comparable results to search_engine_comparison.csv.
Evaluate each engine's evidence-to-cost tradeoff in a spreadsheet.

Scale the benchmark to five questions

The comparison keeps openai/gpt-4o-mini fixed for every request. The engine set contains Exa, Parallel, plus Perplexity.

In compare_search_engines.py from earlier, locate What are the biggest technology news stories this week?.
Use that question as the first item in a five-question collection.
Add What are the biggest artificial intelligence announcements this week? as the second question.
Add What are the biggest cybersecurity stories this week? as the third question.
Add What are the biggest space and science stories this week? as the fourth question.
Add What are the biggest climate and energy stories this week? as the fifth question.

How are 15 results formed?

The outer loop selects one question. The existing engine loop sends that question through all three pinned engines.

Five questions multiplied by three engines produces 15 result records. Including the current question in each record keeps every answer connected to its prompt.

Update the existing execution flow so each question passes through the complete engine loop with the current question stored in its result record.
Save compare_search_engines.py.

The next run sends 15 billable requests. At OpenRouter's published range, the search component totals approximately 1.5 to 10.5 cents before model token charges.

Before you run the expanded benchmark, how many results do you expect each engine to produce?

Use your code editor's run control to run compare_search_engines.py.

You should see each of the five questions processed once by each engine. The completed collection should contain 15 structured result records.

Seeing fewer than 15 results?

Check that the five-question loop surrounds the existing three-engine loop. A loop placed after the engine calls still processes only the original question.

Check that every successful response is added to the shared result collection. A collection recreated inside either loop loses earlier results.

Ask for help with the nested loops in my comparison script.

Write the comparison CSV

A flat table makes every engine run easy to filter. Each row needs enough context to compare the answer's evidence with the reported cost.

What belongs in each row?

Use one row for one question-engine combination. Preserve the values already captured from the API response.

A question column identifies the prompt used for the request.
An engine column identifies Exa, Parallel, or Perplexity.
An answer column contains the returned response text.
A source-links column contains the returned citation or source URLs.
A citation-count column records the number of returned URLs.
Per-call cost or usage columns preserve the response data captured earlier.

Use the language's CSV-writing utility so commas or line breaks inside an answer remain within the correct cell.

Add export logic after all 15 responses finish so the complete result collection replaces search_engine_comparison.csv on every run.
Save compare_search_engines.py.

Before you run the script again, what row count should the new CSV contain?

Use your code editor's run control to run compare_search_engines.py again.

You should see search_engine_comparison.csv appear in the openrouter-search-engine-comparison folder. It should contain 15 data rows beneath its headers.

CSV missing or incomplete?

If the file is missing, confirm the export runs after the request loops finish. If it contains fewer than 15 rows, confirm the export receives the complete shared result collection.

If answers spill across columns, use a CSV-writing utility that quotes field values. Manually combining values cannot safely preserve commas or line breaks in generated answers.

Help me debug my comparison CSV export.

That export turns 15 separate API responses into one comparison dataset you can inspect without scrolling through terminal output.

✔️ Awesome, I've got everything!

Great. Double-check that compare_search_engines.py is saved before you inspect the generated CSV.

ⓧ I'd like to double check the full code

Compare your completed script against this behavior checklist.

Confirm the script still reads OPENROUTER_API_KEY from private local storage.
Confirm every request still uses openai/gpt-4o-mini.
Confirm the five questions each run through Exa, Parallel, and Perplexity.
Confirm every result records its question, engine, answer, returned URLs, citation count, plus per-call cost or usage data.
Confirm the complete result collection writes to search_engine_comparison.csv after the request loops finish.

Compare the engines in a spreadsheet

A spreadsheet groups the 15 rows into a view you can sort or filter. The fixed model helps you focus on changes caused by the search engine.

Use the file sidebar in your code editor to locate search_engine_comparison.csv.
Open search_engine_comparison.csv with a spreadsheet application on your computer.

Before you count the rows, how many groups do you expect to see if each question has one result from every engine?

Count the data rows beneath the header.

You should see 15 data rows arranged as five groups of three engine results. Each group should contain the same question three times.

Filter the spreadsheet to one question.
Compare the source links across its three engine rows.
Review the citation count for each engine row.
Compare the recorded per-call cost or usage values with OpenRouter's approximate $0.001 to $0.007 per-search range.
Repeat the comparison for the other four questions.

You now have a five-question benchmark that shows which engine returns the most useful sources for the cost you observed. The engine choice is measurable through the same model instead of relying on a single answer.

Secret mission

Run a Blind Search Engine Review

Hide the engine names and recorded costs while you score all 15 results for answer relevance and source usefulness. Reveal the labels afterward to see whether your preferred engine also delivers the strongest value.

Clean Up Your Resources

Keeping your local files plus an unused API key creates no new search calls. Choose whether to keep the project, pause your work, or remove everything before another run uses more OpenRouter credits.

Resources you used:

The local openrouter-search-engine-comparison folder containing compare_search_engines.py plus search_engine_comparison.csv.
The OpenRouter API key created for this project.
The local OPENROUTER_API_KEY environment setting or private .env entry.

Keep everything running

No action needed. Choose this if you want to rerun the comparison or inspect the results later.

Keep the openrouter-search-engine-comparison folder on your computer.
Keep the project API key active only if you plan to make more requests.
Keep any private .env file out of source control.
Check the OpenRouter Credits page before running another 15-call comparison.

Pause - I'll come back to this later

No server or background process needs to be stopped. Pausing means closing your project applications while keeping the files available.

Close the spreadsheet displaying search_engine_comparison.csv.
Close the code editor containing openrouter-search-engine-comparison.
Close the terminal session used to run the script.
Leave the local project folder in place for your next session.

Your script plus comparison results remain on your computer. The idle files create no additional API usage.

Delete - I don't want to use this again

Remove the project credential plus the local comparison files when you no longer need this experiment. Any unused OpenRouter credits remain in your account.

Deletion is permanent

Revoking the project key disables future requests that use it. Your OpenRouter account remains available.

Deleting the project folder removes the script plus its CSV results. Any private .env file stored inside that folder is removed too.

Return to the OpenRouter API Keys page from earlier.
Locate the API key created for this project.
Use the available revocation control for that key.
Confirm the revocation if the page asks.

You should see that the project key is no longer active. Requests using its old value can no longer consume your credits.

Remove OPENROUTER_API_KEY from the same environment configuration where you saved it if you used a persistent environment variable.
Remove the OPENROUTER_API_KEY entry from any private .env file stored outside the project folder.
Close any terminal session that still holds the old key.

macOS

Close the spreadsheet application that is displaying search_engine_comparison.csv.
Close the code editor that contains openrouter-search-engine-comparison.
Use macOS file search to locate the openrouter-search-engine-comparison folder.
Move the exact matching folder to the deleted-items area.
Permanently clear the deleted-items area.
Search for the folder name once more.

You should find no folder named openrouter-search-engine-comparison. This confirms that the script plus CSV are gone.

Windows

Close the spreadsheet application that is displaying search_engine_comparison.csv.
Close the code editor that contains openrouter-search-engine-comparison.
Use Windows file search to locate the openrouter-search-engine-comparison folder.
Move the exact matching folder to the deleted-items area.
Permanently clear the deleted-items area.
Search for the folder name once more.

You should find no folder named openrouter-search-engine-comparison. This confirms that the script plus CSV are gone.

Linux

Close the spreadsheet application that is displaying search_engine_comparison.csv.
Close the code editor that contains openrouter-search-engine-comparison.
Use your Linux file manager to locate the openrouter-search-engine-comparison folder.
Move the exact matching folder to the deleted-items area.
Permanently clear the deleted-items area.
Search for the folder name once more.

You should find no folder named openrouter-search-engine-comparison. This confirms that the script plus CSV are gone.

That closes the experiment safely. Your revoked key can no longer make requests.

Nice Work!

Fantastic work! You built a repeatable OpenRouter web search comparison with one fixed model. The finished spreadsheet compares results from Exa, Parallel, and Perplexity across five questions.

You've learned how to:

Validate a private API key with a successful OpenRouter request that returns an answer.
Run a fixed-model comparison across Exa, Parallel, and Perplexity to isolate the search engine as the changing factor.
Analyze a 15-row CSV to compare source usefulness, citation counts, and reported per-call cost against OpenRouter's published range.
Secret Mission: Applied the same comparison method in an optional challenge to push your analysis skills further.

Ready to quiz yourself?