Choose an Embedding Model

Test three embedding models on your documents and compare hit rate and cost.

Introduction

30 Second Summary

A tool can look impressive in a public ranking yet miss the information you care about. The real test happens with the files and questions you use every day.

In this project, you will test three embedding models from OpenRouter against your own documents. Your Python script will turn the retrieval results plus cost data into a comparison table and chart.

What You'll Build

You will open a chart that shows which of three embedding models best retrieved the right documents for your 10 test questions.

By the end of this project, you'll have:

A repeatable model bakeoff that creates a separate searchable index for each model from your own documents.
A per-question retrieval report that shows the top three matches from every model.
An evidence-backed model choice supported by hit rate, vector size, cost per run, and a comparison chart.
Secret Mission: An optional challenge to push your model evaluation skills further.

Are there any prerequisites?

You need Python plus an OpenRouter account with a small credit balance.

You also need a folder containing 20 to 50 text documents.

Before We Start

A useful embedding model comparison depends on each candidate running under the same conditions. A shared local workspace keeps package versions consistent.

You'll prepare Python inside an isolated virtual environment. You'll also create separate folders for documents, indexes, and results.

In this step, get ready to:

Confirm Python 3 is available.
Create an isolated project workspace.
Prepare the project dependencies and folder structure.

Check that Python 3 is ready

Python runs the scripts that embed your documents and compare the models. A version check confirms that your terminal can find a Python 3 interpreter.

macOS

Press Cmd+Space to open Spotlight.
Type Terminal into Spotlight.
Press Enter to open Terminal.
Check the installed Python version by running this command:

python3 --version

What does this command check?

The --version option prints the Python version connected to python3. This confirms that the interpreter is available from Terminal.

You should see output beginning with Python followed by a major version of 3.

Windows

Press the Windows key to open search.
Type PowerShell into search.
Press Enter to open PowerShell.
Check the installed Python version by running this command:

python --version

What does this command check?

The --version option prints the Python version connected to python. This confirms that the interpreter is available from PowerShell.

You should see output beginning with Python followed by a major version of 3.

Linux

Use your application launcher to search for Terminal.
Select Terminal from the search results.
Check the installed Python version by running this command:

python3 --version

What does this command check?

The --version option prints the Python version connected to python3. This confirms that the interpreter is available from Terminal.

You should see output beginning with Python followed by a major version of 3.

✔️ Python is ready

That first check is complete. Python can now create the isolated environment for your model comparison.

ⓧ Python 3 is unavailable

Your computer needs a Python 3 installation before it can create the project environment. Choose your operating system below.

macOS

Visit the official Python downloads for macOS page.
Download the current stable Python 3 installer.
Run the downloaded installer.
Accept the default installation options.
Return to Terminal after the installation finishes.
Repeat the version check from the macOS tab above.

Windows

Visit the official Python downloads for Windows page.
Download the current stable Python 3 installer.
Run the downloaded installer.
Accept the default installation options.
Return to PowerShell after the installation finishes.
Repeat the version check from the Windows tab above.

Linux

Read the official Python guidance for Unix platforms.
Follow the installation path for your Linux distribution.
Return to Terminal after the installation finishes.
Repeat the version check from the Linux tab above.

Still unable to run Python?

Close your terminal after installing Python. Open a new terminal so it can detect the new installation.

If Python remains unavailable, ask for help with your installation.

Create an isolated workspace

The project folder keeps every input and result in one place. The .venv environment keeps this project's packages separate from packages used by other Python projects.

macOS

Create the project folder on your Desktop by running these commands:

cd ~/Desktop
mkdir embedding-model-bakeoff
cd embedding-model-bakeoff
python3 -m venv .venv
source .venv/bin/activate

What do these commands do?

The first command moves your terminal to the Desktop.
The second command creates embedding-model-bakeoff.
The third command enters the new project folder.
The fourth command creates an isolated environment in .venv.
The final command activates that environment for the current terminal session.

Confirm that (.venv) appears near the start of your terminal prompt.

Virtual environment not active?

Confirm that your terminal is inside embedding-model-bakeoff. The activation script lives inside the .venv folder created there.

If the prompt still does not change, help me activate the environment.

Windows

Create the project folder on your Desktop by running these commands:

Set-Location $HOME\Desktop
New-Item -Name "embedding-model-bakeoff" -ItemType Directory
Set-Location "embedding-model-bakeoff"
python -m venv .venv
.\.venv\Scripts\Activate.ps1

What do these commands do?

The first command moves PowerShell to the Desktop.
The second command creates embedding-model-bakeoff.
The third command enters the new project folder.
The fourth command creates an isolated environment in .venv.
The final command activates that environment for the current PowerShell session.

Confirm that (.venv) appears near the start of your PowerShell prompt.

Virtual environment not active?

Confirm that PowerShell is inside embedding-model-bakeoff. The activation script lives inside the .venv folder created there.

PowerShell can block local activation scripts because of its execution policy. Use the activation guidance in the official Python virtual environment documentation.

If you need help with the policy or path, help me activate the environment.

Linux

Create the project folder on your Desktop by running these commands:

cd ~/Desktop
mkdir embedding-model-bakeoff
cd embedding-model-bakeoff
python3 -m venv .venv
source .venv/bin/activate

What do these commands do?

The first command moves your terminal to the Desktop.
The second command creates embedding-model-bakeoff.
The third command enters the new project folder.
The fourth command creates an isolated environment in .venv.
The final command activates that environment for the current terminal session.

Confirm that (.venv) appears near the start of your terminal prompt.

Virtual environment not active?

Confirm that your terminal is inside embedding-model-bakeoff. Some Linux distributions provide virtual environment support as a separate Python package.

If creation or activation still fails, help me activate the environment.

Finish the project workspace

The comparison script needs packages for web requests and numerical vectors. It also needs packages for tables and charts.

Install the project dependencies into the active environment by running this command:

pip install requests numpy pandas matplotlib

What do these packages add?

Requests sends embedding requests over HTTP.
NumPy compares numerical vectors.
pandas builds the model comparison table.
Matplotlib renders the final result chart.

Before you check, do you expect the active environment to contain all four packages?

Display the installed package details by running this command:

pip show requests numpy pandas matplotlib

What does this check show?

The output includes a separate package record for each dependency. Each record includes its name and installed version.

You should see records for Requests, NumPy, pandas, and Matplotlib. Good work. Your isolated environment now has every package needed for the bakeoff.

Package details missing?

Confirm that (.venv) still appears in your terminal prompt. Run the installation command again if the environment was inactive during the first attempt.

A network interruption can stop a package download. Help me fix the package installation.

Your later tests need 20 to 50 personal text documents with content you understand. Familiar documents make it possible to decide which search result is correct.

Identify 20 to 50 personal text documents that you can use for the comparison.
Keep the selected documents in their current location for now.

You add those files during the indexing stage. This checkpoint keeps documents/ empty.

macOS

Create the three project folders by running this command:

mkdir documents indexes results

How are the folders used?

The documents/ folder holds your input documents later in the project.
The indexes/ folder stores one embedding index per model.
The results/ folder stores comparison outputs.

Before you check, which three folder names do you expect the terminal to list?

List the workspace contents by running this command:

ls

What does this output prove?

The command lists the visible contents of embedding-model-bakeoff. Seeing all three names confirms that the folder structure exists.

You should see documents, indexes, and results in the output.

Missing a project folder?

Confirm that your terminal is inside embedding-model-bakeoff. Folder names are case-sensitive on many systems.

If one of the folders remains missing, help me fix the folder structure.

Windows

Create the three project folders by running this command:

New-Item -Path documents, indexes, results -ItemType Directory

How are the folders used?

The documents/ folder holds your input documents later in the project.
The indexes/ folder stores one embedding index per model.
The results/ folder stores comparison outputs.

Before you check, which three folder names do you expect PowerShell to list?

List the workspace contents by running this command:

Get-ChildItem

What does this output prove?

The command lists the contents of embedding-model-bakeoff. Seeing all three names confirms that the folder structure exists.

You should see documents, indexes, and results in the output.

Missing a project folder?

Confirm that PowerShell is inside embedding-model-bakeoff. Compare each folder name with the creation command.

If one of the folders remains missing, help me fix the folder structure.

Linux

Create the three project folders by running this command:

mkdir documents indexes results

How are the folders used?

The documents/ folder holds your input documents later in the project.
The indexes/ folder stores one embedding index per model.
The results/ folder stores comparison outputs.

Before you check, which three folder names do you expect the terminal to list?

List the workspace contents by running this command:

ls

What does this output prove?

The command lists the visible contents of embedding-model-bakeoff. Seeing all three names confirms that the folder structure exists.

You should see documents, indexes, and results in the output.

Missing a project folder?

Confirm that your terminal is inside embedding-model-bakeoff. Folder names are case-sensitive on Linux.

If one of the folders remains missing, help me fix the folder structure.

Your workspace is ready. Python is isolated inside .venv. Your inputs and generated outputs now have separate destinations.

✔️ Awesome, I've got everything!

Great. Keep the virtual environment active while you continue to the next step.

ⓧ I'd like to double check the workspace

The documents/ artifact is an empty project input folder, ready for 20 to 50 learner-provided text documents.
The indexes/ artifact is an empty folder, ready to store one embedding index per model.
The results/ artifact is an empty folder, ready to store comparison outputs.

Your local foundation is in place. Next up, you'll create an OpenRouter key and use it to request your first embedding vector.

Test an OpenRouter Embedding

Your model comparison needs a working connection to OpenRouter before any document testing begins. A failed connection would stop every model in the bakeoff.

In this step, you will create an API key for one embedding request. You will store the key locally before confirming that the model returns a numeric vector.

In this step, get ready to:

Create an OpenRouter API key.
Store the key in a local .env file.
Inspect the vector returned by one embedding request.

Create an OpenRouter account and key

OpenRouter uses account credits to pay for model requests. An API key authorizes your terminal to use those credits.

Visit the OpenRouter homepage in your browser.
Create an account through the sign-up flow if you do not already have one.
Sign in to your OpenRouter account.
Visit the Credits page to confirm that your account has an available balance.

Before You Add Credits

Adding credits changes your account balance. Choose only an amount you are comfortable using for this model comparison.

A credit limit on the API key provides another spending boundary. OpenRouter recommends setting a limit on every key.

Add credits through the Credits page if your available balance is zero.

Your account now has the balance required to send a paid embedding request.

Visit the API key settings page.
Enter embedding-bakeoff as the key name.
Set an optional credit limit that fits your budget.
Create the API key using the key creation form.
Copy the new key immediately.

OpenRouter displays the complete key only when it is created. Your copied value is the credential that the terminal request uses.

Cannot Create the Key?

Confirm that you are signed in to the same OpenRouter account that holds your credits.

Check that the key has a name. Raise its credit limit if the selected limit would block the test request.

Help me create an OpenRouter API key.

Store the key in a local .env file

Credentials can feel risky to handle. Your API key stays in a local file named .env throughout this project.

Keep This File Local

The API key grants access to your OpenRouter credits. Keep the value out of screenshots.

Keep .env out of every commit or repository upload.

Create .env beside documents/ in your project workspace by adding this line:

OPENROUTER_API_KEY=your-api-key-here

What Does This Configuration Do?

The OPENROUTER_API_KEY name gives later terminal commands a standard way to find your credential.
The .env file keeps the actual value outside the request command.
The your-api-key-here text marks where your copied key belongs.

Replace your-api-key-here with the API key you copied.
Save .env.
Confirm that .env appears beside documents/ in your workspace.

You should see .env alongside documents/, indexes/, and results/.

Cannot Find the Environment File?

Check that the filename begins with a full stop. Confirm that your editor did not save it as .env.txt.

Make sure the file is beside the three project folders. A file saved elsewhere cannot be loaded by the terminal command in the next substep.

Help me fix my local environment file.

✔️ Awesome, I've got everything!

Your .env file is saved locally. Keep the file out of commits before continuing.

ⓧ I'd like to double check the full code

Your .env file follows this structure. The local file contains your real key in place of the placeholder.

OPENROUTER_API_KEY=your-api-key-here

Call the embeddings endpoint

The terminal needs to load your saved key before it can authorize a request. You can confirm that the variable exists without printing its sensitive value.

Return to the terminal with your activated virtual environment.
Load the key from .env by running these commands:

set -a
source .env
set +a
python -c 'import os; print("API key loaded:", bool(os.getenv("OPENROUTER_API_KEY")))'

What Do These Commands Do?

The shell loads the value from .env into the current terminal session.
The set -a setting exports the loaded value to commands started from that terminal.
The Python check converts the presence of OPENROUTER_API_KEY into a safe Boolean result.

Check the final line of the terminal output.

You should see API key loaded: True. The key itself stays hidden.

Seeing False?

Confirm that your terminal is in the same workspace as .env.

Check that the file begins with OPENROUTER_API_KEY=. Remove any spaces before the name.

Help me load my API key safely.

Before you run the request, do you expect the model to return words or numbers?

Send one test sentence to the embeddings endpoint by running this command:

curl https://openrouter.ai/api/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -d '{
    "model": "openai/text-embedding-3-small",
    "input": "The quick brown fox jumps over the lazy dog"
  }' | python -c 'import json, sys; vector = json.load(sys.stdin)["data"][0]["embedding"]; print("Embedding vector:", vector); print("Vector length:", len(vector))'

What Does This Request Do?

The request sends one sentence to https://openrouter.ai/api/v1/embeddings.
The Authorization header supplies your loaded key without printing it.
The openai/text-embedding-3-small model converts the sentence into a numeric vector.
The data[0].embedding lookup selects that vector from the response.
The len(vector) check reports how many dimensions the vector contains.

Inspect the first line of the terminal output.
Check the final line for the vector length.

You should see a long list beginning with Embedding vector:. The final line should read Vector length: 1536.

That first live vector proves your local key can reach an embedding model through OpenRouter.

Request Did Not Return a Vector?

If the response reports 401 Unauthorized, replace the value in .env with a valid key.

If the response reports 402 Payment Required, add credits to your account. Raise the key limit if it has been exhausted.

If the response reports 404 Not Found, confirm that the model is openai/text-embedding-3-small.

Help me debug my embedding request.

Your first model call works. Next, you will turn the model shortlist into a comparison config.

Configure Three Embedding Models

Your successful OpenRouter request proved that your key can reach an embedding model. A useful comparison still needs a stable shortlist before every document is processed.

This step turns three candidates into a repeatable configuration. Their prices show the expected request cost. Their vector sizes reveal the storage tradeoff.

In this step, get ready to:

Choose three models from the source article shortlist.
Build a configuration file with three fields per model.
Print the three-model configuration table from your comparison script.

Choose a balanced shortlist

A fair test benefits from candidates with different tradeoffs. The first candidate provides an English retrieval baseline. The second candidate tests a larger multilingual representation.

What are the three candidates?

The model openai/text-embedding-3-small costs $0.02 per million input tokens. Its default vector contains 1,536 values.
The model qwen/qwen3-embedding-8b costs $0.01 per million input tokens. Its default vector contains 4,096 values.
The model perplexity/pplx-embed-v1-0.6b costs $0.004 per million input tokens. Its default vector contains 1,024 values.

Open the source article to review its comparison table.
Select openai/text-embedding-3-small as your English retrieval candidate.
Select qwen/qwen3-embedding-8b as your multilingual candidate.
Select perplexity/pplx-embed-v1-0.6b as your lowest-priced paid candidate.

The configuration file keeps these choices fixed across every later test. This prevents a model detail from changing halfway through the bakeoff.

Create models_config.json inside your project workspace by using your code editor's new-file control.
Paste the three-model configuration into models_config.json using the code below:

{
  "models": [
    {
      "model_id": "openai/text-embedding-3-small",
      "price_per_million_tokens": 0.02,
      "vector_size": 1536
    },
    {
      "model_id": "qwen/qwen3-embedding-8b",
      "price_per_million_tokens": 0.01,
      "vector_size": 4096
    },
    {
      "model_id": "perplexity/pplx-embed-v1-0.6b",
      "price_per_million_tokens": 0.004,
      "vector_size": 1024
    }
  ]
}

What does this configuration do?

The models array keeps the shortlist in one place.
Each model_id value tells OpenRouter which embedding model to use.
Each price_per_million_tokens value supports the later cost calculation.
Each vector_size value records the expected number of values in one embedding.

Save models_config.json.
Count the objects inside the models array. You should find exactly three model entries.

Does the configuration look incomplete?

Check that every model object has a model_id field. Confirm that every object also has a price_per_million_tokens field.

Check that the final model object is followed by the array's closing bracket. A missing brace leaves the file as invalid JSON.

Help me check my model configuration.

✔️ Awesome, I've got everything!

Your configuration now contains exactly three candidates. Keep models_config.json saved in your project workspace.

ⓧ I'd like to double check the full code

Your complete models_config.json file should match this configuration.

{
  "models": [
    {
      "model_id": "openai/text-embedding-3-small",
      "price_per_million_tokens": 0.02,
      "vector_size": 1536
    },
    {
      "model_id": "qwen/qwen3-embedding-8b",
      "price_per_million_tokens": 0.01,
      "vector_size": 4096
    },
    {
      "model_id": "perplexity/pplx-embed-v1-0.6b",
      "price_per_million_tokens": 0.004,
      "vector_size": 1024
    }
  ]
}

Load and print the configuration

The JSON file becomes useful when your Python script can load it consistently. The script also checks that your locally saved key is available before later requests begin.

Create embedding_comparison.py inside your project workspace by using your code editor's new-file control.
Paste the configuration loader into embedding_comparison.py using the code below:

import json
import os
from pathlib import Path
import requests
from dotenv import load_dotenv

# Keep local paths and the remote embeddings endpoint in one place.
CONFIG_PATH = Path("models_config.json")
EMBEDDINGS_URL = "https://openrouter.ai/api/v1/embeddings"

# Load the local key without placing the secret in source code.
load_dotenv()
API_KEY = os.getenv("OPENROUTER_API_KEY")
if not API_KEY:
    raise RuntimeError("OPENROUTER_API_KEY is missing from .env.")

# Read the same model settings for every comparison run.
with CONFIG_PATH.open("r", encoding="utf-8") as config_file:
    MODELS = json.load(config_file)["models"]

def print_config_table(models):
    # Print the three facts that drive the later comparison.
    print("Model ID | Price per 1M tokens | Vector size")
    print("-" * 72)
    for model in models:
        print(f"{model['model_id']} | ${model['price_per_million_tokens']} | {model['vector_size']}")

if __name__ == "__main__":
    print_config_table(MODELS)

What does this script do?

The python-dotenv loader reads your existing .env file. The key becomes available through OPENROUTER_API_KEY.
The CONFIG_PATH constant points to models_config.json.
The MODELS list holds the three model objects loaded from disk.
The print_config_table() function turns those objects into terminal rows.

Save embedding_comparison.py.

Before you run the script, which three model identifiers do you expect to see?

Print the configuration table by running this command in the terminal from earlier:

python embedding_comparison.py

What does this command do?

The python command executes embedding_comparison.py. The script loads the saved configuration before printing its rows.

You'll see one header row. You'll also see rows for openai/text-embedding-3-small, qwen/qwen3-embedding-8b, and perplexity/pplx-embed-v1-0.6b.

Good progress. Your shortlist now prints from disk as a repeatable comparison table.

Table missing or script failing?

Confirm that the activated virtual environment from earlier is still active. This ensures Python can import the installed dependencies.

Confirm that models_config.json sits beside embedding_comparison.py.

Help me fix the configuration table.

Add the embedding request helper

Each candidate needs its own embedding request because its output has a different vector size. A shared helper keeps the request shape consistent across all three models.

In embedding_comparison.py, place the following function directly above print_config_table():

def create_embedding(text, model):
    # Send one text value to the model selected from the configuration.
    response = requests.post(
        EMBEDDINGS_URL,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"model": model["model_id"], "input": text},
        timeout=60,
    )
    response.raise_for_status()

    # Return only the numeric vector needed by the local index.
    return response.json()["data"][0]["embedding"]

How does the request helper work?

The Requests library sends an HTTP request to the OpenRouter embeddings endpoint.
The authorization header reads the existing key from API_KEY. The secret never appears in the script.
The request body reads model_id from the current configuration object.
The return statement extracts the numeric embedding from the response.

Save embedding_comparison.py.

Before you rerun the script, do you expect the new helper to change the printed table?

Verify the completed script by running this command in the terminal from earlier:

python embedding_comparison.py

What are you checking?

This rerun checks that the request helper has valid Python syntax. It also confirms that the configuration loader still reaches all three model objects.

You'll see the same three-row configuration table. Each row shows a model identifier. Each row also shows its price and vector size.

That's the configuration layer finished. Your comparison script can now use one consistent request helper for every candidate.

Seeing a syntax or import problem?

Check that create_embedding() starts at the left edge of the file. Confirm that every line inside the function uses the same four-space indentation.

Check that import requests remains near the top of the file. The installed dependency must be available inside the activated virtual environment.

Help me debug the completed script.

✔️ Awesome, I've got everything!

Your script loads the local key. It reads all three model records. It also prints the configuration table.

ⓧ I'd like to double check the full code

Your complete embedding_comparison.py file should match this version.

import json
import os
from pathlib import Path
import requests
from dotenv import load_dotenv

# Keep local paths and the remote embeddings endpoint in one place.
CONFIG_PATH = Path("models_config.json")
EMBEDDINGS_URL = "https://openrouter.ai/api/v1/embeddings"

# Load the local key without placing the secret in source code.
load_dotenv()
API_KEY = os.getenv("OPENROUTER_API_KEY")
if not API_KEY:
    raise RuntimeError("OPENROUTER_API_KEY is missing from .env.")

# Read the same model settings for every comparison run.
with CONFIG_PATH.open("r", encoding="utf-8") as config_file:
    MODELS = json.load(config_file)["models"]

def create_embedding(text, model):
    # Send one text value to the model selected from the configuration.
    response = requests.post(
        EMBEDDINGS_URL,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"model": model["model_id"], "input": text},
        timeout=60,
    )
    response.raise_for_status()

    # Return only the numeric vector needed by the local index.
    return response.json()["data"][0]["embedding"]

def print_config_table(models):
    # Print the three facts that drive the later comparison.
    print("Model ID | Price per 1M tokens | Vector size")
    print("-" * 72)
    for model in models:
        print(f"{model['model_id']} | ${model['price_per_million_tokens']} | {model['vector_size']}")

if __name__ == "__main__":
    print_config_table(MODELS)

Your three-model bakeoff now has a fixed starting line. Next up, you'll embed your own documents into a separate index for each candidate.

Build the Document Indexes

Your three embedding models are configured. The comparison now needs a shared document corpus.

Each model produces vectors in its own vector space. A separate index keeps those vectors tied to the model that created them.

In this step, you will add 20 to 50 personal text documents to documents/. You will use OpenRouter to build one persistent index per model.

The script also reports the token count for each indexing run. Those totals make the later cost comparison specific to your corpus.

In this step, get ready to:

Fill documents/ with a shared evaluation corpus.
Make embedding_comparison.py load every document in the corpus.
Build three model-specific indexes with token totals.

Add your evaluation documents

A fair model bakeoff uses the same corpus for every candidate. Your corpus should contain the material you actually expect to search.

Switch back to the project workspace from earlier.
Choose between 20 and 50 personal text documents for your evaluation corpus.
Copy the selected documents into documents/.
Confirm the folder's file count falls within 20 to 50.

Your documents/ folder now holds the shared evidence that all three models receive.

Load the shared corpus

Search results need to point back to the document that produced each vector. The loader preserves that connection before any embedding requests begin.

In your project workspace, return to embedding_comparison.py.
Add a corpus loader to embedding_comparison.py.

What should the loader preserve?

Every readable file contributes its full text to the corpus.
Every loaded item keeps its source filename.
The loader reports the number of documents it discovered.

Save embedding_comparison.py.
Predict whether the discovered document count will match the file count in documents/.
Run the script with the same terminal command that produced your successful embedding request earlier.

You should see a corpus count that matches the number of files in documents/.

Is the document count wrong?

Confirm every evaluation document sits directly inside documents/. Remove any unrelated files that the loader could count as documents.

Check that each source file contains readable text. An empty or unreadable file should not enter the evaluation corpus.

Help me troubleshoot my document loader.

Build one index per model

Vectors from one embedding model belong to that model's vector space. Three separate index files prevent vectors from different models from being mixed.

Replace the script's existing single-request path with a model indexing loop.

What should each indexing run do?

Each model run uses the same loaded corpus.
Each saved record includes its source filename.
Each saved record includes its embedding vector.
Each model run accumulates the token usage returned for its requests.
Each completed model run writes one persistent index file to indexes/.
Each completed model run prints its total token count.

This full run uses your OpenRouter credit balance. Your fixed 20 to 50 document corpus keeps the work limited to the evaluation set you chose.

The terminal may pause while the requests complete. That pause reflects three full passes through your corpus.

Save embedding_comparison.py.
Predict how many index files three configured models should create.
Run the script with the same terminal command that produced your successful embedding request earlier.

You should see one token total for each configured model after its indexing run completes.

Expand indexes/ in your project workspace.

You should see exactly three persistent index files. Each file should correspond to one model from models_config.json.

Open each saved index file in your editor.

Each index should contain one record per document. Every record should pair a source filename with its embedding vector.

Missing an index or document?

If you see fewer than three index files, confirm the indexing loop processes every entry in models_config.json.

If an index has too few records, compare its record count with the file count in documents/. Check which source file failed before running the script again.

Help me debug my indexing run.

That is the indexing stage complete. Your three models now have comparable indexes built from the same document corpus.

Next, you will test known-answer questions against every index to see which model retrieves the right documents.

Test Questions Against Each Model

Your three embedding model indexes now represent the same document collection. That gives every model the same starting point.

The next challenge is building a fair retrieval test. You will pair ten questions with known correct filenames before viewing any search results.

The script embeds each question with each model. It uses vector similarity to rank the documents stored in that model's index.

In this step, get ready to:

Create ten evaluation questions with known correct documents.
Extend the comparison script with cross-model similarity search.
Print the top three document matches for every question-model combination.

Write unbiased evaluation questions

An evaluation set gives each model the same test. Each entry in your JSON file contains a question plus the filename that should answer it.

Create questions.json beside embedding_comparison.py using the editor from earlier.
Add exactly ten top-level entries to questions.json.
Write one natural-language question in each entry.
Pair each question with its known correct filename from documents/.
Include the full file extension in every correct filename.
Choose every correct filename before viewing any model output.
Save questions.json.
Count the top-level entries in questions.json.

You should have exactly ten question and filename pairs. Every filename should match a file already stored in documents/.

Why write the questions first?

Writing questions after viewing search results can favour the documents a model already retrieves well. Choosing the correct documents first keeps the test focused on your original expectations.

The known filename becomes the answer key for the scoring step. Your final model choice rests on a consistent test.

Build the cross-model search loop

Each saved index belongs to one model's vector space. A question vector only makes sense when compared with document vectors from that same model.

The evaluation loop sends each question to the matching model through OpenRouter. It ranks the corresponding index using the resulting query vector.

Return to embedding_comparison.py in the editor from earlier.
Build an input-loading section that reads the ten entries from questions.json plus the three saved index files from indexes/.
Save embedding_comparison.py.
Rerun the script with the same terminal command you used in the previous step.

You should see confirmation that the script loaded ten questions. You should also see confirmation that it loaded three model indexes.

Build the evaluation loop that prints the three highest-similarity filenames for every question-model combination.
Save embedding_comparison.py.
Rerun the script with the same terminal command from earlier.

You should see each question tested against all three model indexes. Every question-model result should contain three ranked document filenames.

Missing matches or seeing a similarity failure?

Confirm that questions.json contains valid JSON. Check that all ten entries use the same field structure.

Confirm that every query vector is compared only with the index created by the same model. A vector-size mismatch usually points to the wrong index.

Help me troubleshoot my cross-model similarity search.

Verify every top-three result

The final check focuses on complete coverage. Your output needs one ranked result for every combination of ten questions and three models.

Before you rerun the script, make a quick prediction about whether all three models will agree on the first result.

Rerun embedding_comparison.py with the same terminal command from earlier.

Across the output, you should find 30 question-model combinations. Each combination should list three ranked filenames from documents/.

Confirm that all ten questions appear under each of the three models.
Confirm that every result contains exactly three document filenames.
Compare the printed matches with the known filenames in questions.json.
Frame your evidence screenshot around the retrieval results only.

That completes the retrieval test. Your script can now show where the models agree on your documents and where their rankings differ.

Your ten-question bakeoff now produces comparable retrieval results for all three models. Next up, you will score hit rate plus cost before turning the winner into a chart.

Compare Results and Choose a Model

Your evaluation now shows the top three documents returned for every question. Those ranked filenames prove that retrieval works. They still leave the final model choice unclear.

This step turns those results into evidence you can compare across your three OpenRouter models. You will score retrieval quality. You will estimate indexing cost. You will save a CSV table. You will render a PNG chart that supports your final choice.

In this step, get ready to:

Score the top-three hit rate for each model.
Save quality metrics and cost metrics in a comparison table.
Render a chart that supports your final model choice.

Score retrieval quality and cost

A hit rate measures how often the known correct document appears among a model's top three matches. Your 10 labeled questions turn that definition into a percentage for each model.

How is hit rate calculated?

Count one hit when the known correct filename appears anywhere in the top three matches.
Count zero hits when the correct filename does not appear in those matches.
Divide each model's hit count by 10.
Multiply the result by 100 to express the score as a percentage.

Switch back to embedding_comparison.py in the editor you used earlier.
Locate the existing search loop that prints the top three document matches for each question.
Add a scoring section after each model finishes its 10 questions. This section records every top-three success. It calculates the model's hit-rate percentage.
Save embedding_comparison.py.

Before you rerun the script, which model do you expect to score the most hits on your documents?

Rerun embedding_comparison.py from the activated terminal with the same command you used in the previous step.

You should see a hit count and hit-rate percentage for each of the three models. Your raw search output now has a measurable retrieval score.

Seeing zero hits for every model?

Check that every correct filename in questions.json exactly matches a filename stored in the indexes.
Confirm that your scoring section checks all three returned matches.
Print one correct filename beside its three returned filenames to expose differences in spelling or file extensions.

Help me debug my hit-rate scoring.

Retrieval quality tells you whether a model finds the right document. The saved token count tells you how much text each model indexed. The price in models_config.json converts that usage into an estimated cost per indexing run.

Add a cost-calculation section to embedding_comparison.py. It reads each saved token count. It applies the matching price unit recorded in models_config.json.
Save embedding_comparison.py.

Before you rerun the script, do you expect the cheapest model to have the lowest token count?

Rerun embedding_comparison.py from the activated terminal.

You should see an estimated indexing cost for every model. Models can process the same corpus at different prices even when their token counts are similar.

Missing or implausible costs?

Confirm that each model entry in models_config.json still contains its recorded price.
Check that each cost calculation uses the token count saved for the matching model.
Display more decimal places if a very small estimate appears as zero.

Help me debug my cost estimates.

Terminal output disappears when you close the session. A comparison table preserves the evidence behind your decision.

Add an export section to embedding_comparison.py. It writes one row per selected model to results/model_comparison.csv.
Include the model name in each row. Include its hit rate. Include its vector size. Include its token count. Include its estimated indexing cost.
Save embedding_comparison.py.
Rerun embedding_comparison.py from the activated terminal.

You should see a three-row comparison table in the terminal. You should also see model_comparison.csv inside the existing results/ folder.

Comparison table missing?

Confirm that the export path points to the existing results/ folder.
Check that the export runs after all three model summaries have been collected.
Confirm that each summary row contains the same set of fields.

Help me fix the comparison table.

Render the comparison chart

The table keeps exact values. A bar chart makes the trade-offs visible across models with very different scales.

What should the chart compare?

The finished chart compares hit rate, vector size, and estimated cost per indexing run. Every metric includes one bar for each of your three selected models.

Separate views for the three metrics keep a large vector size from visually flattening a small cost value. Clear model labels connect every bar to the matching table row.

Add a chart-rendering section to embedding_comparison.py. It uses the same three summary rows written to the comparison table.
Create one bar view for hit rate. Create one bar view for vector size. Create one bar view for estimated cost.
Label every bar view with the three selected model names.
Save the complete chart to results/model_comparison.png.
Save embedding_comparison.py.

Before you rerun the script, which metric do you expect to show the largest difference between models?

Rerun embedding_comparison.py from the activated terminal.

You should see confirmation that the comparison table and chart were written to the results/ folder.

Return to the results/ folder in your editor's file sidebar.
Select model_comparison.png to preview the chart.

You should see all three models represented in the hit-rate view. You should see the same models represented in the vector-size view and cost view.

Chart missing or hard to read?

Confirm that the chart section receives all three model summary rows.
Check that the chart saves inside results/ with the filename model_comparison.png.
Shorten the displayed model labels if they overlap beneath the bars.

Help me debug the comparison chart.

✔️ Awesome, I've got everything!

Great work. Your script now scores retrieval quality. It calculates cost. It writes the comparison table. It renders the final chart.

ⓧ I'd like to double check the full code

Compare your completed embedding_comparison.py against these required outcomes.

The existing search logic still prints the top three matches for every question and model.
The scoring logic checks each known correct filename against all three returned matches.
The summary logic calculates hit rate from exactly 10 labeled questions.
The cost logic combines each model's saved token count with its recorded price.
The export logic writes results/model_comparison.csv with one row for each selected model.
The chart logic writes results/model_comparison.png with hit rate, vector size, and estimated cost for all three models.

Choose your preferred model

Your preferred model should meet the retrieval quality your documents need. Cost and vector size help you choose between models with similar hit rates.

Identify the model with the highest hit rate in model_comparison.csv.
Compare the estimated costs of any models with similar hit rates.
Compare vector sizes when storage size matters for your future index.
Choose the model whose quality and operating cost best fit your documents.

Before your final check, do you expect the tallest hit-rate bar to belong to the model you selected?

Compare your selected model with the highest hit-rate row in results/model_comparison.csv.
Confirm that results/ contains model_comparison.csv.
Confirm that results/ contains model_comparison.png.

You should see three model rows in the table. You should see the same three models across every metric in the chart.

You have turned your own documents into a repeatable model benchmark. Your final choice now rests on measured retrieval quality and recorded cost.

Secret mission

Stress-Test Your Model Choice

Re-rank your three embedding models under changing production priorities. Test whether your original winner survives a focus on hit rate. Repeat the decision for vector size. Finish with indexing cost.

Clean Up Your Resources

Your local workspace has no ongoing cost. Choose whether to keep it, pause it with a capped OpenRouter key, or delete it before future embedding requests use more account credit.

Resources you used:

An active OpenRouter API key used for embedding requests.
A local project workspace containing the activated Python virtual environment, installed dependencies, .env file, and embedding_comparison.py script.
Evaluation inputs in documents/, models_config.json, and questions.json.
Generated model indexes in indexes/ plus the comparison record in results/.

Keep everything running

No action is needed. Choose this if you plan to test more models or keep your comparison as a model-choice record.

Keep the local project workspace with its documents, indexes, questions, results, and Python environment.
Keep .env uncommitted so the API key stays outside version control.
Leave the OpenRouter API key active only if you plan to run more comparisons.
Preserve results/model_comparison.csv and results/model_comparison.png as evidence for your preferred model.

Pause - I'll come back to this later

Pausing ends the active local session while preserving every file for later. A spending cap limits what the existing key can consume while you are away.

Close the terminal session from earlier to end the activated Python virtual environment.
Leave the local project workspace unchanged so its indexes and results remain available.
Return to the OpenRouter dashboard from earlier.
Apply a spending cap to the existing OpenRouter API key.
Confirm that the cap now limits future usage through that key.

Delete - I don't want to use this again

Deleting your workspace is permanent. Your documents and comparison record disappear with it.

Copy results/model_comparison.csv to a folder outside the project workspace if you want to preserve the comparison table.
Copy results/model_comparison.png to a folder outside the project workspace if you want to preserve the chart.
Copy any source documents you want to keep to a folder outside the project workspace.
Return to the OpenRouter dashboard from earlier.
Select the API key used for this project.
Use the revoke control to disable the key.
Confirm that the dashboard no longer shows the key as active.

The workspace deletion steps differ by operating system.

macOS

Press Cmd+Space to open system search.
Type Finder and press Enter to open it.
Locate the project workspace folder that contains embedding_comparison.py and models_config.json.
Select the project workspace folder.
Press Cmd+Delete to move the selected folder to Trash.
Review the contents of Trash before permanent deletion.
Empty Trash after confirming that every item there can be deleted.
Confirm that the project workspace no longer appears in its previous location.

Windows

Press the Windows key to open system search.
Type File Explorer and press Enter to open it.
Locate the project workspace folder that contains embedding_comparison.py and models_config.json.
Select the project workspace folder.
Press Shift+Delete to start permanent deletion.
Confirm the deletion when Windows asks.
Confirm that the project workspace no longer appears in its previous location.

Removing the workspace clears .env, documents/, indexes/, models_config.json, questions.json, results/, and embedding_comparison.py together.

Nice Work!

You did it! You turned your own documents into a clear embedding model choice backed by measured results.

You've learned how to:

Build a repeatable model bakeoff with OpenRouter. Use 20 to 50 of your own documents as the corpus. Test each model with 10 labeled questions.
Create a separate saved index for each model. Review the top three matches for every question.
Calculate hit rate from known correct documents. Estimate cost per indexing run from model prices plus token counts. Compare vector size in the final chart. Use results/model_comparison.png as a visual record of your model choice.
Extend your model evaluation workflow through the optional Secret Mission to push your skills further.

Ready to quiz yourself?