Grade AI Answers with Probabilities

Compare Jev and an LLM judge on labels for agreement, cost, and latency.

Introduction

30 Second Summary

A grader can sound completely certain while making the wrong call. That confidence becomes a problem when you need to evaluate many answers consistently.

In this project, you will use Python to compare Jev with an LLM judge on the same labeled answers through OpenRouter. Your script will save each decision before summarizing agreement, cost, and latency.

What You'll Build

Picture running your finished script and seeing a side-by-side table that shows which grader agrees with your labels more often across the same 20 answers.

By the end of this project, you'll have:

  • A 20-answer labeled dataset that prints as a table. You can inspect every answer beside its expected grade.
  • Two grader results files that preserve every decision. The Jev file includes a probability. Both files capture latency. They also capture cost.
  • A side-by-side summary table that reveals which grader matches your labels more often. It also compares total cost. It compares median latency too.
  • Secret Mission: Add a separate calibration split to choose the probability threshold without tuning against the test labels.

Are there any prerequisites?

You need Python 3 on your computer. You also need an OpenRouter account with a small credit balance.

Before We Start

A benchmark only means something when every run starts from the same environment. A missing dependency can stop Jev before it grades a single answer.

Python will run both graders inside one isolated workspace. Git will track the benchmark code plus its results.

You also need one OpenRouter credential for both graders. Keeping it in an environment variable prevents it from landing in repository files.

In this step, get ready to:
  • Confirm that Python and Git are ready.
  • Create a local repository with an isolated Python environment.
  • Install the benchmark dependencies and configure your OpenRouter API key.
Check Python and Git

The tabular output package used later supports Python 3.10 or higher. Git needs to return a version so it can create your local repository.

  • Press Cmd+Space on macOS or the Windows key on Windows to open system search.
  • Type Terminal on macOS or PowerShell on Windows into the search field.
  • Press Enter to open the terminal.

macOS

  • Check your Python and Git versions by running these commands:
python3 --version
git --version

What do these commands check?

  • The first command prints the installed Python version.
  • The second command confirms that Git is available in this terminal.

Windows

  • Check your Python and Git versions by running these commands:
python --version
git --version

What do these commands check?

  • The first command prints the installed Python version.
  • The second command confirms that Git is available in PowerShell.

Choose the Python result that matches your terminal.

✔️ I see version 3.10 or higher

Good, your Python interpreter can run the benchmark dependencies.

ⓧ I see an older version

Your current Python version is too old for the tabular output package used in this project.

  • Visit the official Python downloads page.
  • Download a current Python 3 installer for your operating system.
  • Complete the installation using its default options.
  • Close the terminal after the installation finishes.
  • Repeat the terminal opening steps above.
  • Repeat the Python version check.

ⓧ Command not found

Your terminal cannot find a Python installation yet.

  • Visit the official Python downloads page.
  • Download a current Python 3 installer for your operating system.
  • Complete the installation using its default options.
  • Close the terminal after the installation finishes.
  • Repeat the terminal opening steps above.
  • Repeat the Python version check.

Choose the Git result that matches your terminal.

✔️ I see a Git version

Git is ready to turn your benchmark folder into a repository.

ⓧ Command not found

Your terminal cannot find Git yet.

  • Visit the official Git installation page.
  • Download the installer for your operating system.
  • Complete the installation using its default options.
  • Close the terminal after the installation finishes.
  • Repeat the terminal opening steps above.
  • Repeat the Git version check.

Version check still failing?

  • Close your current terminal so a new session can load the updated command path.
  • Repeat the system search steps to open a fresh terminal.
  • Confirm that the installer finished before repeating the version check.

Still stuck? Help me troubleshoot why Python or Git is not available in my terminal.

Create the repository and virtual environment

A local repository gives the benchmark code plus its result files one named home. A virtual environment keeps this project's Python packages separate from packages used by other projects.

  • Move to your Desktop in the terminal from the previous check by running this command:
cd ~/Desktop

What does this command do?

This changes the terminal's current location to your Desktop. The repository you create next stays easy to find.

macOS

  • Create the local repository by running these commands:
git init jev-llm-grader
cd jev-llm-grader
git status

What do these commands do?

  • The first command creates the `jev-llm-grader` folder as a Git repository.
  • The second command moves your terminal into that folder.
  • The final command reports the repository's current state.
  • Create and activate the isolated Python environment by running these commands:
python3 -m venv .venv
source .venv/bin/activate
python --version

What do these commands do?

  • The first command creates the `.venv` environment inside `jev-llm-grader`.
  • The second command activates that environment for the current terminal session.
  • The final command proves that the active environment can run Python.

Your prompt should now begin with the environment name. The Python version appears underneath it.

Environment did not activate?

  • Confirm that your terminal is inside the `jev-llm-grader` folder.
  • Confirm that the `.venv` directory was created inside that folder.
  • Repeat the activation command from the same terminal session.

Need another pair of eyes? Help me activate my Python virtual environment on macOS.

Windows

  • Create the local repository by running these commands:
git init jev-llm-grader
cd jev-llm-grader
git status

What do these commands do?

  • The first command creates the `jev-llm-grader` folder as a Git repository.
  • The second command moves PowerShell into that folder.
  • The final command reports the repository's current state.
  • Create and activate the isolated Python environment by running these commands:
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python --version

What do these commands do?

  • The first command creates the `.venv` environment inside `jev-llm-grader`.
  • The second command activates that environment for the current PowerShell session.
  • The final command proves that the active environment can run Python.

Your prompt should now begin with the environment name. The Python version appears underneath it.

Environment did not activate?

  • Confirm that PowerShell is inside the `jev-llm-grader` folder.
  • Confirm that the `.venv` directory was created inside that folder.
  • Use Python's official virtual environment guidance if PowerShell blocks the activation script.

Need help with the PowerShell restriction? Help me activate my Python virtual environment on Windows.

Install dependencies and protect the API key

The benchmark needs one library for direct HTTP requests. It also needs libraries for typed OpenRouter responses plus readable terminal tables.

  • Install the required dependencies into the active environment by running these commands:
python -m pip install requests
pip install openrouter
pip install tabulate

What do these packages provide?

  • `requests` sends direct HTTP requests to OpenRouter.
  • `openrouter` provides Python access to typed model responses.
  • Python's built-in `csv` module writes benchmark results without another installation.
  • `tabulate` prints the dataset plus comparison results as readable tables.
  • Confirm that each command finishes without an installation error.

Package installation failed?

  • Confirm that your terminal prompt still shows the active `.venv` environment.
  • Confirm that your Python version is `3.10` or higher.
  • Check your internet connection before repeating the installation commands.

Still seeing an installation error? Help me troubleshoot my benchmark dependencies.

OpenRouter requests use account credit. The credits page tells you whether your account has enough balance to continue.

  • Open your OpenRouter credits page.
  • Confirm that the page shows a positive credit balance.
  • Add a small credit balance if the available balance is zero.

Keep the key contained

This key can spend your OpenRouter balance. A per-key credit limit caps the amount it can use.

The key is shown once after creation. Store it securely before closing that screen.

  • Open your OpenRouter key settings.
  • Sign in to your OpenRouter account if prompted.
  • Select Create API Key.
  • Enter Jev grading benchmark in the Name field.
  • Choose a Credit limit that caps the spend you are willing to allow.
  • Use the confirmation control at the bottom of the form to generate the key.
  • Copy the generated key while it is visible.
  • Store the copied key in your password manager.

The environment variable makes the key available to Python without placing it in the repository. The value lasts for the current terminal session.

macOS

  • Export the key by replacing `sk-or-v1-YOUR-API-KEY` in this command before you run it:
export OPENROUTER_API_KEY="sk-or-v1-YOUR-API-KEY"

What does this command do?

This stores the credential under `OPENROUTER_API_KEY` in your current terminal session. Future Python processes can read it without a project file.

Windows

  • Export the key by replacing `sk-or-v1-YOUR-API-KEY` in this command before you run it:
$env:OPENROUTER_API_KEY="sk-or-v1-YOUR-API-KEY"

What does this command do?

This stores the credential under `OPENROUTER_API_KEY` in your current PowerShell session. Future Python processes can read it without a project file.

Before you run the final check, do you expect it to confirm all three dependencies plus the protected key?

  • Verify the repository, dependencies, and environment variable by running these commands:
python -m pip list
python -c "print('OPENROUTER_API_KEY is set' if __import__('os').environ.get('OPENROUTER_API_KEY') else 'OPENROUTER_API_KEY is not set')"
git status

What should you see?

  • The package list includes `requests`, `openrouter`, and `tabulate`.
  • The environment check prints `OPENROUTER_API_KEY is set` without revealing the credential.
  • Git reports the current repository state from inside `jev-llm-grader`.

Setup check did not pass?

  • Repeat the dependency installation if a required package is absent from the list.
  • Repeat the export command if the environment check reports that the key is missing.
  • Return to the `jev-llm-grader` folder if Git cannot find a repository.

Want help reading the final output? Help me diagnose my benchmark setup check.

That's the setup locked in: your repository is ready with an active Python environment plus a protected OpenRouter key. Next, you'll make one Jev call and inspect its typed probability.

Make a Jev Grading Call

Your benchmark repository is ready from the previous step. The activated Python environment already has the packages this call needs.

Your OpenRouter key remains in an environment variable. The script can use it without storing the credential in your project files.

A written verdict hides how sure the grader is. In this step, you will use Jev to return a typed correctness grade with a probability.

In this step, get ready to:
  • Create a benchmark script that uses the existing OpenRouter key.
  • Ask Jev a typed correctness question.
  • Display the boolean grade with its probability.
Create the grading script

A Noul question asks whether a condition holds. Its answer is the probability of yes between 0 and 1.

For this benchmark, yes means the candidate answer agrees with the supplied knowledge. The script converts that probability into a boolean grade using a starter threshold of 0.5.

  • In the repository from earlier, use your editor's new-file control to create grade_answers.py.

You'll see grade_answers.py in the editor's file sidebar. This confirms that the script is inside the benchmark repository.

  • Paste the following code into grade_answers.py:
import os
from typesafe_sdk import TypeSafeClient

# Send Jev requests through OpenRouter with the existing key
client = TypeSafeClient(api_key=os.environ["OPENROUTER_API_KEY"], base_url="https://openrouter.ai/api")

# Keep the evidence, question, and candidate answer together
state = {"knowledge": "The capital of France is Paris.", "question": "What is the capital of France?", "answer": "Paris."}

# Ask a typed yes or no correctness question
result = client.system_one(
    model="jev-1.13",
    state=state,
    questions={
        "correct": {
            "type": "noul",
            "instructions": "Is the answer correct using only the knowledge?",
            "criteria": {"true": "The answer agrees with the knowledge.", "false": "The answer conflicts with the knowledge."},
        }
    },
)

# Turn the yes probability into a boolean grade
probability = result.answers["correct"].noul
verdict = probability >= 0.5
print(f"Correct: {verdict} | Probability: {probability:.2f}")

What does this code do?

  • The TypeSafeClient sends Jev requests through OpenRouter. It reads the API key from OPENROUTER_API_KEY.
  • The state dictionary keeps the knowledge beside the question and candidate answer.
  • The correct question uses the noul type. Jev returns the probability that the answer is correct.
  • The verdict comparison turns the probability into True or False.
  • Save grade_answers.py.
  • Confirm that the editor tab no longer shows an unsaved-change indicator.

Good work. Your benchmark now contains one complete typed grading request.

Can't find the saved script?

  • Check that grade_answers.py appears inside the benchmark repository from earlier.
  • Make sure the filename ends with .py.

Still stuck? Help me check why grade_answers.py is missing or not saving in my project repository.

✔️ Awesome, I've got everything!

Your complete grading script is saved in grade_answers.py.

ⓧ I'd like to double check the full code

import os
from typesafe_sdk import TypeSafeClient

# Send Jev requests through OpenRouter with the existing key
client = TypeSafeClient(api_key=os.environ["OPENROUTER_API_KEY"], base_url="https://openrouter.ai/api")

# Keep the evidence, question, and candidate answer together
state = {"knowledge": "The capital of France is Paris.", "question": "What is the capital of France?", "answer": "Paris."}

# Ask a typed yes or no correctness question
result = client.system_one(
    model="jev-1.13",
    state=state,
    questions={
        "correct": {
            "type": "noul",
            "instructions": "Is the answer correct using only the knowledge?",
            "criteria": {"true": "The answer agrees with the knowledge.", "false": "The answer conflicts with the knowledge."},
        }
    },
)

# Turn the yes probability into a boolean grade
probability = result.answers["correct"].noul
verdict = probability >= 0.5
print(f"Correct: {verdict} | Probability: {probability:.2f}")
Run the first grade

The saved script sends one candidate answer through the typed grading path. Its final line makes both parts of the grade visible in your terminal.

Before you run the script, picture the verdict you expect for the sample answer.

  • Switch back to the terminal with the activated virtual environment.
  • Run the saved benchmark script with this command:
python grade_answers.py

How to read the grade

You have your first typed Jev grade. The terminal now shows a boolean verdict beside its probability.

  • You'll see Correct: True because the candidate answer agrees with the supplied knowledge.
  • You'll see Probability: followed by a number between 0.00 and 1.00.
  • The exact probability can vary between calls.

Call not returning a grade?

  • Confirm that the project's virtual environment is still activated.
  • Check that OPENROUTER_API_KEY is still available in the current terminal session.
  • Confirm that your internet connection can reach OpenRouter.

Need another pair of eyes? Help me debug why grade_answers.py is not returning a Jev verdict and probability.

Your one-answer grading path now works from request to typed result. Next up, you'll build the labeled answer set that turns this single check into a benchmark.

Create the Labeled Answer Set

Your first Jev request proved that the grader can return a typed verdict with a probability. That result covers one answer.

A grader comparison needs fixed ground truth before either grader sees the work. In this step, you'll build a labeled set of 20 examples.

In this step, get ready to:
  • Define a consistent structure for the labeled answers.
  • Add 20 mixed correct and incorrect examples to the existing script.
  • Print the complete answer set as a table.
Define the answer schema

A consistent schema gives both graders the same evidence. It also gives your benchmark a known result for every answer.

What makes a usable labeled example?

Each example needs four pieces of evidence:

  • A unique ID keeps the result traceable across both graders.
  • A question or reference context defines what a correct answer must contain.
  • A candidate answer gives the grader the exact text to evaluate.
  • A known correct or incorrect label provides the result used to measure agreement.
  • Choose exactly 20 examples with both correct and incorrect candidate answers.
  • Resolve the known label for every example before either grader runs.
Add the labeled dataset

The existing grade_answers.py script already loads OPENROUTER_API_KEY from the environment. The new dataset section keeps that setup untouched.

  • Switch back to grade_answers.py from the previous step.
  • Add an in-script dataset display section containing your 20 examples in the four-field structure above.
  • Save grade_answers.py.

Before you run the script, how many data rows do you expect the table to show?

  • Run the script with the same terminal command you used in the previous step.

You'll see the existing single-answer Jev result. You'll also see a table containing 20 labeled data rows.

Table missing rows or fields?

  • Count the records in the in-script dataset if the table contains fewer than 20 data rows.
  • Use the same four fields in every record if a column is missing from the table.
  • Check each ID for a duplicate if two results are difficult to distinguish.

Still stuck? Help me debug a Python dataset table that is missing records or fields.

Verify the 20 examples

A table can print successfully while still containing weak benchmark data. This final review checks that every row can support a fair comparison.

  • Count the table's data rows.
  • Confirm that every ID appears once.
  • Verify that every candidate answer can be judged from its question or reference context.
  • Compare every known label with its supporting context.
  • Confirm that the table contains both correct and incorrect labels.
  • Fix every incomplete or mislabeled record.
  • Save grade_answers.py.

Before the final run, do you expect every example to appear as one complete row?

  • Run the script again with the same terminal command from the previous step.

That's your benchmark locked in. You'll see exactly 20 data rows with all four required fields.

Dataset check still failing?

  • Remove any repeated ID before running the script again.
  • Complete any record that lacks context or a known label.
  • Correct any label that conflicts with the reference context.

Need another pair of eyes? Help me audit my labeled answer set for incomplete records or inconsistent labels.

Your fixed answer set is ready for repeatable evaluation. Next, you'll grade every example with Jev while recording its probability, cost, and latency.

Grade Answers with Jev

The labeled benchmark now covers 20 answers with known labels. You have also proved that Jev can grade one answer with a typed verdict plus a probability.

A benchmark needs the same grading path applied across every example. This step turns your single request into a repeatable batch.

The batch records evidence for each answer in a CSV file. It also captures the cost reported by OpenRouter.

In this step, get ready to:
  • Send all 20 labeled answers through the existing Jev grader.
  • Measure one latency value for every Jev request.
  • Write one complete result row per answer to jev_results.csv.
Grade the full dataset

Batch grading applies one consistent decision process to every labeled example. Reusing the existing Jev request keeps the comparison fair across all 20 answers.

  • In grade_answers.py, wrap the existing Jev request in a loop over all 20 dataset records.
  • Save grade_answers.py.

The next run uses your OpenRouter balance. Keeping the existing 20-example dataset prevents accidental extra requests.

Before you run the batch, how many verdict plus probability pairs do you expect to see?

  • Run grade_answers.py again using the same method as earlier.

You should see one Jev verdict plus probability pair for each of the 20 example IDs. That confirms every labeled answer reached the same grading path.

What Does the Batch Prove?

Each printed result connects a dataset example to Jev's typed decision. Processing all 20 examples gives you enough rows for the comparison later in the project.

Does the Batch Stop Early?

  • Confirm the active terminal still has access to OPENROUTER_API_KEY.
  • Check that the loop passes one complete dataset record into the existing Jev request each time.

If one example keeps stopping the batch, ask for help tracing that record through the loop.

Capture latency and cost

A useful benchmark keeps operational evidence beside every grade. Latency captures how long each request takes.

Per-call cost shows how much of your OpenRouter balance each decision uses. Recording both values now makes the final comparison reproducible.

  • Add a stored latency value to each printed Jev result in grade_answers.py.
  • Save grade_answers.py.

Before you run the updated batch, do you expect every request to take exactly the same amount of time?

  • Run grade_answers.py again using the same method as earlier.

You should see a numeric latency beside each Jev result. The values can differ because every answer is graded through a separate request.

The same result record also needs the amount reported for its request. Keeping that value beside the latency preserves the evidence at answer level.

  • Extend each printed result record with the cost reported by OpenRouter.
  • Save grade_answers.py.

Before you run the batch again, do you expect each result to contain both operational measurements?

  • Run grade_answers.py again using the same method as earlier.

You should now see latency plus reported cost for every graded answer. Each in-memory result contains the evidence needed for export.

Missing a Measurement?

  • Confirm the latency measurement surrounds the existing Jev request.
  • Confirm the cost comes from the response for the same answer.
  • Check that both measurements are attached before the result record is printed.

If a value remains empty, ask for help following the response into the result record.

Write the Jev results file

The in-memory results disappear when the script exits. Writing them to jev_results.csv creates a durable input for the comparison step.

Each row needs the example ID plus its known label. It also needs the Jev verdict, Jev probability, latency, plus cost.

  • Add a CSV export that writes one complete row per answer to jev_results.csv after the batch finishes.
  • Save grade_answers.py.

Before you run the final check, how many data rows should the results file contain?

  • Run grade_answers.py again using the same method as earlier.

When the batch finishes, you should see jev_results.csv in the project repository. Its presence confirms the in-memory results were exported.

  • Select jev_results.csv from your editor's project file list.

You should see one header row plus 20 data rows. Every data row should contain an example ID, known label, Jev verdict, Jev probability, latency, plus cost.

Is the CSV Missing Rows?

  • Confirm the export runs after the full grading loop finishes.
  • Check that every completed Jev result is added to the collection passed into the CSV writer.
  • Confirm the writer creates one row for every stored result.

If the row count still falls short, ask for help comparing the dataset count with the exported result count.

That is the Jev batch complete. Next up, you will grade the same 20 answers with an LLM judge so both graders can be compared on identical data.

Grade Answers with an LLM Judge

Your Jev batch now grades every answer with a typed verdict. Each result also carries a probability.

The benchmark still needs a second grading path. An LLM judge returns a written verdict that can explain its decision.

Both graders receive the same 20 examples through OpenRouter. Matching inputs keeps the comparison fair.

In this step, get ready to:
  • Define a response contract for the LLM judge prompt.
  • Run all 20 labeled answers through the LLM judge.
  • Save one complete judge record per answer in llm_results.csv.
Define the judge response contract

An LLM judge writes prose instead of returning a guaranteed typed field. The prompt needs a clear response contract so your script can preserve the explanation while extracting a grade when one appears.

What should the judge receive?

  • The judge receives each example's question.
  • The judge also receives the candidate answer.
  • The known label remains outside the prompt.
  • The response states a clear grade when possible.
  • The response explains the grading decision in prose.
  • The script preserves the complete written verdict.
  • The parser records a grade only when the verdict states one clearly.
  • Use this response contract as the design checklist for the LLM judge workflow.
Add and run the judge workflow

The existing Jev workflow already establishes the order of the dataset. The LLM workflow mirrors that order so each result can be matched by example ID.

What should each result row contain?

A CSV row preserves the judge output beside the measurement data for one answer.

  • The example ID links the row to the original labeled example.
  • The known label provides the benchmark answer.
  • The written verdict preserves the full judge response.
  • The parsed grade stores a usable result when the response states one clearly.
  • The latency records how long the request took.
  • The cost records the amount returned for the request.

Your 20-request batch keeps its OpenRouter credit use visible through the recorded cost field.

  • Edit grade_answers.py in the code editor you used earlier by adding the LLM judge batch workflow described above.
  • Save grade_answers.py.

Before you rerun the benchmark, do you expect every written verdict to contain a grade that the parser can recognize?

  • Rerun the updated grade_answers.py from the activated virtual environment using the same method that generated jev_results.csv.

When the run finishes, you should see llm_results.csv in the project repository. The file should contain one result for each of the 20 answers.

You now have a reproducible second grading path for the complete labeled dataset.

Missing judge results?

  • Confirm that the LLM loop processes the same 20 examples as the Jev loop.
  • Check the response text extraction when the written verdict column is blank.
  • Check the CSV-writing logic when llm_results.csv does not appear.
  • Confirm that OPENROUTER_API_KEY remains available in the activated environment when requests stop.

Still stuck? Help me troubleshoot my incomplete LLM judge results.

Verify the LLM result rows

Written verdicts need an integrity check before they can support the final comparison. Every source example needs one complete result row.

Before you inspect the file, do you expect its example IDs to match jev_results.csv exactly?

  • Open llm_results.csv using the table-viewing method you used for the Jev results.
  • Count the data rows in llm_results.csv.
  • Compare the example IDs with the IDs in jev_results.csv.
  • Inspect the written verdict column for complete judge responses.
  • Check the parsed grade column for verdicts that state a clear grade.
  • Review the latency value in every row.
  • Review the cost value in every row.

You should see 20 data rows. Their example IDs should match the 20 rows in jev_results.csv.

Every row should contain its known label. Every row should also preserve the complete written verdict.

A parsed grade should appear when the verdict states one clearly. Latency and cost should appear for every completed request.

That completes your two-grader dataset. Both grading paths are now ready for a direct comparison.

Both result files are ready. Next, you'll compare agreement with the labels plus total cost and median latency in one summary table.

Compare Grader Performance

Your 20 answers have now passed through Jev plus an LLM judge using OpenRouter. Each grader recorded its decisions across the same labeled set.

Individual rows make the overall trade-offs difficult to see. This step turns those results into a side-by-side comparison of agreement, cost, and latency.

In this step, get ready to:
  • Measure how often each grader agrees with the known labels.
  • Calculate the total cost for each grader. Capture its median latency in the same summary.
  • Save the side-by-side results in comparison_summary.csv.
Confirm the inputs match

A fair comparison needs identical input rows. Both CSV result files should cover the same 20 example IDs with the same known labels.

  • Return to the editor you used for grade_answers.py earlier.
  • Select jev_results.csv from the editor's file list.
  • Confirm the file contains one result row for each of the 20 labeled answers.
  • Switch to llm_results.csv from the editor's file list.
  • Confirm its example IDs match the IDs in jev_results.csv.
  • Confirm its known labels match the labels in jev_results.csv.

That check gives you a fair baseline. Both graders cover the same labeled set.

How Are the Metrics Calculated?

  • Agreement rate counts grades that match the known label. It divides that count by all 20 examples.
  • Total cost adds the recorded per-call costs for one grader.
  • Median latency sorts the 20 recorded response times. It averages the two values in the middle.
Calculate the comparison

Agreement measures grading performance against your labels. Cost shows the spend behind that performance. Median latency represents the typical response time without letting one slow request dominate the result.

  • Switch back to grade_answers.py in your editor.
  • Write a final comparison section after both grading workflows finish. Use one row per grader with the three metrics defined above.
  • Save grade_answers.py.

Before you run the benchmark, which grader do you expect to match more labels? Which grader do you expect to respond faster?

  • Run grade_answers.py using the same terminal method you used earlier.

You should see a side-by-side table with one row for Jev. A second row should represent the LLM judge.

Each row should report agreement against the known labels. It should also report total cost plus median latency.

Comparison Values Look Wrong?

  • Confirm the Jev agreement calculation compares the typed verdict with the known label.
  • Confirm the LLM judge calculation compares the parsed grade with the known label.
  • Check that each grader uses its own cost values plus its own latency values.

Need help tracing the calculations? Help me debug my grader comparison metrics.

Export the summary

A saved summary keeps the aggregate results beside the detailed grader outputs. It also makes the benchmark easier to review without repeating the calculations manually.

  • Return to the final comparison section in grade_answers.py.
  • Add an export after the displayed table that writes the same two summary rows to comparison_summary.csv.
  • Save grade_answers.py.

Before the final run, predict which grader will have the lower total cost. Predict which grader will have the lower median latency.

  • Run grade_answers.py using the same terminal method as before.

That is the comparison working. You should see both grader rows in the terminal with agreement, total cost, and median latency.

  • Select comparison_summary.csv from your editor's file list.
  • Confirm the file contains one row for Jev.
  • Confirm the file contains one row for the LLM judge.
  • Confirm both rows include agreement against the labels.
  • Confirm both rows include total OpenRouter cost.
  • Confirm both rows include median latency.

Final Repository Check

  • The repository retains grade_answers.py with both grading workflows plus the aggregate comparison calculations.
  • The repository retains jev_results.csv with 20 detailed Jev results.
  • The repository retains llm_results.csv with 20 detailed LLM judge results.
  • The repository contains comparison_summary.csv with the side-by-side summary.

Summary File Missing?

  • Confirm the export runs after both summary rows have been calculated.
  • Check that the export filename is exactly comparison_summary.csv.
  • Confirm the benchmark reaches the comparison section after both grading workflows finish.

Still missing the file? Help me debug the comparison summary export.

Your benchmark is complete. The repository now preserves both detailed result sets plus a reproducible comparison of grading performance, cost, and speed.

Secret mission

Investigate Low-Confidence Grades

The summary table tells you which grader performed better overall. This challenge investigates Jev's lowest-probability decisions and turns the evidence into a human-review policy.

Clean Up Your Resources

Clean Up Your Resources

Your script and result files live locally, so keeping them creates no ongoing charge. Choose whether to keep the repository, pause your shell session, or delete the local files.

Resources you used:

  • Your local project repository.
  • Your benchmark script: grade_answers.py.
  • Your generated CSV outputs: jev_results.csv, llm_results.csv, and comparison_summary.csv.
  • Your OpenRouter API key exposed through OPENROUTER_API_KEY.

Keep everything running

No action needed. Choose this if you plan to rerun the benchmark or keep its results as evidence of your evaluation work.

  • Keep the project repository with grade_answers.py.
  • Keep jev_results.csv for the typed Jev grades.
  • Keep llm_results.csv for the written LLM judge verdicts.
  • Keep comparison_summary.csv for the agreement, cost, and latency comparison.
  • Keep the OpenRouter API key active only while you expect to run more benchmarks.
  • Run grade_answers.py only when you intend to use OpenRouter credit.

Pause - I'll come back to this later

This option ends the active shell session while preserving the repository for your next benchmark run.

  • Wait for any running benchmark to finish.
  • Close the terminal window that contains the activated Python virtual environment.
  • Keep the project repository in its current location.
  • Open your OpenRouter account's key-management page.
  • Apply a spending cap that matches the amount of credit you are comfortable using.
  • Confirm that grade_answers.py and all three result files remain in the repository.

Delete - I don't want to use this again

This option removes every local benchmark artifact. It also closes the credential used for API requests.

Save What You Need

Deleting the repository removes the benchmark script plus its generated results.

Revoking the API key prevents that credential from authorizing future requests.

macOS

  • Press Cmd+Space to open the search bar.
  • Type Finder into the search bar.
  • Press Enter to open Finder.
  • Navigate to the parent location that contains the project repository from earlier.
  • Select the project repository folder.
  • Drag the selected folder into the system trash.
  • Right-click the system trash icon in the Dock.
  • Choose the option that empties the system trash.

You should no longer see the project repository in its former parent location.

Windows

  • Press the Windows key to open the search bar.
  • Type File Explorer into the search bar.
  • Press Enter to open File Explorer.
  • Navigate to the parent location that contains the project repository from earlier.
  • Select the project repository folder.
  • Drag the selected folder into the system trash.
  • Right-click the system trash icon on the desktop.
  • Choose the option that empties the system trash.

You should no longer see the project repository in its former parent location.

  • Open your OpenRouter account's key-management page.
  • Revoke the API key that supplied OPENROUTER_API_KEY for this project.
  • Confirm that the credential no longer appears as active.
  • Close the terminal session to clear its current OPENROUTER_API_KEY value.

You have closed the loop cleanly. The local benchmark is removed from your machine.

Nice Work!

Nice Work!

You made it! Your Python benchmark now sends 20 labeled answers through OpenRouter. It compares probabilistic grades from Jev with verdicts from an LLM judge.

You've learned how to:

  • Build a 20-example labeled dataset that challenges both graders with correct examples. The same dataset also includes incorrect examples.
  • Save per-answer grading evidence in two CSV files. jev_results.csv records typed verdicts with probabilities. llm_results.csv records written judge verdicts. Both files preserve latency. Both files preserve cost.
  • Create a side-by-side benchmark summary in comparison_summary.csv. It reports agreement against the known labels. It reports total cost. It reports median latency.
  • Secret Mission: Extend the benchmark through an optional challenge that pushes your evaluation skills further.

Ready to quiz yourself?