Score AI Outputs With a Judge Model

Build a calibrated AI judge that scores agent outputs against your criteria.

Introduction

30 Second Summary

A summary can sound convincing while missing the details that matter most. Checking every answer by hand becomes slow once your prompt produces a growing batch of results.

In this project, you will build a scoring script that uses a cheaper judge model through OpenRouter to evaluate ten saved outputs against your own criteria. You will compare its scores with your labels until the pass threshold matches your judgement on at least eight outputs.

What You'll Build

You will run one script and see ten AI outputs scored with reasons before a calibration table reveals exactly where the judge agrees with you.

By the end of this project, you'll have:

A reusable scoring script that evaluates a batch of outputs against criteria written in plain language. Each result includes a score plus the judge's reason.
A calibration table that places judge decisions beside your human labels. Its disagreement count shows how well the threshold reflects your judgement.
A cost comparison between two judge models. You can see whether the cheaper option preserves enough agreement to use regularly.
Secret Mission: An optional challenge to push your skills further.

Are there any prerequisites?

You need ten saved outputs from a prompt you already use. You also need basic Python or Node experience plus an OpenRouter key with a small balance.

Before We Start

Scoring AI outputs needs a reliable place to run code. It also needs access to a judge model that did not produce the outputs being evaluated.

In this step, you will prepare a local Python workspace. You will configure secure access to two judge models through OpenRouter.

In this step, get ready to:

Prepare a local Python workspace.
Configure OpenRouter access outside the project files.
Choose two judge models for your target prompt.

Prepare your Python workspace

Python runs the scoring scripts you build later. A version check confirms that your terminal can start the interpreter.

macOS

Check whether Python is available by running this command:

python3 --version

What does this command do?

The command starts Python long enough to print its installed version. It exits without opening the interactive interpreter.

Windows

Check whether Python is available by running this command in PowerShell:

py --version

What does this command do?

The Python launcher prints the version selected for your current Windows environment. It exits as soon as the version appears.

Choose the tab that matches your version-check result.

✔️ I see a Python version

Your terminal can start Python. That gives the project a working runtime for every scoring script.

ⓧ Command not found

Python needs to be installed before the workspace can run scripts.

Download Python from the official Python downloads page.
Run the downloaded installer for your operating system.
Approve the operating system permission request if one appears.
Return to your terminal after the installation finishes.
Repeat the version check from the platform tab above.

Still unable to start Python?

Close your terminal after the installation. Start a new terminal session so it can detect the new Python command.

Still stuck? Help me troubleshoot my Python installation.

A dedicated folder keeps the scoring scripts separate from your other files. You will place it on your Desktop so it stays easy to find.

macOS

Create the local workspace on your Desktop by running these commands:

mkdir -p ~/Desktop/ai-output-scoring
cd ~/Desktop/ai-output-scoring
pwd

What do these commands do?

The first command creates the ai-output-scoring folder on your Desktop.
The second command makes that folder your current location.
The final command prints the full workspace path.

You should see a path ending in Desktop/ai-output-scoring.

Windows

Create the local workspace on your Desktop by running these commands in PowerShell:

New-Item -ItemType Directory -Path "$([Environment]::GetFolderPath('Desktop'))\ai-output-scoring"
Set-Location -Path "$([Environment]::GetFolderPath('Desktop'))\ai-output-scoring"
Get-Location

What do these commands do?

The first command finds your Desktop location before creating the ai-output-scoring folder.
The second command makes that folder your current location.
The final command displays the full workspace path.

You should see a path ending in Desktop\ai-output-scoring.

Workspace path not showing?

Check that the folder name is exactly ai-output-scoring. A spelling difference creates or targets another location.

Still stuck? Help me create my local scoring workspace.

Configure OpenRouter access

One OpenRouter API key can authenticate requests to both judge models. The key belongs in an environment variable so your project files never contain the credential.

Install the OpenRouter Python SDK by running this command:

pip install openrouter

What does this command do?

The command installs the OpenRouter package for Python. Your future scripts use this package to send requests through one model gateway.

The installation output should finish without a failure message. That visible result confirms that the model-calling method is available.

Package installation failing?

Confirm that your Python version check succeeded before retrying the installation. A package installer linked to another Python installation can cause environment-specific failures.

Still stuck? Help me install the OpenRouter Python SDK.

Keep your credential protected

This step touches a live credential plus a paid balance. You stay in control by setting a low credit limit on the key.

OpenRouter displays the key once. Keep it out of screenshots plus project files.

Visit the OpenRouter API keys page.
Sign in to your OpenRouter account.
Create a new API key named ai-output-judge.
Set a low credit limit for the key.
Copy the key when OpenRouter shows it.

Set the key only in your current terminal session. The value stays outside the ai-output-scoring folder.

macOS

Replace the plain text placeholder with your key before running this command:

export OPENROUTER_API_KEY="sk-or-v1-YOUR-API-KEY"

What does this command do?

The command stores the credential in OPENROUTER_API_KEY for the current terminal session. Python can read it without placing the key inside a project file.

Windows

Replace the plain text placeholder with your key before running this command in PowerShell:

$env:OPENROUTER_API_KEY="sk-or-v1-YOUR-API-KEY"

What does this command do?

The command stores the credential in OPENROUTER_API_KEY for the current PowerShell session. Python can read it without placing the key inside a project file.

Key unavailable to your scripts?

Run future scripts from the same terminal session where you set the environment variable. A different terminal session does not inherit this temporary value.

Never paste your key into a help request. Help me check my environment variable without revealing its value.

Choose your evaluation inputs

Judge calls run once for every output in your batch. Your model choices determine the cost of each calibration run.

Visit the OpenRouter model catalog.
Exclude the model that produced the outputs you plan to evaluate.
Compare the displayed input prices.
Compare the displayed output prices.
Select a low-cost model as your primary judge.
Record the primary judge model ID here: your primary judge model ID.
Select a cheaper model as your second judge.
Record the cheaper judge model ID here: your cheaper judge model ID.

Why choose two judge models?

The primary judge gives you a baseline for agreement with your labels. The cheaper judge shows whether you can preserve that agreement at a lower cost.

Keeping the output-producing model out of the judge role reduces the chance of one model grading its own habits.

Choose one prompt whose outputs you want to evaluate.
Record the selected prompt here: the prompt whose outputs you want to evaluate.

Before you run the final check, do you expect your terminal to report the Python runtime plus the new workspace path?

macOS

Verify the runtime plus your current workspace by running these commands:

python3 --version
pwd

What do these checks prove?

The first command confirms that Python remains available.
The second command confirms that your terminal is inside the local scoring workspace.

You should see a Python version followed by a path ending in Desktop/ai-output-scoring.

Windows

Verify the runtime plus your current workspace by running these commands in PowerShell:

py --version
Get-Location

What do these checks prove?

The first command confirms that Python remains available.
The second command confirms that PowerShell is inside the local scoring workspace.

You should see a Python version followed by a path ending in Desktop\ai-output-scoring.

Final checks not matching?

Return to the platform-specific workspace commands if the path points somewhere else. Keep using the same terminal session so the OpenRouter credential remains available.

Still stuck? Help me verify my Python scoring workspace safely.

✔️ Awesome, I've got everything!

Your local workspace is ready. Your OpenRouter credential stays outside the project files.

You have also selected a primary judge plus a cheaper comparison judge.

ⓧ I'd like to double check the full code

There is no application code to compare yet. Double check that your setup matches this state.

Credentials and access configured for a primary low-cost judge model and a second, cheaper judge model. Secrets are kept outside project files.
Local workspace created for the AI-output scoring project.
The selected primary model is your primary judge model ID.
The selected cheaper model is your cheaper judge model ID.
The target prompt is the prompt whose outputs you want to evaluate.

Your scoring workspace now has a runtime plus secure model access. Next, you will collect ten outputs from your selected prompt.

Collect Evaluation Outputs

An evaluation is only useful when its examples reflect how your prompt behaves across repeated runs. A single polished response can hide inconsistency in the model's other answers.

In this step, you'll run your selected prompt ten times. You'll save every full response in outputs.jsonl with a stable ID.

In this step, get ready to:

Prepare a stable ID sequence for the batch.
Collect ten full outputs from the same prompt.
Verify that outputs.jsonl contains ten complete records.

Create the collection file

JSON Lines stores one JSON record on each physical line. This format lets later steps process each output separately.

Create outputs.jsonl inside your existing local project workspace with your file editor.
Confirm outputs.jsonl appears in the workspace file list.

Good start. outputs.jsonl is now the single source for the batch you will evaluate.

Why use stable IDs?

A stable ID keeps each response tied to its later judge score. The same ID also connects the response to your human label during calibration.

Reserve output-01 through output-10 for the ten runs.
Keep the selected prompt unchanged across the batch.

These choices give every run a fixed place in a comparable batch.

Use the same generating model for every run.

Collect ten complete responses

Each run is one sample of how your selected prompt behaves. The original generated text becomes the evidence that your judge evaluates later.

Run your selected prompt once with the generating model.

You'll see the first response produced by your normal workflow.

Copy the entire first response without rewriting it.
Store the response as one record in outputs.jsonl with output-01 as its stable ID.
Confirm the first record contains its stable ID plus the full generated text.

That first sample is safely captured. You now have a repeatable pattern for the remaining nine.

Repeat the same run-to-record process for output-02 through output-10.
Save outputs.jsonl after adding the tenth record.

Does a saved response look incomplete?

Return to the original model result when a record ends early.
Copy the response again from its first character through its final character.
Use help checking whether my JSONL records preserve each complete model response if the record structure still looks wrong.

Verify the saved batch

The next steps depend on a batch with exactly ten records. A missing ID would separate later scores from the output they belong to.

Before you inspect the file, do you expect the stable IDs to form one unbroken sequence?

Return to outputs.jsonl in your existing workspace.
Check that the IDs run from output-01 through output-10 with no gaps.
Check that each stable ID appears once.
Inspect every record for the full generated text.
Confirm each record occupies one physical line.

You'll see exactly ten records. Every record has one unique stable ID plus the full response from its run.

That's the collection complete. Your batch can now support repeatable scoring without losing track of which result earned each score.

Do the count or IDs look wrong?

Generate a fresh response for each missing ID.
Remove a duplicated record only after confirming that the same response was saved twice.
Use help repairing my ten-record evaluation batch if the sequence still contains gaps or duplicates.

Redact any sensitive generated text before capturing your checkpoint.

Your evaluation batch is ready. Next up, you'll write the plain-language criteria that your judge model uses to score the first output.

Define Your Judge Criteria

Your ten outputs give you a stable evaluation batch. A judge still needs a precise definition of quality.

Vague criteria pull scores toward the middle. The pass mark then becomes arbitrary.

In this step, you’ll define observable criteria. You’ll then score one output with your primary judge model.

In this step, get ready to:

Define plain-language pass criteria with a numeric scoring scale.
Store the primary judge settings without storing credentials.
Prove the judge path with one saved output.

Write observable criteria

An LLM judge compares each output with the same written rubric. Observable requirements help it distinguish a strong response from one that only sounds convincing.

What belongs in your criteria?

The task definition names what the original prompt asks each output to accomplish.
Three to five requirements describe evidence that should appear in a passing output.
Failure conditions identify problems that make an output unusable or misleading.
A scale from 1 to 5 gives every score a distinct meaning.
The response contract requires one whole-number score followed by one concise written reason.

Use concrete checks that you can apply yourself. Examples include covering every required point or avoiding unsupported claims.

Open the existing project workspace with your usual code editor’s folder-opening control.
Create `criteria.md` inside the existing project workspace with the editor’s new-file control.

You should see `criteria.md` listed beside `outputs.jsonl` in the project workspace.

How should the scale work?

A score of 1 means the output fails the core task.
A score of 2 means the output has substantial problems.
A score of 3 means the output needs meaningful revision.
A score of 4 means the output passes with only minor issues.
A score of 5 means the output fully satisfies the criteria.

Use 4 as the initial pass threshold. You’ll calibrate that threshold against your own labels later.

Write `criteria.md` so it covers the task definition and the observable requirements.
Save `criteria.md`.

The saved file should now explain what makes an output pass. It should also define the expected score and reason format.

Configure the primary judge

A JSON configuration keeps the model choice separate from the scoring logic. This separation makes a later model swap a small configuration change.

Record the selected primary judge identifier as your primary judge model identifier.
Create `judge_config.json` inside the existing project workspace with the editor’s new-file control.

You should now see `judge_config.json` beside `criteria.md` in the project workspace.

What should the configuration store?

The primary judge model identifier selects the model used for this first evaluation.
The score range has a minimum of `1` and a maximum of `5`.
The initial pass threshold is `4`.
The model-call settings match the settings supported by your existing access method.

Keep every credential outside this file. The configuration only needs non-secret values.

Add the configuration values above to `judge_config.json` using your primary judge model identifier.
Save `judge_config.json`.

Your editor should report no JSON syntax problems. The file should contain no secret credentials.

Score one output

A one-record smoke test keeps the feedback loop small. It proves that the criteria and model response work together before you score the full batch.

Create `score_outputs.py` inside the existing project workspace with the editor’s new-file control.

You should see `score_outputs.py` listed with the two configuration files.

What should the scorer do?

The script loads `criteria.md` and `judge_config.json`.
The script reads one record from `outputs.jsonl` without changing its stable output ID.
The script sends the full output text with the judging criteria to the configured primary model.
The script reads a numeric score and a written reason from the judge response.
The script attaches the request-cost information exposed by the existing model-call method.

Implement the one-record scoring path in `score_outputs.py` by reusing the working model-call method from your workspace.
Save `score_outputs.py`.

Before you run it, do you expect the judge’s reason to point to a specific criterion?

Run `score_outputs.py` with the same Python method you already use in this workspace.

You should see the stable output ID with a numeric score. You should also see a written reason.

The same result should include request-cost information. This gives you a complete evaluation record for one output.

That first scored record proves your judge can apply the rubric and explain its decision.

Missing a score or reason?

Check that `criteria.md` asks for one whole-number score followed by one written reason.
Confirm that `judge_config.json` contains the selected model identifier without any credentials.
Inspect the model-call result through the same response-handling method that already works in your workspace.
Still stuck? Help me debug my one-output judge script.

✔️ Awesome, I’ve got everything!

Great. Your criteria and primary judge configuration are saved.

Your scorer can now evaluate one output with a visible score and reason. It also preserves the output ID and request cost.

ⓧ I’d like to double check the full code

Compare your workspace against these completed artifact checkpoints.

`criteria.md` contains plain-language pass criteria with the 1 to 5 scoring scale.
`criteria.md` defines the expected numeric score and written reason format.
`judge_config.json` stores the primary judge identifier and the score range.
`judge_config.json` stores the initial pass threshold and non-secret model-call settings.
`score_outputs.py` loads `outputs.jsonl` and `criteria.md`.
`score_outputs.py` returns the output ID and numeric score.
`score_outputs.py` returns the written reason and request-cost information.
`outputs.jsonl` still contains exactly ten records with stable output IDs.

Your judge can now score one saved output against a reusable rubric. Next up, you’ll run the same evaluation across all ten outputs and compare their scores.

Score the Full Batch

A single result proves that your scoring path works. It does not reveal whether the judge model applies your criteria consistently across ten different outputs.

In this step, you'll score the full batch with your primary judge model. You'll save a table that keeps every output ID beside its score, reason, model identifier, cost, and timestamp.

In this step, get ready to:

Extend the scoring script to process every saved output.
Run the primary judge across the full batch.
Confirm the table contains one complete result for every output ID.

Extend the scorer across the batch

Batch evaluation sends each saved record through the same scoring path. Using one path keeps the comparison focused on output quality.

What should the batch flow record?

The loop sends each of the ten records through the existing primary-judge call.
Every result keeps the stable output ID, numeric score, written reason, configured model identifier, per-output cost, and timestamp.
The CSV writer stores the ten records in judge_results.csv.
The console table displays the output ID, score, reason, and cost for quick review.

In score_outputs.py from earlier, extend the existing single-output path with the batch flow described above.
Save score_outputs.py.

Run the full batch

Each judge request adds one result with its own cost entry. Running the batch turns those ten requests into a table you can compare.

Before you run the batch, how many data rows do you expect the printed table to contain?

Run the updated score_outputs.py using the same method that produced your earlier single-output score.

You'll see a table with ten data rows. Each row shows an output ID, score, reason, and cost.

That is the full scoring pass complete. Every saved output now has a comparable judge result.

Seeing fewer than ten rows?

Confirm outputs.jsonl still contains exactly ten records.
Use the last printed output ID to locate the record where the run stopped.
Check that the script appends one result after each successful judge response.
Still stuck? Help me troubleshoot my incomplete batch-scoring run.

Verify every saved result

The console table gives you a quick view. judge_results.csv preserves enough detail to audit the scores later.

Before you compare the files, do you expect every stable ID in outputs.jsonl to appear exactly once in the results?

Open judge_results.csv from the file list in your existing project workspace.
Compare its output IDs with the ten stable IDs in outputs.jsonl.
Confirm that each output ID appears exactly once.
Confirm that every result includes a numeric score.
Confirm that every result includes a written reason.
Confirm that every result identifies the configured primary judge model.
Confirm that every result records its token or request cost.
Confirm that every result records a timestamp.

You should find ten result rows covering all ten stable IDs. The printed table should show the same output ID, score, reason, and cost values saved in judge_results.csv.

You now have a complete batch of judge results that you can inspect and compare. The stable IDs keep every decision traceable to its original output.

Your primary judge has scored the complete batch. Next, you'll label those same outputs yourself to create a human baseline for disagreement.

Calibrate Against Your Labels

Your primary judge has scored all ten outputs. The score table captures one model's opinion.

Calibration checks that opinion against decisions you trust. Your human labels become the reference for that check.

In this step, you'll label each output as pass or fail. You'll then compare those labels with the judge's decisions at the current threshold.

In this step, get ready to:

Assign a human pass or fail label to every output.
Extend the scoring script with threshold-based comparisons.
Generate a calibration table with agreement totals.

Label each output yourself

A human label records your own pass or fail decision for one output. The human_labels.csv file stores those decisions in a two-column CSV table.

Consistent labels come from applying criteria.md to every output.

Create human_labels.csv beside outputs.jsonl with the same file-creation method you used earlier.
Set its first row to the output_id column followed by the human_label column.

You should see human_labels.csv beside the other project files. Its first row should contain the two column names.

Review the first output in outputs.jsonl against criteria.md.
Record that output's stable ID in the output_id column.
Enter pass or fail in the human_label column.

The first data row should now connect one stable output ID to one human decision.

Repeat this process for the remaining nine outputs.
Save human_labels.csv.
Confirm the file contains ten data rows below its header row.

Every stable ID from outputs.jsonl should appear exactly once. Every label should contain only pass or fail.

Missing or duplicate labels?

Copy each stable ID directly from outputs.jsonl to avoid typing differences.
Remove any duplicate row before counting the ten labels again.
Check each value in human_label for spelling differences or extra spaces.
Get tailored help with checking your human-label file.

Add the calibration phase

The numeric threshold turns each primary score into a binary judge decision. Comparing that decision with the human label exposes where the current pass mark aligns with your judgment.

How does the comparison work?

A primary score at or above the threshold becomes pass.
A primary score below the threshold becomes fail.
A matching pair receives the agreement status.
A different pair receives the disagreement status.
Each row in calibration.csv keeps the output ID, human label, primary score, judge decision, status, judge reason, and cost.
The final summary records the total agreement count plus the total disagreement count.

In score_outputs.py, add a calibration phase that implements every comparison rule above.
Save score_outputs.py.

Check the agreement baseline

The current threshold is your starting point. Its disagreement count shows how much calibration work remains.

Before you run the calibration phase, how many judge decisions do you expect to match your labels?

Return to the terminal panel from earlier.
Run the calibration phase of score_outputs.py with the same script-running method you used earlier.

You'll see the score table first. A calibration table should follow with ten comparison rows.

Each row should place the human label beside the threshold-derived judge decision. The table should finish with total agreement and disagreement counts.

Return to your project workspace.
Select calibration.csv.
Confirm all ten output IDs appear exactly once.
Confirm the agreement count plus the disagreement count equals ten.

Good work. Your judge now has a visible agreement baseline against your own decisions.

Calibration table incomplete?

Compare the IDs in human_labels.csv with the IDs in judge_results.csv for exact matches.
Confirm the calibration phase reads the current threshold from judge_config.json.
Count only output comparison rows when calculating the totals.
Get tailored help with debugging your calibration table.

Your human labels and current threshold are now measurable. Next, you'll tune the threshold until the judge matches at least eight of your ten labels before comparing a cheaper judge's agreement and cost.

Tune and Compare Judge Models

Your calibration table now shows where the primary judge model matches your labels. Those disagreement rows expose the problem this step solves. An arbitrary cutoff can turn useful scores into poor pass or fail decisions.

This step turns that cutoff into a calibrated threshold with at least eight agreements out of ten. Your human labels provide the reference point.

A second cheaper judge then faces the same batch. You can compare its agreement with the primary judge. You can also compare its cost.

In this step, get ready to:

Tune the primary threshold until it agrees with at least eight human labels.
Evaluate the ten outputs with the cheaper judge.
Compare both judges across agreement and cost.

Calibrate the primary threshold

The threshold converts each numeric score into a pass or fail decision. Testing every possible threshold reveals which cutoff best matches your human labels.

How does a threshold sweep work?

Mark a primary score as pass when it reaches the candidate threshold.
Mark a primary score as fail when it falls below the candidate threshold.
Compare every derived decision with its label in human_labels.csv.
Count the matches across all ten output IDs.

Switch back to score_outputs.py in the editor from earlier.
Add a threshold sweep that tests the full score range from judge_config.json.
Print a threshold table containing the agreement result for every candidate.
Save score_outputs.py.

Before you run the sweep, which threshold do you expect to match the most human labels?

Run score_outputs.py using the same method you used earlier.

You should see one row for each possible threshold. Each row shows its agreement count. It also shows its disagreement count.

Choose a threshold that reaches at least eight agreements.
Update the pass threshold in judge_config.json with that value.
Save judge_config.json.

Before you rerun the calibration, how many disagreements should remain at most?

Regenerate calibration.csv by running score_outputs.py with the primary judge selected.

You should see at least eight agreements across the ten outputs. The regenerated calibration.csv should use the calibrated threshold for every judge decision.

Strong result. Your primary judge now reaches the required agreement target against labels you assigned yourself.

No threshold reaches eight agreements?

A fresh primary run sends ten more model requests. Use it only after checking the saved data for alignment problems.

Check that every output ID in judge_results.csv matches an ID in human_labels.csv.
Confirm that scores meeting the threshold receive a pass decision.
Clarify criteria.md around any recurring disagreement pattern.
Rerun the primary judge before sweeping the thresholds again.
Still stuck? Help me diagnose why no threshold reaches eight agreements.

Score with the cheaper judge

A fair model comparison holds the evaluation conditions constant. The outputs stay fixed. The criteria stay fixed. The calibrated threshold stays fixed.

Why keep the same batch?

Changing only the judge model isolates the effect of that model choice. Any change in agreement or cost now comes from the judge you swapped.

Switch back to judge_config.json in the editor from earlier.
Add the identifier for the second cheaper judge model.
Save judge_config.json.

Your configuration should now retain both judge model identifiers. It should also retain the calibrated pass threshold.

Switch back to score_outputs.py.
Extend its model-selection path so second-model runs write to cheaper_judge_results.csv.
Preserve the primary result schema for every second-model row.
Save score_outputs.py.

This next run sends ten model requests. The fixed batch keeps that spend focused on a direct comparison.

Before you run the cheaper judge, do you expect it to agree on the same outputs as the primary judge?

Run the ten saved outputs through the second judge using the same script method from earlier.

You should see ten result rows from the second model. You should also see cheaper_judge_results.csv in the project workspace.

What should the result file prove?

Each row links one stable output ID to its numeric score. The row also retains the written reason from the cheaper judge.

The model identifier shows which judge produced the result. The cost plus timestamp make the run auditable.

Missing cheaper judge results?

Confirm that the second model identifier appears in judge_config.json.
Check that the selected model path writes to cheaper_judge_results.csv.
Keep your credentials outside every project file while checking model access.
Still stuck? Help me troubleshoot the second judge run.

Compare agreement and cost

Agreement shows how closely each judge follows your labels. Cost shows whether that behavior is affordable enough to repeat across future batches.

Switch back to score_outputs.py.
Add an agreement comparison that joins both judge result files to human_labels.csv using the calibrated threshold from judge_config.json.
Save score_outputs.py.

Before you compare the saved results, which model do you expect to match more human labels?

Run the saved-results comparison path in score_outputs.py.

You should see one agreement row for the calibrated primary judge. You should see another agreement row for the cheaper judge.

The comparison reads your saved result files. It does not need another model request.

Extend the comparison to save model_comparison.csv with each model's agreement count plus total cost.
Save score_outputs.py.

Before you run the final comparison, which judge do you expect to offer the stronger agreement-to-cost tradeoff?

Run the final comparison using the same script method from earlier.
Select model_comparison.csv in the editor from earlier.
Confirm that the file contains one row for each judge model.

Each model row should show an agreement result against all ten human labels. It should also show the model's total evaluation cost.

Keep criteria.md in the project workspace as your reusable evaluation specification.
Keep the calibrated threshold in judge_config.json for the next evaluation batch.

Comparison totals look wrong?

Confirm that both judge result files contain exactly ten stable output IDs.
Check that the comparison uses the same calibrated threshold for both models.
Verify that each total includes the cost from every row in its result file.
Still stuck? Help me debug the agreement and cost comparison.

You have turned a raw judge score into a calibrated evaluation process. Your reusable criteria plus threshold now let you compare future judges against evidence from your own decisions.

Secret mission

Audit Your Judge's Disagreements

Your judge meets the agreement target. The disagreement rows still show where it can fail. Audit those cases across both models before choosing the strongest agreement-to-cost tradeoff.

Clean Up Your Resources

Choose whether to keep, pause, or delete your local scoring workspace. Keeping its files is free because charges only come from new judge requests.

Resources you used:

The reusable evaluation specification in criteria.md plus the calibrated judge settings in judge_config.json.
The ten saved outputs in outputs.jsonl plus their human labels in human_labels.csv.
The generated reports in judge_results.csv, calibration.csv, cheaper_judge_results.csv, and model_comparison.csv.
The scoring automation in score_outputs.py.

Keep everything running

Everything can stay as it is. Choose this option if you want to score another batch or refine your criteria later.

Your criteria in criteria.md remain ready for the next evaluation batch.
Your calibrated threshold in judge_config.json remains available for reuse.
Your saved outputs plus calibration reports remain available for comparison.
Your model balance stays untouched while score_outputs.py is idle.

Pause - I'll come back to this later

Pausing stops an active scoring run while preserving every reusable artifact. This prevents the script from sending more judge requests.

Return to the terminal from earlier if score_outputs.py is still running.
Press Ctrl+C to stop the active scoring run.

If no scoring run is active, the project is already paused. Your criteria, chosen threshold, outputs, labels, and reports remain in the local workspace.

Delete - I don't want to use this again

Deleting the workspace is permanent, so use score_outputs.py to identify the correct folder before removing anything. Your credentials remain safe because they are stored outside the project files.

macOS

Finder can remove the complete workspace without relying on a folder path that varies between computers.

Press Cmd+Space to open the macOS search bar.
Type Finder and press Enter to open it.
Navigate to the location where you stored the local workspace from earlier.
Select the workspace folder containing score_outputs.py.
Drag the selected workspace folder onto Trash in the Dock.
Open Trash from the Dock.
Permanently remove the selected workspace using the available deletion control.

You should no longer see the workspace folder in its previous location or in the system trash.

Windows

File Explorer can remove the complete workspace from the location where you saved it.

Press the Windows key to open the Windows search bar.
Type File Explorer and press Enter to open it.
Navigate to the location where you stored the local workspace from earlier.
Select the workspace folder containing score_outputs.py.
Press Delete to move the selected workspace to the system recycle bin.
Open the system recycle bin.
Permanently remove the selected workspace using the available deletion control.

You should no longer see the workspace folder in its previous location or in the system recycle bin.

Nice Work!

Strong work. You built a calibrated LLM-as-a-judge workflow for ten saved outputs.

You've learned how to:

Produce a reusable batch evaluation with a score, written reason, plus cost for each saved output.
Set a human-calibrated threshold that matches your own labels on at least eight of ten outputs.
Create a judge comparison that shows the agreement level plus evaluation cost for each model.
Complete an optional Secret Mission that extends your calibrated evaluation workflow.

Ready to quiz yourself?