Build an LLM Eval Harness

Test LLM prompts with a golden dataset, deterministic checks, and LLM-as-judge scoring.

Introduction

30 Second Summary

Every time you tweak an AI prompt, there is a real chance you accidentally make it worse. Production AI teams call this "prompt rot." Without a systematic way to test prompt changes, quality degrades silently and nobody notices until users start complaining.

In this project, you will build an LLM evaluation harness that automatically tests summarization prompts against a golden dataset, scores outputs using both rule-based checks and AI-powered judging, and catches quality regressions before they reach users.

What You'll Build

A complete, portable eval harness that runs locally via Promptfoo's CLI and produces interactive HTML reports you can use to compare prompt variants and track scores over time.

By the end of this project, you'll have:

  • A golden dataset of 6 test cases with deterministic assertions that check for required keywords and enforce length limits.
  • Layered scoring that combines rule-based checks with LLM-as-judge rubric grading and factuality verification.
  • A regression tracking workflow that compares eval runs side-by-side and quantifies score changes after prompt modifications.
  • Secret Mission: Build a custom multi-dimension rubric that scores summaries across 5 weighted criteria for granular quality insights.

Are there any prerequisites?

You should be comfortable using the command line and editing YAML files. You will also need Node.js v20 or higher installed and an OpenAI API key ready to go.

Before We Start

Before We Start...

Set Up Your Eval Project

Before you can test and compare summarization prompts, you need a local project with the right tools. Promptfoo is an open-source evaluation framework that runs your test cases against LLM outputs, scores them, and generates comparison reports.

In this step, you'll confirm Node.js is ready and create your project folder. Then you'll initialize Promptfoo and run a smoke test to confirm it can reach the OpenAI API.

In this step, get ready to:
  • Confirm Node.js is installed on your machine.
  • Create the project directory with subdirectories for prompts and evaluation output.
  • Initialize Promptfoo and verify it connects to the OpenAI API.
Verify Node.js and create the project directory

Promptfoo runs on Node.js, so you need it installed before anything else. Let's confirm it's ready.

  • Check your Node.js version by running this command:
node --version

✔️ I see a version number

You're all set. Make sure the number starts with v20 or higher.

ⓧ Command not found

Node.js isn't installed yet. You'll need version v20 or higher.

  • Go to nodejs.org.
  • Download the LTS version for your operating system.
  • Run the installer and accept the default settings.
  • Close and reopen your terminal, then run node --version again to confirm it works.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and its subdirectories by running these commands:
mkdir -p summarization-eval/prompts summarization-eval/evaluations
cd summarization-eval

What do these commands do?

  • mkdir -p creates folders and any missing parent folders in one go.
  • prompts/ will hold your prompt template files.
  • evaluations/ will store your eval output files like JSON baselines.
  • cd summarization-eval moves you into the new project folder so all future commands run from there.
  • Confirm the folders exist by running:
ls

You should see evaluations and prompts in the output.

Initialize Promptfoo and configure your API key

Promptfoo uses a YAML configuration file to define your prompts, providers, and test cases. The init command scaffolds a starter config that you'll replace entirely in the next step.

  • Initialize Promptfoo by running this command:
  • If npx asks to install the promptfoo package, type y and press Enter to confirm.
npx promptfoo@latest init

What does this command do?

npx promptfoo@latest init creates a promptfooconfig.yaml file in your current folder. This is a starter template with placeholder content.

The @latest tag ensures you always get the most recent version of Promptfoo. No global install is needed.

  • Confirm the config file was created by running:
ls

You should see promptfooconfig.yaml listed alongside your prompts and evaluations folders.

Don't see the config file?

  • Make sure you ran the init command from inside the summarization-eval folder. Run pwd to check your current location.
  • If npx timed out or showed a network error, check your internet connection and try the command again.

Help me troubleshoot why promptfoo init didn't create a config file.

Now Promptfoo needs access to the OpenAI API so it can call models during evaluations.

  • Export your OpenAI API key for this terminal session by running this command:
  • Replace your-api-key-here with your actual OpenAI API key.
export OPENAI_API_KEY=your-api-key-here

This key is session-only

The export command sets the key only for your current terminal session. If you close and reopen your terminal, you'll need to run this command again.

Verify Promptfoo works

A quick smoke test confirms that Promptfoo is installed correctly and can reach the OpenAI API. You'll run the scaffolded eval and open the web viewer to check the results.

  • Run the scaffolded eval against the OpenAI API by running this command:
npx promptfoo@latest eval --no-cache

What does --no-cache do?

The --no-cache flag tells Promptfoo to skip its local cache and make fresh API calls. This ensures you see live results instead of cached data from a previous run.

You should see a table in your terminal with test results showing pass/fail indicators. The specific results come from the placeholder config and are not important yet.

  • Open the web viewer to inspect the results in your browser by running this command:
npx promptfoo@latest view

Your browser opens with an interactive results dashboard. Confirm you can see test results displayed in a table.

  • Press Ctrl+C in your terminal to stop the web viewer when you're done.

Not seeing results?

  • If you see an authentication error, your API key may not be set correctly. Run echo $OPENAI_API_KEY to verify it prints your key.
  • If you see a network error, check your internet connection and confirm your OpenAI API key has available credits.
  • If the browser did not open automatically, check your terminal output for a URL (usually http://localhost:15500) and open it manually.

Help me debug why Promptfoo eval is not working.

Your project is set up and Promptfoo is confirmed working. Next, you'll build a golden dataset with real test cases and deterministic assertions.

Build a Golden Dataset with Deterministic Checks

Your eval environment is set up and Promptfoo can reach the OpenAI API. Now it's time to create the actual tests your harness will run.

In this step, you'll build a golden dataset. This is a curated set of test cases with known-good inputs and specific assertions that check whether the LLM's output meets your requirements.

In this step, get ready to:
  • Create a summarization prompt template.
  • Build a golden dataset of 6 test cases with deterministic assertions.
  • Run your first eval and review pass/fail results.
Create the summarization prompt

Your eval harness needs a prompt template to test. Promptfoo uses Nunjucks templating, where {{variable_name}} placeholders get replaced with values from each test case at runtime.

  • Create a file called prompts/basic.txt and add this content:
Summarize the following article in 2-3 sentences:

{{article}}

What does this prompt do?

The {{article}} placeholder is a Nunjucks template variable. When Promptfoo runs each test case, it swaps this placeholder with the article value defined in that test's vars section.

This is a deliberately simple prompt. You'll compare it against a more detailed one later in the project.

  • Save prompts/basic.txt.
Write the golden dataset with deterministic assertions

A golden dataset is a fixed set of inputs paired with assertions that define "correct" output. For summarization, you'll check that the model's summary contains key terms from the source article.

Deterministic checks are cheap, fast, and perfectly reproducible. They're the foundation you build more sophisticated evaluations on top of.

  • Open promptfooconfig.yaml and replace its entire contents with this configuration:
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Text Summarization Eval Harness

# Point to the prompt template you just created
prompts:
  - file://prompts/basic.txt

# The LLM that generates summaries
providers:
  - openai:chat:gpt-4o-mini

# Golden dataset: 6 test cases with deterministic assertions
tests:
  - description: Tech product launch with figures
    vars:
      article: |
        Orion Technologies announced the release of its Nova X1 processor, which delivers 40% faster performance than its predecessor while consuming 25% less power. The chip is manufactured using a 2-nanometer process and is expected to ship in Q3 2026. CEO Maria Chen stated that the Nova X1 represents the company's largest R&D investment to date, with over $2 billion spent on development.
    assert:
      # Case-insensitive substring match
      - type: icontains
        value: Nova X1
      - type: icontains
        value: '40%'

  - description: Scientific climate discovery
    vars:
      article: |
        Researchers at the Global Climate Institute published findings showing that Arctic permafrost is thawing 70% faster than models predicted a decade ago. The study, based on satellite data collected over 15 years, found that methane emissions from thawing permafrost have doubled since 2015. Lead author Dr. James Park warned that this creates a feedback loop that could accelerate global warming beyond current projections.
    assert:
      - type: icontains
        value: permafrost

  - description: City transportation policy
    vars:
      article: |
        The city of Portland announced a $500 million investment in a new light rail extension connecting the downtown core to the eastern suburbs. Construction is set to begin in early 2027, with an expected completion date of 2030. The project will add 12 new stations and is projected to serve 45,000 daily riders. Mayor Lisa Huang called it the most significant infrastructure investment in the city's history.
    assert:
      - type: icontains
        value: Portland
      # Passes if either variant appears in the output
      - type: icontains-any
        value:
          - $500 million
          - 500 million

  - description: Sports milestone achievement
    vars:
      article: |
        The Denver Altitude became the first expansion team in Major League Soccer history to win the championship in their third season. Forward Diego Ramirez scored twice in the final, bringing his season total to 28 goals. The team's remarkable run included a 15-game unbeaten streak and a league-record 74 points in the regular season. Head coach Sarah Mitchell credited the team's success to a culture of relentless preparation.
    assert:
      - type: icontains
        value: Denver Altitude

  - description: Healthcare breakthrough with reference
    vars:
      article: |
        A clinical trial conducted across 30 hospitals demonstrated that a new mRNA-based treatment reduced symptoms of rheumatoid arthritis by 60% compared to the current standard of care. The treatment, developed by BioGenesis Labs, showed minimal side effects in the 2,400-patient trial. The FDA is expected to begin its review process by the end of 2026, with potential approval in early 2027.
    assert:
      - type: icontains
        value: rheumatoid arthritis

  - description: Short article edge case - should not hallucinate
    vars:
      article: |
        Global coffee prices rose 15% last month due to drought conditions in Brazil.
    assert:
      # Custom JavaScript expression to enforce word count
      - type: javascript
        value: output.split(' ').length <= 30
        metric: brevity

What does this configuration do?

The config has three sections that work together to define your eval harness.

  • prompts tells Promptfoo which prompt template to use. file://prompts/basic.txt points to the prompt you just created.
  • providers defines which LLM generates the summaries. openai:chat:gpt-4o-mini is fast and affordable for summarization.
  • tests is your golden dataset. Each test case has a vars.article input and an assert list of checks to run against the output.

The three assertion types each check something different. icontains checks that the output contains a substring (case-insensitive). icontains-any passes if the output contains any one of the listed substrings. javascript runs a custom expression, like the word count check on test 6.

  • Save promptfooconfig.yaml.

Seeing a YAML syntax error?

  • YAML is whitespace-sensitive. Make sure your indentation uses spaces, not tabs.
  • Check that the article text under vars is indented correctly under the | block scalar character.
  • The value '40%' must be wrapped in quotes because the % character can cause parsing issues in YAML.
  • Help me fix a YAML syntax error in my promptfooconfig.yaml

✔️ Awesome, I've got everything!

Make sure you've saved promptfooconfig.yaml before moving on.

ⓧ I'd like to double check the full code

Here is the full promptfooconfig.yaml for comparison.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Text Summarization Eval Harness

prompts:
  - file://prompts/basic.txt

providers:
  - openai:chat:gpt-4o-mini

tests:
  - description: Tech product launch with figures
    vars:
      article: |
        Orion Technologies announced the release of its Nova X1 processor, which delivers 40% faster performance than its predecessor while consuming 25% less power. The chip is manufactured using a 2-nanometer process and is expected to ship in Q3 2026. CEO Maria Chen stated that the Nova X1 represents the company's largest R&D investment to date, with over $2 billion spent on development.
    assert:
      - type: icontains
        value: Nova X1
      - type: icontains
        value: '40%'

  - description: Scientific climate discovery
    vars:
      article: |
        Researchers at the Global Climate Institute published findings showing that Arctic permafrost is thawing 70% faster than models predicted a decade ago. The study, based on satellite data collected over 15 years, found that methane emissions from thawing permafrost have doubled since 2015. Lead author Dr. James Park warned that this creates a feedback loop that could accelerate global warming beyond current projections.
    assert:
      - type: icontains
        value: permafrost

  - description: City transportation policy
    vars:
      article: |
        The city of Portland announced a $500 million investment in a new light rail extension connecting the downtown core to the eastern suburbs. Construction is set to begin in early 2027, with an expected completion date of 2030. The project will add 12 new stations and is projected to serve 45,000 daily riders. Mayor Lisa Huang called it the most significant infrastructure investment in the city's history.
    assert:
      - type: icontains
        value: Portland
      - type: icontains-any
        value:
          - $500 million
          - 500 million

  - description: Sports milestone achievement
    vars:
      article: |
        The Denver Altitude became the first expansion team in Major League Soccer history to win the championship in their third season. Forward Diego Ramirez scored twice in the final, bringing his season total to 28 goals. The team's remarkable run included a 15-game unbeaten streak and a league-record 74 points in the regular season. Head coach Sarah Mitchell credited the team's success to a culture of relentless preparation.
    assert:
      - type: icontains
        value: Denver Altitude

  - description: Healthcare breakthrough with reference
    vars:
      article: |
        A clinical trial conducted across 30 hospitals demonstrated that a new mRNA-based treatment reduced symptoms of rheumatoid arthritis by 60% compared to the current standard of care. The treatment, developed by BioGenesis Labs, showed minimal side effects in the 2,400-patient trial. The FDA is expected to begin its review process by the end of 2026, with potential approval in early 2027.
    assert:
      - type: icontains
        value: rheumatoid arthritis

  - description: Short article edge case - should not hallucinate
    vars:
      article: |
        Global coffee prices rose 15% last month due to drought conditions in Brazil.
    assert:
      - type: javascript
        value: output.split(' ').length <= 30
        metric: brevity

And here is prompts/basic.txt.

Summarize the following article in 2-3 sentences:

{{article}}
Run your first eval

Time to see your golden dataset in action. Promptfoo will send each test case's article through your prompt, call gpt-4o-mini, and check every assertion against the response.

  • Run the eval with caching disabled to force fresh API calls:
npx promptfoo@latest eval --no-cache

What does --no-cache do?

The --no-cache flag forces Promptfoo to make fresh API calls instead of reusing cached results from previous runs. This ensures you see real, up-to-date outputs every time.

  • Check the terminal output. You should see a table showing each test case with green (pass) or red (fail) indicators for each assertion.
  • All 6 test cases should pass their deterministic checks, since gpt-4o-mini reliably preserves key terms from source text.

Seeing failures or errors?

  • If you see an authentication error, make sure your OPENAI_API_KEY environment variable is still set. Environment variables reset when you close the terminal.
  • If a specific assertion fails, the model may have rephrased a key term. Check the full model response in the output to see if the keyword appears in a slightly different form.
  • If you see a YAML parsing error, double-check your indentation. Every level in YAML must use consistent spacing.
  • Help me debug my Promptfoo eval results

Your golden dataset is running and every test is passing. Next up, you'll add LLM-as-judge scoring to evaluate qualities that keyword checks can't measure, like factual accuracy and summary quality.

Add LLM-as-Judge Scoring

Your deterministic assertions confirm that key terms survive summarization. But they can't tell you whether a summary is actually good. A summary could include "Nova X1" and "40%" while still being rambling, inaccurate, or full of hallucinated details.

This is where LLM-as-judge scoring comes in. You will use a more capable model (gpt-4o) to grade the outputs of a cheaper model (gpt-4o-mini) against rubrics you write in plain English.

In this step, get ready to:
  • Add a global quality rubric that grades every test case automatically.
  • Enhance individual tests with targeted rubric and factuality assertions.
  • Run the eval and explore LLM-judge scores in the web viewer.
Add a default quality rubric

Promptfoo supports a defaultTest block that applies assertions to every test case automatically. Instead of duplicating the same quality checks six times, you define them once here.

  • In promptfooconfig.yaml, find the line that says providers: and its child item.
  • Add the following defaultTest block directly below the providers section and above the tests: line:
defaultTest:
  options:
    # Use gpt-4o as the judge - more capable than the target model
    provider: openai:chat:gpt-4o
  assert:
    # Global word count limit for all summaries
    - type: javascript
      value: output.split(' ').length <= 100
      metric: word-count
    # Global quality rubric scored by the LLM judge
    - type: llm-rubric
      value: |
        The summary is concise, accurate, and captures the main point of the article.
        It does not introduce information not present in the original text.
        It is written in clear, professional language.
      threshold: 0.7
      metric: overall-quality

What does this code do?

The defaultTest block has two parts.

  • options.provider: openai:chat:gpt-4o tells Promptfoo to use gpt-4o as the judge model. The judge should always be more capable than the model being tested.
  • The javascript assertion enforces a hard 100-word limit on every summary. The llm-rubric assertion gives gpt-4o a plain-English rubric and asks it to score the output from 0.0 to 1.0.
  • threshold: 0.7 means the assertion passes only when the judge's score meets or exceeds 0.7.
  • The metric field tags each assertion for filtering in the web viewer.

Why use a separate judge model?

Using the same model to both generate and judge outputs introduces self-enhancement bias. The model tends to rate its own style favorably. A more capable judge model (gpt-4o) evaluating a cheaper target model (gpt-4o-mini) produces more reliable scores.

You won't see a change yet. The next substep adds per-test assertions, then you will run the full eval.

Add per-test LLM-rubric and factuality assertions

The global rubric checks general quality. Now you will add targeted assertions to individual test cases that check for specific facts, figures, and behaviors.

You will use three assertion types: llm-rubric for criteria-based scoring, factuality for reference-based fact checking, and a strict hallucination-detection rubric for edge cases.

  • In promptfooconfig.yaml, find the first test case with description Tech product launch with figures.
  • Add the following three new assertions to the end of this test case's existing assert: section, below the two icontains assertions that are already there. Then do the same for the next two tests as shown in the code comments:
      # --- Add to "Tech product launch with figures" test, below existing assertions ---
      - type: llm-rubric
        value: Mentions the key performance claims (40% faster, 25% less power) and the company name (Orion Technologies)
        metric: fact-preservation

      # --- Add to "Scientific climate discovery" test, below existing assertions ---
      - type: factuality
        value: Arctic permafrost is thawing 70% faster than predicted. Methane emissions from thawing permafrost have doubled since 2015. This could accelerate global warming through a feedback loop.
      - type: llm-rubric
        value: Accurately conveys the 70% faster thawing rate and the methane feedback loop concern
        metric: fact-preservation

      # --- Add to "City transportation policy" test, below existing assertions ---
      - type: llm-rubric
        value: Mentions the light rail extension, the investment amount ($500 million), and the projected timeline
        metric: fact-preservation

What are these assertion types doing?

  • llm-rubric gives the judge a specific criterion to score against. For the tech test, the judge checks whether the summary mentions both the performance claims and the company name.
  • factuality is a reference-based assertion. You provide ground-truth statements, and the judge categorizes the summary's relationship to them (subset, superset, equivalent, or disagreeing). It fails only when the summary directly contradicts the reference.
  • The metric: fact-preservation tag lets you filter all these assertions together in the web viewer later.
  • Now find the remaining three test cases (Sports milestone achievement, Healthcare breakthrough with reference, and Short article edge case - should not hallucinate).
  • Add the following assertions to the end of each test's existing assert: section:
      # --- Add to "Sports milestone achievement" test, below existing assertions ---
      - type: llm-rubric
        value: Captures the historic achievement (first expansion team to win championship in third season) and mentions Diego Ramirez
        threshold: 0.8
        metric: fact-preservation

      # --- Add to "Healthcare breakthrough with reference" test, below existing assertions ---
      - type: factuality
        value: A new mRNA treatment by BioGenesis Labs reduced rheumatoid arthritis symptoms by 60% in a 2,400-patient trial across 30 hospitals with minimal side effects. FDA review expected by end of 2026.
      - type: llm-rubric
        value: Accurately reports the 60% symptom reduction and attributes the treatment to BioGenesis Labs
        threshold: 0.8
        metric: fact-preservation

      # --- Add to "Short article edge case" test, below existing assertions ---
      - type: llm-rubric
        value: |
          The summary does not add any information beyond what is stated in the original article.
          It should not speculate about causes, effects, or future implications not mentioned in the source.
          Fail if the summary introduces any new claims not present in the original.
        threshold: 0.9
        metric: no-hallucination

What do the thresholds and hallucination check do?

  • threshold: 0.8 on the sports and healthcare tests raises the bar. These tests involve specific names and figures that must appear accurately.
  • The short article test uses threshold: 0.9 with a strict anti-hallucination rubric. When the source is only one sentence, the model is most tempted to pad with fabricated context. This assertion catches that.
  • The metric: no-hallucination tag separates this assertion from the fact-preservation checks in the web viewer.
  • Save promptfooconfig.yaml.

YAML indentation errors?

YAML is sensitive to spacing. Each new assertion must be indented at the same level as the existing assertions in that test case (6 spaces from the left edge). If your assertions are mis-indented, Promptfoo will throw a parse error when you run the eval.

Use the double-check tab below to compare your full file against the expected code.

Help me fix a YAML indentation error in my promptfooconfig.yaml

✔️ Awesome, I've got everything!

Great. Double check you have saved promptfooconfig.yaml before moving on.

ⓧ I'd like to double check the full code

Here is the complete promptfooconfig.yaml as it should look now. Compare yours against this:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Text Summarization Eval Harness

prompts:
  - file://prompts/basic.txt

providers:
  - openai:chat:gpt-4o-mini

defaultTest:
  options:
    provider: openai:chat:gpt-4o
  assert:
    - type: javascript
      value: output.split(' ').length <= 100
      metric: word-count
    - type: llm-rubric
      value: |
        The summary is concise, accurate, and captures the main point of the article.
        It does not introduce information not present in the original text.
        It is written in clear, professional language.
      threshold: 0.7
      metric: overall-quality

tests:
  - description: Tech product launch with figures
    vars:
      article: |
        Orion Technologies announced the release of its Nova X1 processor, which delivers 40% faster performance than its predecessor while consuming 25% less power. The chip is manufactured using a 2-nanometer process and is expected to ship in Q3 2026. CEO Maria Chen stated that the Nova X1 represents the company's largest R&D investment to date, with over $2 billion spent on development.
    assert:
      - type: icontains
        value: Nova X1
      - type: icontains
        value: '40%'
      - type: llm-rubric
        value: Mentions the key performance claims (40% faster, 25% less power) and the company name (Orion Technologies)
        metric: fact-preservation

  - description: Scientific climate discovery
    vars:
      article: |
        Researchers at the Global Climate Institute published findings showing that Arctic permafrost is thawing 70% faster than models predicted a decade ago. The study, based on satellite data collected over 15 years, found that methane emissions from thawing permafrost have doubled since 2015. Lead author Dr. James Park warned that this creates a feedback loop that could accelerate global warming beyond current projections.
    assert:
      - type: icontains
        value: permafrost
      - type: factuality
        value: Arctic permafrost is thawing 70% faster than predicted. Methane emissions from thawing permafrost have doubled since 2015. This could accelerate global warming through a feedback loop.
      - type: llm-rubric
        value: Accurately conveys the 70% faster thawing rate and the methane feedback loop concern
        metric: fact-preservation

  - description: City transportation policy
    vars:
      article: |
        The city of Portland announced a $500 million investment in a new light rail extension connecting the downtown core to the eastern suburbs. Construction is set to begin in early 2027, with an expected completion date of 2030. The project will add 12 new stations and is projected to serve 45,000 daily riders. Mayor Lisa Huang called it the most significant infrastructure investment in the city's history.
    assert:
      - type: icontains
        value: Portland
      - type: icontains-any
        value:
          - $500 million
          - 500 million
      - type: llm-rubric
        value: Mentions the light rail extension, the investment amount ($500 million), and the projected timeline
        metric: fact-preservation

  - description: Sports milestone achievement
    vars:
      article: |
        The Denver Altitude became the first expansion team in Major League Soccer history to win the championship in their third season. Forward Diego Ramirez scored twice in the final, bringing his season total to 28 goals. The team's remarkable run included a 15-game unbeaten streak and a league-record 74 points in the regular season. Head coach Sarah Mitchell credited the team's success to a culture of relentless preparation.
    assert:
      - type: icontains
        value: Denver Altitude
      - type: llm-rubric
        value: Captures the historic achievement (first expansion team to win championship in third season) and mentions Diego Ramirez
        threshold: 0.8
        metric: fact-preservation

  - description: Healthcare breakthrough with reference
    vars:
      article: |
        A clinical trial conducted across 30 hospitals demonstrated that a new mRNA-based treatment reduced symptoms of rheumatoid arthritis by 60% compared to the current standard of care. The treatment, developed by BioGenesis Labs, showed minimal side effects in the 2,400-patient trial. The FDA is expected to begin its review process by the end of 2026, with potential approval in early 2027.
    assert:
      - type: icontains
        value: rheumatoid arthritis
      - type: factuality
        value: A new mRNA treatment by BioGenesis Labs reduced rheumatoid arthritis symptoms by 60% in a 2,400-patient trial across 30 hospitals with minimal side effects. FDA review expected by end of 2026.
      - type: llm-rubric
        value: Accurately reports the 60% symptom reduction and attributes the treatment to BioGenesis Labs
        threshold: 0.8
        metric: fact-preservation

  - description: Short article edge case - should not hallucinate
    vars:
      article: |
        Global coffee prices rose 15% last month due to drought conditions in Brazil.
    assert:
      - type: javascript
        value: output.split(' ').length <= 30
        metric: brevity
      - type: llm-rubric
        value: |
          The summary does not add any information beyond what is stated in the original article.
          It should not speculate about causes, effects, or future implications not mentioned in the source.
          Fail if the summary introduces any new claims not present in the original.
        threshold: 0.9
        metric: no-hallucination
Run the eval and explore LLM-judge scores in the web viewer

Your config now combines cheap deterministic checks with expensive LLM-judge scoring. Promptfoo runs the deterministic assertions first, then sends outputs to gpt-4o for rubric grading.

  • Run the eval by running this command:
npx promptfoo@latest eval --no-cache

You should see a table in your terminal with green (pass) and red (fail) indicators for each assertion. The eval takes longer this time because each test case now triggers LLM-judge API calls to gpt-4o.

  • Open the web viewer to inspect the detailed scores by running this command:
npx promptfoo@latest view

Your browser opens to an interactive results table.

  • Click any cell in the results table to see the full grading result. This includes the judge's reason (a natural language explanation), score (0.0 to 1.0), and pass status.
  • Use the Failures filter mode at the top of the table to quickly find any assertions that did not pass.
  • Check the Results Charts section for the Pass Rate and Score Distribution visualizations.
  • Type metric = fact-preservation in the filter bar to isolate only the rubric scores tagged with that metric.

What should I see in the web viewer?

Each test case row now shows multiple assertion results. Green cells passed. Red cells failed. Clicking a cell reveals the judge's reasoning.

Most assertions should pass since gpt-4o-mini is generally good at preserving facts from source text. If you see failures, read the judge's reason field to understand why it scored below the threshold.

  • Press Ctrl+C in your terminal to stop the web viewer when you are done exploring.

Seeing API errors or timeouts?

Make sure your OPENAI_API_KEY environment variable is still set in your current terminal session. If you opened a new terminal window, you need to export it again.

If you see rate limit errors, wait 30 seconds and run the eval again. The LLM-judge assertions make more API calls than the deterministic-only config.

Help me debug API errors when running my Promptfoo eval with LLM-as-judge assertions

Your eval harness now has two layers of quality checks. Deterministic assertions catch missing keywords instantly. LLM-judge assertions evaluate whether the summary is actually well-written and factually grounded. Next up, you will add a second prompt variant and compare them head-to-head.

Compare Prompt Variants

Your eval harness now scores summaries using both rule-based checks and LLM-as-judge rubrics. The next question is whether a different prompt could produce higher scores.

In this step, you'll write a second prompt that takes an expert-persona approach. Promptfoo tests every combination of prompt, provider, and test case automatically, so you can compare both prompts against the same golden dataset in a single eval run.

In this step, get ready to:
  • Create an expert-persona prompt variant.
  • Compare both prompts side-by-side in the web viewer.
  • Export a baseline report for regression tracking.
Create a second prompt variant

Your basic prompt gives the model minimal instructions. A more detailed prompt with a specific role and explicit constraints might produce more accurate, less hallucination-prone summaries.

Let's create an expert-persona prompt and let the eval harness measure whether the extra instructions actually help.

  • Create a new file called prompts/expert.txt and paste in the following content:
You are an expert editor at a major news publication. Summarize the following article in 2-3 concise sentences. Capture all key facts and figures accurately. Do not add opinions or information not present in the original article.

{{article}}

What makes this prompt different?

The expert prompt assigns the model a specific role ("expert editor at a major news publication"). This guides the model toward professional, publication-ready language.

It also explicitly forbids adding opinions or unsourced information. This directly targets the hallucination edge case in your golden dataset. The instruction to capture "all key facts and figures" maps to the fact-preservation assertions you wrote in the previous step.

  • Save prompts/expert.txt.
  • Confirm both prompt files exist by running:
ls prompts/

You should see both basic.txt and expert.txt listed.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

You are an expert editor at a major news publication. Summarize the following article in 2-3 concise sentences. Capture all key facts and figures accurately. Do not add opinions or information not present in the original article.

{{article}}

Now you need to tell Promptfoo about your new prompt. This means adding it to the prompts array in your config file.

  • In promptfooconfig.yaml, find the prompts section near the top of the file.
  • Find this block:
prompts:
  - file://prompts/basic.txt
  • Add a second line referencing your new expert prompt, so the section looks like this:
prompts:
  - file://prompts/basic.txt
  - file://prompts/expert.txt
  • Save promptfooconfig.yaml.

✔️ Awesome, I've got everything!

Great. Double check you've saved the file before running the eval.

ⓧ I'd like to double check the full code

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Text Summarization Eval Harness

prompts:
  - file://prompts/basic.txt
  - file://prompts/expert.txt

providers:
  - openai:chat:gpt-4o-mini

defaultTest:
  options:
    provider: openai:chat:gpt-4o
  assert:
    - type: javascript
      value: output.split(' ').length <= 100
      metric: word-count
    - type: llm-rubric
      value: |
        The summary is concise, accurate, and captures the main point of the article.
        It does not introduce information not present in the original text.
        It is written in clear, professional language.
      threshold: 0.7
      metric: overall-quality

tests:
  - description: Tech product launch with figures
    vars:
      article: |
        Orion Technologies announced the release of its Nova X1 processor, which delivers 40% faster performance than its predecessor while consuming 25% less power. The chip is manufactured using a 2-nanometer process and is expected to ship in Q3 2026. CEO Maria Chen stated that the Nova X1 represents the company's largest R&D investment to date, with over $2 billion spent on development.
    assert:
      - type: icontains
        value: Nova X1
      - type: icontains
        value: '40%'
      - type: llm-rubric
        value: Mentions the key performance claims (40% faster, 25% less power) and the company name (Orion Technologies)
        metric: fact-preservation

  - description: Scientific climate discovery
    vars:
      article: |
        Researchers at the Global Climate Institute published findings showing that Arctic permafrost is thawing 70% faster than models predicted a decade ago. The study, based on satellite data collected over 15 years, found that methane emissions from thawing permafrost have doubled since 2015. Lead author Dr. James Park warned that this creates a feedback loop that could accelerate global warming beyond current projections.
    assert:
      - type: icontains
        value: permafrost
      - type: factuality
        value: Arctic permafrost is thawing 70% faster than predicted. Methane emissions from thawing permafrost have doubled since 2015. This could accelerate global warming through a feedback loop.
      - type: llm-rubric
        value: Accurately conveys the 70% faster thawing rate and the methane feedback loop concern
        metric: fact-preservation

  - description: City transportation policy
    vars:
      article: |
        The city of Portland announced a $500 million investment in a new light rail extension connecting the downtown core to the eastern suburbs. Construction is set to begin in early 2027, with an expected completion date of 2030. The project will add 12 new stations and is projected to serve 45,000 daily riders. Mayor Lisa Huang called it the most significant infrastructure investment in the city's history.
    assert:
      - type: icontains
        value: Portland
      - type: icontains-any
        value:
          - $500 million
          - 500 million
      - type: llm-rubric
        value: Mentions the light rail extension, the investment amount ($500 million), and the projected timeline
        metric: fact-preservation

  - description: Sports milestone achievement
    vars:
      article: |
        The Denver Altitude became the first expansion team in Major League Soccer history to win the championship in their third season. Forward Diego Ramirez scored twice in the final, bringing his season total to 28 goals. The team's remarkable run included a 15-game unbeaten streak and a league-record 74 points in the regular season. Head coach Sarah Mitchell credited the team's success to a culture of relentless preparation.
    assert:
      - type: icontains
        value: Denver Altitude
      - type: llm-rubric
        value: Captures the historic achievement (first expansion team to win championship in third season) and mentions Diego Ramirez
        threshold: 0.8
        metric: fact-preservation

  - description: Healthcare breakthrough with reference
    vars:
      article: |
        A clinical trial conducted across 30 hospitals demonstrated that a new mRNA-based treatment reduced symptoms of rheumatoid arthritis by 60% compared to the current standard of care. The treatment, developed by BioGenesis Labs, showed minimal side effects in the 2,400-patient trial. The FDA is expected to begin its review process by the end of 2026, with potential approval in early 2027.
    assert:
      - type: icontains
        value: rheumatoid arthritis
      - type: factuality
        value: A new mRNA treatment by BioGenesis Labs reduced rheumatoid arthritis symptoms by 60% in a 2,400-patient trial across 30 hospitals with minimal side effects. FDA review expected by end of 2026.
      - type: llm-rubric
        value: Accurately reports the 60% symptom reduction and attributes the treatment to BioGenesis Labs
        threshold: 0.8
        metric: fact-preservation

  - description: Short article edge case - should not hallucinate
    vars:
      article: |
        Global coffee prices rose 15% last month due to drought conditions in Brazil.
    assert:
      - type: javascript
        value: output.split(' ').length <= 30
        metric: brevity
      - type: llm-rubric
        value: |
          The summary does not add any information beyond what is stated in the original article.
          It should not speculate about causes, effects, or future implications not mentioned in the source.
          Fail if the summary introduces any new claims not present in the original.
        threshold: 0.9
        metric: no-hallucination
Run the eval and compare prompts side-by-side

Promptfoo now has two prompts in its config. When you run the eval, it tests every test case against both prompts and produces a side-by-side comparison.

  • Run the eval with both prompts by running this command:
npx promptfoo@latest eval --no-cache

You should see a table in your terminal with two columns, one for each prompt.

  • Open the web viewer to explore the comparison by running:
npx promptfoo@latest view

The web viewer opens in your browser. The results table now shows two columns, one per prompt variant.

  • Click the Different filter mode to highlight test cases where the two prompts scored differently.
  • Open Results Charts and select the Scatter Plot to compare the two prompts head-to-head.

How do I read the scatter plot?

Each dot represents one test case scored by one assertion. Green dots mean the expert prompt scored higher than the basic prompt. Red dots mean the basic prompt scored higher.

A cluster of green dots tells you the expert prompt is consistently stronger. Gray dots mean both prompts scored the same.

  • Click any cell in the results table to see the full grading result, including the judge's reason, score (0.0-1.0), and pass status.
  • Press Ctrl+C in your terminal to stop the viewer when you're done exploring.

Only seeing one column?

Make sure you saved promptfooconfig.yaml after adding the second prompt to the prompts array.

Check that the file path is exactly file://prompts/expert.txt. A typo in the path means Promptfoo can't find the prompt and silently skips it.

Help me debug why Promptfoo is only showing one prompt column

Export your baseline results to JSON

Before modifying any prompts, you should save the current results as a baseline. This gives you a fixed reference point to compare against after future changes.

  • Export both a JSON file and an HTML report by running this command:
npx promptfoo@latest eval --no-cache -o evaluations/baseline.json -o evaluations/baseline.html

What do these files contain?

The JSON file contains machine-readable data: timestamps, pass/fail stats, and individual scores for every test case. You'll use this file to quantify regressions in the next step.

The HTML file is a standalone report you can open in any browser without running the Promptfoo viewer. The -o flag runs a fresh eval and saves the output to the specified file path.

  • Confirm the files were created by running:
ls evaluations/

You should see baseline.json and baseline.html listed.

Don't see your files?

Make sure the evaluations/ directory exists. You created it in Step 1. If it's missing, create it by running mkdir evaluations and try the export command again.

Check that you included both -o flags in the command. Each -o specifies one output file.

Help me debug why my Promptfoo baseline export didn't create the files

Your two prompts are now benchmarked with a saved baseline. Next, you'll intentionally degrade a prompt to see how the harness catches regressions.

Catch a Regression

You have a harness that scores summaries with deterministic checks and LLM-as-judge rubrics. The real test is whether it can catch quality drops when someone changes a prompt.

In this step, you will intentionally break the basic prompt, re-run the eval, and use Promptfoo's comparison tools to measure exactly how much damage the change caused.

In this step, get ready to:
  • Degrade the basic prompt to simulate a careless edit.
  • Compare the degraded eval run against your baseline in the web viewer.
  • Quantify the regression by comparing JSON export stats.
Intentionally degrade the basic prompt

A well-designed eval harness acts as a safety net for prompt quality. To prove yours works, you will introduce a known regression by replacing the basic prompt with a vague, under-specified version.

  • Open prompts/basic.txt in your text editor.
  • Find the current content of the file:
Summarize the following article in 2-3 sentences:

{{article}}
  • Replace the entire contents of the file with this degraded prompt:
Tell me about this:

{{article}}
  • Save prompts/basic.txt.

Why is this prompt so bad?

This degraded prompt does not ask for a summary. It gives no length guidance and lacks any instruction about accuracy.

This simulates what happens when someone makes a careless edit to a production prompt. Without an eval harness, this kind of regression could ship to users undetected.

Re-run the eval and compare in the web viewer

Now you will run the eval with the degraded prompt and use the web viewer's Compare feature to diff it against your baseline run.

  • Run the eval with the degraded prompt by running these commands:
npx promptfoo@latest eval --no-cache
npx promptfoo@latest view

What do these commands do?

  • --no-cache forces fresh API calls so the degraded prompt actually runs against the model instead of returning cached results.
  • promptfoo view opens the interactive web viewer where you can inspect scores and compare eval runs.

You should see the eval table in your terminal with noticeably more red (fail) indicators in the basic prompt column.

  • In the web viewer, click the eval actions menu in the top right corner of the results table.
  • Select Compare.
  • Choose the previous eval run (your baseline) from the list.

The diff view highlights score changes. Red cells indicate regressions and green cells indicate improvements.

  • Look at the basic prompt column. You should see significant drops in pass rate and rubric scores.
  • Check that the expert prompt column remains stable, since you did not change that prompt.

Why does only one column change?

Promptfoo tests every test case against every prompt in the config. Since you only degraded basic.txt, the expert prompt's scores stay the same. This is exactly what side-by-side comparison is designed to reveal.

Not seeing a difference between runs?

  • Make sure you saved prompts/basic.txt before running the eval. Unsaved changes won't be picked up by Promptfoo.
  • Confirm you included the --no-cache flag. Without it, Promptfoo may return cached results from your previous run.
  • Check that you selected the correct baseline run in the Compare menu. The earlier run should have a higher pass rate.

Help me figure out why my Promptfoo Compare view isn't showing a difference between eval runs.

Quantify the regression from JSON output

The visual diff is helpful, but numbers tell a clearer story. You will export the degraded run to JSON and compare its pass rate against your baseline file.

  • Press Ctrl+C in your terminal to stop the web viewer.
  • Export the degraded eval run to JSON by running this command:
npx promptfoo@latest eval --no-cache -o evaluations/degraded.json

What does this command do?

The -o flag tells Promptfoo to write results to a file. The .json extension produces a machine-readable JSON file containing timestamps, stats (successes, failures, errors), and per-test-case scores.

  • Confirm the file was created by running:
ls evaluations/

You should see baseline.json, baseline.html, and degraded.json in the output.

  • Compare the pass rates between baseline and degraded runs by running this script:
echo "=== Baseline ==="
cat evaluations/baseline.json | python3 -c "
import json, sys
data = json.load(sys.stdin)
stats = data['results']['stats']
total = stats['successes'] + stats['failures']
print(f\"Pass rate: {stats['successes']}/{total} ({100*stats['successes']/total:.0f}%)\")
"

echo "=== Degraded ==="
cat evaluations/degraded.json | python3 -c "
import json, sys
data = json.load(sys.stdin)
stats = data['results']['stats']
total = stats['successes'] + stats['failures']
print(f\"Pass rate: {stats['successes']}/{total} ({100*stats['successes']/total:.0f}%)\")
"

What does this script do?

  • It reads the JSON export for each eval run and pulls out the results.stats object, which contains successes and failures counts.
  • It then calculates the pass rate as a percentage so you can compare the two runs at a glance.

You should see a clear drop in pass rate for the degraded run compared to the baseline. The baseline should have a higher pass rate because the expert prompt still performs well, while the degraded basic prompt drags the numbers down.

Seeing a Python error?

  • Make sure python3 is installed on your machine. Run python3 --version to check.
  • If you see a KeyError, your JSON file may be empty or malformed. Re-run the export command and check that evaluations/degraded.json is not empty.

Help me debug the Python comparison script for my Promptfoo JSON exports.

Now that you have proven your harness catches regressions, restore the original prompt so your eval harness is back to its baseline state.

  • Open prompts/basic.txt in your text editor.
  • Find the degraded content:
Tell me about this:

{{article}}
  • Replace the entire contents with the original prompt:
Summarize the following article in 2-3 sentences:

{{article}}
  • Save prompts/basic.txt.

✔️ Awesome, I've got everything!

Double check you've saved prompts/basic.txt with the original prompt restored.

ⓧ I'd like to double check the full code

Here is what each file should look like at this point.

Summarize the following article in 2-3 sentences:

{{article}}
You are an expert editor at a major news publication. Summarize the following article in 2-3 concise sentences. Capture all key facts and figures accurately. Do not add opinions or information not present in the original article.

{{article}}
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Text Summarization Eval Harness

prompts:
  - file://prompts/basic.txt
  - file://prompts/expert.txt

providers:
  - openai:chat:gpt-4o-mini

defaultTest:
  options:
    provider: openai:chat:gpt-4o
  assert:
    - type: javascript
      value: output.split(' ').length <= 100
      metric: word-count
    - type: llm-rubric
      value: |
        The summary is concise, accurate, and captures the main point of the article.
        It does not introduce information not present in the original text.
        It is written in clear, professional language.
      threshold: 0.7
      metric: overall-quality

tests:
  - description: Tech product launch with figures
    vars:
      article: |
        Orion Technologies announced the release of its Nova X1 processor, which delivers 40% faster performance than its predecessor while consuming 25% less power. The chip is manufactured using a 2-nanometer process and is expected to ship in Q3 2026. CEO Maria Chen stated that the Nova X1 represents the company's largest R&D investment to date, with over $2 billion spent on development.
    assert:
      - type: icontains
        value: Nova X1
      - type: icontains
        value: '40%'
      - type: llm-rubric
        value: Mentions the key performance claims (40% faster, 25% less power) and the company name (Orion Technologies)
        metric: fact-preservation

  - description: Scientific climate discovery
    vars:
      article: |
        Researchers at the Global Climate Institute published findings showing that Arctic permafrost is thawing 70% faster than models predicted a decade ago. The study, based on satellite data collected over 15 years, found that methane emissions from thawing permafrost have doubled since 2015. Lead author Dr. James Park warned that this creates a feedback loop that could accelerate global warming beyond current projections.
    assert:
      - type: icontains
        value: permafrost
      - type: factuality
        value: Arctic permafrost is thawing 70% faster than predicted. Methane emissions from thawing permafrost have doubled since 2015. This could accelerate global warming through a feedback loop.
      - type: llm-rubric
        value: Accurately conveys the 70% faster thawing rate and the methane feedback loop concern
        metric: fact-preservation

  - description: City transportation policy
    vars:
      article: |
        The city of Portland announced a $500 million investment in a new light rail extension connecting the downtown core to the eastern suburbs. Construction is set to begin in early 2027, with an expected completion date of 2030. The project will add 12 new stations and is projected to serve 45,000 daily riders. Mayor Lisa Huang called it the most significant infrastructure investment in the city's history.
    assert:
      - type: icontains
        value: Portland
      - type: icontains-any
        value:
          - $500 million
          - 500 million
      - type: llm-rubric
        value: Mentions the light rail extension, the investment amount ($500 million), and the projected timeline
        metric: fact-preservation

  - description: Sports milestone achievement
    vars:
      article: |
        The Denver Altitude became the first expansion team in Major League Soccer history to win the championship in their third season. Forward Diego Ramirez scored twice in the final, bringing his season total to 28 goals. The team's remarkable run included a 15-game unbeaten streak and a league-record 74 points in the regular season. Head coach Sarah Mitchell credited the team's success to a culture of relentless preparation.
    assert:
      - type: icontains
        value: Denver Altitude
      - type: llm-rubric
        value: Captures the historic achievement (first expansion team to win championship in third season) and mentions Diego Ramirez
        threshold: 0.8
        metric: fact-preservation

  - description: Healthcare breakthrough with reference
    vars:
      article: |
        A clinical trial conducted across 30 hospitals demonstrated that a new mRNA-based treatment reduced symptoms of rheumatoid arthritis by 60% compared to the current standard of care. The treatment, developed by BioGenesis Labs, showed minimal side effects in the 2,400-patient trial. The FDA is expected to begin its review process by the end of 2026, with potential approval in early 2027.
    assert:
      - type: icontains
        value: rheumatoid arthritis
      - type: factuality
        value: A new mRNA treatment by BioGenesis Labs reduced rheumatoid arthritis symptoms by 60% in a 2,400-patient trial across 30 hospitals with minimal side effects. FDA review expected by end of 2026.
      - type: llm-rubric
        value: Accurately reports the 60% symptom reduction and attributes the treatment to BioGenesis Labs
        threshold: 0.8
        metric: fact-preservation

  - description: Short article edge case - should not hallucinate
    vars:
      article: |
        Global coffee prices rose 15% last month due to drought conditions in Brazil.
    assert:
      - type: javascript
        value: output.split(' ').length <= 30
        metric: brevity
      - type: llm-rubric
        value: |
          The summary does not add any information beyond what is stated in the original article.
          It should not speculate about causes, effects, or future implications not mentioned in the source.
          Fail if the summary introduces any new claims not present in the original.
        threshold: 0.9
        metric: no-hallucination

Your eval harness just proved its value. A single careless prompt edit caused a measurable quality drop, and your harness caught it with hard numbers. Next up, you will take on the Secret Mission and build a custom multi-dimension rubric for even more granular scoring.

Secret mission

Build a Custom Multi-Dimension Rubric

Your current eval harness gives each summary a single quality grade. What if you could score across 5 separate dimensions instead? Build a custom rubricPrompt that evaluates Accuracy, Key-Fact Coverage, Completeness, Readability, and Brevity with weighted scoring.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs. The only expense was the OpenAI API usage during your eval runs (~$0.10-0.50 total).

Resources you used:

  • summarization-eval/ project directory (local files including prompts, config, and evaluation outputs).
  • Promptfoo local eval history stored in ~/.promptfoo/.
  • OPENAI_API_KEY environment variable (set for the current shell session only).
  • OpenAI API credits consumed (~$0.10-0.50 total, already spent).

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment with more prompt variants and assertion types.

  • Your summarization-eval/ directory stays on your Desktop with all prompts, config, and evaluation reports intact.
  • Your Promptfoo eval history remains in ~/.promptfoo/ so you can compare future runs against your baseline.
  • If you open a new terminal session, re-export your API key by running export OPENAI_API_KEY=your-api-key-here before running evals.

Pause - I'll come back to this later

Keep your project files but clear the API key from your current session. This frees up nothing significant since the project is local, but it is good security hygiene.

  • Clear the OpenAI API key from your current terminal session by running:
unset OPENAI_API_KEY
  • Your summarization-eval/ directory and all evaluation reports remain on your Desktop.
  • When you come back, re-export your API key and pick up where you left off.

Delete - I don't want to use this again

Remove all project files and Promptfoo's local cache to start completely fresh.

  • Delete the project directory by running:
cd ~/Desktop
rm -rf summarization-eval
  • Clear your Promptfoo eval history by running:
rm -rf ~/.promptfoo
  • Clear the API key from your terminal session by running:
unset OPENAI_API_KEY
  • No cloud resources need to be deleted. The OpenAI API credits you used are already consumed and will not incur further charges.

Mission accomplished

That's a Wrap!

Nice work! You just built a production-grade LLM evaluation harness from scratch using Promptfoo. Your harness can test any summarization prompt against a golden dataset, score outputs with layered assertions, and catch regressions before they reach production.

You've learned how to:

  • Build a golden dataset of 6 test cases and score them with layered assertions combining deterministic checks (keyword matching, word count limits) and LLM-as-judge rubric grading with factuality verification.
  • Compare two prompt variants side-by-side using Promptfoo's web viewer and scatter plot to identify which prompt produces better summaries.
  • Establish a regression tracking workflow by exporting baselines to JSON, intentionally degrading a prompt, and quantifying the score drop.
  • Secret Mission: Built a custom multi-dimension rubric that scores summaries across 5 weighted criteria for granular quality insights.

Ready to quiz yourself?