Model Selection with Architecture Records

Compare three free LLMs with promptfoo and write a data-backed ADR.

Introduction

30 Second Summary

Every team building with AI hits the same question: which model should we actually use? Most of the time, someone tries a few prompts, says "this one feels better," and that gut feeling becomes the team's decision.

In this project, you will build a structured evaluation that compares three free-tier LLMs side by side. Then you will write an architecture decision record that captures the winning model with your own numbers, not vendor marketing.

What You'll Build

Picture running one command that opens a color-coded grid of three models scored across 15 questions, alongside a Markdown decision record naming the winner with numbers anyone on your team can reproduce.

By the end of this project, you'll have:

  • A promptfoo evaluation you can rerun with one command, comparing Gemini 2.5 Flash, Gemini 2.5 Flash-Lite, and a free OpenRouter model across 15 test cases drawn from your Project 1 workload.
  • A browser-based comparison grid showing pass/fail results, latency numbers, and quality scores for every model on every question.
  • An architecture decision record in MADR 4.0.0 format where the "Decision Outcome" section cites your own evaluation data.
  • Secret Mission: Test whether the LLM judge is biased by swapping the grading model and comparing rankings.

Are there any prerequisites?

This project builds on Project 1 (C4 AI Architecture to Working API). You will need a code editor, Git, and a Gemini API key. If you are starting fresh, the first step includes setup paths for everything you need.

Before We Start

Before running any evaluations or writing any decisions, let's lock in what you are about to build and why it matters.

Set Up Your Evaluation Environment

Your Project 1 service works. But right now, you have no data proving that Gemini 2.5 Flash is the right model for the job.

Before you can generate that data, you need your evaluation tools and API keys ready. In this step, you will verify Node.js, confirm promptfoo works, set your API keys, and create the project folder.

In this step, get ready to:
  • Verify Node.js and promptfoo are ready to run evaluations.
  • Configure API keys for Google and OpenRouter.
  • Create the project folder and prompt file.
Verify Node.js and promptfoo

promptfoo is the evaluation framework you will use to compare multiple LLMs side by side. It requires Node.js >=22.22.0 and runs on demand through npx, so you do not need to install it globally.

  • Check your Node.js version by running this command:
node --version

✔️ I see version 22.22.0 or higher

Your Node.js version meets the minimum requirement. Move on to confirming promptfoo below.

ⓧ I see an older version

promptfoo requires Node.js >=22.22.0. Your current version is too old.

  • Download and install Node.js 24 LTS from the official Node.js download page.
  • The installer will replace your current version.
  • After the install completes, close and reopen your terminal.
  • Re-run the version check to confirm:
node --version

You should now see v24 or higher.

ⓧ Command not found

Node.js is not installed on your machine yet.

  • Download and install Node.js 24 LTS from the official Node.js download page.
  • Run through the installer and accept the defaults.
  • After the install completes, close and reopen your terminal.
  • Verify the installation by running:
node --version

You should see v24 or higher.

Before you run the next command, what do you expect to see when you ask promptfoo for its version?

  • Confirm promptfoo works by running this command:
npx promptfoo@latest --version

You should see a version number like 0.122.0 or later. promptfoo downloads on demand through npx, so nothing is installed permanently on your machine.

What is promptfoo?

promptfoo is an open-source LLM evaluation framework. You give it a YAML config listing models, prompts, and test cases. It calls every model on every test case and presents the results in a side-by-side grid.

Because it runs via npx, you always get the latest version without managing a global install.

🙋‍♀️ Seeing an error when running npx?

Make sure your Node.js version is 22.22.0 or higher. Run node --version to double check.

If you see a network error, check your internet connection. npx needs to download promptfoo the first time you run it.

help me debug a promptfoo npx error

Set up your API keys

promptfoo needs two API keys to call the models you will compare. The Google provider looks for an environment variable called GOOGLE_API_KEY. This is different from the GEMINI_API_KEY you used in Project 1.

Windows PowerShell users

The commands below use export, which works on macOS and Linux. If you are using Windows PowerShell, replace export with $env: syntax instead. For example: $env:GOOGLE_API_KEY = $env:GEMINI_API_KEY.

First, set your GOOGLE_API_KEY.

✔️ I have a Gemini API key from Project 1

Your GEMINI_API_KEY from Project 1 needs to be active in this terminal session before you can map it.

  • Check if GEMINI_API_KEY is set in this session by running:
echo $GEMINI_API_KEY

If you see your key printed, skip to the export command below. If this prints a blank line, navigate to your Project 1 folder and load your .env file by running:

source .env
  • Set GOOGLE_API_KEY by running this command:
export GOOGLE_API_KEY=$GEMINI_API_KEY
  • Verify it was set by running:
echo $GOOGLE_API_KEY

You should see your API key value printed in the terminal.

ⓧ I need a new Gemini API key

You can get a free Gemini API key from Google AI Studio.

  • Open Google AI Studio in your browser.
  • Sign in with your Google account.
  • Click Get API key.
  • Copy the key. Do not share it or commit it to version control.
  • Set it as GOOGLE_API_KEY in your terminal by running this command (replace the placeholder with your actual key):
export GOOGLE_API_KEY=your-api-key-here
  • Verify it was set by running:
echo $GOOGLE_API_KEY

You should see your API key value printed in the terminal.

Next, set your OPENROUTER_API_KEY. OpenRouter gives you access to free-tier models from other providers, adding a third model to your comparison.

✔️ I have an OpenRouter API key

  • Set your OPENROUTER_API_KEY by running this command (replace the placeholder with your actual key):
export OPENROUTER_API_KEY=your-api-key-here
  • Verify it was set by running:
echo $OPENROUTER_API_KEY

You should see your API key value printed in the terminal.

ⓧ I need to create an OpenRouter account

OpenRouter's free tier gives you 50 requests per day with no credit card required.

  • Open openrouter.ai in your browser and click Sign Up.
  • Create your account using your email or a social login.
  • After signing in, go to openrouter.ai/settings/keys.
  • Click Create Key, give it a name, and copy the key.
  • Set it in your terminal by running this command (replace the placeholder with your actual key):
export OPENROUTER_API_KEY=your-api-key-here
  • Verify it was set by running:
echo $OPENROUTER_API_KEY

You should see your API key value printed in the terminal.

Why two different API keys?

Each model provider authenticates requests with its own key. GOOGLE_API_KEY lets promptfoo call Gemini models through Google AI Studio. OPENROUTER_API_KEY lets promptfoo call open-weight models through OpenRouter's free tier.

Having two providers in your comparison adds real diversity to the evaluation. It prevents you from only seeing what one vendor offers.

🙋‍♀️ Echo prints a blank line?

If echo $GOOGLE_API_KEY prints nothing, the variable was not set. Make sure you ran the export command in the same terminal session. Opening a new terminal tab resets environment variables.

If you used the export GOOGLE_API_KEY=$GEMINI_API_KEY shortcut, confirm that GEMINI_API_KEY is set too by running echo $GEMINI_API_KEY.

help me fix my API key environment variables

Create the project folder and prompt file

You need a dedicated folder for your evaluation config, prompt template, and decision record. The prompt file defines the exact workload you are evaluating models for.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir model-evaluation
cd model-evaluation
  • Confirm you are in the right place by running:
pwd

You should see a path ending in model-evaluation.

Now create the prompt template that promptfoo will send to each model.

  • Open the model-evaluation folder in your code editor.
  • Create a new file called prompt.txt and paste the following content:
You are a helpful assistant. Respond concisely.

User question: {{question}}

What does this prompt template do?

This file is the prompt that every model will receive during evaluation. The {{question}} placeholder is a promptfoo template variable. When promptfoo runs, it replaces {{question}} with each test case's question.

By using the same prompt for every model, you ensure a fair comparison. The only variable is the model itself.

  • Save prompt.txt.
  • Verify the file exists by running:
ls

You should see prompt.txt listed in the output.

Don't see prompt.txt in the output?

Make sure you saved the file inside the model-evaluation folder, not in a parent directory. Run pwd to confirm your terminal is inside model-evaluation.

If the file is in the wrong place, move it by running mv or delete it and recreate it in the correct folder.

help me find my prompt.txt file

✔️ Awesome, I've got everything!

Great. Double check you have saved prompt.txt and move on.

ⓧ I'd like to double check the full code

Here is the complete content of prompt.txt. Make sure your file matches exactly:

You are a helpful assistant. Respond concisely.

User question: {{question}}

Your evaluation environment is ready. You have Node.js, promptfoo, two API keys, and a prompt template. Next up, you will run your first model comparison and see three LLMs go head to head.

Run Your First Model Comparison

Your evaluation environment is set up. You have promptfoo ready via npx, your API keys loaded, and a prompt file that mirrors the workload from Project 1.

Before you can design meaningful evaluation criteria, you need to see what a model comparison actually looks like. In this step, you'll pit three LLMs against each other on real questions and pick a winner based purely on what you read.

In this step, get ready to:
  • Create a promptfoo evaluation config with three models and five test cases.
  • Run the evaluation and open a side-by-side comparison grid in your browser.
  • Pick a winner based on your first impression of the outputs.
Create the evaluation config with three models and five test cases

promptfoo uses a single YAML config file to define everything about an evaluation: which models to compare, which prompt to send, and what questions to test.

  • In your code editor, create a new file called promptfooconfig.yaml inside your model-evaluation folder.
  • Paste the following configuration into the file:
description: Model selection evaluation for AI Question Answering Service

# Reference the prompt template
prompts:
  - file://prompt.txt

# Three models to compare across two providers
providers:
  - id: google:gemini-2.5-flash
  - id: google:gemini-2.5-flash-lite
  - id: openrouter:meta-llama/llama-3.2-3b-instruct:free

# Configures the LLM judge for rubric assertions (added later)
defaultTest:
  options:
    provider: google:gemini-2.5-flash

# Five software architecture questions from your Project 1 workload
tests:
  - vars:
      question: What is the C4 model in software architecture?
  - vars:
      question: Explain the difference between containers and components in C4.
  - vars:
      question: What is architecture drift and why does it matter?
  - vars:
      question: How does FastAPI handle request validation?
  - vars:
      question: What is a virtual environment in Python and why use one?

What does this config do?

  • prompts points to your prompt.txt file. promptfoo reads it and injects each test case's question variable into the {{question}} placeholder.
  • providers lists three models: Gemini 2.5 Flash and Flash-Lite from Google, plus Meta Llama 3.2 3B Instruct via OpenRouter. Each model gets every test question.
  • defaultTest.options.provider sets the LLM judge for rubric-based grading. You'll add assertions in the next step. For now, this line does nothing because there are no assertions to grade.
  • tests defines five software architecture questions. These are the exact kinds of questions your Project 1 /ask endpoint handles.
  • Save promptfooconfig.yaml.
  • Confirm the file exists by running this in your terminal from inside the model-evaluation folder:
ls

You should see both prompt.txt and promptfooconfig.yaml listed.

Only seeing one file?

  • Make sure you saved promptfooconfig.yaml inside the model-evaluation folder, not one level above it.
  • Check that your terminal is inside the model-evaluation folder. Run pwd to verify.

Still stuck? Ask for help.

✔️ Awesome, I've got everything!

Great. Double check you've saved promptfooconfig.yaml and you're ready to run your first evaluation.

ⓧ I'd like to double check the full code

Here is the complete promptfooconfig.yaml file. Make sure yours matches exactly:

description: Model selection evaluation for AI Question Answering Service

prompts:
  - file://prompt.txt

providers:
  - id: google:gemini-2.5-flash
  - id: google:gemini-2.5-flash-lite
  - id: openrouter:meta-llama/llama-3.2-3b-instruct:free

defaultTest:
  options:
    provider: google:gemini-2.5-flash

tests:
  - vars:
      question: What is the C4 model in software architecture?
  - vars:
      question: Explain the difference between containers and components in C4.
  - vars:
      question: What is architecture drift and why does it matter?
  - vars:
      question: How does FastAPI handle request validation?
  - vars:
      question: What is a virtual environment in Python and why use one?
Run the evaluation and read the grid

Your config tells promptfoo what to test. Now let it loose on all three models.

  • Run the evaluation by entering this command:
npx promptfoo@latest eval

What's happening?

promptfoo sends each of the five questions to all three models in parallel. You'll see a progress bar as it works through 15 total API calls (5 questions x 3 models). This may take a minute or two depending on API response times.

Once the eval finishes, you'll see a summary table in the terminal. The real payoff is the visual comparison grid.

Before you open it, take a guess: will all three models produce similar answers, or will you spot obvious quality differences?

  • Open the results in your browser by running:
npx promptfoo@latest view

A browser tab opens with a color-coded results table. Each column is one of your three models. Each row is one of your five questions. You can read every model's answer side by side.

What should I see?

The grid shows three columns (Gemini 2.5 Flash, Gemini 2.5 Flash-Lite, and Llama 3.2 3B Instruct) and five rows of outputs. Since you have no assertions yet, there are no pass/fail indicators. You are just reading raw answers.

Scroll through and compare. Some models will be more verbose. Some may hedge their answers. Some may miss parts of the question entirely.

🙋‍♀️ Browser tab didn't open?

  • The terminal should show a local URL (like http://localhost:15500). Copy and paste it into your browser manually.
  • If you see an error about no results, make sure your npx promptfoo@latest eval command finished successfully before running npx promptfoo@latest view.
  • If the eval failed with an authentication error, double check your GOOGLE_API_KEY and OPENROUTER_API_KEY environment variables are set in the same terminal session.

Need help troubleshooting?

Commit your initial impression

Look at the grid. Read through the five answers from each model. One model probably stands out to you as the "best" option.

  • Pick the model whose answers you like the most.
  • Note the name of the model and why you chose it.

What just happened?

You just made a model selection decision based on reading five answers. No latency data. No consistency check across a larger question set. No scoring criteria. Just your gut feeling after scanning outputs.

If a teammate asked "why that model?" right now, your only answer would be "the answers looked good to me." Hold onto your pick. You will revisit it in the next step.

You have a comparison grid and a winner in mind. Next, you'll add structured assertions that measure quality, latency, and conciseness. Your vibes-based pick might not survive the numbers.

Add Criteria That Actually Measure Something

You have a working comparison grid with three models answering five questions. You even picked a winner. But if a teammate asks "why that model?", your only answer is "the answers looked good."

Structured assertions change that. They measure specific properties like accuracy, response speed, and answer length across every test case. In this step, you will expand your test suite to 15 cases and add assertions that turn your evaluation from opinion into measurement.

In this step, get ready to:
  • Add 10 more test cases covering a range of question types.
  • Add three structured assertions for quality, latency, and conciseness.
  • Re-run the evaluation and compare structured results to your vibes-based pick.
Add 10 more test cases

Five test cases is too small a sample to reveal patterns. Fifteen cases covering factual, conceptual, comparison, and practical questions give the assertions a meaningful dataset to score against.

  • In promptfooconfig.yaml, scroll to the bottom of the tests section and find the last entry:
  - vars:
      question: What is a virtual environment in Python and why use one?
  • Add these 10 new test cases directly below that last entry:
  - vars:
      question: Explain the purpose of a .gitignore file.
  - vars:
      question: What is the difference between a monolith and a modular architecture?
  - vars:
      question: How do API keys work for authentication?
  - vars:
      question: What is the role of an orchestrator in a service architecture?
  - vars:
      question: Explain what Mermaid diagrams are and why developers use them.
  - vars:
      question: What is the difference between GET and POST HTTP methods?
  - vars:
      question: How does dependency injection improve testability?
  - vars:
      question: What is the purpose of a health check endpoint?
  - vars:
      question: Explain the concept of separation of concerns in software design.
  - vars:
      question: What are the tradeoffs between using a single file versus multiple modules?

Why these questions?

These 15 questions are drawn from the software architecture domain that your Project 1 /ask endpoint serves. They cover different question types so the evaluation tests each model across a realistic workload, not just a narrow slice.

Some are straightforward definitions ("What is a health check endpoint?"). Others require nuance ("What are the tradeoffs between a single file versus multiple modules?"). A model that shines on simple questions may stumble on the complex ones.

  • Save promptfooconfig.yaml.
  • Scroll through the tests section and count the entries. You should have 15 test cases total.

Count doesn't add up to 15?

  • Make sure the new entries start right after the last original test case. No blank lines between entries.
  • Check that each entry starts with - vars: at the same indent level as the existing entries. YAML is sensitive to indentation.
  • Confirm you did not accidentally delete any of the original five test cases.

Still stuck? Help me fix my promptfooconfig.yaml test cases. I should have 15 entries but my count is off.

Add structured assertions

Right now, promptfoo just shows raw model outputs with no judgment. Assertions tell promptfoo what "good" looks like so it can score each answer as pass or fail.

  • In promptfooconfig.yaml, find the defaultTest section near the top of the file. It currently looks like this:
defaultTest:
  options:
    provider: google:gemini-2.5-flash
  • Add this assert block directly below the provider line, at the same indent level as options:
  assert:
    - type: llm-rubric
      value: The answer is factually accurate and directly addresses the question without hedging or unnecessary caveats
    - type: latency
      threshold: 10000
    - type: word-count
      value:
        max: 150

What do these assertions measure?

  • llm-rubric sends each model's answer to a judge LLM (Gemini 2.5 Flash, configured in defaultTest.options.provider) and asks it to grade the answer against your rubric text. This is your quality check.
  • latency checks whether the model responded in under 10,000 milliseconds (10 seconds). This is your speed check.
  • word-count fails any answer longer than 150 words. This is your conciseness check.

💡 Why is Gemini judging itself?

The defaultTest.options.provider setting means Gemini 2.5 Flash grades all three models, including its own answers. That is a conflict of interest. You will investigate whether this bias matters in the Secret Mission.

  • Save promptfooconfig.yaml.
  • Confirm the assert block is indented under defaultTest at the same level as options. Incorrect indentation will cause a YAML parsing error when you run the evaluation.

Not sure about the indentation?

The word assert should start at the same column as the word options (two spaces in from defaultTest). If you mix tabs and spaces, YAML will reject the file. Use spaces only.

Need a hand? Help me check my promptfooconfig.yaml indentation for the assert block under defaultTest.

Run the structured evaluation and compare to your vibes pick

Think back to the model you picked as the winner in the last step. Do you think it will still come out on top when measured against accuracy, latency, and word count criteria?

  • Run the evaluation with the --no-cache flag by running this command:
npx promptfoo@latest eval --no-cache

Why --no-cache?

The --no-cache flag forces promptfoo to call each model fresh instead of reusing saved results. Without it, the latency assertion would measure near-zero because it is reading from cache, not from a real API call.

This run takes longer than the first one. promptfoo is calling all three models on 15 questions (45 API calls) plus running the llm-rubric judge on each response (another 45 calls to Gemini 2.5 Flash).

  • Once the evaluation finishes, open the results in your browser by running:
npx promptfoo@latest view

The grid now looks different from the last step. Each cell has green or red pass/fail indicators for the three assertions. A model that looked great by vibes may now show red cells for busting the word count or missing the latency threshold.

What should I look for?

  • Which model has the most green (passing) cells across all 15 test cases?
  • Which model fails the word-count check most often? Verbose answers are a common trait of larger models.
  • Which model is fastest? Check the latency pass/fail column.
  • Does the llm-rubric quality score match what you thought the "best" answers were when you read them in the last step?

🙋‍♀️ Seeing YAML errors or no pass/fail cells?

  • If the eval command fails with a parsing error, check the indentation of your assert block. It must be at the same level as options under defaultTest.
  • If you see outputs but no pass/fail colors, confirm you added the assert block inside defaultTest, not at the root level of the YAML file.
  • If latency shows all green even on slow models, make sure you used the --no-cache flag. Without it, cached results report near-zero latency.

Need help? My promptfoo eval is not showing pass/fail results for my assertions. Help me debug my promptfooconfig.yaml.

✔️ Awesome, I've got everything!

Great. Double check that you have saved promptfooconfig.yaml with all 15 test cases and the three assertions.

ⓧ I'd like to double check the full code

Your promptfooconfig.yaml is now too long to show in one block. Scroll through the file and confirm these sections are present, in order:

  • description line at the top.
  • prompts section referencing file://prompt.txt.
  • providers section with three provider IDs: google:gemini-2.5-flash, google:gemini-2.5-flash-lite, and openrouter:meta-llama/llama-3.2-3b-instruct:free.
  • defaultTest section with options.provider set to google:gemini-2.5-flash, followed by an assert block containing llm-rubric, latency, and word-count assertions.
  • tests section with 15 test case entries, starting with "What is the C4 model in software architecture?" and ending with "What are the tradeoffs between using a single file versus multiple modules?".

The prompt.txt file is unchanged from the previous step.

Your evaluation now has real criteria behind it. You can point to pass rates, latency numbers, and word counts instead of "it seemed good." Next up, you will capture this decision in a formal architecture decision record that anyone on your team can read and reproduce.

Write the Architecture Decision Record

Your evaluation grid now shows hard numbers for three models across 15 test cases. But that data lives only in your promptfoo results. If you leave the team, the reasoning behind your model choice leaves with you.

An architecture decision record captures the decision, the options, the criteria, and the evidence in a format anyone on the team can read and re-evaluate. In this step, you'll write one using the MADR 4.0.0 template, citing your own promptfoo data in every section.

In this step, get ready to:
  • Create the ADR file using the MADR 4.0.0 template structure.
  • Fill in every section with your own evaluation data from the promptfoo grid.
  • Commit and push the entire project to GitHub.
Create the ADR file with MADR structure

MADR stands for Markdown Any Decision Records. It is a lightweight template for recording architecture decisions in a structured, version-controlled format.

  • Create the docs/decisions/ directory inside your model-evaluation folder by running this command:
mkdir -p docs/decisions
  • Confirm the directory was created by running:
ls docs

You should see decisions listed in the output.

What is an architecture decision record?

An ADR is a short Markdown document that records a single architecture decision: the context, the options considered, and the outcome chosen. It lives alongside your code in version control.

The MADR 4.0.0 template gives you a fixed set of sections to fill in. This means anyone reviewing your repository can find the reasoning behind a technical choice without asking the person who made it.

  • In your code editor, create a new file called 0001-model-selection.md inside the docs/decisions/ folder.
  • Paste the following content into the file:
---
status: accepted
date: 2026-08-25
decision-makers: Your Name
---

# Use Gemini 2.5 Flash for the Question Answering Service

## Context and Problem Statement

Our AI Question Answering Service needs a production-ready LLM for the /ask endpoint. Three team members each prefer a different model. Rather than argue from vendor benchmarks, we ran our own evaluation on 15 test cases drawn from our actual workload and scored each model on factual accuracy, response latency, and conciseness.

## Decision Drivers

* Must operate within a free tier with no budget allocated
* Response latency under 10 seconds for acceptable user experience
* Answers must be concise, under 150 words, and factually accurate
* Decision must be reproducible by running a single command

## Considered Options

* Gemini 2.5 Flash via Google AI Studio free tier
* Gemini 2.5 Flash-Lite via Google AI Studio free tier
* Meta Llama 3.2 3B Instruct via OpenRouter free tier

What do these sections do?

  • The YAML frontmatter (status, date, decision-makers) records who made the decision and when. Replace Your Name with your actual name.
  • Context and Problem Statement explains why you need to make this decision at all.
  • Decision Drivers lists the constraints that will determine the winner. These map directly to your promptfoo assertions: free tier, latency under 10 seconds, concise answers, and reproducibility.
  • Considered Options names the three models you evaluated.
  • Save 0001-model-selection.md.
  • If your editor has a Markdown preview (in VS Code, press Cmd+Shift+V on macOS or Ctrl+Shift+V on Windows), open it. You should see the title, context paragraph, four decision drivers as a bullet list, and three considered options.

File not showing in the sidebar?

  • Make sure you created the file inside docs/decisions/, not in the root of model-evaluation/.
  • Check that the filename is exactly 0001-model-selection.md with the .md extension.

Help me find where my ADR file ended up.

Fill in the Decision Outcome and Confirmation sections

The next three sections are where your promptfoo data turns into a documented decision. This is the part a teammate reads first when they open the ADR.

  • In 0001-model-selection.md, add the following content below the Considered Options section:
## Decision Outcome

Chosen option: "Gemini 2.5 Flash", because it achieved the highest pass rate on the quality rubric across 15 test cases while meeting latency and conciseness constraints. Evaluation command: `npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache`.

### Consequences

* Good, because the model passes all quality assertions on test cases drawn from our actual workload
* Good, because latency stays under 10 seconds on the free tier
* Good, because the same GOOGLE_API_KEY from our existing setup works with no new accounts required
* Bad, because the free tier has rate limits that would not support production traffic
* Bad, because free-tier data is used to improve Google products, which may not be acceptable for sensitive workloads
* Neutral, because this is a reversible decision. Switching models requires changing one line in the promptfoo config and one constant in model_provider.py

### Confirmation

Re-run the evaluation to verify: `npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache && npx promptfoo@latest view`. The results grid should show Gemini 2.5 Flash passing all assertions. If the model, free-tier terms, or our workload changes, re-evaluate.

Note: This evaluation used Gemini 2.5 Flash as the LLM judge for rubric-based assertions. Self-assessment bias is possible. A follow-up evaluation should re-run with an independent judge model to verify rankings.

What do these sections capture?

  • Decision Outcome names the winner and includes the exact command to reproduce the evaluation. Anyone can run it and verify the claim.
  • Consequences lists the Good, Bad, and Neutral outcomes of choosing this model. The "Neutral" entry flags this as a reversible decision.
  • Confirmation tells a future reader exactly how to re-verify the decision. It also flags the LLM-as-judge bias, because Gemini 2.5 Flash grades itself.
  • Save 0001-model-selection.md.
  • Check the Markdown preview. You should now see the Decision Outcome paragraph with the embedded command, a Consequences bullet list, and a Confirmation section that mentions the self-assessment bias.

Markdown preview not rendering?

  • Make sure you saved the file before refreshing the preview.
  • Check that the YAML frontmatter (the block between the two --- lines) has no stray characters. A missing dash or extra space can break the parser and stop everything below from rendering.

Help me fix my Markdown preview.

  • Next, add the Pros and Cons section below the Confirmation section. This covers the first two models. Paste the following:
## Pros and Cons of the Options

### Gemini 2.5 Flash

Google's mid-range model with enhanced reasoning and thinking capabilities.

* Good, because highest quality scores on the llm-rubric across 15 test cases
* Good, because consistent latency within the 10-second threshold
* Good, because already integrated in our Project 1 codebase via google-genai SDK
* Neutral, because free-tier rate limits are moderate
* Bad, because free-tier usage data is used to improve Google products

### Gemini 2.5 Flash-Lite

Google's cost-optimized model for high-volume, latency-sensitive tasks.

* Good, because fastest response times of all three options
* Good, because same API key and SDK as Flash with zero migration cost
* Neutral, because quality scores slightly lower than Flash on complex questions
* Bad, because less capable on nuanced or multi-part questions
  • Save the file and check the Markdown preview. You should see a Pros and Cons section with two model subsections, each listing Good, Neutral, and Bad bullet points.
  • Add the final model's pros and cons, plus the More Information section, below the Gemini 2.5 Flash-Lite subsection. Paste the following:
### Meta Llama 3.2 3B Instruct via OpenRouter

Meta's small open-weight model accessed through OpenRouter's free tier.

* Good, because completely independent provider, reducing vendor lock-in
* Good, because open-weight model with no data-training concerns
* Bad, because noticeably lower quality on complex or nuanced questions due to 3B parameter size
* Bad, because OpenRouter free tier limited to 50 requests per day
* Bad, because requires a separate API key and account

## More Information

Evaluation data generated by promptfoo (run via npx). Full results viewable by running `npx promptfoo@latest view` in the project directory. Test cases were drawn from the software architecture question-answering domain served by the Project 1 /ask endpoint. The prompt template mirrors Project 1's orchestrator system context.

Why document pros AND cons for every option?

A common mistake in decision records is only documenting the winner. When conditions change (a model gets a price increase, a new model launches, your workload shifts), the team needs to re-evaluate from the original set of options.

By documenting pros and cons for all three models, a future reader can see immediately whether a change in constraints (for example, a budget increase) would flip the decision.

  • Save 0001-model-selection.md.
  • Check the Markdown preview. You should see the complete ADR with all sections: frontmatter, title, Context, Decision Drivers, Considered Options, Decision Outcome, Consequences, Confirmation, Pros and Cons for all three models, and More Information.

✔️ Awesome, I've got everything!

Great. Make sure you have saved 0001-model-selection.md and that the Markdown preview shows all sections rendering correctly.

ⓧ I'd like to double check the full code

The full ADR is longer than what fits in a single block. Scroll up through this step and confirm you have all four content chunks in order:

  • Chunk 1: YAML frontmatter (status, date, decision-makers) through Considered Options (three bullet points).
  • Chunk 2: Decision Outcome, Consequences (six bullet points), and Confirmation (two paragraphs including the judge bias note).
  • Chunk 3: Pros and Cons of the Options heading, then Gemini 2.5 Flash (five bullet points) and Gemini 2.5 Flash-Lite (four bullet points).
  • Chunk 4: Meta Llama 3.2 3B Instruct via OpenRouter (five bullet points), then More Information (one paragraph).

Confirm each chunk is present in order with no blank gaps between sections.

Why call out the LLM-as-judge limitation?

Your llm-rubric assertion uses Gemini 2.5 Flash as the judge (set in defaultTest.options.provider). That same model is also a candidate being judged. It is grading its own answers.

Flagging this in the Confirmation section is honest engineering. It tells a future reader exactly which result to question and how to verify it independently.

Commit and push to GitHub

Your ADR is complete, but it only exists on your machine. Pushing it to GitHub makes it readable by anyone with access to the repository and renders the Markdown automatically.

  • Initialize a Git repository in your model-evaluation folder and stage all files by running these commands:
git init
git add .
git commit -m "Add model evaluation config and architecture decision record"
  • You should see a list of new files in the commit output: prompt.txt, promptfooconfig.yaml, and docs/decisions/0001-model-selection.md.
  • Create a new repository on GitHub called model-evaluation. You can do this through the GitHub website or using the GitHub CLI.
  • Push your local repository to GitHub by running these commands (replace the URL with your own repository URL):
git branch -M main
git remote add origin https://github.com/your-username/model-evaluation.git
git push -u origin main

Before you check GitHub, what do you expect the ADR to look like in the browser? Will it render as formatted Markdown or show raw text?

  • Open your repository on GitHub in the browser.
  • Navigate to docs/decisions/0001-model-selection.md.

GitHub automatically renders Markdown files. You should see the full ADR with formatted headings, bullet lists, and the YAML frontmatter displayed as a table at the top. The Decision Outcome section cites the exact evaluation command anyone can run to reproduce your results.

Push rejected or authentication error?

  • If you see a "permission denied" or "authentication failed" error, check that you are logged into Git with your GitHub credentials. Run gh auth status if you have the GitHub CLI installed.
  • If the remote URL is wrong, remove it and re-add it with the correct URL by running git remote remove origin followed by the git remote add command again with the correct URL.

Help me push my repository to GitHub.

Your model selection is now documented, reproducible, and version-controlled. Anyone on your team can open the ADR, read the reasoning, and re-run the evaluation with a single command. Next up, you'll take on an optional challenge to test whether your evaluation itself is trustworthy.

Secret mission

Test the Judge for Bias

You used Gemini 2.5 Flash as both a candidate and the judge in your evaluation. That is a conflict of interest. Does the model grade itself more favorably than it grades the competition? Swap the grading model to a different provider, re-run the evaluation, and find out whether the rankings hold up.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely locally with free-tier API calls, so there are no ongoing costs.

Resources you used:

  • The model-evaluation project folder on your Desktop.
  • Your GitHub repository containing the evaluation config and ADR.
  • Your Google AI Studio API key (free tier).
  • Your OpenRouter API key (free tier).

Keep everything running

No action needed. Choose this if you plan to revisit your evaluation, add more models, or extend your test cases.

  • Your model-evaluation folder stays on your Desktop, ready for future evaluations.
  • Your API keys remain active on the free tier with no charges.
  • Your ADR and evaluation configs stay in your GitHub repository for reference.

Pause - I'll come back to this later

Close any running processes to free up your terminal. Your files and API keys stay in place for when you return.

  • If the promptfoo web view is still open in your terminal, press Ctrl+C to stop the server.
  • Close your terminal window. The GOOGLE_API_KEY and OPENROUTER_API_KEY environment variables are session-scoped, so they disappear when the terminal closes.
  • When you return, re-export your API keys in a new terminal and navigate back into the model-evaluation folder to pick up where you left off.

Delete - I don't want to use this again

Remove all project files and revoke your API keys to clean up completely.

  • If the promptfoo web view is still running, press Ctrl+C in that terminal to stop it.

Delete the model-evaluation folder from your Desktop.

macOS

  • Run this command to delete the project folder:
rm -rf ~/Desktop/model-evaluation

Windows

  • Run this command to delete the project folder:
rmdir /s /q "%USERPROFILE%\Desktop\model-evaluation"
  • Delete your GitHub repository. Open the repository on GitHub, click Settings, scroll down to the Danger Zone section, and click Delete this repository.
  • Revoke your Google AI Studio API key. Visit aistudio.google.com, open your API keys list, and delete the key you created for this project.
  • Revoke your OpenRouter API key. Visit openrouter.ai/keys and delete the key you created for this project.

Nice Work!

Nice Work!

You just built a reproducible model evaluation and wrote an architecture decision record that cites your own data. No vendor benchmarks, no gut feelings, just numbers your team can re-run with a single command.

You've learned how to:

  • Set up promptfoo to compare multiple LLMs side by side using a YAML config, with providers spanning Google AI Studio and OpenRouter free tiers.
  • Design structured evaluation criteria (llm-rubric, latency, word-count) that expose the gap between subjective "vibes" assessment and reproducible measurement.
  • Write an architecture decision record in MADR 4.0.0 format where every section is backed by your own reproducible evaluation data.
  • Secret Mission: An optional challenge to push your evaluation skills further.

Ready to quiz yourself?