Evaluate RAG Reliability with FastAPI

Introduction

30 Second Summary

A polished answer can feel trustworthy even when it comes from the wrong source. Without a clear check, confidence can hide a factual mistake.

In this project, you will build a small policy-question API that exposes a deliberate RAG failure. A deterministic evaluator will compare weak lexical retrieval with improved semantic retrieval.

What You'll Build

The finished lab turns one policy question into a visible before-and-after scorecard where the wrong 90-day answer sits beside the correct 30-day result.

By the end of this project, you'll have:

• A FastAPI demo packaged with Docker Compose that you can query through interactive documentation in your browser.
• A designed failure where contractor-access produces 90 days while employee-mobility proves the answer should be 30 calendar days.
• A deterministic scorecard that saves Hit@1, Recall@2, MRR, groundedness, and reference correctness as JSON and Markdown reports.
• Secret Mission: Compare the deterministic verdict with a Gemini LLM-as-judge score to expose evaluator disagreement.

Are there any prerequisites?

You should be comfortable with Python and have a Google account for the included Gemini API-key setup. The local lab requires Python 3.10 or newer plus Docker Desktop.

Before We Start

This first checkpoint defines the failure you are setting out to prove before any hands-on work begins. Your goal is to reproduce a fluent 90-day international remote-work answer grounded in the wrong contractor policy. Your scorecard must separate retrieval quality, groundedness, and reference correctness.

Set Up the Lab

Before the lab can expose the wrong 90-day answer, every run needs a known Python environment. Pinned packages keep the evaluation foundation repeatable.

The live demo also connects to the Gemini Developer API. This setup keeps the credential outside your project before any RAG request runs.

In this step, get ready to:

• Prepare a local virtual environment with Python 3.10 or newer.
• Install the pinned dependencies inside the lab folder.
• Store GEMINI_API_KEY in ~/.zshrc without adding the credential to Git.

Prepare the Python workspace

The project dependencies require Python 3.10 or newer. Checking the runtime first prevents compatibility problems during installation.

• Press Cmd+Space on macOS to open the search bar.
• Type Terminal into the search bar.
• Press Enter to open Terminal.
• Check the installed Python version by running this command:

python3 --version

What does this check do?

The --version option prints the Python version before exiting. Use the result to choose the matching path below.

✔️ I see version 3.10 or higher

That clears the first compatibility gate. Your Python runtime can install every pinned package in this lab.

ⓧ I see an older version

macOS can keep its Apple-managed Python beside a newer installation. The Python.org installer keeps both versions intact while placing the newer python3 first on your shell path.

• Visit the official Python releases for macOS page.
• Download a current macOS installer.
• Run the downloaded installer using its default options.
• Close Terminal after the installer finishes.
• Press Cmd+Space to open the macOS search bar.
• Type Terminal into the search bar.
• Press Enter to start a fresh Terminal session.
• Check the updated version by running this command:

python3 --version

What should you see?

The command now prints a Python version that meets or exceeds the project requirement.

ⓧ Command not found

Your shell cannot find a Python installation yet. The official macOS installer provides the runtime plus the python3 command.

• Visit the official Python releases for macOS page.
• Download a current macOS installer.
• Run the downloaded installer using its default options.
• Close Terminal after the installer finishes.
• Press Cmd+Space to open the macOS search bar.
• Type Terminal into the search bar.
• Press Enter to start a fresh Terminal session.
• Confirm Python is available by running this command:

python3 --version

What should you see?

The command now prints a Python version that meets or exceeds the project requirement.

Still seeing the wrong Python version?

Close every Terminal window after installing Python. A fresh Terminal session reloads the shell path created by the installer.

If the older version remains, confirm that you used the Python.org macOS installer. Leave the Apple-managed Python installation unchanged.

Get help checking which Python installation your shell is using.

Keeping the lab on your Desktop gives the project a predictable location. The folder name also makes it easy to identify in Terminal and VS Code.

• Create the rag-evaluation-lab folder on your Desktop by running these commands:

cd ~/Desktop
mkdir rag-evaluation-lab
ls
cd rag-evaluation-lab

What do these commands do?

• The first command moves Terminal to your Desktop.
• The second command creates the rag-evaluation-lab folder.
• The third command lists the Desktop contents so you can confirm the folder exists.
• The final command moves Terminal into the new folder.

You should see rag-evaluation-lab in the folder listing. Terminal then uses that folder as its current location.

Folder already exists?

An existing folder with the same name makes the creation command report that the directory already exists. Reuse that folder only if it does not contain another project.

Get help checking the folder safely before continuing.

• Create and activate .venv inside rag-evaluation-lab by running these commands:

python3 -m venv .venv
source .venv/bin/activate

Why isolate the lab?

The first command creates a dedicated Python environment inside .venv. The second command makes that environment active for the current Terminal session.

Installing packages globally can mix versions from unrelated projects. The virtual environment keeps this lab's dependencies together.

Your Terminal prompt now includes the .venv environment name. That visible prefix confirms the isolated environment is active.

Virtual environment not active?

Confirm that Terminal is inside the rag-evaluation-lab folder. The activation path only works from the folder containing .venv.

Get help locating or activating the environment.

• Press Cmd+Space on macOS or the Windows key on Windows to open the search bar.
• Type VS Code into the search bar.
• Press Enter to open VS Code.
• Click File in the top menu.
• Click Open Folder.
• Select the rag-evaluation-lab folder on your Desktop.
• Click Open.

The Explorer sidebar now shows rag-evaluation-lab as the open folder.

Install the pinned dependencies

A pinned requirements.txt file records the exact package versions used by the lab. Reinstalling from this file recreates the same dependency set.

• Click the New File button in the VS Code Explorer sidebar.
• Name the file requirements.txt.
• Add the pinned dependencies by pasting this content:

fastapi==0.141.1
uvicorn==0.53.0
google-genai==2.24.0
pydantic==2.13.5
pytest==9.1.1
httpx==0.28.1

What does each dependency provide?

• FastAPI defines the HTTP API.
• Uvicorn runs the application locally.
• Google Gen AI Python SDK connects the live provider to Gemini.
• Pydantic validates request data plus structured responses.
• pytest runs the deterministic tests.
• HTTPX supports FastAPI's test client.

• Save requirements.txt.
• Switch back to the Terminal window from earlier.
• Install every pinned dependency into the active .venv by running this command:

python3 -m pip install -r requirements.txt

What does this command do?

Python runs pip from the active environment. The -r option tells it to install every package listed in requirements.txt.

The installation can take a couple of minutes while the packages download. Terminal continues printing progress during the process.

When installation finishes, Terminal returns to the active .venv prompt without an installation failure.

Dependencies not installing?

Confirm that the Terminal prompt still shows .venv. Check that every line in requirements.txt matches the pinned file above.

A network interruption can also stop a package download. Restore the connection before running the same installation command again.

Get help diagnosing the installation output.

Configure the Gemini credential

An API key authorizes the live provider to call Gemini. This credential stays in your shell configuration outside the project folder.

Credential setup can feel risky. Keep the key off-screen whenever you take screenshots.

• Return to the Explorer sidebar in VS Code.
• Click the New File button.
• Name the file .gitignore.
• Protect the local environment files by pasting this content:

.venv/
__pycache__/
.pytest_cache/
.env
reports/

What does this file protect?

The ignore rules keep the virtual environment plus generated caches out of future Git commits. They also exclude local .env files plus generated reports.

The Gemini key lives in ~/.zshrc outside this folder. No credential belongs in the project files.

• Save .gitignore.
• Confirm that requirements.txt plus .gitignore appear in the VS Code Explorer sidebar.

✔️ Awesome, I've got everything!

Both project files are saved. Your dependency pins plus ignore rules are ready.

ⓧ I'd like to double check the full code

Compare your requirements.txt file with this complete version.

fastapi==0.141.1
uvicorn==0.53.0
google-genai==2.24.0
pydantic==2.13.5
pytest==9.1.1
httpx==0.28.1

Compare your .gitignore file with this complete version.

.venv/
__pycache__/
.pytest_cache/
.env
reports/

• Open Google AI Studio in your browser.
• Complete the Google account sign-in if prompted.
• Open the API Keys page.
• Click Create API key if you do not already have an auth key for this lab.
• Copy the new key when Google AI Studio displays it.
• Keep the copied key out of screenshots plus project files.

• Switch back to the Terminal window from earlier.
• Open ~/.zshrc in nano by running this command:

nano ~/.zshrc

Why use this file?

The ~ path points to your macOS home folder. The ~/.zshrc file loads shell settings when Zsh starts.

Nano opens the file directly in Terminal. Its shortcut guide appears along the bottom of the screen.

• Move the cursor to a blank line at the end of ~/.zshrc.
• Add the Gemini API key export by copying this line into nano:

export GEMINI_API_KEY=<YOUR_API_KEY_HERE>

How does the key reach the SDK?

The export creates the GEMINI_API_KEY environment variable for your shell sessions. The Google Gen AI Python SDK reads that variable when it creates a Gemini client.

The placeholder must be replaced before saving. Your real key should never appear in this project folder.

• Replace <YOUR_API_KEY_HERE> inside nano with the key you copied.
• Press Ctrl+O to save the file.
• Press Enter to confirm the filename.
• Press Ctrl+X to exit nano.
• Load the updated shell configuration plus reactivate the virtual environment by running these commands:

source ~/.zshrc
source .venv/bin/activate

What happens when these commands run?

The first command loads the new environment variable into the current Terminal session. The second command restores .venv as the active Python environment.

Neither command prints the key. Terminal returns to the prompt with the environment ready.

Your credential now lives in ~/.zshrc outside rag-evaluation-lab. The project files contain no copy of the key.

Credential not loading?

Reopen ~/.zshrc and confirm the export line has no spaces around the equals sign. Check that the placeholder was fully replaced.

If both GEMINI_API_KEY plus GOOGLE_API_KEY already exist in your shell, the SDK gives GOOGLE_API_KEY precedence.

Get help checking the environment variable without revealing its value.

Before you run the final check, predict whether both imports will succeed in the active environment.

• Verify the installed dependencies by running this command:

python3 -c "from google import genai; import fastapi; print('dependencies ready')"

What does this check prove?

Python imports the Gemini SDK plus FastAPI from the active environment. It prints the confirmation only after both imports succeed.

The terminal prints dependencies ready. You've cleared the setup gate with a reproducible environment plus a separately stored credential.

Import check not passing?

Confirm that the prompt shows .venv. An inactive environment makes Python search outside the packages you installed.

Check that the dependency installation finished successfully before rerunning the import check.

Get help tracing the failed import.

Your lab now has a pinned Python environment plus a protected hosted-model credential. Next, you'll use that foundation to reproduce the deliberately misleading RAG answer.

Build the Designed Retrieval Failure

Your Python environment is ready to host the evaluation lab. The next challenge is creating a RAG path that can sound convincing after retrieving the wrong policy.

This step builds that failure deliberately. Similar wording across employee and contractor policies gives lexical retrieval a realistic trap.

In this step, get ready to:

• Define the policy contracts and labeled evaluation cases.
• Build hosted and deterministic providers behind one protocol.
• Run the lexical request through FastAPI to expose the misleading answer.

Define the policy contracts and cases

The lab needs a shared data shape before retrieval can rank anything. Pydantic contracts keep questions, chunks, answers, citations, and scorecards consistent across the application.

• Create an app folder from the VS Code file sidebar.
• Create an empty app/__init__.py file inside that folder.
• Create app/models.py inside the app folder.
• Create app/knowledge.py inside the app folder.

You should now see __init__.py, models.py, and knowledge.py under the app folder.

• Add the core chunk and answer contracts to app/models.py by pasting this code:

from typing import Literal

from pydantic import BaseModel, Field


class Chunk(BaseModel):
    id: str
    title: str
    text: str


class EvalCase(BaseModel):
    question: str
    expected_chunk_id: str
    required_facts: list[str]


class Claim(BaseModel):
    text: str
    citation_id: str


class AnswerPayload(BaseModel):
    answer: str
    claims: list[Claim]

What do these contracts represent?

• The Chunk contract gives every policy passage a stable ID, title, and body.
• The EvalCase contract records the correct passage and required facts for a labeled question.
• The AnswerPayload contract pairs a generated answer with cited factual claims.

• Add the request, retrieval, and response contracts below AnswerPayload by pasting this code:

class RetrievedChunk(BaseModel):
    id: str
    title: str
    text: str
    score: float


class AskRequest(BaseModel):
    question: str
    retriever: Literal["lexical", "semantic"] = "lexical"
    top_k: int = Field(default=1, ge=1, le=3)


class AskResponse(BaseModel):
    answer: str
    claims: list[Claim]
    retrieved: list[RetrievedChunk]

How is the API boundary controlled?

The request limits retrieval to the supported strategies. It also keeps top_k between one and three so each response stays focused.

• Save app/models.py.

VS Code should show the new classes without Python syntax warnings.

• Complete app/models.py by adding the evaluation contracts below AskResponse.

class MetricSummary(BaseModel):
    hit_at_1: float
    recall_at_2: float
    mrr: float
    groundedness: float
    reference_correctness: float


class EvaluationRun(BaseModel):
    retriever: str
    metrics: MetricSummary


class Scorecard(BaseModel):
    baseline: EvaluationRun
    semantic: EvaluationRun
    delta: MetricSummary

Why define evaluation models now?

These contracts establish the scorecard shape used by later evaluation routes. A stable schema lets the lexical baseline and semantic result use the same metric fields.

• Save app/models.py.
• Confirm VS Code shows no syntax warnings in app/models.py.

Seeing model syntax warnings?

Check that every class body remains indented by four spaces. Confirm that each field stays inside its matching class.

If a name appears unresolved, confirm the first two imports match the code above.

Need another pair of eyes? Help me inspect my Pydantic model definitions.

The policy collection contains three deliberately confusing pairs. Each labeled case identifies the passage that should rank highest even when another passage shares more obvious words.

• Start app/knowledge.py with the model import and the first two policy chunks:

from app.models import Chunk, EvalCase


CHUNKS = [
    Chunk(
        id="contractor-access",
        title="Contractor International Access",
        text=(
            "Contractors requesting international remote-work system access must "
            "submit the access request 90 days before departure."
        ),
    ),
    Chunk(
        id="employee-mobility",
        title="Employee Temporary Mobility",
        text=(
            "Employees planning to work temporarily from another country must send "
            "the mobility form at least 30 calendar days before departure."
        ),
    ),
]

Where does the ambiguity come from?

Both passages discuss international work before departure. The contractor passage repeats more words from the designed question, while the employee passage contains the correct rule.

• Insert the two meal-policy chunks immediately above the closing bracket of CHUNKS.

    Chunk(
        id="team-event-meals",
        title="Team Event Meals",
        text=(
            "Employees must submit a meal receipt for team events costing more than "
            "80 US dollars."
        ),
    ),
    Chunk(
        id="individual-meals",
        title="Individual Meal Expenses",
        text="A receipt becomes mandatory when one meal costs more than 25 US dollars.",
    ),

What does this policy pair test?

The meal passages share employee, receipt, and expense language. Their different thresholds make ranking quality measurable against a specific required fact.

• Save app/knowledge.py.

The file should now contain four valid Chunk entries inside CHUNKS.

• Insert the final two policy chunks immediately above the closing bracket of CHUNKS.

    Chunk(
        id="damaged-equipment",
        title="Damaged Home-Office Equipment",
        text=(
            "New hires have 14 calendar days to report damaged home-office equipment."
        ),
    ),
    Chunk(
        id="workspace-stipend",
        title="New Hire Workspace Stipend",
        text=(
            "People joining the company may request the workspace stipend during "
            "their first 60 calendar days."
        ),
    ),

Why add another confusing pair?

The final pair separates damaged-equipment reporting from the workspace allowance. This tests whether retrieval follows meaning when the question uses different wording.

• Add the three labeled cases below the closing bracket of CHUNKS.

EVAL_CASES = [
    EvalCase(
        question=(
            "How early must employees submit an international remote-work request?"
        ),
        expected_chunk_id="employee-mobility",
        required_facts=["30 calendar days"],
    ),
    EvalCase(
        question="When must employees submit a receipt for a meal expense?",
        expected_chunk_id="individual-meals",
        required_facts=["more than 25 US dollars"],
    ),
    EvalCase(
        question=(
            "How long do new hires have to claim the home-office equipment allowance?"
        ),
        expected_chunk_id="workspace-stipend",
        required_facts=["60 calendar days"],
    ),
]

What makes these cases labeled?

Each question includes an expected chunk ID and a required fact. Those labels provide an external reference for checking retrieval and answer correctness.

• Save app/knowledge.py.
• Confirm the file contains six chunks and three evaluation cases.

Seeing a list or indentation error?

Confirm every Chunk entry sits inside the CHUNKS brackets. Check that EVAL_CASES begins after the chunk list closes.

Check the commas after each model instance. A missing comma can make the following entry look invalid.

Still blocked? Help me find the structure error in my policy fixtures.

Build the providers and retrieval path

A provider protocol keeps retrieval independent from the model implementation. The live path can use Gemini, while deterministic fixtures can produce repeatable answers and vectors.

Why use a provider protocol?

A structural protocol defines the capabilities the application needs. Any provider with matching embedding and answer methods can be substituted without changing retrieval.

• Create app/providers.py inside the app folder.
• Create app/retrieval.py inside the app folder.

You should now see both provider and retrieval files beside the policy contracts.

• Start app/providers.py with the model IDs and provider protocol:

from __future__ import annotations

from typing import Protocol

from google import genai

from app.models import AnswerPayload, Chunk, Claim


GENERATION_MODEL = "gemini-3.8-flash"
EMBEDDING_MODEL = "gemini-embedding-2"


class ModelProvider(Protocol):
    def embed_documents(self, chunks: list[Chunk]) -> dict[str, list[float]]:
        ...

    def embed_query(self, question: str) -> list[float]:
        ...

    def answer(self, question: str, contexts: list[Chunk]) -> AnswerPayload:
        ...

What does the protocol guarantee?

• Document embeddings map every chunk ID to a vector.
• Query embeddings convert a question into the same vector space.
• The answer method returns the shared AnswerPayload contract.

• Add the Gemini client and embedding methods below ModelProvider.

class GeminiProvider:
    def __init__(self) -> None:
        self.client = genai.Client()

    def embed_documents(self, chunks: list[Chunk]) -> dict[str, list[float]]:
        vectors: dict[str, list[float]] = {}
        for chunk in chunks:
            content = f"title: {chunk.title} | text: {chunk.text}"
            result = self.client.models.embed_content(
                model=EMBEDDING_MODEL,
                contents=content,
            )
            vectors[chunk.id] = list(result.embeddings[0].values or [])
        return vectors

    def embed_query(self, question: str) -> list[float]:
        content = f"task: question answering | query: {question}"
        result = self.client.models.embed_content(
            model=EMBEDDING_MODEL,
            contents=content,
        )
        return list(result.embeddings[0].values or [])

How are hosted embeddings prepared?

Documents include their title and text. Questions use the question-answering query prefix so the embedding model can distinguish a search query from searchable content.

• Save app/providers.py.

VS Code should recognize GeminiProvider as an implementation with both embedding methods.

• Complete GeminiProvider by adding this answer method below embed_query.

    def answer(self, question: str, contexts: list[Chunk]) -> AnswerPayload:
        context_text = "\n\n".join(
            f"[{chunk.id}] {chunk.title}: {chunk.text}" for chunk in contexts
        )
        prompt = f"""
Answer the question using only the supplied context. Do not use prior knowledge.
Return a concise fluent answer. For each factual claim, include an exact supporting
quote copied from the context and the matching citation ID.

Question: {question}

Context:
{context_text}
"""
        interaction = self.client.interactions.create(
            model=GENERATION_MODEL,
            input=prompt,
            response_format={
                "type": "text",
                "mime_type": "application/json",
                "schema": AnswerPayload.model_json_schema(),
            },
        )
        return AnswerPayload.model_validate_json(interaction.output_text)

How is the answer constrained?

The prompt limits the answer to retrieved context. Structured output makes the model return an answer plus exact supporting quotes and citation IDs that Pydantic can validate.

• Start FixtureProvider below GeminiProvider with the fixed document vectors and first query vector.

class FixtureProvider:
    _vectors = {
        "employee-mobility": [1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
        "contractor-access": [0.0, 1.0, 0.0, 0.0, 0.0, 0.0],
        "individual-meals": [0.0, 0.0, 1.0, 0.0, 0.0, 0.0],
        "team-event-meals": [0.0, 0.0, 0.0, 1.0, 0.0, 0.0],
        "workspace-stipend": [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
        "damaged-equipment": [0.0, 0.0, 0.0, 0.0, 0.0, 1.0],
    }

    _query_vectors = {
        "How early must employees submit an international remote-work request?": [
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0,
        ],
    }

Why use fixed vectors?

Each expected policy receives its own vector direction. Matching query vectors make later semantic checks deterministic without a network request.

• Insert the remaining query vectors immediately above the closing bracket of _query_vectors.

        "When must employees submit a receipt for a meal expense?": [
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            0.0,
        ],
        "How long do new hires have to claim the home-office equipment allowance?": [
            0.0,
            0.0,
            0.0,
            0.0,
            1.0,
            0.0,
        ],

What do the remaining vectors cover?

The extra vectors map the meal question to individual-meals and the allowance question to workspace-stipend.

• Save app/providers.py.

The fixture now contains six document vectors and three query vectors.

• Add the first three deterministic answers below _query_vectors.

    _answers = {
        "contractor-access": (
            "Employees should submit the international remote-work request "
            "90 days before departure.",
            "90 days before departure",
        ),
        "employee-mobility": (
            "Employees must submit the mobility form at least 30 calendar days "
            "before departure.",
            "at least 30 calendar days before departure",
        ),
        "team-event-meals": (
            "Employees must submit a meal receipt when the event costs more than "
            "80 US dollars.",
            "more than 80 US dollars",
        ),
    }

What makes an answer deterministic?

Every chunk ID maps to a fixed answer and supporting quote. The same retrieved passage therefore produces the same output on every offline run.

• Insert the final three deterministic answers immediately above the closing bracket of _answers.

        "individual-meals": (
            "A receipt is required when one meal costs more than 25 US dollars.",
            "more than 25 US dollars",
        ),
        "damaged-equipment": (
            "New hires have 14 calendar days to report damaged home-office equipment.",
            "14 calendar days",
        ),
        "workspace-stipend": (
            "New hires may request the workspace stipend during their first "
            "60 calendar days.",
            "first 60 calendar days",
        ),

How do the fixtures preserve citations?

Each tuple contains a fluent answer and an exact quote. The provider assigns the retrieved chunk ID to that quote when it creates a claim.

• Complete FixtureProvider by adding its protocol methods below _answers.

    def embed_documents(self, chunks: list[Chunk]) -> dict[str, list[float]]:
        return {chunk.id: self._vectors[chunk.id] for chunk in chunks}

    def embed_query(self, question: str) -> list[float]:
        return self._query_vectors[question]

    def answer(self, question: str, contexts: list[Chunk]) -> AnswerPayload:
        primary = contexts[0]
        answer, quote = self._answers[primary.id]
        return AnswerPayload(
            answer=answer,
            claims=[Claim(text=quote, citation_id=primary.id)],
        )

How does the fixture satisfy the protocol?

The methods expose fixed vectors and select the answer tied to the first retrieved chunk. That behavior makes retrieval order directly visible in the returned answer.

• Save app/providers.py.
• Confirm VS Code shows no unresolved indentation around the two provider classes.

Provider class not lining up?

Confirm the Gemini methods sit inside GeminiProvider. Confirm every fixed dictionary and fixture method sits inside FixtureProvider.

Check that the inserted dictionary entries appear before their dictionary's closing brace.

Need help checking the class boundaries? Help me inspect my provider indentation.

Lexical retrieval converts the question and each policy into token sets. It ranks passages by the number of shared lowercase words.

• Add tokenization and lexical ranking to app/retrieval.py.

import math
import re

from app.knowledge import CHUNKS
from app.models import RetrievedChunk
from app.providers import ModelProvider


def tokenize(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower()))


def lexical_retrieve(question: str, top_k: int) -> list[RetrievedChunk]:
    query_terms = tokenize(question)
    ranked = [
        RetrievedChunk(
            id=chunk.id,
            title=chunk.title,
            text=chunk.text,
            score=float(len(query_terms & tokenize(chunk.text))),
        )
        for chunk in CHUNKS
    ]
    return sorted(ranked, key=lambda item: item.score, reverse=True)[:top_k]

How does lexical ranking work?

The tokenizer keeps lowercase letters and numbers. Each score counts the tokens shared by the question and a policy chunk.

• Add cosine similarity below lexical_retrieve.

def cosine_similarity(left: list[float], right: list[float]) -> float:
    if len(left) != len(right) or not left:
        return 0.0
    dot = sum(a * b for a, b in zip(left, right))
    left_norm = math.sqrt(sum(value * value for value in left))
    right_norm = math.sqrt(sum(value * value for value in right))
    if left_norm == 0.0 or right_norm == 0.0:
        return 0.0
    return dot / (left_norm * right_norm)

Why include cosine similarity here?

The shared retrieval module supports both lexical and semantic strategies. Cosine similarity provides the vector comparison used by the semantic path in the next improvement stage.

• Save app/retrieval.py.

The file should now contain a lexical ranking function and a vector-similarity helper.

• Add semantic ranking below cosine_similarity.

def semantic_retrieve(
    question: str,
    top_k: int,
    provider: ModelProvider,
) -> list[RetrievedChunk]:
    document_vectors = provider.embed_documents(CHUNKS)
    query_vector = provider.embed_query(question)
    ranked = [
        RetrievedChunk(
            id=chunk.id,
            title=chunk.title,
            text=chunk.text,
            score=cosine_similarity(query_vector, document_vectors[chunk.id]),
        )
        for chunk in CHUNKS
    ]
    return sorted(ranked, key=lambda item: item.score, reverse=True)[:top_k]

How is semantic ranking isolated?

The retriever asks only for the protocol's embedding methods. It does not need to know whether Gemini or a deterministic fixture produced the vectors.

• Complete app/retrieval.py with the strategy selector.

def retrieve(
    question: str,
    retriever: str,
    top_k: int,
    provider: ModelProvider,
) -> list[RetrievedChunk]:
    if retriever == "semantic":
        return semantic_retrieve(question, top_k, provider)
    return lexical_retrieve(question, top_k)

What does the selector control?

A semantic request uses provider embeddings. Every other supported request follows the lexical baseline that creates this step's designed failure.

• Save app/retrieval.py.
• Confirm VS Code shows no syntax warnings in the retrieval functions.

Retrieval imports unresolved?

Confirm knowledge.py, models.py, and providers.py are all inside the same app folder.

Check that app/__init__.py exists and remains empty.

Still seeing an import problem? Help me trace my application imports.

Run the designed failure

The API connects the request contract, retrieval strategy, and model provider. Its response exposes both the fluent answer and the ranked passages that produced it.

• Create app/main.py inside the app folder.
• Add the imports and default provider factory to app/main.py.

from collections.abc import Callable
from typing import Annotated

from fastapi import Depends, FastAPI

from app.models import AskRequest, AskResponse, Chunk
from app.providers import GeminiProvider, ModelProvider
from app.retrieval import retrieve


def get_provider() -> ModelProvider:
    return GeminiProvider()

Why use a provider factory?

The default factory creates the hosted provider for live requests. Passing a different factory later lets tests replace it without changing the route.

• Add the application factory and root route below get_provider.

def create_app(
    provider_factory: Callable[[], ModelProvider] = get_provider,
) -> FastAPI:
    application = FastAPI(
        title="Deterministic RAG Evaluation Lab",
        version="1.0.0",
    )

    @application.get("/")
    def root() -> dict[str, str]:
        return {"message": "RAG evaluation lab is ready"}

    return application


app = create_app()

What does the application factory provide?

The factory creates the FastAPI application and its readiness route. The module-level app gives Uvicorn the application object it needs to serve.

• Save app/main.py.

The application now has a valid import target and a root response.

• Insert the request route inside create_app immediately above return application.

    @application.post("/ask", response_model=AskResponse)
    def ask(
        request: AskRequest,
        provider: Annotated[ModelProvider, Depends(provider_factory)],
    ) -> AskResponse:
        ranked = retrieve(
            request.question,
            request.retriever,
            request.top_k,
            provider,
        )
        contexts = [
            Chunk(id=item.id, title=item.title, text=item.text) for item in ranked
        ]
        answer = provider.answer(request.question, contexts)
        return AskResponse(
            answer=answer.answer,
            claims=answer.claims,
            retrieved=ranked,
        )

What happens inside the request route?

• Dependency injection supplies the active model provider.
• Retrieval ranks the policy chunks with the requested strategy.
• The provider answers from those chunks before the route returns the answer, claims, and ranking evidence.

• Save app/main.py.
• Confirm the ask function remains indented inside create_app.

Application route not recognized?

Confirm the route decorator and ask function use four spaces of indentation. Keep return application after the route definition.

Check that app = create_app() remains outside the factory with no indentation.

Need help with the factory structure? Help me inspect my FastAPI application factory.

Use this final comparison to catch missing lines before starting the server.

✔️ Awesome, I've got everything!

Great. Save every file before moving to the live request.

ⓧ I'd like to double check the full code

Your app/__init__.py file should be empty.

from typing import Literal

from pydantic import BaseModel, Field


class Chunk(BaseModel):
    id: str
    title: str
    text: str


class EvalCase(BaseModel):
    question: str
    expected_chunk_id: str
    required_facts: list[str]


class Claim(BaseModel):
    text: str
    citation_id: str


class AnswerPayload(BaseModel):
    answer: str
    claims: list[Claim]


class RetrievedChunk(BaseModel):
    id: str
    title: str
    text: str
    score: float


class AskRequest(BaseModel):
    question: str
    retriever: Literal["lexical", "semantic"] = "lexical"
    top_k: int = Field(default=1, ge=1, le=3)


class AskResponse(BaseModel):
    answer: str
    claims: list[Claim]
    retrieved: list[RetrievedChunk]


class MetricSummary(BaseModel):
    hit_at_1: float
    recall_at_2: float
    mrr: float
    groundedness: float
    reference_correctness: float


class EvaluationRun(BaseModel):
    retriever: str
    metrics: MetricSummary


class Scorecard(BaseModel):
    baseline: EvaluationRun
    semantic: EvaluationRun
    delta: MetricSummary

What should match in the contracts?

Your file should contain ten Pydantic models. The request must support lexical and semantic retrieval with top_k limited to one through three.

from app.models import Chunk, EvalCase


CHUNKS = [
    Chunk(
        id="contractor-access",
        title="Contractor International Access",
        text=(
            "Contractors requesting international remote-work system access must "
            "submit the access request 90 days before departure."
        ),
    ),
    Chunk(
        id="employee-mobility",
        title="Employee Temporary Mobility",
        text=(
            "Employees planning to work temporarily from another country must send "
            "the mobility form at least 30 calendar days before departure."
        ),
    ),
    Chunk(
        id="team-event-meals",
        title="Team Event Meals",
        text=(
            "Employees must submit a meal receipt for team events costing more than "
            "80 US dollars."
        ),
    ),
    Chunk(
        id="individual-meals",
        title="Individual Meal Expenses",
        text="A receipt becomes mandatory when one meal costs more than 25 US dollars.",
    ),
    Chunk(
        id="damaged-equipment",
        title="Damaged Home-Office Equipment",
        text=(
            "New hires have 14 calendar days to report damaged home-office equipment."
        ),
    ),
    Chunk(
        id="workspace-stipend",
        title="New Hire Workspace Stipend",
        text=(
            "People joining the company may request the workspace stipend during "
            "their first 60 calendar days."
        ),
    ),
]


EVAL_CASES = [
    EvalCase(
        question=(
            "How early must employees submit an international remote-work request?"
        ),
        expected_chunk_id="employee-mobility",
        required_facts=["30 calendar days"],
    ),
    EvalCase(
        question="When must employees submit a receipt for a meal expense?",
        expected_chunk_id="individual-meals",
        required_facts=["more than 25 US dollars"],
    ),
    EvalCase(
        question=(
            "How long do new hires have to claim the home-office equipment allowance?"
        ),
        expected_chunk_id="workspace-stipend",
        required_facts=["60 calendar days"],
    ),
]

What should match in the policy data?

Your file should contain six fictional chunks and three labeled cases. The expected IDs must point to the employee, individual meal, and workspace stipend policies.

from __future__ import annotations

from typing import Protocol

from google import genai

from app.models import AnswerPayload, Chunk, Claim


GENERATION_MODEL = "gemini-3.8-flash"
EMBEDDING_MODEL = "gemini-embedding-2"


class ModelProvider(Protocol):
    def embed_documents(self, chunks: list[Chunk]) -> dict[str, list[float]]:
        ...

    def embed_query(self, question: str) -> list[float]:
        ...

    def answer(self, question: str, contexts: list[Chunk]) -> AnswerPayload:
        ...


class GeminiProvider:
    def __init__(self) -> None:
        self.client = genai.Client()

    def embed_documents(self, chunks: list[Chunk]) -> dict[str, list[float]]:
        vectors: dict[str, list[float]] = {}
        for chunk in chunks:
            content = f"title: {chunk.title} | text: {chunk.text}"
            result = self.client.models.embed_content(
                model=EMBEDDING_MODEL,
                contents=content,
            )
            vectors[chunk.id] = list(result.embeddings[0].values or [])
        return vectors

    def embed_query(self, question: str) -> list[float]:
        content = f"task: question answering | query: {question}"
        result = self.client.models.embed_content(
            model=EMBEDDING_MODEL,
            contents=content,
        )
        return list(result.embeddings[0].values or [])

    def answer(self, question: str, contexts: list[Chunk]) -> AnswerPayload:
        context_text = "\n\n".join(
            f"[{chunk.id}] {chunk.title}: {chunk.text}" for chunk in contexts
        )
        prompt = f"""
Answer the question using only the supplied context. Do not use prior knowledge.
Return a concise fluent answer. For each factual claim, include an exact supporting
quote copied from the context and the matching citation ID.

Question: {question}

Context:
{context_text}
"""
        interaction = self.client.interactions.create(
            model=GENERATION_MODEL,
            input=prompt,
            response_format={
                "type": "text",
                "mime_type": "application/json",
                "schema": AnswerPayload.model_json_schema(),
            },
        )
        return AnswerPayload.model_validate_json(interaction.output_text)


class FixtureProvider:
    _vectors = {
        "employee-mobility": [1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
        "contractor-access": [0.0, 1.0, 0.0, 0.0, 0.0, 0.0],
        "individual-meals": [0.0, 0.0, 1.0, 0.0, 0.0, 0.0],
        "team-event-meals": [0.0, 0.0, 0.0, 1.0, 0.0, 0.0],
        "workspace-stipend": [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
        "damaged-equipment": [0.0, 0.0, 0.0, 0.0, 0.0, 1.0],
    }

    _query_vectors = {
        "How early must employees submit an international remote-work request?": [
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0,
        ],
        "When must employees submit a receipt for a meal expense?": [
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            0.0,
        ],
        "How long do new hires have to claim the home-office equipment allowance?": [
            0.0,
            0.0,
            0.0,
            0.0,
            1.0,
            0.0,
        ],
    }

    _answers = {
        "contractor-access": (
            "Employees should submit the international remote-work request "
            "90 days before departure.",
            "90 days before departure",
        ),
        "employee-mobility": (
            "Employees must submit the mobility form at least 30 calendar days "
            "before departure.",
            "at least 30 calendar days before departure",
        ),
        "team-event-meals": (
            "Employees must submit a meal receipt when the event costs more than "
            "80 US dollars.",
            "more than 80 US dollars",
        ),
        "individual-meals": (
            "A receipt is required when one meal costs more than 25 US dollars.",
            "more than 25 US dollars",
        ),
        "damaged-equipment": (
            "New hires have 14 calendar days to report damaged home-office equipment.",
            "14 calendar days",
        ),
        "workspace-stipend": (
            "New hires may request the workspace stipend during their first "
            "60 calendar days.",
            "first 60 calendar days",
        ),
    }

    def embed_documents(self, chunks: list[Chunk]) -> dict[str, list[float]]:
        return {chunk.id: self._vectors[chunk.id] for chunk in chunks}

    def embed_query(self, question: str) -> list[float]:
        return self._query_vectors[question]

    def answer(self, question: str, contexts: list[Chunk]) -> AnswerPayload:
        primary = contexts[0]
        answer, quote = self._answers[primary.id]
        return AnswerPayload(
            answer=answer,
            claims=[Claim(text=quote, citation_id=primary.id)],
        )

What should match in the providers?

Your file should define one protocol and two implementations. The hosted provider uses Gemini, while the fixture provider keeps vectors and cited answers fixed.

import math
import re

from app.knowledge import CHUNKS
from app.models import RetrievedChunk
from app.providers import ModelProvider


def tokenize(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower()))


def lexical_retrieve(question: str, top_k: int) -> list[RetrievedChunk]:
    query_terms = tokenize(question)
    ranked = [
        RetrievedChunk(
            id=chunk.id,
            title=chunk.title,
            text=chunk.text,
            score=float(len(query_terms & tokenize(chunk.text))),
        )
        for chunk in CHUNKS
    ]
    return sorted(ranked, key=lambda item: item.score, reverse=True)[:top_k]


def cosine_similarity(left: list[float], right: list[float]) -> float:
    if len(left) != len(right) or not left:
        return 0.0
    dot = sum(a * b for a, b in zip(left, right))
    left_norm = math.sqrt(sum(value * value for value in left))
    right_norm = math.sqrt(sum(value * value for value in right))
    if left_norm == 0.0 or right_norm == 0.0:
        return 0.0
    return dot / (left_norm * right_norm)


def semantic_retrieve(
    question: str,
    top_k: int,
    provider: ModelProvider,
) -> list[RetrievedChunk]:
    document_vectors = provider.embed_documents(CHUNKS)
    query_vector = provider.embed_query(question)
    ranked = [
        RetrievedChunk(
            id=chunk.id,
            title=chunk.title,
            text=chunk.text,
            score=cosine_similarity(query_vector, document_vectors[chunk.id]),
        )
        for chunk in CHUNKS
    ]
    return sorted(ranked, key=lambda item: item.score, reverse=True)[:top_k]


def retrieve(
    question: str,
    retriever: str,
    top_k: int,
    provider: ModelProvider,
) -> list[RetrievedChunk]:
    if retriever == "semantic":
        return semantic_retrieve(question, top_k, provider)
    return lexical_retrieve(question, top_k)

What should match in retrieval?

Your file should contain tokenization, lexical ranking, cosine similarity, semantic ranking, and the shared strategy selector.

from collections.abc import Callable
from typing import Annotated

from fastapi import Depends, FastAPI

from app.models import AskRequest, AskResponse, Chunk
from app.providers import GeminiProvider, ModelProvider
from app.retrieval import retrieve


def get_provider() -> ModelProvider:
    return GeminiProvider()


def create_app(
    provider_factory: Callable[[], ModelProvider] = get_provider,
) -> FastAPI:
    application = FastAPI(
        title="Deterministic RAG Evaluation Lab",
        version="1.0.0",
    )

    @application.get("/")
    def root() -> dict[str, str]:
        return {"message": "RAG evaluation lab is ready"}

    @application.post("/ask", response_model=AskResponse)
    def ask(
        request: AskRequest,
        provider: Annotated[ModelProvider, Depends(provider_factory)],
    ) -> AskResponse:
        ranked = retrieve(
            request.question,
            request.retriever,
            request.top_k,
            provider,
        )
        contexts = [
            Chunk(id=item.id, title=item.title, text=item.text) for item in ranked
        ]
        answer = provider.answer(request.question, contexts)
        return AskResponse(
            answer=answer.answer,
            claims=answer.claims,
            retrieved=ranked,
        )

    return application


app = create_app()

What should match in the application?

Your application should expose the readiness route and POST /ask. The request route should return the generated answer beside its claims and retrieved chunks.

The server watches your application files while it runs. Keep this terminal attached because it hosts the API until you stop it.

• Start the application with Uvicorn by running this command:

python3 -m uvicorn app.main:app --reload

What does this command do?

Uvicorn imports app from app.main and serves it locally. Reload mode restarts the process when a project file changes.

You should see startup messages showing that the application is running. The terminal remains occupied while the server accepts requests.

Server failed to start?

Confirm your terminal still shows the activated .venv environment. Check the first traceback line that points to one of your application files.

If the traceback mentions an import, confirm every Python file is inside app and that app/__init__.py exists.

Need help reading the traceback? Help me diagnose my Uvicorn startup error.

• Open http://127.0.0.1:8000/docs in your browser.
• Expand the POST /ask endpoint.
• Enable editing for the request body.
• Replace the example request with this JSON:

{
    "question": "How early must employees submit an international remote-work request?",
    "retriever": "lexical",
    "top_k": 1
}

What does this request test?

The question asks about employees, but it also repeats international remote-work and request language. A top_k value of one forces the model to answer from the single highest lexical result.

Before you submit the request, which policy do you think word overlap ranks first?

• Submit the POST /ask request from the interactive documentation.

You should see contractor-access as the first retrieved chunk. The answer says the request should be submitted 90 days before departure.

That result is the intended failure. The answer is fluent and cited from its retrieved passage, while the labeled employee policy requires 30 calendar days.

What did this failure prove?

The generator followed the supplied contractor passage faithfully. Retrieval selected the wrong source, so a grounded answer still became incorrect against the labeled reference.

Request did not return the designed result?

Confirm the request uses lexical with top_k set to 1. Check that the question matches the labeled international remote-work question exactly.

If the request fails before returning JSON, inspect the Uvicorn terminal for a provider or credential error. Keep your API key out of any message you share.

Still stuck? Help me diagnose my designed lexical request.

You now have a reproducible RAG failure with its source evidence exposed. Next, you will measure why groundedness alone cannot certify correctness.

Measure the Failure Deterministically

Your FastAPI endpoint now exposes the designed RAG failure. The lexical retriever selects the contractor policy, so the answer confidently claims that employees need 90 days.

That answer remains grounded in the retrieved distractor. You need separate metrics to prove that groundedness can stay high while reference correctness falls to zero.

In this step, get ready to:

• Implement deterministic retrieval metrics plus answer-quality metrics.
• Evaluate every labeled case through the fixed FixtureProvider data.
• Expose POST /evaluate through the running API.

Implement the answer metrics

Retrieval metrics measure where the labeled policy appears in the ranking. Answer metrics test whether each cited quote exists in its source plus whether the final answer contains the required fact.

What do the metrics measure?

Hit@1 checks whether the expected policy ranks first. Recall@2 checks whether it appears within the first two results.

Mean reciprocal rank rewards an expected policy for appearing closer to the top. Its score falls as the policy moves lower in the ranking.

Groundedness checks cited evidence against the retrieved text. Reference correctness checks the answer against the labeled fact.

• Create app/evaluation.py inside the app folder using the VS Code Explorer sidebar.
• Add the normalization helper plus the groundedness metric by pasting this code into app/evaluation.py:

import re

from app.knowledge import CHUNKS, EVAL_CASES
from app.models import (
    AnswerPayload,
    Chunk,
    EvaluationRun,
    MetricSummary,
    Scorecard,
)
from app.providers import ModelProvider
from app.retrieval import retrieve


EVALUATION_K = 2


def normalize(text: str) -> str:
    return " ".join(re.findall(r"[a-z0-9]+", text.lower()))


def groundedness(answer: AnswerPayload, contexts: list[Chunk]) -> float:
    if not answer.claims:
        return 0.0
    sources = {chunk.id: normalize(chunk.text) for chunk in contexts}
    supported = sum(
        1
        for claim in answer.claims
        if claim.citation_id in sources
        and normalize(claim.text) in sources[claim.citation_id]
    )
    return supported / len(answer.claims)

How does groundedness work?

• The normalize() helper converts text into lowercase alphanumeric words for consistent comparisons.
• The sources dictionary connects each retrieved chunk ID to its normalized text.
• The supported-claim count increases only when a cited quote appears in the chunk named by its citation.

• Save app/evaluation.py.

Uvicorn reloads the application in the terminal from earlier. You should see the reload complete without a Python traceback.

Seeing a reload failure?

Check that every imported model name matches app/models.py. Confirm that the indentation inside groundedness() matches the snippet.

Ask for help with the exact traceback if the reload still fails: Help me debug the groundedness metric in app/evaluation.py.

• Add the reference-correctness metric plus the averaging helper below groundedness() by pasting this code:

def reference_correctness(answer: AnswerPayload, required_facts: list[str]) -> float:
    if not required_facts:
        return 1.0
    normalized_answer = normalize(answer.answer)
    matched = sum(
        1 for fact in required_facts if normalize(fact) in normalized_answer
    )
    return matched / len(required_facts)


def _mean(values: list[float]) -> float:
    return sum(values) / len(values)

How does correctness work?

• The reference_correctness() function looks for every labeled fact in the generated answer.
• The result becomes the proportion of required facts found in the answer.
• The _mean() helper combines per-case values into one scorecard metric.

• Save app/evaluation.py.

Uvicorn should reload the module without a traceback. The file now contains separate rules for source support plus labeled correctness.

Does the module fail to reload?

Place reference_correctness() after the closing line of groundedness(). Keep _mean() at the left edge with the other function definitions.

Use the traceback to locate a spacing problem: Help me fix the function placement in app/evaluation.py.

Evaluate the fixed cases offline

The FixtureProvider supplies fixed answer text plus fixed embedding vectors. Every evaluation run therefore uses the same inputs without sending a request to Gemini.

• Start the evaluation runner below _mean() by adding its metric collections:

def evaluate_run(retriever_name: str, provider: ModelProvider) -> EvaluationRun:
    hits: list[float] = []
    recalls: list[float] = []
    reciprocal_ranks: list[float] = []
    grounded_scores: list[float] = []
    correctness_scores: list[float] = []

Why collect separate score lists?

Each labeled case contributes one value to every metric list. The evaluator averages those values after all three policy questions have run.

• Save app/evaluation.py.

Uvicorn should reload successfully. The new runner is valid Python while you add its evaluation loop.

Is the runner outside the module?

Keep def evaluate_run aligned with the other function definitions. Indent each metric list by four spaces.

Ask for a structure check if the terminal shows a syntax error: Help me align evaluate_run correctly.

• Add the case-processing loop inside evaluate_run() below correctness_scores by pasting this code:

    for case in EVAL_CASES:
        ranked = retrieve(
            case.question,
            retriever_name,
            len(CHUNKS),
            provider,
        )
        ranked_ids = [item.id for item in ranked]
        hits.append(float(ranked_ids[0] == case.expected_chunk_id))
        recalls.append(float(case.expected_chunk_id in ranked_ids[:EVALUATION_K]))
        rank = ranked_ids.index(case.expected_chunk_id) + 1
        reciprocal_ranks.append(1.0 / rank)

        selected = [
            Chunk(id=ranked[0].id, title=ranked[0].title, text=ranked[0].text)
        ]
        answer = provider.answer(case.question, selected)
        grounded_scores.append(groundedness(answer, selected))
        correctness_scores.append(
            reference_correctness(answer, case.required_facts)
        )

What happens for each case?

• The retriever ranks every policy chunk so the evaluator can find the expected chunk's exact position.
• The first ranked chunk becomes the answer context because the designed failure uses a single retrieved result.
• The provider answer receives one groundedness score plus one reference-correctness score.

• Save app/evaluation.py.

Uvicorn should complete another clean reload. The runner can now score all three labeled questions.

Seeing an indentation traceback?

Keep the entire loop indented inside evaluate_run(). Keep the statements under for case in EVAL_CASES indented one additional level.

Use the traceback line in this prompt: Help me repair the evaluate_run loop indentation.

• Finish evaluate_run() by adding this return block after the loop:

    return EvaluationRun(
        retriever=retriever_name,
        metrics=MetricSummary(
            hit_at_1=_mean(hits),
            recall_at_2=_mean(recalls),
            mrr=_mean(reciprocal_ranks),
            groundedness=_mean(grounded_scores),
            reference_correctness=_mean(correctness_scores),
        ),
    )

What does the runner return?

The return block averages every metric list into one MetricSummary. The surrounding EvaluationRun records which retriever produced those values.

• Save app/evaluation.py.

The terminal should show another successful reload. The evaluator now returns a complete metric summary for one retrieval method.

Does the return block fail?

Align return EvaluationRun with the for statement inside the function. This keeps the return outside the loop.

Ask for help if a bracket or indentation error remains: Help me fix the EvaluationRun return block.

• Add build_scorecard() below evaluate_run() by pasting this code:

def build_scorecard(provider: ModelProvider) -> Scorecard:
    baseline = evaluate_run("lexical", provider)
    semantic = evaluate_run("semantic", provider)
    delta = MetricSummary(
        **{
            field: getattr(semantic.metrics, field) - getattr(baseline.metrics, field)
            for field in MetricSummary.model_fields
        }
    )
    return Scorecard(baseline=baseline, semantic=semantic, delta=delta)

How is the comparison built?

The same provider evaluates the lexical baseline plus the semantic run. Using one fixed provider keeps the comparison focused on retrieval.

The delta object subtracts every baseline metric from its semantic counterpart. Positive values expose measurable improvement.

• Save app/evaluation.py.

Uvicorn should reload the complete evaluation module without a traceback. Both retrieval methods now pass through the same deterministic evaluator.

Does scorecard construction fail?

Check that both retriever names match lexical plus semantic exactly. Confirm that MetricSummary.model_fields remains inside the dictionary comprehension.

Share the traceback through this prompt if needed: Help me debug build_scorecard.

✔️ Awesome, I've got everything!

Your deterministic evaluator is complete. Make sure app/evaluation.py is saved.

ⓧ I'd like to double check the full code

import re

from app.knowledge import CHUNKS, EVAL_CASES
from app.models import (
    AnswerPayload,
    Chunk,
    EvaluationRun,
    MetricSummary,
    Scorecard,
)
from app.providers import ModelProvider
from app.retrieval import retrieve


EVALUATION_K = 2


def normalize(text: str) -> str:
    return " ".join(re.findall(r"[a-z0-9]+", text.lower()))


def groundedness(answer: AnswerPayload, contexts: list[Chunk]) -> float:
    if not answer.claims:
        return 0.0
    sources = {chunk.id: normalize(chunk.text) for chunk in contexts}
    supported = sum(
        1
        for claim in answer.claims
        if claim.citation_id in sources
        and normalize(claim.text) in sources[claim.citation_id]
    )
    return supported / len(answer.claims)


def reference_correctness(answer: AnswerPayload, required_facts: list[str]) -> float:
    if not required_facts:
        return 1.0
    normalized_answer = normalize(answer.answer)
    matched = sum(
        1 for fact in required_facts if normalize(fact) in normalized_answer
    )
    return matched / len(required_facts)


def _mean(values: list[float]) -> float:
    return sum(values) / len(values)


def evaluate_run(retriever_name: str, provider: ModelProvider) -> EvaluationRun:
    hits: list[float] = []
    recalls: list[float] = []
    reciprocal_ranks: list[float] = []
    grounded_scores: list[float] = []
    correctness_scores: list[float] = []

    for case in EVAL_CASES:
        ranked = retrieve(
            case.question,
            retriever_name,
            len(CHUNKS),
            provider,
        )
        ranked_ids = [item.id for item in ranked]
        hits.append(float(ranked_ids[0] == case.expected_chunk_id))
        recalls.append(float(case.expected_chunk_id in ranked_ids[:EVALUATION_K]))
        rank = ranked_ids.index(case.expected_chunk_id) + 1
        reciprocal_ranks.append(1.0 / rank)

        selected = [
            Chunk(id=ranked[0].id, title=ranked[0].title, text=ranked[0].text)
        ]
        answer = provider.answer(case.question, selected)
        grounded_scores.append(groundedness(answer, selected))
        correctness_scores.append(
            reference_correctness(answer, case.required_facts)
        )

    return EvaluationRun(
        retriever=retriever_name,
        metrics=MetricSummary(
            hit_at_1=_mean(hits),
            recall_at_2=_mean(recalls),
            mrr=_mean(reciprocal_ranks),
            groundedness=_mean(grounded_scores),
            reference_correctness=_mean(correctness_scores),
        ),
    )


def build_scorecard(provider: ModelProvider) -> Scorecard:
    baseline = evaluate_run("lexical", provider)
    semantic = evaluate_run("semantic", provider)
    delta = MetricSummary(
        **{
            field: getattr(semantic.metrics, field) - getattr(baseline.metrics, field)
            for field in MetricSummary.model_fields
        }
    )
    return Scorecard(baseline=baseline, semantic=semantic, delta=delta)

How to use this reference

Compare this reference with your saved file from top to bottom. Match every function name plus each indentation level.

Expose the evaluation scorecard

The API needs a response route for the evaluator. A small reporting module also preserves each result as a machine-readable scorecard plus a readable summary.

• Create app/reporting.py inside the app folder using the VS Code Explorer sidebar.
• Add the report location plus the first scorecard writer by pasting this code into app/reporting.py:

from pathlib import Path

from app.models import Scorecard


REPORT_DIR = Path("reports")


def save_scorecard(scorecard: Scorecard) -> None:
    REPORT_DIR.mkdir(parents=True, exist_ok=True)
    (REPORT_DIR / "scorecard.json").write_text(
        scorecard.model_dump_json(indent=2),
        encoding="utf-8",
    )

What does the writer prepare?

The function creates the reports folder when needed. It serializes the complete scorecard into scorecard.json.

• Save app/reporting.py.

Uvicorn should reload the new module without a traceback. The report writer can now preserve the structured result.

Does the report module fail?

Confirm that Path uses a capital letter. Keep the write_text() call indented inside save_scorecard().

Ask for help with the traceback if needed: Help me debug the scorecard writer.

• Extend save_scorecard() below the JSON writer by pasting this indented code:

    rows = []
    for name in MetricSummaryFields:
        baseline = getattr(scorecard.baseline.metrics, name)
        semantic = getattr(scorecard.semantic.metrics, name)
        delta = getattr(scorecard.delta, name)
        rows.append(f"| {name} | {baseline:.3f} | {semantic:.3f} | {delta:+.3f} |")

    markdown = "\n".join(
        [
            "# RAG Evaluation Scorecard",
            "",
            "| Metric | Lexical baseline | Semantic | Delta |",
            "| --- | ---: | ---: | ---: |",
            *rows,
            "",
            "Groundedness can remain high while reference correctness is low. "
            "This indicates that the generator faithfully used the wrong retrieved source.",
        ]
    )
    (REPORT_DIR / "scorecard.md").write_text(markdown, encoding="utf-8")

How is the readable report built?

Each metric becomes one table row with baseline, semantic, plus delta values. The closing note records the exact lesson demonstrated by the designed failure.

The finished summary is written to scorecard.md inside the same report folder.

• Save app/reporting.py.

Uvicorn should reload the expanded writer cleanly. The module can now produce both report formats.

Is the Markdown writer misplaced?

Keep rows plus markdown inside save_scorecard(). Preserve the indentation before the final write_text() call.

Use this prompt for a bracket check: Help me fix the Markdown scorecard writer.

• Add the ordered metric field names at the bottom of app/reporting.py by pasting this code:

MetricSummaryFields = (
    "hit_at_1",
    "recall_at_2",
    "mrr",
    "groundedness",
    "reference_correctness",
)

Why define the field order?

The tuple controls which metrics appear in the readable report. Its order keeps retrieval metrics before answer-quality metrics.

• Save app/reporting.py.

Uvicorn should reload without a traceback. The reporting module now contains every scorecard field.

Seeing a missing field error?

Match every tuple value to a field in MetricSummary exactly. Keep the tuple outside save_scorecard() at the left edge.

Ask for a field-name comparison if needed: Help me check MetricSummaryFields.

✔️ Awesome, I've got everything!

Your report writer is complete. Make sure app/reporting.py is saved.

ⓧ I'd like to double check the full code

from pathlib import Path

from app.models import Scorecard


REPORT_DIR = Path("reports")


def save_scorecard(scorecard: Scorecard) -> None:
    REPORT_DIR.mkdir(parents=True, exist_ok=True)
    (REPORT_DIR / "scorecard.json").write_text(
        scorecard.model_dump_json(indent=2),
        encoding="utf-8",
    )

    rows = []
    for name in MetricSummaryFields:
        baseline = getattr(scorecard.baseline.metrics, name)
        semantic = getattr(scorecard.semantic.metrics, name)
        delta = getattr(scorecard.delta, name)
        rows.append(f"| {name} | {baseline:.3f} | {semantic:.3f} | {delta:+.3f} |")

    markdown = "\n".join(
        [
            "# RAG Evaluation Scorecard",
            "",
            "| Metric | Lexical baseline | Semantic | Delta |",
            "| --- | ---: | ---: | ---: |",
            *rows,
            "",
            "Groundedness can remain high while reference correctness is low. "
            "This indicates that the generator faithfully used the wrong retrieved source.",
        ]
    )
    (REPORT_DIR / "scorecard.md").write_text(markdown, encoding="utf-8")


MetricSummaryFields = (
    "hit_at_1",
    "recall_at_2",
    "mrr",
    "groundedness",
    "reference_correctness",
)

How to use this reference

Compare your report writer with this complete file. Pay close attention to the indentation inside save_scorecard().

The running application can now call the evaluator plus save its result. The final route uses FixtureProvider directly, which keeps this endpoint offline.

• Replace the import section at the top of app/main.py with this complete import group:

from collections.abc import Callable
from typing import Annotated

from fastapi import Depends, FastAPI

from app.evaluation import build_scorecard
from app.models import AskRequest, AskResponse, Chunk, Scorecard
from app.providers import FixtureProvider, GeminiProvider, ModelProvider
from app.reporting import save_scorecard
from app.retrieval import retrieve

What do the new imports connect?

The application imports build_scorecard() for deterministic evaluation. It imports save_scorecard() for report generation.

The FixtureProvider import supplies fixed local data to the route. The Scorecard model defines the response contract.

• Save app/main.py.

Uvicorn should reload without an import error. The application now recognizes the evaluator plus the report writer.

Seeing an import failure?

Confirm that both app/evaluation.py plus app/reporting.py are saved. Check each imported function name against its definition.

Ask for help with the import traceback: Help me debug the app.main imports.

• Find the closing parenthesis of the existing return AskResponse( block in create_app().
• Add the evaluation route before return application by pasting this code:

    @application.post("/evaluate", response_model=Scorecard)
    def evaluate() -> Scorecard:
        scorecard = build_scorecard(FixtureProvider())
        save_scorecard(scorecard)
        return scorecard

What does the endpoint do?

• The route builds both scorecard runs from fixed fixture data.
• The report writer saves the same result to the local report files.
• The endpoint returns the complete scorecard as its response.

• Save app/main.py.

Uvicorn should reload the route without a traceback. The running API now exposes POST /evaluate.

Is the endpoint missing?

Keep the route indented inside create_app(). Place it before the function's final return application line.

Use this prompt if the route still does not load: Help me place the evaluate endpoint.

✔️ Awesome, I've got everything!

Your application now exposes the offline evaluation endpoint. Make sure app/main.py is saved.

ⓧ I'd like to double check the full code

from collections.abc import Callable
from typing import Annotated

from fastapi import Depends, FastAPI

from app.evaluation import build_scorecard
from app.models import AskRequest, AskResponse, Chunk, Scorecard
from app.providers import FixtureProvider, GeminiProvider, ModelProvider
from app.reporting import save_scorecard
from app.retrieval import retrieve


def get_provider() -> ModelProvider:
    return GeminiProvider()


def create_app(
    provider_factory: Callable[[], ModelProvider] = get_provider,
) -> FastAPI:
    application = FastAPI(
        title="Deterministic RAG Evaluation Lab",
        version="1.0.0",
    )

    @application.get("/")
    def root() -> dict[str, str]:
        return {"message": "RAG evaluation lab is ready"}

    @application.post("/ask", response_model=AskResponse)
    def ask(
        request: AskRequest,
        provider: Annotated[ModelProvider, Depends(provider_factory)],
    ) -> AskResponse:
        ranked = retrieve(
            request.question,
            request.retriever,
            request.top_k,
            provider,
        )
        contexts = [
            Chunk(id=item.id, title=item.title, text=item.text) for item in ranked
        ]
        answer = provider.answer(request.question, contexts)
        return AskResponse(
            answer=answer.answer,
            claims=answer.claims,
            retrieved=ranked,
        )

    @application.post("/evaluate", response_model=Scorecard)
    def evaluate() -> Scorecard:
        scorecard = build_scorecard(FixtureProvider())
        save_scorecard(scorecard)
        return scorecard

    return application


app = create_app()

How to use this reference

Compare this file with your saved app/main.py. Confirm that the evaluation route remains inside create_app().

Before you submit the evaluation, do you expect a fully grounded answer to receive a fully correct score?

• Switch back to the FastAPI documentation page from earlier.
• Submit the POST /evaluate endpoint using its request controls.

The lexical baseline should report hit_at_1 of 0.0. It should report groundedness of 1.0 plus reference_correctness of 0.0.

That split is the proof: the fixture answer quotes its retrieved contractor policy faithfully while contradicting the labeled employee policy. The response also includes Recall@2 plus mean reciprocal rank from the same offline run.

Do the baseline values look different?

Confirm that the endpoint constructs FixtureProvider() directly. Check that EVALUATION_K equals 2.

Compare the metric functions with the full-file reference before trying again. If the mismatch remains, help me trace the unexpected evaluation values.

You now have reproducible evidence that source support does not guarantee factual correctness. Next, you will improve retrieval and use the same evaluator to measure the change.

Improve Retrieval and Generate Scorecards

Your FastAPI app already proves the core RAG failure. The lexical baseline retrieves the wrong passage while groundedness still scores 1.0.

Semantic embeddings compare the meaning of each question with the policy passages. The same evaluator then proves whether retrieval quality improves.

In this step, get ready to:

• Compare lexical retrieval with semantic retrieval through the shared provider contract.
• Generate deterministic before-and-after metrics through the offline evaluator.
• Inspect the JSON and Markdown scorecards.

Compare lexical and semantic retrieval

Lexical retrieval ranks passages by shared words. Semantic retrieval uses meaning to connect the employee question with the mobility policy despite their different wording.

This live check uses your hosted model quota. The request stays small because it sends only fictional policy text.

• Visit http://127.0.0.1:8000/docs in your browser.
• Expand POST /ask in the interactive documentation.
• Set question to How early must employees submit an international remote-work request?.
• Set retriever to semantic.
• Set top_k to 1.
• Submit the request from the endpoint's request panel.

You should see employee-mobility as the first retrieved chunk. Good, the answer now uses the policy requiring 30 calendar days.

How Semantic Retrieval Stays Swappable

The ModelProvider protocol gives retrieval a shared interface for embeddings. The retrieve() function can therefore use a live provider or a fixed fixture without changing its ranking flow.

Gemini Embedding 2 receives documents as title: {title} | text: {content}. It receives questions as task: question answering | query: {content}.

Semantic Result Looks Unchanged

• Check the submitted body for retriever set to semantic.
• Restart the running app from a shell that already has GEMINI_API_KEY configured.

Help me diagnose the semantic request.

Generate the deterministic scorecard

A hosted embedding result can evolve over time. The FixtureProvider keeps the scorecard reproducible by returning fixed document vectors, fixed query vectors, and fixed cited answers.

Before the check, consider whether those fixed vectors should make every semantic metric perfect.

• Return to the interactive documentation in your browser.
• Expand POST /evaluate.
• Submit the evaluation from the endpoint's request panel.

The response should contain baseline, semantic, and delta sections. The semantic section should report 1.0 for Hit@1, Recall@2, MRR, groundedness, and reference correctness.

That evaluation now writes reports/scorecard.json and reports/scorecard.md. Your retrieval fix now has repeatable evidence without another Gemini call.

Why the Metrics Stay Separate

The baseline groundedness remains 1.0 because the fixture answer quotes its retrieved source exactly. Reference correctness remains 0.0 because that source contains the wrong policy fact.

The semantic run improves the ranking first. The corrected source then allows the deterministic answer to satisfy the labeled reference.

Scorecard Files Missing

• Check that you submitted POST /evaluate instead of POST /ask.
• Check the terminal running Uvicorn for the latest traceback.
• Check that the running process can write inside the project folder.

Help me restore scorecard generation.

Inspect the saved reports

The JSON report preserves the complete scorecard for other tools. The Markdown report turns the same values into a table that is easy to review or share.

• Switch back to VS Code from earlier.
• Expand the reports folder in the Explorer sidebar.
• Open scorecard.json.

You should see the lexical baseline, semantic run, and metric deltas stored as one scorecard object.

• Open scorecard.md from the same folder.

Before the final check, consider which metric should remain unchanged when both answers quote their retrieved passages exactly.

• Compare the five metric rows across the lexical baseline, semantic, and delta columns.

The semantic column should show 1.000 for every metric. The groundedness delta should remain +0.000 while reference correctness rises by +1.000.

You have the evidence now. The semantic retriever reaches the correct source across every labeled case while preserving a transparent baseline.

Your retrieval improvement now has reproducible evidence. Next, you will protect that evidence with offline tests before packaging the API with Docker.

Test and Containerize the Lab

Your scorecard now proves that semantic retrieval fixes the ranking failure while the lexical answer remains grounded in the wrong source. The evidence is clear on your machine.

Another developer still needs a reliable way to reproduce that evidence. In this step, pytest checks the lab without calling Gemini. Docker Desktop packages the FastAPI service into a portable demo.

In this step, get ready to:

• Test the lexical failure, semantic improvement, metric separation, and deterministic API output offline.
• Package the API with a pinned Python image and a mounted reports directory.
• Run the tests and launch the containerized interactive documentation.

Build the offline test suite

The test suite injects FixtureProvider through create_app(). Every assertion runs against fixed answers and vectors, so these checks consume no model credits.

• In the VS Code Explorer sidebar from earlier, create a folder named tests beside the app folder.

You should see the new tests folder in the Explorer sidebar.

• Inside tests, create a file named test_lab.py.

You should see test_lab.py nested under tests.

• Set up the offline client and add the lexical failure test by copying this code into tests/test_lab.py:

from fastapi.testclient import TestClient

from app.evaluation import build_scorecard
from app.knowledge import EVAL_CASES
from app.main import create_app
from app.providers import FixtureProvider


provider = FixtureProvider()
client = TestClient(create_app(lambda: provider))


def test_lexical_failure_is_visible() -> None:
    response = client.post(
        "/ask",
        json={
            "question": EVAL_CASES[0].question,
            "retriever": "lexical",
            "top_k": 1,
        },
    )
    assert response.status_code == 200
    payload = response.json()
    assert payload["retrieved"][0]["id"] == "contractor-access"
    assert "90 days" in payload["answer"]

What Does This Test Prove?

• TestClient sends a request directly to the FastAPI application without starting a network server.
• FixtureProvider replaces the hosted model path with fixed local answers.
• The assertions preserve the designed failure by requiring contractor-access and the misleading 90 days answer.

• Save tests/test_lab.py.
• Run the first offline test from the terminal by running:

python3 -m pytest -q

What Does This Check Do?

The quiet test run discovers functions beginning with test_ and reports whether their assertions pass. This first run checks the designed lexical failure.

You should see one passing test. That result confirms the wrong 90-day answer remains reproducible without a Gemini call.

First Test Not Passing?

Check that test_lab.py sits inside tests. Confirm that your terminal is still using the virtual environment from setup.

Compare the imports and assertions with the block above. A misspelled project identifier prevents test collection or changes the expected result.

Help me diagnose the first offline test.

• Add the semantic improvement test at the bottom of tests/test_lab.py by copying this code:

def test_semantic_retrieval_fixes_the_top_result() -> None:
    response = client.post(
        "/ask",
        json={
            "question": EVAL_CASES[0].question,
            "retriever": "semantic",
            "top_k": 1,
        },
    )
    assert response.status_code == 200
    payload = response.json()
    assert payload["retrieved"][0]["id"] == "employee-mobility"
    assert "30 calendar days" in payload["answer"]

What Does This Test Prove?

• The request keeps the same labeled question while switching the retriever to semantic.
• The assertions require employee-mobility to become the top result.
• The expected answer contains the correct fact from the employee policy.

• Save tests/test_lab.py.
• Check both retrieval tests from the terminal by running:

python3 -m pytest -q

What Does This Check Do?

This run compares the intentional lexical failure with the semantic correction. Both outcomes stay fixed because FixtureProvider supplies the same vectors every time.

You should see two passing tests. The pair proves that the retrieval method changes the answer while the evaluation inputs stay constant.

Semantic Test Not Passing?

Confirm that the second request uses semantic and top_k is set to 1. Check that the assertions use employee-mobility and 30 calendar days.

Help me debug the semantic retrieval test.

• Add the metric separation test at the bottom of tests/test_lab.py by copying this code:

def test_scorecard_separates_groundedness_from_correctness() -> None:
    scorecard = build_scorecard(provider)
    assert scorecard.baseline.metrics.hit_at_1 == 0.0
    assert scorecard.baseline.metrics.groundedness == 1.0
    assert scorecard.baseline.metrics.reference_correctness == 0.0
    assert scorecard.semantic.metrics.hit_at_1 == 1.0
    assert scorecard.semantic.metrics.reference_correctness == 1.0

What Does This Test Prove?

• The lexical baseline misses the expected top chunk while remaining fully grounded in its retrieved distractor.
• Reference correctness stays at zero because the generated answer omits the labeled fact.
• The semantic run restores the top result and the required fact.

• Save tests/test_lab.py.
• Check all three tests from the terminal by running:

python3 -m pytest -q

What Does This Check Do?

This run adds direct assertions for the distinction at the center of the lab. A grounded answer can still fail reference correctness when retrieval selects the wrong policy.

You should see three passing tests. Your central RAG failure claim now has an executable check.

Metric Test Not Passing?

Check each nested attribute name against the code block. Confirm that build_scorecard(provider) receives the shared FixtureProvider instance.

Help me trace the scorecard assertion failure.

• Add the deterministic endpoint test at the bottom of tests/test_lab.py by copying this code:

def test_evaluate_endpoint_is_offline_and_deterministic() -> None:
    first = client.post("/evaluate")
    second = client.post("/evaluate")
    assert first.status_code == 200
    assert first.json() == second.json()

What Does This Test Prove?

• The test submits the same offline evaluation twice.
• The first assertion confirms that the endpoint accepts the request.
• The final assertion requires both scorecards to match exactly.

• Save tests/test_lab.py.
• Check the complete offline suite from the terminal by running:

python3 -m pytest -q

What Does This Check Do?

This run executes the complete four-test suite. Every test uses the fixture path, including both calls to /evaluate.

You should see four passing tests. Excellent work. The lab can now prove its failure and improvement without a network call or model charge.

Complete Suite Not Passing?

Read the failing test name first. It identifies whether the problem belongs to retrieval, metrics, or repeated endpoint output.

Confirm that the client still uses create_app(lambda: provider). Creating the default app would select the hosted provider.

Help me diagnose the failing offline suite.

✔️ Awesome, I've got everything!

Great. Double-check that tests/test_lab.py is saved before you package the service.

ⓧ I'd like to double check the full code

from fastapi.testclient import TestClient

from app.evaluation import build_scorecard
from app.knowledge import EVAL_CASES
from app.main import create_app
from app.providers import FixtureProvider


provider = FixtureProvider()
client = TestClient(create_app(lambda: provider))


def test_lexical_failure_is_visible() -> None:
    response = client.post(
        "/ask",
        json={
            "question": EVAL_CASES[0].question,
            "retriever": "lexical",
            "top_k": 1,
        },
    )
    assert response.status_code == 200
    payload = response.json()
    assert payload["retrieved"][0]["id"] == "contractor-access"
    assert "90 days" in payload["answer"]


def test_semantic_retrieval_fixes_the_top_result() -> None:
    response = client.post(
        "/ask",
        json={
            "question": EVAL_CASES[0].question,
            "retriever": "semantic",
            "top_k": 1,
        },
    )
    assert response.status_code == 200
    payload = response.json()
    assert payload["retrieved"][0]["id"] == "employee-mobility"
    assert "30 calendar days" in payload["answer"]


def test_scorecard_separates_groundedness_from_correctness() -> None:
    scorecard = build_scorecard(provider)
    assert scorecard.baseline.metrics.hit_at_1 == 0.0
    assert scorecard.baseline.metrics.groundedness == 1.0
    assert scorecard.baseline.metrics.reference_correctness == 0.0
    assert scorecard.semantic.metrics.hit_at_1 == 1.0
    assert scorecard.semantic.metrics.reference_correctness == 1.0


def test_evaluate_endpoint_is_offline_and_deterministic() -> None:
    first = client.post("/evaluate")
    second = client.post("/evaluate")
    assert first.status_code == 200
    assert first.json() == second.json()

Package the API for Docker

A container captures the runtime and dependency installation steps needed to serve the lab. The pinned python:3.12.14-slim image gives every build the same Python base.

• In the VS Code Explorer sidebar, create a file named Dockerfile beside requirements.txt.
• Define the container build by copying this code into Dockerfile:

# syntax=docker/dockerfile:1
FROM python:3.12.14-slim

WORKDIR /app

COPY requirements.txt .
RUN python -m pip install --no-cache-dir -r requirements.txt

COPY app ./app
COPY tests ./tests

EXPOSE 8000

CMD ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

What Does This Build Define?

• The base image supplies the pinned Python runtime.
• The dependency layer installs the exact versions from requirements.txt.
• The copy instructions place the application and tests inside /app.
• The final command serves app.main:app on container port 8000.

• Save Dockerfile.

You should see Dockerfile listed beside requirements.txt in the Explorer sidebar.

Dockerfile Not Recognized?

Confirm that the filename is exactly Dockerfile with no extension. Check that it sits beside requirements.txt.

Help me check my Dockerfile location and contents.

✔️ Awesome, I've got everything!

Your container build definition is ready.

ⓧ I'd like to double check the full code

# syntax=docker/dockerfile:1
FROM python:3.12.14-slim

WORKDIR /app

COPY requirements.txt .
RUN python -m pip install --no-cache-dir -r requirements.txt

COPY app ./app
COPY tests ./tests

EXPOSE 8000

CMD ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

• In the VS Code Explorer sidebar, create a file named compose.yaml beside Dockerfile.
• Define the local service by copying this code into compose.yaml:

services:
  api:
    build: .
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      GEMINI_API_KEY: ${GEMINI_API_KEY}
    volumes:
      - ./reports:/app/reports

What Does This Service Configure?

• The api service builds from the Dockerfile beside this configuration.
• The port mapping exposes the containerized API on your local browser address.
• The environment mapping passes your existing Gemini credential into the container.
• The bind mount connects the local reports directory to /app/reports inside the container.

• Save compose.yaml.

You should see compose.yaml beside Dockerfile in the Explorer sidebar.

Compose File Showing a Warning?

Check the YAML indentation carefully. The build, ports, environment, and volumes entries all belong under api.

Help me inspect my Compose configuration.

✔️ Awesome, I've got everything!

Your Compose service now connects the image, local port, credential, and report mount.

ⓧ I'd like to double check the full code

services:
  api:
    build: .
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      GEMINI_API_KEY: ${GEMINI_API_KEY}
    volumes:
      - ./reports:/app/reports

• In the VS Code Explorer sidebar, create a file named .dockerignore beside compose.yaml.
• Keep local-only files out of the build context by copying this code into .dockerignore:

.venv
__pycache__
.pytest_cache
.git
.env
reports

What Does This File Exclude?

The ignore list keeps the local virtual environment, caches, Git history, environment file, and generated reports out of the image build context. The container installs clean dependencies and writes new reports through the mounted directory.

• Save .dockerignore.

You should see .dockerignore beside the other container files in the Explorer sidebar.

Ignore File Missing from Explorer?

Confirm that the filename starts with a full stop. Check that VS Code is showing hidden files in the project Explorer.

Help me find or correct the Docker ignore file.

✔️ Awesome, I've got everything!

Your Docker build context now excludes local artifacts.

ⓧ I'd like to double check the full code

.venv
__pycache__
.pytest_cache
.git
.env
reports

Run the reproducible demo

The final check proves both layers of reproducibility. The tests verify behavior offline, while the container verifies that the packaged service starts with the same application code.

Before you run the complete suite, do you expect any test to call the hosted Gemini provider?

• Run the complete offline suite from the terminal by running:

python3 -m pytest -q

What Does the Final Test Run Confirm?

All four tests use TestClient and FixtureProvider. The suite checks the intended failure, semantic correction, metric values, and identical repeated scorecards.

You should see four passing tests. No hosted-model response is needed for this verification.

Still Seeing a Test Failure?

Use the failing function name to return to its matching code chunk. Confirm that tests/test_lab.py matches the full-code tab before continuing.

Help me resolve the final pytest failure.

• Return to the terminal that was serving the local API earlier.
• Press Control-C to release port 8000.
• Press Cmd+Space on macOS or the Windows key on Windows to open system search.
• Type Docker Desktop.
• Press Enter to open Docker Desktop.
• Wait until Docker Desktop finishes starting.

The first build can take a few minutes while Docker downloads the image and installs the pinned dependencies. Steady build output means the process is working.

Before you start the service, do you expect the container to expose the same interactive documentation as the local API?

• Build the image and start the API service by running:

docker compose up --build

What Does This Command Do?

Docker Compose builds the image before starting the api service. The command stays attached to the running container so you can watch its logs.

Keep this terminal running during the demo. Press Control-C when you are ready to stop the service.

You should see the build complete before Uvicorn begins serving the application. The terminal remains attached to the running API.

Container Not Starting?

Confirm that Docker Desktop has finished starting. Check that the earlier local server released port 8000.

If the build fails while copying files, confirm that Dockerfile, compose.yaml, requirements.txt, app, and tests share the same top-level location.

Help me diagnose the container startup failure.

• Switch back to the browser from earlier.
• Open http://127.0.0.1:8000/docs.
• Expand POST /evaluate in the interactive documentation.

Before you submit the evaluation, where do you expect the container to write the two scorecard files?

• Submit the POST /evaluate request from the endpoint panel.
• Return to the VS Code Explorer sidebar.
• Expand the existing reports folder.

You should receive the deterministic baseline, semantic, and delta values in the browser. The local reports folder should contain scorecard.json and scorecard.md after the container handles the request.

Reports Not Updating Locally?

Confirm that the request returned a scorecard in the browser. Check that the volumes entry in compose.yaml maps ./reports to /app/reports.

Help me debug the mounted reports directory.

That completes the reproducibility proof. Another developer can now run the same offline checks, start the same container, and inspect the same scorecard artifacts.

Secret mission

Add an LLM-as-Judge Disagreement Check

Add a Gemini judge that scores the designed retrieval failure beside your deterministic metrics. You will expose disagreements without changing the reproducible scorecard that acts as your audit trail.

Clean Up Your Resources

Your running local service does not call Gemini on its own. Requests to POST /ask or POST /judge-disagreement can consume free-tier quota or billed tokens.

Nice Work!

Nice Work!

You did it. Your reproducible RAG evaluation lab now exposes a fluent 90-day answer as wrong against the 30-day policy reference.

What you learned:

• Built a provider-swappable FastAPI service with live lexical retrieval through /ask. Each response shows the answer beside its citations. It also shows the ranked policy chunks with retrieval scores.
• Reproduced the designed lexical failure that returns the grounded 90-day answer. Proved semantic retrieval restores the labeled 30-day result. Captured Hit@1 plus Recall@2 plus MRR in JSON scorecards. Generated the same evidence as a Markdown report. Kept groundedness separate from reference correctness.
• Proved four behaviors offline using pytest with TestClient plus FixtureProvider. Packaged the API with Docker Compose. Served the containerized demo through FastAPI's interactive documentation.
• Secret Mission: Compared deterministic scores with a Gemini LLM-as-judge verdict through /judge-disagreement. Preserved the deterministic scorecard as the audit trail for every disagreement.

Ready to quiz yourself?