RAG Evaluation Lab: Measure What Matters

Build a RAG pipeline and measure retrieval quality with real metrics.

Introduction

30 Second Summary

Everyone is building RAG chatbots. But when someone asks "how good is your retrieval?", most builders shrug and say "it seems to work." The gap between vibes and measurements is what separates a demo from a production system.

In this project, you will build a RAG evaluation lab that produces real numbers comparing different retrieval configurations. You'll measure which setup retrieves the right passages from 215 Paul Graham essays and prove your generated answers are grounded in context.

What You'll Build

You run a single Python script and watch a formatted results table print to your terminal, showing exactly which RAG configuration wins across retrieval accuracy and answer faithfulness.

By the end of this project, you'll have:

  • A working RAG pipeline you can query about Paul Graham's essays, seeing retrieved passages and a generated answer from Groq's free Llama 3.3 70B model.
  • A comparison table showing Hit Rate@3 and MRR across 4 configurations (2 chunking strategies x 2 embedding models), proving which setup retrieves the right passages.
  • Faithfulness scores showing what percentage of generated claims are actually supported by the retrieved context, not hallucinated.
  • Secret Mission: Add BM25 keyword search, fuse it with vector search using Reciprocal Rank Fusion, and prove hybrid beats pure vector on your eval set.

Are there any prerequisites?

You need Python installed on your machine and a Google account to sign up for Groq's free API tier. No credit card is required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

This project is not a typical chatbot tutorial. You are building a measurement system that produces real numbers comparing retrieval strategies, so you can prove which configuration works best and why.

Set Up Your Environment

You need a working Python environment with the RAG toolkit installed before you can build or measure anything. This step gets your local machine ready with the packages that power retrieval, embedding, and generation.

You will create an isolated project folder, install the packages that handle vector search and LLM calls, and grab a free API key from Groq for answer generation.

In this step, get ready to:
  • Create a project folder with a Python virtual environment.
  • Install the RAG toolkit packages (ChromaDB, sentence-transformers, Groq SDK, and more).
  • Sign up for Groq and set your API key as an environment variable.
Create your project folder and virtual environment

A virtual environment keeps this project's packages isolated from the rest of your system. You will create a dedicated folder on your Desktop and set up a fresh environment inside it.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Move to your Desktop and create the project folder by running these commands:

macOS

cd ~/Desktop
mkdir rag-eval-lab
cd rag-eval-lab
python3 -m venv venv

Windows

cd %USERPROFILE%\Desktop
mkdir rag-eval-lab
cd rag-eval-lab
python -m venv venv
  • Confirm the folder was created by running:
ls

You should see a venv folder listed in the output.

What just happened?

You created a new folder called rag-eval-lab on your Desktop and set up a Python virtual environment inside it. The venv folder contains a self-contained copy of Python and pip that keeps your project's dependencies separate from everything else on your machine.

  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

Your terminal prompt now shows (venv) at the beginning of the line. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

On Windows, you may need to allow script execution first. Run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned in PowerShell, then try the activate command again.

On macOS, make sure you used source before the path. Running the path alone will not activate the environment.

Help me activate my Python virtual environment.

Install the dependencies

With your virtual environment active, you can install the six packages that power the RAG pipeline. This includes the vector store, embedding models, LLM client, dataset loader, and keyword search library.

  • Install all project dependencies by running:
pip install groq chromadb sentence-transformers datasets rank-bm25 numpy

What are these packages?

  • groq connects to Groq's free LLM API for answer generation and evaluation.
  • chromadb is a local vector database that stores and searches your document embeddings.
  • sentence-transformers runs embedding models locally on your CPU to convert text into vectors.
  • datasets loads the Paul Graham essays corpus from Hugging Face.
  • rank-bm25 provides BM25 keyword search (used in the Secret Mission).
  • numpy handles array math for metrics computation.

The install will download several hundred megabytes (the embedding models are included in sentence-transformers). Wait until you see a success message with no errors before continuing.

Seeing installation errors?

If you see a permissions error, make sure your (venv) prefix is visible in your prompt. Installing without an active virtual environment can cause permission failures.

If the install hangs or times out, check your internet connection and try running the command again.

Help me fix my pip install errors.

Sign up for Groq and set your API key

Groq provides free access to large language models through a simple API. You need an API key to send prompts and receive generated answers from their servers.

  • Open https://console.groq.com in your browser.
  • Sign up using your Google account (or email).
  • Once logged in, navigate to the API Keys section in the left sidebar.
  • Click Create API Key and give it any name (e.g. rag-eval-lab).
  • Copy the key immediately. It is only shown once.

Why Groq?

Groq offers a generous free tier with no credit card required. You get thousands of tokens per minute, which is more than enough for this project's evaluation runs.

  • Back in your terminal, set the API key as an environment variable by running:

macOS

export GROQ_API_KEY=your-key-here

Windows

set GROQ_API_KEY=your-key-here
  • Replace your-key-here with the key you just copied from the Groq console.

Why an environment variable?

Storing your key in an environment variable keeps it out of your code. Your Python script reads it at runtime with os.environ.get("GROQ_API_KEY"), so the key never appears in a file you might accidentally share.

Having trouble creating an API key?

Some users report that the key creation button is disabled after signup. If this happens, try signing out and back in, or use a different browser. If the issue persists, email support@groq.com for help.

Help me get my Groq API key working.

Before you run the verification below, what do you think will happen if one of the packages failed to install or the API key is not set correctly?

  • Verify everything is working by running:
python -c "from groq import Groq; import chromadb; from sentence_transformers import SentenceTransformer; print('All imports OK')"

You should see All imports OK printed with no errors.

Seeing an import error?

If you see ModuleNotFoundError, the package did not install correctly. Make sure (venv) appears in your prompt, then re-run the pip install command from earlier.

If you see a different error related to the Groq import, confirm your Python version is 3.7 or higher by running python --version.

Help me fix my import errors.

Your environment is ready and every package imports cleanly. Next up, you will build the naive RAG pipeline and ask it your first question about Paul Graham's essays.

Build the Naive RAG Pipeline

You have a fully equipped environment. Now you need something to measure.

Before any evaluation metrics make sense, you need a working end-to-end RAG pipeline that loads real documents, retrieves relevant passages, and generates answers. This step builds that pipeline from scratch using Paul Graham's 215 essays as the corpus.

In this step, get ready to:
  • Load and chunk 215 Paul Graham essays into a local vector store.
  • Build retrieval and answer generation functions powered by Groq.
  • Run an end-to-end test query and see retrieved passages plus a generated answer.
Load and chunk the essay corpus

A RAG pipeline starts with a corpus of documents split into small, searchable pieces. You will load the essays from Hugging Face and slice them into fixed-size text chunks.

  • Create a new file called rag_eval_lab.py in your rag-eval-lab folder.
  • Add the imports, configuration constants, and the essay-loading function by pasting this code:
import os
import numpy as np
from datasets import load_dataset
from sentence_transformers import SentenceTransformer
import chromadb
from groq import Groq
from rank_bm25 import BM25Okapi

# --- Configuration ---
GROQ_MODEL = "llama-3.3-70b-versatile"
EMBEDDING_MODELS = {
    "MiniLM": "all-MiniLM-L6-v2",
    "MPNet": "all-mpnet-base-v2",
}
CHUNK_CONFIGS = {
    "small_200": {"size": 200, "overlap": 0},
    "large_1000": {"size": 1000, "overlap": 200},
}

# --- Load Corpus ---
def load_essays():
    ds = load_dataset("sgoel9/paul_graham_essays", split="train")
    essays = []
    for row in ds:
        essays.append({"title": row["title"], "text": row["text"]})
    return essays

What does this code do?

The configuration section defines your LLM model, two embedding models, and two chunking strategies. You will use these later to compare performance across configurations.

The load_essays() function pulls 215 Paul Graham essays from the Hugging Face dataset sgoel9/paul_graham_essays and returns them as a list of dicts with title and text fields.

  • Now add the chunking functions below the load_essays function by pasting this code:
# --- Chunking ---
def chunk_text(text, title, chunk_size=200, overlap=0):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append({"text": chunk, "source": title})
        start = end - overlap if overlap > 0 else end
    return chunks

def chunk_corpus(essays, chunk_size=200, overlap=0):
    all_chunks = []
    for essay in essays:
        all_chunks.extend(chunk_text(essay["text"], essay["title"], chunk_size, overlap))
    return all_chunks

What does this code do?

The chunk_text() function slices a single essay into fixed-size pieces of 200 characters with no overlap. Each chunk records the source essay title as metadata so you can trace results back later.

The chunk_corpus() function runs chunk_text() across every essay and returns one flat list of all chunks.

  • Save rag_eval_lab.py.
  • Verify the data loads correctly by running this command:
python -c "from rag_eval_lab import load_essays, chunk_corpus; essays = load_essays(); chunks = chunk_corpus(essays); print(f'Loaded {len(essays)} essays, created {len(chunks)} chunks')"

You should see Loaded 215 essays followed by a chunk count in the thousands.

Seeing an import error?

  • Make sure your virtual environment is still active. You should see (venv) at the start of your terminal prompt.
  • Check that rag_eval_lab.py is saved in the rag-eval-lab folder and your terminal is in that same folder.
  • help me debug this import error in rag_eval_lab.py
Embed and store the chunks in ChromaDB

With the text chunks ready, you need to convert them into numeric vectors and store them in a vector database for similarity search.

  • Add the embedding and indexing function below the chunking code in rag_eval_lab.py:
# --- Embedding and Indexing ---
def build_collection(chunks, model_name, collection_name, chroma_client):
    model = SentenceTransformer(model_name)
    texts = [c["text"] for c in chunks]
    embeddings = model.encode(texts, show_progress_bar=True, batch_size=64)

    try:
        chroma_client.delete_collection(name=collection_name)
    except Exception:
        pass

    collection = chroma_client.create_collection(
        name=collection_name,
        embedding_function=None,
        metadata={"hnsw:space": "cosine"}
    )

    batch_size = 5000
    for i in range(0, len(texts), batch_size):
        batch_end = min(i + batch_size, len(texts))
        collection.add(
            ids=[f"chunk_{j}" for j in range(i, batch_end)],
            embeddings=embeddings[i:batch_end].tolist(),
            documents=texts[i:batch_end],
            metadatas=[{"source": chunks[j]["source"]} for j in range(i, batch_end)],
        )

    return collection, model

What does this code do?

This function loads a SentenceTransformer embedding model, encodes every chunk into a vector, and stores the vectors in a ChromaDB collection. It uses embedding_function=None because you are providing pre-computed embeddings rather than letting ChromaDB embed for you.

The function returns both the collection (for querying) and the embedding model (so queries use the same model).

  • Save rag_eval_lab.py.
  • Verify the module still loads without syntax errors by running:
python -c "import rag_eval_lab; print('Module loads OK')"

You should see Module loads OK printed with no tracebacks.

Build the retrieval and generation functions

The final two pieces of the pipeline: a function to query the vector store and a function to send retrieved context to Groq for answer generation.

  • Add the retrieval function below build_collection in rag_eval_lab.py:
# --- Retrieval ---
def retrieve(query, collection, model, n_results=3):
    query_embedding = model.encode([query])
    results = collection.query(
        query_embeddings=query_embedding.tolist(),
        n_results=n_results,
        include=["documents", "metadatas", "distances"],
    )
    return results

What does this code do?

The retrieve() function embeds the user's question with the same model used for indexing, then asks ChromaDB for the top-3 most similar chunks. It returns the matching documents, their metadata (source essay), and distance scores.

  • Add the generation function below retrieve:
# --- Generation ---
def generate_answer(query, contexts, groq_client):
    context_str = "\n\n---\n\n".join(contexts)
    prompt = f"""Answer the question based ONLY on the following context. If the context doesn't contain the answer, say "I don't have enough information."

Context:
{context_str}

Question: {query}

Answer:"""

    chat_completion = groq_client.chat.completions.create(
        messages=[
            {"role": "system", "content": "You are a helpful assistant that answers questions based only on the provided context."},
            {"role": "user", "content": prompt},
        ],
        model=GROQ_MODEL,
        temperature=0,
        max_tokens=300,
    )
    return chat_completion.choices[0].message.content

What does this code do?

This function joins the retrieved chunks into a single context string, builds a prompt that instructs the LLM to answer ONLY from the provided context, and sends it to Groq's llama-3.3-70b-versatile model.

Setting temperature=0 ensures deterministic answers. The system message reinforces that the model should not hallucinate beyond what the context provides.

  • Save rag_eval_lab.py.

✔️ Awesome, I've got everything!

Great. Make sure rag_eval_lab.py is saved. You are ready to test the full pipeline.

ⓧ I'd like to double check the full code

Scroll through rag_eval_lab.py and confirm these sections appear in order:

  • Imports (os, numpy, datasets, sentence_transformers, chromadb, groq, rank_bm25).
  • Configuration constants: GROQ_MODEL, EMBEDDING_MODELS, CHUNK_CONFIGS.
  • The load_essays() function.
  • The chunk_text() and chunk_corpus() functions.
  • The build_collection() function.
  • The retrieve() function.
  • The generate_answer() function.

ⓧ Something looks off

If your file has syntax errors, check that the f-string inside generate_answer uses triple quotes (f""") and that all braces in the prompt match.

help me find the issue in my rag_eval_lab.py file

Run your first query

Every piece of the pipeline is in place. Time to wire them together and see if the system can actually answer a question about Paul Graham's essays.

Before you run this, think about it: with 200-character chunks and no overlap, do you expect the retrieved passages to contain enough context for a good answer?

  • Run the full pipeline end-to-end by running this command:
python -c "from rag_eval_lab import load_essays, chunk_corpus, build_collection, retrieve, generate_answer; import chromadb, os; from groq import Groq; essays = load_essays(); chunks = chunk_corpus(essays); print(f'Loaded {len(essays)} essays, {len(chunks)} chunks'); client = chromadb.Client(); collection, model = build_collection(chunks, 'all-MiniLM-L6-v2', 'baseline', client); print(f'Indexed {collection.count()} chunks'); groq_client = Groq(api_key=os.environ.get('GROQ_API_KEY')); query = 'What did Paul Graham work on before Y Combinator?'; results = retrieve(query, collection, model); print('\nRetrieved chunks:'); [print(f'  {i+1}. {doc[:100]}...') for i, doc in enumerate(results['documents'][0])]; answer = generate_answer(query, results['documents'][0], groq_client); print(f'\nAnswer: {answer}')"

What should I see?

The command takes 1-2 minutes because it downloads the dataset and embeds all chunks. You will see a progress bar from sentence-transformers as it encodes.

Once complete, you will see 3 retrieved text passages (each truncated to 100 characters), followed by a generated answer about Paul Graham's work before Y Combinator.

Seeing an API key error?

  • Confirm your GROQ_API_KEY environment variable is still set. Run echo %GROQ_API_KEY% (Windows) or echo $GROQ_API_KEY (macOS) to check.
  • If the variable is empty, set it again with your key from the Groq console.
  • help me fix my Groq API key configuration

You should see 3 retrieved passages printed, followed by a generated answer referencing Paul Graham's work on Lisp, painting, or Viaweb.

Why are the retrieved chunks so short?

Each chunk is only 200 characters. That is barely two sentences of context. The generated answer might still be reasonable because Groq's LLM is powerful, but you have no way to know if the retrieval found the RIGHT passages.

This is the core problem: the pipeline "works" but you cannot tell if it is working WELL. Step 3 builds the measurement instrument that reveals exactly how good (or bad) this naive setup really is.

Your pipeline answers questions about Paul Graham's essays. Next up, you will build an evaluation set that reveals exactly how much retrieval quality this naive chunking strategy is leaving on the table.

Build Your Eval Set and Measure the Baseline

Your RAG pipeline answers questions. But you have no idea whether it retrieves the right passages.

In this step, you will build a measurement instrument: 10 question-answer pairs where you already know which essay contains the answer. When you run those questions through the pipeline and score the results, the numbers will tell you exactly how well your naive 200-character chunking strategy performs.

In this step, get ready to:
  • Hand-craft an evaluation set of 10 question-answer pairs with known source essays.
  • Implement retrieval metrics (Hit Rate@3 and MRR).
  • Run the baseline evaluation and see your first real numbers.
Create the evaluation set

Every evaluation needs ground truth. You will write 10 questions where you already know the correct essay source. This lets you check whether the retriever actually pulled a chunk from the right essay.

  • In rag_eval_lab.py, scroll to the bottom of the file (after the generate_answer function).
  • Add the evaluation set by pasting this code:
# --- Evaluation Set ---
EVAL_SET = [
    {"question": "What programming language did Paul Graham use to write the first version of Viaweb?", "ground_truth_answer": "Lisp", "source_essay": "Beating the Averages"},
    {"question": "What did Paul Graham study in college before switching to AI?", "ground_truth_answer": "Philosophy", "source_essay": "What I Worked On"},
    {"question": "What is Paul Graham's advice on how to get startup ideas?", "ground_truth_answer": "Notice problems in your own life rather than trying to think of startup ideas", "source_essay": "How to Get Startup Ideas"},
    {"question": "What year was Y Combinator founded?", "ground_truth_answer": "2005", "source_essay": "What I Worked On"},
    {"question": "What does Paul Graham say about the relationship between wealth and money?", "ground_truth_answer": "Wealth is not money. Wealth is what you want, not the money to buy it.", "source_essay": "How to Make Wealth"},
    {"question": "What painting technique did Paul Graham learn in Florence?", "ground_truth_answer": "Painting from life, focusing on careful observation", "source_essay": "What I Worked On"},
    {"question": "According to Paul Graham, what makes a good essay?", "ground_truth_answer": "Surprise the reader with something true they did not already know", "source_essay": "The Age of the Essay"},
    {"question": "What does Paul Graham say about the number of founders a startup should have?", "ground_truth_answer": "Two or three founders is ideal", "source_essay": "The 18 Mistakes That Kill Startups"},
    {"question": "What does Paul Graham mean by schlep blindness?", "ground_truth_answer": "The tendency to unconsciously avoid hard or tedious problems even if they are valuable to solve", "source_essay": "Schlep Blindness"},
    {"question": "What did Paul Graham build with Robert Morris at Y Combinator's first batch?", "ground_truth_answer": "They funded and advised the first batch of startups in summer 2005", "source_essay": "What I Worked On"},
]

What makes this eval set useful?

Each dict has three fields: question (what you ask the retriever), ground_truth_answer (the correct answer for reference), and source_essay (which essay the answer lives in).

The source_essay field is the key. It lets you verify whether the retriever pulled chunks from the correct essay, not just any chunk that sounds vaguely related.

  • Save rag_eval_lab.py.

Why hand-craft instead of auto-generate?

Synthetic eval sets (generated by an LLM) tend to ask questions the retriever easily answers, because the LLM writes questions based on what it sees in the text. Hand-crafted questions let you target specific difficulty levels: factual recall, synthesis across paragraphs, and specific quotes.

Implement retrieval metrics

Two metrics will score your retriever. Hit Rate@3 asks: did at least one of the top-3 retrieved chunks come from the correct essay? MRR (Mean Reciprocal Rank) goes further: if the correct chunk was ranked 1st it scores 1.0, if 2nd it scores 0.5, if 3rd it scores 0.33.

  • Below EVAL_SET, add the metrics function:
# --- Metrics ---
def compute_retrieval_metrics(eval_set, collection, model, n_results=3):
    hits = 0
    reciprocal_ranks = []

    for item in eval_set:
        results = retrieve(item["question"], collection, model, n_results)
        sources = [m["source"] for m in results["metadatas"][0]]

        if item["source_essay"] in sources:
            hits += 1
            rank = sources.index(item["source_essay"]) + 1
            reciprocal_ranks.append(1.0 / rank)
        else:
            reciprocal_ranks.append(0.0)

    hit_rate = hits / len(eval_set)
    mrr = np.mean(reciprocal_ranks)
    return {"hit_rate_at_3": hit_rate, "mrr": mrr}

What does this code do?

  • For each question, it retrieves the top-3 chunks and checks the source metadata against source_essay.
  • If any retrieved chunk came from the correct essay, it counts as a hit. The rank position of that first correct chunk determines the reciprocal rank (1/rank).
  • If none of the top-3 came from the right essay, the reciprocal rank is 0.0.
  • The function returns averages across all 10 questions.
  • Save rag_eval_lab.py.

Seeing a NameError for retrieve?

Make sure compute_retrieval_metrics is defined after all the earlier functions in the file. It calls retrieve() which must exist above it.

Help me debug this

Run the baseline evaluation

You have the eval set and the scoring function. Time to see what your naive 200-character chunking actually scores.

Before you run this, what do you think the Hit Rate@3 will be? A score of 1.0 means every question found the right essay in the top 3. A score of 0.0 means it never found the right essay. Where do you think 200-character chunks will land?

  • Run the baseline evaluation by executing this command in your terminal:
python -c "
from rag_eval_lab import *
import chromadb

print('Loading essays...')
essays = load_essays()
chunks = chunk_corpus(essays, 200, 0)
print(f'Created {len(chunks)} chunks')

chroma_client = chromadb.Client()
print('Building collection (this may take a minute)...')
collection, emb_model = build_collection(chunks, 'all-MiniLM-L6-v2', 'baseline', chroma_client)

print('Running evaluation...')
metrics = compute_retrieval_metrics(EVAL_SET, collection, emb_model)
print(f'\nBaseline Results (small_200 + MiniLM):')
print(f'  Hit Rate@3: {metrics[\"hit_rate_at_3\"]:.2f}')
print(f'  MRR:        {metrics[\"mrr\"]:.3f}')
"

What should I see?

After the embedding progress bar completes, you will see baseline results printed. Expect Hit Rate@3 somewhere around 0.40 to 0.60 and MRR around 0.3 to 0.5.

Those numbers mean the retriever only finds the correct essay about half the time. For a production system, that is poor. The 200-character chunks are too small to carry enough semantic meaning for the embedding model to match them accurately.

Why are these scores mediocre?

A 200-character chunk is about one sentence. Sentence-level embeddings often capture only a fragment of the idea, missing the broader context that connects a question to the right passage. The next steps will fix this by testing larger chunks and better embedding models.

Script hangs or runs very slowly?

The all-MiniLM-L6-v2 model downloads on first use (about 90 MB). If your connection is slow, the initial run can take several minutes. Subsequent runs use the cached model.

If you see a CUDA error, the script runs on CPU by default. Check that sentence-transformers installed correctly in your virtual environment.

Help me debug my evaluation script

✔️ Awesome, I've got everything!

Great. Make sure you have saved rag_eval_lab.py with both the EVAL_SET list and the compute_retrieval_metrics function.

ⓧ I'd like to double check the full code

Your rag_eval_lab.py should contain the following sections in order. Scroll through your file and confirm each is present:

  • Imports (os, numpy, datasets, sentence_transformers, chromadb, groq, rank_bm25).
  • Configuration section with GROQ_MODEL, EMBEDDING_MODELS, and CHUNK_CONFIGS.
  • load_essays() function.
  • chunk_text() and chunk_corpus() functions.
  • build_collection() function.
  • retrieve() function.
  • generate_answer() function.
  • EVAL_SET list with 10 dictionaries (added in this step).
  • compute_retrieval_metrics() function (added in this step).

You now have real numbers proving the naive chunking strategy leaves performance on the table. Next, you will test whether larger chunks with overlap can close that gap.

Compare Chunking Strategies

Your baseline numbers from the previous step gave you a real measurement. But a Hit Rate@3 in the 0.4-0.6 range means the pipeline fails to retrieve the right essay nearly half the time.

Chunk size is one of the highest-leverage settings in RAG. Your 200-character chunks slice sentences in half, destroying the context an embedding model needs. By switching to 1000-character chunks with 200-character overlap, you give each chunk a full paragraph of meaning. The numbers will prove whether that matters.

In this step, get ready to:
  • Build a faithfulness evaluation function that scores whether generated answers are grounded in context.
  • Run both chunking strategies through your 10-question eval set.
  • Print a comparison table with Hit Rate@3, MRR, and faithfulness scores side by side.
Add the faithfulness evaluation function

Retrieval metrics tell you whether the right passages are found. But they say nothing about whether the LLM's answer is actually grounded in those passages or hallucinated from its training data.

A faithfulness check sends the retrieved context, the question, and the generated answer back to the LLM acting as a judge. It scores from 0.0 (completely unfaithful) to 1.0 (perfectly grounded).

  • In rag_eval_lab.py, scroll to the bottom of the file (after the compute_retrieval_metrics() function) and add this new function:
# --- Faithfulness Evaluation ---
def evaluate_faithfulness(query, answer, contexts, groq_client):
    context_str = "\n\n".join(contexts)
    prompt = f"""You are an evaluation judge. Given a question, an answer, and the context used to generate the answer, score the faithfulness of the answer.

Faithfulness means: every claim in the answer is supported by the context. Score from 0.0 (completely unfaithful) to 1.0 (perfectly faithful).

Context: {context_str}

Question: {query}

Answer: {answer}

Respond with ONLY a number between 0.0 and 1.0, nothing else."""

    chat_completion = groq_client.chat.completions.create(
        messages=[{"role": "user", "content": prompt}],
        model=GROQ_MODEL,
        temperature=0,
        max_tokens=10,
    )
    try:
        score = float(chat_completion.choices[0].message.content.strip())
        return min(max(score, 0.0), 1.0)
    except ValueError:
        return 0.0

What does this code do?

This function uses the LLM as a judge. It sends the retrieved context, the question, and the generated answer to Groq with a scoring prompt.

  • context_str joins the retrieved passages into a single string for the judge to review.
  • The prompt asks the LLM to return ONLY a number between 0.0 and 1.0. This makes parsing reliable.
  • The try/except block handles cases where the LLM returns something other than a clean number, defaulting to 0.0.
  • Save rag_eval_lab.py.

Unsure where to add the function?

Place it directly below the closing line of compute_retrieval_metrics(). Make sure it is NOT indented inside another function. It should be a top-level function at the same indentation level as all your other def statements.

Help me place my evaluate_faithfulness function in the right spot

Run the chunking comparison

Now you have all the measurement tools ready. Time to run both chunking strategies through the eval set and see which one wins.

  • At the very bottom of rag_eval_lab.py, add this runner block that iterates over both chunking strategies:
# --- Run Chunking Comparison ---
if __name__ == "__main__":
    print("Loading essays...")
    essays = load_essays()
    print(f"Loaded {len(essays)} essays.")

    groq_client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
    chroma_client = chromadb.Client()

    results_table = []
    model_path = EMBEDDING_MODELS["MiniLM"]

    for chunk_name, chunk_config in CHUNK_CONFIGS.items():
        chunks = chunk_corpus(essays, chunk_config["size"], chunk_config["overlap"])
        print(f"\nChunking: {chunk_name} -> {len(chunks)} chunks")

        collection_name = f"{chunk_name}_miniLM"
        collection, emb_model = build_collection(
            chunks, model_path, collection_name, chroma_client
        )

What does this code do?

This block loads all 215 essays, then loops over your CHUNK_CONFIGS dictionary. For each config (200-char no overlap and 1000-char with 200-char overlap), it chunks the entire corpus and builds a separate ChromaDB collection embedded with all-MiniLM-L6-v2.

  • Directly below the code you just added (still inside the for loop), add the metrics computation and faithfulness check:
        # Compute retrieval metrics
        metrics = compute_retrieval_metrics(EVAL_SET, collection, emb_model, n_results=3)

        # Faithfulness on 3 sample questions
        faithfulness_scores = []
        for item in EVAL_SET[:3]:
            results = retrieve(item["question"], collection, emb_model, n_results=3)
            contexts = results["documents"][0]
            answer = generate_answer(item["question"], contexts, groq_client)
            score = evaluate_faithfulness(
                item["question"], answer, contexts, groq_client
            )
            faithfulness_scores.append(score)

        avg_faithfulness = np.mean(faithfulness_scores)
        results_table.append({
            "config": f"{chunk_name} + MiniLM",
            "hit_rate_at_3": metrics["hit_rate_at_3"],
            "mrr": metrics["mrr"],
            "faithfulness": avg_faithfulness,
        })

What does this code do?

  • compute_retrieval_metrics() runs all 10 eval questions against the collection and returns Hit Rate@3 and MRR.
  • The faithfulness loop takes the first 3 eval questions, retrieves context, generates an answer, then asks the LLM judge to score how grounded that answer is.
  • Each configuration's results get appended to results_table for printing at the end.
  • After the for loop (back at the if __name__ indentation level), add the results printer:
    # Print Results
    print("\n" + "=" * 70)
    print("CHUNKING STRATEGY COMPARISON")
    print("=" * 70)
    print(f"{'Configuration':<30} {'Hit Rate@3':<12} {'MRR':<8} {'Faithfulness':<12}")
    print("-" * 70)
    for row in results_table:
        print(f"{row['config']:<30} {row['hit_rate_at_3']:<12.2f} {row['mrr']:<8.3f} {row['faithfulness']:<12.2f}")
    print("=" * 70)

What does this code do?

This formats the comparison as a clean table. Each row shows one chunking configuration with its three metrics, so you can see the difference at a glance.

  • Save rag_eval_lab.py.

Getting an indentation error?

The metrics and faithfulness code must be indented 8 spaces (inside the for loop). The print block must be indented 4 spaces (inside if __name__ but outside the for loop). Check that nothing is at zero indentation except function definitions.

Help me fix my indentation in the runner block

See which chunking strategy wins

Before you run this, think about what you expect. The 1000-character chunks give the embedding model full paragraphs of context. Do you think that will help or hurt retrieval accuracy?

  • Run the comparison by executing:
python rag_eval_lab.py

What should I expect?

This takes a few minutes. The script downloads the embedding model (if not cached), embeds all chunks for both strategies, then runs the eval queries and faithfulness checks against Groq. You will see progress bars for the embedding steps.

You should see a formatted table showing two rows. The large_1000 + MiniLM row should show noticeably higher Hit Rate@3 and MRR values than small_200 + MiniLM. The larger chunks with overlap give the embedding model enough context to capture meaning, and give the retriever full paragraphs to match against.

Why does chunk size matter so much?

A 200-character chunk often slices mid-sentence. The embedding model can only encode what it sees, so a truncated thought gets a meaningless vector. A 1000-character chunk preserves full paragraphs.

The 200-character overlap between large chunks means important sentences that fall on a boundary still appear in the next chunk. This prevents "seam blindness" where a key passage sits right at the split point.

Seeing a rate limit error from Groq?

The faithfulness checks make multiple Groq API calls in sequence. If you hit the rate limit, wait 60 seconds and run the script again. Groq's free tier allows 30 requests per minute for llama-3.3-70b-versatile.

Help me debug my Groq rate limit error

Clean up the runner for the next step

In the next step, you will expand this comparison to include a second embedding model. To keep the file clean, remove the if __name__ == "__main__": block you just ran.

  • In rag_eval_lab.py, delete everything from the # --- Run Chunking Comparison --- comment down to the end of the file.
  • Save rag_eval_lab.py. Your file should now end with the evaluate_faithfulness() function.

✔️ Awesome, I've got everything!

Confirm you have saved your file and that it ends with the evaluate_faithfulness() function (no runner code at the bottom).

ⓧ I'd like to double check the full code

Your file is longer than 30 lines, so confirm each section is present in order by scrolling through rag_eval_lab.py:

  • Imports (os, numpy, datasets, sentence_transformers, chromadb, groq, rank_bm25).
  • Configuration constants: GROQ_MODEL, EMBEDDING_MODELS, CHUNK_CONFIGS.
  • Functions: load_essays(), chunk_text(), chunk_corpus().
  • Functions: build_collection(), retrieve(), generate_answer().
  • EVAL_SET list with 10 question-answer pairs.
  • Functions: compute_retrieval_metrics(), evaluate_faithfulness().

The file should end after evaluate_faithfulness() with no runner code below it.

You now have hard numbers proving larger chunks with overlap beat the naive strategy. In the next step, you will test whether a better embedding model can push those numbers even higher.

Compare Embedding Models and Print Final Results

You now have data showing that larger chunks with overlap beat naive small chunks. But chunking is only half the retrieval equation.

The embedding model determines how well your system captures semantic similarity between a question and a passage. By swapping all-MiniLM-L6-v2 (384 dimensions, fast) for all-mpnet-base-v2 (768 dimensions, higher quality), you can test whether a bigger model actually helps on your corpus. The eval set gives you a definitive answer.

In this step, get ready to:
  • Write the main evaluation loop that tests all 4 configurations (2 chunking strategies x 2 embedding models).
  • Add faithfulness scoring to every configuration.
  • Print a formatted results dashboard identifying the best configuration.
Write the evaluation loop over all configurations

Instead of running each configuration manually, you need a function that iterates over every combination of chunking strategy and embedding model automatically.

  • In rag_eval_lab.py, scroll to the bottom of the file (below the evaluate_faithfulness function) and add the following code:
# --- Main Evaluation Loop ---
def run_full_evaluation():
    print("Loading essays...")
    essays = load_essays()
    print(f"Loaded {len(essays)} essays.")
    
    groq_client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
    chroma_client = chromadb.Client()
    
    results_table = []
    
    for chunk_name, chunk_config in CHUNK_CONFIGS.items():
        chunks = chunk_corpus(essays, chunk_config["size"], chunk_config["overlap"])
        print(f"\nChunking: {chunk_name} -> {len(chunks)} chunks")
        
        for model_name, model_path in EMBEDDING_MODELS.items():
            collection_name = f"{chunk_name}_{model_name}".replace(" ", "_").lower()
            print(f"  Embedding: {model_name} ({model_path})")
            
            collection, emb_model = build_collection(
                chunks, model_path, collection_name, chroma_client
            )
            
            metrics = compute_retrieval_metrics(EVAL_SET, collection, emb_model, n_results=3)

What does this code do?

The outer loop iterates over CHUNK_CONFIGS (small_200 and large_1000). The inner loop iterates over EMBEDDING_MODELS (MiniLM and MPNet). That gives you 2 x 2 = 4 combinations.

For each combination, it chunks the corpus, embeds it into a new ChromaDB collection, and runs compute_retrieval_metrics() against your 10-question eval set.

Add faithfulness scoring to each configuration

Retrieval metrics tell you if the right passages are found. Faithfulness tells you if the generated answer actually uses those passages instead of hallucinating.

  • Directly below the metrics = compute_retrieval_metrics(...) line you just added (still inside the inner loop), add this code:
            # Faithfulness on 3 sample questions
            faithfulness_scores = []
            for item in EVAL_SET[:3]:
                results = retrieve(item["question"], collection, emb_model, n_results=3)
                contexts = results["documents"][0]
                answer = generate_answer(item["question"], contexts, groq_client)
                score = evaluate_faithfulness(item["question"], answer, contexts, groq_client)
                faithfulness_scores.append(score)
            
            avg_faithfulness = np.mean(faithfulness_scores)
            
            results_table.append({
                "config": f"{chunk_name} + {model_name}",
                "hit_rate_at_3": metrics["hit_rate_at_3"],
                "mrr": metrics["mrr"],
                "faithfulness": avg_faithfulness,
            })

What does this code do?

For each of the 4 configurations, it picks the first 3 eval questions and runs a full generate-then-judge cycle. This keeps API calls manageable on Groq's free tier (12 generation calls + 12 faithfulness calls total).

Each result gets appended to results_table with the config name, Hit Rate@3, MRR, and average faithfulness score.

Print the results dashboard and run the full evaluation

The final piece formats all 4 rows into a readable table and identifies the winning configuration by MRR.

  • Below the loop (still inside run_full_evaluation() but outside both for loops), add the results printing code:
    # Print Results
    print("\n" + "=" * 70)
    print("RAG EVALUATION RESULTS")
    print("=" * 70)
    print(f"{'Configuration':<30} {'Hit Rate@3':<12} {'MRR':<8} {'Faithfulness':<12}")
    print("-" * 70)
    for row in results_table:
        print(f"{row['config']:<30} {row['hit_rate_at_3']:<12.2f} {row['mrr']:<8.3f} {row['faithfulness']:<12.2f}")
    print("=" * 70)
    
    best = max(results_table, key=lambda x: x["mrr"])
    print(f"\nBest configuration: {best['config']} (MRR: {best['mrr']:.3f})")

What does this code do?

The formatted table uses Python f-string alignment (:<30 for left-padding) to create a clean columnar output. The final two lines find the row with the highest MRR and print it as the winner.

  • At the very bottom of rag_eval_lab.py (outside the function, at the top level), add the entry point:
if __name__ == "__main__":
    run_full_evaluation()
  • Save rag_eval_lab.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before running it.

ⓧ I'd like to double check the full code

Your rag_eval_lab.py is large now. Scroll through and confirm these sections exist in order:

  • Imports and configuration constants (GROQ_MODEL, EMBEDDING_MODELS, CHUNK_CONFIGS).
  • load_essays(), chunk_text(), and chunk_corpus() functions.
  • build_collection(), retrieve(), and generate_answer() functions.
  • EVAL_SET list with 10 question-answer pairs.
  • compute_retrieval_metrics() and evaluate_faithfulness() functions.
  • run_full_evaluation() function with the nested loop, faithfulness scoring, and results printing.
  • if __name__ == "__main__": run_full_evaluation() at the very end.

This will take a few minutes

The script downloads and runs two embedding models across thousands of chunks. Expect 3-5 minutes depending on your machine. You will see progress bars for each embedding step and status messages as each configuration completes.

Before you run this, think about what you expect. You have 4 configurations being tested. Do you think the larger embedding model will always win, or could the smaller model beat it on certain chunk sizes?

  • Run the full evaluation by executing this command:
python rag_eval_lab.py

When it finishes, you will see a formatted table with 4 rows. Each row shows a configuration name, its Hit Rate@3, MRR, and faithfulness score. The final line identifies the best configuration by MRR.

What do these numbers mean?

Hit Rate@3 tells you what fraction of questions had the correct source essay somewhere in the top 3 results. MRR (Mean Reciprocal Rank) rewards finding the right document at rank 1 more than rank 3. Faithfulness measures whether the LLM's answers are grounded in the retrieved context.

The best configuration is the one where good retrieval and high faithfulness combine. If retrieval is poor but faithfulness is high, the LLM is faithfully saying "I don't know." If retrieval is good but faithfulness is low, the LLM is ignoring context and hallucinating.

Seeing a rate limit error?

Groq's free tier has per-minute request limits. If you hit a rate limit, wait 60 seconds and run the script again. The script makes about 24 Groq API calls total (12 for generation, 12 for faithfulness judging), so it usually stays within limits when calls run sequentially.

If you still hit rate limits, try running the script during off-peak hours (early morning or late evening).

Help me fix Groq rate limit errors in my evaluation script

You now have hard numbers proving which RAG configuration works best on this corpus. That table is not a vibe check. It is a measurement. Next, take on the Secret Mission to see if hybrid search can beat your best pure-vector result.

Secret mission

Hybrid Search Beats Pure Vector

Your best vector configuration is good. But what if you could make it better by combining two completely different search methods? In this challenge, you'll implement BM25 keyword search, fuse it with vector search using Reciprocal Rank Fusion, and prove with real numbers that hybrid beats pure vector.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with a free-tier API, so there are no ongoing costs.

Resources you used:

  • rag-eval-lab project folder on your Desktop (includes virtual environment, Python scripts, and installed packages).
  • Hugging Face cache (downloaded embedding models and dataset files stored in ~/.cache/huggingface/ on macOS/Linux or C:\Users\username\.cache\huggingface\ on Windows).
  • GROQ_API_KEY environment variable (session-scoped, cleared when you close your terminal).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with different chunking strategies, embedding models, or evaluation questions.

  • Your rag-eval-lab folder stays on your Desktop ready to run again.
  • The cached embedding models remain on disk so future runs start faster without re-downloading.
  • Your Groq API key environment variable is only active in the current terminal session. It disappears when you close the terminal.

Pause - I'll come back to this later

Free up disk space from the large cached models while keeping your project files intact.

  • Delete the Hugging Face model and dataset cache to reclaim disk space. The models will re-download automatically the next time you run your script.
  • On macOS/Linux, run:
rm -rf ~/.cache/huggingface/
  • On Windows, run:
rmdir /s /q %USERPROFILE%\.cache\huggingface\
  • Keep the rag-eval-lab folder. Your scripts and virtual environment stay ready for your return.

Delete - I don't want to use this again

Remove all project resources and cached data from your machine.

  • Delete the project folder. On macOS/Linux, run:
rm -rf ~/Desktop/rag-eval-lab
  • On Windows, run:
rmdir /s /q %USERPROFILE%\Desktop\rag-eval-lab
  • Delete the Hugging Face cache (embedding models and datasets). On macOS/Linux, run:
rm -rf ~/.cache/huggingface/
  • On Windows, run:
rmdir /s /q %USERPROFILE%\.cache\huggingface\
  • Delete your Groq API key from the Groq Console. Open https://console.groq.com, navigate to the API Keys page, and delete the key you created for this project.
  • Your GROQ_API_KEY environment variable is session-scoped and disappears when you close the terminal. No action needed.

Nice Work!

Nice Work!

You just built a RAG evaluation lab that produces real numbers comparing retrieval strategies across 215 Paul Graham essays. This is not a toy chatbot. It is a measurement instrument that proves which configuration works best and why.

You've learned how to:

  • Build a complete RAG pipeline over a real corpus using free tools: Groq for LLM generation, local sentence-transformers for embeddings, and ChromaDB for vector storage.
  • Design a hand-crafted evaluation set with ground-truth references and measure retrieval quality using Hit Rate@3 and MRR to compare chunking strategies and embedding models with real data.
  • Measure answer faithfulness using an LLM-as-judge approach, proving generated answers are grounded in retrieved context rather than hallucinated.
  • (Secret Mission) Implemented hybrid search combining BM25 keyword search with vector search using Reciprocal Rank Fusion, and proved it beats pure vector retrieval with real numbers.

Ready to quiz yourself?