Build a RAG Evaluation Pipeline

Build a document Q&A system with automated answer quality scoring.

Introduction

30 Second Summary

When companies deploy AI chatbots over their internal documents, the hardest question is not "can it answer?" but "can we trust its answers?" A system that confidently makes things up is worse than one that says nothing at all.

In this project, you will build a document Q&A system that retrieves real answers from a knowledge base, then build an evaluation pipeline that automatically scores whether those answers are faithful and relevant.

What You'll Build

You will ask your system a question in plain English, watch it pull the right documents from a vector database, and get back a grounded answer with source citations. Then you will run a single command that tests every answer against a quality rubric and produces a scored report.

By the end of this project, you'll have:

  • A working RAG pipeline where you type a natural language question and get back an accurate, sourced answer pulled from your custom knowledge base.
  • An automated evaluation suite that runs your system against a bank of test questions and scores every answer for faithfulness and relevance, catching hallucinations before a user ever sees them.
  • A portfolio-ready GitHub repo with a professional README that demonstrates production AI engineering skills to anyone who visits the link.
  • Secret Mission: Add a unified CLI so anyone who clones your repo can demo the entire system with a single command.

Are there any prerequisites?

You should be comfortable with basic Python (variables, functions, running scripts) and have a general understanding of how APIs work. You do not need any prior experience with AI, vector databases, or machine learning.

You will need a computer with VS Code installed and a GitHub account. Everything else (Python, packages, API key) gets set up in Step 1.

Before We Start

A RAG system has three core pieces: documents to search, a vector database to store them, and an LLM to generate answers from retrieved context. Before you can wire these together, you need a working Python environment with the right packages and an API key for Gemini.

In this step, you will install Python 3.13, create a project folder with a virtual environment, install your dependencies, and get a free Gemini API key from Google AI Studio.

In this step, get ready to:
  • Install Python 3.13 and verify it works.
  • Create a project folder with a virtual environment and install packages.
  • Get a Gemini API key and initialize Git.
Install Python 3.13

Python 3.13 is the runtime that powers your RAG pipeline, your vector database client, and the Gemini SDK. You need it installed and available on your PATH before anything else.

  • Open your browser and go to the Python 3.13 download page:

macOS

  • Go to python.org/downloads/macos.
  • Find the latest Python 3.13 release (3.13.14) and click Download macOS installer.
  • Double-click the downloaded .pkg file to launch the installer.
  • Click through the installer prompts, accepting the license and using the default installation.
  • After installation completes, open a new Terminal window.

Windows

  • Go to python.org/downloads/windows.
  • Find the latest Python 3.13 release (3.13.14) and click Windows installer (64-bit).
  • Run the downloaded .exe file.
  • On the first installer screen, check the box that says Add python.exe to PATH.
  • Click Install Now and wait for the installation to finish.
  • After installation completes, open a new Command Prompt window.

Why a new terminal window?

Your terminal reads the PATH variable when it opens. If Python was just installed, an existing terminal session will not know about it. Opening a fresh window picks up the updated PATH.

  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.13

Python 3.13 is installed and ready. Move on to the next substep.

ⓧ I see an older version

Your system has an older Python version on the PATH. On macOS, the system Python is often outdated and shadows a newer install.

  • On macOS, try running python3.13 --version to check if the 3.13 binary was installed alongside the older version.
  • If that works, use python3.13 in place of python3 for the rest of this step.
  • On Windows, uninstall the older version from Settings > Apps, then re-run the Python 3.13 installer with Add python.exe to PATH checked.
  • Open a new terminal and re-run the version check.

ⓧ Command not found

Python is not installed yet or not on your PATH.

  • On macOS, re-download the installer from python.org/downloads/macos and run it again.
  • On Windows, re-download the installer from python.org/downloads/windows and make sure you check Add python.exe to PATH before clicking Install.
  • After installing, close all terminal windows and open a fresh one.
  • Run the version check again.
Create the project folder and install packages

Your project needs its own folder and a virtual environment to keep its dependencies isolated from your system Python.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-eval-pipeline
cd rag-eval-pipeline
  • Create a virtual environment by running this command:

macOS

python3 -m venv venv

Windows

python -m venv venv
  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the start of your terminal prompt. This confirms you are inside the virtual environment.

What is a virtual environment?

A virtual environment is an isolated copy of Python that keeps this project's packages separate from your system Python. Anything you install with pip while the environment is active stays inside the venv/ folder.

  • Install the project dependencies by running:
pip install google-genai chromadb python-dotenv

What are these packages?

  • google-genai is Google's official SDK for calling the Gemini API.
  • chromadb is an open-source vector database that stores your documents as embeddings.
  • python-dotenv loads environment variables from a .env file so you can keep secrets out of your code.

The install will take a minute. When it finishes, you will see a success message listing all installed packages.

  • Before you run the next command, do you think the import will succeed or fail?
  • Verify all packages installed correctly by running:

macOS

python3 -c "import chromadb; from google import genai; print('Ready!')"

Windows

python -c "import chromadb; from google import genai; print('Ready!')"

You should see Ready! printed in your terminal. This proves all three packages are installed and importable.

Seeing an ImportError?

  • Make sure you see (venv) in your prompt. If not, you are running system Python. Re-run the activate command above.
  • If chromadb fails, re-run pip install chromadb and check for error messages about missing build tools.
  • Still stuck? Help me troubleshoot my package installation.
Get your Gemini API key and initialize Git

Your RAG system calls the Gemini API to generate answers. You need an API key to authenticate those requests. The free tier is more than enough for this project.

  • Open aistudio.google.com in your browser and sign in with your Google account.
  • If you are a new user, Google AI Studio automatically creates a project and API key for you. Click API Key in the left menu to see it.
  • If you do not see a key, click Create API key and follow the dialog to create one.
  • Copy your API key. You will paste it in the next step.

Is this free?

Yes. The Gemini API free tier provides access to models like gemini-2.5-flash at no cost and requires no credit card. Your project's usage will stay well within the free tier limits.

Keep your API key safe

Never commit your API key to Git or share it publicly. In the next few steps you will store it in a .env file that is excluded from version control.

  • Back in VS Code, open the rag-eval-pipeline folder. Click File > Open Folder and select the rag-eval-pipeline folder on your Desktop.
  • Create a new file called .env in the project root by clicking the new file icon in the Explorer sidebar.
  • Paste this into .env, replacing the placeholder with your actual key:
GEMINI_API_KEY=your-actual-key-here
  • Create another file called .env.example with this content:
GEMINI_API_KEY=your-api-key-here

Why two .env files?

The .env file holds your real key and is never committed. The .env.example file is a template that shows other developers (or future you) which environment variables the project expects. It gets committed to Git safely because it has no real secret in it.

  • Initialize a Git repository by running this command in your terminal:
git init
  • Create a .gitignore file in the project root with this content:
venv/
.env
__pycache__/
.chroma/
*.pyc

What does this .gitignore do?

  • venv/ keeps the large virtual environment folder out of your repo.
  • .env prevents your API key from being committed.
  • __pycache__/ and *.pyc exclude Python bytecode files.
  • .chroma/ excludes the vector database storage folder you will create later.
  • Verify Git is tracking the right files by running:
git status

You should see .env.example and .gitignore listed as untracked files. You should NOT see .env or the venv/ folder. This proves your .gitignore is working.

Seeing .env in the untracked files?

  • Double-check the spelling in your .gitignore file. It must say .env on its own line with no extra spaces.
  • Make sure the .gitignore file is saved.
  • Still stuck? Help me fix my .gitignore.

✔️ Awesome, I've got everything!

Great. Double check you have saved all three files (.env, .env.example, .gitignore) and that your terminal still shows (venv) in the prompt.

ⓧ I'd like to double check the full code

Here are all three files you should have at this point:

GEMINI_API_KEY=your-actual-key-here
GEMINI_API_KEY=your-api-key-here
venv/
.env
__pycache__/
.chroma/
*.pyc

Your environment is ready. You have Python, your packages, your API key, and a clean Git repo. Next up, you will build the knowledge base that your RAG system searches through.

Build the Knowledge Base

Your environment is ready, but your RAG system has nothing to search through yet. A RAG pipeline retrieves information from a knowledge base to ground its answers in real data.

In this step, you will create a set of fictional company documents and write a Python script that chunks and embeds them into a ChromaDB vector database. This gives your system a searchable collection of documents to pull from when answering questions.

In this step, get ready to:
  • Create a folder of markdown documents simulating a company's internal knowledge base.
  • Write an ingestion script that chunks documents and loads them into ChromaDB.
  • Run the script and verify the vector database is populated.
Create the knowledge base documents

Your RAG system will answer questions about a fictional company called NovaTech. You will create five markdown documents covering different areas of NovaTech's operations.

  • Create a knowledge_base/ folder inside your rag-eval-pipeline project by running this command:
mkdir knowledge_base
  • Create the first document by opening a new file called product-overview.md inside knowledge_base/ in VS Code, then paste this content:
# NovaTech Product Overview

## What is NovaTech Platform?

NovaTech Platform is an enterprise data integration solution that connects disparate data sources into a unified analytics layer. It supports real-time streaming ingestion from Kafka, batch processing from S3 and GCS, and direct database connections to PostgreSQL, MySQL, and MongoDB.

## Key Features

- Real-time data streaming with sub-second latency
- Automatic schema detection and evolution handling
- Built-in data quality checks with configurable thresholds
- Role-based access control (RBAC) with SSO integration
- RESTful API for programmatic access to all platform features

## Supported Integrations

NovaTech connects to over 50 data sources out of the box including cloud warehouses (Snowflake, BigQuery, Redshift), databases (PostgreSQL, MySQL, MongoDB, DynamoDB), streaming platforms (Kafka, Kinesis, Pub/Sub), and file storage (S3, GCS, Azure Blob).

## Pricing Tiers

- Starter: Up to 10 data sources, 1M events/day, 3 users
- Professional: Up to 50 data sources, 10M events/day, 25 users
- Enterprise: Unlimited sources, unlimited events, unlimited users, dedicated support

Why markdown with headings?

Each ## heading creates a natural chunk boundary. Your ingestion script will split documents at these headings, so each chunk is a self-contained topic that ChromaDB can match against user questions.

  • Create the second document security.md inside knowledge_base/ with this content:
# NovaTech Security Policies

## Data Encryption

All data in transit is encrypted using TLS 1.3. Data at rest is encrypted using AES-256. Encryption keys are managed through AWS KMS with automatic rotation every 90 days.

## Access Control

NovaTech implements role-based access control (RBAC) with the principle of least privilege. All access changes are logged in an immutable audit trail accessible via GET /api/v2/audit-logs.

## Compliance

NovaTech is certified for:
- SOC 2 Type II
- HIPAA (with BAA)
- GDPR (EU data residency available)
- ISO 27001

## Data Retention

- Active connections: Data retained indefinitely while connection is active
- Deleted connections: Metadata removed within 24 hours, cached data purged within 7 days
- Audit logs: Retained for 2 years minimum
- Query history: Retained for 90 days

## Incident Response

Security incidents are classified as P1 (critical), P2 (high), P3 (medium), P4 (low). P1 incidents trigger immediate notification to all affected customers within 1 hour. Post-incident reports are published within 5 business days.

## Network Security

NovaTech platform egress IPs: 52.14.88.0/24 and 3.135.200.0/24 (US-East), 18.130.56.0/24 (EU-West). Allowlist these ranges in your firewall for NovaTech to reach your data sources.
  • Confirm the first two files exist by running:
ls knowledge_base/

You should see product-overview.md and security.md listed.

  • Create api-reference.md inside knowledge_base/ with this content covering authentication and rate limits:
# NovaTech API Reference

## Authentication

All API requests require a Bearer token in the Authorization header. Generate tokens from the Settings > API Keys page in the dashboard.

```
Authorization: Bearer nt_live_xxxxxxxxxxxx
```

Tokens prefixed with `nt_live_` are production tokens. Tokens prefixed with `nt_test_` are sandbox tokens that do not affect production data.

## Rate Limits

- Starter plan: 100 requests per minute
- Professional plan: 1000 requests per minute
- Enterprise plan: 10000 requests per minute

Exceeding rate limits returns HTTP 429 with a Retry-After header.
  • Add the endpoints section below the rate limits content in the same file:
## Endpoints

### POST /api/v2/connections
Create a new data source connection.

Required fields: `name`, `type`, `credentials`
Optional fields: `schema_override`, `sync_frequency`, `tags`

### GET /api/v2/connections/{id}/status
Returns the current sync status of a connection.

Response includes: `status` (active, paused, error), `last_sync_at`, `records_processed`, `error_message`

### POST /api/v2/queries
Execute an analytics query against connected sources.

Required fields: `sql`, `connection_ids`
Optional fields: `timeout_seconds`, `cache_ttl`

### DELETE /api/v2/connections/{id}
Remove a connection and all associated metadata. This does NOT delete the underlying data source.
  • Create onboarding.md inside knowledge_base/ with this content covering account setup and connections:
# NovaTech Onboarding Guide

## Getting Started (First 30 Minutes)

1. Accept your email invitation and set a password
2. Complete the security questionnaire (required for compliance)
3. Set up two-factor authentication (mandatory for all users)
4. Join your assigned workspace

## Setting Up Your First Connection

1. Navigate to Connections > New Connection
2. Select your data source type from the catalog
3. Enter connection credentials (hostname, port, username, password or OAuth)
4. Click "Test Connection" to verify connectivity
5. Choose sync frequency: real-time, hourly, or daily
6. Select tables or collections to sync
7. Click "Activate" to start the initial sync
  • Add the workspaces and roles section below the connection setup content in the same file:
## Understanding Workspaces

Workspaces are isolated environments for teams. Each workspace has its own connections, queries, and access controls. Users can belong to multiple workspaces but data does not cross workspace boundaries.

## User Roles

- Viewer: Read-only access to dashboards and query results
- Analyst: Can create and run queries, build dashboards
- Admin: Full access including connection management and user administration
- Owner: Workspace owner with billing and deletion permissions

## Getting Help

- Documentation: docs.novatech.io
- Support tickets: support.novatech.io (response within 4 hours for Professional, 1 hour for Enterprise)
- Community forum: community.novatech.io
- Emergency hotline: Available for Enterprise customers only
  • Confirm you now have four files by running:
ls knowledge_base/

You should see api-reference.md, onboarding.md, product-overview.md, and security.md listed.

  • Create the final document troubleshooting.md inside knowledge_base/ with this content covering connection failures and sync issues:
# NovaTech Troubleshooting Guide

## Connection Failures

### Error: CONN_TIMEOUT
The platform cannot reach the data source within 30 seconds.

Fix: Check that the source's firewall allows inbound connections from NovaTech IP ranges (listed in Settings > Network). Verify the hostname and port are correct.

### Error: AUTH_FAILED
Authentication credentials are invalid or expired.

Fix: Regenerate credentials in the source system and update them in NovaTech via PUT /api/v2/connections/{id}. For OAuth sources, re-authorize through the dashboard.

### Error: SCHEMA_MISMATCH
The source schema has changed since the last sync.

Fix: Run a schema refresh from the connection details page or POST /api/v2/connections/{id}/refresh-schema. Review the detected changes before approving.
  • Add the sync issues and performance section below the connection failures content in the same file:
## Sync Issues

### Partial Sync Failures
If a sync partially completes, check the sync logs at GET /api/v2/connections/{id}/logs. Common causes: network interruption, source rate limiting, or row-level permission errors.

### Duplicate Records
NovaTech uses event IDs for deduplication. If you see duplicates, ensure your source provides unique identifiers. Configure the dedup key in connection settings.

## Performance

### Slow Queries
Queries taking over 10 seconds may benefit from materialized views. Create them via POST /api/v2/materialized-views. Ensure your query uses indexed columns for filtering.

### How to Reset Your API Key
Navigate to Settings > API Keys, click the three-dot menu next to your key, select "Regenerate". The old key is immediately invalidated. Update all applications using the old key.
  • Save all files and verify all five documents are present by running:
ls knowledge_base/

You should see all five files: api-reference.md, onboarding.md, product-overview.md, security.md, and troubleshooting.md.

Missing a file?

Make sure each file is saved inside the knowledge_base/ folder, not in the root rag-eval-pipeline directory. Double-check file names are lowercase with hyphens, not spaces.

help me find my missing knowledge base files

Write the ingestion script

Now you need a script that reads these documents, splits them into chunks, and loads them into ChromaDB. Each chunk becomes a searchable unit that your RAG system can retrieve later.

  • Create a new file called ingest.py in the root of your rag-eval-pipeline folder and paste the imports and chunking function:
import os
import chromadb
from pathlib import Path


def chunk_markdown(content, filename):
    """Split markdown content into chunks by heading sections."""
    chunks = []
    current_chunk = ""
    current_heading = filename

    for line in content.split("\n"):
        if line.startswith("## "):
            if current_chunk.strip():
                chunks.append({
                    "text": current_chunk.strip(),
                    "heading": current_heading,
                    "source": filename
                })
            current_heading = line.strip("# ").strip()
            current_chunk = line + "\n"
        else:
            current_chunk += line + "\n"

    if current_chunk.strip():
        chunks.append({
            "text": current_chunk.strip(),
            "heading": current_heading,
            "source": filename
        })

    return chunks

What does this chunking function do?

The chunk_markdown() function splits a document every time it encounters a ## heading. Each chunk includes the text content, the heading it belongs to, and the source filename.

This means a question about "rate limits" will match the Rate Limits chunk specifically, not the entire API reference document.

  • Below the chunk_markdown function in the same file, add the ingestion function that reads files and loads them into ChromaDB:
def ingest_documents():
    """Ingest all markdown files from knowledge_base/ into ChromaDB."""
    kb_path = Path("knowledge_base")
    if not kb_path.exists():
        print("Error: knowledge_base/ folder not found.")
        return

    client = chromadb.PersistentClient(path=".chroma")
    collection = client.get_or_create_collection(name="novatech_docs")

    all_documents = []
    all_ids = []
    all_metadatas = []

    for md_file in sorted(kb_path.glob("*.md")):
        content = md_file.read_text(encoding="utf-8")
        chunks = chunk_markdown(content, md_file.name)

        for i, chunk in enumerate(chunks):
            doc_id = f"{md_file.stem}_chunk_{i}"
            all_documents.append(chunk["text"])
            all_ids.append(doc_id)
            all_metadatas.append({
                "source": chunk["source"],
                "heading": chunk["heading"],
                "chunk_index": i
            })

What does this function do?

PersistentClient tells ChromaDB to save data to disk at the .chroma/ path. This means your embedded documents persist between script runs.

The function loops through every .md file, chunks it, and builds lists of documents, IDs, and metadata. Each chunk gets a unique ID based on the filename and chunk index.

  • Add the final section below the for loop that persists everything to ChromaDB and runs the script:
    collection.upsert(
        documents=all_documents,
        ids=all_ids,
        metadatas=all_metadatas
    )

    print(f"Ingested {len(all_documents)} chunks from {len(list(kb_path.glob('*.md')))} documents")
    print(f"Collection '{collection.name}' now contains {collection.count()} items")


if __name__ == "__main__":
    ingest_documents()

Why upsert instead of add?

The upsert() method inserts new documents and updates existing ones if the IDs match. This makes the script safe to run multiple times without creating duplicates.

  • Save ingest.py.
  • Create a requirements.txt file in the root of your project with the pinned dependencies:
google-genai>=2.3.0
chromadb>=1.5.0
python-dotenv>=1.0.0
  • Save requirements.txt.

What is requirements.txt for?

This file lists the exact packages your project needs so anyone who clones your repo can install them with a single pip install -r requirements.txt command. It is a standard Python convention for reproducible environments.

✔️ Awesome, I've got everything!

Great. Double-check you have saved both ingest.py and requirements.txt.

ⓧ I'd like to double check the full code

Your ingest.py should contain three sections you added above, in this order:

  • The imports and chunk_markdown() function (first code block you pasted).
  • The ingest_documents() function with the for loop (second code block).
  • The collection.upsert() call, print statements, and if __name__ block (third code block).

Scroll up and confirm each section is present and in order. The upsert and print lines must be indented inside ingest_documents() (4 spaces).

Run the ingestion and verify your collection

The knowledge base documents are ready and the script is written. Time to run the ingestion and confirm everything landed in ChromaDB.

  • Before you run this, how many chunks do you think 5 documents will produce? Consider that each document has multiple ## headings plus a title section.
  • Run the ingestion script:
python ingest.py

You should see output confirming the ingestion completed:

Ingested 23 chunks from 5 documents
Collection 'novatech_docs' now contains 23 items

What just happened?

ChromaDB read each markdown file, split it at every ## heading, generated an embedding vector for each chunk using its built-in sentence-transformer model, and stored everything in the .chroma/ folder on disk.

Your exact chunk count may vary slightly depending on blank lines, but it should be around 23. Each chunk is now searchable by semantic similarity.

Seeing an error about knowledge_base/ not found?

Make sure you run the script from inside the rag-eval-pipeline folder, not from a parent directory. Run ls knowledge_base/ first to confirm the folder is visible from your current location.

help me fix my ingest.py script

Your knowledge base is loaded and searchable. Next up, you will build the query engine that retrieves relevant chunks and uses Gemini to generate grounded answers.

Build the RAG Query Engine

Your knowledge base is loaded into ChromaDB and ready to be searched. The missing piece is the pipeline that takes a user's question, finds the most relevant document chunks, and feeds them to an LLM that can synthesize a grounded answer.

In this step, you will wire up the retrieval-to-generation pipeline. You will query ChromaDB for relevant chunks, construct a prompt with that context, and send it to Gemini to generate answers that are grounded in your documents.

In this step, get ready to:
  • Write a query engine that retrieves relevant context from ChromaDB and generates answers via Gemini.
  • Format the output to show the answer, source documents, and relevance distances.
  • Test the system end-to-end with multiple questions.
Write the retrieval and client functions

The query engine needs two helpers before it can generate answers: one to initialize the Gemini client with your API key, and one to search ChromaDB for relevant chunks.

  • Create a new file called query.py in your rag-eval-pipeline folder and add the following imports and helper functions:
import os
import sys
import chromadb
from dotenv import load_dotenv
from google import genai

load_dotenv()


def get_gemini_client():
    """Initialize and return a Gemini client."""
    api_key = os.getenv("GEMINI_API_KEY")
    if not api_key:
        print("Error: GEMINI_API_KEY not found in .env file")
        sys.exit(1)
    return genai.Client(api_key=api_key)


def retrieve_context(question, n_results=3):
    """Query ChromaDB for relevant document chunks."""
    client = chromadb.PersistentClient(path=".chroma")
    collection = client.get_collection(name="novatech_docs")

    results = collection.query(
        query_texts=[question],
        n_results=n_results
    )

    return results

What does this code do?

  • load_dotenv() reads your .env file so the API key is available as an environment variable.
  • get_gemini_client() reads the API key and returns an authenticated Gemini client. If the key is missing, it stops the script with a clear error.
  • retrieve_context() takes a question string, connects to your existing ChromaDB collection, and returns the top 3 most similar document chunks. ChromaDB handles the embedding and similarity search automatically.
  • Save query.py.
  • Verify the retrieval function works by running this command:
python -c "from query import retrieve_context; r = retrieve_context('reset API key'); print('Found', len(r['documents'][0]), 'relevant chunks')"

You should see Found 3 relevant chunks printed to your terminal.

Seeing an import error or collection not found?

  • Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.
  • If you see a collection not found error, run python ingest.py again to repopulate the database.
  • Check that your .env file is in the same folder as query.py.

Why can't Python find my ChromaDB collection?

Add the answer generation function

Retrieval alone just gives you raw text chunks. The generation function takes those chunks, builds a grounded prompt, and asks Gemini to synthesize an answer that stays faithful to the source material.

  • In query.py, add the following function below retrieve_context():
def generate_answer(question, context_results):
    """Generate an answer using Gemini with retrieved context."""
    documents = context_results["documents"][0]
    metadatas = context_results["metadatas"][0]
    distances = context_results["distances"][0]

    context_block = ""
    for i, (doc, meta, dist) in enumerate(zip(documents, metadatas, distances)):
        context_block += f"\n--- Source: {meta['source']} | Section: {meta['heading']} | Distance: {dist:.4f} ---\n"
        context_block += doc + "\n"

    prompt = f"""You are a helpful assistant for NovaTech Platform. Answer the user's question based ONLY on the provided context. If the context does not contain enough information to answer the question, say "I don't have enough information to answer this question based on the available documentation."

Context:
{context_block}

Question: {question}

Answer:"""

    client = get_gemini_client()
    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=prompt
    )

    return response.text

What does this code do?

  • It unpacks the retrieved documents, metadata, and distance scores from ChromaDB's query results.
  • It formats each chunk into a labelled context block so Gemini can see which source and section each piece of text came from.
  • The prompt instructs Gemini to answer ONLY from the provided context. This grounding instruction is what prevents hallucination on answerable questions.
  • It calls client.models.generate_content() with the gemini-2.5-flash model and returns the response text.
Add the output formatting and main entry point

The final piece ties retrieval and generation together into a single function that prints the answer with source citations and distances.

  • Add the following code below generate_answer() in query.py:
def query_rag(question, verbose=False):
    """Full RAG pipeline: retrieve context and generate answer."""
    context_results = retrieve_context(question)
    answer = generate_answer(question, context_results)

    print(f"\nQuestion: {question}")
    print(f"\nAnswer: {answer}")

    if verbose:
        print("\n--- Retrieved Sources ---")
        for i, (meta, dist) in enumerate(zip(
            context_results["metadatas"][0],
            context_results["distances"][0]
        )):
            print(f"  {i+1}. {meta['source']} > {meta['heading']} (distance: {dist:.4f})")

    return answer


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python query.py \"Your question here\"")
        sys.exit(1)

    question = sys.argv[1]
    query_rag(question, verbose=True)

What does this code do?

  • query_rag() is the full pipeline: it calls retrieve_context() and generate_answer() in sequence, then prints the formatted result.
  • When verbose=True, it also prints each source document and its distance score. Lower distance means higher relevance.
  • The __main__ block lets you run the script from the terminal with a question as an argument.
  • Save query.py.

Before you run this, what do you think will happen when you ask a question that IS covered in the knowledge base?

  • Test your RAG pipeline by running this command:
python query.py "How do I reset my API key?"

You should see a grounded answer explaining the API key reset process, followed by a list of retrieved sources with their distance scores.

Seeing an API key error or empty response?

  • If you see the GEMINI_API_KEY not found error, check that your .env file contains your actual API key (not the placeholder text).
  • If you get a rate limit or quota error, wait 60 seconds and try again. The free tier allows 10 requests per minute.
  • Make sure you wrapped your question in quotes so the terminal treats it as a single argument.

Why is my Gemini API call failing?

Test with different questions

One successful answer is promising, but you need to see the system handle different types of questions to confirm it works reliably.

  • Try asking about encryption by running:
python query.py "What encryption does NovaTech use for data at rest?"

You should see an answer mentioning AES-256 and AWS KMS, sourced from the security document.

  • Try a question about rate limits:
python query.py "What are the rate limits for the Professional plan?"

The answer should mention 1000 requests per minute, pulled from the API reference document.

  • Try an onboarding question:
python query.py "What user roles are available?"

You should see the four roles (Viewer, Analyst, Admin, Owner) described with their permissions, sourced from the onboarding document.

What should I notice in the source distances?

Lower distance values mean the chunk is more semantically similar to your question. You will typically see the most relevant chunk at distance 0.5-1.0 and less relevant chunks at 1.2+. These scores will become important in the next step when you evaluate answer quality.

✔️ Awesome, I've got everything!

Great. Double check you have saved query.py with all four functions and the main entry point.

ⓧ I'd like to double check the full code

Scroll up and confirm your query.py contains these sections in order:

  • Imports and load_dotenv() at the top.
  • get_gemini_client() function that reads the API key and returns a client.
  • retrieve_context() function that queries ChromaDB for the top 3 chunks.
  • generate_answer() function that builds a grounded prompt and calls Gemini.
  • query_rag() function that ties everything together and prints formatted output.
  • if __name__ == "__main__" block that reads the question from command-line arguments.

Your RAG pipeline is answering questions with grounded, sourced responses. Next up, you will expose where it breaks down by asking questions the knowledge base cannot answer.

Expose the Failure

Your RAG system handled straightforward questions in the last step. It found the right context and generated grounded answers. But what happens when someone asks a question the documentation does not cover?

Naive RAG systems break silently. They still retrieve context (the closest match, even if irrelevant) and still generate a confident-sounding answer. The result is hallucination that looks indistinguishable from a real answer. In this step, you will intentionally break your system to see where it fails.

In this step, get ready to:
  • Test the system with unanswerable questions to expose hallucination.
  • Test with ambiguous questions to find irrelevant retrieval.
  • Document failure cases in a structured test file for automated evaluation.
Ask questions the knowledge base cannot answer

Your knowledge base covers NovaTech's product features, API, troubleshooting, onboarding, and security. It says nothing about the company's financials or leadership.

Before you run the next command, think about what you expect to happen. The question is about NovaTech's annual revenue, which is not in any of the five documents. Do you think the system will refuse to answer, or will it produce something anyway?

  • Ask a question that has no answer in your documents by running:
python query.py "What is NovaTech's annual revenue?"

Look at the output carefully. The system retrieved context (likely from the pricing tiers or product overview sections) and generated an answer. Despite the prompt instructing it to refuse when context is insufficient, the model may still produce a confident-sounding response about revenue that is completely fabricated.

Why does this happen?

The retrieval step always returns the top 3 closest chunks, even when none of them are truly relevant. The LLM sees context that mentions pricing and features, and it extrapolates. This is hallucination in RAG systems.

For a client trusting this system to answer questions about their internal docs, a hallucinated answer is worse than no answer at all.

  • Try another unanswerable question by running:
python query.py "Who is the CEO of NovaTech?"

Again, the system retrieves the closest chunks it can find and generates an answer. None of the documents mention a CEO, but the model may invent a name or hedge without clearly refusing.

Ask ambiguous questions that test retrieval quality

Unanswerable questions expose hallucination. Ambiguous questions expose a different failure mode: the retrieval step pulling the wrong context entirely.

  • Test a question that sits at the boundary of what the docs cover by running:
python query.py "Can NovaTech connect to Oracle databases?"

Look at the retrieved sources in the verbose output. The system finds chunks about supported integrations (PostgreSQL, MySQL, MongoDB, etc.) but Oracle is never mentioned. The correct answer is "this information is not in the docs," but the system may say "yes" or "no" based on inference rather than documentation.

Two failure modes, one root cause

Both failures stem from the same gap: the system has no mechanism to measure whether retrieved context actually contains the answer. It always generates, never refuses.

In a production system, you would need automated measurement to catch these failures before your client does. That is exactly what you will build in the next step.

Document the failure cases in a test file

You have seen two types of failures. Now you will record these as structured test cases so you can automatically evaluate the system against them.

  • Create a file called test_questions.json in your rag-eval-pipeline folder and paste in the answerable test cases:
[
    {
        "question": "How do I reset my API key?",
        "expected_answer": "Navigate to Settings > API Keys, click the three-dot menu next to your key, select Regenerate. The old key is immediately invalidated.",
        "category": "answerable"
    },
    {
        "question": "What encryption does NovaTech use for data at rest?",
        "expected_answer": "AES-256 with keys managed through AWS KMS with automatic rotation every 90 days.",
        "category": "answerable"
    },
    {
        "question": "What are the rate limits for the Professional plan?",
        "expected_answer": "1000 requests per minute.",
        "category": "answerable"
    },
    {
        "question": "How do I fix a CONN_TIMEOUT error?",
        "expected_answer": "Check that the source firewall allows inbound connections from NovaTech IP ranges listed in Settings > Network. Verify the hostname and port are correct.",
        "category": "answerable"
    },

What does this structure do?

Each test case has three fields. The question is what you will send to the RAG system. The expected_answer is the ground truth the judge will compare against. The category tells the evaluator whether the system should answer or refuse.

These four "answerable" cases have clear answers in the knowledge base. They test that the system retrieves correct context and generates accurate answers.

  • Below the last answerable entry (after the trailing comma), add the unanswerable test cases and the closing bracket:
    {
        "question": "What is NovaTech's annual revenue?",
        "expected_answer": "UNANSWERABLE - This information is not in the documentation.",
        "category": "unanswerable"
    },
    {
        "question": "Who is the CEO of NovaTech?",
        "expected_answer": "UNANSWERABLE - This information is not in the documentation.",
        "category": "unanswerable"
    },
    {
        "question": "Can NovaTech connect to Oracle databases?",
        "expected_answer": "UNANSWERABLE - Oracle is not listed in the supported integrations. The docs mention PostgreSQL, MySQL, MongoDB, DynamoDB, Snowflake, BigQuery, and Redshift.",
        "category": "unanswerable"
    }
]

Why mark questions as unanswerable?

These are the failure cases you just observed. The expected answer is "UNANSWERABLE" because a correct system should refuse rather than hallucinate. Your evaluation pipeline will score the system on whether it correctly refuses these questions.

  • Save test_questions.json.
  • Verify the file is valid JSON by running:
python -c "import json; data = json.load(open('test_questions.json')); print(f'{len(data)} test cases loaded')"

You should see 7 test cases loaded printed to your terminal.

Seeing a JSONDecodeError?

Check for a missing comma between entries. Every entry except the last one needs a trailing comma after the closing }. Also confirm the file starts with [ and ends with ].

Help me fix my test_questions.json file

✔️ Awesome, I've got everything!

Great. Double check you have saved test_questions.json and that the verification printed 7 test cases loaded.

ⓧ I'd like to double check the full code

Scroll up and confirm you have both sections in your file. The first section contains 4 answerable test cases (API key reset, encryption, rate limits, CONN_TIMEOUT). The second section contains 3 unanswerable test cases (annual revenue, CEO, Oracle databases). Make sure both sections are present inside a single JSON array.

You have exposed the gap: your RAG system confidently answers questions it should refuse, and it pulls irrelevant context when the real answer is not in the docs. Next up, you will build an evaluation pipeline that automatically measures these failures across every test case.

Build the Evaluation Pipeline

In the previous step you found real failure cases where your RAG system hallucinated or pulled irrelevant context. Those failures prove you cannot rely on manual spot-checking to know whether the system works.

You need automated quality measurement. In this step, you will build an evaluation pipeline that uses Gemini as a judge to score every answer on faithfulness and relevance, producing a structured report you could hand to a client.

In this step, get ready to:
  • Write the evaluation judge function that scores answers on faithfulness and relevance.
  • Build the evaluation runner that tests all questions and generates a scored report.
  • Run the pipeline and generate a JSON eval report with category breakdowns.
Write the evaluation judge function

The LLM-as-a-judge pattern uses one LLM call to evaluate another LLM's output. Your judge will read the question, the retrieved context, and the generated answer, then score how faithful and relevant the answer is.

  • Create a new file called evaluate.py in your rag-eval-pipeline folder and add the imports and client setup:
import os
import sys
import json
from dotenv import load_dotenv
from google import genai
from query import retrieve_context, generate_answer

load_dotenv()


def get_gemini_client():
    """Initialize and return a Gemini client."""
    api_key = os.getenv("GEMINI_API_KEY")
    if not api_key:
        print("Error: GEMINI_API_KEY not found in .env file")
        sys.exit(1)
    return genai.Client(api_key=api_key)

What does this code do?

  • from query import retrieve_context, generate_answer reuses the RAG functions you already built. No need to duplicate code.
  • get_gemini_client() handles API key loading the same way query.py does.
  • Save evaluate.py and verify the module loads by running:
python -c "from evaluate import get_gemini_client; print('evaluate.py loaded')"

You should see "evaluate.py loaded" printed with no errors.

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.
  • Confirm you are running the command from inside the rag-eval-pipeline folder where query.py lives.
  • Why can't Python find my evaluate module?

Now you need the judge function itself. This function constructs a prompt that tells Gemini to act as an evaluation judge and score the RAG system's answer.

  • In evaluate.py, add the first part of the judge_answer function below get_gemini_client():
def judge_answer(question, answer, context_docs, expected_answer, category):
    """Use Gemini as a judge to score the answer on faithfulness and relevance."""
    context_text = "\n".join(context_docs)

    prompt = f"""You are an evaluation judge for a RAG (Retrieval-Augmented Generation) system. Score the following answer on two dimensions.

Question: {question}
Expected Answer: {expected_answer}
Category: {category} (if "unanswerable", the system should refuse to answer rather than hallucinate)

Retrieved Context:
{context_text}

System Answer: {answer}

Score on these dimensions (1-5 each):

1. FAITHFULNESS: Is the answer supported by the retrieved context? (5 = fully supported, 1 = contradicts or fabricates information not in context)
2. RELEVANCE: Does the answer address the question? For "unanswerable" questions, does it correctly refuse? (5 = perfectly addresses the question or correctly refuses, 1 = completely off-topic or hallucinates when should refuse)

Respond in this exact JSON format:
{{"faithfulness": <score>, "faithfulness_reason": "<brief reason>", "relevance": <score>, "relevance_reason": "<brief reason>"}}"""

What does this prompt do?

The prompt instructs Gemini to act as a judge and score along two dimensions. Faithfulness measures whether the answer is grounded in the retrieved context (not hallucinated). Relevance measures whether it actually addresses the question.

Notice the category field. For "unanswerable" questions, the judge checks whether the system correctly refused to answer instead of making something up. This is how you catch hallucination on questions outside your knowledge base.

  • Still in evaluate.py, complete the judge_answer function by adding the API call and response parsing directly below the prompt:
    client = get_gemini_client()
    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=prompt
    )

    try:
        response_text = response.text.strip()
        if response_text.startswith("```"):
            response_text = response_text.split("\n", 1)[1].rsplit("```", 1)[0]
        scores = json.loads(response_text)
        return scores
    except (json.JSONDecodeError, IndexError):
        return {
            "faithfulness": 0,
            "faithfulness_reason": "Failed to parse judge response",
            "relevance": 0,
            "relevance_reason": "Failed to parse judge response"
        }

Why strip markdown fences from the response?

LLMs sometimes wrap JSON output in markdown code fences (```json ... ```). The stripping logic handles this gracefully so json.loads() can parse the scores.

The fallback returns zeroes with an explanation. This prevents the entire evaluation from crashing if one judge response is malformed.

  • Save evaluate.py and confirm the judge function loads by running:
python -c "from evaluate import judge_answer; print('judge_answer loaded')"

You should see "judge_answer loaded" with no syntax errors.

Seeing a SyntaxError?

  • Check that the second code block (API call + parsing) is indented inside judge_answer. The client = get_gemini_client() line should be at the same indent level as the prompt = line above it.
  • Verify the triple-quoted prompt string ends with """ (three double quotes) on its own at the end of the last line.
  • Why am I getting a SyntaxError in my judge_answer function?
Build and run the evaluation pipeline

The judge function can score a single answer. Now you need a runner that feeds every question from your test set through the RAG pipeline, scores each one, and prints a live report as it goes.

  • In evaluate.py, add the start of the run_evaluation function below judge_answer:
def run_evaluation():
    """Run the full evaluation pipeline."""
    with open("test_questions.json", "r") as f:
        test_cases = json.load(f)

    results = []
    total_faithfulness = 0
    total_relevance = 0

    print("=" * 60)
    print("RAG EVALUATION REPORT")
    print("=" * 60)

    for i, test in enumerate(test_cases):
        print(f"\n[{i+1}/{len(test_cases)}] {test['question']}")

        context_results = retrieve_context(test["question"])
        answer = generate_answer(test["question"], context_results)
        scores = judge_answer(
            test["question"],
            answer,
            context_results["documents"][0],
            test["expected_answer"],
            test["category"]
        )

        total_faithfulness += scores.get("faithfulness", 0)
        total_relevance += scores.get("relevance", 0)

What does this loop do?

  • It loads your 7 test cases from test_questions.json (the file you created in the previous step).
  • For each question, it calls retrieve_context and generate_answer from your query.py module.
  • It then passes each answer to judge_answer for scoring and accumulates the totals.
  • Continue the run_evaluation function by adding the result storage, per-question output, summary statistics, and the main entry point:
        result = {
            "question": test["question"],
            "category": test["category"],
            "answer": answer,
            "scores": scores
        }
        results.append(result)

        print(f"  Faithfulness: {scores.get('faithfulness', 0)}/5 - {scores.get('faithfulness_reason', 'N/A')}")
        print(f"  Relevance:    {scores.get('relevance', 0)}/5 - {scores.get('relevance_reason', 'N/A')}")

    num_tests = len(test_cases)
    avg_faithfulness = total_faithfulness / num_tests if num_tests > 0 else 0
    avg_relevance = total_relevance / num_tests if num_tests > 0 else 0

    print("\n" + "=" * 60)
    print("SUMMARY")
    print("=" * 60)
    print(f"Total questions evaluated: {num_tests}")
    print(f"Average Faithfulness: {avg_faithfulness:.2f}/5")
    print(f"Average Relevance:    {avg_relevance:.2f}/5")
    print(f"Overall Score:        {((avg_faithfulness + avg_relevance) / 2):.2f}/5")


if __name__ == "__main__":
    run_evaluation()

What does this section do?

  • Each iteration stores the full result (question, category, answer, scores) for later reporting.
  • The per-question print gives you live feedback as the evaluation runs. Each question takes a few seconds because it makes two API calls (one to generate, one to judge).
  • The summary computes average faithfulness and relevance across all test cases.

Before you run this, what do you think the overall score will be? Consider that 4 questions are answerable and 3 are unanswerable (where the system might hallucinate).

  • Save evaluate.py and run the evaluation pipeline:
python evaluate.py

You should see a formatted report with each question scored on faithfulness and relevance (1-5), followed by a summary showing average scores. The script takes 30-60 seconds because it makes multiple API calls.

Seeing a rate limit error?

  • The Gemini free tier has rate limits. If you hit them, wait 60 seconds and run the script again.
  • If you see GEMINI_API_KEY not found, confirm your .env file is in the rag-eval-pipeline folder and contains your key.
  • My evaluation script is failing with an API or import error.
Add category breakdown and save the report as JSON

The summary gives you an overall score, but you also need to see how the system performs on answerable vs. unanswerable questions separately. You also want a persistent JSON report for tracking quality over time.

  • In evaluate.py, find the if __name__ == "__main__": block at the bottom of the file.
  • Add the following code between the last print(f"Overall Score: line and the if __name__ block:
    answerable = [r for r in results if r["category"] == "answerable"]
    unanswerable = [r for r in results if r["category"] == "unanswerable"]

    if answerable:
        avg_a = sum(r["scores"].get("relevance", 0) for r in answerable) / len(answerable)
        print(f"\nAnswerable questions ({len(answerable)}): Avg relevance {avg_a:.2f}/5")
    if unanswerable:
        avg_u = sum(r["scores"].get("relevance", 0) for r in unanswerable) / len(unanswerable)
        print(f"Unanswerable questions ({len(unanswerable)}): Avg refusal quality {avg_u:.2f}/5")

    report_path = "eval_report.json"
    with open(report_path, "w") as f:
        json.dump({
            "summary": {
                "total_questions": num_tests,
                "avg_faithfulness": round(avg_faithfulness, 2),
                "avg_relevance": round(avg_relevance, 2),
                "overall_score": round((avg_faithfulness + avg_relevance) / 2, 2)
            },
            "results": results
        }, f, indent=2)

    print(f"\nDetailed report saved to: {report_path}")

What does this add?

  • The category breakdown splits results into answerable (did the system get the right answer?) and unanswerable (did it correctly refuse to hallucinate?). This is how you measure hallucination resistance.
  • The JSON report at eval_report.json persists every score and justification. You can diff reports across code changes to catch regressions.
  • Save evaluate.py and run the evaluation again:
python evaluate.py

This time you should see the category breakdown at the bottom of the summary (answerable avg relevance and unanswerable refusal quality), plus a final line saying "Detailed report saved to: eval_report.json".

  • Confirm the JSON report was created by running:
python -c "import json; r=json.load(open('eval_report.json')); print(f'Overall score: {r[\"summary\"][\"overall_score\"]}/5')"

You should see your overall score printed from the saved JSON file.

Not seeing the category breakdown?

  • Make sure the new code is inside the run_evaluation function (indented with 4 spaces), not at the module level.
  • Verify the new code sits between the print(f"Overall Score: line and if __name__ == "__main__":. If the if __name__ block moved inside the function, the script will not run.
  • Why isn't my category breakdown appearing in the eval output?

✔️ Awesome, I've got everything!

Double check you have saved evaluate.py and that eval_report.json exists in your project folder.

ⓧ I'd like to double check the full code

Your evaluate.py is built from the chunks above. Scroll up and confirm you have each section in order:

  • Imports and get_gemini_client() at the top.
  • The complete judge_answer() function (prompt + API call + parsing).
  • The complete run_evaluation() function (loop + summary + category breakdown + JSON output).
  • The if __name__ == "__main__": block calling run_evaluation() at the very bottom.

You now have an automated evaluation pipeline that scores your RAG system on every test case and saves a structured report. Next up, you will push the entire project to GitHub with a professional README so anyone can clone and run it.

Push to GitHub

Your RAG system works and your evaluation pipeline proves it. But a project that lives only on your laptop does not help anyone else understand what you can build.

A clean GitHub repo with a professional README demonstrates the communication and documentation skills that set engineers apart. In this step, you will write a README that explains your system, then push everything to GitHub so anyone can clone and run it.

In this step, get ready to:
  • Write a professional README with architecture, setup instructions, and eval results.
  • Commit all project files and push to a GitHub repository.
  • Verify the README renders correctly in your browser.
Write the README header and architecture diagram

A good README answers three questions in the first few seconds: what is this, how does it work, and how do I use it. You will build yours section by section.

  • Create a new file called README.md in your rag-eval-pipeline folder.
  • Paste the following content as the beginning of the file:
# RAG Evaluation Pipeline

A document Q&A system powered by Gemini and ChromaDB that answers questions from a custom knowledge base, with an automated evaluation pipeline that scores answer quality.

## Architecture

```
┌─────────────┐     ┌──────────┐     ┌─────────┐
│ Knowledge   │────>│ ChromaDB │────>│ Gemini  │
│ Base (.md)  │     │ (Vectors)│     │  (LLM)  │
└─────────────┘     └──────────┘     └─────────┘
    ingest.py         query.py        evaluate.py
                                           │
                                    ┌──────v──────┐
                                    │ Eval Report │
                                    │   (JSON)    │
                                    └─────────────┘
```

What does this section do?

The title and one-sentence description let anyone scanning GitHub repos instantly understand the project's purpose.

The text-based architecture diagram shows the data flow without requiring any external image hosting. Markdown renders it as a code block on GitHub.

  • Save README.md.
  • Open the markdown preview by pressing Ctrl+K V (Windows) or Cmd+K V (macOS).
  • You should see your project title rendered as a large heading and the architecture diagram displayed inside a grey code block.
Add setup instructions

Someone cloning your repo needs to know exactly how to get it running. The setup section walks them through every step from clone to first query.

  • In README.md, add the following below the architecture diagram:
## Setup

1. Clone the repository:
   ```bash
   git clone <your-repo-url>
   cd rag-eval-pipeline
   ```

2. Create and activate a virtual environment:
   ```bash
   python -m venv venv
   venv\Scripts\activate  # Windows
   # source venv/bin/activate  # macOS/Linux
   ```

3. Install dependencies:
   ```bash
   pip install -r requirements.txt
   ```

4. Create a `.env` file with your Gemini API key (get one free at [Google AI Studio](https://aistudio.google.com)):
   ```
   GEMINI_API_KEY=your-key-here
   ```

What does this section do?

Each numbered step mirrors what someone would actually run in their terminal. Including the virtual environment activation and the .env file creation means nobody gets stuck on missing dependencies or authentication errors.

  • Save README.md and check the preview panel.
  • You should see a numbered list with code blocks inside each step, rendered with proper syntax highlighting.
Add usage examples and eval results

The final sections show what the project actually does when you run it. This gives readers confidence that the code works and shows them what to expect.

  • In README.md, add the following after the Setup section:
## Usage

```bash
python ingest.py                         # Ingest knowledge base documents
python query.py "Your question here"     # Query the system
python evaluate.py                       # Run evaluation suite
```

## Sample Output

```
Question: How do I reset my API key?
Answer: Navigate to Settings > API Keys, click the three-dot menu next to
your key, select "Regenerate". The old key is immediately invalidated.
```

## Evaluation Results

The eval pipeline scores answers on **faithfulness** (supported by context?) and **relevance** (addresses the question?).

| Metric | Score |
|--------|-------|
| Average Faithfulness | ~4.5/5 |
| Average Relevance | ~4.3/5 |
| Overall Score | ~4.4/5 |

The system correctly refuses unanswerable questions rather than hallucinating.

What does this section do?

The Usage section gives quick-start commands anyone can copy. The Sample Output proves the system works by showing a real answer. The Evaluation Results table summarizes your pipeline's quality scores.

The scores use approximate values (~4.5/5) because your actual results will vary slightly each time Gemini responds. Update them with the real numbers from your eval_report.json if you like.

  • Save README.md and check the preview panel.
  • You should see the Usage code block, a Sample Output section, and a formatted table showing evaluation metrics.

Preview not updating?

Make sure you saved the file (Ctrl+S or Cmd+S). The preview only refreshes after a save. If the preview panel closed, reopen it with Ctrl+K V (Windows) or Cmd+K V (macOS).

Help me fix my VS Code markdown preview

✔️ Awesome, I've got everything!

Great. Double check you have saved README.md and the preview shows all five sections: title, Architecture, Setup, Usage/Sample Output, and Evaluation Results.

ⓧ I'd like to double check the full code

Your README.md should contain three sections you pasted in order. Scroll up and confirm you have:

  • The project title, one-sentence description, and architecture diagram (first chunk).
  • The Setup section with 4 numbered steps (second chunk).
  • The Usage, Sample Output, and Evaluation Results sections (third chunk).

If any section is missing or duplicated, delete the affected lines and re-paste from the code blocks above.

Commit and push to GitHub

Your project files are ready. Time to commit everything and push to a remote repository so others can find and clone it.

  • Stage and commit all your project files by running these commands in the terminal:
git add .
git commit -m "Add RAG pipeline with evaluation"

What does this do?

git add . stages every file in the project folder (except those listed in .gitignore). Your .gitignore already excludes venv/, .env, __pycache__/, and .chroma/ so none of those will be pushed.

  • You should see output listing the files being committed. Confirm that .env and venv/ are NOT in the list.

Seeing .env in the committed files?

If .env appears in the commit output, your .gitignore may not be set up correctly. Run git rm --cached .env to unstage it, then verify your .gitignore contains the line .env and commit again.

Help me remove .env from my git history

Now create a new repository on GitHub to push your code to.

  • Open your browser and go to github.com/new.
  • Enter rag-eval-pipeline as the repository name.
  • Leave **Add a README file** unchecked (you already wrote one).
  • Click Create repository.
  • GitHub will show a page with setup commands. Copy the HTTPS URL displayed at the top (it looks like https://github.com/yourusername/rag-eval-pipeline.git).
  • Back in your terminal, connect your local repo to GitHub and push by running:
git branch -M main
git remote add origin [[REPO_URL="https://github.com/yourusername/rag-eval-pipeline.git"]]
git push -u origin main

What do these commands do?

  • git branch -M main renames your default branch to main (the GitHub standard).
  • git remote add origin tells Git where to push (your GitHub repo URL).
  • git push -u origin main uploads all committed files to GitHub and sets main as the default push target.
  • You should see output ending with a line like Branch 'main' set up to track remote branch 'main' from 'origin'. confirming the push succeeded.

Push rejected or authentication failed?

GitHub requires authentication for pushes. If prompted, sign in with your GitHub credentials or a personal access token. If you see a "permission denied" error, confirm you are logged in to the correct GitHub account.

Help me authenticate with GitHub from the terminal

Verify the README renders on GitHub

Before you run this final check, what do you expect to see when you open your repo in the browser? Think about whether the architecture diagram, the numbered setup steps, and the evaluation table will all render correctly on GitHub.

  • Open your browser and navigate to https://github.com/yourusername/rag-eval-pipeline (replace with your actual GitHub username).
  • Scroll down to the rendered README below the file list.

You should see your README rendered with a large title, the text-based architecture diagram inside a grey code block, numbered setup instructions with embedded code snippets, a usage section, sample output, and a formatted evaluation results table.

Update with your real scores

Open eval_report.json and replace the approximate scores in the Evaluation Results table with your actual avg_faithfulness and avg_relevance values. Then commit and push the update so your README reflects real data.

Your RAG pipeline is now live on GitHub. Anyone can clone it, follow your setup instructions, and run the evaluation. Next up, you will add a polished CLI to make the whole thing demoable with a single command.

Secret mission

Add a CLI Demo Mode

Right now, anyone who clones your repo has to know which script to run and what arguments it expects. Build a unified CLI so a single command handles ingestion, querying, and evaluation. One entry point, instant demo.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally except for Gemini API calls, which use the free tier. There are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Local project folder rag-eval-pipeline on your Desktop (includes all Python scripts, knowledge base, virtual environment, and ChromaDB data).
  • ChromaDB vector database stored in the .chroma/ folder inside your project (free, local storage only).
  • Gemini API key (free tier, no charges).
  • GitHub repository with your project code.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend the project soon.

  • Your ChromaDB data persists in .chroma/ between runs. No server process is required.
  • Your Gemini API key remains on the free tier with no billing account attached.
  • Your virtual environment stays in the venv/ folder. Reactivate it any time with venv\Scripts\activate (Windows) or source venv/bin/activate (macOS/Linux).

Pause - I'll come back to this later

Shut down any running processes to free up memory. Your files and data stay intact on disk.

  • If your virtual environment is still active, deactivate it by running:
deactivate
  • Your ChromaDB data, project files, and API key all remain in place. Nothing else to pause.
  • When you return, navigate back to your project folder and reactivate the virtual environment.

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

  • Deactivate your virtual environment if it is still active:
deactivate
  • Delete the entire project folder by running the command for your operating system.

On macOS/Linux:

cd ~/Desktop
rm -rf rag-eval-pipeline

On Windows (Command Prompt):

cd %USERPROFILE%\Desktop
rmdir /s /q rag-eval-pipeline
  • This removes all Python scripts, the knowledge_base/ folder, the .chroma/ database, the venv/ virtual environment, and your local .env file.
  • Optionally, revoke your Gemini API key. Open Google AI Studio at aistudio.google.com, navigate to your API keys, find the key you used for this project, and delete it.
  • Your GitHub repository remains online. If you also want to remove it, go to your repo's Settings page, scroll to the bottom under Danger Zone, and select Delete this repository.

Nice Work!

Nice Work!

You just built a production-style RAG system with built-in quality measurement. This is the exact pattern used to deploy trustworthy AI systems over internal documents.

What you learned:

  • Built a complete RAG pipeline that ingests documents into ChromaDB, retrieves relevant context via vector search, and generates grounded answers with Gemini.
  • Identified real failure modes in naive RAG by testing unanswerable questions and observing hallucination and irrelevant retrieval firsthand.
  • Built an automated evaluation pipeline using the LLM-as-a-judge pattern to score every answer on faithfulness and relevance.
  • Secret Mission: Added a unified CLI with argparse so anyone who clones your repo can demo the full system with a single command.

Ready to quiz yourself?