Build a RAG Pipeline with Python

Build a Q&A bot that answers questions grounded in your own documents.

Introduction

30 Second Summary

You've probably asked an AI chatbot a question and gotten a confident-sounding answer that turned out to be completely wrong. This happens because the model generates text from patterns it learned during training, not from any real source of truth.

In this project, you will build a RAG (Retrieval-Augmented Generation) pipeline that grounds an AI's answers in your own documents. You'll create a command-line Q&A bot that reads your study notes, stores them in a local vector database, and answers questions using only the information it retrieves.

What You'll Build

Ask your bot a question about your study notes and watch it pull the most relevant passages, feed them to Gemini as context, and return a grounded answer with source citations.

By the end of this project, you'll have:

  • A command-line Q&A bot that answers natural-language questions about your documents with accurate, sourced answers.
  • A local ChromaDB vector database storing your embedded document chunks that you can search by meaning instead of keywords.
  • A clear view of why retrieval matters by comparing the same question answered with and without context side by side.
  • Secret Mission: Add a multi-document filter so you can ask questions about specific topics only.

Do I need any AI or Python experience?

No prior experience with AI, embeddings, or vector databases is needed. This project walks you through every step from scratch, including installing all required tools.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Python Environment and Gemini API Key

Every RAG pipeline has two dependencies: an embedding model to convert text into vectors, and a generation model to produce answers. Both live behind the Gemini API.

In this step, you'll install the Python packages that talk to Gemini and ChromaDB, grab a free API key, and prove everything works by generating your first embedding.

In this step, get ready to:
  • Install google-genai and chromadb in a virtual environment.
  • Get a free Gemini API key from Google AI Studio.
  • Verify the setup by generating a test embedding.
Create your project folder and check Python

You need a dedicated folder to keep your RAG project files together, and Python 3.10 or higher to run the packages you'll install.

  • Open VS Code's integrated terminal by clicking Terminal in the top menu bar, then clicking New Terminal.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-pipeline
cd rag-pipeline
  • Confirm the folder was created by running:
pwd

You should see a path ending in rag-pipeline.

  • Check your Python version by running:
python3 --version

✔️ I see version 3.10 or higher

You're all set. Move on to the next instruction below.

ⓧ I see an older version

Your system has a Python version below 3.10. The google-genai package requires Python 3.10 or higher.

On macOS, the system Python is often outdated. Install a newer version using Homebrew or download directly from python.org.

  • Download Python 3.10+ from python.org/downloads and run the installer.
  • After installing, close and reopen your terminal.
  • Run python3 --version again to confirm you see 3.10 or higher.

macOS default Python

macOS often ships with an older Python that shadows newer installs. If python3 --version still shows the old version after installing, use the full path (e.g. /usr/local/bin/python3.10) or install via pyenv to manage versions.

ⓧ Command not found

Python is not installed on your machine. Install it now.

  • Download Python 3.10+ from python.org/downloads and run the installer.
  • On Windows, check the box labeled Add Python to PATH during installation.
  • Close and reopen your terminal after installing.
  • Run python3 --version (or python --version on Windows) to confirm you see 3.10 or higher.
Install packages in a virtual environment

A virtual environment keeps your project's packages isolated from the rest of your system. This prevents version conflicts between projects.

  • Create a virtual environment by running:
python3 -m venv venv
  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

  • Install the two packages your project needs by running:
pip install google-genai chromadb

What are these packages?

The google-genai package is Google's official Python SDK for the Gemini API. It handles embedding text and generating responses.

The chromadb package is a local vector database. It stores your embeddings on disk and lets you search them by meaning instead of keywords.

The install may take a minute. When it finishes, you should see a success message listing the installed packages.

Seeing install errors?

Make sure your virtual environment is active (you see (venv) in your prompt). If not, re-run the activate command above.

If you see a permissions error, do not use sudo. Deactivate, delete the venv folder, and re-create it.

Help me fix a pip install error in my virtual environment.

Get a free Gemini API key

Your Python packages need an API key to authenticate with Google's servers. The Gemini API free tier provides both embeddings and text generation at no cost.

  • Open your browser and go to aistudio.google.com.
  • Sign in with your Google account if prompted.
  • Click Get API key in the left sidebar to open the API keys page.
  • Click Create API key.
  • Select a project when prompted (the default project is fine for new users).
  • Copy your new API key. You will only see it once, so keep it somewhere safe temporarily.

Why is this free?

Google offers a free tier for the Gemini API that includes both embedding and text generation. No billing setup is required. You get generous rate limits for prototyping and learning.

Now set the key as an environment variable so your Python scripts can access it.

  • In your terminal (still inside the rag-pipeline folder with (venv) active), run the export command with your copied key:

macOS

export GEMINI_API_KEY=your-api-key-here

Windows

set GEMINI_API_KEY=your-api-key-here
  • Replace your-api-key-here with the key you copied from Google AI Studio.

Why an environment variable?

Storing your key in an environment variable keeps it out of your code files. This means you won't accidentally share it if you push code to GitHub. The GEMINI_API_KEY name is the standard variable that ChromaDB's embedding function reads automatically.

This key only lasts for your terminal session

The export command sets the variable for your current terminal session only. If you close the terminal, you'll need to re-run the export command next time.

Verify your setup

Time to prove everything is connected. You'll write a short script that calls the Gemini embedding API and prints the result.

  • Create a new file called verify_setup.py in your rag-pipeline folder. In VS Code, click File then New File.
  • Save it immediately as verify_setup.py using Cmd+S (macOS) or Ctrl+S (Windows).
  • Paste the following code into verify_setup.py:
import os
from google import genai
from google.genai import types

# Create a client using our API key from the environment
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

# Generate an embedding for a test sentence
result = client.models.embed_content(
    model="gemini-embedding-001",
    contents="Hello, this is a test embedding!",
    config=types.EmbedContentConfig(output_dimensionality=768),
)

# Print the results to confirm everything works
embedding = result.embeddings[0].values
print(f"Embedding generated successfully!")
print(f"Dimensions: {len(embedding)}")
print(f"First 5 values: {embedding[:5]}")

What does this code do?

It creates a Gemini client using your API key, sends the text "Hello, this is a test embedding!" to the gemini-embedding-001 model, and asks for a 768-dimension vector back.

If the API key and packages are set up correctly, it prints the vector's length and first few values. This proves your entire chain (packages, key, network) works.

  • Save verify_setup.py.

Before you run this, what do you expect to see? Think about what a 768-dimension embedding would look like when printed.

  • Run the verification script:
python verify_setup.py

You should see output like this:

Embedding generated successfully!
Dimensions: 768
First 5 values: [0.0123, -0.0456, 0.0789, ...]

The exact numbers will differ, but you should see Dimensions: 768 confirming the embedding was generated with the correct size.

✔️ Awesome, I've got everything!

Great. Double check you've saved verify_setup.py and move on to the next step.

ⓧ I'd like to double check the full code

Here is the complete verify_setup.py file:

import os
from google import genai
from google.genai import types

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

result = client.models.embed_content(
    model="gemini-embedding-001",
    contents="Hello, this is a test embedding!",
    config=types.EmbedContentConfig(output_dimensionality=768),
)

embedding = result.embeddings[0].values
print(f"Embedding generated successfully!")
print(f"Dimensions: {len(embedding)}")
print(f"First 5 values: {embedding[:5]}")

Seeing an error when you run the script?

If you see KeyError: 'GEMINI_API_KEY', your environment variable is not set. Re-run the export (or set on Windows) command from the previous substep.

If you see ModuleNotFoundError, your virtual environment may not be active. Check for (venv) in your terminal prompt and re-activate if needed.

If you see a permissions or authentication error, double-check that you pasted the full API key with no extra spaces.

Help me debug my verify_setup.py script.

Your environment is ready. Next up, you'll load real documents, split them into chunks, and prepare them for embedding.

Load and Chunk Your Documents

In Step 1 you proved your environment works by generating a single embedding. Now you need documents to feed your RAG pipeline.

Raw documents are too long to embed as a single block. If you embed a whole file, your search will either return everything or nothing useful. Chunking splits documents into smaller overlapping pieces so retrieval can pinpoint the exact passage that answers a question.

In this step, get ready to:
  • Create sample markdown study notes for the pipeline to ingest.
  • Write a chunking function that splits text into 500-character overlapping pieces.
  • Run the script and confirm chunks are created from each file.
Create your study notes

Your RAG pipeline needs documents to search through. You will create three short markdown files covering Python topics. These act as the knowledge base your bot will answer questions from.

  • Create a notes/ folder inside your project directory by running this command:
mkdir notes
  • Create a file called python-basics.md inside notes/ and paste in the first section of content:
# Python Basics

## What is Python?

Python is a high-level, interpreted programming language created by Guido van Rossum and first released in 1991. It emphasizes code readability with its use of significant indentation. Python supports multiple programming paradigms including procedural, object-oriented, and functional programming.

## Variables

A variable in Python is a name that refers to a value stored in memory. Unlike some other languages, Python variables do not need explicit type declarations. You create a variable by assigning a value to a name using the equals sign.

For example: `message = "Hello, World!"` creates a variable called message that holds a string value.

Python uses dynamic typing, which means the type of a variable is determined at runtime based on the value assigned to it. You can reassign a variable to a different type at any time.
  • Below the Variables section in the same file, paste the remaining content:
## Data Types

Python has several built-in data types:
- Integers (int): whole numbers like 42, -7, 0
- Floats (float): decimal numbers like 3.14, -0.5
- Strings (str): text enclosed in quotes like "hello"
- Booleans (bool): True or False
- Lists (list): ordered, mutable collections like [1, 2, 3]
- Dictionaries (dict): key-value pairs like {"name": "Alice", "age": 30}

## Print Function

The print() function outputs text to the console. It can accept multiple arguments separated by commas, and it adds a newline at the end by default.

Examples:
- print("Hello") outputs: Hello
- print("Age:", 25) outputs: Age: 25
- print(1, 2, 3, sep="-") outputs: 1-2-3
  • Save python-basics.md.
  • Confirm the file exists by running:
ls notes/

You should see python-basics.md listed.

  • Create a file called data-structures.md inside notes/ and paste the first section:
# Data Structures in Python

## Lists

A list is an ordered, mutable collection of items. Lists are created using square brackets and can contain items of different types.

Creating lists:
- empty_list = []
- numbers = [1, 2, 3, 4, 5]
- mixed = [1, "hello", True, 3.14]

Common list operations:
- append(item): adds an item to the end
- insert(index, item): adds an item at a specific position
- remove(item): removes the first occurrence of an item
- pop(index): removes and returns the item at the given index
- len(list): returns the number of items
- sort(): sorts the list in place
  • Below the Lists section in the same file, paste the Dictionaries section:
## Dictionaries

A dictionary is an unordered collection of key-value pairs. Keys must be immutable (strings, numbers, tuples) and unique. Values can be any type.

Creating dictionaries:
- empty_dict = {}
- person = {"name": "Alice", "age": 30, "city": "New York"}

Common dictionary operations:
- dict[key]: access a value by key
- dict[key] = value: add or update a key-value pair
- dict.get(key, default): safely access a value with a fallback
- dict.keys(): returns all keys
- dict.values(): returns all values
- dict.items(): returns all key-value pairs as tuples
  • Below the Dictionaries section, paste the Tuples and Sets sections:
## Tuples

A tuple is an ordered, immutable collection. Once created, you cannot modify its contents. Tuples are created using parentheses.

- coordinates = (10, 20)
- rgb = (255, 128, 0)
- single = (42,)  # note the trailing comma for single-item tuples

Tuples are useful when you want to ensure data cannot be accidentally modified, or as dictionary keys.

## Sets

A set is an unordered collection of unique items. Sets automatically remove duplicates and support mathematical set operations.

- numbers = {1, 2, 3, 4, 5}
- unique_letters = set("hello")  # {'h', 'e', 'l', 'o'}

Set operations:
- union: set1 | set2
- intersection: set1 & set2
- difference: set1 - set2
  • Save data-structures.md.
  • Create a file called functions.md inside notes/ and paste the first section:
# Functions in Python

## What is a Function?

A function is a reusable block of code that performs a specific task. Functions help organize code, reduce repetition, and make programs easier to understand and maintain.

## Defining Functions

You define a function using the def keyword, followed by the function name, parentheses for parameters, and a colon. The function body is indented.

def greet(name):
    return f"Hello, {name}!"

## Parameters and Arguments

Parameters are the variables listed in a function definition. Arguments are the actual values passed when calling the function.

Types of parameters:
- Positional: def add(a, b) - must be provided in order
- Default: def greet(name, greeting="Hello") - has a fallback value
- Keyword: called as greet(name="Alice") - specified by name
- *args: accepts any number of positional arguments as a tuple
- **kwargs: accepts any number of keyword arguments as a dictionary
  • Below the Parameters section in the same file, paste the remaining content:
## Return Values

Functions can return values using the return statement. A function without a return statement returns None by default.

def add(a, b):
    return a + b

result = add(3, 5)  # result is 8

Functions can return multiple values as a tuple:
def get_stats(numbers):
    return min(numbers), max(numbers), sum(numbers) / len(numbers)

## Scope

Variables defined inside a function are local to that function. They cannot be accessed from outside. Variables defined outside all functions are global and can be read (but not modified without the global keyword) inside functions.

## Lambda Functions

Lambda functions are small, anonymous functions defined with the lambda keyword. They can have any number of arguments but only one expression.

square = lambda x: x ** 2
add = lambda a, b: a + b

Lambdas are often used with functions like map(), filter(), and sorted().
  • Save functions.md.
  • Confirm all three files exist by running:
ls notes/

You should see data-structures.md, functions.md, and python-basics.md listed.

Why markdown files?

Markdown is plain text with light formatting. It is easy for a chunking function to process without needing HTML parsers or PDF libraries. In production, you would add loaders for other file types, but the RAG pipeline logic stays the same.

Write the chunking and loading script

Now you will create the Python script that reads those notes, splits them into overlapping chunks, and prints what it found. This is the first half of the RAG pipeline: turning raw documents into searchable pieces.

  • Create a new file called rag.py in your project folder and paste the imports and configuration:
import os
import sys
import glob

import chromadb
from chromadb.utils.embedding_functions import GoogleGeminiEmbeddingFunction
from google import genai
from google.genai import types

GEMINI_API_KEY = os.environ["GEMINI_API_KEY"]
NOTES_DIR = "notes"
CHROMA_DIR = ".chroma"
COLLECTION_NAME = "study_notes"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 100
TOP_K = 3

client = genai.Client(api_key=GEMINI_API_KEY)

google_ef = GoogleGeminiEmbeddingFunction(
    model_name="gemini-embedding-001",
    task_type="RETRIEVAL_DOCUMENT",
    dimension=768,
)

chroma_client = chromadb.PersistentClient(path=CHROMA_DIR)

What does this code do?

This sets up the building blocks your script needs. CHUNK_SIZE = 500 means each chunk will be at most 500 characters. CHUNK_OVERLAP = 100 means chunks share 100 characters with their neighbour so sentences at the boundary are not cut off.

The GoogleGeminiEmbeddingFunction tells ChromaDB to use Gemini for embedding automatically when you add or query documents later.

  • Below the configuration block in rag.py, paste the chunking function:
def chunk_text(text, size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    if len(text) <= size:
        return [text]
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start += size - overlap
    return chunks

What does this code do?

The chunk_text() function slides a 500-character window across the text, advancing by 400 characters each time (500 minus the 100-character overlap). This means every chunk shares its last 100 characters with the start of the next chunk.

Overlap prevents retrieval misses. If a key sentence sits at the boundary between two chunks, the overlap guarantees at least one chunk contains it in full.

  • Below chunk_text(), paste the document loading function:
def load_documents():
    """Load all markdown files from the notes directory."""
    files = glob.glob(os.path.join(NOTES_DIR, "*.md"))
    if not files:
        print(f"No .md files found in '{NOTES_DIR}/' directory.")
        sys.exit(1)

    all_chunks = []
    for filepath in files:
        with open(filepath, encoding="utf-8") as f:
            text = f.read()
        filename = os.path.basename(filepath)
        topic = filename.replace(".md", "")
        chunks = chunk_text(text)
        for i, chunk in enumerate(chunks):
            all_chunks.append(
                {
                    "id": f"{filename}::{i}",
                    "text": chunk,
                    "metadata": {"source": filename, "topic": topic, "chunk_index": i},
                }
            )
        print(f"  {filename}: {len(chunks)} chunks")

    print(f"\nLoaded {len(files)} files, created {len(all_chunks)} chunks total.")
    print(f"\nSample chunk:\n---\n{all_chunks[0]['text'][:200]}...\n---")
    return all_chunks

What does this code do?

This function finds every .md file in the notes/ folder, reads it, and passes the text through chunk_text(). Each chunk gets an ID, the raw text, and metadata recording which file and topic it came from.

The topic field is derived from the filename (e.g. python-basics from python-basics.md). You will use this metadata for filtering in a later step.

  • At the bottom of rag.py, paste the entry point that wires the load command to load_documents():
def main():
    if len(sys.argv) < 2:
        print("Usage: python rag.py <command> [arguments]")
        print("Commands: load, embed, search, ask, ask-no-context, chat")
        sys.exit(1)

    command = sys.argv[1]

    if command == "load":
        load_documents()
    else:
        print(f"Unknown command: {command}")
        sys.exit(1)


if __name__ == "__main__":
    main()

What does this code do?

The main() function reads the first argument from the command line and dispatches to the matching function. Right now only load is supported. You will add more commands in later steps.

  • Save rag.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved rag.py and move on to the next substep.

ⓧ I'd like to double check the full code

Your rag.py should contain these sections in order. Scroll up and confirm each is present:

  • Imports and configuration constants (GEMINI_API_KEY through chroma_client).
  • The chunk_text() function.
  • The load_documents() function.
  • The main() function and if __name__ == "__main__": block at the bottom.
Test the chunking pipeline

Everything is wired up. Before you run it, think about what you expect to see. You have 3 markdown files of different lengths being split into 500-character chunks with 100-character overlap.

  • How many chunks do you think 3 short markdown files will produce? Make a guess before running the command.
  • Run the load command by running:
python rag.py load

You should see output listing each file with its chunk count, a total, and a sample chunk. Something like:

  python-basics.md: 3 chunks
  data-structures.md: 4 chunks
  functions.md: 4 chunks

Loaded 3 files, created 11 chunks total.

Sample chunk:
---
# Python Basics

## What is Python?

Python is a high-level, interpreted programming language created by Guido van Rossum and first released in 1991. It emphasizes code readability with its use of significant indentation. Python supports multiple prog...
---

Why overlapping chunks?

Without overlap, a sentence that starts at character 490 and ends at character 520 would be split across two chunks. Neither chunk would contain the full sentence. With 100-character overlap, the second chunk starts at character 400, capturing that sentence completely.

This is why your pipeline retrieves more relevant results. The overlapping window guarantees that information is never lost at chunk boundaries.

Seeing "No .md files found"?

Make sure you are running the command from the same directory that contains the notes/ folder. Run ls and confirm you see both notes/ and rag.py in the listing.

If the files are in the wrong location, move them into a notes/ subfolder relative to where rag.py lives.

Help me debug why my rag.py can't find the notes folder.

Your documents are chunked and ready. Next up, you will embed these chunks into vectors and store them in a ChromaDB database so you can search them by meaning instead of keywords.

Embed and Store Chunks in ChromaDB

In the last step, you split your study notes into manageable chunks. Right now those chunks are just plain text strings sitting in memory.

To search them by meaning, you need to convert each chunk into a vector embedding and store it in a vector database. In this step, you will embed all chunks into ChromaDB and then query the collection to prove that semantic search works.

In this step, get ready to:
  • Embed all document chunks into a persistent ChromaDB collection.
  • Search the collection with a test question to verify semantic retrieval.
Write the embedding function

Your script already initializes a GoogleGeminiEmbeddingFunction and a PersistentClient at the top of the file. Now you need a function that uses them to create a collection, embed every chunk, and store the results.

  • In rag.py, find the end of the load_documents() function. Add the following embed_documents() function below it:
def embed_documents():
    """Embed and store all document chunks in ChromaDB."""
    chunks = load_documents()

    # Delete existing collection if it exists so we start fresh
    try:
        chroma_client.delete_collection(name=COLLECTION_NAME)
    except Exception:
        pass

    # Create a new collection with the Gemini embedding function
    collection = chroma_client.create_collection(
        name=COLLECTION_NAME,
        embedding_function=google_ef,
    )

    # Add chunks in batches of 20 to avoid API rate limits
    batch_size = 20
    for i in range(0, len(chunks), batch_size):
        batch = chunks[i : i + batch_size]
        collection.add(
            ids=[c["id"] for c in batch],
            documents=[c["text"] for c in batch],
            metadatas=[c["metadata"] for c in batch],
        )

    print(f"\nEmbedded and stored {collection.count()} chunks in ChromaDB.")

What does this code do?

  • chroma_client.delete_collection() wipes any previous version of the collection so you always start fresh when re-embedding.
  • create_collection() makes a new collection and attaches google_ef as its embedding function. This means ChromaDB will automatically call Gemini to generate embeddings whenever you add documents or run a query.
  • collection.add() sends each batch of chunks to ChromaDB. It passes the text, the IDs, and the metadata (source file, topic, chunk index) together.

Now wire this function into the CLI so you can run it. In the main() function, find the line:

    if command == "load":
        load_documents()
    else:
        print(f"Unknown command: {command}")
        sys.exit(1)
  • Replace the else: block with an elif for the embed command, so the section now looks like this:
    if command == "load":
        load_documents()
    elif command == "embed":
        embed_documents()
    else:
        print(f"Unknown command: {command}")
        sys.exit(1)
  • Save rag.py.
  • Run the embed command:
python rag.py embed

You should see the file-loading output from before, followed by a line like Embedded and stored 18 chunks in ChromaDB. The exact number depends on how your notes split, but it should be around 15-20.

Seeing a rate limit or API error?

  • Check that your GEMINI_API_KEY environment variable is still set. Run echo $GEMINI_API_KEY to confirm it prints your key.
  • If you see a 429 rate limit error, wait 60 seconds and try again. The free tier has per-minute limits.

Help me debug my ChromaDB embedding error

Write the search function

Your chunks are now stored as vectors. The next piece is a function that takes a plain-English question, embeds it the same way, and finds the closest matching chunks by distance.

  • In rag.py, add the following two functions below embed_documents():
def search(query, topic=None):
    """Search for relevant chunks."""
    collection = chroma_client.get_collection(
        name=COLLECTION_NAME,
        embedding_function=google_ef,
    )

    # Build an optional metadata filter
    where_filter = {"topic": topic} if topic else None

    # Query returns the closest chunks by embedding distance
    results = collection.query(
        query_texts=[query],
        n_results=TOP_K,
        where=where_filter,
        include=["documents", "metadatas", "distances"],
    )

    return results


def print_search_results(results):
    """Display search results."""
    for i in range(len(results["ids"][0])):
        doc = results["documents"][0][i]
        meta = results["metadatas"][0][i]
        dist = results["distances"][0][i]
        print(f"\n[{i + 1}] (distance: {dist:.4f}) source: {meta['source']}")
        print(f"    {doc[:150]}...")

What does this code do?

  • search() opens the existing collection, embeds your query using the same Gemini model, and finds the top TOP_K (3) closest chunks. The where_filter is optional metadata filtering you will use in the Secret Mission.
  • print_search_results() loops through the returned results and prints each chunk's distance score, source file, and a 150-character preview. Lower distance means a closer semantic match.

What is a distance score?

ChromaDB measures how similar two embeddings are by calculating the distance between them. A score of 0 means a perfect match. A score above 1.0 typically means the result is not very relevant to your query.

Now wire the search command into main(). Find the section you just edited:

    elif command == "embed":
        embed_documents()
    else:
        print(f"Unknown command: {command}")
        sys.exit(1)
  • Add a new elif for the search command between embed and else, so it looks like this:
    elif command == "embed":
        embed_documents()
    elif command == "search":
        if len(sys.argv) < 3:
            print("Usage: python rag.py search \"your query\"")
            sys.exit(1)
        query = sys.argv[2]
        results = search(query)
        print_search_results(results)
    else:
        print(f"Unknown command: {command}")
        sys.exit(1)
  • Save rag.py.

Before you run the search, what do you think will come back for the query "what is a variable"? Which of your three note files do you expect to score closest?

  • Test semantic search by running:
python rag.py search "what is a variable"

You should see 3 results printed, each with a distance score, source file, and a text preview. The top results should come from python-basics.md since that file has a section dedicated to variables.

Seeing "Collection not found" or no results?

  • Make sure you ran python rag.py embed first. The search command reads from the stored collection. If you skipped the embed step, there is nothing to search.
  • Check that COLLECTION_NAME is "study_notes" in both the embed and search functions.
  • Confirm a .chroma/ folder exists in your project directory. Run ls -a to check.

Help me debug my ChromaDB search query

✔️ Awesome, I've got everything!

Great. Double check you have saved rag.py and that both the embed and search commands are working.

ⓧ I'd like to double check the full code

Your rag.py is now over 100 lines. Scroll through and confirm these sections are present in order:

  • Imports and constants at the top (unchanged from before).
  • The chunk_text() function.
  • The load_documents() function.
  • The new embed_documents() function that creates a collection and adds chunks in batches.
  • The new search(query, topic=None) function that queries the collection.
  • The new print_search_results(results) function that displays results.
  • The main() function with load, embed, and search commands.

Your vector database is loaded and searchable. Next up, you will feed those retrieved chunks to Gemini and generate real answers to your questions.

Generate Answers with Retrieved Context

Your search() function finds the most relevant chunks for a question. But raw text snippets aren't a complete, readable answer.

The generation step is where an LLM synthesises those retrieved chunks into a coherent response. In this step, you'll build that final piece and see firsthand what happens when the LLM has no context versus when it has your retrieved chunks to ground its answer.

In this step, get ready to:
  • Write a function that retrieves top-k chunks and formats them into a numbered context block for the LLM.
  • Build a generate function that sends context plus question to Gemini with a constraining system prompt.
  • Compare answers with and without retrieval to see why RAG matters.
Format retrieved chunks into numbered context

Before sending anything to Gemini, you need to format the raw search results into a clean, numbered context block that the model can reference.

  • In rag.py, find the print_search_results() function. Add the following new function below it:
def generate_answer(query, topic=None):
    """Retrieve context and generate an answer."""
    results = search(query, topic=topic)

    context = "\n\n".join(
        [
            f"[{i + 1}] (source: {results['metadatas'][0][i]['source']})\n{results['documents'][0][i]}"
            for i in range(len(results["ids"][0]))
        ]
    )

    sources = "\n".join(
        [
            f"[{i + 1}] {results['metadatas'][0][i]['source']}"
            for i in range(len(results["ids"][0]))
        ]
    )

What does this code do?

  • search(query, topic=topic) retrieves the top-k most relevant chunks from ChromaDB for the given question.
  • context joins the retrieved documents into a numbered list so the LLM can cite specific snippets by number.
  • sources builds a separate reference list showing which file each chunk came from.
Send context to Gemini and generate an answer

Now you need to send those formatted chunks to Gemini with a system prompt that constrains its answer to only the provided context.

  • Still inside the generate_answer function, add this code directly below the sources variable you just wrote:
    system_prompt = (
        "You answer questions using ONLY the numbered context snippets below. "
        "If the context does not contain enough information to answer the question, "
        "respond with: I don't have enough information in the provided context to answer that. "
        "When you use information from a snippet, cite it by number like [1]."
    )

    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=f"Context:\n{context}\n\nQuestion: {query}",
        config=types.GenerateContentConfig(
            system_instruction=system_prompt,
            temperature=0.0,
        ),
    )

    answer = response.text
    print(f"\nAnswer: {answer}")
    print(f"\nSources:\n{sources}")
    return answer

What does this code do?

  • system_prompt tells Gemini to ONLY use the numbered context. If the context doesn't contain the answer, it must say so.
  • temperature=0.0 makes the output as deterministic as possible. No creative embellishment.
  • The function prints the answer and lists the source files so you always know where information came from.

What is prompt engineering?

The system_prompt is a form of prompt engineering. You're shaping how the LLM behaves by giving it clear constraints.

The constraint here is critical: without it, the model would happily make up answers beyond what's in the documents. The system prompt forces it to cite sources or admit when it doesn't know.

Now you need to wire up the ask command in main() so you can run this from the terminal.

  • In the main() function, find the elif command == "search": block. Add this new command block directly below it (before the else at the bottom):
    elif command == "ask":
        if len(sys.argv) < 3:
            print("Usage: python rag.py ask \"your question\"")
            sys.exit(1)
        query = sys.argv[2]
        generate_answer(query)
  • Save rag.py.

Before you run this, what do you think the answer will look like? Will it quote your notes directly, or paraphrase them?

  • Test the grounded generation by running this command:
python rag.py ask "what is a variable in Python"

You should see a coherent answer that cites chunk numbers like [1] and [2], followed by a list of source files.

Seeing a KeyError or empty response?

  • Make sure you ran python rag.py embed in the previous step. The collection must exist before you can query it.
  • Check that your GEMINI_API_KEY environment variable is still set in this terminal session.
  • Verify the indentation of generate_answer is consistent. The system_prompt, response, and return must all be inside the function body.

Why isn't my RAG ask command returning an answer?

See what happens without retrieval

That grounded answer worked because your pipeline retrieved relevant chunks first. But what happens when the LLM gets the same constraining prompt with zero context?

  • In rag.py, add this new function directly below generate_answer:
def generate_no_context(query):
    """Generate an answer without any context to show the difference."""
    system_prompt = (
        "You answer questions using ONLY the numbered context snippets below. "
        "If the context does not contain enough information to answer the question, "
        "respond with: I don't have enough information in the provided context to answer that. "
        "When you use information from a snippet, cite it by number like [1]."
    )

    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=f"Context:\n(no context provided)\n\nQuestion: {query}",
        config=types.GenerateContentConfig(
            system_instruction=system_prompt,
            temperature=0.0,
        ),
    )

    print(f"\nAnswer (no context): {response.text}")

What does this code do?

This function uses the exact same system prompt and model as generate_answer. The only difference is that it sends (no context provided) instead of real retrieved chunks.

Because the system prompt says "answer ONLY from the context below", the model will refuse to answer when there's nothing to work with. This proves retrieval is the ingredient that unlocks useful answers.

  • In main(), add the ask-no-context command directly below the ask block you just added:
    elif command == "ask-no-context":
        if len(sys.argv) < 3:
            print("Usage: python rag.py ask-no-context \"your question\"")
            sys.exit(1)
        query = sys.argv[2]
        generate_no_context(query)
  • Save rag.py.

What do you think Gemini will say when asked about something clearly outside your notes, with no context to draw from?

  • Run this command to see the model respond without any retrieved context:
python rag.py ask-no-context "what is the capital of Mars"

You should see the model respond with: "I don't have enough information in the provided context to answer that."

Why does this matter?

This is the core of RAG in action. Without retrieval, the LLM correctly refuses because it has nothing to ground on. With retrieval, it gives a useful, cited answer from your actual documents.

In a production system without this constraint, the LLM would happily hallucinate an answer about Mars. The system prompt plus retrieved context is what keeps responses honest.

Model gave an answer instead of refusing?

  • Double check that your generate_no_context function passes (no context provided) in the contents string, not actual documents.
  • Verify the system_instruction is set correctly in the GenerateContentConfig. The constraining instruction must go in system_instruction, not in contents.

Why is my model answering instead of refusing when there's no context?

✔️ Awesome, I've got everything!

Great. Double check you've saved rag.py and both commands work before moving on.

ⓧ I'd like to double check the full code

Your rag.py is now quite long. Scroll through and confirm these sections are all present in order:

  • Imports and constants at the top (GEMINI_API_KEY, CHUNK_SIZE, TOP_K, etc).
  • Client setup: client, google_ef, chroma_client.
  • chunk_text() function.
  • load_documents() function.
  • embed_documents() function.
  • search() and print_search_results() functions.
  • generate_answer() function (new this step).
  • generate_no_context() function (new this step).
  • main() with commands: load, embed, search, ask, ask-no-context.

You now have a complete generation pipeline that turns raw chunks into grounded, cited answers. Next up, you'll wrap this into an interactive chat loop so you can explore your notes conversationally.

Build an Interactive Query Loop

Your RAG pipeline can now answer questions grounded in real documents. The ask command proved that retrieval changes answers from generic guesses to cited, accurate responses.

A one-shot command works for a single question, but real exploration needs a conversation. In this step, you'll build an interactive loop that lets you ask question after question without restarting the script each time.

In this step, get ready to:
  • Add an interactive chat_mode() function that loops until you type "quit".
  • Wire up the chat command in main() to launch the loop.
  • Test the loop with multiple questions and see grounded answers with sources each time.
Add the interactive chat function

The interactive loop needs to do three things: accept questions repeatedly, display sources alongside every answer, and handle edge cases like empty input or a keyboard interrupt.

  • In rag.py, find the generate_no_context() function.
  • Add the following chat_mode() function below it (above main()):
def chat_mode():
    """Interactive chat loop."""
    print("RAG Chat Mode - Ask questions about your study notes!")
    print("Commands: /topic <name> to filter, /topic all to reset, quit to exit\n")

    current_topic = None

    while True:
        try:
            query = input("You: ").strip()
        except (EOFError, KeyboardInterrupt):
            print("\nGoodbye!")
            break

        if not query:
            continue
        if query.lower() == "quit":
            print("Goodbye!")
            break
        if query.startswith("/topic "):
            topic_value = query[7:].strip()
            if topic_value == "all":
                current_topic = None
                print("Filter cleared. Searching all topics.\n")
            else:
                current_topic = topic_value
                print(f"Now filtering to topic: {current_topic}\n")
            continue

        generate_answer(query, topic=current_topic)
        print()

What does this code do?

  • current_topic tracks which topic filter is active. When None, the search covers all documents.
  • The try/except block handles the case where the user presses Ctrl+C or the input stream ends unexpectedly.
  • if not query: continue skips empty input so the user can press Enter without triggering an API call.
  • The /topic commands let you scope retrieval to a specific document's chunks. This calls generate_answer() with the active topic, which passes it to search() as a metadata filter.
Connect the chat command to main

The function exists but the CLI does not route to it yet. You need to add a branch in main() so that python rag.py chat launches the loop.

  • In rag.py, scroll down to the main() function.
  • Find the else block at the bottom:
    else:
        print(f"Unknown command: {command}")
        sys.exit(1)
  • Add the following elif branch directly above that else block, so the end of main() looks like this:
    elif command == "chat":
        chat_mode()
    else:
        print(f"Unknown command: {command}")
        sys.exit(1)
  • Save rag.py.
Test the interactive loop

Before you run this, what do you expect to see when you type a question about Python variables?

  • Launch the interactive chat by running:
python rag.py chat

You should see the welcome message and a You: prompt waiting for input.

  • Type a question about your study notes and press Enter:
what is a variable in Python

You should see a grounded answer citing chunk numbers like [1] and [2], followed by a sources list showing which files the answer came from.

  • Ask a second question to confirm the loop continues:
what is the difference between a list and a tuple

You should see another grounded answer pulling from the data structures notes.

  • Type quit and press Enter to exit the loop cleanly.

You should see "Goodbye!" and be returned to your normal terminal prompt.

Chat mode exits immediately or shows an error?

  • Make sure you saved rag.py after adding both chat_mode() and the elif branch in main().
  • Check that chat_mode() is defined above main(). Python reads top to bottom, so calling a function before it is defined causes a NameError.
  • Verify your GEMINI_API_KEY environment variable is still set in this terminal session.

Why is my chat mode not working?

✔️ Awesome, I've got everything!

Great. Double check you have saved rag.py with both the new chat_mode() function and the elif command == "chat" branch in main().

ⓧ I'd like to double check the full code

Scroll through your rag.py and confirm these sections are present, in order:

  • Imports and constants at the top (GEMINI_API_KEY, NOTES_DIR, CHROMA_DIR, etc.).
  • The client, google_ef, and chroma_client setup.
  • chunk_text(), load_documents(), embed_documents(), search(), print_search_results().
  • generate_answer() and generate_no_context().
  • The new chat_mode() function (with the while loop, quit handling, and /topic commands).
  • main() with all six commands: load, embed, search, ask, ask-no-context, chat.

Secret mission

Add Topic Filtering

Real-world RAG systems need to scope retrieval to specific document subsets. In this mission, you'll discover how the metadata filtering you already built enables production-style scoped queries, then verify it works end-to-end.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder on your Desktop containing rag.py, verify_setup.py, and notes/.
  • A .chroma/ directory storing your local vector database (no cost).
  • A Python virtual environment inside your project folder.
  • A Gemini API key on the free tier (no charges).

Keep everything running

No action needed. Choose this if you plan to keep exploring your RAG pipeline or want to add more notes later.

  • Your vector database persists in the .chroma/ directory. You can re-run python rag.py chat anytime without re-embedding.
  • To index new documents, drop additional .md files into the notes/ folder and run python rag.py embed again.
  • Your Gemini API key remains active on the free tier with no charges.

Pause - I'll come back to this later

Shut down any running processes to free up memory. Your files and database stay on disk for when you return.

  • If you have the interactive chat running, type quit to exit.
  • Close the terminal window you were using for this project.
  • Everything persists on disk. When you return, open a new terminal, navigate to your project folder, activate your virtual environment, and set GEMINI_API_KEY again before running commands.

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

  • Delete the entire project folder (which includes .chroma/, notes/, scripts, and the virtual environment) by running:
cd ~/Desktop
rm -rf rag-pipeline

On Windows, run:

cd ~\Desktop
Remove-Item -Recurse -Force rag-pipeline
  • Optionally, delete your Gemini API key from Google AI Studio. Go to aistudio.google.com/apikey, find your key in the list, and delete or revoke it.
  • Unset the environment variable in your current terminal session by running:
unset GEMINI_API_KEY

On Windows PowerShell, run:

Remove-Item Env:GEMINI_API_KEY

Nice Work!

Nice Work!

You just built a complete RAG pipeline from scratch. Your Q&A bot reads documents, embeds them into a vector database, retrieves the most relevant passages, and generates grounded answers with source citations.

You've learned how to:

  • Build a RAG pipeline end-to-end: chunking documents with overlap, embedding them with Gemini, storing them in ChromaDB, retrieving by semantic similarity, and generating grounded answers.
  • Write a system prompt that constrains an LLM to answer only from retrieved context, and prove it works by comparing answers with and without retrieval.
  • Build an interactive query loop that displays source references alongside every answer so users always know where information came from.
  • Secret Mission: Added metadata filtering to scope retrieval to specific document topics, a technique used in production RAG systems.

Ready to quiz yourself?