Build a RAG Document Q&A Bot

Build a RAG pipeline that answers questions from your own documents.

Introduction

30 Second Summary

AI chatbots are impressive, but they can only answer from what they were trained on. When you need an AI to answer questions about your own documents, reports, or notes, you need a way to feed it the right information at the right time.

In this project, you will build a RAG (Retrieval-Augmented Generation) system from scratch that ingests your own text files, finds the most relevant passages for any question, and generates grounded answers with source citations.

What You'll Build

A command-line Document Q&A Bot that converts your text files into searchable vectors, stores them in a local database, and answers natural language questions by retrieving relevant context before generating a response.

By the end of this project, you'll have:

  • A Python script that loads documents, splits them into chunks, and stores them as embeddings in a local vector database.
  • A retrieval pipeline that finds the most relevant chunks for any question and generates cited answers using an LLM.
  • An interactive terminal loop where you can keep asking questions and see the full retrieval and generation process in action.
  • Secret Mission: Add a reranking step that re-scores retrieved chunks for dramatically better answer quality.

Are there any prerequisites?

You'll need a beginner-level understanding of Python (variables, functions, loops) and an OpenAI API key. Step 1 walks you through installing Python and setting up your project from scratch.

Install Python and Set Up Your Project

To build a RAG system, you need a Python environment with a few key libraries. Python handles the scripting, OpenAI provides the AI models, and ChromaDB stores your document vectors locally.

In this step, you'll confirm that Python, your virtual environment, and all required packages are correctly installed and ready to use.

In this step, get ready to:
  • Verify Python 3.14 is installed and accessible from your terminal.
  • Confirm your virtual environment is active.
  • Check that your packages and API key are configured.
Verify your Python installation

Python is the language your entire RAG pipeline runs on. Let's confirm the correct version is installed and accessible from your terminal.

  • Open the terminal in VS Code by pressing Ctrl+`.
  • Check your Python version by running this command:
python --version

What should I see?

You should see Python 3.14.5 (or another 3.14.x patch version) printed in your terminal. This confirms Python is installed and your system PATH is configured correctly.

Not seeing Python 3.14?

If you see a different version or an error, make sure you checked Add python.exe to PATH during installation. You may need to close and reopen your terminal for PATH changes to take effect.

help me fix my Python installation

Confirm your virtual environment is active

A virtual environment keeps your project's packages isolated from other Python projects on your machine. Your terminal prompt should show that the environment is active.

  • Look at the beginning of your terminal prompt for (venv).
  • If you don't see (venv), activate the virtual environment by running:
venv\Scripts\activate

What is a virtual environment?

A virtual environment is an isolated space where Python installs packages just for your current project. Without one, installing packages globally can cause version conflicts between projects.

The (venv) prefix in your terminal confirms that any pip install commands will only affect this project.

Still not seeing (venv)?

Make sure you are in your project folder (the one containing the venv folder). On Windows, the activation script is at venv\Scripts\activate.

help me activate my virtual environment on Windows

Check your packages and API key

Your project depends on three packages: the OpenAI SDK for embeddings and chat, ChromaDB for vector storage, and python-dotenv for loading your API key securely.

  • Verify all three packages are installed by running:
pip list

You should see chromadb, openai, and python-dotenv in the list of installed packages.

What do these packages do?

  • openai connects to OpenAI's API for generating embeddings and chat completions.
  • chromadb is a local vector database that stores and searches your document embeddings.
  • python-dotenv loads environment variables from your .env file so your API key stays out of your code.
  • In the VS Code Explorer sidebar, confirm that a .env file exists in your project folder.
  • Open .env and verify it contains your OpenAI API key in this format:
OPENAI_API_KEY=your-api-key-here
  • Replace your-api-key-here with your actual OpenAI API key if you haven't already.
  • Save the file.

Keep your API key safe

Never commit your .env file to version control. If you plan to use Git with this project, add .env to your .gitignore file.

Missing a package?

If any of the three packages are missing from the pip list output, install them by running:

pip install openai chromadb python-dotenv

Make sure (venv) is showing in your terminal before running this command.

help me install the Python packages for my RAG project

Your development environment is confirmed and ready. Next up, you'll create your knowledge base and write the code that loads and chunks your documents.

Create Your Knowledge Base and Load Documents

Your project environment is ready with Python, your virtual environment, and all the packages installed. Now you need to give your RAG system something to search through: a set of documents containing the knowledge you want to query.

In this step, you'll create a small knowledge base of text files and write Python functions to load and chunk those documents. Chunking is critical because embeddings work best on focused passages, not entire documents.

In this step, get ready to:
  • Create a documents folder with 3 text files covering your chosen topic.
  • Write a load_documents() function that reads .txt files and tracks their source filename.
  • Write a chunk_text() function that splits documents into overlapping passages.
Create your knowledge base folder and files

A RAG system needs a source of truth. Your knowledge base is a folder of plain text files that contain the information you want your bot to answer questions about.

  • Create the documents folder by running this command in your terminal:
mkdir documents
  • Confirm the folder exists by running:
ls

You should see documents listed in the output alongside your .env file and venv folder.

Now you'll create 3 text files inside the documents folder. Each file should be 200-500 words and cover a topic of your choice.

  • In the VS Code sidebar, right-click the documents folder and select New File.
  • Name the first file something descriptive, like onboarding.txt or pasta_recipes.txt or python_basics.txt.
  • Write or paste 200-500 words of content into the file.
  • Repeat for 2 more files covering different subtopics within your chosen theme.

Picking your topic

Choose something you find interesting. Good options include a fictional company handbook (policies, benefits, team structure), a collection of recipes, or notes on a programming language.

The more factual detail you include, the better your RAG system will perform later. Avoid very short or vague content.

Don't see the documents folder in VS Code?

Make sure you ran mkdir documents from within your project folder (the one open in VS Code). If the folder appeared elsewhere, delete it and re-run the command while your terminal is inside the correct directory.

Why isn't my documents folder showing up in VS Code?

Write the document loading script

With your knowledge base ready, you need a Python script that can read those files and prepare them for embedding. You'll build this in a file called ingest.py which will eventually handle the full ingestion pipeline.

  • In the VS Code sidebar, right-click your project folder and select New File.
  • Name it ingest.py.
  • Add the following imports and initialization code to ingest.py:
import os
from openai import OpenAI
import chromadb
from dotenv import load_dotenv

# Load environment variables from .env
load_dotenv()

# Initialize the OpenAI client (uses OPENAI_API_KEY from .env automatically)
client = OpenAI()
# Initialize ChromaDB with persistent storage so data survives between runs
chroma_client = chromadb.PersistentClient(path="./chroma_db")

What does this code do?

load_dotenv() reads your .env file and loads OPENAI_API_KEY into the environment so the OpenAI client can authenticate automatically.

ChromaDB's PersistentClient stores data to disk in a ./chroma_db folder. This means you can ingest your documents once and query them across multiple sessions without re-embedding.

You won't see output yet. The next chunk adds the function that actually loads your documents.

  • Add the load_documents() function below the initialization code in ingest.py:

def load_documents(folder_path):
    """Load all .txt files from a folder."""
    documents = []
    for filename in os.listdir(folder_path):
        if filename.endswith(".txt"):
            filepath = os.path.join(folder_path, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read()
            documents.append({"text": text, "source": filename})
            print(f"Loaded: {filename} ({len(text)} characters)")
    return documents

What does this function do?

os.listdir(folder_path) scans the folder for all files. The .endswith(".txt") filter ensures only text files are processed.

Each document is stored as a dictionary with two keys: text (the file content) and source (the filename). Tracking the source is important because your final RAG system will cite which file each answer came from.

  • Save ingest.py.
  • Test the function by running this command in your terminal:
python -c "from ingest import load_documents; docs = load_documents('documents'); print(f'{len(docs)} documents loaded')"

You should see each of your text files listed with their character counts, followed by the total count (e.g. 3 documents loaded).

Seeing a ModuleNotFoundError?

Make sure your virtual environment is activated. Your terminal prompt should show (venv) at the beginning. If not, run venv\Scripts\activate to reactivate it.

Help me fix this ModuleNotFoundError.

Add the text chunking function

Your documents are loaded, but they're too long to embed effectively. Embedding models produce the best results on focused, passage-length text rather than entire documents.

Why do we chunk documents?

Embeddings capture the meaning of text as numerical vectors. A 2000-character document about multiple topics produces a blurry, unfocused vector that doesn't match well against specific questions.

Splitting into ~500-character chunks means each vector represents one focused idea. The 50-character overlap ensures that sentences on a chunk boundary aren't cut mid-thought.

  • Add the chunk_text() function below load_documents() in ingest.py:

def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        # Move forward by chunk_size minus overlap to create overlapping windows
        start = end - overlap
    return chunks

What does this function do?

The function slides a 500-character window across the text, stepping forward 450 characters each time (500 minus 50 overlap). Each window becomes one chunk.

The overlap means the last 50 characters of one chunk appear as the first 50 characters of the next. This prevents important sentences from being split at a boundary where neither chunk has the full context.

  • Save ingest.py.
  • Test both functions together by running:
python -c "from ingest import load_documents, chunk_text; docs = load_documents('documents'); chunks = chunk_text(docs[0]['text']); print(f'\nChunks created: {len(chunks)}'); print(f'\nFirst chunk:\n{chunks[0]}')"

You should see the loaded documents listed, followed by the number of chunks created from the first document and a preview of the first chunk's content.

Seeing 0 chunks or an IndexError?

An IndexError on docs[0] means no .txt files were found. Double-check that your text files are saved inside the documents folder with the .txt extension.

If you see only 1 chunk, your file might be very short. Each file should contain at least 200 words (roughly 1000+ characters) to produce multiple chunks.

Help me debug my chunking results.

✔️ Awesome, I've got everything!

Great. Double check you've saved ingest.py and that your test command printed chunks successfully.

ⓧ I'd like to double check the full code

import os
from openai import OpenAI
import chromadb
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()
chroma_client = chromadb.PersistentClient(path="./chroma_db")


def load_documents(folder_path):
    """Load all .txt files from a folder."""
    documents = []
    for filename in os.listdir(folder_path):
        if filename.endswith(".txt"):
            filepath = os.path.join(folder_path, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read()
            documents.append({"text": text, "source": filename})
            print(f"Loaded: {filename} ({len(text)} characters)")
    return documents


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start = end - overlap
    return chunks

Your knowledge base is loaded and chunked. Next up, you'll turn those text chunks into numerical vectors and store them in ChromaDB so you can search them by meaning.

Generate Embeddings and Store in ChromaDB

Your documents are chunked and ready to go. The next challenge is converting those text chunks into a format a computer can actually compare and search through.

In this step, you'll turn plain text into numerical vectors called embeddings, store them in a vector database, and watch similarity search find the most relevant chunks for any query.

In this step, get ready to:
  • Write a function to generate embeddings using the OpenAI API.
  • Build the full ingestion pipeline that processes, embeds, and stores document chunks in ChromaDB.
  • Run the pipeline and test similarity search on your stored vectors.
Write the embedding function

An embedding is a list of numbers (a vector) that captures the meaning of a piece of text. Words and sentences with similar meanings end up close together in this numerical space.

OpenAI's text-embedding-3-small model converts any text into a 1536-dimensional vector. You'll use this to transform each chunk into a searchable representation.

  • In ingest.py, find the chunk_text function and add this new function below it:
def get_embeddings(texts):
    """Generate embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    return [item.embedding for item in response.data]

What does this code do?

This function sends a list of text strings to OpenAI's embedding model and gets back a list of vectors.

  • client.embeddings.create() calls the OpenAI embeddings API with the text-embedding-3-small model.
  • Each item in the response contains a 1536-dimensional vector representing that text's meaning.
  • The list comprehension extracts just the embedding vectors from each response item.

What is an embedding?

Think of embeddings as coordinates on a map of meaning. The sentence "I love dogs" and "I adore puppies" would land near each other because they mean similar things. "The stock market crashed" would land far away from both.

This numerical closeness is what makes similarity search possible. Instead of matching exact keywords, you're matching meaning.

  • Save ingest.py.

You won't see a change yet. The next function will call get_embeddings as part of the full pipeline.

Build the full ingestion pipeline

Now you need a function that orchestrates the entire process: loading documents, chunking them, generating embeddings, and storing everything in ChromaDB.

  • In ingest.py, add the ingest_documents function below get_embeddings:
def ingest_documents(folder_path):
    """Full ingestion pipeline: load, chunk, embed, store."""
    # Load documents
    documents = load_documents(folder_path)
    if not documents:
        print("No .txt files found in the documents folder!")
        return

    # Create or get collection
    collection = chroma_client.get_or_create_collection(
        name="document_qa",
        metadata={"hnsw:space": "cosine"}
    )

    # Process each document
    all_chunks = []
    all_metadatas = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["text"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadatas.append({
                "source": doc["source"],
                "chunk_index": i
            })
            all_ids.append(f"{doc['source']}_chunk_{i}")

    print(f"\nTotal chunks created: {len(all_chunks)}")

    # Generate embeddings
    print("Generating embeddings...")
    embeddings = get_embeddings(all_chunks)
    print(f"Generated {len(embeddings)} embeddings (dimension: {len(embeddings[0])})")

    # Store in ChromaDB
    collection.upsert(
        ids=all_ids,
        embeddings=embeddings,
        documents=all_chunks,
        metadatas=all_metadatas
    )
    print(f"\nStored {len(all_chunks)} chunks in ChromaDB!")
    print("Ingestion complete. You can now run query.py to ask questions.")

What does this pipeline do?

This function runs through four stages:

  • Load all .txt files from the documents folder using your existing load_documents function.
  • Chunk each document into overlapping pieces using chunk_text, and assign each chunk a unique ID and metadata (source filename and position).
  • Embed all chunks at once by passing them to get_embeddings.
  • Store the chunks, embeddings, and metadata in a ChromaDB collection using upsert (which inserts new items or updates existing ones).

What is a vector database?

A vector database like ChromaDB indexes your embedding vectors so you can find the closest ones to any query in milliseconds. Without it, you'd have to compare your query against every single chunk one by one.

The "hnsw:space": "cosine" setting tells ChromaDB to measure distance using cosine similarity. This measures the angle between two vectors. Vectors pointing in a similar direction have a small cosine distance, meaning their text is semantically similar.

  • Save ingest.py.

You won't see a change yet. Next, you'll add the entry point and run the script.

Run the ingestion and test similarity search

Your pipeline functions are ready. Now you need an entry point that kicks everything off when you run the script.

  • At the very bottom of ingest.py, add the main entry point:
if __name__ == "__main__":
    ingest_documents("documents")

What does this code do?

The if __name__ == "__main__": block means this code only runs when you execute the file directly (python ingest.py). It won't run if another script imports functions from this file.

  • Save ingest.py.
  • Run the ingestion script by running this command in your terminal:
python ingest.py

You should see output showing each file loaded, the total chunks created, embeddings generated with their dimension (1536), and a confirmation that chunks were stored in ChromaDB.

Seeing an error?

  • If you see an authentication error, double-check that your .env file has a valid OpenAI API key and that you've saved the file.
  • If you see "No .txt files found", make sure your documents folder contains .txt files and is in the same directory as ingest.py.
  • If the command is not found, confirm your virtual environment is activated (you should see (venv) in your terminal prompt).

Help me debug my ingest.py script

Now test similarity search to see the vector database in action. You'll query your stored chunks with a test string and see which ones come back as most relevant.

  • Open an interactive Python session by running:
python
  • Run this test query in the Python session to search your chunks:
from ingest import get_embeddings, chroma_client

# Connect to the collection you just created
collection = chroma_client.get_or_create_collection(
    name="document_qa",
    metadata={"hnsw:space": "cosine"}
)

# Generate an embedding for a test question
test_embedding = get_embeddings(["What is this about?"])[0]

# Query for the 2 most similar chunks
results = collection.query(
    query_embeddings=[test_embedding],
    n_results=2
)

# Print results with distance scores
for i, doc in enumerate(results["documents"][0]):
    print(f"Result {i+1} (distance: {results['distances'][0][i]:.4f}):")
    print(f"  {doc[:100]}...")
    print()

What do the distance scores mean?

The distance values show how far each result is from your query in vector space. A distance of 0 means identical. Smaller distances mean the chunk is more semantically relevant to your question.

Because we configured the collection with cosine distance, these scores measure the angle between vectors. Two chunks about the same topic will have a small cosine distance even if they use different words.

  • Exit the Python session by typing:
exit()

✔️ Awesome, I've got everything!

Great. Double check you've saved ingest.py and that you can see the chroma_db folder in your project directory.

ⓧ I'd like to double check the full code

import os
from openai import OpenAI
import chromadb
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()
chroma_client = chromadb.PersistentClient(path="./chroma_db")


def load_documents(folder_path):
    """Load all .txt files from a folder."""
    documents = []
    for filename in os.listdir(folder_path):
        if filename.endswith(".txt"):
            filepath = os.path.join(folder_path, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read()
            documents.append({"text": text, "source": filename})
            print(f"Loaded: {filename} ({len(text)} characters)")
    return documents


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start = end - overlap
    return chunks


def get_embeddings(texts):
    """Generate embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    return [item.embedding for item in response.data]


def ingest_documents(folder_path):
    """Full ingestion pipeline: load, chunk, embed, store."""
    # Load documents
    documents = load_documents(folder_path)
    if not documents:
        print("No .txt files found in the documents folder!")
        return

    # Create or get collection
    collection = chroma_client.get_or_create_collection(
        name="document_qa",
        metadata={"hnsw:space": "cosine"}
    )

    # Process each document
    all_chunks = []
    all_metadatas = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["text"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadatas.append({
                "source": doc["source"],
                "chunk_index": i
            })
            all_ids.append(f"{doc['source']}_chunk_{i}")

    print(f"\nTotal chunks created: {len(all_chunks)}")

    # Generate embeddings
    print("Generating embeddings...")
    embeddings = get_embeddings(all_chunks)
    print(f"Generated {len(embeddings)} embeddings (dimension: {len(embeddings[0])})")

    # Store in ChromaDB
    collection.upsert(
        ids=all_ids,
        embeddings=embeddings,
        documents=all_chunks,
        metadatas=all_metadatas
    )
    print(f"\nStored {len(all_chunks)} chunks in ChromaDB!")
    print("Ingestion complete. You can now run query.py to ask questions.")


if __name__ == "__main__":
    ingest_documents("documents")

Your documents are now embedded and stored as searchable vectors. Next up, you'll build the retrieval pipeline that finds the most relevant chunks for any question you ask.

Build the Retrieval Pipeline

Your documents are chunked, embedded, and stored in ChromaDB. The vector database is loaded and ready to answer questions. But how do you actually find the right chunks for a given question?

The answer is similarity search. You convert the question into an embedding using the same model, then ask the database to find the stored chunks whose vectors are closest to your question vector. In this step, you will write a retrieval function, test it with real queries, and add metadata filtering for production-ready control.

In this step, get ready to:
  • Write a retrieval function that finds the most relevant chunks for any question.
  • Test retrieval with different queries to see how similarity search behaves.
  • Add metadata filtering to scope retrieval to specific documents.
Write the retrieve function

The retrieval function is the bridge between a user's natural language question and the stored document chunks. It converts the question into a vector and searches for the nearest neighbors in the database.

  • In VS Code, create a new file called query.py in your project folder (the same folder where ingest.py lives).
  • Add the imports and client setup by pasting this code at the top of query.py:
from openai import OpenAI
import chromadb
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(
    name="document_qa",
    metadata={"hnsw:space": "cosine"}
)

What does this code do?

This setup mirrors what you did in ingest.py. You load environment variables, initialize the OpenAI client, and connect to the same persistent ChromaDB database.

The key line is get_or_create_collection. It reconnects to the existing document_qa collection you created during ingestion, giving you access to all your stored chunks and embeddings.

  • Add the get_query_embedding function below the setup code:
def get_query_embedding(text):
    """Generate embedding for a query string."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

Why a separate embedding function?

In ingest.py, you embedded batches of chunks. Here you only need to embed a single question at a time, so this function takes one string and returns one embedding vector.

You must use the same model (text-embedding-3-small) for queries as you used for ingestion. Embeddings from different models live in different vector spaces and cannot be compared.

  • Add the retrieve function below get_query_embedding:
def retrieve(question, n_results=3):
    """Retrieve the most relevant chunks for a question."""
    query_embedding = get_query_embedding(question)

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results
    )

    retrieved_chunks = []
    for i in range(len(results["documents"][0])):
        retrieved_chunks.append({
            "text": results["documents"][0][i],
            "source": results["metadatas"][0][i]["source"],
            "distance": results["distances"][0][i]
        })

    return retrieved_chunks

What does this code do?

The retrieve function is your similarity search engine. It embeds the question, then calls collection.query() to find the 3 closest chunks in vector space.

ChromaDB returns results organized by query. Since you send one query at a time, everything lives at index [0]. Each result includes the original text, its metadata (source filename), and the distance score.

Lower distance means higher similarity. A distance of 0 would mean an exact match.

  • Save query.py.

Seeing a red underline on the imports?

Make sure your virtual environment is activated in VS Code. Check the bottom status bar for the Python interpreter path pointing to your venv folder.

help me fix Python import errors in VS Code

Test retrieval with different queries

Your retrieval function is ready. Now you will test it by running real queries against your stored documents and examining the distance scores that come back.

  • Open your terminal in VS Code and test retrieval by running this command:
python -c "from query import retrieve; chunks = retrieve('What is the main topic?'); [print(f'[{i+1}] Source: {c[chr(39)+chr(39)}source{chr(39)+chr(39)}] | Distance: {c[chr(39)+chr(39)}distance{chr(39)+chr(39)}]:.4f}\n    {c[chr(39)+chr(39)}text{chr(39)+chr(39)}][:100]}...') for i, c in enumerate(chunks)]"

Actually, running complex Python in a one-liner is messy. A cleaner approach is to use the interactive Python shell.

  • Start the Python interactive shell by running:
python
  • Test retrieval with a question that your documents should answer by running these lines:
from query import retrieve

# Try a question your documents cover
chunks = retrieve("What is the main topic?")
for i, chunk in enumerate(chunks, 1):
    print(f"[{i}] Source: {chunk['source']} | Distance: {chunk['distance']:.4f}")
    print(f"    {chunk['text'][:100]}...")
    print()

What should I see?

You should see 3 retrieved chunks, each showing the source filename, a distance score (a number between 0 and 2 for cosine distance), and a preview of the chunk text. Lower distance values mean the chunk is more semantically similar to your question.

  • Now try a question that your documents definitely do NOT cover:
# Try a question your documents DON'T cover
chunks = retrieve("How do black holes form?")
for i, chunk in enumerate(chunks, 1):
    print(f"[{i}] Source: {chunk['source']} | Distance: {chunk['distance']:.4f}")
    print(f"    {chunk['text'][:100]}...")
    print()

Why does it still return results?

Top-k retrieval always returns k results, even when none are truly relevant. The vector database finds the closest chunks to your question no matter what. It does not know whether those chunks actually answer the question.

Notice the distance scores for the irrelevant question are higher (farther) than for the relevant question. In a production system, you would set a distance threshold and discard chunks above it. For now, observe the difference.

  • Exit the Python shell by typing:
exit()
Add metadata filtering to your retrieval

In production, you often need to restrict which documents the search considers. A user might only have permission to see certain files. Or you might want to search only the most recent documents.

What is metadata filtering?

When you ingested documents, each chunk was stored with metadata including its source filename. ChromaDB lets you pass a where filter that restricts which chunks are even considered during the search. This is much more efficient than retrieving everything and filtering after.

  • In query.py, find the retrieve function. Locate this line:
def retrieve(question, n_results=3):
  • Replace the entire retrieve function with this updated version that adds a source_filter parameter:
def retrieve(question, n_results=3, source_filter=None):
    """Retrieve the most relevant chunks for a question."""
    query_embedding = get_query_embedding(question)

    where_filter = None
    if source_filter:
        where_filter = {"source": source_filter}

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
        where=where_filter
    )

    retrieved_chunks = []
    for i in range(len(results["documents"][0])):
        retrieved_chunks.append({
            "text": results["documents"][0][i],
            "source": results["metadatas"][0][i]["source"],
            "distance": results["distances"][0][i]
        })

    return retrieved_chunks

What changed here?

The function now accepts an optional source_filter parameter. When provided, it builds a where dictionary that tells ChromaDB to only consider chunks from that specific source file.

When source_filter is None (the default), the where parameter passes None to ChromaDB, which means search all chunks with no restriction.

  • Save query.py.
  • Test metadata filtering by opening the Python shell again:
python
  • Run a filtered query using one of your document filenames:
from query import retrieve

# Search only within a specific document
chunks = retrieve("What is the main topic?", source_filter="your-filename.txt")
for i, chunk in enumerate(chunks, 1):
    print(f"[{i}] Source: {chunk['source']} | Distance: {chunk['distance']:.4f}")
    print(f"    {chunk['text'][:100]}...")
    print()
  • Replace your-filename.txt with the actual name of one of your text files in the documents folder.
  • Verify that all returned chunks come from only that file.
  • Exit the Python shell by typing exit().

Getting an error about the collection?

If you see an error mentioning the collection is empty or not found, make sure you ran ingest.py successfully in the previous step. The chroma_db folder should exist in your project directory.

help me debug ChromaDB collection errors in my RAG project

✔️ Awesome, I've got everything!

Great. Double check you have saved query.py with the updated retrieve function that includes source_filter.

ⓧ I'd like to double check the full code

from openai import OpenAI
import chromadb
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(
    name="document_qa",
    metadata={"hnsw:space": "cosine"}
)


def get_query_embedding(text):
    """Generate embedding for a query string."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding


def retrieve(question, n_results=3, source_filter=None):
    """Retrieve the most relevant chunks for a question."""
    query_embedding = get_query_embedding(question)

    where_filter = None
    if source_filter:
        where_filter = {"source": source_filter}

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
        where=where_filter
    )

    retrieved_chunks = []
    for i in range(len(results["documents"][0])):
        retrieved_chunks.append({
            "text": results["documents"][0][i],
            "source": results["metadatas"][0][i]["source"],
            "distance": results["distances"][0][i]
        })

    return retrieved_chunks

Your retrieval pipeline is working. You can now search your document store with natural language questions and see exactly which chunks are most relevant. Next up, you will connect this retrieval to an LLM and generate grounded answers with source citations.

Build the Generation Step and Complete the RAG Pipeline

Your retrieval pipeline can find the most relevant chunks from your documents. But raw chunks alone don't answer a question. You need a language model to read those chunks, synthesize them, and produce a human-friendly answer.

This is where the "Generation" in Retrieval-Augmented Generation comes in. In this step, you'll connect your retriever to gpt-5.4-mini with a carefully crafted prompt that forces the model to answer only from retrieved context and cite its sources.

In this step, get ready to:
  • Write a generation function that turns retrieved chunks into grounded, cited answers.
  • Combine retrieval and generation into a single RAG pipeline function.
  • Build an interactive terminal loop to keep asking questions.
Write the generate_answer function

The generation function is the bridge between raw retrieved text and a polished answer. It constructs a prompt that provides the retrieved chunks as context and instructs the model to stay grounded in that context.

  • In query.py, add the following generate_answer function below the retrieve function:
def generate_answer(question, chunks):
    """Generate an answer using retrieved context."""
    context = ""
    for i, chunk in enumerate(chunks, 1):
        context += f"\n[Source {i}: {chunk['source']}]\n{chunk['text']}\n"

    prompt = f"""You are a helpful assistant that answers questions based ONLY on the provided context.
If the context does not contain enough information to answer the question, say "I don't have enough information to answer that question."
Always cite which source(s) your answer comes from using [Source N] notation.

Context:
{context}

Question: {question}

Answer:"""

    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2
    )

    return response.choices[0].message.content

What does this code do?

The function builds a context string by formatting each retrieved chunk with a numbered source label. This makes it easy for the model to reference where information came from.

The prompt is the critical piece of prompt engineering. It gives the model three strict instructions: only use the provided context, say when it doesn't know, and cite sources with [Source N] notation. Without these constraints, the model would hallucinate answers from its training data.

Setting temperature=0.2 keeps the output focused and deterministic. A higher value would introduce more randomness, which you don't want in a factual Q&A system.

  • Save query.py

Why constrain the model this way?

Without the "answer ONLY from context" instruction, the model will blend its training knowledge with the retrieved text. This defeats the purpose of RAG. The whole point is to get answers grounded in YOUR documents, not the model's general knowledge.

The citation requirement creates accountability. If the model claims something, you can trace it back to the specific chunk that contained that information.

Combine retrieval and generation into the ask function

Now you'll create a single function that ties the entire pipeline together: embed the question, retrieve relevant chunks, display them, and generate a grounded answer.

  • In query.py, add the following ask function below generate_answer:
def ask(question, source_filter=None):
    """Complete RAG pipeline: retrieve + generate."""
    print(f"\n{'='*60}")
    print(f"Question: {question}")
    print(f"{'='*60}")

    # Retrieve relevant chunks
    chunks = retrieve(question, n_results=3, source_filter=source_filter)

    print(f"\nRetrieved {len(chunks)} chunks:")
    for i, chunk in enumerate(chunks, 1):
        print(f"\n  [{i}] Source: {chunk['source']} | Distance: {chunk['distance']:.4f}")
        print(f"      {chunk['text'][:100]}...")

    # Generate answer
    answer = generate_answer(question, chunks)

    print(f"\nAnswer:\n{answer}")
    print(f"{'='*60}")

    return answer

What does this code do?

The ask function is your complete RAG pipeline in one call. It retrieves the top 3 most relevant chunks, displays them with their cosine distance scores, then passes everything to generate_answer for the final response.

Printing the retrieved chunks and distances lets you see exactly what context the model is working with. Lower distance scores mean higher similarity. This transparency helps you debug when answers are wrong.

The optional source_filter parameter lets you restrict retrieval to a specific document, which is the metadata filtering you built in the previous step.

  • Save query.py.

Seeing an API error?

If you see an authentication error, double-check that your .env file contains a valid API key and that load_dotenv() is at the top of query.py. The OpenAI client reads OPENAI_API_KEY from your environment automatically.

If you see a model not found error, confirm you typed gpt-5.4-mini exactly as shown in the code above.

help me debug my OpenAI API connection

Add an interactive question loop

The final piece turns your script into a proper interactive tool. You'll add a loop that keeps accepting questions from the terminal until the user types quit.

  • At the bottom of query.py, add the following __main__ block:
if __name__ == "__main__":
    print("RAG Document Q&A Bot")
    print("Type 'quit' to exit.\n")

    while True:
        question = input("\nYour question: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not question:
            continue
        ask(question)

What does this code do?

The if __name__ == "__main__": guard means this code only runs when you execute the file directly (not when importing it as a module).

The while True loop continuously prompts for questions. Empty inputs are skipped, and typing quit, exit, or q breaks the loop cleanly.

  • Save query.py.
  • Run your RAG pipeline by executing this command in your terminal:
python query.py
  • Ask a question that is clearly covered by your documents. You should see retrieved chunks with distance scores followed by a generated answer with [Source N] citations.
  • Ask a question that is NOT covered by your documents. The model should respond with something like "I don't have enough information to answer that question."
  • Type quit to exit when you're done testing.

What should I see?

For each question, the terminal prints three things: the question itself in a header, the top 3 retrieved chunks with their source filenames and distance scores, and then the generated answer with citations pointing back to those sources.

Lower distance values (closer to 0) mean the chunk is more semantically similar to your question. A distance of 0.2 is very relevant. A distance above 0.5 means the chunk is only loosely related.

Getting empty results or no chunks?

Make sure you ran python ingest.py first. The chroma_db folder must exist with your indexed documents before querying.

If you see chunks but the answer seems unrelated, check that your documents actually contain information about the topic you're asking about.

my RAG bot isn't returning useful answers

✔️ Awesome, I've got everything!

Great. Double check you've saved query.py and confirmed the full pipeline works end-to-end.

ⓧ I'd like to double check the full code

Here is the complete query.py file at this point:

from openai import OpenAI
import chromadb
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(
    name="document_qa",
    metadata={"hnsw:space": "cosine"}
)


def get_query_embedding(text):
    """Generate embedding for a query string."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding


def retrieve(question, n_results=3, source_filter=None):
    """Retrieve the most relevant chunks for a question."""
    query_embedding = get_query_embedding(question)

    where_filter = None
    if source_filter:
        where_filter = {"source": source_filter}

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
        where=where_filter
    )

    retrieved_chunks = []
    for i in range(len(results["documents"][0])):
        retrieved_chunks.append({
            "text": results["documents"][0][i],
            "source": results["metadatas"][0][i]["source"],
            "distance": results["distances"][0][i]
        })

    return retrieved_chunks


def generate_answer(question, chunks):
    """Generate an answer using retrieved context."""
    context = ""
    for i, chunk in enumerate(chunks, 1):
        context += f"\n[Source {i}: {chunk['source']}]\n{chunk['text']}\n"

    prompt = f"""You are a helpful assistant that answers questions based ONLY on the provided context.
If the context does not contain enough information to answer the question, say "I don't have enough information to answer that question."
Always cite which source(s) your answer comes from using [Source N] notation.

Context:
{context}

Question: {question}

Answer:"""

    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2
    )

    return response.choices[0].message.content


def ask(question, source_filter=None):
    """Complete RAG pipeline: retrieve + generate."""
    print(f"\n{'='*60}")
    print(f"Question: {question}")
    print(f"{'='*60}")

    chunks = retrieve(question, n_results=3, source_filter=source_filter)

    print(f"\nRetrieved {len(chunks)} chunks:")
    for i, chunk in enumerate(chunks, 1):
        print(f"\n  [{i}] Source: {chunk['source']} | Distance: {chunk['distance']:.4f}")
        print(f"      {chunk['text'][:100]}...")

    answer = generate_answer(question, chunks)

    print(f"\nAnswer:\n{answer}")
    print(f"{'='*60}")

    return answer


if __name__ == "__main__":
    print("RAG Document Q&A Bot")
    print("Type 'quit' to exit.\n")

    while True:
        question = input("\nYour question: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not question:
            continue
        ask(question)

Your RAG system is fully operational. You've built every layer from raw documents to cited answers. Next up, take on the Secret Mission to add reranking and dramatically improve your retrieval quality.

Secret mission

Add Reranking for Better Retrieval

Your RAG pipeline works, but vector similarity alone is an imperfect filter. Sometimes chunks that are semantically close to a query don't actually help answer it. Reranking adds a second, more careful evaluation pass that dramatically improves answer quality.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with no running services or ongoing costs. OpenAI API charges only apply when you actively run the scripts.

Resources you used:

  • Project folder on your Desktop (contains ingest.py, query.py, .env, documents/, chroma_db/, and venv/).
  • Python 3.14 installation on your system.

Keep everything running

No action needed. Choose this if you plan to keep building on this RAG system or want Python available for future projects.

  • Your project folder stays on your Desktop, ready to use whenever you want.
  • The chroma_db/ folder persists your vector database between sessions. Run query.py anytime to ask new questions.
  • Python 3.14 stays installed and available for other projects.

Pause - I'll come back to this later

Deactivate your virtual environment and close VS Code. Your files stay on disk, ready to pick back up later.

  • If your virtual environment is still active, deactivate it by running:
  • Close VS Code.
deactivate

When you return, reactivate the environment with:

venv\Scripts\activate

Delete - I don't want to use this again

Remove all project resources and optionally uninstall Python from your system.

  • Deactivate the virtual environment if it is still active:
  • Close VS Code.
deactivate
  • Delete the entire project folder from your Desktop. Open File Explorer, navigate to your Desktop, right-click your project folder, and select Delete.
  • To uninstall the Python 3.14 runtime, open a terminal and run:
py uninstall 3.14
  • Confirm the removal when prompted. This removes the Python 3.14 runtime but keeps the Python install manager.
  • If you also want to remove the Python install manager itself, click Start, open Installed apps, search for Python, and uninstall the Python install manager entry.

Nice Work!

Nice Work!

You just built a complete Retrieval-Augmented Generation system from the ground up. Every component in a production RAG pipeline is now something you understand hands-on, from raw text all the way to cited, grounded AI answers.

What you learned:

  • Built a full RAG pipeline from scratch that loads documents, splits them into overlapping chunks, generates embeddings with OpenAI, and stores them in a ChromaDB vector database for millisecond similarity search.
  • Created a retrieval function that embeds a natural language question and finds the most relevant chunks using cosine distance, with support for metadata filtering by source document.
  • Assembled a generation step that constructs a grounded prompt from retrieved context, calls gpt-5.4-mini, and produces answers with source citations while refusing to hallucinate when context is insufficient.
  • Added a reranking step that re-scores retrieved chunks for relevance, dramatically improving answer quality by filtering out semantically similar but unhelpful passages.

Ready to quiz yourself?