Build a RAG System with ChromaDB

Build a RAG pipeline that answers questions about your own documents.

Introduction

30 Second Summary

Have you ever wished you could ask an AI questions about your own documents and get accurate answers? Standard AI models only know what they were trained on, so they have no way to access your specific files or data.

In this project, you will build a RAG (Retrieval-Augmented Generation) system from scratch. Your system will ingest text documents, store them in a vector database, and answer natural language questions by retrieving relevant context before generating a response.

What You'll Build

A complete RAG pipeline built with Python, ChromaDB, and OpenAI that transforms raw text documents into a queryable knowledge base you can interact with from the command line.

By the end of this project, you'll have:

  • A document loader and chunker that splits text files into overlapping passages with metadata tracking.
  • A ChromaDB vector database storing embedded document chunks you can query using semantic similarity search.
  • A complete end-to-end RAG pipeline that retrieves relevant context and generates grounded LLM answers with source citations.
  • Secret Mission: A multi-document comparison feature that retrieves across multiple collections and synthesizes cross-document answers.

Are there any prerequisites?

You'll need basic Python knowledge including variables, functions, and loops. No prior AI or machine learning experience is required.

Set Up Your Environment

Before you can build a RAG system, you need a proper Python environment with the right packages installed.

In this step, you will create a project folder, set up an isolated virtual environment, install the core libraries, and configure your OpenAI API key so the RAG pipeline can generate embeddings and answers.

In this step, get ready to:
  • Install Python and create a virtual environment.
  • Install the project dependencies.
  • Configure your OpenAI API key.
Install Python and create a virtual environment

This project requires Python 3.9 or higher. Let's check which version you have installed.

  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.9 or higher

You're all set. Move on to the next instruction below.

ⓧ I see an older version

You have Python installed, but it is below 3.9. You need to upgrade.

  • Download the latest Python installer from python.org/downloads.
  • Run the installer. On Windows, make sure to check Add Python to PATH before clicking Install Now.
  • On macOS, after installing, use python3 --version to confirm the new version appears. The macOS system Python may still point to an older version. Use python3 explicitly for all commands in this project.

ⓧ Command not found

Python is not installed on your machine yet.

  • Download Python from python.org/downloads.
  • Run the installer. On Windows, check Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal, then run the version check command again to confirm it works.

Now let's create a project folder on your Desktop and set up an isolated virtual environment inside it.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-system
cd rag-system
  • Create and activate your virtual environment by running these commands:

macOS

python3 -m venv .venv
source .venv/bin/activate

Windows

python -m venv .venv
.venv\Scripts\activate

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python, so different projects can use different package versions without conflicts.

After activation, your terminal prompt changes to show (.venv) at the beginning. This confirms you are working inside the virtual environment.

  • Confirm your terminal prompt now starts with (.venv). This means the virtual environment is active.

Don't see (.venv) in your prompt?

  • Make sure you ran the activation command for your operating system. macOS uses source .venv/bin/activate and Windows uses .venv\Scripts\activate.
  • On Windows PowerShell, you may need to run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned first to allow script execution.
  • If you closed and reopened your terminal, you need to cd back into the rag-system folder and rerun the activate command.

Help me activate my Python virtual environment.

Install dependencies

With your virtual environment active, you can install the three packages this project needs: ChromaDB for vector storage, the OpenAI client for embeddings and chat completions, and python-dotenv for loading environment variables.

  • Install the required packages by running this command:
pip install chromadb==1.5.9 openai==2.41.0 python-dotenv==1.1.0

What do these packages do?

  • chromadb is an open-source vector database that stores your document embeddings and handles similarity search locally.
  • openai is the official Python client for OpenAI's API. You will use it to generate text embeddings and chat completions.
  • python-dotenv loads environment variables from a .env file so you can keep secrets out of your code.

The installation may take a minute or two since ChromaDB has several dependencies to download.

  • Verify the packages installed correctly by running:
pip list
  • You should see chromadb, openai, and python-dotenv in the output list along with their version numbers.

Seeing an installation error?

  • Make sure your virtual environment is active. You should see (.venv) in your terminal prompt. If not, rerun the activate command.
  • If you see a permissions error, do not use sudo. Instead, make sure you are inside the activated virtual environment.
  • If a specific package fails to build, make sure your Python version is 3.9 or higher. ChromaDB requires it.

Help me fix my pip install error.

Configure your OpenAI API key

The RAG system uses OpenAI's embedding model to convert text into vectors and a chat model to generate answers. Both require an API key.

  • Open VS Code in your project folder by running this command from your terminal:
code .

What does code . do?

The code . command launches VS Code and opens the current directory as your workspace. The dot (.) means "this folder".

If this command does not work, open VS Code manually. Press Cmd+Space (macOS) or the Windows key (Windows), type Visual Studio Code, and press Enter. Then go to File > Open Folder and select your rag-system folder on the Desktop.

  • In the VS Code Explorer sidebar, right-click in the empty area and select New File.
  • Name the file .env and press Enter.
  • Paste the following content into the file:
OPENAI_API_KEY=your-api-key-here

Where do I get my API key?

  • Go to platform.openai.com/api-keys and sign in to your OpenAI account.
  • Click Create new secret key.
  • Copy the key immediately. OpenAI only shows it once.
  • Back in VS Code, replace your-api-key-here with the key you just copied.

If you do not have an OpenAI account, you can create one for free. New accounts typically receive a small amount of free credit. This project costs approximately $0.01 total.

  • Save the .env file by pressing Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Make sure you have saved your .env file with your actual API key replacing the placeholder.

ⓧ I'd like to double check the full code

Your .env file should look exactly like this (with your real key in place of the placeholder):

OPENAI_API_KEY=your-api-key-here

Worried about keeping your API key safe?

The .env file stays local on your machine. Never commit it to version control. If you use Git later, add .env to your .gitignore file to keep it out of your repository.

Your environment is ready. Next up, you will build the document loader and chunker that splits your files into retrieval-ready pieces.

Build the Document Loader and Chunker

Your environment is ready with Python, ChromaDB, and your OpenAI API key configured. Now it's time to build the first layer of the RAG pipeline.

Every RAG system starts with ingestion. Raw documents need to be broken into smaller pieces that a vector database can store and search through efficiently. In this step, you'll build the loader and chunker that handles this transformation.

In this step, get ready to:
  • Create sample documents for your RAG system to ingest.
  • Build a document loader that reads text files from a folder.
  • Implement a text chunker that splits documents into overlapping passages.
Create sample documents to ingest

Your RAG system needs source documents to retrieve from. You'll create two text files that describe a fictional company and its products.

  • Create a documents folder inside your rag-system project folder by running this command:
mkdir documents
  • Confirm the folder exists by running:
ls

You should see documents listed alongside your .env file and .venv folder.

  • Create a new file called company_overview.txt inside the documents folder, then paste in the following content:
Acme Corporation was founded in 2015 by Jane Smith and John Doe. The company specializes in developing sustainable energy solutions for residential and commercial buildings. Headquartered in Austin, Texas, Acme has grown from a small startup to a company with over 500 employees across three offices.

Acme's flagship product is the SolarSync system, which combines solar panels with intelligent battery storage. The system uses machine learning algorithms to optimize energy usage patterns based on weather forecasts and household consumption data. SolarSync has been installed in over 50,000 homes across the United States.

The company's revenue reached $200 million in 2025, representing a 40% year-over-year growth. Acme went public in 2023 on the NASDAQ exchange under the ticker symbol ACME. The current CEO is Jane Smith, who has been recognized as one of the top 50 women in technology by Forbes magazine.

Acme's research division is currently developing next-generation solid-state batteries that promise to double energy storage capacity while reducing costs by 30%. The company holds over 150 patents related to energy storage and solar technology. Their R&D center in Boulder, Colorado employs 80 scientists and engineers working on breakthrough energy solutions.

What is this document for?

This file gives your RAG system factual content to retrieve from later. When you ask a question like "Who founded Acme?", the retriever will search these chunks for the most relevant passage.

  • Create a second file called product_specs.txt inside the documents folder, then paste in this content:
SolarSync Pro System Specifications

The SolarSync Pro is Acme Corporation's premium residential energy system. It consists of three main components: solar panels, a battery storage unit, and the SyncController hub.

Solar Panels: The system uses monocrystalline silicon panels with 22.5% efficiency rating. Each panel produces 400 watts under standard test conditions. A typical residential installation includes 20-30 panels, generating 8-12 kW of power. The panels come with a 25-year performance warranty guaranteeing at least 85% output retention.

Battery Storage: The PowerVault battery unit uses lithium iron phosphate (LFP) chemistry for maximum safety and longevity. Available in 10 kWh and 20 kWh configurations. The battery supports 10,000 charge cycles before reaching 80% capacity, which translates to approximately 27 years of daily use. Operating temperature range is -20C to 50C.

SyncController Hub: This is the brain of the system. It connects to home WiFi and uses Acme's proprietary AI algorithms to manage energy flow. Features include real-time monitoring via mobile app, automatic grid switching during outages, time-of-use optimization to minimize electricity costs, and integration with smart home platforms including Google Home and Amazon Alexa. The controller updates firmware automatically and requires no user maintenance.

Installation typically takes 2-3 days for a standard residential setup. Acme provides certified installers in all 50 states. The total system cost ranges from $25,000 to $45,000 before federal tax credits, which currently cover 30% of the installation cost.
  • Save both files.

You now have two documents totalling around 600 words each. This is enough to demonstrate chunking clearly.

Don't see both files in your documents folder?

Make sure the files are saved inside the documents folder (not directly in rag-system). The file names must end in .txt for the loader to find them.

help me figure out why my text files aren't in the right folder

Write the document loader and chunking logic

Now you'll build the Python script that reads those documents and splits them into smaller, overlapping chunks. Overlapping chunks ensure that no important context is lost at the boundary between two pieces.

  • Create a new file called loader.py in the root of your rag-system folder.
  • Add the import and the document loading function by pasting this code:
import os


def load_documents(directory="documents"):
    """Load all .txt files from the specified directory."""
    documents = []
    for filename in os.listdir(directory):
        if filename.endswith(".txt"):
            filepath = os.path.join(directory, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                content = f.read()
            documents.append({"content": content, "source": filename})
    return documents

What does this code do?

  • load_documents() scans the documents/ folder for any file ending in .txt.
  • For each text file it finds, it reads the full content and stores it alongside the filename as metadata.
  • The result is a list of dictionaries, each holding the file's content and its source name.
  • Below the load_documents function, add the chunking function:
def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start = end - overlap
    return chunks

What does this code do?

chunk_text() slides a 500-character window across the document text. After each window, it steps forward by 450 characters (500 minus the 50-character overlap).

The overlap means the end of one chunk appears again at the start of the next. This prevents sentences that span a boundary from being split without context on either side.

  • Below chunk_text, add the orchestration function and the main block that ties everything together:
def process_documents(directory="documents"):
    """Load documents and split them into chunks with metadata."""
    documents = load_documents(directory)
    all_chunks = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        total_chunks = len(chunks)
        for i, chunk in enumerate(chunks):
            all_chunks.append({
                "text": chunk,
                "metadata": {
                    "source": doc["source"],
                    "chunk_index": i,
                    "total_chunks": total_chunks,
                },
                "id": f"{doc['source']}_chunk_{i}",
            })

    return all_chunks


if __name__ == "__main__":
    chunks = process_documents()
    print(f"Total chunks created: {len(chunks)}")
    for chunk in chunks[:3]:
        print(f"\n--- {chunk['id']} ---")
        print(f"Source: {chunk['metadata']['source']}")
        print(f"Chunk {chunk['metadata']['chunk_index'] + 1} of {chunk['metadata']['total_chunks']}")
        print(f"Text preview: {chunk['text'][:100]}...")

What does this code do?

  • process_documents() calls load_documents() then runs chunk_text() on each document's content.
  • Each chunk gets metadata tracking its source filename, position index, and the total number of chunks from that document.
  • The id field gives every chunk a unique identifier by combining the filename and chunk number.
  • The if __name__ == "__main__" block runs a quick test. It prints the total chunk count and previews the first 3 chunks so you can verify everything works.
  • Save loader.py.
  • Run the loader to verify it works:
python loader.py

You should see output showing the total number of chunks created and a preview of the first 3 chunks, each with its source filename and chunk index.

Seeing a FileNotFoundError?

This means Python cannot find the documents/ folder. Make sure you're running the command from inside your rag-system folder and that documents/ is a direct subfolder of rag-system.

help me debug why Python can't find my documents folder

✔️ Awesome, I've got everything!

Great. Double check you've saved loader.py and move on to the next section.

ⓧ I'd like to double check the full code

Here's the complete loader.py file for comparison:

import os


def load_documents(directory="documents"):
    """Load all .txt files from the specified directory."""
    documents = []
    for filename in os.listdir(directory):
        if filename.endswith(".txt"):
            filepath = os.path.join(directory, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                content = f.read()
            documents.append({"content": content, "source": filename})
    return documents


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start = end - overlap
    return chunks


def process_documents(directory="documents"):
    """Load documents and split them into chunks with metadata."""
    documents = load_documents(directory)
    all_chunks = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        total_chunks = len(chunks)
        for i, chunk in enumerate(chunks):
            all_chunks.append({
                "text": chunk,
                "metadata": {
                    "source": doc["source"],
                    "chunk_index": i,
                    "total_chunks": total_chunks,
                },
                "id": f"{doc['source']}_chunk_{i}",
            })

    return all_chunks


if __name__ == "__main__":
    chunks = process_documents()
    print(f"Total chunks created: {len(chunks)}")
    for chunk in chunks[:3]:
        print(f"\n--- {chunk['id']} ---")
        print(f"Source: {chunk['metadata']['source']}")
        print(f"Chunk {chunk['metadata']['chunk_index'] + 1} of {chunk['metadata']['total_chunks']}")
        print(f"Text preview: {chunk['text'][:100]}...")
Understand the architecture so far

What role does the loader play in RAG?

The loader is the "ingestion" layer of your RAG pipeline. It transforms raw documents into retrieval-ready units that a vector database can store and search.

The 50-character overlap ensures context is not lost at chunk boundaries. If a key sentence happens to fall at the edge of one chunk, the overlap means the next chunk captures it too.

No cost was incurred in this step. The text files and the Python script all run locally.

Your ingestion layer is complete. Next up, you'll embed these chunks into vectors and store them in ChromaDB so they become searchable by meaning.

Embed and Store in ChromaDB

Your document loader can break raw text into retrieval-ready chunks. The next challenge is making those chunks searchable by meaning, not just by exact keywords.

In this step, you will convert your text chunks into numerical vectors (embeddings) and store them in ChromaDB. This is the "index" layer of your RAG pipeline. Once chunks are embedded, you can find the most relevant passages for any question using semantic similarity.

In this step, get ready to:
  • Create an embedding and storage pipeline that sends chunks to ChromaDB.
  • Verify your vector store contains all expected chunks with correct metadata.
  • Understand how embeddings and vector databases fit into the RAG architecture.
Create the embedding and storage pipeline

The embedder is the bridge between your text chunks and the vector database. It takes each chunk, converts it into a 1536-dimensional vector using OpenAI's embedding model, and stores both the vector and original text in ChromaDB for fast similarity search.

  • Create a new file called embedder.py in your rag-system folder.
  • Paste the following code into embedder.py:
import os

import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
from dotenv import load_dotenv

from loader import process_documents

load_dotenv()


def create_vector_store():
    """Create a ChromaDB collection and populate it with document chunks."""
    # Initialize a persistent ChromaDB client that saves data to disk
    client = chromadb.PersistentClient(path="./chroma_data")

    # Set up the OpenAI embedding function for automatic vectorization
    embedding_function = OpenAIEmbeddingFunction(
        api_key=os.getenv("OPENAI_API_KEY"),
        model_name="text-embedding-3-small",
    )

    # Create or connect to the collection with the embedding function attached
    collection = client.get_or_create_collection(
        name="rag_documents",
        embedding_function=embedding_function,
    )

    # Load and chunk all documents using the loader module
    chunks = process_documents()

    # Prepare parallel lists for ChromaDB's upsert method
    ids = [chunk["id"] for chunk in chunks]
    documents = [chunk["text"] for chunk in chunks]
    metadatas = [chunk["metadata"] for chunk in chunks]

    # Upsert chunks into the collection (embeds automatically on insert)
    collection.upsert(ids=ids, documents=documents, metadatas=metadatas)

    # Print confirmation with chunk count and persistence location
    print(f"Stored {collection.count()} chunks in ChromaDB")
    print(f"Database persisted to ./chroma_data")

    # Peek at one record to verify metadata was preserved
    sample = collection.peek(limit=1)
    print(f"\nSample metadata: {sample['metadatas'][0]}")

    return collection


if __name__ == "__main__":
    create_vector_store()

What does this code do?

This script builds the embedding and storage layer of your RAG system in one function:

  • chromadb.PersistentClient(path="./chroma_data") creates a local database that persists to disk. Your vectors survive between script runs.
  • OpenAIEmbeddingFunction tells ChromaDB to use OpenAI's text-embedding-3-small model to convert text into 1536-dimensional vectors automatically whenever you add documents.
  • get_or_create_collection creates the rag_documents collection on first run, and reconnects to it on subsequent runs.
  • collection.upsert() inserts or updates chunks. ChromaDB handles embedding each chunk behind the scenes using the attached embedding function.
  • collection.peek(limit=1) retrieves one record to confirm metadata (source filename, chunk index) was stored correctly.
  • Save embedder.py.

You won't see output yet. In the next substep, you'll run this script and confirm the vector store is populated.

Seeing a red underline on the import?

If your editor shows an error on from loader import process_documents, make sure loader.py is in the same folder as embedder.py. Both files must be at the top level of your rag-system folder.

Help me fix an import error in my embedder.py file.

✔️ Awesome, I've got everything!

Great. Make sure you've saved embedder.py before continuing.

ⓧ I'd like to double check the full code

Here is the complete embedder.py file for comparison:

import os

import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
from dotenv import load_dotenv

from loader import process_documents

load_dotenv()


def create_vector_store():
    """Create a ChromaDB collection and populate it with document chunks."""
    client = chromadb.PersistentClient(path="./chroma_data")

    embedding_function = OpenAIEmbeddingFunction(
        api_key=os.getenv("OPENAI_API_KEY"),
        model_name="text-embedding-3-small",
    )

    collection = client.get_or_create_collection(
        name="rag_documents",
        embedding_function=embedding_function,
    )

    chunks = process_documents()

    ids = [chunk["id"] for chunk in chunks]
    documents = [chunk["text"] for chunk in chunks]
    metadatas = [chunk["metadata"] for chunk in chunks]

    collection.upsert(ids=ids, documents=documents, metadatas=metadatas)

    print(f"Stored {collection.count()} chunks in ChromaDB")
    print(f"Database persisted to ./chroma_data")

    sample = collection.peek(limit=1)
    print(f"\nSample metadata: {sample['metadatas'][0]}")

    return collection


if __name__ == "__main__":
    create_vector_store()
Run and verify the vector store

Now you will run the embedder to populate your vector database. This sends each chunk to the OpenAI API for embedding and stores the results in ChromaDB.

  • Run the embedder script in your terminal:
python embedder.py

What should I see?

Your terminal should print output like this:

Stored 7 chunks in ChromaDB
Database persisted to ./chroma_data

Sample metadata: {'chunk_index': 0, 'source': 'company_overview.txt', 'total_chunks': 4}

The exact chunk count depends on your document sizes. The sample metadata confirms that provenance data (which file each chunk came from) was preserved through the pipeline.

  • Confirm the chroma_data/ directory was created by running:
ls chroma_data

You should see files and folders inside chroma_data/. This is your persistent vector database on disk.

Seeing an authentication error?

If you see an error mentioning an invalid API key, open your .env file and confirm OPENAI_API_KEY is set to a valid key from your OpenAI account. Make sure there are no extra spaces or quotes around the key value.

If you see a module not found error for chromadb, make sure your virtual environment is active. Your terminal prompt should show (.venv) at the beginning.

Help me fix my embedder.py errors.

Understand the architecture so far

What are embeddings?

The embedder converts each text chunk into a vector of 1536 numbers. These numbers capture the semantic meaning of the text. Chunks about similar topics end up with similar vectors, even if they use completely different words.

ChromaDB stores these vectors and enables fast similarity search. When you query later, your question gets embedded into the same vector space. ChromaDB then finds the chunks whose vectors are closest to your question's vector. This is the "index" layer of RAG.

Why upsert instead of add?

The upsert method inserts new records or updates existing ones if the ID already exists. This means you can safely re-run the script without creating duplicate entries. The add method would throw an error if you tried to add a chunk with an ID that already exists.

How much does this cost?

Embedding ~3000 words costs approximately $0.001 with text-embedding-3-small ($0.02 per 1M tokens). For this project's small document set, the total embedding cost is effectively zero.

Your documents are now embedded and stored as searchable vectors. Next up, you will build a retrieval pipeline that takes a natural language question and finds the most relevant chunks from your vector database.

Build the Retrieval Pipeline

You have a vector database full of embedded document chunks. The next question is: how do you find the right chunks when someone asks a question?

This is where retrieval comes in. You will build a module that takes a natural language question, converts it into the same vector space as your stored chunks, and finds the closest matches using semantic similarity.

In this step, get ready to:
  • Create a retriever module that queries your ChromaDB vector store.
  • Test semantic search to see how it finds relevant passages even when your query uses different words than the source.
Create the retriever module

The retriever is the bridge between a user's question and the stored knowledge in your vector database. It embeds the query using the same model that embedded your documents, then uses ChromaDB to find the nearest vectors.

  • Create a new file called retriever.py in your rag-system folder.
  • Add the imports and connection function by pasting this code:
import os

import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
from dotenv import load_dotenv

load_dotenv()


def get_collection():
    """Connect to the existing ChromaDB collection."""
    client = chromadb.PersistentClient(path="./chroma_data")

    embedding_function = OpenAIEmbeddingFunction(
        api_key=os.getenv("OPENAI_API_KEY"),
        model_name="text-embedding-3-small",
    )

    collection = client.get_or_create_collection(
        name="rag_documents",
        embedding_function=embedding_function,
    )

    return collection

What does this code do?

This sets up the connection to the same ChromaDB collection you built in the previous step.

  • PersistentClient(path="./chroma_data") connects to the existing database on disk rather than creating a new one.
  • The OpenAIEmbeddingFunction must use the same model (text-embedding-3-small) that embedded the documents. Mismatched models produce incompatible vectors.
  • get_or_create_collection retrieves the existing collection by name. If it does not exist, it creates one.

Now you need the function that actually performs the search. The retrieve() function takes a natural language query, passes it to ChromaDB, and returns the most similar chunks.

  • Add the retrieve function below get_collection():
def retrieve(query, n_results=3):
    """Retrieve the most relevant chunks for a given query."""
    collection = get_collection()

    results = collection.query(query_texts=[query], n_results=n_results)

    retrieved_chunks = []
    for i in range(len(results["ids"][0])):
        retrieved_chunks.append({
            "id": results["ids"][0][i],
            "text": results["documents"][0][i],
            "metadata": results["metadatas"][0][i],
            "distance": results["distances"][0][i],
        })

    return retrieved_chunks

What does this code do?

This is where semantic search happens.

  • collection.query(query_texts=[query], n_results=n_results) embeds your query using the same model, then finds the 3 closest vectors in the database.
  • The results come back as parallel lists (ids, documents, metadatas, distances). The loop zips them into a cleaner list of dictionaries.
  • Lower distance scores mean higher relevance. A distance of 0 would be a perfect match.

Why does ChromaDB handle embedding the query automatically?

When you created the collection with an embedding_function, ChromaDB uses that function for both storing documents and querying. You pass in raw text, and ChromaDB converts it to a vector behind the scenes. This ensures your query lives in the same vector space as your documents.

  • Save retriever.py.
Test semantic search

The power of semantic search is that it finds relevant passages even when your question uses completely different words than the source text. Let's add a test block and run it.

  • Add the following test code at the bottom of retriever.py:
if __name__ == "__main__":
    query = "What is the battery capacity of the SolarSync system?"
    print(f"Query: {query}\n")

    results = retrieve(query)
    for i, chunk in enumerate(results):
        print(f"--- Result {i + 1} (distance: {chunk['distance']:.4f}) ---")
        print(f"Source: {chunk['metadata']['source']}")
        print(f"Text: {chunk['text'][:200]}...")
        print()

What does this code do?

This test block runs a sample query about battery capacity. It prints the top 3 results with their distance scores (relevance) and the source file each chunk came from. You can see how ChromaDB ranks results by semantic closeness to your question.

  • Save retriever.py.
  • Run the retriever by executing this command in your terminal:
python retriever.py
  • You should see 3 results printed, each with a distance score and the source filename. The chunk from product_specs.txt about the PowerVault battery should appear with the lowest distance (highest relevance).

Not seeing any results?

  • Make sure you ran embedder.py in the previous step. The chroma_data/ directory must already exist with data.
  • Confirm your virtual environment is active and your .env file contains a valid API key.
  • Check that your retriever.py file is saved in the same rag-system folder as embedder.py.

Help me debug my retriever.py results

Try experimenting with different queries. Ask something that uses different words than the source material.

  • Try changing the query string to something like "How much does the system cost?" or "Who runs the company?" and run the script again.
  • Notice how the retriever finds relevant chunks even though your question doesn't use the exact words from the documents. This is semantic similarity in action.

What is semantic similarity?

Traditional keyword search looks for exact word matches. Semantic similarity compares the meaning of text by measuring the distance between vectors. "battery capacity" and "energy storage" are far apart in keyword search but close together in vector space.

Lower distance values mean higher relevance. You can also try changing n_results to 1 or 5 to see how the number of retrieved chunks affects precision.

✔️ Awesome, I've got everything!

Great. Double check you have saved retriever.py before moving on.

ⓧ I'd like to double check the full code

import os

import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
from dotenv import load_dotenv

load_dotenv()


def get_collection():
    """Connect to the existing ChromaDB collection."""
    client = chromadb.PersistentClient(path="./chroma_data")

    embedding_function = OpenAIEmbeddingFunction(
        api_key=os.getenv("OPENAI_API_KEY"),
        model_name="text-embedding-3-small",
    )

    collection = client.get_or_create_collection(
        name="rag_documents",
        embedding_function=embedding_function,
    )

    return collection


def retrieve(query, n_results=3):
    """Retrieve the most relevant chunks for a given query."""
    collection = get_collection()

    results = collection.query(query_texts=[query], n_results=n_results)

    retrieved_chunks = []
    for i in range(len(results["ids"][0])):
        retrieved_chunks.append({
            "id": results["ids"][0][i],
            "text": results["documents"][0][i],
            "metadata": results["metadatas"][0][i],
            "distance": results["distances"][0][i],
        })

    return retrieved_chunks


if __name__ == "__main__":
    query = "What is the battery capacity of the SolarSync system?"
    print(f"Query: {query}\n")

    results = retrieve(query)
    for i, chunk in enumerate(results):
        print(f"--- Result {i + 1} (distance: {chunk['distance']:.4f}) ---")
        print(f"Source: {chunk['metadata']['source']}")
        print(f"Text: {chunk['text'][:200]}...")
        print()

Your retrieval pipeline is working. You can now find the most relevant chunks from your documents for any natural language question. Next up, you will connect this retriever to an LLM so it can generate grounded answers using the context you retrieve.

Complete the RAG Pipeline with Generation

Your retrieval pipeline can now find the most relevant chunks from your vector database for any natural language query. The missing piece is generation: taking those retrieved chunks and producing a coherent, grounded answer.

In this step, you will build the generation module that sends retrieved context to OpenAI's gpt-4o-mini model along with a user question. Then you will wire everything together into one command-line application you can run end-to-end.

In this step, get ready to:
  • Create the generation module that retrieves context and calls OpenAI.
  • Build the end-to-end RAG pipeline with an interactive command-line interface.
  • Understand how every component in the architecture connects.
Create the generation module

The generator is the final layer that turns raw retrieved chunks into a human-readable answer. It constructs a prompt that includes the retrieved context, instructs the LLM to answer only from that context, and parses the response.

  • Create a new file called generator.py in your rag-system folder and add the following code:
import os

from dotenv import load_dotenv
from openai import OpenAI

from retriever import retrieve

load_dotenv()

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))


def generate_answer(query, n_results=3):
    """Retrieve context and generate a grounded answer."""
    # Use the retriever to find relevant chunks
    chunks = retrieve(query, n_results=n_results)

    # Build context string with source labels for each chunk
    context_parts = []
    for i, chunk in enumerate(chunks):
        source = chunk["metadata"]["source"]
        context_parts.append(f"[Source: {source}]\n{chunk['text']}")

    context = "\n\n".join(context_parts)

    # Construct the prompt that constrains the LLM to use only retrieved context
    prompt = f"""You are a helpful assistant that answers questions based only on the provided context. 
If the answer cannot be found in the context, say "I don't have enough information to answer that question."
Always cite which source document your answer comes from.

Context:
{context}

Question: {query}

Answer:"""

    # Call OpenAI with low temperature for factual responses
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
    )

    return {
        "answer": response.choices[0].message.content,
        "sources": [chunk["metadata"]["source"] for chunk in chunks],
        "chunks_used": len(chunks),
    }


if __name__ == "__main__":
    query = "Who founded Acme Corporation and when?"
    print(f"Query: {query}\n")

    result = generate_answer(query)
    print(f"Answer: {result['answer']}")
    print(f"\nSources: {', '.join(set(result['sources']))}")

What does this code do?

The generate_answer function ties retrieval to generation. It calls your existing retrieve() function to get relevant chunks, then formats them into a context string with [Source: filename] labels so the LLM knows where each piece of information came from.

The prompt explicitly tells the model to answer only from the provided context and to cite its sources. Setting temperature=0.2 makes the responses factual and consistent rather than creative. The function returns a dictionary with the answer text, source filenames, and how many chunks were used.

  • Save generator.py.
  • Test the generator by running:
python generator.py

You should see the query printed, followed by an answer about Jane Smith and John Doe founding Acme in 2015, with source filenames listed below.

Seeing an authentication error?

Make sure your .env file contains a valid OpenAI API key. The key must have credits available for both embeddings and chat completions.

If you see a module import error for retriever, confirm that retriever.py exists in the same folder as generator.py.

help me debug my generator.py error

✔️ Awesome, I've got everything!

Great. Double check you have saved generator.py before moving on.

ⓧ I'd like to double check the full code

import os

from dotenv import load_dotenv
from openai import OpenAI

from retriever import retrieve

load_dotenv()

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))


def generate_answer(query, n_results=3):
    """Retrieve context and generate a grounded answer."""
    chunks = retrieve(query, n_results=n_results)

    context_parts = []
    for i, chunk in enumerate(chunks):
        source = chunk["metadata"]["source"]
        context_parts.append(f"[Source: {source}]\n{chunk['text']}")

    context = "\n\n".join(context_parts)

    prompt = f"""You are a helpful assistant that answers questions based only on the provided context. 
If the answer cannot be found in the context, say "I don't have enough information to answer that question."
Always cite which source document your answer comes from.

Context:
{context}

Question: {query}

Answer:"""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
    )

    return {
        "answer": response.choices[0].message.content,
        "sources": [chunk["metadata"]["source"] for chunk in chunks],
        "chunks_used": len(chunks),
    }


if __name__ == "__main__":
    query = "Who founded Acme Corporation and when?"
    print(f"Query: {query}\n")

    result = generate_answer(query)
    print(f"Answer: {result['answer']}")
    print(f"\nSources: {', '.join(set(result['sources']))}")
Build the end-to-end pipeline

Right now each module works in isolation. The final piece is an orchestrator script that rebuilds the vector store, then lets you ask questions in an interactive loop.

  • Create a new file called rag.py in your rag-system folder and add the following code:
from embedder import create_vector_store
from generator import generate_answer


def main():
    """Run the complete RAG pipeline interactively."""
    print("=== RAG System ===")
    print("Building vector store from documents...")
    # Re-embed all documents to ensure the vector store is up to date
    create_vector_store()
    print("\nReady! Ask questions about your documents.")
    print("Type 'quit' to exit.\n")

    # Interactive question-answer loop
    while True:
        query = input("Question: ").strip()
        if query.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not query:
            continue

        print("\nRetrieving relevant context...")
        result = generate_answer(query)
        print(f"\nAnswer: {result['answer']}")
        print(f"Sources: {', '.join(set(result['sources']))}")
        print(f"Chunks used: {result['chunks_used']}")
        print("-" * 50 + "\n")


if __name__ == "__main__":
    main()

What does this code do?

The main() function first calls create_vector_store() to re-embed all documents. This ensures the vector database is always up to date with whatever is in your documents/ folder.

Then it enters a while True loop that reads user input, passes it through the full RAG pipeline (retrieve + generate), and prints the answer with its sources. Typing quit, exit, or q breaks the loop.

  • Save rag.py.
  • Run the complete pipeline by running:
python rag.py

You should see the vector store build, then a Question: prompt waiting for your input.

  • Try asking a question about your documents, such as: What is the battery capacity of the SolarSync system?.
  • Then try asking something NOT in your documents, such as: What is the capital of France?.
  • The system should respond that it does not have enough information for questions outside your documents.
  • Type quit to exit when you are done testing.

Not getting grounded answers?

If the model answers questions that are not in your documents, check that your prompt in generator.py includes the instruction to answer only from the provided context. The exact phrasing matters.

If you see an empty answer or an error about the chroma_data directory, make sure you ran embedder.py successfully in the previous step. The vector store must exist before the pipeline can query it.

help me debug my RAG pipeline

✔️ Awesome, I've got everything!

Great. Double check you have saved rag.py before moving on.

ⓧ I'd like to double check the full code

from embedder import create_vector_store
from generator import generate_answer


def main():
    """Run the complete RAG pipeline interactively."""
    print("=== RAG System ===")
    print("Building vector store from documents...")
    create_vector_store()
    print("\nReady! Ask questions about your documents.")
    print("Type 'quit' to exit.\n")

    while True:
        query = input("Question: ").strip()
        if query.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not query:
            continue

        print("\nRetrieving relevant context...")
        result = generate_answer(query)
        print(f"\nAnswer: {result['answer']}")
        print(f"Sources: {', '.join(set(result['sources']))}")
        print(f"Chunks used: {result['chunks_used']}")
        print("-" * 50 + "\n")


if __name__ == "__main__":
    main()
Understand the complete architecture

How the full RAG pipeline works

Every question you ask flows through six stages: your query gets embedded into a vector, ChromaDB searches for the closest chunk vectors (semantic search), the top-k chunks are retrieved with their metadata, those chunks are formatted into a context block with source labels, the context plus your question are sent to the LLM, and the LLM generates a grounded answer citing its sources.

Each component is modular. You could swap ChromaDB for Pinecone, change the embedding model, use a different LLM, or adjust the chunk size. The architecture stays the same.

What does this cost?

Each query uses the text-embedding-3-small model to embed your question ($0.02 per 1M tokens) and gpt-4o-mini for generation ($0.75 per 1M input tokens). For a small document set and a handful of queries, your total cost stays under $0.01.

Secret mission

Multi-Collection Cross-Document Retrieval

Your RAG system currently searches a single collection. Real-world systems often need to pull information from multiple knowledge bases and synthesize answers across them. Extend your pipeline to query multiple ChromaDB collections, merge results by relevance, and generate cross-collection answers.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with no ongoing costs. The only charges were the one-time OpenAI API calls for embeddings and chat completions (approximately $0.01 total).

Resources you used:

  • rag-system project folder on your Desktop (contains all Python files, documents, .venv, and chroma_data/).
  • OpenAI API key (stored in your .env file and on your OpenAI account).

Keep everything running

No action needed. Choose this if you're still actively building or want to keep querying your documents.

  • Your local ChromaDB vector database persists between runs inside chroma_data/. You can keep adding documents and querying without re-embedding.
  • Your OpenAI API key remains active. You are only charged when you run the scripts and make API calls.
  • The virtual environment (.venv) stays on disk and can be reactivated any time.

Pause - I'll come back to this later

There are no running processes or ongoing costs to pause. Everything is already idle when you close your terminal.

  • Close any terminal windows running the RAG system.
  • Your files, vector database, and virtual environment will remain on disk until you return.
  • When you come back, reactivate your virtual environment and run rag.py to pick up where you left off.

Delete - I don't want to use this again

Remove all project files and reclaim disk space. This removes your code, documents, virtual environment, and the ChromaDB database.

  • Delete the entire project folder by running this command from your Desktop:

macOS

cd ~/Desktop
rm -rf rag-system

Windows

cd ~\Desktop
rmdir /s /q rag-system
  • Confirm the folder is gone by running:
ls

You should no longer see rag-system in the output.

  • Revoke your OpenAI API key if you no longer need it:

Nice Work!

Nice Work!

Nice work! You just built a complete RAG system from scratch, taking documents from raw text all the way to grounded, source-cited LLM answers.

You've learned how to:

  • Build a document ingestion pipeline that loads text files, splits them into overlapping chunks, and preserves source metadata throughout the system.
  • Store and query vector embeddings in a ChromaDB database using OpenAI's text-embedding-3-small model for semantic similarity search.
  • Connect retrieval to generation by constructing context-aware prompts that instruct the LLM to answer only from retrieved chunks and cite its sources.
  • Secret Mission: Extended the system with multi-collection cross-document retrieval that queries multiple ChromaDB collections, filters by distance threshold, and synthesizes cross-collection answers.

Ready to quiz yourself?