Build a Document Q&A System with RAG

Build a RAG pipeline that answers questions from your own documents.

Introduction

30 Second Summary

AI chatbots are powerful, but they have no idea what's in your private documents. Retrieval-Augmented Generation (RAG) is how modern AI products solve this problem by feeding relevant information from your own files into an LLM at query time.

In this project, you will build a Python command-line tool that ingests your own documents, stores them in a vector database, and answers natural-language questions with cited sources.

What You'll Build

A Python CLI application that chunks Markdown files, embeds them into a local ChromaDB vector database, retrieves the most relevant passages for a question, and generates a grounded answer using OpenAI.

By the end of this project, you'll have:

  • A working RAG pipeline that chunks, embeds, and retrieves your documents for semantic search.
  • A local ChromaDB vector database populated with your embedded document data.
  • An interactive terminal Q&A loop that generates grounded answers with source citations.
  • Secret Mission: Add a Gradio web interface so anyone can chat with your documents in a browser.

Are there any prerequisites?

You should be comfortable with basic Python (variables, functions, loops) and know how to run commands in a terminal. No prior AI or machine learning experience is needed.

Set Up Your Environment

Before you can ask an AI questions about your documents, you need a working Python environment with two key libraries installed. Python is the language you will write your RAG application in.

You also need access to the OpenAI API for generating text embeddings and answers. In this step, you will install Python, set up a project folder with an isolated environment, and confirm that your API key connects correctly.

In this step, get ready to:
  • Install and verify Python 3.9+.
  • Create a project folder with a virtual environment and install dependencies.
  • Get an OpenAI API key and confirm it works.
Install Python 3.9+ and verify

This project requires Python version 3.9 or higher. Both of the libraries you will install (openai and chromadb) depend on this minimum version.

  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Check your Python version by running this command:
python3 --version

You should see a version number printed, like Python 3.14.5.

✔️ I see version 3.9 or higher

Your Python install is ready. Move on to the next substep below.

ⓧ I see an older version

Your system has Python installed, but the version is below 3.9. Some macOS systems ship with an older Python by default that shadows newer installs.

  • Download and install the latest Python from python.org/downloads.
  • During installation on Windows, check the box that says Add Python to PATH.
  • Close and reopen your terminal after installing.
  • Run python3 --version again to confirm the new version appears.

macOS shadows older Python

If python3 --version still shows the old version after installing, try running the full path to the new binary (e.g. /Library/Frameworks/Python.framework/Versions/3.14/bin/python3) or use a version manager like pyenv to manage multiple Python versions.

ⓧ Command not found

Python is not installed on your system yet.

  • Download the latest Python installer from python.org/downloads.
  • Run the installer. On Windows, check the box that says Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal after installation finishes.
  • Run python3 --version to confirm it prints a version number of 3.9 or higher.

Windows uses python instead of python3

On Windows, the command is often python --version instead of python3 --version. If python3 is not recognized, try python --version instead.

Create your project folder and install dependencies

A virtual environment keeps your project's packages isolated from other Python projects on your machine. This prevents version conflicts between different projects.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-project
cd rag-project
  • Confirm the folder exists by running:
ls

You should see an empty listing since the folder is brand new.

  • Create and activate a virtual environment by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

What does this do?

The first command creates a venv folder containing a self-contained Python installation. The second command activates it so that any packages you install go into this folder, not your system Python.

You will know it worked when your terminal prompt changes to show (venv) at the beginning of the line.

  • Install the two libraries your project needs by running:
pip install openai==2.40.0 chromadb==1.5.9

What are these libraries?

The openai package lets you call OpenAI's API to create text embeddings and generate answers. The chromadb package is a vector database that stores and searches your document chunks by similarity.

The install may take a minute or two. When it finishes, you should see a line that says Successfully installed followed by a list of packages.

Install failing?

Make sure you see (venv) in your terminal prompt. If you do not, the virtual environment is not active. Re-run the activation command from the step above.

If you see a permissions error, do not use sudo. Activate your virtual environment first, then retry the pip install command.

Help me troubleshoot my pip install error.

Get an OpenAI API key and verify it works

Your RAG system needs an API key to call OpenAI's embedding and chat models. You will store this key in a .env file and then run a quick test to confirm everything connects.

  • Open platform.openai.com in your browser.
  • Sign in or create an account if you do not have one.
  • Click your profile icon in the top-right corner, then click API keys.
  • Click Create new secret key.
  • Give it a name like rag-project and click Create secret key.
  • Copy the key immediately. You will not be able to see it again after closing the dialog.

What is an API key?

An API key is a unique string that identifies your account when your code talks to OpenAI's servers. It works like a password for your application. Keep it secret and never commit it to version control.

Now open your code editor so you can create files in your project folder.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Cursor and press Enter to open it.
  • In Cursor, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the rag-project folder, then click Open.

Use any code editor you prefer

This project uses Cursor as the recommended editor, but VS Code or any text editor that can create files will work fine.

  • Create a new file called .env in the root of your rag-project folder by right-clicking in the file explorer sidebar and selecting New File.
  • Paste the following into .env, replacing your-api-key-here with the key you just copied:
OPENAI_API_KEY=your-api-key-here
  • Save .env.

Next, load your API key into your terminal session so the test script can read it.

  • Back in your terminal (where the virtual environment is active), load the key by running:

macOS

export $(cat .env)

Windows

Open your .env file, copy the value after OPENAI_API_KEY=, and run:

$env:OPENAI_API_KEY="your-api-key-here"

Replace your-api-key-here with your actual key value.

Now create a test script to verify your key and the OpenAI connection work.

  • Create a new file called test_setup.py in your rag-project folder.
  • Paste the following code into test_setup.py:
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

response = client.embeddings.create(
    input="Hello, world!",
    model="text-embedding-3-small"
)

print(f"Success! Embedding has {len(response.data[0].embedding)} dimensions.")

What does this script do?

This script sends the text "Hello, world!" to OpenAI's text-embedding-3-small model and asks it to convert that text into a numeric vector (an embedding).

If the key is valid and the connection works, it prints how many dimensions the embedding has. You should see 1536, which is the default dimension count for this model.

  • Save test_setup.py.
  • Run the test script in your terminal:
python3 test_setup.py

You should see this output:

Success! Embedding has 1536 dimensions.

Seeing an authentication error?

If you see a message about an invalid API key, double-check that your .env file has no extra spaces around the = sign and that you copied the full key.

Make sure you loaded the key into your terminal session using the export command above. You can verify by running echo $OPENAI_API_KEY (macOS) or echo $env:OPENAI_API_KEY (Windows) to check it is set.

Help me fix my OpenAI API key authentication error.

✔️ Awesome, I've got everything!

Great. Double check you have saved both .env and test_setup.py.

ⓧ I'd like to double check the full code

Here is what each file should look like at the end of this step.

.env:

OPENAI_API_KEY=your-api-key-here

test_setup.py:

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

response = client.embeddings.create(
    input="Hello, world!",
    model="text-embedding-3-small"
)

print(f"Success! Embedding has {len(response.data[0].embedding)} dimensions.")

Your environment is ready and confirmed working. Next up, you will create the documents that your RAG system will search through and answer questions about.

Create Your Knowledge Base

Your environment is ready and your API key is working. Now it's time to give your RAG system something to work with.

An LLM on its own has no knowledge of your private documents. In this step, you'll create a small knowledge base of Markdown files and write the Python functions that load and chunk those documents into pieces a vector database can store.

In this step, get ready to:
  • Create a docs/ folder with sample Markdown documents.
  • Write a document loading function that reads all .md files with metadata.
  • Write a chunking function that splits documents into overlapping pieces.
Create a docs/ folder and add sample documents

Your RAG system needs documents to search through. You'll create a folder of Markdown files that act as your private knowledge base.

  • Create the docs/ folder inside your project directory by running this command:
mkdir docs
  • Confirm the folder exists by running:
ls

You should see docs listed alongside your existing files.

  • In your code editor, create a new file called company-policy.md inside the docs/ folder.
  • Paste the following content into company-policy.md:
# Remote Work Policy

## Eligibility
All full-time employees who have completed their 90-day probation period are eligible for remote work. Contractors must get written approval from their team lead.

## Schedule
Employees may work remotely up to 3 days per week. Core hours are 10 AM to 3 PM in the employee's local timezone. All team meetings must be attended during core hours.

## Equipment
The company provides a laptop and one external monitor for remote workers. Employees are responsible for their own internet connection, which must be at least 50 Mbps download speed.

## Communication
All team communication should happen in Slack during work hours. Response time expectation is 30 minutes during core hours. Video must be on during all meetings with more than 2 participants.

## Security
All work must be done on company-provided devices. VPN must be connected when accessing internal systems. Public Wi-Fi is prohibited for accessing sensitive data.

Why this content?

This document has distinct sections with specific facts (dates, speeds, rules). Later, when you ask your RAG system a question like "How many days can I work from home?", it needs to find and return the correct section.

  • Save company-policy.md.
  • Create a second file called product-faq.md inside the docs/ folder.
  • Paste the following content into product-faq.md:
# Product FAQ

## What is AcmeSync?
AcmeSync is a file synchronization tool designed for teams. It keeps your project files in sync across all team members' devices in real-time.

## How much does it cost?
AcmeSync offers three plans:
- Free: Up to 5 users, 10 GB storage
- Pro: Up to 50 users, 500 GB storage, $12/user/month
- Enterprise: Unlimited users, unlimited storage, custom pricing

## What file types are supported?
AcmeSync supports all file types. There is no restriction on file formats. Individual file size limit is 5 GB on Free and Pro plans, and 25 GB on Enterprise.

## How does conflict resolution work?
When two people edit the same file simultaneously, AcmeSync keeps both versions and notifies the team. Users can then choose which version to keep or merge changes manually.

## Is there an API?
Yes, AcmeSync provides a REST API for Pro and Enterprise plans. API documentation is available at docs.acmesync.example.com. Rate limit is 1000 requests per hour for Pro and 10000 for Enterprise.

What does this document do?

This FAQ-style document tests a different format. It contains pricing numbers, feature details, and technical limits. Your RAG system will need to retrieve the right section depending on what question you ask.

  • Save product-faq.md.
  • Confirm both files exist by running:
ls docs

You should see company-policy.md and product-faq.md in the output.

Don't see both files?

Make sure you created the files inside the docs/ folder, not in your project root. Check that the file names end in .md with no extra spaces.

Help me find my docs files.

Write the document loading function

Now you'll create the main Python script that powers your RAG system. The first function it needs is a document loader that finds every Markdown file in your docs/ folder and reads its content.

  • Create a new file called rag.py in your project root (the same level as your docs/ folder).
  • Add the imports and configuration by pasting this code into rag.py:
import os
import sys
from pathlib import Path
from openai import OpenAI

import chromadb

# Configuration
DOCS_DIR = "./docs"
DB_DIR = "./chroma_db"
EMBEDDING_MODEL = "text-embedding-3-small"
CHAT_MODEL = "gpt-5.4-mini"
COLLECTION_NAME = "my_documents"

# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path=DB_DIR)

What does this code do?

The configuration section defines all the key settings in one place. DOCS_DIR points to your Markdown folder. DB_DIR is where ChromaDB will persist your vector data on disk.

The two client initializations create your connections to OpenAI (for generating embeddings) and ChromaDB (for storing and searching those embeddings). PersistentClient means your data survives between script runs.

  • Below the client initialization, add the load_documents function by pasting this code:

def load_documents(docs_dir):
    """Load all markdown files from the docs directory."""
    documents = []
    docs_path = Path(docs_dir)
    for filepath in docs_path.glob("*.md"):
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
        documents.append({"content": content, "source": filepath.name})
    return documents

What does this function do?

This function uses Python's pathlib to scan a directory for all .md files. For each file it finds, it reads the full text content and stores it alongside the filename as metadata.

The source metadata is important. When your system answers a question later, it uses this filename to tell you which document the answer came from.

  • Save rag.py.
  • Test the function by running this command in your terminal:
python3 -c "from rag import load_documents; docs = load_documents('./docs'); print(f'Loaded {len(docs)} documents: {[d[\"source\"] for d in docs]}')"

You should see output confirming 2 documents were loaded with their filenames.

Seeing 0 documents loaded?

Make sure your terminal is in the same folder as rag.py and the docs/ folder. The path ./docs is relative to where you run the script.

Also check that your files end in .md (not .txt or .markdown). The glob pattern *.md only matches that exact extension.

Why isn't my load_documents function finding my files?

Write the chunking function

A full document is too long to embed as one piece. Chunking splits each document into smaller overlapping segments so the system can retrieve just the relevant part when answering a question.

  • In rag.py, add the chunk_text function below load_documents by pasting this code:

def chunk_text(text, chunk_size=500, overlap=100):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append(chunk)
        start += chunk_size - overlap
    return chunks

What does this function do?

This function slides a 500-character window across the text, advancing 400 characters each time (500 minus 100 overlap). Each window captures one chunk.

The 100-character overlap is deliberate. Without it, a sentence split across two chunks might lose its meaning in both. The overlap ensures context around chunk boundaries is preserved.

Why 500 characters with 100 overlap?

Smaller chunks mean more precise retrieval. If a chunk is too large, it might contain information about multiple topics. If it's too small, it might lack enough context to be useful.

500 characters is roughly 2-3 sentences. This hits a good balance for a beginner project. Production systems often use semantic chunking that splits at paragraph boundaries, but character-based chunking keeps the logic simple and transparent.

  • Save rag.py.
  • Test the chunking function by running:
python3 -c "from rag import load_documents, chunk_text; docs = load_documents('./docs'); chunks = chunk_text(docs[0]['content']); print(f'{len(chunks)} chunks from first document'); print(f'First chunk: {chunks[0][:80]}...')"

You should see several chunks created from your first document, with the beginning of the first chunk printed as a preview.

Seeing an import error?

If you see a ModuleNotFoundError for chromadb or openai, make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.

Why am I getting an import error in my RAG script?

✔️ Awesome, I've got everything!

Great. Double check you've saved rag.py and both docs files before moving on.

ⓧ I'd like to double check the full code

Here's the complete rag.py so far. Make sure yours matches:

import os
import sys
from pathlib import Path
from openai import OpenAI

import chromadb

# Configuration
DOCS_DIR = "./docs"
DB_DIR = "./chroma_db"
EMBEDDING_MODEL = "text-embedding-3-small"
CHAT_MODEL = "gpt-5.4-mini"
COLLECTION_NAME = "my_documents"

# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path=DB_DIR)


def load_documents(docs_dir):
    """Load all markdown files from the docs directory."""
    documents = []
    docs_path = Path(docs_dir)
    for filepath in docs_path.glob("*.md"):
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
        documents.append({"content": content, "source": filepath.name})
    return documents


def chunk_text(text, chunk_size=500, overlap=100):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append(chunk)
        start += chunk_size - overlap
    return chunks

Your knowledge base is ready and your loading and chunking functions work. Next up, you'll embed these chunks into vectors and store them in ChromaDB so your system can search through them.

Embed and Store in ChromaDB

In the previous step, you built functions that load and split your Markdown documents into digestible chunks. But text chunks alone aren't searchable by meaning.

To make your documents queryable, you need to convert text into numerical vectors called embeddings. These vectors capture the semantic meaning of text, letting you find relevant chunks by similarity rather than exact keyword matches. In this step, you'll write the functions that embed your chunks and store them in ChromaDB.

In this step, get ready to:
  • Understand how ChromaDB collections organize your embedded data.
  • Write an embedding function that converts text into vectors using OpenAI.
  • Build an ingestion pipeline that chunks, embeds, and stores your documents.
Understand your ChromaDB client and collections

Your rag.py file already initializes a ChromaDB PersistentClient at the top of the file. This client saves your data to disk in the ./chroma_db folder, so your embedded documents persist between script runs.

What is a ChromaDB collection?

A collection is like a table in a traditional database, but designed for vectors. It stores your embeddings alongside the original text and metadata (like which file a chunk came from).

You'll use get_or_create_collection() to access the collection. This method creates it the first time you run ingestion, and reuses the existing one on future runs.

Write the embedding function

Text embeddings are lists of numbers (vectors) that represent the meaning of a piece of text. Texts with similar meanings produce vectors that are close together in mathematical space.

  • In rag.py, find the chunk_text() function. Add the following get_embeddings() function below it:
def get_embeddings(texts):
    """Get embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(input=texts, model=EMBEDDING_MODEL)
    return [item.embedding for item in response.data]

What does this code do?

This function sends a list of text strings to OpenAI's text-embedding-3-small model and gets back a vector for each one.

  • client.embeddings.create() calls the OpenAI embeddings API with your texts and model name.
  • The response contains a data list where each item has an embedding field holding the vector (a list of 1536 numbers).
  • The list comprehension extracts just the embedding vectors from the response.
  • Save rag.py.

Why text-embedding-3-small?

This model costs $0.02 per 1M tokens, making it extremely affordable for small projects. Each vector has 1536 dimensions, which provides good quality for semantic search without being overkill for your use case.

Seeing a RateLimitError?

Free-tier OpenAI accounts have rate limits (100 requests per minute for embeddings). If you're hitting this, wait a few seconds and try again. For this project, you won't come close to the limit with just two documents.

help me debug a RateLimitError from OpenAI embeddings

Write the ingestion pipeline

The ingestion pipeline ties everything together. It loads your documents, chunks them, embeds the chunks, and stores everything in ChromaDB with metadata so you can trace answers back to their source files.

  • In rag.py, add the following ingest() function below get_embeddings():
def ingest():
    """Load documents, chunk them, embed, and store in ChromaDB."""
    documents = load_documents(DOCS_DIR)
    if not documents:
        print("No .md files found in ./docs/ directory.")
        return

    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)

    all_chunks = []
    all_metadata = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadata.append({"source": doc["source"], "chunk_index": i})
            all_ids.append(f"{doc['source']}_chunk_{i}")

    embeddings = get_embeddings(all_chunks)

    collection.upsert(
        ids=all_ids,
        embeddings=embeddings,
        documents=all_chunks,
        metadatas=all_metadata,
    )
    print(f"Ingested {len(all_chunks)} chunks from {len(documents)} documents.")

What does this code do?

  • load_documents(DOCS_DIR) reads all your Markdown files from ./docs.
  • get_or_create_collection() opens the my_documents collection (or creates it if this is the first run).
  • The loop chunks each document and builds three parallel lists: the chunk text, metadata (source filename and chunk position), and a unique ID for each chunk.
  • get_embeddings(all_chunks) sends all chunks to OpenAI in one batch for efficiency.
  • collection.upsert() stores everything in ChromaDB. Using upsert instead of add means re-running ingestion updates existing chunks instead of creating duplicates.

Why upsert instead of add?

If you edit a document and re-run ingestion, upsert overwrites chunks with the same ID. With add, you'd get an error because the IDs already exist. This makes your pipeline safe to run multiple times.

  • Save rag.py.

Seeing an AuthenticationError?

Make sure your OPENAI_API_KEY environment variable is set in your current terminal session. If you opened a new terminal, you'll need to export it again or source your .env file.

help me fix an AuthenticationError with my OpenAI API key

✔️ Awesome, I've got everything!

Great. Double check you've saved rag.py with both new functions.

ⓧ I'd like to double check the full code

Here's the complete rag.py file as it should look at the end of this step:

import os
import sys
from pathlib import Path
from openai import OpenAI

import chromadb

# Configuration
DOCS_DIR = "./docs"
DB_DIR = "./chroma_db"
EMBEDDING_MODEL = "text-embedding-3-small"
CHAT_MODEL = "gpt-5.4-mini"
COLLECTION_NAME = "my_documents"

# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path=DB_DIR)


def load_documents(docs_dir):
    """Load all markdown files from the docs directory."""
    documents = []
    docs_path = Path(docs_dir)
    for filepath in docs_path.glob("*.md"):
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
        documents.append({"content": content, "source": filepath.name})
    return documents


def chunk_text(text, chunk_size=500, overlap=100):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append(chunk)
        start += chunk_size - overlap
    return chunks


def get_embeddings(texts):
    """Get embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(input=texts, model=EMBEDDING_MODEL)
    return [item.embedding for item in response.data]


def ingest():
    """Load documents, chunk them, embed, and store in ChromaDB."""
    documents = load_documents(DOCS_DIR)
    if not documents:
        print("No .md files found in ./docs/ directory.")
        return

    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)

    all_chunks = []
    all_metadata = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadata.append({"source": doc["source"], "chunk_index": i})
            all_ids.append(f"{doc['source']}_chunk_{i}")

    embeddings = get_embeddings(all_chunks)

    collection.upsert(
        ids=all_ids,
        embeddings=embeddings,
        documents=all_chunks,
        metadatas=all_metadata,
    )
    print(f"Ingested {len(all_chunks)} chunks from {len(documents)} documents.")

Your embedding and storage functions are ready. Next up, you'll write the retrieval and generation functions that actually answer questions using your stored knowledge.

Retrieve and Generate Answers

Your documents are chunked and ready to be embedded. Now you need to build the other half of the RAG pipeline: taking a user's question, finding the most relevant chunks, and feeding them to an LLM that generates a grounded answer.

This is where the magic happens. Instead of asking the LLM to answer from its general training data, you'll force it to answer using only the specific text from your documents. This prevents hallucination and lets the model cite its sources.

In this step, get ready to:
  • Write the retrieval function that searches ChromaDB for relevant chunks.
  • Write the generation function that prompts the LLM with retrieved context.
  • Build the interactive Q&A loop with source citations.
Write the retrieval function

When a user asks a question, you need to find the document chunks most likely to contain the answer. The retrieval function converts the question into an embedding and asks ChromaDB for the closest matches.

  • In rag.py, scroll to the bottom of your file (below the ingest() function) and add the retrieval function:
def retrieve(query, k=4):
    """Find the most relevant chunks for a query."""
    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)
    query_embedding = get_embeddings([query])[0]

    results = collection.query(query_embeddings=[query_embedding], n_results=k)

    chunks = []
    for doc, meta in zip(results["documents"][0], results["metadatas"][0]):
        chunks.append({"text": doc, "source": meta["source"]})
    return chunks

What does this code do?

The retrieve() function takes a natural-language question and converts it into a vector embedding using the same model that embedded your documents.

It then calls collection.query() with that embedding to find the 4 most similar chunks (controlled by k=4). ChromaDB handles the similarity math for you. The function returns a list of dictionaries containing each chunk's text and its source filename.

Why does semantic search work here?

When you embed the question "What is the remote work schedule?" and the chunk "Employees may work remotely up to 3 days per week", both end up as nearby points in vector space. The question doesn't need to contain the same keywords as the answer. It just needs to be about the same concept.

  • Save rag.py.
  • You won't see a change yet. The next function will use retrieve() to feed context into the LLM.
Write the generation function

Retrieval alone gives you raw text chunks. The generation function takes those chunks, constructs a prompt that constrains the LLM to answer using only that context, and returns a grounded response.

  • Below the retrieve() function in rag.py, add the generation function:
def generate(query, chunks):
    """Generate an answer using retrieved chunks as context."""
    context = "\n\n".join(
        f"[Source: {c['source']}]\n{c['text']}" for c in chunks
    )

    system_prompt = (
        "You answer questions using ONLY the provided context. "
        "If the context does not contain enough information to answer, "
        "say 'I don't have that information in the provided documents.' "
        "Cite which source file(s) your answer comes from."
    )

    response = client.chat.completions.create(
        model=CHAT_MODEL,
        temperature=0,
        messages=[
            {"role": "system", "content": system_prompt},
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}",
            },
        ],
    )
    return response.choices[0].message.content

What does this code do?

  • The context variable joins all retrieved chunks into a single string, labeling each chunk with its source filename.
  • The system_prompt tells the model to answer ONLY from the provided context. If the answer isn't there, it must say so instead of making something up.
  • Setting temperature=0 makes the output deterministic. The same context and question always produce the same answer.
  • The user message passes both the context and the question to the model in a structured format.

Why is the system prompt so strict?

This is the core of what makes RAG different from a regular chatbot. By constraining the LLM to answer only from retrieved context, you prevent it from hallucinating facts that aren't in your documents. The "cite your sources" instruction also makes it easy to verify answers.

  • Save rag.py.
  • You won't see a change yet. The next chunk adds the interactive loop that ties retrieval and generation together.
Build the interactive Q&A loop

You need a way to actually use your retrieval and generation functions together. The interactive loop reads questions from the terminal, pipes them through the full RAG pipeline, and prints answers with source citations.

  • Below the generate() function in rag.py, add the chat loop:
def chat():
    """Interactive Q&A loop."""
    print("RAG Document Q&A (type 'quit' to exit)")
    print("-" * 40)
    while True:
        query = input("\nYour question: ").strip()
        if query.lower() in ("quit", "exit", "q"):
            break
        if not query:
            continue

        chunks = retrieve(query)
        answer = generate(query, chunks)

        print(f"\nAnswer: {answer}")
        print(f"\nSources: {', '.join(set(c['source'] for c in chunks))}")

What does this code do?

  • The while True loop keeps prompting for questions until the user types quit, exit, or q.
  • Each question runs through retrieve() to find relevant chunks, then generate() to produce a grounded answer.
  • The sources line uses a set() to deduplicate filenames, so each source file appears only once.
  • Now add the entry point at the very bottom of rag.py (below the chat() function):
if __name__ == "__main__":
    if len(sys.argv) > 1 and sys.argv[1] == "ingest":
        ingest()
    else:
        chat()

What does this entry point do?

This routes the script based on command-line arguments. Running python3 rag.py ingest populates the database. Running python3 rag.py with no arguments starts the interactive chat. One file handles both halves of the pipeline.

  • Save rag.py.
  • Your script is now complete. In the next step, you'll run the ingestion command to populate the database and then test the Q&A system with real questions.

Seeing a syntax error when you save?

  • Check that the if __name__ block is at the top level of the file (no indentation), not inside another function.
  • Make sure you have matching quotes around all strings in the system_prompt variable. Each line of the prompt is wrapped in its own pair of double quotes.
  • Verify the f-string in generate() uses curly braces {context} and {query} for variable substitution.

Help me debug a syntax error in my rag.py file

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here's the complete rag.py file with all functions:

import os
import sys
from pathlib import Path
from openai import OpenAI

import chromadb

# Configuration
DOCS_DIR = "./docs"
DB_DIR = "./chroma_db"
EMBEDDING_MODEL = "text-embedding-3-small"
CHAT_MODEL = "gpt-5.4-mini"
COLLECTION_NAME = "my_documents"

# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path=DB_DIR)


def load_documents(docs_dir):
    """Load all markdown files from the docs directory."""
    documents = []
    docs_path = Path(docs_dir)
    for filepath in docs_path.glob("*.md"):
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
        documents.append({"content": content, "source": filepath.name})
    return documents


def chunk_text(text, chunk_size=500, overlap=100):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append(chunk)
        start += chunk_size - overlap
    return chunks


def get_embeddings(texts):
    """Get embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(input=texts, model=EMBEDDING_MODEL)
    return [item.embedding for item in response.data]


def ingest():
    """Load documents, chunk them, embed, and store in ChromaDB."""
    documents = load_documents(DOCS_DIR)
    if not documents:
        print("No .md files found in ./docs/ directory.")
        return

    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)

    all_chunks = []
    all_metadata = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadata.append({"source": doc["source"], "chunk_index": i})
            all_ids.append(f"{doc['source']}_chunk_{i}")

    embeddings = get_embeddings(all_chunks)

    collection.upsert(
        ids=all_ids,
        embeddings=embeddings,
        documents=all_chunks,
        metadatas=all_metadata,
    )
    print(f"Ingested {len(all_chunks)} chunks from {len(documents)} documents.")


def retrieve(query, k=4):
    """Find the most relevant chunks for a query."""
    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)
    query_embedding = get_embeddings([query])[0]

    results = collection.query(query_embeddings=[query_embedding], n_results=k)

    chunks = []
    for doc, meta in zip(results["documents"][0], results["metadatas"][0]):
        chunks.append({"text": doc, "source": meta["source"]})
    return chunks


def generate(query, chunks):
    """Generate an answer using retrieved chunks as context."""
    context = "\n\n".join(
        f"[Source: {c['source']}]\n{c['text']}" for c in chunks
    )

    system_prompt = (
        "You answer questions using ONLY the provided context. "
        "If the context does not contain enough information to answer, "
        "say 'I don't have that information in the provided documents.' "
        "Cite which source file(s) your answer comes from."
    )

    response = client.chat.completions.create(
        model=CHAT_MODEL,
        temperature=0,
        messages=[
            {"role": "system", "content": system_prompt},
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}",
            },
        ],
    )
    return response.choices[0].message.content


def chat():
    """Interactive Q&A loop."""
    print("RAG Document Q&A (type 'quit' to exit)")
    print("-" * 40)
    while True:
        query = input("\nYour question: ").strip()
        if query.lower() in ("quit", "exit", "q"):
            break
        if not query:
            continue

        chunks = retrieve(query)
        answer = generate(query, chunks)

        print(f"\nAnswer: {answer}")
        print(f"\nSources: {', '.join(set(c['source'] for c in chunks))}")


if __name__ == "__main__":
    if len(sys.argv) > 1 and sys.argv[1] == "ingest":
        ingest()
    else:
        chat()

Your RAG script is feature-complete. Next up, you'll populate the vector database and put the system to the test with real questions.

Test Your RAG System

Your RAG pipeline is fully built. Every piece is in place: document loading, chunking, embedding, retrieval, and generation.

Now it's time to run the system end-to-end and prove it works. You'll populate your vector database, ask real questions, and verify that answers come grounded in your documents with source citations.

In this step, get ready to:
  • Run the ingestion pipeline to populate ChromaDB with your document chunks.
  • Ask questions and verify accurate, source-cited answers.
  • Test edge cases to confirm the system refuses to hallucinate.
Run the ingestion pipeline

Before you can ask questions, your documents need to be chunked, embedded, and stored in ChromaDB. The ingest command handles the entire pipeline in one shot.

  • Make sure your virtual environment is still active (you should see (venv) in your terminal prompt).
  • Populate your vector database by running this command:
python3 rag.py ingest

You should see output confirming how many chunks were ingested from your documents. The exact number depends on your document lengths, but expect something like:

Ingested 7 chunks from 2 documents.

What just happened?

The ingest command ran the full ingestion pipeline. It loaded both Markdown files, split them into ~500 character overlapping chunks, sent each chunk to OpenAI's text-embedding-3-small model to get vector embeddings, and stored everything in ChromaDB.

A new chroma_db/ directory now exists in your project folder. This is your persistent vector database.

  • Confirm the database directory was created by running:
ls

You should see chroma_db/ listed alongside docs/, rag.py, and your other project files.

Seeing an API key error?

If you see an authentication error, your OPENAI_API_KEY environment variable may not be loaded. Re-export it by running source .env in your terminal, then try the ingest command again.

If you see "No .md files found", make sure your docs/ folder is in the same directory as rag.py and contains .md files.

Help me debug my RAG ingestion error

Ask questions and verify grounded answers

With your documents embedded and stored, the retrieval and generation functions can now find relevant chunks and produce answers. The interactive Q&A loop lets you chat with your documents in real time.

  • Start the interactive Q&A loop by running:
python3 rag.py

You should see a prompt that says Your question: waiting for your input.

  • Type a question about your documents and press Enter. Try asking:
How many days per week can employees work remotely?

The system retrieves the most relevant chunks from your documents, passes them as context to the LLM, and generates an answer. You should see an answer that mentions "3 days per week" along with a source citation pointing to company-policy.md.

  • Ask another question that pulls from a different document:
How much does AcmeSync Pro cost?

You should see an answer mentioning "$12/user/month" with a source citation pointing to product-faq.md. The answer is grounded entirely in your document content.

Why are source citations important?

Source citations let users verify that an answer actually comes from your documents. Without them, there's no way to distinguish a grounded answer from a hallucination.

The Sources: line at the bottom of each answer shows exactly which files the system retrieved from. This is the "R" in RAG doing its job.

Test edge cases with out-of-scope questions

A well-built RAG system knows when it doesn't have the answer. The system prompt you wrote in the previous step instructs the LLM to refuse to answer when the retrieved context doesn't contain relevant information.

  • In the same Q&A session, ask a question that has nothing to do with your documents:
What is the capital of France?

The system should respond with something like "I don't have that information in the provided documents" instead of answering from its general knowledge. This is the key difference between a RAG system and a regular chatbot.

Why does this matter?

LLMs know the capital of France. But your RAG system is designed to answer only from your documents. By refusing to answer out-of-scope questions, it proves that RAG successfully constrains the model to your data.

This is exactly how enterprise AI assistants work. A customer support bot should only answer from support docs, not from general internet knowledge.

  • Try one more edge case. Ask something partially related but not in your documents:
What is the company's vacation policy?

Your documents cover remote work but not vacation. The system should acknowledge it doesn't have this information rather than guessing.

  • Type quit and press Enter to exit the Q&A loop.

System answering out-of-scope questions anyway?

If the system answers questions that aren't in your documents, check the system_prompt variable in your generate() function. Make sure it includes the instruction to only use the provided context.

Also verify that temperature=0 is set in the client.chat.completions.create() call. Higher temperatures make the model more creative and more likely to go off-script.

My RAG system is hallucinating answers for questions not in my documents

Secret mission

Add a Gradio Web Interface

Your RAG system works great in the terminal, but what if you could share it with anyone through a browser? In this secret mission, you'll add a web-based chat interface using Gradio so anyone on your computer can ask questions about your documents without touching the command line.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. The OpenAI API charges only when you make requests, and no resources are left running after you close your terminal.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Your project folder on the Desktop (contains rag.py, app.py, docs/, chroma_db/, venv/, and .env).
  • Your OpenAI API key (created on platform.openai.com).

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with your RAG system.

  • Your project folder stays on your Desktop, ready to use whenever you want.
  • Your OpenAI API key remains active but only costs money when you make API calls.
  • The chroma_db/ folder persists your vector database on disk, so you can query it again without re-ingesting.

Pause - I'll come back to this later

Stop any running processes to free up your terminal. Your files and database stay intact for next time.

  • If the Gradio server is still running, press Ctrl+C in the terminal to stop it.
  • Deactivate your virtual environment by running:
deactivate
  • Everything else stays in place. When you return, activate the virtual environment again and pick up where you left off.

Delete - I don't want to use this again

Remove all project files and optionally revoke your API key.

  • If the Gradio server is still running, press Ctrl+C in the terminal to stop it.
  • Deactivate your virtual environment by running:
deactivate
  • Delete your project folder from the Desktop. On macOS/Linux, run:
cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]
  • On Windows, run:
cd ~\Desktop
rmdir /s /q [[PROJECT_FOLDER="your project folder name"]]
  • To revoke your OpenAI API key, go to platform.openai.com/api-keys, find the key you created for this project, click the trash icon next to it, and confirm the deletion.

Nice Work!

Nice Work!

You just built the same architecture that powers enterprise AI assistants, customer support bots, and knowledge base search tools.

You've learned how to:

  • Built a complete RAG pipeline from scratch using raw API calls, with no framework abstractions hiding the mechanics.
  • Used ChromaDB as a local vector database to store and semantically search document chunks converted into text embeddings.
  • Engineered grounded prompts that limit LLM responses to retrieved context, preventing hallucination by design.
  • Added a Gradio web interface for browser-based document chat with source citations.

Ready to quiz yourself?