Build a RAG-Powered Q&A System

Build a Python RAG system that answers questions grounded in your own documents.

Introduction

30 Second Summary

Ever asked an LLM a question about your company's internal docs and gotten a completely made-up answer? That's because LLMs can only work with what they were trained on.

In this project, you will build a RAG (Retrieval-Augmented Generation) system in Python that answers questions grounded in your own documents. Instead of hallucinating from training data, your system retrieves real context and cites its sources.

What You'll Build

A Python command-line application that ingests Markdown documents, stores them as vector embeddings in a local ChromaDB database, and uses GPT-4o mini to answer questions based only on your documents.

By the end of this project, you'll have:

  • A document ingestion pipeline that chunks text and embeds it into a ChromaDB vector database.
  • A semantic search retrieval system that finds the most relevant document chunks for any question.
  • An interactive Q&A loop that generates grounded, cited answers using GPT-4o mini.
  • Secret Mission: Add a Streamlit web interface so anyone can ask questions through a browser instead of the terminal.

Are there any prerequisites?

You need basic Python knowledge (variables, functions, loops). No prior AI or machine learning experience is needed.

Before We Start

Before We Start...

Set Up Your Environment

Before you can build a RAG system, you need a working Python environment with the right libraries installed.

In this step, you will set up Python, create an isolated project directory, and connect to the OpenAI API. This gives you the foundation to start building.

In this step, get ready to:
  • Verify Python is installed and create a project with a virtual environment.
  • Install the openai and chromadb packages.
  • Configure your OpenAI API key as an environment variable.
Install Python and create a project

Python is the language you will use to build the entire RAG pipeline. You need version 3.9 or higher installed on your machine.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Check your Python version by running this command:
python3 --version

What should I see?

You should see a version number like Python 3.12.4 or Python 3.14.5. Any version 3.9 or higher works for this project.

✔️ I see version 3.9 or higher

Your Python installation is ready. Move on to the next instruction below.

ⓧ I see a version lower than 3.9

You need to upgrade Python. Download the latest version from the official Python website.

  • Go to python.org/downloads in your browser.
  • Click the large Download Python button for the latest version.
  • Run the installer. On Windows, make sure to check Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal, then run python3 --version again to confirm.

ⓧ Command not found

Python is not installed yet. Download and install it from the official website.

  • Go to python.org/downloads in your browser.
  • Click the large Download Python button for the latest version.
  • Run the installer. On Windows, make sure to check Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal, then run python3 --version again to confirm.
  • On Windows, if python3 still does not work, try python --version instead.

Now open your code editor so you can work in a terminal and edit files side by side.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code (or Cursor if you use Cursor) and press Enter to open it.
  • Open the integrated terminal by pressing Ctrl+` (backtick key, same shortcut on macOS and Windows).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-qa-system
cd rag-qa-system
  • Create a virtual environment by running:
python3 -m venv venv

What is a virtual environment?

A virtual environment is an isolated copy of Python for this project only. Packages you install inside it stay separate from other projects on your machine.

This prevents version conflicts between projects that need different versions of the same library.

  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate
  • Confirm the virtual environment is active by checking that your terminal prompt now starts with (venv).

Don't see (venv) in your prompt?

On Windows, you may need to allow script execution first. Run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned in PowerShell, then try activating again.

On macOS, if you see a "permission denied" error, make sure you typed source before the path.

Help me activate my Python virtual environment.

Install dependencies

Your RAG system needs two main libraries. The openai package connects to OpenAI's embedding and chat models. The chromadb package provides the local vector database that stores your document chunks.

  • Install both packages by running this command:
pip install openai chromadb

What does this install?

The openai package is the official Python SDK for calling OpenAI's API. It handles authentication, request formatting, and response parsing.

The chromadb package is a lightweight vector database that runs entirely on your machine. It stores document embeddings and finds semantically similar chunks when you search.

  • Verify the installation succeeded by running:
pip show openai chromadb
  • You should see package details including a Name: and Version: line for each package.

Seeing an installation error?

Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt. If not, re-run the activation command from the previous substep.

If you see a permissions error, do not use sudo. Deactivate and recreate the virtual environment instead.

Help me fix my pip install error.

Configure your OpenAI API key

The OpenAI API requires an API key for authentication. You will generate one from the OpenAI dashboard and store it as an environment variable so the Python SDK can find it automatically.

  • Go to platform.openai.com in your browser.
  • Sign up for a free account if you do not have one, or log in to your existing account.
  • Navigate to the API Keys page in your dashboard settings.
  • Click Create new secret key.
  • Give it a name like rag-project and click Create secret key.
  • Copy the key immediately. You will not be able to see it again after closing the dialog.

Why store the key as an environment variable?

The OpenAI Python SDK automatically reads from the OPENAI_API_KEY environment variable when you create a client with OpenAI(). This keeps your key out of your code, so you never accidentally commit it to version control.

  • Back in your VS Code terminal (with the virtual environment still active), set the environment variable by running the command for your operating system:

macOS

export OPENAI_API_KEY="your_api_key_here"

Windows

setx OPENAI_API_KEY "your_api_key_here"
  • Replace your_api_key_here with the actual key you copied from the OpenAI dashboard.

Keep your key for the session

On macOS, the export command sets the variable for your current terminal session only. If you close the terminal, you will need to export it again. To make it permanent, add the export line to your ~/.zshrc file.

On Windows, setx saves the variable permanently but it only takes effect in new terminal sessions. Close and reopen your terminal after running it.

  • Verify the key is set by running this quick test:

macOS

echo $OPENAI_API_KEY

Windows

echo %OPENAI_API_KEY%
  • You should see your API key printed in the terminal. If you see a blank line, re-run the export or setx command.

Seeing a blank output?

Make sure you did not include extra spaces around the = sign in the export command.

On Windows with setx, remember to close and reopen your terminal before the variable is available.

Help me set my OpenAI API key as an environment variable.

Your environment is ready. You have Python, a virtual environment, the required packages, and a connected API key. Next up, you will create the documents that your RAG system will search through.

Create Your Knowledge Base

Your environment is ready and your packages are installed. Now you need something for your RAG system to search through.

A RAG system is only as good as its knowledge base. In this step, you'll create a set of sample documents that contain information a general-purpose LLM wouldn't know. This is where RAG proves its value.

In this step, get ready to:
  • Create a docs/ folder with three sample Markdown files.
  • Understand why document choice matters for RAG.
Create your sample documents

Your RAG system needs a set of documents to ingest and search. You'll create three Markdown files representing a fictional company's internal knowledge.

  • Create a docs folder inside your project directory by running this command:
mkdir docs
  • Confirm the folder exists by running:
ls

You should see docs listed in the output.

  • In VS Code, right-click the docs folder in the Explorer sidebar and select New File.
  • Name the file company-policies.md and paste the following content:
# Remote Work Policy

All employees may work remotely up to 3 days per week. Remote work days must be coordinated with your team lead at least 24 hours in advance. Employees are expected to be available on Slack during core hours (10am-4pm local time).

# Expense Reimbursement

Employees can submit expense reports for work-related purchases up to $500 without manager approval. Purchases over $500 require written approval from your direct manager before the purchase is made. All expense reports must be submitted within 30 days of the purchase date.

# Vacation Policy

Full-time employees receive 20 days of paid time off per year. PTO requests must be submitted at least 2 weeks in advance for periods longer than 3 days. Unused PTO does not roll over to the next calendar year.

What does this document cover?

This file contains HR policies like remote work rules, expense limits, and vacation allowances. These are internal details that a general-purpose LLM has no way of knowing.

  • Save company-policies.md.
  • Create another new file in the docs folder named tech-stack.md and paste this content:
# Frontend Architecture

Our web application uses React 18 with TypeScript. We use Tailwind CSS for styling and Zustand for state management. All new components should be functional components using hooks.

# Backend Services

The backend is built with Python using FastAPI. We use PostgreSQL as our primary database and Redis for caching. All API endpoints must include OpenAPI documentation.

# Deployment

We deploy to AWS using Terraform for infrastructure-as-code. The CI/CD pipeline runs on GitHub Actions. Production deployments happen every Tuesday and Thursday after passing all automated tests.

Why is this useful for RAG?

Engineering decisions like which database, framework, or deployment schedule a company uses are proprietary. An LLM can't answer "what database do we use?" without this document as context.

  • Save tech-stack.md.
  • Create one more file in docs named onboarding.md and paste this content:
# First Week Checklist

New employees should complete the following in their first week:
1. Set up your development environment using the setup guide in Confluence
2. Complete security training in the LMS portal
3. Schedule 1-on-1 meetings with your team members
4. Read the architecture decision records (ADRs) in the docs folder

# Communication Channels

We use Slack for day-to-day communication. The #engineering channel is for technical discussions. The #general channel is for company-wide announcements. Direct messages should be used sparingly - prefer public channels so others can benefit from the discussion.

# Code Review Process

All code changes require at least two approvals before merging. Reviews should be completed within 24 hours of being requested. Use conventional comments (e.g., "nit:", "suggestion:", "blocking:") to indicate the severity of your feedback.

What makes this document valuable?

Onboarding procedures, communication norms, and code review rules are specific to one organization. When you ask your RAG system "how does code review work here?", this document provides the grounded answer.

  • Save onboarding.md.
  • Confirm all three files exist by running this command in your terminal:
ls docs/

You should see company-policies.md, onboarding.md, and tech-stack.md in the output.

Don't see all three files?

Make sure each file was created inside the docs folder, not in the project root. Check the VS Code Explorer sidebar to confirm the files are nested under docs/.

Also verify each filename ends with .md (not .txt or no extension). The ingestion code in the next step looks specifically for .md files.

Why aren't my files showing up in the docs folder?

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved before moving on.

ⓧ I'd like to double check the full code

Here are all three files for comparison:

# Remote Work Policy

All employees may work remotely up to 3 days per week. Remote work days must be coordinated with your team lead at least 24 hours in advance. Employees are expected to be available on Slack during core hours (10am-4pm local time).

# Expense Reimbursement

Employees can submit expense reports for work-related purchases up to $500 without manager approval. Purchases over $500 require written approval from your direct manager before the purchase is made. All expense reports must be submitted within 30 days of the purchase date.

# Vacation Policy

Full-time employees receive 20 days of paid time off per year. PTO requests must be submitted at least 2 weeks in advance for periods longer than 3 days. Unused PTO does not roll over to the next calendar year.
# Frontend Architecture

Our web application uses React 18 with TypeScript. We use Tailwind CSS for styling and Zustand for state management. All new components should be functional components using hooks.

# Backend Services

The backend is built with Python using FastAPI. We use PostgreSQL as our primary database and Redis for caching. All API endpoints must include OpenAPI documentation.

# Deployment

We deploy to AWS using Terraform for infrastructure-as-code. The CI/CD pipeline runs on GitHub Actions. Production deployments happen every Tuesday and Thursday after passing all automated tests.
# First Week Checklist

New employees should complete the following in their first week:
1. Set up your development environment using the setup guide in Confluence
2. Complete security training in the LMS portal
3. Schedule 1-on-1 meetings with your team members
4. Read the architecture decision records (ADRs) in the docs folder

# Communication Channels

We use Slack for day-to-day communication. The #engineering channel is for technical discussions. The #general channel is for company-wide announcements. Direct messages should be used sparingly - prefer public channels so others can benefit from the discussion.

# Code Review Process

All code changes require at least two approvals before merging. Reviews should be completed within 24 hours of being requested. Use conventional comments (e.g., "nit:", "suggestion:", "blocking:") to indicate the severity of your feedback.
Understand why document choice matters

You just created a knowledge base of internal company information. This is the type of content where RAG delivers the most value.

Why does document choice matter for RAG?

RAG shines when your documents contain domain-specific information that the LLM doesn't have in its training data. Internal company policies, proprietary engineering decisions, and team-specific processes are perfect examples.

If you asked GPT-4o mini "what's our vacation policy?" without RAG, it would either hallucinate an answer or say it doesn't know. With RAG, it retrieves the exact policy from your documents and cites the source.

Use your own documents

After completing this project, try swapping in your own Markdown files. Meeting notes, personal wikis, study guides, or recipe collections all work great. The more specific and niche the content, the more impressive the RAG results will be.

Your knowledge base is ready. Next up, you'll build the ingestion pipeline that reads these documents, splits them into chunks, and stores them as vector embeddings in ChromaDB.

Build the Ingestion Pipeline

You have a knowledge base of Markdown documents ready to go. Now you need a way to feed those documents into a format that an LLM can search through quickly.

In this step, you'll build the ingestion pipeline. This is the code that reads your documents, splits them into small overlapping chunks, converts each chunk into a numerical vector embedding, and stores everything in a ChromaDB vector database on disk.

In this step, get ready to:
  • Write functions that load and chunk your Markdown documents.
  • Embed and store those chunks in a persistent ChromaDB collection.
  • Run the ingestion pipeline and verify your chunks are stored.
Load and chunk documents

Before you can search your documents with AI, you need to break them into smaller pieces called chunks. An LLM has a limited context window, so feeding it entire files is wasteful. Chunking lets you retrieve only the most relevant pieces later.

Why overlapping chunks?

When you split text at a fixed boundary, you might cut a sentence or idea in half. Overlap means each chunk shares some characters with its neighbors, so important context at the boundary is preserved in at least one chunk.

  • Create a new file called rag.py in your project folder by running:
touch rag.py
  • Confirm the file exists by running:
ls

You should see rag.py and the docs/ folder in the output.

  • Open rag.py in VS Code and add the following imports and configuration at the top of the file:
import os
import chromadb
from openai import OpenAI

# --- Configuration ---
DOCS_DIR = "./docs"
CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "my_documents"
EMBEDDING_MODEL = "text-embedding-3-small"
GENERATION_MODEL = "gpt-4o-mini"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 100

client = OpenAI()
chroma_client = chromadb.PersistentClient(path=CHROMA_PATH)

What does this code do?

  • DOCS_DIR points to the folder where your Markdown files live.
  • CHROMA_PATH is where ChromaDB will store its database files on disk.
  • CHUNK_SIZE and CHUNK_OVERLAP control how your documents are split (500 characters per chunk with 100 characters of overlap).
  • client = OpenAI() creates the OpenAI client. It automatically reads your OPENAI_API_KEY environment variable.
  • chroma_client = chromadb.PersistentClient(path=CHROMA_PATH) creates a ChromaDB client that saves data to disk so it persists between runs.
  • Save rag.py.

You won't see a visible change yet. The next chunk adds the functions that use this configuration.

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is activated. You should see the environment name in your terminal prompt.
  • Confirm you installed both packages by running pip list and checking that openai and chromadb appear in the list.

Still stuck? Why am I getting a ModuleNotFoundError for openai or chromadb?

  • Below the configuration section in rag.py, add the load_documents function:

# --- Ingestion ---
def load_documents(directory):
    """Load all markdown files from a directory."""
    documents = []
    for filename in os.listdir(directory):
        if filename.endswith(".md"):
            filepath = os.path.join(directory, filename)
            with open(filepath, "r") as f:
                content = f.read()
            documents.append({"content": content, "source": filename})
    return documents

What does this code do?

  • load_documents() scans a directory for files ending in .md.
  • For each Markdown file, it reads the full text content and stores it alongside the filename as metadata.
  • The result is a list of dictionaries, each with a "content" key (the text) and a "source" key (the filename). The source is how you'll cite answers later.
  • Save rag.py.

You won't see a change yet. The next function uses these loaded documents.

  • Directly below load_documents, add the chunk_text function:

def chunk_text(text, size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start += size - overlap
    return chunks

What does this code do?

  • chunk_text() takes a string and slides a window of 500 characters across it.
  • After each window, it moves forward by 400 characters (size minus overlap), leaving 100 characters of overlap with the previous chunk.
  • This ensures no sentence gets completely lost at a chunk boundary.
  • Save rag.py.

You won't see a visible change yet. The next substep adds embedding and storage.

Embed and store chunks in ChromaDB

Your chunks are just strings of text right now. To search them by meaning (not just keywords), you need to convert them into vector embeddings. These are arrays of numbers that capture the semantic meaning of the text.

  • Below chunk_text in rag.py, add the get_embeddings function:

def get_embeddings(texts):
    """Generate embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(
        model=EMBEDDING_MODEL,
        input=texts
    )
    return [item.embedding for item in response.data]

What does this code do?

  • get_embeddings() sends a list of text strings to OpenAI's text-embedding-3-small model.
  • The model returns a 1536-dimensional vector for each text. Texts with similar meanings produce vectors that are close together in this high-dimensional space.
  • This costs $0.02 per 1 million tokens. Your three small documents will cost a fraction of a cent.
  • Save rag.py.

You won't see a change yet. The next function ties everything together.

  • Below get_embeddings, add the ingest_documents function that orchestrates the full pipeline:

def ingest_documents():
    """Load, chunk, embed, and store documents in ChromaDB."""
    documents = load_documents(DOCS_DIR)
    if not documents:
        print(f"No markdown files found in {DOCS_DIR}/")
        return

    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)

    all_chunks = []
    all_metadatas = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadatas.append({"source": doc["source"]})
            all_ids.append(f"{doc['source']}_chunk_{i}")

    embeddings = get_embeddings(all_chunks)

    collection.add(
        documents=all_chunks,
        embeddings=embeddings,
        metadatas=all_metadatas,
        ids=all_ids
    )

    print(f"Ingested {len(all_chunks)} chunks from {len(documents)} documents.")

What does this code do?

  • ingest_documents() is the orchestrator. It calls load_documents(), then chunks each document, then embeds all chunks in one API call.
  • get_or_create_collection() creates the ChromaDB collection if it doesn't exist, or returns the existing one. This prevents errors if you run ingestion more than once.
  • Each chunk gets a unique ID built from the filename and chunk index (e.g. company-policies.md_chunk_0). This lets ChromaDB track each piece individually.
  • collection.add() stores the chunks, their embeddings, and their metadata all at once.
  • Save rag.py.

You won't see a change yet. The next substep adds a way to run this pipeline from the command line.

Run the ingestion

You've written all the ingestion logic. Now you need a way to trigger it from the terminal so you can verify it works.

  • At the bottom of rag.py, add the main block:

# --- Main ---
if __name__ == "__main__":
    import sys

    if len(sys.argv) > 1 and sys.argv[1] == "ingest":
        ingest_documents()

What does this code do?

  • if __name__ == "__main__" ensures this block only runs when you execute the file directly (not when importing from it later).
  • sys.argv[1] == "ingest" checks if you passed the word ingest as a command-line argument.
  • This pattern lets you add more commands later (like a Q&A mode) without changing the ingestion logic.
  • Save rag.py.
  • Run the ingestion pipeline by executing:
python rag.py ingest

You should see output similar to:

Ingested 9 chunks from 3 documents.

The exact chunk count may vary slightly depending on your document lengths. The important thing is that it reports chunks from 3 documents without errors.

What just happened?

Your script read all three Markdown files, split them into ~500-character overlapping chunks, sent those chunks to OpenAI to generate vector embeddings, and stored everything in a new chroma_db/ folder on disk. That folder is now your searchable vector database.

  • Confirm the database was created by running:
ls

You should see a new chroma_db/ folder alongside your docs/ folder and rag.py.

Seeing an AuthenticationError?

  • This means your OPENAI_API_KEY environment variable is not set or is invalid.
  • Make sure you exported the key in the same terminal session where you're running the script.
  • If you opened a new terminal, you need to set the variable again.

Still stuck? Why is my OpenAI API key not being recognized when I run rag.py?

Seeing 'Add of existing embedding ID' warnings?

This happens if you run the ingestion command a second time. ChromaDB is warning you that chunks with those IDs already exist. Your data is fine. To re-ingest cleanly, delete the chroma_db/ folder first and run the command again.

✔️ Awesome, I've got everything!

Great. Double check you've saved rag.py and that the chroma_db/ folder exists in your project directory.

ⓧ I'd like to double check the full code

Here is the complete rag.py up to this point:

import os
import chromadb
from openai import OpenAI

# --- Configuration ---
DOCS_DIR = "./docs"
CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "my_documents"
EMBEDDING_MODEL = "text-embedding-3-small"
GENERATION_MODEL = "gpt-4o-mini"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 100

client = OpenAI()
chroma_client = chromadb.PersistentClient(path=CHROMA_PATH)


# --- Ingestion ---
def load_documents(directory):
    """Load all markdown files from a directory."""
    documents = []
    for filename in os.listdir(directory):
        if filename.endswith(".md"):
            filepath = os.path.join(directory, filename)
            with open(filepath, "r") as f:
                content = f.read()
            documents.append({"content": content, "source": filename})
    return documents


def chunk_text(text, size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start += size - overlap
    return chunks


def get_embeddings(texts):
    """Generate embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(
        model=EMBEDDING_MODEL,
        input=texts
    )
    return [item.embedding for item in response.data]


def ingest_documents():
    """Load, chunk, embed, and store documents in ChromaDB."""
    documents = load_documents(DOCS_DIR)
    if not documents:
        print(f"No markdown files found in {DOCS_DIR}/")
        return

    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)

    all_chunks = []
    all_metadatas = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadatas.append({"source": doc["source"]})
            all_ids.append(f"{doc['source']}_chunk_{i}")

    embeddings = get_embeddings(all_chunks)

    collection.add(
        documents=all_chunks,
        embeddings=embeddings,
        metadatas=all_metadatas,
        ids=all_ids
    )

    print(f"Ingested {len(all_chunks)} chunks from {len(documents)} documents.")


# --- Main ---
if __name__ == "__main__":
    import sys

    if len(sys.argv) > 1 and sys.argv[1] == "ingest":
        ingest_documents()

Your documents are now embedded and searchable. Next up, you'll build the retrieval and generation pipeline that finds relevant chunks and uses them to answer questions.

Build the Retrieval and Generation Pipeline

Your documents are chunked and embedded in ChromaDB. The ingestion half of the RAG pipeline is done.

Now you need the other half: the part that takes a user's question, finds the most relevant chunks, and feeds them to an LLM as context. This is where your system stops being a database and starts being an intelligent Q&A tool.

In this step, get ready to:
  • Write a retrieval function that finds the most relevant chunks for any question.
  • Write a generation function that sends those chunks to GPT-4o mini for a grounded answer.
  • Build an interactive Q&A loop in the terminal.
Write the retrieval function

When a user asks a question, you need to find which chunks in the database are most relevant. This is semantic search. You embed the question into a vector, then ask ChromaDB to find the closest stored vectors.

  • In rag.py, find the # --- Main --- comment near the bottom of your file.
  • Add the following retrieval function directly above that comment:
# --- Retrieval ---
def retrieve(query, n_results=3):
    """Find the most relevant chunks for a query."""
    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)
    query_embedding = get_embeddings([query])[0]

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results
    )

    return [
        {"text": doc, "source": meta["source"]}
        for doc, meta in zip(results["documents"][0], results["metadatas"][0])
    ]

What does this code do?

  • get_or_create_collection connects to your existing ChromaDB collection where the ingested chunks live.
  • get_embeddings([query])[0] converts the user's question into a vector using the same embedding model you used for ingestion.
  • collection.query() performs a similarity search and returns the 3 closest chunks along with their metadata.
  • The function returns a clean list of dictionaries with the chunk text and which file it came from.

Why use the same embedding model for queries?

Vectors can only be meaningfully compared if they were generated by the same model. If you embed your documents with text-embedding-3-small but embed your question with a different model, the similarity scores would be meaningless.

  • Save rag.py.
  • You won't see a change yet. The next function will use retrieve() to build a complete answer.
Write the generation function

Retrieval alone gives you raw text chunks. The generation function takes those chunks, formats them into a prompt, and asks GPT-4o mini to synthesize an answer using only that context.

  • In rag.py, add the following function directly below the retrieve() function you just wrote (and above the # --- Main --- comment):
# --- Generation ---
def answer(query):
    """Retrieve context and generate a grounded answer."""
    chunks = retrieve(query)
    context = "\n\n".join(
        f"[{i+1}] {c['text']}" for i, c in enumerate(chunks)
    )
    sources = "\n".join(
        f"[{i+1}] {c['source']}" for i, c in enumerate(chunks)
    )

    response = client.chat.completions.create(
        model=GENERATION_MODEL,
        temperature=0,
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant that answers questions based "
                    "only on the provided context. If the answer is not in the "
                    "context, reply exactly: \"I don't have that information in "
                    "the provided documents.\" Cite sources using [1], [2], etc."
                ),
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}",
            },
        ],
    )

    answer_text = response.choices[0].message.content
    return f"{answer_text}\n\nSources:\n{sources}"

What does this code do?

  • retrieve(query) fetches the top 3 relevant chunks from ChromaDB.
  • The context variable numbers each chunk with [1], [2], [3] so the LLM can cite them.
  • The system prompt instructs the model to answer only from the provided context. If the answer isn't there, it must say exactly "I don't have that information in the provided documents."
  • temperature=0 prevents the model from being creative. You want factual answers, not embellishments.
  • The function appends source filenames to the answer so the user knows where the information came from.

Why is the "I don't know" instruction important?

Without this instruction, the LLM would happily make up answers when the retrieved chunks don't contain the answer. This is called hallucination. The explicit "say you don't know" rule is what makes RAG trustworthy.

  • Save rag.py.
  • You won't see a change yet. The next chunk builds the loop that connects everything.
Build the interactive Q&A loop

The retrieval and generation functions are ready. Now you need a way to actually use them. You'll update the main block to include a terminal loop that accepts questions and prints answers.

  • In rag.py, scroll down to the # --- Main --- section at the bottom.
  • Find the existing if __name__ == "__main__": block.
  • Replace the entire main block with this updated version that adds the Q&A loop:
# --- Main ---
if __name__ == "__main__":
    import sys

    if len(sys.argv) > 1 and sys.argv[1] == "ingest":
        ingest_documents()
    else:
        print("RAG Q&A System (type 'quit' to exit)")
        print("-" * 40)
        while True:
            query = input("\nQuestion: ")
            if query.lower() in ("quit", "exit"):
                break
            print(f"\n{answer(query)}")

What does this code do?

  • The if sys.argv[1] == "ingest" branch still handles ingestion when you run python rag.py ingest.
  • The new else branch starts a loop that prompts for a question, passes it through answer(), and prints the result with sources.
  • Typing quit or exit breaks out of the loop cleanly.
  • Save rag.py.
  • Test that the Q&A loop starts by running this command in your terminal:
python rag.py

You should see a header that says RAG Q&A System (type 'quit' to exit) followed by a Question: prompt waiting for your input.

  • Type quit and press **Enter** to exit for now. You'll test it properly in the next step.

Not seeing the Question prompt?

  • Make sure you saved rag.py after making changes.
  • Check that the else: block is indented at the same level as the if len(sys.argv) line above it.
  • If you see an error about OPENAI_API_KEY, make sure your environment variable is still set in this terminal session.

Why isn't my RAG Q&A loop starting when I run python rag.py?

✔️ Awesome, I've got everything!

Great. Double check you've saved rag.py and you're ready to move on.

ⓧ I'd like to double check the full code

Here's the complete rag.py file at this point:

import os
import chromadb
from openai import OpenAI

# --- Configuration ---
DOCS_DIR = "./docs"
CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "my_documents"
EMBEDDING_MODEL = "text-embedding-3-small"
GENERATION_MODEL = "gpt-4o-mini"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 100

client = OpenAI()
chroma_client = chromadb.PersistentClient(path=CHROMA_PATH)


# --- Ingestion ---
def load_documents(directory):
    """Load all markdown files from a directory."""
    documents = []
    for filename in os.listdir(directory):
        if filename.endswith(".md"):
            filepath = os.path.join(directory, filename)
            with open(filepath, "r") as f:
                content = f.read()
            documents.append({"content": content, "source": filename})
    return documents


def chunk_text(text, size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start += size - overlap
    return chunks


def get_embeddings(texts):
    """Generate embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(
        model=EMBEDDING_MODEL,
        input=texts
    )
    return [item.embedding for item in response.data]


def ingest_documents():
    """Load, chunk, embed, and store documents in ChromaDB."""
    documents = load_documents(DOCS_DIR)
    if not documents:
        print(f"No markdown files found in {DOCS_DIR}/")
        return

    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)

    all_chunks = []
    all_metadatas = []
    all_ids = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_metadatas.append({"source": doc["source"]})
            all_ids.append(f"{doc['source']}_chunk_{i}")

    embeddings = get_embeddings(all_chunks)

    collection.add(
        documents=all_chunks,
        embeddings=embeddings,
        metadatas=all_metadatas,
        ids=all_ids
    )

    print(f"Ingested {len(all_chunks)} chunks from {len(documents)} documents.")


# --- Retrieval ---
def retrieve(query, n_results=3):
    """Find the most relevant chunks for a query."""
    collection = chroma_client.get_or_create_collection(name=COLLECTION_NAME)
    query_embedding = get_embeddings([query])[0]

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results
    )

    return [
        {"text": doc, "source": meta["source"]}
        for doc, meta in zip(results["documents"][0], results["metadatas"][0])
    ]


# --- Generation ---
def answer(query):
    """Retrieve context and generate a grounded answer."""
    chunks = retrieve(query)
    context = "\n\n".join(
        f"[{i+1}] {c['text']}" for i, c in enumerate(chunks)
    )
    sources = "\n".join(
        f"[{i+1}] {c['source']}" for i, c in enumerate(chunks)
    )

    response = client.chat.completions.create(
        model=GENERATION_MODEL,
        temperature=0,
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant that answers questions based "
                    "only on the provided context. If the answer is not in the "
                    "context, reply exactly: \"I don't have that information in "
                    "the provided documents.\" Cite sources using [1], [2], etc."
                ),
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}",
            },
        ],
    )

    answer_text = response.choices[0].message.content
    return f"{answer_text}\n\nSources:\n{sources}"


# --- Main ---
if __name__ == "__main__":
    import sys

    if len(sys.argv) > 1 and sys.argv[1] == "ingest":
        ingest_documents()
    else:
        print("RAG Q&A System (type 'quit' to exit)")
        print("-" * 40)
        while True:
            query = input("\nQuestion: ")
            if query.lower() in ("quit", "exit"):
                break
            print(f"\n{answer(query)}")

Your full RAG pipeline is wired up. Next, you'll put it through its paces by asking questions it can and cannot answer.

Test Your RAG System

Your RAG pipeline is fully built. You have ingestion, retrieval, and generation all wired together in a single script.

Now it's time to put it to the test. You'll ask questions the documents CAN answer, questions they CAN'T answer, and then tweak the chunking settings to see how they affect quality.

In this step, get ready to:
  • Test questions your documents can answer.
  • Test questions your documents cannot answer.
  • Experiment with chunk size and overlap settings.
Ask questions your documents can answer

The real test of a RAG system is whether its answers are grounded in your documents with proper citations. If retrieval is working, the system should pull relevant chunks and produce an accurate answer.

  • Start the Q&A loop by running this command:
python rag.py

You should see the system print RAG Q&A System (type 'quit' to exit) followed by a line of dashes.

  • At the Question: prompt, type What is the remote work policy? and press Enter.
  • Read the response. It should mention that employees can work remotely up to 3 days per week, and cite company-policies.md as a source.
  • Try another question: type What database does the backend use? and press Enter.
  • The answer should mention PostgreSQL as the primary database and cite tech-stack.md.

What should I see?

Each answer ends with a Sources: section listing the filenames the system pulled context from. These citations prove the answer is grounded in YOUR documents, not hallucinated from the model's training data.

Seeing an error about your API key?

Make sure your OPENAI_API_KEY environment variable is still set in this terminal session. If you opened a new terminal, you'll need to export it again.

help me fix my OpenAI API key environment variable

Ask questions your documents cannot answer

RAG's biggest advantage over a raw LLM call is hallucination resistance. When the answer isn't in your documents, the system should say so instead of making something up.

  • At the Question: prompt, type What is the company's stock price? and press Enter.
  • The response should say: I don't have that information in the provided documents.
  • Try one more: type Who is the CEO? and press Enter.
  • Again, you should get the same "I don't have that information" response since none of your documents mention a CEO.

Why does this matter?

This is the core value of RAG. A plain GPT-4o mini call would confidently invent a stock price or a CEO name. Your system refuses to answer because the temperature=0 setting and the system prompt instruction work together to keep responses honest.

Experiment with chunk size

Chunk size controls how much text each retrieved result carries. Smaller chunks are more precise but carry less context. Larger chunks carry more context but might include irrelevant information.

  • Type quit in the Q&A loop to exit.
  • In rag.py, find these two lines near the top of the file:
CHUNK_SIZE = 500
CHUNK_OVERLAP = 100
  • Change them to use smaller chunks:
CHUNK_SIZE = 200
CHUNK_OVERLAP = 50

With smaller chunks, your documents will be split into more pieces. Each piece carries less surrounding context.

  • Save rag.py.
  • Delete the existing database so you can re-ingest with the new chunk size.

macOS

  • Run this command to delete the old database:
rm -rf chroma_db

Windows

  • Run this command to delete the old database:
Remove-Item -Recurse -Force chroma_db
  • Re-ingest your documents with the new chunk size by running:
python rag.py ingest

Notice the ingestion now reports more chunks than before. That's because smaller chunks mean more pieces from the same documents.

  • Start the Q&A loop again by running:
python rag.py
  • Ask the same questions you tried earlier, like What is the remote work policy? and What database does the backend use?.
  • Compare the answers to what you saw before. You may notice the responses are more focused but occasionally miss broader context that lived in a larger chunk.

How does chunk size affect answers?

Smaller chunks (like 200 characters) retrieve more targeted snippets. This works well for short factual questions.

Larger chunks (like 500 or 1000 characters) retrieve more surrounding context. This works better for questions that need broader understanding. There's no single "best" size. It depends on your documents and the types of questions you expect.

  • Type quit to exit the Q&A loop.
  • In rag.py, find the chunk configuration lines and set them back to the original values:
CHUNK_SIZE = 500
CHUNK_OVERLAP = 100
  • Save rag.py.
  • Delete the database and re-ingest one more time to restore the original chunks.

macOS

  • Run these commands to reset your database:
rm -rf chroma_db
python rag.py ingest

Windows

  • Run these commands to reset your database:
Remove-Item -Recurse -Force chroma_db
python rag.py ingest

You should see the original chunk count restored in the ingestion output.

Getting fewer or more chunks than expected?

Make sure you changed both CHUNK_SIZE and CHUNK_OVERLAP back to 500 and 100 respectively. Also confirm you deleted the chroma_db folder before re-ingesting. If the folder still exists, the old chunks remain and new chunks won't replace them.

help me fix my chunk count after re-ingesting

Secret mission

Add a Streamlit Web Interface

Your RAG system works great in the terminal, but what if you could share it with teammates who don't use the command line? In this secret mission, you'll add a browser-based interface using Streamlit so anyone can ask questions through a simple web page.

Clean Up Your Resources

Clean Up Your Resources

This project runs locally and the OpenAI API is pay-per-use, so there are no ongoing costs when you stop using it. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder on your Desktop (contains rag.py, app.py, docs/, chroma_db/, and your virtual environment).
  • OpenAI API key stored as an environment variable.

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with different documents and chunk sizes.

  • Your project folder, ChromaDB database, and virtual environment stay on disk ready to use.
  • Your OpenAI API key remains set as an environment variable.
  • To start the Q&A system again, activate your virtual environment and run python rag.py or streamlit run app.py.

Pause - I'll come back to this later

Stop any running processes to free up memory. Your files stay on disk so you can pick up where you left off.

  • If Streamlit is still running, press Ctrl+C in the terminal to stop it.
  • Deactivate your virtual environment by running:
deactivate
  • Your project folder and ChromaDB database remain on disk. No re-ingestion needed when you come back.

Delete - I don't want to use this again

Remove all project files and clean up your environment variable. This frees disk space and removes the API key from your shell.

  • If Streamlit is still running, press Ctrl+C in the terminal to stop it.
  • Deactivate your virtual environment by running:
deactivate
  • Delete the entire project folder by running:

macOS

cd ~/Desktop
rm -rf rag-qa-system

Windows

cd ~\Desktop
rmdir /s /q rag-qa-system
  • Remove the OpenAI API key from your shell profile. If you added it to your ~/.zshrc (macOS) or ~/.bashrc (Linux), open that file and delete the line containing OPENAI_API_KEY.
  • If you only set the key for a single terminal session (without editing your shell profile), it is already gone once you close that terminal.
  • To unset it from your current session, run:

macOS

unset OPENAI_API_KEY

Windows

Remove-Item Env:OPENAI_API_KEY

Nice Work!

Nice Work!

You just built a Retrieval-Augmented Generation system from scratch. Your Python application ingests documents, embeds them into a vector database, and answers natural-language questions grounded in real data.

You've learned how to:

  • Built a document ingestion pipeline that chunks text and embeds it into vectors using OpenAI's text-embedding-3-small model.
  • Stored embeddings in a persistent ChromaDB vector database and implemented semantic search to find relevant context for any question.
  • Connected retrieval to GPT-4o mini for grounded, cited answers that say "I don't know" when the answer isn't in the corpus.
  • Added a Streamlit web interface so anyone can query your RAG system from a browser.

Ready to quiz yourself?