Build a RAG Chatbot in Python

Build a RAG chatbot that answers questions grounded in your own documents.

Introduction

30 Second Summary

Ever asked an AI chatbot a question and gotten a confident-sounding answer that was completely wrong? Real-world AI systems solve this by looking up information in actual documents before answering.

In this project, you will build a command-line chatbot that answers questions using your own documents as its source of truth. You'll learn how retrieval-augmented generation (RAG) works by building the full pipeline from scratch.

What You'll Build

A Python RAG chatbot that ingests text documents, stores them in a local vector database, and generates answers grounded in retrieved context using OpenAI.

By the end of this project, you'll have:

  • A document ingestion pipeline that chunks and embeds text files into a local ChromaDB vector database.
  • A retrieval and answer system that finds the most relevant chunks and generates grounded answers using GPT-4o mini.
  • A clear mental model of the RAG pipeline: chunk, embed, store, retrieve, generate.
  • Secret Mission: Add source citations so the chatbot tells you exactly which chunks it used to answer each question.

Are there any prerequisites?

You'll need basic Python knowledge (variables, functions, loops). No prior AI or machine learning experience is required.

Set Up Your Python Environment

Every tool in your RAG chatbot runs on Python. The OpenAI SDK, ChromaDB, and the scripts you write all need a working Python environment to function.

In this step, you'll install Python, create a dedicated project folder with an isolated virtual environment, and add the two packages that power your chatbot's AI brain and memory.

In this step, get ready to:
  • Install and verify Python 3.10+.
  • Create a project folder and virtual environment.
  • Install the openai and chromadb packages.
Install Python 3.10+ and verify the installation

Python 3.10 or higher is required because the libraries you'll use depend on features introduced in that version.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.10 or higher

You're all set. Move on to the next substep.

ⓧ I see an older version

Your system has an older Python that won't work with this project's dependencies. You need to install Python 3.10 or higher alongside it.

  • Go to https://www.python.org/downloads/ and download the latest Python 3 installer.
  • Run the installer. On Windows, check the box that says Add python.exe to PATH before clicking Install Now.
  • Close and reopen your terminal, then run the version check again.

macOS still showing the old version?

macOS often ships with an older system Python. After installing from python.org, try running python3.10 --version or python3.12 --version to use the specific version you installed. Use whichever version-specific command works for you throughout this project in place of python3.

ⓧ Command not found

Python isn't installed on your machine yet. Let's fix that.

  • Go to https://www.python.org/downloads/ and download the latest Python 3 installer for your operating system.
  • Run the installer. On Windows, check the box that says Add python.exe to PATH before clicking Install Now.
  • Close and reopen your terminal after installation completes.
  • Run the version check command again to confirm it works.
Create your project folder and virtual environment

A virtual environment keeps your project's packages separate from the rest of your system. This prevents version conflicts between different projects on your machine.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-chatbot
cd rag-chatbot
  • Confirm the folder exists by running:
ls ~/Desktop

You should see rag-chatbot in the output.

  • Create a virtual environment inside the project folder by running this command:

macOS

python3 -m venv venv

Windows

python -m venv venv
  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

On Windows PowerShell, you may get a permissions error. Run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser first, then try the activate command again.

On macOS, make sure you used source before the path. Running the path alone won't activate the environment.

Help me activate my Python virtual environment.

What is a virtual environment?

A virtual environment is an isolated copy of Python for one project. Packages you install inside it won't affect other projects on your machine.

When the environment is active, pip install puts packages only in this project's venv folder. Deactivate it later by typing deactivate.

Install the required packages

With the virtual environment active, you'll install the two packages that power your chatbot. The openai package connects to OpenAI's API for embeddings and text generation. The chromadb package gives you a local vector database to store and search your document chunks.

  • Install both packages by running this command:
pip install openai chromadb

What does this command do?

pip install openai chromadb downloads both packages and their dependencies from PyPI (Python's package registry) into your virtual environment. This may take a minute since ChromaDB has several dependencies.

You should see output ending with Successfully installed followed by a list of packages.

Seeing an installation error?

Make sure you see (venv) in your terminal prompt. If it's missing, your virtual environment isn't active. Go back and run the activation command.

If you see a permissions error, you may be installing outside the virtual environment. Never use sudo pip install. Activate the venv first.

Help me fix my pip install error.

  • Open your code editor. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Cursor (or Visual Studio Code) and press Enter to open it.
  • In your editor, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the rag-chatbot folder, then click Open.

You should see rag-chatbot as the root folder in your editor's file explorer sidebar. Inside it, you'll see the venv folder you created.

✔️ Awesome, I've got everything!

Great. Your Python environment, virtual environment, and packages are all ready to go.

ⓧ I'd like to double check my setup

Run this command in your terminal (with the virtual environment active) to confirm both packages are installed:

pip list | grep -E "openai|chromadb"

You should see both openai and chromadb listed with version numbers.

Your Python environment is locked and loaded. Next up, you'll get your OpenAI API key so your chatbot can tap into the power of GPT-4o mini and OpenAI's embedding models.

Get Your OpenAI API Key

Your Python environment is ready with all the packages installed. Now you need an API key so your code can talk to OpenAI's models.

An API key is like a password that identifies your application to OpenAI's servers. Without it, OpenAI won't process your requests for embeddings or chat completions.

In this step, get ready to:
  • Create an OpenAI account and generate an API key.
  • Set the API key as an environment variable.
  • Test the connection with a simple API call.
Create an OpenAI account and generate an API key

OpenAI requires you to create a developer account before you can use their API. You will also need to add a payment method, since embedding and chat completion calls cost a small amount per request.

  • Open your browser and go to platform.openai.com.
  • Click Sign up and create your account (or sign in if you already have one).
  • Once signed in, click API keys in the left sidebar. You can also navigate directly to platform.openai.com/api-keys.
  • Click Create new secret key.
  • Give the key a name like rag-chatbot and leave permissions set to All.
  • Copy the key immediately and paste it somewhere safe (a text file, password manager, or sticky note on your desk).

Your key is shown only once

OpenAI displays the full secret key only at the moment you create it. If you close the dialog without copying, you will need to delete that key and create a new one.

How much does this project cost?

This project uses text-embedding-3-small at $0.02 per 1M tokens and gpt-4o-mini at $0.15 per 1M input tokens. For a small document and a handful of questions, expect to spend less than $0.05 total.

Set the API key as an environment variable

Storing your key in an environment variable keeps it out of your code. This prevents accidentally sharing it if you push your project to GitHub.

  • Return to your terminal (the one with your virtual environment activated).
  • Set the OPENAI_API_KEY environment variable by running the command below, replacing your-api-key-here with the key you just copied:

macOS

export OPENAI_API_KEY=your-api-key-here

Windows

set OPENAI_API_KEY=your-api-key-here

Why not put the key directly in code?

Hardcoding secrets in source files is a security risk. If you ever share or upload your code, anyone who sees it gets full access to your OpenAI account. Environment variables keep the secret outside your project files.

The OPENAI_API_KEY variable name is the standard that OpenAI's Python SDK recognizes. Your code will read it with os.environ.get("OPENAI_API_KEY").

This variable only lasts for your session

The export (macOS) or set (Windows) command sets the variable for the current terminal session only. If you close the terminal, you will need to run this command again before using the chatbot.

Test the connection with a simple API call

Before building anything, confirm that your key works by sending a quick chat request to OpenAI.

  • Run this test command in your terminal:
python -c "
import os
from openai import OpenAI

# Create a client using the API key from the environment
client = OpenAI(api_key=os.environ.get('OPENAI_API_KEY'))

# Send a simple test message
response = client.chat.completions.create(
    model='gpt-4o-mini',
    messages=[{'role': 'user', 'content': 'Say hello in one sentence.'}]
)

print(response.choices[0].message.content)
"

What does this command do?

This runs a short Python script directly from the terminal. It creates an OpenAI client using your API key, sends a one-sentence prompt to gpt-4o-mini, and prints the response.

If everything is connected correctly, you will see a friendly greeting printed to your terminal.

  • Check the terminal output. You should see a short greeting from the model (something like "Hello! How can I help you today?").

Seeing an authentication error?

If you see a message about an incorrect API key, double-check that you copied the full key with no extra spaces. Run your export or set command again and make sure your terminal session is the same one where you set the variable.

If you see a billing or quota error, visit your OpenAI billing page at platform.openai.com/settings/organization/billing and add a payment method.

Help me debug my OpenAI API connection error.

Your API key is working and your environment is fully connected to OpenAI. Next up, you will create the knowledge base document that your chatbot will learn from.

Create Your Knowledge Base

Your OpenAI connection is live and ready to go. Now it's time to give your chatbot something to be smart about.

A RAG chatbot can only answer questions from information it has access to. In this step, you'll create a knowledge base document and write the code that breaks it into smaller pieces called chunks. Chunking is essential because language models have limited context windows, so you need to feed them only the most relevant pieces of a document rather than the entire thing.

In this step, get ready to:
  • Create a sample text document that serves as your chatbot's knowledge base.
  • Write a Python script that reads the document and splits it into overlapping chunks.
  • Verify the chunking output by running the script.
Create a sample knowledge base document

Your chatbot needs a source of truth. This is the document it will search through when answering questions.

  • Create a new file called knowledge.txt in your project folder by running this command:
touch knowledge.txt
  • Open knowledge.txt in your editor and paste the following content:
The Solar System consists of the Sun and the objects that orbit it, either directly or indirectly. Of the objects that orbit the Sun directly, the largest are the eight planets. The rest are smaller objects, such as dwarf planets and small Solar System bodies.

Mercury is the smallest planet in the Solar System and the closest to the Sun. Its orbit around the Sun takes 87.97 Earth days, the shortest of all the planets. Mercury has no atmosphere to retain heat, so temperatures vary drastically from 430 degrees Celsius during the day to minus 180 degrees Celsius at night.

Venus is the second planet from the Sun. It is sometimes called Earth's sister planet because of their similar size and mass. Venus has a thick atmosphere composed mainly of carbon dioxide, which creates a greenhouse effect making it the hottest planet in the Solar System with surface temperatures around 465 degrees Celsius.

Earth is the third planet from the Sun and the only known planet to harbor life. It has one natural satellite, the Moon. Earth's atmosphere is composed of 78 percent nitrogen, 21 percent oxygen, and trace amounts of other gases. The planet's surface is about 71 percent water.

Mars is the fourth planet from the Sun and is often called the Red Planet due to its reddish appearance caused by iron oxide on its surface. Mars has two small moons named Phobos and Deimos. The planet has the largest volcano in the Solar System, Olympus Mons, which stands about 21.9 kilometers high.

Jupiter is the fifth planet from the Sun and the largest in the Solar System. It is a gas giant with a mass more than two and a half times that of all the other planets combined. Jupiter's Great Red Spot is a giant storm that has been raging for at least 350 years. The planet has at least 95 known moons.

Saturn is the sixth planet from the Sun and is known for its prominent ring system. It is the second-largest planet in the Solar System after Jupiter. Saturn's rings are made mostly of ice particles, rocky debris, and dust. The planet has 146 known moons, with Titan being the largest.

Uranus is the seventh planet from the Sun. It is unique because it rotates on its side, with an axial tilt of about 98 degrees. Uranus has 28 known moons and a faint ring system. The planet appears blue-green due to methane in its atmosphere.

Neptune is the eighth and farthest known planet from the Sun. It has the strongest winds in the Solar System, reaching speeds up to 2,100 kilometers per hour. Neptune has 16 known moons, with Triton being the largest. Triton is unusual because it orbits Neptune in the opposite direction to the planet's rotation.

What is this document for?

This is your chatbot's entire knowledge base. When someone asks a question, the chatbot will search through this text to find the most relevant pieces.

You can replace this with any text you like later. Company FAQs, product documentation, research notes, or course material all work great as knowledge bases.

  • Save knowledge.txt.
  • Confirm the file exists by running:
ls

You should see knowledge.txt listed in your project folder.

Write the ingestion script

Now you'll build the Python script that reads your document and splits it into smaller chunks. Each chunk will eventually become a searchable unit in your vector database.

  • Create a new file called ingest.py in your project folder and add the following setup code:
import os
import chromadb
from openai import OpenAI

# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path="./chroma_db")

# Create or get collection
collection = chroma_client.get_or_create_collection(name="my_documents")


def read_document(file_path):
    """Read a text file and return its contents."""
    with open(file_path, "r", encoding="utf-8") as f:
        return f.read()

What does this code do?

  • The imports bring in the OpenAI SDK for generating embeddings and ChromaDB for storing them locally.
  • PersistentClient(path="./chroma_db") creates a local database folder so your data survives between script runs.
  • get_or_create_collection(name="my_documents") sets up a named container for your embedded chunks.
  • read_document() is a helper that opens a text file and returns its full contents as a string.

You won't see any output yet. The next chunk uses these building blocks to do the real work.

  • Below the read_document function, add the chunk_text function:
def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start += chunk_size - overlap
    return chunks

What does chunk_text do?

This function slides a window across your text, grabbing 500 characters at a time. Each window overlaps the previous one by 50 characters.

The overlap ensures that sentences split across chunk boundaries still appear in at least one complete chunk. Without overlap, the chatbot might miss context that falls exactly at a boundary.

Why 500 characters and 50 overlap?

500 characters keeps each chunk well under the text-embedding-3-small model's 8192-token limit while still holding a meaningful paragraph of information. 50-character overlap is roughly one sentence, enough to preserve context at boundaries.

  • Save ingest.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

import os
import chromadb
from openai import OpenAI

# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path="./chroma_db")

# Create or get collection
collection = chroma_client.get_or_create_collection(name="my_documents")


def read_document(file_path):
    """Read a text file and return its contents."""
    with open(file_path, "r", encoding="utf-8") as f:
        return f.read()


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start += chunk_size - overlap
    return chunks
Verify the chunking output

Time to make sure your chunking logic works correctly before moving on to embeddings.

  • Test your functions by running this command in the terminal:
python -c "from ingest import read_document, chunk_text; text = read_document('knowledge.txt'); chunks = chunk_text(text); print(f'Number of chunks: {len(chunks)}'); print(f'\nFirst chunk:\n{chunks[0]}')"

What should I see?

Your terminal should print the total number of chunks (around 7-9 depending on the exact text length) followed by the full text of the first chunk. The first chunk should start with "The Solar System consists of the Sun..." and be around 500 characters long.

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • Confirm you're running the command from your project folder where ingest.py and knowledge.txt both live.
  • Check that openai and chromadb are installed by running pip list and looking for them in the output.

Help me debug a ModuleNotFoundError when importing from ingest.py

Your knowledge base is chunked and ready to go. Next up, you'll turn these plain-text chunks into numerical embeddings and store them in ChromaDB so your chatbot can search through them.

Embed and Store Your Chunks in ChromaDB

In the last step, you split your knowledge base into bite-sized chunks that are ready for processing. Now comes the magic: turning those text chunks into numerical vectors that a computer can compare for meaning.

In this step, you'll use OpenAI's text-embedding-3-small model to generate embeddings for each chunk, then store them in ChromaDB. This is what allows your chatbot to later search for the most relevant context when answering a question.

In this step, get ready to:
  • Understand the ChromaDB client and collection already in your file.
  • Write a function that generates embeddings using OpenAI's API.
  • Build and run a full ingestion pipeline that stores your chunks in ChromaDB.
Review the ChromaDB client and collection

At the top of your ingest.py file, you already have three lines that set up ChromaDB. Let's understand what they do before building on top of them.

  • In ingest.py, find these lines near the top of the file:
# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path="./chroma_db")

# Create or get collection
collection = chroma_client.get_or_create_collection(name="my_documents")

What does this code do?

chromadb.PersistentClient(path="./chroma_db") creates a ChromaDB client that saves data to a folder called ./chroma_db in your project directory. This means your embedded data survives between script runs.

get_or_create_collection(name="my_documents") creates a collection (think of it like a database table) named my_documents. If the collection already exists from a previous run, it reuses it instead of crashing.

What is a vector database?

A vector database stores data as numerical vectors (lists of numbers) and lets you search by meaning instead of exact keywords. When you ask "What's the hottest planet?", the database finds chunks about Venus's temperature even if those chunks never use the word "hottest".

Write the embedding function

Embeddings are numerical representations of text that capture meaning. Two sentences about the same topic will have similar embedding vectors, even if they use different words.

  • In ingest.py, find the end of your chunk_text() function, then add the following function below it:
def get_embeddings(texts):
    """Get embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(
        input=texts,
        model="text-embedding-3-small"
    )
    return [item.embedding for item in response.data]

What does this code do?

  • client.embeddings.create() sends your text chunks to OpenAI's API and gets back a vector for each one.
  • model="text-embedding-3-small" specifies OpenAI's most cost-effective embedding model at $0.02 per 1M tokens.
  • The list comprehension extracts just the embedding vectors from the API response.
  • Save ingest.py.

You won't see a change yet. The next function will call get_embeddings() as part of the full ingestion pipeline.

Seeing a red underline on client.embeddings?

Make sure your get_embeddings() function is at the module level (no extra indentation), not nested inside another function.

Also check that client = OpenAI(...) still exists near the top of your file.

Help me debug my get_embeddings function in ingest.py

Build the ingestion pipeline and store your chunks

Now you'll tie everything together. The ingestion function reads your document, chunks it, embeds the chunks, and stores them in ChromaDB with metadata that tracks where each chunk came from.

  • In ingest.py, add the following function below get_embeddings():
def ingest_document(file_path):
    """Ingest a document: read, chunk, embed, and store in ChromaDB."""
    print(f"Reading document: {file_path}")
    text = read_document(file_path)

    print("Chunking document...")
    chunks = chunk_text(text)
    print(f"Created {len(chunks)} chunks")

    print("Generating embeddings...")
    embeddings = get_embeddings(chunks)

    print("Storing in ChromaDB...")
    ids = [f"{os.path.basename(file_path)}_chunk_{i}" for i in range(len(chunks))]
    metadatas = [{"source": file_path, "chunk_index": i} for i in range(len(chunks))]

    collection.add(
        ids=ids,
        embeddings=embeddings,
        documents=chunks,
        metadatas=metadatas
    )

    print(f"Successfully ingested {len(chunks)} chunks from {file_path}")

What does this code do?

  • The function orchestrates the full pipeline: read, chunk, embed, store.
  • ids gives each chunk a unique identifier like knowledge.txt_chunk_0. ChromaDB requires unique IDs for each record.
  • metadatas attaches the source filename and chunk index to each record, so you can trace answers back to their origin later.
  • collection.add() saves everything to ChromaDB: the text, the embeddings, the IDs, and the metadata.
  • Now add the main block at the very bottom of ingest.py so the script runs when you execute it:
if __name__ == "__main__":
    ingest_document("knowledge.txt")

What does this code do?

if __name__ == "__main__": means this code only runs when you execute the file directly (not when you import it from another script). It kicks off the ingestion pipeline on knowledge.txt.

  • Save ingest.py.
  • Run the ingestion script by executing this command in your terminal:
python ingest.py
  • You should see output like this:
Reading document: knowledge.txt
Chunking document...
Created 6 chunks
Generating embeddings...
Storing in ChromaDB...
Successfully ingested 6 chunks from knowledge.txt
  • Confirm the ChromaDB data directory was created by running:
ls chroma_db

You should see files inside the chroma_db folder. This is your local vector database on disk.

Seeing an authentication error?

Make sure your OPENAI_API_KEY environment variable is still set in this terminal session. If you opened a new terminal, you'll need to export it again.

Also verify your virtual environment is still active. You should see the environment name in parentheses at the start of your terminal prompt.

Help me fix my OpenAI authentication error when running ingest.py

Seeing a 'file not found' error?

Make sure knowledge.txt is in the same folder as ingest.py. Run ls to check that both files appear in the current directory.

Help me fix a file not found error in my RAG ingestion script

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

Here's the complete ingest.py file at this point:

import os
import chromadb
from openai import OpenAI

# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path="./chroma_db")

# Create or get collection
collection = chroma_client.get_or_create_collection(name="my_documents")


def read_document(file_path):
    """Read a text file and return its contents."""
    with open(file_path, "r", encoding="utf-8") as f:
        return f.read()


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start += chunk_size - overlap
    return chunks


def get_embeddings(texts):
    """Get embeddings for a list of texts using OpenAI."""
    response = client.embeddings.create(
        input=texts,
        model="text-embedding-3-small"
    )
    return [item.embedding for item in response.data]


def ingest_document(file_path):
    """Ingest a document: read, chunk, embed, and store in ChromaDB."""
    print(f"Reading document: {file_path}")
    text = read_document(file_path)

    print("Chunking document...")
    chunks = chunk_text(text)
    print(f"Created {len(chunks)} chunks")

    print("Generating embeddings...")
    embeddings = get_embeddings(chunks)

    print("Storing in ChromaDB...")
    ids = [f"{os.path.basename(file_path)}_chunk_{i}" for i in range(len(chunks))]
    metadatas = [{"source": file_path, "chunk_index": i} for i in range(len(chunks))]

    collection.add(
        ids=ids,
        embeddings=embeddings,
        documents=chunks,
        metadatas=metadatas
    )

    print(f"Successfully ingested {len(chunks)} chunks from {file_path}")


if __name__ == "__main__":
    ingest_document("knowledge.txt")

Your knowledge base is now embedded and stored in a local vector database. Next up, you'll build the retrieval and answer pipeline that searches ChromaDB and generates grounded answers.

Build the Retrieval and Answer Pipeline

Your knowledge base is embedded and stored in ChromaDB. Now you need a way to search it and generate answers from what you find.

In this step, you will build a second Python script that takes a user's question, finds the most relevant chunks from your vector database, and passes them to GPT-4o mini as context for generating a grounded answer.

In this step, get ready to:
  • Write a function that embeds a user question and queries ChromaDB for the most similar chunks.
  • Write a function that sends the retrieved chunks and user question to GPT-4o mini with a system prompt.
  • Combine everything into an interactive chat loop.
Write the embedding and retrieval functions

The retrieval side of RAG has two jobs. First, convert the user's question into an embedding vector. Second, search the vector database for chunks whose embeddings are closest to that question vector.

  • Create a new file called chat.py in your project folder.
  • Add the following code to set up the clients and define the embedding and retrieval functions:
import os
import chromadb
from openai import OpenAI

# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(name="my_documents")


def get_embedding(text):
    """Get embedding for a single text."""
    response = client.embeddings.create(
        input=text,
        model="text-embedding-3-small"
    )
    return response.data[0].embedding


def retrieve_relevant_chunks(query, n_results=3):
    """Retrieve the most relevant chunks for a query."""
    query_embedding = get_embedding(query)
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results
    )
    return results

What does this code do?

The imports and initialization connect to the same OpenAI client and ChromaDB database you used in ingest.py. The collection variable points at your existing my_documents collection that already holds your embedded chunks.

The get_embedding() function converts a single piece of text (your question) into a vector using the same text-embedding-3-small model you used to embed the chunks. This ensures the question and chunks live in the same vector space.

The retrieve_relevant_chunks() function embeds the user's query and passes it to ChromaDB's collection.query() method. ChromaDB compares the query vector against all stored chunk vectors and returns the 3 closest matches.

  • Save chat.py.
  • You won't see a change yet. The next chunk uses these functions to generate answers.
Write the answer generation function

Retrieval alone just gives you raw text chunks. The generation step sends those chunks as context to a language model along with the user's question, so the model can synthesize a coherent answer grounded in real data.

  • In chat.py, add the following function below retrieve_relevant_chunks():
def generate_answer(query, context_chunks):
    """Generate an answer using GPT-4o mini with retrieved context."""
    context = "\n\n---\n\n".join(context_chunks)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant that answers questions based on "
                    "the provided context. If the context does not contain enough "
                    "information to answer the question, say so clearly. "
                    "Do not make up information."
                )
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}"
            }
        ],
        temperature=0.2
    )

    return response.choices[0].message.content

What does this code do?

The function first joins all retrieved chunks into a single string, separated by --- dividers. This becomes the context block the model reads.

The system prompt tells gpt-4o-mini to answer only from the provided context. If the context does not contain enough information, the model should say so rather than hallucinate. Setting temperature=0.2 keeps responses factual and consistent.

This is the core of RAG. Instead of relying on the model's training data alone, you feed it the exact information it needs to answer correctly.

  • Save chat.py.
  • You won't see a change yet. The next chunk ties retrieval and generation together in a chat loop.
Combine everything into an interactive chat loop

The final piece connects your retrieval and generation functions into a loop that reads user input, fetches context, generates an answer, and prints it.

  • In chat.py, add the following code below generate_answer():
def chat():
    """Run the interactive chat loop."""
    print("RAG Chatbot Ready! Ask questions about your documents.")
    print("Type 'quit' to exit.\n")

    while True:
        query = input("You: ").strip()
        if query.lower() == "quit":
            print("Goodbye!")
            break
        if not query:
            continue

        results = retrieve_relevant_chunks(query)
        context_chunks = results["documents"][0]

        answer = generate_answer(query, context_chunks)
        print(f"\nAssistant: {answer}\n")


if __name__ == "__main__":
    chat()

What does this code do?

The chat() function runs an infinite loop. Each iteration reads a line of input, retrieves relevant chunks, generates an answer, and prints it.

Typing quit breaks the loop. Empty input is skipped. The results["documents"][0] line extracts the actual text strings from ChromaDB's query response (it returns nested lists because you can batch multiple queries at once).

The if __name__ == "__main__" block ensures the chat loop starts when you run chat.py directly.

  • Save chat.py.
  • Run your chatbot by executing this command in your terminal:
python chat.py
  • You should see RAG Chatbot Ready! Ask questions about your documents. printed in your terminal, followed by a You: prompt waiting for input.
  • Type quit to exit for now. You will test it properly in the next step.

Not seeing the chatbot prompt?

  • Make sure your virtual environment is still activated. You should see the environment name in your terminal prompt.
  • Confirm that your OPENAI_API_KEY environment variable is still set. If you opened a new terminal window, you may need to export it again.
  • Check that the ./chroma_db folder exists in your project directory. If it is missing, re-run python ingest.py first.

Why isn't my RAG chatbot starting?

✔️ Awesome, I've got everything!

Great. Double check you have saved chat.py and confirmed the chatbot prompt appears.

ⓧ I'd like to double check the full code

import os
import chromadb
from openai import OpenAI

# Initialize clients
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(name="my_documents")


def get_embedding(text):
    """Get embedding for a single text."""
    response = client.embeddings.create(
        input=text,
        model="text-embedding-3-small"
    )
    return response.data[0].embedding


def retrieve_relevant_chunks(query, n_results=3):
    """Retrieve the most relevant chunks for a query."""
    query_embedding = get_embedding(query)
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results
    )
    return results


def generate_answer(query, context_chunks):
    """Generate an answer using GPT-4o mini with retrieved context."""
    context = "\n\n---\n\n".join(context_chunks)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant that answers questions based on "
                    "the provided context. If the context does not contain enough "
                    "information to answer the question, say so clearly. "
                    "Do not make up information."
                )
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}"
            }
        ],
        temperature=0.2
    )

    return response.choices[0].message.content


def chat():
    """Run the interactive chat loop."""
    print("RAG Chatbot Ready! Ask questions about your documents.")
    print("Type 'quit' to exit.\n")

    while True:
        query = input("You: ").strip()
        if query.lower() == "quit":
            print("Goodbye!")
            break
        if not query:
            continue

        results = retrieve_relevant_chunks(query)
        context_chunks = results["documents"][0]

        answer = generate_answer(query, context_chunks)
        print(f"\nAssistant: {answer}\n")


if __name__ == "__main__":
    chat()

Your retrieval and generation pipeline is complete. Next up, you will put it to the test by asking questions your chatbot can only answer from the document.

Test Your RAG Chatbot

Your full RAG pipeline is wired up. You have a knowledge base ingested into ChromaDB, an embedding function that turns questions into vectors, a retrieval function that finds relevant chunks, and a generation function that produces grounded answers.

Now it's time to put your chatbot through its paces. You'll ask questions it should be able to answer, questions it definitely cannot answer, and compare the results to what a model without your document context would say.

In this step, get ready to:
  • Ask questions the chatbot can answer from your document.
  • Test how the chatbot handles questions outside its knowledge.
  • Observe the difference RAG makes compared to a model without context.
Ask questions your document can answer

The best way to verify your RAG pipeline works is to ask questions where you already know the answer from your knowledge base. If the chatbot returns accurate, specific information from your document, you know retrieval is working.

  • Start the chatbot by running this command in your terminal:
python chat.py

You should see the message RAG Chatbot Ready! Ask questions about your documents. followed by a You: prompt waiting for your input.

  • Type a question that your knowledge.txt file can answer. For example:
How many moons does Jupiter have?

The chatbot should respond with a specific answer pulled from your document. In this case, it should mention that Jupiter has at least 95 known moons.

  • Try a few more questions to confirm retrieval is working consistently:
What is the hottest planet in the Solar System?
How high is Olympus Mons?

What should I see?

The chatbot should give specific answers with data from your document. For the hottest planet, it should mention Venus at around 465 degrees Celsius. For Olympus Mons, it should say about 21.9 kilometers high.

These precise numbers come directly from the chunks your pipeline retrieved. The chatbot is not guessing from general knowledge.

Seeing an error when you run the chatbot?

  • If you see an authentication error, make sure your OPENAI_API_KEY environment variable is still set in your current terminal session.
  • If you see a collection error, confirm you ran ingest.py successfully in a previous step and that the ./chroma_db folder exists in your project directory.
  • If the chatbot gives vague or generic answers instead of specific facts, your chunks may not have been embedded correctly. Try running ingest.py again.

Help me debug my RAG chatbot

Test the chatbot's boundaries

A well-built RAG chatbot should know what it does not know. Your system prompt instructs the model to clearly say when the context does not contain enough information. This is what prevents hallucinations in production RAG systems.

  • Ask a question that your knowledge.txt does not cover at all. For example:
What is the capital of France?

The chatbot should tell you that it cannot find the answer in the provided context. It will not make up an answer even though GPT-4o mini obviously knows that Paris is the capital of France.

Why does it refuse to answer?

Your system prompt in generate_answer() explicitly tells the model: "If the context does not contain enough information to answer the question, say so clearly. Do not make up information."

This is the key difference between RAG and a plain chatbot. The model is constrained to only use the retrieved chunks as evidence. This makes answers trustworthy and verifiable.

  • Try one more question that partially relates to your document but asks for information it does not contain:
When was Neptune discovered?

Your document mentions Neptune but does not include its discovery date. The chatbot should share what it can find about Neptune from the context and clearly state that the discovery date is not included.

Compare RAG-grounded answers to general model responses

To truly appreciate what RAG adds, compare your chatbot's answers to what a model without context would produce. This reveals why retrieval matters.

  • Ask your chatbot this question:
How many moons does Saturn have?

Your chatbot should respond with 146 moons, because that is the specific number in your document.

  • Now ask the same question to any general-purpose AI chatbot (like ChatGPT or a web search). Notice the difference in specificity.

What does this comparison show?

A general model might give a different number because its training data could be outdated or aggregated from multiple sources. Your RAG chatbot gives the exact answer from your document every time.

This is the core value of RAG in production. Enterprise knowledge bases, customer support bots, and internal tools all use this pattern to ensure answers are grounded in authoritative, up-to-date sources rather than the model's training data.

  • When you're done exploring, type quit to exit the chatbot.

Why RAG over fine-tuning?

You might wonder why not just fine-tune a model on your documents instead. Fine-tuning bakes knowledge into model weights permanently, making it expensive to update and impossible to trace which source a specific answer came from.

RAG keeps your data separate from the model. You can update, add, or remove documents without retraining anything. The model always shows its work by pointing to the retrieved context.

Secret mission

Add Source Citations

Your chatbot gives answers, but where did those answers come from? In this secret mission, you'll make your RAG chatbot transparent by adding source citations to every response. Each answer will show exactly which chunks were used, so you can verify the chatbot's reasoning.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs locally with a small amount of OpenAI API usage, so there are no ongoing costs unless you continue making API calls.

Resources you used:

  • Project folder on your Desktop containing ingest.py, chat.py, and knowledge.txt.
  • Local ChromaDB database stored in the chroma_db directory.
  • Python virtual environment inside your project folder.
  • OpenAI API key set as an environment variable.
  • OpenAI account.

Keep everything running

No action needed. Choose this if you're still actively experimenting with your RAG chatbot or plan to add more documents to your knowledge base.

  • Your project folder, virtual environment, and ChromaDB database will remain on your Desktop ready to use.
  • Your OPENAI_API_KEY environment variable will persist until you close your terminal session.
  • You will only incur further OpenAI charges if you run the scripts again.

Pause - I'll come back to this later

Shut down running processes to free up memory while keeping your files intact for later.

  • Deactivate your virtual environment by running:
deactivate
  • Unset your API key from the current terminal session by running:
unset OPENAI_API_KEY
  • Your project folder and ChromaDB database remain on disk. When you come back, reactivate the virtual environment and re-export your API key to pick up where you left off.

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

  • Deactivate your virtual environment first by running:
deactivate
  • Unset the API key environment variable by running:
unset OPENAI_API_KEY
  • Delete the entire project folder (including all Python files, the virtual environment, and the ChromaDB database). On macOS/Linux, run:
cd ~/Desktop
rm -rf rag-chatbot
  • On Windows, run:
cd ~\Desktop
rmdir /s /q rag-chatbot

Nice Work!

Nice Work!

You just built a fully functional RAG chatbot from scratch. Your chatbot can ingest documents, retrieve relevant context using semantic search, and generate grounded answers that stay faithful to your source material.

Here's what you accomplished:

  • Built a document ingestion pipeline that reads, chunks, and embeds text into a local ChromaDB vector database.
  • Created a semantic search system that retrieves the most relevant chunks for any user question using OpenAI embeddings.
  • Connected retrieval to generation using GPT-4o mini, completing the full RAG pipeline: chunk, embed, store, retrieve, generate.
  • Added source citations so the chatbot tells you exactly which chunks it used to answer each question.

Ready to quiz yourself?