Build a RAG System With Your Own Docs

Build a local RAG pipeline that answers questions from your own documents.

Introduction

30 Second Summary

AI chatbots are great at answering general questions. But what happens when you need answers from your own specific documents, like company policies, research notes, or product manuals? That is where Retrieval-Augmented Generation comes in.

In this project, you will build a RAG system that reads your own text documents, stores them in a vector database, and answers questions using only the information they contain.

What You'll Build

A command-line question-answering system that retrieves relevant information from your documents and generates grounded answers using NVIDIA NIM's Llama 3.1 model, hosted for free with no credit card required.

By the end of this project, you'll have:

  • A working RAG pipeline that loads, chunks, and embeds your documents into a local ChromaDB vector database.
  • A retrieval system that finds the most relevant pieces of your documents for any question you ask.
  • An interactive question-answering interface that generates answers grounded only in your documents using NVIDIA NIM's Llama 3.1 model.
  • Secret Mission: Add source citations so every answer tells you exactly which document and chunk the information came from.

Are there any prerequisites?

You should have basic Python knowledge (variables, functions, and loops). You will also need a free NVIDIA Developer account, which you will set up during the project.

Set Up Your Python Environment

Your RAG system needs Python and a few key packages to work. Python runs the code. The packages handle storing documents as vectors, connecting to the AI model, and managing configuration.

In this step, you will install Python, create an isolated project environment, and install the three packages your project depends on.

In this step, get ready to:
  • Install and verify Python 3.10+.
  • Create a project folder and virtual environment.
  • Install the required Python packages.
Check your Python version

This project requires Python 3.10 or higher. Let's check whether you already have it installed.

  • Open a terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.10 or higher

You are ready to go. Move on to the next substep.

ⓧ I see an older version

You have Python installed, but it is below 3.10. You need to install a newer version.

  • Go to python.org/downloads and download the latest Python 3 installer for your operating system.
  • Run the installer. On Windows, check the box that says Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal, then run the version check command again to confirm you see 3.10 or higher.

Why a newer version?

ChromaDB (the vector database you will use) requires SQLite > 3.35, which ships with Python 3.10+. Older versions may cause silent database errors.

ⓧ Command not found

Python is not installed on your machine yet.

  • Go to python.org/downloads and download the latest Python 3 installer for your operating system.
  • Run the installer. On Windows, check the box that says Add Python to PATH before clicking Install Now.
  • On macOS, double-click the downloaded .pkg file and follow the installer prompts.
  • Close and reopen your terminal, then run the version check command again to confirm Python is working.
Create your project folder and virtual environment

A virtual environment is an isolated space where your project's packages live. It prevents conflicts with other Python projects on your machine.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-project
cd rag-project
  • Confirm the folder exists by running:
ls

You should see an empty output since the folder is brand new.

  • Create a virtual environment by running this command:

macOS

python3 -m venv venv

Windows

python -m venv venv
  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

On Windows, if you get an error about execution policies, run this command first:

Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned

Then try the activation command again. On macOS, make sure you typed source before the path.

Help me activate my Python virtual environment.

Install the required packages

Your project needs three packages: ChromaDB for storing document vectors, the OpenAI Python client for talking to the AI model, and python-dotenv for loading your API key from a file.

  • Install all three packages by running this command:
pip install chromadb==1.5.9 openai==2.37.0 python-dotenv==1.1.0

What does this command do?

The == syntax pins each package to a specific version. This ensures your project works exactly as expected, regardless of future updates to these libraries.

  • chromadb==1.5.9 is the vector database that stores your document chunks as embeddings.
  • openai==2.37.0 is the Python client that connects to NVIDIA NIM's OpenAI-compatible API.
  • python-dotenv==1.1.0 loads environment variables from a .env file so you can keep secrets out of your code.

The install may take a minute or two. You will see a stream of download and install messages in your terminal.

  • Verify the packages installed correctly by running:
pip list

You should see chromadb, openai, and python-dotenv in the list along with their dependencies.

Seeing installation errors?

Make sure your virtual environment is active. You should see (venv) in your terminal prompt. If not, run the activation command from the previous substep again.

If you see a build error related to hnswlib, try upgrading pip first with pip install --upgrade pip and then re-run the install command.

Help me fix my pip install errors.

Your Python environment is ready. Next up, you will configure your NVIDIA NIM API key so your project can talk to the Llama 3.1 AI model.

Configure Your NVIDIA NIM API Key

Your Python environment is ready with all the packages installed. Now you need a way to talk to an AI model that will generate answers from your documents.

NVIDIA NIM provides free hosted AI model endpoints that you can call over the internet. In this step, you will grab an API key from NVIDIA and write the Python code that connects to their servers.

In this step, get ready to:
  • Generate an NVIDIA NIM API key.
  • Store the key securely in a .env file.
  • Create config.py to connect to NVIDIA's API and verify the connection works.
Generate your NVIDIA NIM API key

NVIDIA NIM hosts AI models for free in the cloud. To use them, you need an API key that proves your identity when your code sends requests.

  • Open your browser and go to build.nvidia.com.
  • Click Login in the top right corner.
  • Sign in with your NVIDIA Developer account.
  • Once signed in, navigate to your API keys page at build.nvidia.com/settings/api-keys.
  • Generate a new API key.
  • Copy the key immediately and save it somewhere safe. It will start with nvapi-.

What is NVIDIA NIM?

NVIDIA NIM is a service that hosts AI models in the cloud and exposes them through a standard API. You send a question, the model processes it on NVIDIA's GPUs, and you get a response back.

The API is compatible with the OpenAI Python package, so you can use the same openai library you may have seen in other tutorials. The only difference is that you point it at NVIDIA's URL instead of OpenAI's.

Keep your key private

Never commit your API key to version control or share it publicly. In the next substep, you will store it in a .env file that stays on your local machine only.

Store your API key in a .env file

A .env file keeps sensitive values like API keys separate from your code. The python-dotenv package reads this file automatically so your scripts can access the key without it being hardcoded.

  • In VS Code, right-click in the Explorer sidebar (the file panel on the left).
  • Click New File.
  • Name the file .env and press Enter.
  • Paste the following content into the file, replacing nvapi-your-key-here with the actual API key you copied:
NVIDIA_API_KEY=nvapi-your-key-here
  • Save .env.

Why use a .env file?

Storing secrets in a .env file means your API key never appears in your Python source code. If you ever share your project or push it to GitHub, the .env file stays behind (as long as you add it to .gitignore).

Create config.py to connect to NVIDIA NIM

This file will load your API key and set up a reusable client that all your other scripts can import. It also includes a test function to confirm everything connects properly.

  • In VS Code, right-click in the Explorer sidebar and click New File.
  • Name the file config.py and press Enter.
  • Add the following code to load your API key and create the NVIDIA NIM client:
import os
from dotenv import load_dotenv
from openai import OpenAI

# Load environment variables from .env file
load_dotenv()

# Get the API key from the environment
NVIDIA_API_KEY = os.getenv("NVIDIA_API_KEY")

# Stop the script if no key is found
if not NVIDIA_API_KEY:
    raise ValueError("NVIDIA_API_KEY not found in .env file. Please add it.")

# Initialize the OpenAI client pointed at NVIDIA's endpoint
client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=NVIDIA_API_KEY
)

What does this code do?

  • load_dotenv() reads your .env file and loads its values into the environment so your script can access them.
  • os.getenv("NVIDIA_API_KEY") pulls the key from the environment into a Python variable.
  • The OpenAI client is initialized with NVIDIA's base URL instead of OpenAI's default. This is possible because NVIDIA NIM uses the same API format as OpenAI.
  • You won't see a change yet. The next chunk adds a test function to verify everything connects.
  • Below the client initialization, add the test connection function:
# Test connection
def test_connection():
    response = client.chat.completions.create(
        model="meta/llama-3.1-8b-instruct",
        messages=[{"role": "user", "content": "Say hello in one word."}],
        max_tokens=10
    )
    print(f"Connection successful! Response: {response.choices[0].message.content}")

if __name__ == "__main__":
    test_connection()

What does this code do?

  • test_connection() sends a simple prompt to the meta/llama-3.1-8b-instruct model hosted on NVIDIA NIM.
  • If the API key is valid and the connection works, it prints a success message with the model's one-word response.
  • The if __name__ == "__main__" block means the test only runs when you execute this file directly, not when other scripts import the client.
  • Save config.py.
  • Run the test by executing this command in your terminal:
python config.py
  • You should see output like: Connection successful! Response: Hello (the exact response word may vary).

Not seeing a successful connection?

  • If you see NVIDIA_API_KEY not found in .env file, double-check that your .env file is in the same folder as config.py and that the variable name is spelled exactly NVIDIA_API_KEY.
  • If you see an authentication error, make sure your API key starts with nvapi- and that you copied the full key without any extra spaces.
  • If the command hangs or times out, NVIDIA's free endpoints can occasionally be slow. Wait a moment and try again.

Help me debug my NVIDIA NIM connection error

✔️ Awesome, I've got everything!

Great. Double check you have saved both .env and config.py.

ⓧ I'd like to double check the full code

Here is the complete .env file:

NVIDIA_API_KEY=nvapi-your-key-here

Here is the complete config.py file:

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

NVIDIA_API_KEY = os.getenv("NVIDIA_API_KEY")

if not NVIDIA_API_KEY:
    raise ValueError("NVIDIA_API_KEY not found in .env file. Please add it.")

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=NVIDIA_API_KEY
)

# Test connection
def test_connection():
    response = client.chat.completions.create(
        model="meta/llama-3.1-8b-instruct",
        messages=[{"role": "user", "content": "Say hello in one word."}],
        max_tokens=10
    )
    print(f"Connection successful! Response: {response.choices[0].message.content}")

if __name__ == "__main__":
    test_connection()

Your connection to NVIDIA NIM is confirmed. Next up, you will create the knowledge base documents that your RAG system will search through.

Prepare Your Knowledge Base Documents

Your connection to NVIDIA NIM is working. Now you need to give your RAG system something to search through.

A RAG system is only as useful as the documents you feed it. In this step, you will create a knowledge base of text files and build a loader that reads them into Python.

In this step, get ready to:
  • Create a data folder with text files your system can search.
  • Build a document loader that reads all text files and returns their contents.
  • Test the loader to confirm it reads your files correctly.
Create your data folder and documents

Your RAG system needs a folder of text documents to use as its knowledge base. The system will only answer questions using information found in these files.

  • Create a data folder inside your project directory by running this command:
mkdir data
  • Confirm the folder exists by running:
ls

You should see data listed alongside your existing project files (config.py, .env, venv).

Now you need to add 2-3 text files with content you want to query. These should be plain .txt files with a paragraph or more of information.

  • In VS Code, right-click the data folder in the Explorer sidebar.
  • Click New File.
  • Name your first file something descriptive, like company_faq.txt or python_basics.txt.
  • Paste or type a few paragraphs of content into the file.
  • Repeat to create at least 2 more text files with different topics.
  • Save all your files.

What makes good test documents?

Pick content where you already know the answers. A company FAQ, product documentation, or your own notes on a topic work well.

Each file should have at least 2-3 paragraphs so there is enough text for the chunking step later. The more distinct the topics are between files, the easier it will be to test that retrieval is working correctly.

Build the document loader

With your documents in place, you need a way to read them into Python. The loader will scan the data/ folder, read every .txt file, and return the contents along with each filename as metadata.

  • Create a new file called loader.py in your project root (the same folder as config.py).
  • Add the load_documents function by copying this code into loader.py:
import os

def load_documents(data_dir="data"):
    documents = []
    for filename in os.listdir(data_dir):
        if filename.endswith(".txt"):
            filepath = os.path.join(data_dir, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                content = f.read()
            documents.append({
                "content": content,
                "source": filename
            })
            print(f"Loaded: {filename} ({len(content)} characters)")
    print(f"\nTotal documents loaded: {len(documents)}")
    return documents

What does this code do?

  • os.listdir(data_dir) scans the data/ folder and returns every file name inside it.
  • The .endswith(".txt") filter ignores any non-text files so only your documents get loaded.
  • Each document is stored as a dictionary with two keys: content (the full text) and source (the filename). The source metadata will help you trace answers back to their origin later.
  • Below the load_documents function, add a test block that runs the loader and previews each document:
if __name__ == "__main__":
    docs = load_documents()
    for doc in docs:
        print(f"\n--- {doc['source']} ---")
        print(doc["content"][:200] + "...")

What does this test block do?

The if __name__ == "__main__" pattern means this code only runs when you execute the file directly (not when another file imports it). It calls load_documents() and prints the first 200 characters of each document so you can verify everything loaded.

  • Save loader.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import os

def load_documents(data_dir="data"):
    documents = []
    for filename in os.listdir(data_dir):
        if filename.endswith(".txt"):
            filepath = os.path.join(data_dir, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                content = f.read()
            documents.append({
                "content": content,
                "source": filename
            })
            print(f"Loaded: {filename} ({len(content)} characters)")
    print(f"\nTotal documents loaded: {len(documents)}")
    return documents

if __name__ == "__main__":
    docs = load_documents()
    for doc in docs:
        print(f"\n--- {doc['source']} ---")
        print(doc["content"][:200] + "...")
Test the loader
  • Run the loader to confirm it reads your files correctly:
python loader.py

You should see each filename printed with its character count, followed by a preview of the first 200 characters of each document.

Seeing a FileNotFoundError?

  • Make sure you have a data folder in the same directory as loader.py.
  • Confirm the folder contains at least one .txt file. If you saved your files with a different extension (like .text), the loader will skip them.
  • Check that your terminal is in the project root folder (the folder that contains both loader.py and the data/ folder).

Why is my loader not finding any files?

Your knowledge base is ready and your loader can read it into Python. Next up, you will split these documents into smaller chunks so the system can retrieve just the relevant pieces instead of entire files.

Chunk Your Documents

Your document loader is working and your text files are ready to process.

But there is a problem: sending an entire document to an AI model wastes its limited context window when only a small section is relevant to the question. The solution is text chunking. You will split each document into smaller, overlapping pieces so the system can later retrieve only the relevant parts.

In this step, get ready to:
  • Create a text splitting function with overlapping chunks.
  • Process all documents through the chunker while preserving source metadata.
  • Test and verify that chunks overlap correctly.
Create the text splitting function

Chunking means slicing a long document into fixed-size pieces. Each chunk overlaps with the previous one so that sentences sitting on a boundary are never cut off and lost.

  • Create a new file called chunker.py in your project folder and add the following function:
def chunk_text(text, chunk_size=500, overlap=100):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start += chunk_size - overlap
    return chunks

What does this code do?

This function uses a sliding window to walk through the text.

  • chunk_size=500 means each chunk is up to 500 characters long.
  • overlap=100 means each new chunk starts 100 characters before the previous one ended. This overlap ensures context is never lost at a boundary.
  • start += chunk_size - overlap advances the window by 400 characters (500 minus 100), creating that overlap.

You won't see a change yet. The next function will use chunk_text() to process your documents.

Process documents into chunks with metadata

Each chunk needs to remember which document it came from and its position in that document. This metadata is critical for tracing answers back to their source later.

  • Add the following function below chunk_text() in chunker.py:
def chunk_documents(documents):
    all_chunks = []
    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append({
                "text": chunk,
                "source": doc["source"],
                "chunk_index": i
            })
    print(f"Created {len(all_chunks)} chunks from {len(documents)} documents")
    return all_chunks

What does this code do?

  • It loops through each document returned by your loader and passes the content to chunk_text().
  • For every chunk, it stores the chunk text, the original filename (source), and the chunk's position number (chunk_index).
  • This metadata lets your RAG system trace any retrieved chunk back to its original file.

You won't see a change yet. The next section adds a test block so you can run the file and see your chunks.

Test the chunker

Now you will add a test block that loads your documents, chunks them, and prints the first few results so you can confirm everything works.

  • Add the following test block at the bottom of chunker.py:
if __name__ == "__main__":
    from loader import load_documents
    docs = load_documents()
    chunks = chunk_documents(docs)
    print(f"\nFirst 3 chunks:")
    for chunk in chunks[:3]:
        print(f"\n[{chunk['source']} - Chunk {chunk['chunk_index']}]")
        print(chunk["text"][:100] + "...")

What does this test block do?

  • It imports your load_documents() function from loader.py and feeds the results into chunk_documents().
  • It prints the first 3 chunks so you can inspect the text, source filename, and chunk index for each.
  • Save chunker.py.
  • Run the chunker by entering this command in your terminal:
python chunker.py

You should see output showing your loaded documents, the total number of chunks created, and the first 3 chunks with their source filenames and chunk indices.

How do I know the overlap is working?

Look at the end of chunk 0 and the beginning of chunk 1. The last ~100 characters of chunk 0 should appear again at the start of chunk 1. This overlap means no sentence gets lost between two chunks.

Seeing an error?

  • If you see ModuleNotFoundError: No module named 'loader', make sure you are running the command from the same folder that contains both chunker.py and loader.py.
  • If you see FileNotFoundError, confirm your data/ folder exists and contains at least one .txt file.
  • If you see 0 chunks created, check that your text files are not empty.

Help me debug my chunker.py output.

✔️ Awesome, I've got everything!

Great. Double check you have saved chunker.py.

ⓧ I'd like to double check the full code

def chunk_text(text, chunk_size=500, overlap=100):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start += chunk_size - overlap
    return chunks

def chunk_documents(documents):
    all_chunks = []
    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            all_chunks.append({
                "text": chunk,
                "source": doc["source"],
                "chunk_index": i
            })
    print(f"Created {len(all_chunks)} chunks from {len(documents)} documents")
    return all_chunks

if __name__ == "__main__":
    from loader import load_documents
    docs = load_documents()
    chunks = chunk_documents(docs)
    print(f"\nFirst 3 chunks:")
    for chunk in chunks[:3]:
        print(f"\n[{chunk['source']} - Chunk {chunk['chunk_index']}]")
        print(chunk["text"][:100] + "...")

Your documents are now split into bite-sized chunks with metadata attached. Next up, you will store these chunks in a vector database so the system can search them by meaning.

Store Chunks in ChromaDB

Your documents are split into overlapping chunks, but right now they're just Python lists sitting in memory. To answer questions later, you need a way to find the chunks that are most relevant to a question based on meaning, not just keywords.

A vector database stores text as numerical representations called embeddings. When you ask a question, the database finds the chunks whose embeddings are mathematically closest to your question. In this step, you'll store your chunks in ChromaDB, which handles embedding automatically using a built-in model.

In this step, get ready to:
  • Initialize a persistent ChromaDB vector store on your local machine.
  • Add all document chunks with their source metadata.
  • Verify the collection count and test a similarity query.
Create the vector store file

ChromaDB's PersistentClient saves your data to disk so you don't have to re-embed documents every time you restart your script. It creates a folder on your machine that persists between sessions.

  • Create a new file called vector_store.py in your project folder.
  • Add the following code to initialize the ChromaDB client and create a collection:
import chromadb

def get_vector_store(persist_dir="./chroma_db"):
    client = chromadb.PersistentClient(path=persist_dir)
    collection = client.get_or_create_collection(name="my_documents")
    return collection

What does this code do?

  • chromadb.PersistentClient(path=persist_dir) creates a ChromaDB client that saves all data to the ./chroma_db folder on your machine.
  • get_or_create_collection(name="my_documents") either opens an existing collection or creates a new one. This means you can run the script multiple times without errors.
  • The function returns the collection object, which you'll use to add and query documents.

Why persistent storage instead of in-memory?

An in-memory client loses all data when your script ends. With PersistentClient, your chunks are embedded once and stored permanently. You can query your documents across multiple sessions without re-processing them every time.

  • You won't see a change yet. The next chunk adds the function that populates this collection with your document chunks.
Add document chunks with metadata

Each chunk needs to be stored with metadata so you can trace answers back to their source file later. ChromaDB requires three parallel lists: unique IDs, the text content, and metadata dictionaries.

  • Add the add_chunks_to_store function below your get_vector_store function in vector_store.py:
def add_chunks_to_store(collection, chunks):
    ids = [f"{chunk['source']}_{chunk['chunk_index']}" for chunk in chunks]
    documents = [chunk["text"] for chunk in chunks]
    metadatas = [{"source": chunk["source"], "chunk_index": chunk["chunk_index"]} for chunk in chunks]

    collection.add(
        ids=ids,
        documents=documents,
        metadatas=metadatas
    )
    print(f"Added {len(chunks)} chunks to the vector store")
    print(f"Collection now has {collection.count()} items")

What does this code do?

  • ids creates a unique identifier for each chunk by combining the source filename and chunk index (e.g., faq.txt_0).
  • documents is the list of actual text content that ChromaDB will embed and store.
  • metadatas attaches the source filename and chunk index to each entry so you can trace results back to their origin document.
  • collection.add() handles embedding the text automatically using ChromaDB's built-in model and stores everything together.
  • You won't see a change yet. The next chunk adds a query function and a test script that runs everything together.
Verify the store with a test query

Now you need a way to query the collection and a test script that runs the full pipeline: load documents, chunk them, store them, and search.

  • Add the query_store function and the test script below your add_chunks_to_store function in vector_store.py:
def query_store(collection, question, n_results=3):
    results = collection.query(
        query_texts=[question],
        n_results=n_results
    )
    return results

if __name__ == "__main__":
    from loader import load_documents
    from chunker import chunk_documents

    docs = load_documents()
    chunks = chunk_documents(docs)

    collection = get_vector_store()
    add_chunks_to_store(collection, chunks)

    # Test query
    test_question = "What is this document about?"
    results = query_store(collection, test_question)
    print(f"\nQuery: {test_question}")
    print(f"Top results:")
    for i, doc in enumerate(results["documents"][0]):
        distance = results["distances"][0][i]
        print(f"\n  Result {i+1} (distance: {distance:.4f}):")
        print(f"  {doc[:150]}...")

What does this code do?

  • query_store() takes a question string and returns the top 3 most similar chunks from the collection using ChromaDB's built-in similarity search.
  • The if __name__ block runs the full pipeline: loads your documents, chunks them, stores them in ChromaDB, and then tests a query.
  • The distance values show how similar each result is to your question. Lower distance means higher similarity.
  • Save vector_store.py.
  • Run the script by entering this command in your terminal:
python vector_store.py
  • You should see output showing your documents being loaded, chunks being created, chunks being added to the vector store, and then the top 3 query results with their distance scores.
  • Check that the Collection now has X items line matches the total number of chunks you saw printed by the chunker.

What about the first run being slow?

The first time ChromaDB processes text, it downloads a small embedding model (approximately 80MB). This happens once and is cached locally for all future runs. Subsequent runs will be much faster.

Seeing an error when running the script?

  • If you see a SQLite version error, your Python installation may be too old. ChromaDB requires SQLite version 3.35 or higher, which ships with Python 3.10+.
  • If you see a duplicate ID error, your chroma_db/ folder already contains data from a previous run. Delete the chroma_db/ folder and run the script again.
  • If you see an import error for chromadb, make sure your virtual environment is activated and that you installed the packages in Step 1.

Help me debug my ChromaDB vector store script

  • Confirm you see a new chroma_db/ folder in your project directory by running:
ls
  • You should see chroma_db listed alongside your other project files. This folder contains your persistent vector database.

✔️ Awesome, I've got everything!

Great. Double check you've saved vector_store.py and that your chroma_db/ folder exists.

ⓧ I'd like to double check the full code

import chromadb

def get_vector_store(persist_dir="./chroma_db"):
    client = chromadb.PersistentClient(path=persist_dir)
    collection = client.get_or_create_collection(name="my_documents")
    return collection

def add_chunks_to_store(collection, chunks):
    ids = [f"{chunk['source']}_{chunk['chunk_index']}" for chunk in chunks]
    documents = [chunk["text"] for chunk in chunks]
    metadatas = [{"source": chunk["source"], "chunk_index": chunk["chunk_index"]} for chunk in chunks]

    collection.add(
        ids=ids,
        documents=documents,
        metadatas=metadatas
    )
    print(f"Added {len(chunks)} chunks to the vector store")
    print(f"Collection now has {collection.count()} items")

def query_store(collection, question, n_results=3):
    results = collection.query(
        query_texts=[question],
        n_results=n_results
    )
    return results

if __name__ == "__main__":
    from loader import load_documents
    from chunker import chunk_documents

    docs = load_documents()
    chunks = chunk_documents(docs)

    collection = get_vector_store()
    add_chunks_to_store(collection, chunks)

    # Test query
    test_question = "What is this document about?"
    results = query_store(collection, test_question)
    print(f"\nQuery: {test_question}")
    print(f"Top results:")
    for i, doc in enumerate(results["documents"][0]):
        distance = results["distances"][0][i]
        print(f"\n  Result {i+1} (distance: {distance:.4f}):")
        print(f"  {doc[:150]}...")

Your documents are now stored as vectors in a persistent database. Next up, you'll build the retrieval function that finds the most relevant chunks for any question you ask.

Generate Answers with NVIDIA NIM

Your retrieval function can now find the top matching chunks for any question. The "R" in RAG is done.

Now you need the "G." In this step, you will build a generation pipeline that feeds retrieved context into NVIDIA NIM's Llama 3.1 model and wrap it in an interactive question loop.

In this step, get ready to:
  • Create a RAG module that constrains the model to answer only from retrieved context.
  • Build an orchestration function that ties retrieval and generation into a single call.
  • Create an interactive command-line interface for querying your documents.
Create the RAG module with a grounded system prompt

The generation step is where the model actually reads your retrieved chunks and formulates an answer. A carefully written system prompt is what keeps the model grounded in your documents instead of hallucinating from its training data.

  • Create a new file called rag.py in your project folder.
  • Add the imports and system prompt by copying this code:
from config import client
from vector_store import get_vector_store, query_store

# System prompt that constrains the model to answer only from provided context
SYSTEM_PROMPT = """You are a helpful assistant that answers questions based ONLY on the provided context.

Rules:
- Only use information from the provided context to answer
- If the context does not contain enough information to answer, say "I don't know based on the available documents."
- Keep answers concise and direct
- Do not make up information or use knowledge outside the context"""

Why does the system prompt matter so much?

Without explicit constraints, a language model will happily answer questions using its general training knowledge. The system prompt is what makes your RAG system trustworthy.

By telling the model to say "I don't know" when the context is insufficient, you prevent hallucinated answers. This is the core principle of grounded generation.

You won't see a change yet. The next chunk uses this prompt to generate answers.

  • Add the generate_answer() function below the SYSTEM_PROMPT constant:
def generate_answer(question, context_chunks):
    # Join all retrieved chunks with separators into one context string
    context = "\n\n---\n\n".join(context_chunks)

    # Build the messages list with system instructions and the user's question
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
    ]

    # Call NVIDIA NIM with low temperature for factual answers
    response = client.chat.completions.create(
        model="meta/llama-3.1-8b-instruct",
        messages=messages,
        temperature=0.2,
        max_tokens=500
    )

    return response.choices[0].message.content

What does this function do?

  • context = "\n\n---\n\n".join(context_chunks) combines the retrieved chunks into a single string, separated by dividers so the model can distinguish between them.
  • The messages list uses the Chat Completions format. The system message sets the rules. The user message passes both the context and the question.
  • temperature=0.2 keeps the model's responses factual and consistent rather than creative.

You won't see output yet. The next function ties retrieval and generation together.

Add the orchestration function

The ask() function is the single entry point for the full RAG pipeline. It retrieves chunks from ChromaDB, then passes them to the generation function.

  • Add the ask() function and test block below generate_answer():
def ask(question):
    # Get the vector store collection
    collection = get_vector_store()
    # Retrieve the top matching chunks for this question
    results = query_store(collection, question)

    # Extract just the document text from results
    context_chunks = results["documents"][0]

    # Generate an answer using the retrieved context
    answer = generate_answer(question, context_chunks)
    return answer

if __name__ == "__main__":
    test_question = "What is this document about?"
    print(f"Question: {test_question}")
    print(f"Answer: {ask(test_question)}")

What does this code do?

  • ask() is the full RAG pipeline in one call. It connects to your ChromaDB collection, retrieves the 3 most relevant chunks, and feeds them into generate_answer().
  • results["documents"][0] extracts the actual text content from ChromaDB's query results.
  • The if __name__ block lets you test the pipeline by running this file directly.
  • Save rag.py.
  • Test the full RAG pipeline by running:
python rag.py
  • You should see the question printed, followed by a generated answer based on your documents.

Not getting an answer?

  • If you see an authentication error, confirm your .env file has a valid NVIDIA_API_KEY that starts with nvapi-.
  • If you see a collection error, make sure you ran vector_store.py in Step 5 to populate your chroma_db/ directory.
  • If the response is slow, that is normal for a free NVIDIA NIM endpoint. Wait 10-15 seconds.

Why isn't my RAG pipeline returning an answer?

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

from config import client
from vector_store import get_vector_store, query_store

SYSTEM_PROMPT = """You are a helpful assistant that answers questions based ONLY on the provided context.

Rules:
- Only use information from the provided context to answer
- If the context does not contain enough information to answer, say "I don't know based on the available documents."
- Keep answers concise and direct
- Do not make up information or use knowledge outside the context"""

def generate_answer(question, context_chunks):
    context = "\n\n---\n\n".join(context_chunks)

    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
    ]

    response = client.chat.completions.create(
        model="meta/llama-3.1-8b-instruct",
        messages=messages,
        temperature=0.2,
        max_tokens=500
    )

    return response.choices[0].message.content

def ask(question):
    collection = get_vector_store()
    results = query_store(collection, question)

    context_chunks = results["documents"][0]

    answer = generate_answer(question, context_chunks)
    return answer

if __name__ == "__main__":
    test_question = "What is this document about?"
    print(f"Question: {test_question}")
    print(f"Answer: {ask(test_question)}")
Build the interactive question loop

The RAG pipeline works, but running a script with a hardcoded question is not very useful. An interactive loop lets you ask multiple questions in a session without restarting the program.

  • Create a new file called main.py in your project folder and add the interactive interface:
from rag import ask

def main():
    # Display a welcome banner
    print("=" * 50)
    print("  RAG Question-Answering System")
    print("  Ask questions about your documents!")
    print("  Type 'quit' to exit")
    print("=" * 50)

    # Loop until the user types quit, exit, or q
    while True:
        print()
        question = input("Your question: ").strip()

        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break

        if not question:
            print("Please enter a question.")
            continue

        # Run the full RAG pipeline and display the answer
        print("\nSearching documents and generating answer...")
        answer = ask(question)
        print(f"\nAnswer: {answer}")

if __name__ == "__main__":
    main()

What does this code do?

  • The while True loop keeps accepting questions until the user types quit, exit, or q.
  • Each question goes through ask(), which retrieves relevant chunks from ChromaDB and generates an answer via NVIDIA NIM.
  • The .strip() and empty-check prevent blank submissions from hitting the API.
  • Save main.py.
  • Run your RAG system by running:
python main.py
  • You should see the welcome banner, then a Your question: prompt waiting for input.
  • Type a question about your documents and press Enter.
  • The system retrieves relevant chunks and prints a grounded answer.
  • Try asking something your documents do NOT cover. The model should respond with "I don't know based on the available documents."
  • Type quit to exit when done.

Not seeing the welcome banner?

  • Make sure you are running python main.py from your project folder (the one containing rag.py and config.py).
  • If you see an import error for rag, confirm that rag.py is saved in the same folder as main.py.
  • If your virtual environment is not activated, the installed packages will not be available. Run venv\Scripts\activate first.

Why is my main.py not running correctly?

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

from rag import ask

def main():
    print("=" * 50)
    print("  RAG Question-Answering System")
    print("  Ask questions about your documents!")
    print("  Type 'quit' to exit")
    print("=" * 50)

    while True:
        print()
        question = input("Your question: ").strip()

        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break

        if not question:
            print("Please enter a question.")
            continue

        print("\nSearching documents and generating answer...")
        answer = ask(question)
        print(f"\nAnswer: {answer}")

if __name__ == "__main__":
    main()

Secret mission

Add Source Citations

Your RAG system answers questions, but it does not tell you where the answer came from. In a production system, this makes it impossible to verify claims or trace errors back to their source. In this secret mission, you will add source citations so every answer includes exactly which document and chunk number the information came from.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder containing all Python files (config.py, loader.py, chunker.py, vector_store.py, rag.py, main.py).
  • Virtual environment (venv/) with installed pip packages.
  • Persistent ChromaDB database (chroma_db/ directory).
  • Your .env file containing your NVIDIA API key.
  • The data/ folder with your text documents.

Keep everything running

No action needed. Choose this if you plan to keep adding documents to your RAG system or experimenting with different queries.

  • Your project folder stays on your Desktop with all files intact.
  • The chroma_db/ directory retains your embedded chunks, so you can query them again without re-processing.
  • Your NVIDIA API key remains in your .env file for future use.

Pause - I'll come back to this later

Deactivate your virtual environment to free up your terminal session. Everything else can stay as-is since there are no running processes or cloud resources.

  • Deactivate your virtual environment by running:
deactivate

When you come back, reactivate the environment and pick up where you left off.

  • On Windows, reactivate with:
venv\Scripts\activate
  • On macOS/Linux, reactivate with:
source venv/bin/activate

Delete - I don't want to use this again

Remove all project files from your machine. Since everything lives inside your project folder on the Desktop, one delete command removes it all.

  • First, deactivate your virtual environment if it is still active:
deactivate
  • Navigate to your Desktop:
cd ~/Desktop
  • On macOS/Linux, remove the project folder by running:
rm -rf rag-project
  • On Windows, remove the project folder by running:
rmdir /s /q rag-project

This removes all Python files, your virtual environment, the chroma_db/ database, the data/ folder, and your .env file.

Your NVIDIA API key still exists

Deleting the project folder removes the local .env file, but your API key still exists in your NVIDIA Developer account. You can revoke it from build.nvidia.com if you no longer need it.

Nice Work!

Nice Work!

You just built a complete Retrieval-Augmented Generation system from scratch. Your system reads documents, chunks them, stores them as vector embeddings, retrieves the most relevant pieces for any question, and generates grounded answers using a real LLM.

You've learned how to:

  • Built a persistent vector database using ChromaDB that stores chunked documents as embeddings for fast similarity search.
  • Created a retrieval pipeline that splits text into overlapping chunks, embeds them, and finds the most relevant pieces for any question using vector similarity search.
  • Connected NVIDIA NIM's Llama 3.1 model to generate answers grounded only in retrieved context, refusing to answer when the documents don't contain the information.
  • Added source citations so every answer is traceable back to exactly which document and chunk it came from.

Ready to quiz yourself?