Build a RAG System with Google Gemini

Build a document Q&A pipeline using embeddings, ChromaDB, and Gemini.

Introduction

30 Second Summary

AI chatbots are great at general knowledge, but they have no idea what's in your company documents, personal notes, or custom data. The solution that 40% of AI engineering job postings ask for is called RAG (Retrieval-Augmented Generation).

In this project, you will build a complete RAG system that feeds your own documents into Google Gemini and answers natural language questions grounded in your source material.

What You'll Build

A command-line Python application that ingests text documents, converts them into searchable vector embeddings, and answers questions using Google Gemini.

By the end of this project, you'll have:

  • A working RAG pipeline that loads documents, creates embeddings, and answers questions using the Gemini API.
  • A local vector database (ChromaDB) storing your document chunks as searchable embeddings.
  • An interactive query system that retrieves relevant context and generates grounded answers using prompt engineering.
  • Secret Mission: Add source citations and a confidence indicator to make your answers trustworthy and transparent.

Are there any prerequisites?

You need basic programming experience. A Java background is more than sufficient. No Python experience is required.

Set Up Your Python Environment

Before you can build anything with AI, you need the tools installed on your machine. Python is the language of choice for AI engineering because nearly every AI library and SDK supports it first.

In this step, you will install Python, create an isolated project environment, connect to Google's Gemini API, and confirm everything works with a test call.

In this step, get ready to:
  • Install Python and create a project folder with a virtual environment.
  • Install the required Python packages.
  • Get a free Gemini API key and verify your connection.
Install Python and create a project

Python is the runtime that executes all of the code in this project. You need version 3.14 or higher because the packages you will install require at least Python 3.10.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Open the integrated terminal by pressing Ctrl+` (both macOS and Windows).

Check whether Python is already installed by running the version command for your operating system:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.14 or higher

You are all set. Skip ahead to the project folder creation below.

ⓧ I see an older version

You have Python installed, but this project requires 3.14 or higher. Install the latest version alongside your existing one.

macOS

  • Open https://www.python.org/downloads/release/python-3146/ in your browser.
  • Click Download macOS installer.
  • Double-click the downloaded .pkg file and follow the prompts to install.
  • After installation, close and reopen VS Code's terminal, then run python3 --version to confirm it shows 3.14.

Windows

  • Open https://www.python.org/downloads/release/python-3146/ in your browser.
  • Click Download Windows installer (64-bit).
  • Run the installer. On the first screen, check Add Python to PATH, then click Install Now.
  • After installation, close and reopen VS Code's terminal, then run python --version to confirm it shows 3.14.

ⓧ Command not found

Python is not installed yet. Follow the instructions for your operating system below.

macOS

  • Open https://www.python.org/downloads/release/python-3146/ in your browser.
  • Click Download macOS installer.
  • Double-click the downloaded .pkg file and follow the prompts to install.
  • Open the Install Certificates.command file located in /Applications/Python 3.14/ to finish SSL setup.
  • Close and reopen VS Code's terminal, then run python3 --version to confirm it shows 3.14.

Windows

  • Open https://www.python.org/downloads/release/python-3146/ in your browser.
  • Click Download Windows installer (64-bit).
  • Run the installer. On the first screen, check Add Python to PATH, then click Install Now.
  • Wait for installation to complete, then click Close.
  • Close and reopen VS Code's terminal, then run python --version to confirm it shows 3.14.

Why do I need a specific version?

The google-genai SDK requires Python 3.10 or higher. Using the latest stable release (3.14) means you get the best performance and compatibility with all packages in this project.

Now create your project folder. Every terminal command in this project assumes you are inside this folder.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-qa-system
cd rag-qa-system
  • Confirm the folder was created by running:
ls

The output should be empty because your new folder has no files yet.

Next, create a virtual environment. A virtual environment isolates this project's packages from any other Python projects on your machine.

  • Create the virtual environment by running this command:

macOS

python3 -m venv venv

Windows

python -m venv venv
  • Activate the virtual environment:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate
  • Check that your terminal prompt now starts with (venv).

What is a virtual environment?

A virtual environment is a self-contained folder that holds a separate copy of Python and its packages. Anything you install with pip while the environment is active only affects this project.

This prevents version conflicts between projects. You can always tell the environment is active because your terminal shows (venv) at the beginning of each line.

Don't see (venv) in your prompt?

Make sure you ran the activation command for your operating system. On Windows, if you see a permissions error, run this first:

Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned

Then try the activation command again. Help me activate my virtual environment.

Install required packages

Your project needs two core packages: the Google GenAI SDK for talking to Gemini, and ChromaDB for storing document embeddings locally.

  • Install both packages by running this command:
pip install google-genai==2.11.0 chromadb==1.5.9

What does this command do?

pip is Python's package manager. The == syntax pins each package to an exact version so your project behaves identically to this tutorial.

google-genai is Google's official SDK for calling the Gemini AI models. chromadb is a vector database that runs entirely on your machine with no account required.

This installation may take a minute or two because ChromaDB has several dependencies.

  • Verify both packages installed correctly by running this quick import test:

macOS

python3 -c "import chromadb; from google import genai; print('All packages installed successfully')"

Windows

python -c "import chromadb; from google import genai; print('All packages installed successfully')"

You should see All packages installed successfully printed in your terminal.

Seeing an import error?

Make sure your virtual environment is still active. You should see (venv) at the start of your prompt. If it disappeared, run the activation command again before retrying the install.

Help me fix my package installation error.

Get your free Gemini API key

The Gemini API lets your Python code send text to Google's AI models and get responses back. Both the text generation model (gemini-2.5-flash) and the embedding model (gemini-embedding-2) are free of charge on the free tier.

  • Open https://aistudio.google.com in your browser.
  • Sign in with your Google account.
  • Click Get API key in the left sidebar.
  • Click Create API key.
  • Copy the key that appears. You will only see it once.

Why is the Gemini API free?

Google offers a free tier for the Gemini API that does not require billing or a credit card. Both gemini-2.5-flash and gemini-embedding-2 are free of charge on this tier. You get a generous quota that is more than enough for this project.

Now store your key safely in an environment file so your code can access it without hardcoding secrets.

  • In VS Code's Explorer sidebar, click the New File icon at the top of your rag-qa-system folder.
  • Name the file .env and press Enter.
  • Paste the following into .env, replacing your-api-key-here with the key you just copied:
GEMINI_API_KEY=your-api-key-here
  • Save .env by pressing Cmd+S (macOS) or Ctrl+S (Windows).

Why use a .env file?

A .env file keeps secrets out of your source code. Your Python script reads the key from this file at runtime using a library called python-dotenv. This way you never accidentally share your API key if you push code to GitHub.

  • Install the python-dotenv package so Python can read your .env file:
pip install python-dotenv==1.1.0

Now create a requirements.txt file to track all your project dependencies.

  • Create a new file called requirements.txt in your rag-qa-system folder using the New File icon.
  • Paste the following into requirements.txt:
google-genai==2.11.0
chromadb==1.5.9
python-dotenv==1.1.0
  • Save requirements.txt.

What is requirements.txt?

This file lists every package your project depends on, pinned to exact versions. Anyone who clones your project can run pip install -r requirements.txt to get the same environment.

Finally, verify your API connection works by running a test call to Gemini.

  • Create a new file called test_api.py in your rag-qa-system folder.
  • Paste the following code into test_api.py:
import os
from google import genai
from dotenv import load_dotenv

# Load the API key from .env
load_dotenv()

# Create a Gemini client
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

# Send a test message
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Say hello in one sentence.",
)

# Print the response
print(response.text)

What does this code do?

load_dotenv() reads your .env file and loads GEMINI_API_KEY into the environment.

genai.Client(api_key=...) creates a connection to Google's API. client.models.generate_content(...) sends a prompt to gemini-2.5-flash and returns the AI's response.

  • Save test_api.py, then run it:

macOS

python3 test_api.py

Windows

python test_api.py

You should see a short greeting printed in your terminal. The exact wording will vary each time because the AI generates fresh responses.

Seeing an API error?

Double-check that your .env file has no extra spaces around the = sign. The format must be exactly GEMINI_API_KEY=your-actual-key with no quotes.

If you see a rate limit error, wait 10 seconds and try again. The free tier sometimes throttles rapid consecutive requests.

Help me troubleshoot my Gemini API connection.

✔️ Awesome, I've got everything!

Great. Double check you have saved both .env and requirements.txt.

ⓧ I'd like to double check the full code

Here is what your .env file should contain:

GEMINI_API_KEY=your-api-key-here

And here is requirements.txt:

google-genai==2.11.0
chromadb==1.5.9
python-dotenv==1.1.0

Your environment is ready. You have Python running, packages installed, and a confirmed connection to the Gemini API. Next up, you will create the documents that your RAG system will search through.

Load and Chunk Your Documents

Your Python environment is ready and connected to Gemini. Now it is time to give your RAG system something to work with.

Large language models only know what they were trained on. To answer questions about your data, you need to feed it in. In this step, you will create sample documents, then build a Python script that loads and splits them into searchable chunks.

In this step, get ready to:
  • Create a folder of sample documents for your knowledge base.
  • Build a document loader and text chunker in Python.
  • Test the chunker to verify it splits documents correctly.
Create sample documents

Every RAG system needs a knowledge base. You will create a small set of text files about Python programming concepts that your system can search through and answer questions about.

  • Create a documents folder inside your project folder by running this command:
mkdir documents
  • Confirm the folder was created by running:
ls

You should see documents listed in the output alongside your existing files.

  • In VS Code's Explorer sidebar, expand the documents folder.
  • Create a new file inside documents called python_basics.txt and paste the following content:
Python is a high-level, interpreted programming language created by Guido van Rossum and first released in 1991. Python's design philosophy emphasizes code readability with its notable use of significant whitespace. It supports multiple programming paradigms, including structured, object-oriented, and functional programming.

Python uses dynamic typing and a combination of reference counting and a cycle-detecting garbage collector for memory management. It features a comprehensive standard library that is one of its greatest strengths, earning the nickname "batteries included."

Variables in Python do not need explicit declaration to reserve memory space. The declaration happens automatically when you assign a value to a variable. Python is dynamically typed, which means you don't have to declare what type each variable is.
  • Create a second file inside documents called python_functions.txt and paste the following content:
Functions in Python are defined using the def keyword. A function is a block of organized, reusable code that is used to perform a single, related action. Functions provide better modularity for your application and a high degree of code reusing.

Python gives you many built-in functions like print(), but you can also create your own functions. These functions are called user-defined functions. You can define functions to provide the required functionality.

Parameters are specified after the function name, inside the parentheses. You can add as many parameters as you want, just separate them with a comma. A return statement is used to exit a function and go back to the place from where it was called.

Default parameter values can be specified by using the assignment operator. If a function is called without the argument, the default value is used. Lambda functions are small anonymous functions defined with the lambda keyword. They can take any number of arguments but can only have one expression.
  • Create a third file inside documents called python_data_structures.txt and paste the following content:
Python has four built-in data structures: lists, tuples, dictionaries, and sets. Lists are ordered, mutable collections that can contain elements of different types. They are defined using square brackets.

Tuples are similar to lists but are immutable, meaning once created they cannot be changed. They are defined using parentheses. Tuples are faster than lists and can be used as dictionary keys.

Dictionaries are unordered collections of key-value pairs. They are defined using curly braces with key-value pairs separated by colons. Keys must be unique and immutable. Dictionaries provide fast lookup by key.

Sets are unordered collections of unique elements. They are defined using curly braces or the set() constructor. Sets support mathematical operations like union, intersection, and difference. They are useful for removing duplicates and membership testing.
  • Save all three files.

Why does a RAG system need documents?

LLMs like Gemini only know what they were trained on. If you ask about your company's internal docs or a niche topic, the model will either hallucinate or say it does not know.

RAG solves this by giving the LLM your own documents as context at query time. The model reads your data and answers based on facts, not guesses.

Build the document loader and chunker

Now you will write the Python script that loads those text files and splits them into smaller pieces. This script will grow throughout the project as you add embedding, storage, and query capabilities.

  • Create a new file in your project folder called rag_system.py and paste the following imports and initialization code:
import os
import chromadb
from google import genai
from google.genai import types
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize the Gemini client
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

# --- Step 2: Load and Chunk Documents ---

What does this code do?

  • The imports bring in the libraries you installed in Step 1: os for file operations, chromadb for the vector database, genai for the Gemini API, and dotenv for loading your API key.
  • load_dotenv() reads your .env file and makes GEMINI_API_KEY available as an environment variable.
  • genai.Client(api_key=...) creates your connection to the Gemini API using the key from your .env file.
  • Save rag_system.py.

Seeing an import error?

Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt. If not, reactivate it:

venv\Scripts\activate

Help me troubleshoot this

  • Below the comment line # --- Step 2: Load and Chunk Documents ---, add the load_documents function:
def load_documents(folder_path):
    """Load all .txt files from a folder."""
    documents = []
    for filename in os.listdir(folder_path):
        if filename.endswith(".txt"):
            filepath = os.path.join(folder_path, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read()
            documents.append({"filename": filename, "text": text})
    return documents

What does this function do?

  • os.listdir(folder_path) scans the folder and returns every file name inside it.
  • The if filename.endswith(".txt") filter ensures you only load text files.
  • Each document is stored as a dictionary with the filename (for tracking the source later) and the full text content.
  • Below load_documents, add a blank line and then the chunk_text function:
def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk.strip())
        start = end - overlap
    return chunks

Why do we chunk documents?

LLMs have context limits. If you pass an entire book as context, most of it will be irrelevant to a specific question. Smaller chunks let you retrieve only the most relevant piece.

The 50-character overlap prevents sentences from being cut in half at chunk boundaries. If a sentence spans two chunks, the overlap ensures it appears completely in at least one of them.

  • Below chunk_text, add a blank line and then the prepare_chunks function:
def prepare_chunks(documents):
    """Load documents and split them into chunks with metadata."""
    all_chunks = []
    for doc in documents:
        chunks = chunk_text(doc["text"])
        for i, chunk in enumerate(chunks):
            all_chunks.append({
                "id": f"{doc['filename']}_chunk_{i}",
                "text": chunk,
                "metadata": {"source": doc["filename"], "chunk_index": i},
            })
    return all_chunks

What does prepare_chunks do?

  • It loops through every document, splits it using chunk_text, and attaches metadata to each chunk.
  • Each chunk gets a unique id (filename + chunk number), the text content, and a metadata dictionary recording which file it came from.
  • This metadata is critical. When your system finds a relevant chunk later, you will know exactly which document it came from.
  • Save rag_system.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved rag_system.py before moving on.

ⓧ I'd like to double check the full code

import os
import chromadb
from google import genai
from google.genai import types
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize the Gemini client
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

# --- Step 2: Load and Chunk Documents ---

def load_documents(folder_path):
    """Load all .txt files from a folder."""
    documents = []
    for filename in os.listdir(folder_path):
        if filename.endswith(".txt"):
            filepath = os.path.join(folder_path, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read()
            documents.append({"filename": filename, "text": text})
    return documents


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk.strip())
        start = end - overlap
    return chunks


def prepare_chunks(documents):
    """Load documents and split them into chunks with metadata."""
    all_chunks = []
    for doc in documents:
        chunks = chunk_text(doc["text"])
        for i, chunk in enumerate(chunks):
            all_chunks.append({
                "id": f"{doc['filename']}_chunk_{i}",
                "text": chunk,
                "metadata": {"source": doc["filename"], "chunk_index": i},
            })
    return all_chunks
Test the chunker

Before moving on, you should verify that the loader and chunker work correctly. You will add a temporary test block to the bottom of your script, run it, then remove the test code.

  • At the very bottom of rag_system.py, add this temporary test code:
# --- Temporary test (remove after verifying) ---
if __name__ == "__main__":
    documents = load_documents("./documents")
    chunks = prepare_chunks(documents)
    print(f"Loaded {len(documents)} documents, split into {len(chunks)} chunks.\n")
    for i, chunk in enumerate(chunks):
        print(f"Chunk {i} [{chunk['metadata']['source']}]: {chunk['text'][:80]}...")
        print()
  • Save rag_system.py and run it by entering this command:
python rag_system.py

You should see output listing the number of documents loaded and each chunk with its source filename and a preview of the text content.

What should I see?

Your terminal should print something like Loaded 3 documents, split into X chunks followed by numbered chunks. Each chunk shows the source filename in square brackets and the first 80 characters of text.

The exact number of chunks depends on how long your documents are. With the sample files above, expect around 6-9 chunks total.

Seeing 0 documents loaded?

Make sure your documents folder is inside the same directory where rag_system.py lives. The path ./documents is relative to where you run the script from.

Also check that your files have the .txt extension. Files saved without an extension will not be picked up by the loader.

Help me debug this

  • Once you have confirmed the output looks correct, delete the temporary test block from the bottom of rag_system.py (everything below the prepare_chunks function).
  • Save rag_system.py.

Your documents are loaded, split, and tagged with metadata. Next up, you will convert these chunks into vector embeddings and store them in a vector database so your system can search them by meaning.

Create Embeddings and Store in ChromaDB

In the last step, you loaded your documents and split them into overlapping chunks. Those chunks are just plain text right now. They're not searchable by meaning.

In this step, you'll convert those text chunks into numerical vectors called embeddings. Then you'll store them in ChromaDB, a vector database that can find similar text by comparing numbers instead of matching exact words.

In this step, get ready to:
  • Generate embeddings for your document chunks using Gemini.
  • Store embeddings in a ChromaDB vector database.
  • Verify that documents are stored and retrievable.
Generate embeddings with Gemini

Right now, your chunks are just strings of characters. A computer can check if two strings are identical, but it can't tell that "functions" and "def keyword" are related concepts.

Embeddings solve this by converting text into arrays of numbers (vectors) that capture meaning. Text with similar meaning gets similar numbers, which makes "finding related content" a math problem instead of a keyword-matching problem.

  • In rag_system.py, scroll to the bottom of the file (below the prepare_chunks() function).
  • Add a new section comment and the get_embeddings() function by copying this code:


# --- Step 3: Create Embeddings and Store in ChromaDB ---

def get_embeddings(texts):
    """Generate embeddings for a list of texts using Gemini."""
    result = client.models.embed_content(
        model="gemini-embedding-2",
        contents=texts,
    )
    return [embedding.values for embedding in result.embeddings]

What does this code do?

This function takes a list of text strings and sends them to Gemini's embedding model (gemini-embedding-2) to get back numerical vectors.

Each text string becomes a list of 3072 numbers. The client.models.embed_content() call handles both single texts and batches, so you can embed all your chunks in one API call.

  • Save rag_system.py.

What are embeddings?

Think of embeddings as coordinates on a map of meaning. The sentence "Python uses dynamic typing" would land close to "variables don't need type declarations" because they describe the same concept. The sentence "pizza recipe" would land far away.

This is what makes semantic search possible. Instead of matching exact keywords, you compare how close two points are on this meaning map.

Store embeddings in ChromaDB

You need somewhere to store these vectors so you can search through them quickly. A regular database stores rows and columns. A vector database stores vectors and is optimized for finding the most similar ones fast.

  • In rag_system.py, add the build_vector_store() function directly below get_embeddings():


def build_vector_store(chunks, db_path="./chroma_db"):
    """Create ChromaDB collection and store document embeddings."""
    chroma_client = chromadb.PersistentClient(path=db_path)
    collection = chroma_client.get_or_create_collection(
        name="documents",
        embedding_function=None,
    )

    # Check if documents are already stored
    if collection.count() > 0:
        print(f"Vector store already contains {collection.count()} chunks. Skipping ingestion.")
        return collection

    # Generate embeddings for all chunks
    texts = [chunk["text"] for chunk in chunks]
    print(f"Generating embeddings for {len(texts)} chunks...")
    embeddings = get_embeddings(texts)

    # Add to ChromaDB
    collection.add(
        ids=[chunk["id"] for chunk in chunks],
        embeddings=embeddings,
        documents=texts,
        metadatas=[chunk["metadata"] for chunk in chunks],
    )
    print(f"Stored {len(chunks)} chunks in the vector database.")
    return collection

What does this code do?

  • chromadb.PersistentClient(path=db_path) creates a database that saves to disk. Your embeddings persist between script runs.
  • get_or_create_collection() creates a named collection (like a table) or reuses an existing one.
  • embedding_function=None tells ChromaDB that you'll provide your own embeddings from Gemini rather than using a built-in model.
  • The collection.count() > 0 check prevents duplicate ingestion if you run the script multiple times.
  • collection.add() stores the chunk IDs, embedding vectors, original text, and metadata together.
  • Save rag_system.py.

What is a vector database?

A vector database is purpose-built for storing and searching through high-dimensional vectors. When you query it, it compares the "direction" of your query vector against all stored vectors using a method called cosine similarity.

ChromaDB runs entirely on your local machine. There is no cloud account or server to manage. It saves everything to a folder on disk, which makes it perfect for learning and prototyping.

Seeing a syntax error?

Make sure there are exactly two blank lines between get_embeddings() and build_vector_store(). Python expects functions to be separated by blank lines. Also check that the indentation inside each function uses consistent spaces (4 spaces per level).

Help me fix a syntax error in my rag_system.py file

Verify the database

Your functions are written, but you haven't run them yet. You need to confirm that documents are being stored and that a test query returns relevant results.

  • At the very bottom of rag_system.py, add this temporary test block:


# --- Temporary test (delete after verifying) ---
if __name__ == "__main__":
    # Load and chunk documents
    documents = load_documents("./documents")
    chunks = prepare_chunks(documents)
    print(f"Prepared {len(chunks)} chunks from {len(documents)} documents.")

    # Build vector store
    collection = build_vector_store(chunks)
    print(f"\nCollection now contains {collection.count()} chunks.")

    # Test query
    test_embedding = get_embeddings(["What are Python functions?"])
    results = collection.query(
        query_embeddings=test_embedding,
        n_results=2,
        include=["documents", "metadatas"],
    )
    print(f"\nTest query results:")
    for doc, meta in zip(results["documents"][0], results["metadatas"][0]):
        print(f"  Source: {meta['source']}, Chunk: {meta['chunk_index']}")
        print(f"  Preview: {doc[:80]}...")

What does this test do?

This temporary block loads your documents, chunks them, generates embeddings, and stores everything in ChromaDB. Then it runs a test query asking about Python functions to confirm that the most relevant chunks come back.

  • Save rag_system.py.
  • Run the script by entering this command in your terminal:
python rag_system.py
  • You should see output showing that chunks were prepared, embeddings were generated, chunks were stored in the database, and the test query returned results from python_functions.txt.

What should I see?

Your output will look something like this:

Prepared 9 chunks from 3 documents.
Generating embeddings for 9 chunks...
Stored 9 chunks in the vector database.

Collection now contains 9 chunks.

Test query results:
  Source: python_functions.txt, Chunk: 0
  Preview: Functions in Python are defined using the def keyword. A function is a block of...
  Source: python_functions.txt, Chunk: 1
  Preview: ...

The exact chunk count depends on your document lengths, but the key thing is that the test query about "Python functions" returned chunks from python_functions.txt. That means semantic search is working.

Getting an API error?

If you see an authentication error, check that your .env file contains a valid GEMINI_API_KEY value. Make sure there are no extra spaces or quotes around the key.

If you see a rate limit error, wait 10-15 seconds and try again. The free tier has per-minute request limits.

Help me debug a Gemini API error in my RAG system

  • Now that you've confirmed everything works, delete the temporary test block from the bottom of rag_system.py (everything from the # --- Temporary test comment down to the end of the file).
  • Save rag_system.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file and deleted the temporary test block.

ⓧ I'd like to double check the full code

Here's the complete rag_system.py file up to this point:

import os
import chromadb
from google import genai
from google.genai import types
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize the Gemini client
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

# --- Step 2: Load and Chunk Documents ---

def load_documents(folder_path):
    """Load all .txt files from a folder."""
    documents = []
    for filename in os.listdir(folder_path):
        if filename.endswith(".txt"):
            filepath = os.path.join(folder_path, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read()
            documents.append({"filename": filename, "text": text})
    return documents


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk.strip())
        start = end - overlap
    return chunks


def prepare_chunks(documents):
    """Load documents and split them into chunks with metadata."""
    all_chunks = []
    for doc in documents:
        chunks = chunk_text(doc["text"])
        for i, chunk in enumerate(chunks):
            all_chunks.append({
                "id": f"{doc['filename']}_chunk_{i}",
                "text": chunk,
                "metadata": {"source": doc["filename"], "chunk_index": i},
            })
    return all_chunks


# --- Step 3: Create Embeddings and Store in ChromaDB ---

def get_embeddings(texts):
    """Generate embeddings for a list of texts using Gemini."""
    result = client.models.embed_content(
        model="gemini-embedding-2",
        contents=texts,
    )
    return [embedding.values for embedding in result.embeddings]


def build_vector_store(chunks, db_path="./chroma_db"):
    """Create ChromaDB collection and store document embeddings."""
    chroma_client = chromadb.PersistentClient(path=db_path)
    collection = chroma_client.get_or_create_collection(
        name="documents",
        embedding_function=None,
    )

    # Check if documents are already stored
    if collection.count() > 0:
        print(f"Vector store already contains {collection.count()} chunks. Skipping ingestion.")
        return collection

    # Generate embeddings for all chunks
    texts = [chunk["text"] for chunk in chunks]
    print(f"Generating embeddings for {len(texts)} chunks...")
    embeddings = get_embeddings(texts)

    # Add to ChromaDB
    collection.add(
        ids=[chunk["id"] for chunk in chunks],
        embeddings=embeddings,
        documents=texts,
        metadatas=[chunk["metadata"] for chunk in chunks],
    )
    print(f"Stored {len(chunks)} chunks in the vector database.")
    return collection

Your documents are now stored as searchable vectors. In the next step, you'll build the query pipeline that retrieves relevant chunks and generates grounded answers using Gemini.

Build the RAG Query Pipeline

Your documents are chunked and stored as vectors in ChromaDB. Now you need a way to actually use that knowledge.

In this step, you'll build the query pipeline that takes a natural language question, finds the most relevant chunks, and uses Gemini to generate a grounded answer. This is the core of any RAG system.

In this step, get ready to:
  • Build the retrieval function that searches your vector database for relevant chunks.
  • Write a prompt-engineered generation function that grounds Gemini's answers in your documents.
  • Create an interactive query loop where you can ask questions and get sourced answers.
Build the retrieval step

When you ask a question, the system needs to find which document chunks are most relevant. It does this by converting your question into a vector (using the same embedding model) and searching for the closest vectors in the database.

How does vector similarity search work?

Vectors that represent similar meanings point in similar directions in high-dimensional space. The database compares the "direction" of your question's vector against every stored chunk vector using a distance metric.

A lower distance score means a closer match. The top results are the chunks most likely to contain the answer to your question.

  • In rag_system.py, scroll to the bottom of the file (below build_vector_store) and add the retrieval function by copying this code:
# --- Step 4: Build the RAG Query Pipeline ---

def retrieve_context(question, collection, n_results=3):
    """Find the most relevant document chunks for a question."""
    question_embedding = get_embeddings([question])
    results = collection.query(
        query_embeddings=question_embedding,
        n_results=n_results,
        include=["documents", "metadatas", "distances"],
    )
    return results

What does this code do?

  • get_embeddings([question]) converts the user's question into a vector using the same Gemini embedding model that encoded your documents.
  • collection.query() searches ChromaDB for the 3 chunks whose vectors are closest to the question vector.
  • The include parameter tells ChromaDB to return the original text, metadata, and distance scores alongside each result.
  • Save the file. You won't see a change yet. The next function will use these retrieved chunks to generate an answer.
Build the generation step with prompt engineering

Retrieval alone gives you raw text chunks. You need prompt engineering to instruct Gemini on how to use those chunks to produce a useful answer.

What is prompt engineering?

Prompt engineering is the practice of designing instructions that guide an LLM to produce the output you want. A well-crafted prompt reduces hallucinations by telling the model exactly what information to use and what to do when it doesn't know the answer.

By including your retrieved chunks as "context" in the prompt, you ground the model's answer in real facts from your documents instead of letting it guess from its training data.

  • Below the retrieve_context function, add the generation function by copying this code:
def generate_answer(question, context_results):
    """Generate an answer using Gemini with retrieved context."""
    # Build context string from retrieved chunks
    context_parts = []
    for i, (doc, metadata) in enumerate(
        zip(context_results["documents"][0], context_results["metadatas"][0])
    ):
        context_parts.append(f"[Source: {metadata['source']}]\n{doc}")

    context_string = "\n\n---\n\n".join(context_parts)

    # Construct the prompt
    prompt = f"""You are a helpful assistant that answers questions based on the provided context.
Use ONLY the information from the context below to answer the question.
If the context does not contain enough information to answer the question, say so clearly.

Context:
{context_string}

Question: {question}

Answer:"""

    # Call Gemini to generate the answer
    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=prompt,
    )
    return response.text

What does this code do?

  • The first section loops through the retrieved chunks and labels each one with its source filename. This gives Gemini context about where each piece of information came from.
  • The prompt instructs Gemini to use ONLY the provided context and to admit when it doesn't have enough information. This is the key to preventing hallucinations.
  • client.models.generate_content() sends the assembled prompt to gemini-2.5-flash and returns the generated text.
  • Save the file. You won't see a change yet. The next chunk ties retrieval and generation together into a runnable pipeline.
Create the interactive query loop

You now have the two core pieces: retrieval and generation. The final step is to wire them together and give yourself a command-line interface to ask questions.

  • Below the generate_answer function, add the pipeline wrapper and main loop by copying this code:
def ask_question(question, collection):
    """Full RAG pipeline: retrieve context and generate answer."""
    # Retrieve relevant chunks
    results = retrieve_context(question, collection)

    # Generate answer
    answer = generate_answer(question, results)

    # Display results
    print(f"\nQuestion: {question}")
    print(f"\nAnswer: {answer}")
    print("\n--- Sources Used ---")
    for metadata in results["metadatas"][0]:
        print(f"  - {metadata['source']} (chunk {metadata['chunk_index']})")
    print()


# --- Main ---

def main():
    # Load and chunk documents
    documents = load_documents("./documents")
    if not documents:
        print("No documents found in ./documents folder. Please add .txt files.")
        return

    chunks = prepare_chunks(documents)
    print(f"Loaded {len(documents)} documents, split into {len(chunks)} chunks.")

    # Build vector store
    collection = build_vector_store(chunks)

    # Interactive query loop
    print("\n=== RAG Document Q&A System ===")
    print("Ask questions about your documents. Type 'quit' to exit.\n")

    while True:
        question = input("Your question: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not question:
            print("Please enter a question.\n")
            continue
        ask_question(question, collection)


if __name__ == "__main__":
    main()

What does this code do?

  • ask_question() ties the full RAG pipeline together: retrieve relevant chunks, generate an answer, then display both the answer and the source documents that informed it.
  • main() loads your documents, builds (or reconnects to) the vector store, then enters an infinite loop waiting for questions.
  • The loop handles edge cases: typing quit, exit, or q exits gracefully, and pressing Enter on an empty line prompts for input instead of crashing.
  • Save rag_system.py.
  • Run your RAG system by running this command:
python rag_system.py
  • You should see output showing your documents loaded and the prompt Your question: waiting for input.
  • Try asking a question about your documents, such as:
What are the four built-in data structures in Python?

You should see Gemini's answer followed by a "Sources Used" section listing which document chunks it drew from.

  • Type quit to exit the loop when you're done testing.

Not getting an answer?

  • If you see a rate limit error, wait 10-15 seconds and try again. The Gemini free tier has per-minute request limits.
  • If you see an API key error, double check that your .env file is in the same folder as rag_system.py and contains a valid key.
  • If the script says "No documents found", make sure your documents/ folder is in the same directory you're running the script from.

Help me debug my RAG pipeline

✔️ Awesome, I've got everything!

Great. Double check you've saved your file and your query loop is working.

ⓧ I'd like to double check the full code

Here's the complete rag_system.py file at this point:

import os
import chromadb
from google import genai
from google.genai import types
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize the Gemini client
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

# --- Step 2: Load and Chunk Documents ---

def load_documents(folder_path):
    """Load all .txt files from a folder."""
    documents = []
    for filename in os.listdir(folder_path):
        if filename.endswith(".txt"):
            filepath = os.path.join(folder_path, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read()
            documents.append({"filename": filename, "text": text})
    return documents


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk.strip())
        start = end - overlap
    return chunks


def prepare_chunks(documents):
    """Load documents and split them into chunks with metadata."""
    all_chunks = []
    for doc in documents:
        chunks = chunk_text(doc["text"])
        for i, chunk in enumerate(chunks):
            all_chunks.append({
                "id": f"{doc['filename']}_chunk_{i}",
                "text": chunk,
                "metadata": {"source": doc["filename"], "chunk_index": i},
            })
    return all_chunks


# --- Step 3: Create Embeddings and Store in ChromaDB ---

def get_embeddings(texts):
    """Generate embeddings for a list of texts using Gemini."""
    result = client.models.embed_content(
        model="gemini-embedding-2",
        contents=texts,
    )
    return [embedding.values for embedding in result.embeddings]


def build_vector_store(chunks, db_path="./chroma_db"):
    """Create ChromaDB collection and store document embeddings."""
    chroma_client = chromadb.PersistentClient(path=db_path)
    collection = chroma_client.get_or_create_collection(
        name="documents",
        embedding_function=None,
    )

    # Check if documents are already stored
    if collection.count() > 0:
        print(f"Vector store already contains {collection.count()} chunks. Skipping ingestion.")
        return collection

    # Generate embeddings for all chunks
    texts = [chunk["text"] for chunk in chunks]
    print(f"Generating embeddings for {len(texts)} chunks...")
    embeddings = get_embeddings(texts)

    # Add to ChromaDB
    collection.add(
        ids=[chunk["id"] for chunk in chunks],
        embeddings=embeddings,
        documents=texts,
        metadatas=[chunk["metadata"] for chunk in chunks],
    )
    print(f"Stored {len(chunks)} chunks in the vector database.")
    return collection


# --- Step 4: Build the RAG Query Pipeline ---

def retrieve_context(question, collection, n_results=3):
    """Find the most relevant document chunks for a question."""
    question_embedding = get_embeddings([question])
    results = collection.query(
        query_embeddings=question_embedding,
        n_results=n_results,
        include=["documents", "metadatas", "distances"],
    )
    return results


def generate_answer(question, context_results):
    """Generate an answer using Gemini with retrieved context."""
    # Build context string from retrieved chunks
    context_parts = []
    for i, (doc, metadata) in enumerate(
        zip(context_results["documents"][0], context_results["metadatas"][0])
    ):
        context_parts.append(f"[Source: {metadata['source']}]\n{doc}")

    context_string = "\n\n---\n\n".join(context_parts)

    # Construct the prompt
    prompt = f"""You are a helpful assistant that answers questions based on the provided context.
Use ONLY the information from the context below to answer the question.
If the context does not contain enough information to answer the question, say so clearly.

Context:
{context_string}

Question: {question}

Answer:"""

    # Call Gemini to generate the answer
    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=prompt,
    )
    return response.text


def ask_question(question, collection):
    """Full RAG pipeline: retrieve context and generate answer."""
    # Retrieve relevant chunks
    results = retrieve_context(question, collection)

    # Generate answer
    answer = generate_answer(question, results)

    # Display results
    print(f"\nQuestion: {question}")
    print(f"\nAnswer: {answer}")
    print("\n--- Sources Used ---")
    for metadata in results["metadatas"][0]:
        print(f"  - {metadata['source']} (chunk {metadata['chunk_index']})")
    print()


# --- Main ---

def main():
    # Load and chunk documents
    documents = load_documents("./documents")
    if not documents:
        print("No documents found in ./documents folder. Please add .txt files.")
        return

    chunks = prepare_chunks(documents)
    print(f"Loaded {len(documents)} documents, split into {len(chunks)} chunks.")

    # Build vector store
    collection = build_vector_store(chunks)

    # Interactive query loop
    print("\n=== RAG Document Q&A System ===")
    print("Ask questions about your documents. Type 'quit' to exit.\n")

    while True:
        question = input("Your question: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not question:
            print("Please enter a question.\n")
            continue
        ask_question(question, collection)


if __name__ == "__main__":
    main()

Your RAG pipeline is fully operational. You can now ask any question about your documents and get grounded, sourced answers. Next up, you'll take on a secret mission to add source citations and confidence scoring to make your answers even more trustworthy.

Secret mission

Add Source Citations and Confidence Scoring

Your RAG system answers questions, but how does the user know which document the answer came from? And what happens when someone asks a question your documents can't answer? In this secret mission, you'll add numbered source citations and a confidence indicator that warns users when answers might not be well-supported.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your project files for future experimentation or delete them to free up disk space.

Resources you used:

  • Project folder on your Desktop (contains rag_system.py, documents/, chroma_db/, venv/, .env, and requirements.txt).
  • Gemini API key stored in Google AI Studio (free tier, no billing).
  • Python 3.14 installation on your system.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your RAG system or want to add more documents later.

  • Your project folder contains everything. The ChromaDB data persists between runs.
  • To use the system again, open VS Code, activate your virtual environment, and run python rag_system.py.
  • Your Gemini API key remains active and free to use.
  • Python stays installed on your system for any future projects.

Pause - I'll come back to this later

Deactivate your virtual environment to free up terminal resources. Everything else stays on disk and costs nothing.

  • In your VS Code terminal, deactivate the virtual environment by running:
deactivate
  • Your project folder, ChromaDB database, and API key all remain intact for next time.
  • When you return, reactivate the virtual environment with venv\Scripts\activate (Windows) or source venv/bin/activate (macOS/Linux).

Delete - I don't want to use this again

Remove all project resources completely. This deletes your code, documents, vector database, and virtual environment.

  • Deactivate your virtual environment first by running:
deactivate
  • Delete your project folder from the Desktop by running:
cd ~/Desktop
rm -rf rag-document-qa

On Windows, use this command instead:

cd ~\Desktop
rmdir /s /q rag-document-qa

Use your actual folder name

Replace rag-document-qa with whatever you named your project folder when you created it in Step 1.

  • To delete your Gemini API key, go to aistudio.google.com.
  • Click API Keys in the left sidebar.
  • Find the key you created, click the three-dot menu icon, and select Delete.
  • To uninstall Python, open Windows Settings, go to Apps, then Installed apps.
  • Search for Python 3.14, click the three-dot menu, and select Uninstall.

Nice Work!

Nice Work!

You just built a complete RAG pipeline from scratch using Python, Google Gemini, and ChromaDB. This is the same architecture pattern that powers document search, customer support bots, and internal knowledge tools at companies worldwide.

Here's what you learned:

  • Converted text documents into vector embeddings using the Gemini API and stored them in a ChromaDB vector database for fast semantic search.
  • Built a retrieval pipeline that finds relevant document chunks by meaning, not just keyword matching.
  • Applied prompt engineering to ground LLM answers in source documents, reducing hallucinations and producing accurate, contextual responses.
  • Secret Mission: Added source citations and a confidence indicator to make answers trustworthy and transparent.

Ready to quiz yourself?