Build a RAG Document Q&A System

Build a RAG pipeline that answers questions about your own text documents.

Introduction

30 Second Summary

You have probably asked an AI chatbot about your own notes or a company policy and gotten a vague or completely made-up answer. That happens because large language models do not have access to your private documents.

In this project, you will build a RAG-powered Q&A system that solves this problem. You will feed text documents into a Python pipeline that retrieves the most relevant passages and sends them to Google's Gemini API to generate accurate, grounded answers.

What You'll Build

When you finish, you will type questions about your documents into the terminal and get back precise answers, with the exact source passages printed as proof.

By the end of this project, you'll have:

  • Load any .txt document and search it by meaning using vector embeddings stored in ChromaDB.
  • Ask natural language questions and get accurate, source-backed answers from Gemini.
  • Test with off-topic questions and watch the system refuse to hallucinate, telling you it does not have enough information instead.
  • Secret Mission: Upgrade the system to handle multiple documents with metadata filtering, so you can target questions at one specific source.

Are there any prerequisites?

This project assumes basic Python knowledge (variables, functions, and file handling). Everything else, including the Gemini API setup and ChromaDB, is covered step by step.

Before We Start

Before you dive in, take a moment to lock in what you are building and why it matters for your career as an AI/ML engineer.

Set Up Your Environment and Get a Gemini API Key

Every RAG pipeline depends on two capabilities: turning text into searchable vector embeddings, and generating natural language answers from retrieved context.

In this project, those capabilities come from the Gemini API and ChromaDB. In this step, you will create your project folder, install the required Python packages, and grab a free API key from Google AI Studio.

In this step, get ready to:
  • Create a project folder with a Python virtual environment.
  • Install the google-genai and chromadb packages.
  • Get a free Gemini API key from Google AI Studio.
Create a project folder and virtual environment
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder....
  • Navigate to your Desktop in the file dialog.
  • Click New Folder.
  • Name the folder rag-qa.
  • Select the rag-qa folder, then click Open (macOS) or Select Folder (Windows).

Seeing a trust prompt?

VS Code may ask if you trust the authors of the files in this folder. Click Trust to enable all features. This is a security feature that restricts code execution in folders you have not approved.

You now have the rag-qa folder open in VS Code. Next, open a terminal so you can create your virtual environment.

  • Click Terminal in the top menu bar.
  • Click New Terminal.

A terminal panel opens at the bottom of VS Code, already pointed at your rag-qa folder.

  • Create and activate a virtual environment by running the commands for your operating system:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

What is a virtual environment?

A virtual environment is an isolated Python workspace. Packages you install inside it stay separate from your system Python, so different projects can use different package versions without conflicting.

You should see (venv) at the beginning of your terminal prompt. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

  • Make sure you ran the commands for the correct operating system tab above.
  • On Windows, if you see an error about running scripts being disabled, run this command in PowerShell, then try the activate command again:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Still stuck? Help me activate my Python virtual environment in VS Code.

Install the required Python packages

Two packages power this project. The google-genai SDK is Google's official Python library for calling the Gemini API. chromadb is an open-source vector database that runs entirely in-memory on your machine.

  • Install both packages by running this command:
pip install google-genai chromadb

This may take a minute or two. When it finishes, you should see a line that includes Successfully installed followed by a list of package names.

Seeing an install error?

  • Make sure (venv) appears in your terminal prompt. If it does not, go back and activate the virtual environment first, then re-run the install command.
  • If you see a permissions error, do not use sudo. Instead, confirm you are inside the virtual environment.

Still stuck? Help me fix my pip install error for google-genai and chromadb.

Get a free Gemini API key

The Gemini API requires an API key to authenticate your requests. Google offers a free tier with no billing required, so this project costs nothing to run.

  • Open Google AI Studio in your browser.
  • Sign in with your Google account.
  • Click Create API key.
  • Copy the key that appears. You will need it in the next step.

Keep your API key private

Never share your API key or commit it to a public repository. If you ever push this project to GitHub, add .env to your .gitignore file so the key stays out of version control.

Now store the key in your project so your code can access it.

  • In VS Code's Explorer sidebar, click the New File... button at the top of the file list.
  • Name the file .env and press Enter.
  • Paste the following line into the file, replacing your-api-key-here with the actual key you just copied:
GEMINI_API_KEY=your-api-key-here

Why a .env file?

A .env file keeps your API key separate from your code. Your application reads the key from this file at runtime, so you never paste the key directly into your Python scripts.

  • Save .env by pressing Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Make sure you have saved your .env file with your real API key in place of your-api-key-here.

ⓧ I'd like to double check the full code

Your .env file should contain exactly one line (with your actual key replacing the placeholder):

GEMINI_API_KEY=your-api-key-here
Verify your setup

Before you run this, what do you think will happen if both packages imported correctly?

  • Verify that everything is installed correctly by running this command in your terminal:
python -c "from google import genai; import chromadb; print('Setup complete!')"

You should see Setup complete! printed in the terminal. This confirms that both google-genai and chromadb are installed and importable.

Seeing a ModuleNotFoundError?

  • Confirm (venv) appears in your terminal prompt. If it does not, activate your virtual environment first, then re-run the pip install command.
  • If you closed and reopened the terminal, the virtual environment deactivates. Reactivate it using the platform-specific command from earlier in this step.

Still stuck? Help me fix my ModuleNotFoundError for google-genai or chromadb.

Your environment is ready and your API key is in place. Next up, you will load a document and ask Gemini a question directly to see what happens without RAG.

Load a Document and Ask Gemini a Question Directly

Your environment is ready with Python, the Gemini SDK, and ChromaDB installed. Now it is time to see why you need all of these tools.

Before building any retrieval system, you need to understand the problem it solves. In this step, you will ask Gemini a question about a private document and see firsthand that it cannot answer accurately.

In this step, get ready to:
  • Create a sample text document with facts that Gemini has never seen.
  • Write a Python script that sends a question about your document directly to Gemini.
  • Run the script and observe Gemini's inability to answer, exposing the problem that RAG solves.
Create a sample text document

You need a document containing specific facts that a general-purpose LLM would not know. A fictional company handbook is perfect because none of its details exist in Gemini's training data.

  • In VS Code's Explorer sidebar, right-click inside the rag-qa folder and select New File.
  • Name the file data.txt and press Enter.
  • Paste the following content into data.txt:
NextWork Corp Employee Handbook - Remote Work Policy

All full-time employees at NextWork Corp are eligible for remote work up to three days per week. Remote work days must be approved by the employee's direct manager at least 48 hours in advance. Employees must be available during core hours of 10 AM to 3 PM in their local time zone.

Equipment and Expenses: NextWork Corp provides a one-time home office stipend of $500 for new employees. This covers essential equipment such as a monitor, keyboard, and ergonomic chair. Ongoing internet expenses are reimbursed up to $75 per month upon submission of receipts through the expense portal.

Performance and Accountability: Remote employees are expected to maintain the same productivity standards as in-office staff. Managers will conduct monthly check-ins to review progress on deliverables. If performance issues arise, the remote work arrangement may be modified or revoked with two weeks notice.

Security Requirements: All remote work must be conducted on company-approved devices with VPN enabled. Employees must not access company systems from public Wi-Fi networks without using the corporate VPN. Any security incidents must be reported to the IT helpdesk within one hour.

What is this document?

This is a fictional employee handbook for "NextWork Corp." It contains specific facts like a $500 home office stipend, $75/month internet reimbursement, and a one-hour security incident reporting window.

These details do not exist anywhere on the internet. If Gemini answers correctly about them later, you know it came from your RAG pipeline, not from its training data.

  • Save data.txt by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Confirm the file exists by running this in your terminal:
ls
  • You should see data.txt listed alongside your .env file and venv folder.
Write a script to query Gemini directly

Now you will build a Python script that loads your API key and asks Gemini a question about the document you just created. The script sends the question directly, with no document context attached.

  • Create a new file called app.py in your rag-qa folder by right-clicking in the Explorer sidebar and selecting New File.
  • Set up the API key loading and Gemini client by pasting this code into app.py:
import os
from google import genai

# Load API key
api_key = None
if os.path.exists(".env"):
    with open(".env") as f:
        for line in f:
            if line.startswith("GEMINI_API_KEY="):
                api_key = line.strip().split("=", 1)[1]

if not api_key:
    print("Error: Set your GEMINI_API_KEY in the .env file")
    exit(1)

# Initialize Gemini client
gemini_client = genai.Client(api_key=api_key)

What does this code do?

  • The script reads your .env file line by line, looking for a line that starts with GEMINI_API_KEY=. It splits on the first = sign to grab the key value.
  • If the key is missing, the script prints an error and stops immediately with exit(1).
  • genai.Client(api_key=api_key) creates a Gemini client that all future API calls will go through.
  • Save app.py.
  • Verify your setup code works by running this in your terminal:
python app.py
  • The script should exit with no output and no errors. That means your API key was loaded and the Gemini client was created successfully.

Seeing an error?

  • If you see "Error: Set your GEMINI_API_KEY in the .env file", open your .env file and make sure the line starts with exactly GEMINI_API_KEY= followed by your key, with no spaces around the = sign.
  • If you see "ModuleNotFoundError: No module named 'google'", your virtual environment may not be active. Run venv\Scripts\activate (Windows) or source venv/bin/activate (macOS) and try again.

Why is my app.py setup not running?

Now add the code that actually asks Gemini a question. This question is about a specific fact from your document.

  • Add the following code to the bottom of app.py, below the line that says gemini_client = genai.Client(api_key=api_key):
# Ask Gemini a question directly (without RAG)
question = "What is the home office stipend amount for new employees at NextWork Corp?"

response = gemini_client.models.generate_content(
    model="gemini-2.0-flash",
    contents=question,
)
print(response.text)

What does this code do?

  • question holds a specific question about the $500 home office stipend from your data.txt file. This is a fact Gemini has never seen.
  • client.models.generate_content() sends the question to the gemini-2.0-flash model. Notice that you are only sending the question. The document itself is never passed to Gemini.
  • Save app.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete app.py file so far. Make sure yours matches exactly.

import os
from google import genai

# Load API key
api_key = None
if os.path.exists(".env"):
    with open(".env") as f:
        for line in f:
            if line.startswith("GEMINI_API_KEY="):
                api_key = line.strip().split("=", 1)[1]

if not api_key:
    print("Error: Set your GEMINI_API_KEY in the .env file")
    exit(1)

# Initialize Gemini client
gemini_client = genai.Client(api_key=api_key)

# Ask Gemini a question directly (without RAG)
question = "What is the home office stipend amount for new employees at NextWork Corp?"

response = gemini_client.models.generate_content(
    model="gemini-2.0-flash",
    contents=question,
)
print(response.text)
Run the script and observe the gap

Your script asks Gemini about the home office stipend at NextWork Corp. The answer is $500, according to your data.txt file. But you never gave Gemini that file.

Before you run this, what do you think Gemini will answer? Will it know the correct stipend amount, or will it make something up?

  • Run the script by entering this command in your terminal:
python app.py

Gemini responds with a generic or incorrect answer. It might invent a stipend amount, give a vague answer about remote work policies in general, or admit it does not have that information.

Why is the answer wrong?

Gemini was trained on public internet data. Your data.txt file is a private document that never appeared in that training data. So Gemini has zero knowledge of what NextWork Corp's policies say.

This is the core problem that Retrieval-Augmented Generation (RAG) solves. Instead of hoping the model already knows your data, you retrieve the relevant parts of your documents and pass them to the model as context. The model then generates an answer grounded in your actual content.

Script hanging or returning an API error?

  • If the script hangs for more than 30 seconds, press Ctrl+C to stop it. Check your internet connection and try again.
  • If you see an authentication or invalid API key error, open your .env file and verify your key is pasted correctly with no extra spaces or line breaks.

Help me debug my Gemini API call in app.py.

You have just seen the gap that RAG fills. Gemini is powerful, but it is blind to your private documents. In the next step, you will break your document into chunks, convert them into embeddings, and store them in ChromaDB so Gemini can actually search through your content.

Chunk the Document and Store Embeddings in ChromaDB

In the last step, you saw Gemini fall flat when asked about your document's content. The model gave a generic or made-up answer because it has no access to your private data.

The fix is to break your document into small pieces, convert each piece into a numerical embedding vector, and store those vectors in a vector database. This lets you search by meaning instead of by keywords. In this step, you will build the chunking and storage layer using ChromaDB.

In this step, get ready to:
  • Write a function to load your document and split it into overlapping text chunks.
  • Generate embedding vectors for each chunk and store them in ChromaDB.
Update your imports and add the chunking function

Your current app.py sends a question straight to Gemini. You need to replace that direct query with a chunking pipeline that prepares your document for search.

  • In app.py, find the import lines at the top of the file:
import os
from google import genai
  • Replace those two lines with these four imports:
import os
import chromadb
from google import genai
from google.genai import types
  • Next, find the comment and line that initialize the Gemini client:
# Initialize Gemini client
gemini_client = genai.Client(api_key=api_key)
  • Replace those two lines with this updated initialization that adds a ChromaDB client:
# Initialize clients
gemini_client = genai.Client(api_key=api_key)
chroma_client = chromadb.Client()
  • Delete everything below the chroma_client line you just added, starting from # Ask Gemini a question directly (without RAG) all the way to the end of the file.

Why delete the direct query?

That code was a one-time experiment to prove Gemini cannot answer questions about your private document. You have seen the gap. Now you are replacing it with the pipeline that closes it.

  • Below the chroma_client = chromadb.Client() line, add the following function and main code by copying it to the end of your file:
def load_and_chunk(filepath, chunk_size=500, overlap=50):
    """Load a text file and split it into overlapping chunks."""
    with open(filepath, "r", encoding="utf-8") as f:
        text = f.read()

    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start += chunk_size - overlap

    return chunks


# Load and chunk the document
print("Loading and chunking document...")
chunks = load_and_chunk("data.txt")
print(f"Created {len(chunks)} chunks")

What does this code do?

The load_and_chunk function reads your entire text file and walks through it in steps of chunk_size characters. Each chunk overlaps the previous one by overlap characters so no sentence gets cut off at a boundary without context.

The main code at the bottom calls this function on data.txt and prints how many chunks were created.

  • Save app.py.
  • Run the script to confirm chunking works:
python app.py

You should see output like this:

Loading and chunking document...
Created 4 chunks

Why 4 chunks?

Your data.txt is roughly 1,200 characters. With a chunk size of 500 characters and 50 characters of overlap, the function steps forward 450 characters at a time. That produces about 4 chunks depending on the exact length of your file.

Not seeing any output?

Make sure you deleted all the old direct-query code below the initialization section. If you see an error about chromadb or types, check that your import lines match the four-line block above exactly.

Help me debug my chunking script.

Generate embeddings and store them in ChromaDB

Your document is now split into chunks, but chunks are just text. To search them by meaning, you need to convert each chunk into a numerical vector called an embedding. The Gemini API generates these vectors, and ChromaDB stores and searches them.

  • In app.py, add the following function between the load_and_chunk function and the # Load and chunk the document comment:
def store_chunks(collection, chunks, source_name):
    """Embed chunks and store them in ChromaDB."""
    for i, chunk in enumerate(chunks):
        result = gemini_client.models.embed_content(
            model="gemini-embedding-001",
            contents=chunk,
            config=types.EmbedContentConfig(output_dimensionality=768),
        )
        embedding = result.embeddings[0].values

        collection.add(
            ids=[f"{source_name}_chunk_{i}"],
            embeddings=[embedding],
            documents=[chunk],
            metadatas=[{"source": source_name}],
        )

    print(f"Stored {len(chunks)} chunks from '{source_name}' in ChromaDB")

What does this code do?

  • gemini_client.models.embed_content() sends each chunk to the gemini-embedding-001 model, which returns a 768-dimension numerical vector representing the chunk's meaning.
  • collection.add() stores each chunk's text, its embedding vector, a unique ID, and a source metadata tag in ChromaDB.
  • The output_dimensionality=768 setting uses the smallest recommended dimension to keep memory usage low and speed up searches.
  • Now scroll to the bottom of app.py and find the line print(f"Created {len(chunks)} chunks").
  • Add the following code directly below it:
# Store chunks with embeddings in ChromaDB
print("\nGenerating embeddings and storing in ChromaDB...")
collection = chroma_client.create_collection(name="documents")
store_chunks(collection, chunks, "data.txt")

What does this code do?

chroma_client.create_collection(name="documents") creates a new in-memory collection in ChromaDB to hold your chunks and their embeddings. Then store_chunks loops through every chunk, embeds it, and adds it to that collection.

  • Save app.py.

Before you run this, what do you think the output will look like? You already saw the chunk count. What new line do you expect?

  • Run the script to see the full pipeline in action:
python app.py

You should see output confirming both steps:

Loading and chunking document...
Created 4 chunks

Generating embeddings and storing in ChromaDB...
Stored 4 chunks from 'data.txt' in ChromaDB

What just happened?

Your script read data.txt, split it into overlapping chunks, sent each chunk to the Gemini embedding model, and stored everything in an in-memory ChromaDB collection. Your document is now searchable by meaning.

Seeing an API error?

If you see an authentication error, double-check that your .env file contains a valid API key with no extra spaces or quotes around the value.

If you see a model-not-found error, make sure the model string is exactly gemini-embedding-001 with no typos.

Help me fix my embedding storage errors.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py and that your terminal shows both the chunk count and the storage confirmation.

ⓧ I'd like to double check the full code

Your app.py should contain these sections in order. Scroll through your file and confirm each one is present:

  • Four import lines: os, chromadb, genai, and types.
  • The API key loading block (unchanged from before).
  • The # Initialize clients section with both gemini_client and chroma_client.
  • The load_and_chunk function (reads a file and splits it into overlapping chunks).
  • The store_chunks function (embeds each chunk and adds it to ChromaDB).
  • Main code that calls load_and_chunk("data.txt") and then store_chunks(collection, chunks, "data.txt").

Make sure the old direct-query code (the question variable, generate_content call, and print(response.text)) is completely removed.

Your document is now loaded, chunked, and stored as searchable vectors. Next up, you will build the retrieval and generation layer so you can ask a question and get an accurate, grounded answer from your own data.

Retrieve Relevant Chunks and Generate a Grounded Answer

Your document is now stored as searchable embeddings inside ChromaDB. In Step 2, you saw that Gemini could not answer questions about your private data on its own. Now it is time to close that gap.

This step builds the core of the RAG pattern: retrieve the most relevant chunks for a question, then pass them as context to Gemini so it generates an answer grounded in your actual document.

In this step, get ready to:
  • Build a retrieval function that searches ChromaDB by semantic similarity.
  • Build a generation function that feeds retrieved chunks as context to Gemini.
  • Run the complete pipeline and compare the grounded answer to Step 2's generic response.
Build the retrieval function

To find the right chunks for a question, you need to convert the question itself into an embedding. Then you query ChromaDB for the chunks whose embeddings are closest in meaning.

  • In app.py, scroll down to the end of the store_chunks() function.
  • Add the following retrieve() function below it, with a blank line separating the two functions:
def retrieve(collection, question, n_results=3):
    """Retrieve the most relevant chunks for a question."""
    result = gemini_client.models.embed_content(
        model="gemini-embedding-001",
        contents=question,
        config=types.EmbedContentConfig(output_dimensionality=768),
    )
    query_embedding = result.embeddings[0].values

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
    )

    return results["documents"][0], results["metadatas"][0]

What does this code do?

  • gemini_client.models.embed_content() converts the user's question into a 768-dimension embedding vector, using the same model and settings that encoded your document chunks.
  • collection.query() performs a semantic search across all stored chunks. It returns the top 3 chunks whose embeddings are closest in meaning to the question embedding.
  • The function returns both the chunk text and their metadata, so you can trace each piece of evidence back to its source file.

This function cannot be tested on its own yet. The next function uses the chunks it retrieves to generate an answer.

Build the generation-with-context function

Retrieving chunks is only half of the RAG pattern. The other half is sending those chunks to Gemini as context, with instructions to answer only from the provided evidence.

  • Below the retrieve() function, add the generate_answer() function:
def generate_answer(question, context_chunks):
    """Generate an answer using retrieved context."""
    context = "\n\n---\n\n".join(context_chunks)

    prompt = f"""You are a helpful assistant that answers questions based ONLY on the provided context.
If the context does not contain enough information to answer the question, say:
"I don't have enough information in the provided documents to answer this question."

Do not make up information. Only use what is explicitly stated in the context.

Context:
{context}

Question: {question}

Answer:"""

    response = gemini_client.models.generate_content(
        model="gemini-2.0-flash",
        contents=prompt,
    )
    return response.text

What does this code do?

  • context = "\n\n---\n\n".join(context_chunks) joins the retrieved chunks into a single string, separated by dividers so Gemini can distinguish each source passage.
  • The prompt is the key to grounding. It tells Gemini to answer ONLY from the provided context and to say "I don't know" when the context is not sufficient. This is prompt engineering at work. Without these constraints, Gemini would fall back to its general training data and potentially hallucinate.
  • client.models.generate_content() calls gemini-2.0-flash with the constructed prompt to produce a grounded answer.
  • Save app.py.

Why does the prompt matter so much?

In Step 2, Gemini had no context and made up an answer (or gave a vague one). The prompt in generate_answer() fixes that by feeding the retrieved evidence directly into the request. It also tells Gemini to refuse when the evidence is insufficient.

This is the difference between a chatbot that hallucinates and one that stays honest.

Wire up the main function and test the pipeline

Right now, the loading and storing code sits as loose lines at the bottom of app.py. You need to wrap it in a main() function and add the retrieval and generation steps to complete the pipeline.

  • In app.py, scroll to the bottom and find this block of loose code:
# Load and chunk the document
print("Loading and chunking document...")
chunks = load_and_chunk("data.txt")
print(f"Created {len(chunks)} chunks")

# Store chunks with embeddings in ChromaDB
print("\nGenerating embeddings and storing in ChromaDB...")
collection = chroma_client.create_collection(name="documents")
store_chunks(collection, chunks, "data.txt")
  • Delete that entire block (all 9 lines).
  • Replace it with the following main() function, which wraps the existing load/store logic and adds retrieval and generation:
def main():
    # Load and chunk the document
    print("Loading and chunking document...")
    chunks = load_and_chunk("data.txt")
    print(f"Created {len(chunks)} chunks")

    # Store chunks with embeddings in ChromaDB
    print("\nGenerating embeddings and storing in ChromaDB...")
    collection = chroma_client.create_collection(name="documents")
    store_chunks(collection, chunks, "data.txt")

    # Ask a question
    question = "What is the home office stipend amount for new employees at NextWork Corp?"
    retrieved_chunks, metadatas = retrieve(collection, question)

    print("\n--- Retrieved Evidence ---")
    for i, (chunk, meta) in enumerate(zip(retrieved_chunks, metadatas)):
        print(f"[Source: {meta['source']}] {chunk[:100]}...")

    answer = generate_answer(question, retrieved_chunks)
    print(f"\n--- Answer ---\n{answer}\n")


if __name__ == "__main__":
    main()

What does this code do?

  • main() runs the entire RAG pipeline in order: load the document, chunk it, embed and store the chunks, then retrieve and generate an answer.
  • The question is hardcoded for now so you can compare the grounded answer with Step 2's generic response. You will add an interactive loop in the next step.
  • if __name__ == "__main__": ensures main() only runs when you execute the file directly, not when it is imported as a module.
  • Save app.py.

Before you run this, think back to Step 2. When you asked Gemini about the home office stipend without providing any context, it gave a generic or incorrect answer. Do you think the response will be different this time?

  • Run the complete RAG pipeline by running this command:
python app.py

You should see three sections in the output. First, the loading and embedding logs you saw before. Then, a "Retrieved Evidence" section showing the chunks that matched your question. Finally, an "Answer" section with an accurate, specific response mentioning the $500 stipend from data.txt.

That is the RAG difference. The same model that could not answer this question in Step 2 now gives a precise, grounded response because it has the relevant document chunks as context.

Seeing an error or an incorrect answer?

  • If you see a NameError for retrieve or generate_answer, make sure both functions are defined above the main() function in app.py.
  • If you see an IndentationError, check that all lines inside main() are indented with exactly 4 spaces.
  • If the answer is still vague, confirm you deleted all of the loose code at the bottom and replaced it with the main() function. The old code did not call retrieve() or generate_answer().

Help me debug my RAG pipeline in app.py

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py.

ⓧ I'd like to double check the full code

Your app.py is now over 30 lines, so scroll through and confirm each section is present in this order:

  • Imports (os, chromadb, genai, types) and API key loading at the top.
  • Client initialization: gemini_client and chroma_client.
  • load_and_chunk() function (from Step 3).
  • store_chunks() function (from Step 3).
  • retrieve() function (added in this step).
  • generate_answer() function (added in this step).
  • main() function and if __name__ == "__main__": block at the very bottom.

Make sure no loose loading or storing code remains outside of main(). All pipeline logic should be inside the function.

Your RAG pipeline is working. Gemini now answers from evidence instead of guessing. Next up, you will add an interactive question loop so you can keep asking questions, and teach the system to gracefully say "I don't know" when a question falls outside your document.

Add a Question Loop and Handle Out-of-Scope Questions

In the last step, you proved that RAG works. Your system retrieved relevant chunks and gave an accurate, grounded answer to a question about the NextWork Corp handbook.

There is a problem though. The system only answers one hardcoded question and exits. A real Q&A system lets users ask anything, as many times as they want. It also needs to handle questions the documents cannot answer without making something up.

In this step, get ready to:
  • Add an interactive question loop so users can ask multiple questions.
  • Test out-of-scope question handling to confirm the system does not hallucinate.
Add an interactive question loop

Right now, main() runs one hardcoded question and stops. You need a loop that keeps asking the user for questions until they decide to quit.

  • In app.py, scroll down to the main() function.
  • Find this block of code that starts with the # Ask a question comment:
    # Ask a question
    question = "What is the home office stipend amount for new employees at NextWork Corp?"
    retrieved_chunks, metadatas = retrieve(collection, question)

    print("\n--- Retrieved Evidence ---")
    for i, (chunk, meta) in enumerate(zip(retrieved_chunks, metadatas)):
        print(f"[Source: {meta['source']}] {chunk[:100]}...")

    answer = generate_answer(question, retrieved_chunks)
    print(f"\n--- Answer ---\n{answer}\n")
  • Delete that entire block (from # Ask a question down to the last line of main()) and replace it with this interactive loop:
    # Step 3: Interactive question loop
    print("\n--- RAG Q&A System Ready ---")
    print("Ask questions about your documents. Type 'quit' to exit.\n")

    while True:
        question = input("Your question: ").strip()
        if question.lower() == "quit":
            print("Goodbye!")
            break
        if not question:
            continue

        # Retrieve relevant chunks
        retrieved_chunks, metadatas = retrieve(collection, question)

        print("\n--- Retrieved Evidence ---")
        for i, (chunk, meta) in enumerate(zip(retrieved_chunks, metadatas)):
            print(f"[Source: {meta['source']}] {chunk[:100]}...")

        # Generate answer
        answer = generate_answer(question, retrieved_chunks)
        print(f"\n--- Answer ---\n{answer}\n")

What does this code do?

  • while True: creates an infinite loop that keeps prompting for questions until the user types quit.
  • input("Your question: ").strip() waits for the user to type a question. .strip() removes any accidental whitespace.
  • if not question: continue skips blank inputs so the system does not waste an API call on an empty string.
  • Each iteration retrieves relevant chunks and generates an answer, exactly the same pipeline you built in the previous step.
  • While you are in main(), update the two comments above your new loop so they include step numbers. Change # Load and chunk the document to # Step 1: Load and chunk the document.
  • Change # Store chunks with embeddings in ChromaDB to # Step 2: Store chunks with embeddings in ChromaDB.
  • Save app.py.
  • Run the system by entering this command in your terminal:
python app.py

You should see the loading output from previous steps, followed by a new prompt: --- RAG Q&A System Ready ---.

  • Type a question about the handbook content, for example:
How many days per week can employees work remotely?

The system retrieves evidence chunks and prints a grounded answer. You should see it correctly answer that employees can work remotely up to three days per week.

Seeing an error when you run the script?

  • If you see an IndentationError, make sure the entire while True: block is indented with 4 spaces inside main().
  • If the script loads chunks but never shows the Your question: prompt, check that you deleted the old hardcoded question block completely before pasting the new loop.
  • If you see a NameError for collection, make sure the lines that create the collection and call store_chunks() are still above the loop.

Still stuck? Help me debug why my interactive question loop is not working in app.py.

Test out-of-scope question handling

A Q&A system that invents answers for topics outside its documents is worse than one that says "I don't know." This is the hallucination problem that employers test for in AI engineering interviews.

Your generate_answer() function already handles this. The system prompt you built in the previous step tells Gemini to respond with a fallback message when the context is not relevant. Now you can test it with the interactive loop.

  • In app.py, find the prompt string inside generate_answer(). Look for this line:
"I don't have enough information in the provided documents to answer this question."

How does out-of-scope detection work?

The system prompt tells Gemini to only answer based on the provided context. When a user asks about a topic not in the document, the retrieved chunks won't contain relevant information.

Gemini reads those irrelevant chunks, recognizes they don't answer the question, and returns the fallback message instead of making something up. This is prompt engineering at work.

Before you test this, think about what you expect to happen. If you ask a question about a topic that is completely absent from the NextWork Corp handbook, do you think Gemini will still try to answer, or will it use the fallback message?

  • If your app is no longer running, start it again by running:
python app.py
  • At the Your question: prompt, type a question about something completely unrelated to the handbook, for example:
What is the capital of France?

The system still retrieves chunks (it always finds the closest matches), but Gemini recognizes that none of them answer the question. You should see a response along the lines of "I don't have enough information in the provided documents to answer this question."

  • Type quit to exit the loop.

Gemini still answered the out-of-scope question?

  • LLMs are probabilistic, so Gemini might occasionally attempt an answer even when the context is not relevant. Try a more obviously unrelated question like "What year did the Titanic sink?" to confirm the fallback works.
  • Check that the prompt string inside generate_answer() still contains the instruction starting with "If the context does not contain enough information". If it was accidentally deleted, add it back.

Need help? Help me understand why Gemini is not using the fallback message for out-of-scope questions.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py.

ⓧ I'd like to double check the full code

Your app.py is over 30 lines, so scroll through the file from top to bottom and confirm these sections are all present, in order:

  • Imports: os, chromadb, genai, and types.
  • API key loading: reads .env and exits if no key is found.
  • Client initialization: gemini_client and chroma_client.
  • Functions: load_and_chunk(), store_chunks(), retrieve(), and generate_answer() are all unchanged from previous steps.
  • main() function: loads and chunks the document, stores embeddings, then runs the while True interactive loop. The old hardcoded question is gone.
  • Comments inside main() read # Step 1:, # Step 2:, and # Step 3:.

Your RAG Q&A system is complete. It loads a document, chunks it, stores embeddings, retrieves relevant context, and generates grounded answers. It even knows when to say "I don't know." Ready for a challenge? The Secret Mission takes this further with multi-document support and metadata filtering.

Secret mission

Multi-Document Support with Metadata Filtering

Your RAG system handles one document, but real-world systems handle dozens. Can you upgrade yours to load multiple documents and let users filter queries by source?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine and uses the Gemini API free tier, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • rag-qa project folder on your Desktop (contains app.py, .env, data.txt, data2.txt, and the venv virtual environment).
  • Gemini API key stored in Google AI Studio.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend this project.

  • Your code, documents, and virtual environment stay in the rag-qa folder on your Desktop.
  • Your Gemini API key remains active in Google AI Studio for future projects.
  • ChromaDB ran in-memory, so its data is already gone once you closed the script.

Pause - I'll come back to this later

There are no running processes or cloud resources to shut down. Everything is stored locally.

  • Close your terminal and VS Code.
  • Your project files and API key will be right where you left them when you come back.
  • To resume, open the rag-qa folder in VS Code, activate the virtual environment, and run python app.py.

Delete - I don't want to use this again

Remove all project files and revoke your API key to start completely fresh.

  • Close your terminal and VS Code if they are still open.
  • Delete the rag-qa folder from your Desktop by running this command in a new terminal window:

macOS

rm -rf ~/Desktop/rag-qa

Windows

Remove-Item -Recurse -Force "$HOME\Desktop\rag-qa"

Linux

rm -rf ~/Desktop/rag-qa
  • Revoke your Gemini API key so it cannot be used again:
  • Open the Google AI Studio API Keys page.
  • Find the API key you created for this project and delete it from the list.

Nice Work!

Nice Work!

You just built a complete RAG-powered Q&A system from scratch using Python, the Gemini API, and ChromaDB. This is the exact retrieval pipeline pattern that comes up in AI engineering interviews and production systems alike.

You learned how to:

  • Built a complete RAG pipeline from scratch, covering document loading, text chunking, embedding generation, vector retrieval, and grounded answer generation.
  • Used the Gemini API for both text generation with client.models.generate_content and embeddings with client.models.embed_content, storing and searching vector embeddings in ChromaDB.
  • Handled out-of-scope questions to prevent hallucination, a key reliability pattern that employers test for in AI engineering roles.
  • Secret Mission: Extended the system to handle multiple documents with metadata filtering, a production RAG pattern used in enterprise systems.

Ready to quiz yourself?