Build a RAG Q&A System with Claude

Introduction

30 Second Summary

Large language models like Claude are powerful, but they can only answer from their training data. Ask about your company's internal handbook or your personal research notes, and the model will either refuse or confidently make something up.

In this project, you will build a RAG (Retrieval-Augmented Generation) pipeline that connects Claude to your own documents. You'll chunk text, store it in a vector database, and retrieve relevant passages at query time so Claude can generate accurate, grounded answers.

What You'll Build

You'll type a question into your terminal, and within seconds Claude responds with a sourced answer pulled directly from your private documents.

By the end of this project, you'll have:

• A command-line Q&A tool that retrieves relevant passages from your documents and generates grounded answers with citations.
• A side-by-side comparison showing how RAG eliminates the hallucination problem by providing Claude with real context instead of letting it guess.
• A reusable Python codebase you can point at any set of documents (research papers, company policies, personal notes) for instant Q&A.
• Secret Mission: Add a re-ranking step that scores retrieved chunks by relevance before passing them to Claude.

Are there any prerequisites?

You'll need Python 3.10+ installed, an Anthropic API key, and basic familiarity with Python scripting and APIs. The project walks you through everything else from scratch.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Environment

A RAG pipeline depends on three distinct capabilities: generating embeddings from text, storing and searching those embeddings, and calling an LLM to generate answers.

Each capability comes from a dedicated library. In this step, you will create an isolated project environment and install all three so your pipeline has everything it needs.

In this step, get ready to:

• Create a project directory and virtual environment.
• Install the three core libraries (sentence-transformers, ChromaDB, and Anthropic SDK).
• Set your API key and verify everything works.

Create your project directory and virtual environment

A virtual environment keeps your project's packages separate from the rest of your system. This prevents version conflicts between projects.

• Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Terminal (macOS) or cmd (Windows) and press Enter to open it.
• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir rag-qa-system
cd rag-qa-system

• Create a virtual environment inside your project folder by running:

python -m venv venv

• Confirm the venv folder was created by running:

ls

You should see a venv folder listed in the output.

• Activate the virtual environment so packages install into it instead of your system Python:

macOS

• Run this activation command:

source venv/bin/activate

Windows

• Run this activation command:

venv\Scripts\activate

Your terminal prompt now shows (venv) at the beginning. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

Make sure you ran the activation command from inside the rag-qa-system folder. Run ls and confirm you see a venv directory listed.

On Windows PowerShell, you may need to run .\venv\Scripts\Activate.ps1 instead. If you see an execution policy error, run Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass first, then try the activation again.

Help me activate my Python virtual environment.

Install the three core libraries

Your RAG pipeline needs three libraries: one to generate embeddings, one to store and search them, and one to call Claude for answer generation.

• Install all three libraries by running this command:

pip install sentence-transformers==5.6.1 chromadb==1.5.9 anthropic==0.120.2

What does this install?

• sentence-transformers runs embedding models locally on your machine. It converts text into numerical vectors that capture meaning.
• chromadb is a vector database that stores those embeddings and lets you search them by similarity.
• anthropic is the official Python SDK for calling Claude's API to generate answers.

The install takes a few minutes because sentence-transformers downloads PyTorch-related dependencies. You will see a stream of package names scrolling past.

When it finishes, you should see a line ending with Successfully installed followed by a list of package names.

Seeing a permissions error or install failure?

Make sure your virtual environment is active. Your prompt should show (venv) at the beginning. If it does not, re-run the activation command from the previous substep.

If you see a network timeout, try running the pip install command again. Temporary network issues can interrupt large downloads.

Help me fix my pip install error.

Set up your API key and verify

Claude needs your API key to authenticate requests. You will set it as an environment variable so the SDK can find it automatically.

• Set your ANTHROPIC_API_KEY environment variable using the command for your platform:

macOS

• Run this command, replacing the placeholder with your actual API key:

export ANTHROPIC_API_KEY=your-api-key-here

Windows

• Run this command, replacing the placeholder with your actual API key:

set ANTHROPIC_API_KEY=your-api-key-here

Where do I find my API key?

You can find or create your API key at the Anthropic console (console.anthropic.com). Copy the key value and paste it in place of your-api-key-here in the command above.

Before you run the next command, what do you expect to see if all three libraries installed correctly?

• Verify that all libraries are installed and your API key is set by running:

python -c "import sentence_transformers; import chromadb; import anthropic; print('All libraries ready!')"

You should see All libraries ready! printed in your terminal. This confirms all three packages imported successfully.

Seeing a ModuleNotFoundError?

This means one of the packages did not install into your active environment. Check that (venv) appears in your prompt. If it does not, activate the virtual environment and run the pip install command again.

If the error names a specific module, reinstall just that package. For example: pip install chromadb==1.5.9 to reinstall chromadb alone.

Help me fix my ModuleNotFoundError.

Your environment is locked and loaded. Next up, you will ask Claude a question without any context and witness the hallucination problem firsthand.

Ask Claude Without Context

Your environment is ready with all the libraries installed. Now it's time to see the problem that the rest of this project solves.

Claude is a powerful language model, but it cannot access your private documents. When you ask it a specific factual question about content it has never seen, it will either admit ignorance or confidently make up an answer. You need to witness this firsthand before building the retrieval system that fixes it.

In this step, get ready to:

• Create sample documents with specific, verifiable facts that Claude has never seen.
• Ask Claude a factual question about those documents without providing any context.
• Observe the hallucinated response and understand the problem RAG solves.

Create sample documents

You need documents containing specific facts that Claude could not possibly know. A fictional company handbook with exact dollar amounts, dates, and policies is perfect for this. If Claude "answers" correctly about a company that doesn't exist, you'll know it's hallucinating.

• Create a documents folder inside your rag-qa-system directory by running this command:

mkdir documents

• Confirm the folder was created by running:

ls

You should see documents listed in the output.

• Create a new file called company_handbook.txt inside the documents folder in your code editor.
• Paste in the following content:

Nexora Technologies Employee Handbook - Last Updated January 2026

Remote Work Policy:
All employees are eligible for remote work up to 3 days per week. Fully remote arrangements require VP-level approval and a minimum of 18 months tenure. Remote workers receive a one-time $1,500 home office stipend upon approval.

Time Off Policy:
New employees receive 15 days of PTO in their first year, increasing to 20 days after 2 years and 25 days after 5 years. Unused PTO carries over up to a maximum of 5 days per year. PTO requests require 2 weeks advance notice for absences longer than 3 consecutive days.

Performance Reviews:
Performance reviews are conducted quarterly using the IMPACT framework (Innovation, Mentorship, Performance, Accountability, Communication, Teamwork). Ratings range from 1-5, with 4+ qualifying for promotion consideration. Annual bonuses are calculated as base salary multiplied by rating percentage (e.g., rating 4 = 4% bonus).

Engineering On-Call:
On-call rotations are 1 week per engineer per quarter. On-call engineers receive $500 per rotation plus $150 per incident responded to outside business hours. Escalation threshold is 15 minutes for P1 incidents and 1 hour for P2 incidents.

Why these specific details?

This handbook contains exact figures ($500 per rotation, 15-minute P1 threshold, 18-month tenure requirement) for a completely fictional company called Nexora Technologies. Claude has never seen these documents, so any "correct" answer it gives will be fabricated.

• Create a second file called product_roadmap.txt inside the documents folder.
• Paste in the following content:

Nexora Technologies Product Roadmap Q1-Q2 2026

Project Aurora (Q1 2026):
Launch the new real-time analytics dashboard powered by Apache Flink. Target: reduce data latency from 15 minutes to under 30 seconds. Budget: $2.3M. Team size: 12 engineers. Key dependency: completion of the Kafka migration by February 15.

Project Beacon (Q1-Q2 2026):
Migrate authentication system from custom OAuth implementation to Auth0. Estimated 6-week implementation with 2-week parallel running period. Rollback plan: maintain legacy system for 90 days post-migration. Risk: 3rd party dependency on Auth0 SLA (99.99% uptime guaranteed).

Project Cascade (Q2 2026):
Implement AI-powered code review assistant for internal engineering teams. Phase 1: static analysis integration. Phase 2: LLM-based suggestions using Claude API. Success metric: reduce average PR review time from 4.2 hours to under 2 hours. Budget: $890K including API costs.

Infrastructure Goals:
- Achieve 99.95% uptime SLA (current: 99.91%)
- Reduce cloud spend by 15% through reserved instance optimization
- Complete SOC 2 Type II certification by end of Q2

• Save both files.
• Confirm both documents exist by running:

ls documents/

You should see company_handbook.txt and product_roadmap.txt listed.

Don't see both files?

Make sure you created the files inside the documents folder, not in the project root. Check that the filenames match exactly (underscores, not hyphens).

Help me figure out why my document files aren't showing up.

Ask Claude a question without document context

Now you'll ask Claude a detailed factual question about Nexora's on-call policy. Claude has never seen these documents, so it has no way to answer accurately.

• Create a new file called naive_query.py in the root of your rag-qa-system folder (not inside documents/).
• Paste in the following code:

import os
from anthropic import Anthropic

client = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))

question = "What is Nexora's on-call compensation policy and what are the escalation thresholds?"

print("=" * 60)
print("NAIVE QUERY (No Context Provided)")
print("=" * 60)
print(f"\nQuestion: {question}\n")

message = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": question,
        }
    ],
)

naive_response = message.content[0].text
print(f"Claude's Response:\n{naive_response}")
print("\n" + "=" * 60)
print("NOTE: Claude has never seen Nexora's documents.")
print("Any specific details above are likely hallucinated.")
print("=" * 60)

What does this code do?

• It creates an Anthropic client using the API key from your environment variable.
• It asks Claude a very specific question about Nexora's on-call compensation (exact dollar amounts and time thresholds) without providing any document context.
• It prints Claude's response alongside a reminder that any specific details are likely hallucinated.

• Save naive_query.py.

✔️ Awesome, I've got everything!

Great. Make sure you've saved naive_query.py before running it.

ⓧ I'd like to double check the full code

Here is the complete naive_query.py file:

import os
from anthropic import Anthropic

client = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))

question = "What is Nexora's on-call compensation policy and what are the escalation thresholds?"

print("=" * 60)
print("NAIVE QUERY (No Context Provided)")
print("=" * 60)
print(f"\nQuestion: {question}\n")

message = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": question,
        }
    ],
)

naive_response = message.content[0].text
print(f"Claude's Response:\n{naive_response}")
print("\n" + "=" * 60)
print("NOTE: Claude has never seen Nexora's documents.")
print("Any specific details above are likely hallucinated.")
print("=" * 60)

Run the naive query and observe the result

Before you run this script, pause and think. What do you expect Claude to say about Nexora's on-call compensation? Does Claude have any way to know the exact dollar amounts or escalation thresholds?

• Run the script by executing this command:

python naive_query.py

Watch Claude's response carefully. It will either say it doesn't know about Nexora, or it will confidently produce specific dollar amounts, time thresholds, and policy details that are completely fabricated.

What should I see?

Claude might invent numbers like "$200 per rotation" or "30-minute escalation window." These are wrong. The real values in your document are $500 per rotation and 15 minutes for P1 incidents.

This is the hallucination problem. Claude sounds confident and fluent, but without access to the actual documents, it has no grounding. This is exactly what RAG fixes.

Take note of this response. You will compare it directly against the RAG-augmented answer you build in a later step.

Seeing an authentication error?

If you see an error about an invalid API key, make sure your ANTHROPIC_API_KEY environment variable is still set in your current terminal session. Environment variables reset when you close the terminal.

• Re-set it by running:

set ANTHROPIC_API_KEY=your-api-key-here

Help me debug my Anthropic API authentication error.

You have now seen the core problem with asking an LLM about private documents. It cannot access information it was never trained on. In the next step, you'll build the retrieval system that solves this by feeding Claude the relevant context before it answers.

Chunk, Embed, and Store Your Documents

In the previous step, you saw Claude confidently answer questions about Nexora documents it has never seen. The response was either hallucinated or vague.

To fix this, you need to give Claude the right context at query time. That means splitting your documents into searchable pieces, converting them to numerical representations (embeddings), and storing them in a vector database where you can retrieve the most relevant ones later.

In this step, get ready to:

• Write a chunking function that splits documents into overlapping pieces.
• Embed all chunks using a local sentence-transformers model.
• Store the embedded chunks in a ChromaDB collection for later retrieval.

Write the chunking logic

An embedding model works best on small, focused pieces of text. If you feed it an entire document at once, the meaning gets diluted across too many topics.

Chunking splits your documents into shorter passages. Overlapping the chunks ensures that context at boundaries is not lost.

• Create a new file called ingest.py in your rag-qa-system folder and add the imports and chunking function:

import os
from sentence_transformers import SentenceTransformer
import chromadb


def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start = end - overlap
    return chunks

What does this code do?

The imports bring in the three libraries you installed: os for file operations, SentenceTransformer for computing embeddings, and chromadb for vector storage.

The chunk_text function slides a 500-character window across the text, stepping forward by 450 characters each time (500 minus the 50-character overlap). This overlap means each chunk shares its edges with its neighbours, so no sentence gets cut off without context.

• Save ingest.py.
• Test that the chunking function works by running this command:

python -c "from ingest import chunk_text; chunks = chunk_text('A' * 1200); print(f'{len(chunks)} chunks created from 1200 characters')"

You should see 3 chunks created from 1200 characters printed. The text is 1200 characters long, so with a window of 500 and a step of 450, you get 3 overlapping pieces.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is still active. You should see (venv) at the start of your terminal prompt. If not, activate it again with venv\Scripts\activate.

Also confirm that ingest.py is saved in the same folder where you are running the command.

Help me debug a ModuleNotFoundError when importing from ingest.py

Embed chunks and store in ChromaDB

Now that you have a working chunker, you need a function that reads your documents, chunks them, embeds them into vectors, and stores everything in ChromaDB for fast similarity search.

• In ingest.py, add the ingest_documents function below the chunk_text function. This first part loads the model, sets up the database, and reads your documents:

def ingest_documents(documents_dir="documents", collection_name="nexora_docs"):
    """Read documents, chunk them, embed them, and store in ChromaDB."""
    model = SentenceTransformer("all-MiniLM-L6-v2")

    client = chromadb.PersistentClient(path="./chroma_data")

    existing_collections = [c.name for c in client.list_collections()]
    if collection_name in existing_collections:
        client.delete_collection(collection_name)

    collection = client.create_collection(name=collection_name)

    all_chunks = []
    all_ids = []
    all_metadatas = []

    for filename in os.listdir(documents_dir):
        if not filename.endswith(".txt"):
            continue

        filepath = os.path.join(documents_dir, filename)
        with open(filepath, "r", encoding="utf-8") as f:
            text = f.read()

        chunks = chunk_text(text)
        for i, chunk in enumerate(chunks):
            chunk_id = f"{filename}_{i}"
            all_chunks.append(chunk)
            all_ids.append(chunk_id)
            all_metadatas.append({"source": filename, "chunk_index": i})

What does this code do?

This function does four things before any embedding happens:

• Loads the all-MiniLM-L6-v2 model locally. This is a free, 80MB embedding model that produces 384-dimensional vectors.
• Creates a PersistentClient so your vector database is saved to disk (not just memory). Data survives between script runs.
• Deletes any existing collection with the same name so you get a fresh start each time you re-ingest.
• Loops through every .txt file, chunks it, and tracks metadata (source filename and chunk index) so you know where each piece came from.

• Still in ingest.py, add the embedding and storage logic to complete the function. Paste this directly below the last line you just added (continuing inside ingest_documents):

    embeddings = model.encode(all_chunks).tolist()

    collection.add(
        ids=all_ids,
        documents=all_chunks,
        embeddings=embeddings,
        metadatas=all_metadatas,
    )

    print(f"Ingested {len(all_chunks)} chunks into ChromaDB from {documents_dir}/")
    print(f"Collection: {collection_name}")
    print(f"Files processed: {[f for f in os.listdir(documents_dir) if f.endswith('.txt')]}")


if __name__ == "__main__":
    ingest_documents()

What does this code do?

The model.encode() call converts all your text chunks into numerical vectors in one batch. Each chunk becomes a list of 384 numbers that represent its meaning.

The collection.add() call stores those vectors in ChromaDB alongside the original text and metadata. Later, you can search by meaning instead of exact keywords.

The if __name__ == "__main__" block makes the script runnable from the command line.

• Save ingest.py.

Before you run this, what do you think the output will tell you? How many chunks do you expect from two short documents?

• Run the ingestion script:

python ingest.py

The first run downloads the all-MiniLM-L6-v2 model (~80MB). After that, you should see output confirming the number of chunks ingested, something like:

Ingested 12 chunks into ChromaDB from documents/
Collection: nexora_docs
Files processed: ['company_handbook.txt', 'product_roadmap.txt']

Why is the chunk count around 12?

Each document is roughly 1000-1200 characters. With a 500-character chunk size and 450-character step, each document produces about 5-7 chunks. The exact number depends on file length. As long as you see a non-zero count and both files listed, your pipeline is working.

Seeing a RuntimeError or download failure?

The model download requires an internet connection. If you see a connection error, check your network and try again.

If you see a FileNotFoundError for the documents folder, make sure you are running the command from inside your rag-qa-system directory where the documents/ folder exists.

Help me debug an error when running ingest.py

• Confirm that the chroma_data folder was created by running:

ls

You should see chroma_data listed alongside your other files. This folder holds your persistent vector database on disk.

✔️ Awesome, I've got everything!

Great. Double check you have saved ingest.py and you can see the chroma_data folder in your project directory.

ⓧ I'd like to double check the full code

Your ingest.py file should contain these sections in order. Scroll up and confirm each is present:

• The three imports (os, SentenceTransformer, chromadb).
• The chunk_text function with chunk_size=500 and overlap=50.
• The ingest_documents function that loads the model, sets up the client, loops through files, then embeds and stores.
• The if __name__ == "__main__" block at the bottom calling ingest_documents().

Your documents are now embedded and searchable. Next up, you will build the retrieval and generation pipeline that queries this vector store and feeds the results to Claude for grounded, accurate answers.

Retrieve and Generate Grounded Answers

In the previous step, you chunked your documents, embedded them, and stored everything in ChromaDB. Your vector database is loaded and ready to search.

Now you connect retrieval to generation. The same question that produced hallucinations in naive_query.py will now get a grounded, cited answer because Claude receives the relevant context before responding.

In this step, get ready to:

• Build a retrieval function that embeds your query and searches ChromaDB for the most similar chunks.
• Build a generation function that sends retrieved context to Claude with a grounding prompt.
• Run the full RAG pipeline and compare the result against your earlier naive response.

Build the retrieval function

Retrieval is the "R" in RAG. You take the user's question, convert it to an embedding using the same model that embedded your documents, and ask ChromaDB to return the chunks with the highest similarity.

• Create a new file called rag_query.py in your rag-qa-system folder by adding this code:

import os
import sys
from sentence_transformers import SentenceTransformer
import chromadb
from anthropic import Anthropic


def retrieve(query, collection, model, n_results=3):
    """Embed the query and retrieve the most similar chunks."""
    query_embedding = model.encode([query]).tolist()

    results = collection.query(
        query_embeddings=query_embedding,
        n_results=n_results,
    )

    return results

What does this code do?

• The imports bring in all the tools you need: SentenceTransformer for embeddings, chromadb for vector search, and Anthropic for calling Claude.
• model.encode([query]).tolist() converts the user's question into the same 384-dimension embedding space as your stored chunks.
• collection.query() searches ChromaDB and returns the top n_results most similar chunks, along with their metadata and distances.

• Save rag_query.py.
• Verify the imports work by running:

python -c "from rag_query import retrieve; print('Retrieve function loaded')"

You should see Retrieve function loaded printed with no errors.

Seeing an import error?

• Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.
• Check that the file is saved as rag_query.py (with an underscore, not a hyphen) in the rag-qa-system folder.

Why is my retrieve function failing to import?

Build the generation function with prompt engineering

The generation function is where RAG becomes powerful. You assemble the retrieved chunks into a context block, then instruct Claude to answer only from that context. This grounding prevents hallucination.

• In rag_query.py, add the first part of the generate_answer function below retrieve(). This assembles the prompt:

def generate_answer(query, retrieved_chunks, retrieved_sources):
    """Send the query + retrieved context to Claude for grounded generation."""
    client = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))

    context_block = ""
    for i, (chunk, source) in enumerate(zip(retrieved_chunks, retrieved_sources)):
        context_block += f"\n[Source {i+1}: {source}]\n{chunk}\n"

    system_prompt = (
        "You are a helpful assistant that answers questions based ONLY on the "
        "provided context. If the context does not contain enough information to "
        "answer the question, say 'I don't have enough information in the provided "
        "documents to answer this question.' Always cite which source your information "
        "comes from."
    )

    user_message = f"""Context from documents:
{context_block}

Question: {query}

Answer based only on the context above:"""

What does this prompt do?

The system_prompt constrains Claude to answer only from the provided context. If the retrieved chunks do not contain the answer, Claude will say so instead of making something up.

The user_message places the retrieved chunks first (labeled by source file), then the question. This structure gives Claude the evidence before the query, mimicking how a researcher reads source material before answering.

• Now complete the function by adding the API call directly below the user_message variable (still inside generate_answer):

    message = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=1024,
        system=system_prompt,
        messages=[
            {
                "role": "user",
                "content": user_message,
            }
        ],
    )

    return message.content[0].text

What does this API call do?

• claude-haiku-4-5 is the most cost-effective Claude model at $1/MTok input and $5/MTok output. Perfect for a RAG system with small context windows.
• The system parameter passes the grounding instruction separately from the conversation messages.
• message.content[0].text extracts Claude's text response from the API response object.

• Save rag_query.py.
• Confirm both functions load correctly by running:

python -c "from rag_query import retrieve, generate_answer; print('Both functions ready')"

You should see Both functions ready printed.

Seeing a syntax error?

• Check that the user_message f-string uses triple quotes f"""...""" and that the closing """ is on the same line as the last text.
• Make sure the message = client.messages.create(...) block is indented at the same level as user_message (inside the function body).

Help me fix a syntax error in my generate_answer function.

Run the full pipeline and compare results

Now you wire everything together with a main() function that loads the model, retrieves relevant chunks, displays them for transparency, and calls generate_answer to produce the final response.

• Add the first part of the main() function below generate_answer() in rag_query.py. This handles query setup, retrieval, and chunk display:

def main():
    if len(sys.argv) < 2:
        query = "What is Nexora's on-call compensation policy and what are the escalation thresholds?"
    else:
        query = " ".join(sys.argv[1:])

    model = SentenceTransformer("all-MiniLM-L6-v2")
    client = chromadb.PersistentClient(path="./chroma_data")
    collection = client.get_collection(name="nexora_docs")

    print("=" * 60)
    print("RAG QUERY (With Retrieved Context)")
    print("=" * 60)
    print(f"\nQuestion: {query}\n")

    results = retrieve(query, collection, model, n_results=3)

    retrieved_chunks = results["documents"][0]
    retrieved_sources = [m["source"] for m in results["metadatas"][0]]
    distances = results["distances"][0]

    print("-" * 40)
    print("Retrieved Chunks:")
    print("-" * 40)
    for i, (chunk, source, distance) in enumerate(
        zip(retrieved_chunks, retrieved_sources, distances)
    ):
        print(f"\n[Chunk {i+1}] Source: {source} | Distance: {distance:.4f}")
        print(f"{chunk[:200]}...")

What does this main function do so far?

• It accepts a question from the command line, or defaults to the on-call question if none is provided.
• It loads the same all-MiniLM-L6-v2 model used during ingestion. Using the same model for embedding queries and documents is essential for meaningful similarity scores.
• It prints the retrieved chunks with their source file and distance score so you can see exactly what context Claude will receive.

• Complete the script by adding the generation call and entry point below the chunk display loop:

    print("\n" + "-" * 40)
    print("Claude's RAG-Augmented Response:")
    print("-" * 40)

    answer = generate_answer(query, retrieved_chunks, retrieved_sources)
    print(f"\n{answer}")


if __name__ == "__main__":
    main()

• Save rag_query.py.

Before you run this, think about what you expect to see. Will Claude give you the same vague or hallucinated answer as before, or something different?

• Run the RAG pipeline:

python rag_query.py

You should see the retrieved chunks printed first (with source filenames and distance scores), followed by Claude's response. The response now cites specific numbers from your documents: $500 per rotation, $150 per incident, 15 minutes for P1, and 1 hour for P2.

Compare this to the output you saw from naive_query.py. The naive version either admitted ignorance or fabricated details. The RAG version answers with exact figures pulled from your actual documents.

Why is this so much better?

Without context, Claude has no information about Nexora. It can only guess. With RAG, you fed Claude the exact handbook section containing on-call details. Claude reads the evidence, then answers. The grounding prompt prevents it from adding anything beyond what the context provides.

This is the core pattern behind every enterprise AI assistant, customer support bot, and document Q&A tool in production today.

Getting an error about the collection not found?

• Make sure you ran python ingest.py in the previous step. The chroma_data folder must exist with your embedded documents.
• Confirm you are running rag_query.py from inside the rag-qa-system folder where chroma_data lives.
• Check that your ANTHROPIC_API_KEY environment variable is still set in this terminal session.

Help me debug my RAG query script.

✔️ Awesome, I've got everything!

Great. Double check you have saved rag_query.py and that your terminal shows the grounded response with citations.

ⓧ I'd like to double check the full code

Your rag_query.py file should contain these sections in order. Scroll up and confirm each is present:

• Imports (os, sys, SentenceTransformer, chromadb, Anthropic).
• retrieve() function that encodes the query and calls collection.query().
• generate_answer() function that builds context_block, system_prompt, and user_message, then calls client.messages.create().
• main() function that parses the query, loads the model and collection, retrieves chunks, prints them, and calls generate_answer().
• if __name__ == "__main__": main() entry point at the bottom.

You have a working RAG pipeline. The same question that tripped up Claude in the naive query now produces an accurate, cited answer grounded in your actual documents. Next up, you will take this further with a Secret Mission that improves retrieval precision.

Secret mission

Add Re-Ranking to Your RAG Pipeline

Your RAG pipeline retrieves chunks by vector similarity, but similarity is not the same as relevance. A chunk can be semantically close to the query without actually answering it. In this mission, you'll add a cross-encoder re-ranking step that scores each retrieved chunk directly against the query, dramatically improving precision.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally except for a handful of Claude API calls that have already been consumed. There are no ongoing costs.

Resources you used:

• The rag-qa-system project folder containing all Python scripts, sample documents, and virtual environment.
• ChromaDB local database stored in ./chroma_data inside your project folder.
• A few Claude Haiku 4.5 API calls (~$0.01-0.05 total, already consumed).

Keep everything running

No action needed. Choose this if you want to keep experimenting with your RAG system or swap in new documents.

• Your RAG system is entirely local. There are no ongoing costs or background processes.
• To use this system for a different domain, replace the text files in documents/ and re-run python ingest.py to rebuild the vector store.
• The only cost you'll incur is from future Claude API calls when you run rag_query.py or naive_query.py.

Pause - I'll come back to this later

Deactivate your virtual environment to free up your terminal session. Your files and vector database stay on disk for when you return.

• Deactivate your virtual environment by running:

deactivate

• When you come back, navigate to your rag-qa-system folder and reactivate the virtual environment with venv\Scripts\activate.
• Remember to set your ANTHROPIC_API_KEY environment variable again since it was session-only.

Delete - I don't want to use this again

Remove all project files, the virtual environment, and the local vector database. This cannot be undone.

• Deactivate your virtual environment first if it is still active:

deactivate

• Navigate up to your Desktop:

cd ~/Desktop

• Delete the entire project folder. On macOS/Linux:

rm -rf rag-qa-system

• On Windows:

rmdir /s /q rag-qa-system

• This removes all Python scripts, sample documents, the venv virtual environment, and the chroma_data vector database.
• The Claude API calls you made are already consumed. There is nothing else to clean up on Anthropic's side.

Nice Work!

Nice Work!

Nice work! You just built a complete RAG pipeline from scratch that retrieves relevant context from private documents and generates accurate, grounded answers.

You've learned how to:

• Build the full embedding-to-generation loop: chunking documents, computing embeddings with sentence-transformers, storing them in ChromaDB, retrieving by similarity, and generating answers with Claude Haiku 4.5.
• Experience the hallucination problem firsthand by asking Claude about documents it has never seen, then solve it with retrieval-augmented generation that grounds answers in real context.
• Engineer prompts that instruct Claude to answer ONLY from provided context and cite its sources, turning a general-purpose LLM into a trustworthy document Q&A system.
• Secret Mission: Implemented cross-encoder re-ranking to score retrieved chunks against the query and improve retrieval precision before passing context to Claude.

Ready to quiz yourself?