Build a RAG Pipeline from Scratch

Build a RAG pipeline that answers questions from your own docs with citations.

Introduction

30 Second Summary

When you ask an AI chatbot a question it doesn't know the answer to, it doesn't say "I don't know." It fabricates a confident, detailed answer instead.

In this project, you will build a Retrieval-Augmented Generation (RAG) pipeline from scratch in Python. Your pipeline will answer questions about your own documents with grounded, source-cited responses.

What You'll Build

Picture typing a question about a company's remote work policy into your terminal and watching your script pull the exact answer from the actual employee handbook, complete with source citations.

By the end of this project, you'll have:

  • First-hand experience with the hallucination problem by watching Claude confidently fabricate policies for a company that does not exist.
  • A working document pipeline that chunks text, stores it as searchable vector embeddings in ChromaDB, and retrieves the most relevant pieces ranked by similarity.
  • A complete RAG pipeline that produces accurate, cited answers from your documents and honestly refuses when the answer is not in the data.
  • Secret Mission: Add source filtering so queries only search specific documents.

Are there any prerequisites?

This project assumes basic Python familiarity with variables, functions, and loops. No prior AI or API experience is required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Python Environment

Your first task is building the environment where your RAG pipeline will run. There is a catch: ChromaDB, the vector database you will use to store and search document chunks, has known compatibility issues with Python 3.14.

To avoid crashes, you will install Python 3.13 in its own virtual environment and add the two Python packages that power your pipeline. By the end of this step, you will have a working connection to Claude ready to go.

In this step, get ready to:
  • Install Python 3.13 and create a virtual environment for your project.
  • Install the chromadb and anthropic packages.
  • Set your API key and verify Claude connectivity.
Install Python 3.13 and create your project folder

Python 3.14 is installed on your machine, but ChromaDB's native components can crash on that version. Python 3.13 avoids these issues entirely.

  • Press the Windows key to open your search bar.
  • Type Command Prompt and press Enter to open it.
  • Check whether Python 3.13 is already installed by running this command:
py -3.13 --version

✔️ I see Python 3.13.x

Python 3.13 is already installed. You are ready to create your project folder.

ⓧ I see an error about Python 3.13 not being installed

The Python Launcher found no Python 3.13 installation. You need to install it alongside your existing Python 3.14.

  • Open python.org/downloads in your browser.
  • Scroll down to find the latest Python 3.13 release and click it.
  • Download the Windows installer (64-bit) and run it.
  • On the installer's first screen, check Add python.exe to PATH, then click Install Now.
  • After installation finishes, close and reopen Command Prompt.
  • Confirm by running:
py -3.13 --version

You should now see a version number starting with 3.13.

ⓧ py command is not recognized

The Python Launcher is not available on your system. Installing Python 3.13 from the official installer will add it.

  • Open python.org/downloads in your browser.
  • Scroll down to find the latest Python 3.13 release and click it.
  • Download the Windows installer (64-bit) and run it.
  • On the installer's first screen, check Add python.exe to PATH, then click Install Now.
  • After installation finishes, close and reopen Command Prompt.
  • Confirm by running:
py -3.13 --version

You should now see a version number starting with 3.13.

  • Navigate to your Desktop and create the project folder by running these commands:
cd %USERPROFILE%\Desktop
mkdir rag-from-scratch
cd rag-from-scratch
  • Create a Python 3.13 virtual environment and activate it by running these commands:
py -3.13 -m venv .venv
.venv\Scripts\activate

What is a virtual environment?

A virtual environment is an isolated Python installation tied to a single project. Packages you install inside it stay separate from your system Python.

You will see (.venv) appear at the start of your terminal prompt. That confirms the virtual environment is active.

  • Confirm the virtual environment is using Python 3.13 by running:
python --version

You should see a version starting with Python 3.13. This confirms your virtual environment is using the right Python, not your system's 3.14.

Still seeing Python 3.14?

If python --version still shows 3.14, your virtual environment may not be active. Check that (.venv) appears in your prompt. If it does not, run .venv\Scripts\activate again.

If the activate command itself fails, make sure you ran py -3.13 -m venv .venv from inside the rag-from-scratch folder.

Still stuck? Help me troubleshoot my Python virtual environment on Windows.

Install chromadb and anthropic
  • Install the two packages your project needs by running:
pip install chromadb==1.5.9 anthropic==1.0.0

What are these packages?

chromadb is an open-source vector database. It stores text as numerical vectors and finds the most similar ones when you search. It runs entirely on your machine with no account or cloud service needed.

anthropic is the official Python SDK for the Anthropic API. It lets your code send prompts to Claude and receive responses.

The installation may take a minute. ChromaDB pulls in several dependencies including its built-in embedding model.

  • Verify both packages installed correctly by running:
python -c "import chromadb; print(chromadb.__version__)"
python -c "import anthropic; print(anthropic.__version__)"

The first command should print 1.5.9 and the second should print 1.0.0.

Seeing an import error?

Make sure (.venv) still appears in your prompt. If you accidentally closed and reopened the terminal, you need to navigate back to rag-from-scratch and run .venv\Scripts\activate before running pip install again.

Need more help? Help me fix a package import error in my Python virtual environment.

Set your API key and test Claude

The Anthropic SDK reads your API key from an environment variable called ANTHROPIC_API_KEY. Setting it here means your Python scripts can connect to Claude without hardcoding the key in your code.

  • Set the environment variable by running this command, replacing your-api-key-here with your actual Anthropic API key:
set ANTHROPIC_API_KEY=your-api-key-here

Why an environment variable?

Storing your key in an environment variable keeps it out of your source code. If you ever share or publish your scripts, the key stays private on your machine.

This set command only lasts for the current terminal session. If you close and reopen Command Prompt, you will need to run it again.

  • Open the project in VS Code by running:
code .

VS Code opens with the rag-from-scratch folder loaded. If VS Code asks whether you trust the authors of the files in this folder, click Yes, I trust the authors.

  • In the Explorer sidebar on the left, hover over the RAG-FROM-SCRATCH heading and click the New File icon (the page with a plus sign).
  • Name the file test_api.py and press Enter.
  • Paste the following code into test_api.py:
from anthropic import Anthropic

# Create a client (reads ANTHROPIC_API_KEY from your environment)
client = Anthropic()

# Send a test message to Claude
message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Say hello in one sentence."}
    ],
)

# Print Claude's response
for block in message.content:
    if block.type == "text":
        print(block.text)

What does this code do?

Anthropic() creates a client that automatically reads your ANTHROPIC_API_KEY environment variable. No need to paste the key into your code.

client.messages.create() sends a prompt to the claude-sonnet-5 model and returns Claude's response. The for loop at the bottom extracts and prints the text from the response.

  • Save test_api.py by pressing Ctrl+S.
  • Switch back to your Command Prompt window (the one with (.venv) in the prompt).
  • Run the test script:
python test_api.py

You should see Claude reply with a short greeting. The exact wording will vary, but any response confirms your API key is working and your code can reach Claude.

Seeing an authentication error?

Double-check that your set ANTHROPIC_API_KEY=... command used your actual API key with no extra spaces or quotes around it. The key should start with sk-ant-.

If you closed and reopened Command Prompt since setting the key, you need to run the set command again. The variable only lasts for the current session.

Still stuck? Help me troubleshoot an Anthropic API authentication error on Windows.

✔️ Awesome, I've got everything!

Great. Double check you have saved test_api.py and move on to the final verification below.

ⓧ I'd like to double check the full code

Here is the complete test_api.py file. Compare it with yours line by line:

from anthropic import Anthropic

# Create a client (reads ANTHROPIC_API_KEY from your environment)
client = Anthropic()

# Send a test message to Claude
message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Say hello in one sentence."}
    ],
)

# Print Claude's response
for block in message.content:
    if block.type == "text":
        print(block.text)

Before you run this final check, what do you expect to see if both packages and your API key are all working correctly?

  • Run this command to confirm everything is ready:
python -c "import chromadb; import anthropic; print('Ready!')"

You should see Ready! printed. Both packages import without errors, which means your environment is fully set up.

Your environment is locked and loaded. Next up, you will see exactly why all this setup matters by watching Claude hallucinate an answer to a question it cannot possibly know.

See the Hallucination Problem

Your Python environment is set up and both packages are ready to go. Now it's time to see the problem that makes this entire project necessary.

LLMs like Claude will confidently fabricate plausible-sounding answers when they don't have real information. You're about to see this happen firsthand.

In this step, get ready to:
  • Ask Claude about a fictional company with no additional context.
  • See Claude's hallucinated response firsthand.
Ask Claude about a company that doesn't exist

Moonstone Technologies is a company that does not exist. Claude has never been trained on information about it, so it has zero factual knowledge to draw from.

  • In VS Code, create a new file called no_rag.py inside your rag-from-scratch folder.
  • Paste the following code into the file:
import os
from anthropic import Anthropic

# Create an Anthropic client (reads ANTHROPIC_API_KEY from environment)
client = Anthropic()

# Ask Claude about a fictional company with no context provided
message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "What is Moonstone Technologies' remote work policy?",
        }
    ],
)

# Print Claude's response
for block in message.content:
    if block.type == "text":
        print(block.text)

What does this code do?

  • Anthropic() creates a client that automatically reads your ANTHROPIC_API_KEY environment variable.
  • client.messages.create() sends a single question to claude-sonnet-5 with no additional context. Claude only has its training data to work with.
  • The for loop at the bottom iterates through the response content blocks and prints any text blocks.
  • Save no_rag.py.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

Here's the complete no_rag.py file. Compare it with yours line by line.

import os
from anthropic import Anthropic

client = Anthropic()

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "What is Moonstone Technologies' remote work policy?",
        }
    ],
)

for block in message.content:
    if block.type == "text":
        print(block.text)
Run the script and observe the hallucination

Before you run this script, think about what you expect Claude to say. Moonstone Technologies doesn't exist. Do you think Claude will admit it doesn't know, or do something else?

  • Run the script by entering this command in your terminal:
python no_rag.py

Read Claude's response carefully. It will invent specific details like the number of remote days allowed, approval processes, and core hours. Every single detail is fabricated. Moonstone Technologies does not exist, so none of this is real.

What is hallucination?

Hallucination is when an LLM generates confident, plausible-sounding information that is completely made up. Claude didn't warn you that it had no data. It wrote a polished answer full of invented facts.

This is the core problem Retrieval-Augmented Generation solves. Instead of letting the model guess, RAG feeds it real documents so every answer is grounded in actual data. Save this output mentally as your "before" baseline.

Seeing an authentication error?

  • Make sure your ANTHROPIC_API_KEY environment variable is still set. If you opened a new terminal window, you need to reactivate your virtual environment and set the variable again.
  • Confirm your virtual environment is active. You should see (.venv) at the start of your terminal prompt.
  • Check that your API key is valid and has not expired on the Anthropic console.

Why is my script failing with an authentication error?

You've just witnessed the hallucination problem firsthand. Claude sounded authoritative, but every detail was fabricated. Next up, you'll build a knowledge base of real documents and teach Claude to use them instead of guessing.

Build Your Knowledge Base

You saw Claude invent a detailed remote work policy for a company that does not exist. The problem is clear: Claude has no access to your actual information.

To fix this, you need to give Claude real documents to reference. In this step, you'll create a local knowledge base by breaking documents into bite-sized chunks and storing them as searchable vectors in ChromaDB.

In this step, get ready to:
  • Create sample documents with specific, verifiable facts about a fictional company.
  • Write a chunking function that splits documents into fixed-size pieces.
  • Ingest the chunks into a ChromaDB vector database and confirm the count.
Create the sample documents

Your knowledge base needs documents packed with specific facts that Claude could never guess on its own. You'll create two documents for a fictional company called Moonstone Technologies: an employee handbook and a product guide.

  • Create a new file called rag.py in your rag-from-scratch folder and paste in the imports and the first document:
import os
import chromadb
from anthropic import Anthropic


# --- Sample Documents ---
DOCUMENTS = {
    "employee_handbook.txt": (
        "Moonstone Technologies Employee Handbook\n\n"
        "Remote Work Policy:\n"
        "All full-time employees are eligible for remote work up to 3 days "
        "per week. Employees must be available during core hours of 10 AM to "
        "3 PM in their local timezone. Remote work arrangements must be "
        "approved by the employee's direct manager. Employees working remotely "
        "are expected to maintain a dedicated workspace and reliable internet "
        "connection.\n\n"
        "Paid Time Off:\n"
        "Full-time employees receive 20 days of paid time off per year, "
        "accruing at a rate of 1.67 days per month. Unused PTO can be carried "
        "over up to a maximum of 5 days into the following year. PTO requests "
        "must be submitted at least 2 weeks in advance for absences of 3 or "
        "more consecutive days.\n\n"
        "Professional Development:\n"
        "Moonstone Technologies provides each employee with a $2,000 annual "
        "professional development budget. This can be used for conferences, "
        "courses, certifications, or books related to their role. Requests are "
        "submitted through the internal learning portal and approved by HR "
        "within 5 business days."
    ),
}

What does this code do?

The DOCUMENTS dictionary maps filenames to their full text content. Each filename key acts as a label you can trace back to later.

The employee handbook includes three sections with verifiable facts: a remote work policy (3 days/week, core hours 10 AM to 3 PM), PTO details (20 days/year), and a professional development budget ($2,000/year). These are facts Claude cannot know because Moonstone Technologies is fictional.

  • Now add the product guide document. In rag.py, find the closing } of the DOCUMENTS dictionary and add this entry above it:
    "product_guide.txt": (
        "MoonBeam Product Guide\n\n"
        "Overview:\n"
        "MoonBeam is Moonstone Technologies' flagship data analytics platform. "
        "It enables teams to build interactive dashboards, run complex queries, "
        "and share insights across the organization.\n\n"
        "Key Features:\n"
        "- Real-time Dashboard Builder: Create custom dashboards with "
        "drag-and-drop widgets. Supports up to 50 widgets per dashboard with "
        "auto-refresh intervals as low as 30 seconds.\n"
        "- Query Engine: Write SQL queries against connected data sources. "
        "Supports PostgreSQL, MySQL, BigQuery, and Snowflake. Query results "
        "are cached for 15 minutes by default.\n"
        "- Collaboration: Share dashboards and queries with team members. "
        "Role-based access control with Viewer, Editor, and Admin roles. "
        "Comments and annotations on any dashboard widget.\n"
        "- API Access: RESTful API for programmatic access. Rate limited to "
        "100 requests per minute per API key. JSON and CSV export formats.\n\n"
        "Pricing:\n"
        "- Starter Plan: $49/month, up to 5 users, 10 dashboards, "
        "100 queries/day.\n"
        "- Team Plan: $199/month, up to 25 users, unlimited dashboards, "
        "1000 queries/day.\n"
        "- Enterprise Plan: Custom pricing, unlimited users, dedicated "
        "support, SSO integration."
    ),

What does this document contain?

The product guide covers MoonBeam, a fictional analytics platform. It lists concrete specs like supported databases (PostgreSQL, MySQL, BigQuery, Snowflake), API rate limits (100 requests/minute), and exact pricing tiers ($49/month Starter, $199/month Team).

These specific numbers are the test. If Claude later quotes them correctly, you know it pulled the answer from your documents, not from its training data.

  • Save rag.py.
  • Verify your documents loaded correctly by running this command:
python -c "from rag import DOCUMENTS; print(len(DOCUMENTS))"

You should see 2 printed, confirming both documents are in the dictionary.

Seeing a SyntaxError?

  • Make sure each document entry ends with ), (closing parenthesis followed by a comma). A missing comma between entries causes a syntax error.
  • Check that the DOCUMENTS dictionary has an opening { and a closing } wrapping both entries.
  • Confirm you are running the command from inside the rag-from-scratch folder with your virtual environment activated.

Need help debugging? Why is my rag.py throwing a SyntaxError when I try to import DOCUMENTS?

Write the chunking and ingestion functions

Your documents are long blocks of text, but a vector database works best with smaller pieces. A chunking function splits each document into fixed-size segments that can be individually matched to a question.

  • Add the chunk size constants and the chunk_text function below the DOCUMENTS dictionary in rag.py:
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50


def chunk_text(text, chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end].strip()
        if chunk:
            chunks.append(chunk)
        start = end - overlap
    return chunks

What does this function do?

  • CHUNK_SIZE = 500 sets each chunk to 500 characters. This is small enough to be focused on one topic, but large enough to hold a complete thought.
  • CHUNK_OVERLAP = 50 means consecutive chunks share 50 characters at their boundaries. This prevents facts from being split across two chunks and lost.
  • The while loop slides a window across the text, grabbing chunk_size characters at a time and advancing by chunk_size - overlap each iteration.
  • Save rag.py and verify the chunking function works by running:
python -c "from rag import chunk_text; chunks = chunk_text('A' * 1100); print(f'{len(chunks)} chunks from 1100 chars')"

You should see 3 chunks from 1100 chars printed. The 1100 characters were split into three overlapping pieces of 500 characters each.

Seeing 0 chunks or an import error?

  • Make sure you saved rag.py after adding the function.
  • Check that chunk_text is defined at the module level (not indented inside the DOCUMENTS dictionary or another block).

Still stuck? Why is my chunk_text function returning 0 chunks or failing to import?

  • Now add the ingest_documents function below chunk_text in rag.py:
def ingest_documents(collection, documents):
    """Chunk and store documents in ChromaDB."""
    all_chunks = []
    all_ids = []
    all_metadatas = []

    for doc_name, content in documents.items():
        chunks = chunk_text(content)
        for i, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            all_ids.append(f"{doc_name}_chunk_{i}")
            all_metadatas.append({"source": doc_name, "chunk_index": i})

    collection.add(
        documents=all_chunks,
        ids=all_ids,
        metadatas=all_metadatas,
    )
    return len(all_chunks)

What does this function do?

  • It loops through each document, calling chunk_text to split the text into pieces.
  • Each chunk gets a unique ID like employee_handbook.txt_chunk_0 so you can trace any chunk back to its source document and position.
  • The metadata dictionary stores the source filename and chunk index alongside each chunk. This is how you will later know which document an answer came from.
  • collection.add() sends everything to ChromaDB in one batch. ChromaDB automatically converts each text chunk into a vector embedding using its built-in model.
  • Save rag.py.
Run the ingestion pipeline

With the documents and functions in place, you need a main block that wires everything together: create a database, create a collection, ingest the documents, and report the result.

  • Add the main block at the bottom of rag.py:
if __name__ == "__main__":
    chroma_client = chromadb.Client()
    collection = chroma_client.create_collection(name="knowledge_base")

    num_chunks = ingest_documents(collection, DOCUMENTS)
    print(f"Ingested {num_chunks} chunks from {len(DOCUMENTS)} documents.\n")

What does this block do?

  • chromadb.Client() creates an in-memory ChromaDB instance. No files, no server, no setup. Everything lives in RAM for this session.
  • create_collection(name="knowledge_base") sets up a named container for your document vectors. Think of it like a database table dedicated to your knowledge base.
  • ingest_documents chunks all your documents and stores them. The returned count tells you exactly how many pieces were created.
  • Save rag.py.

Before you run this, how many chunks do you think two documents (roughly 800 and 1100 characters each) will produce with a 500-character chunk size?

  • Run the ingestion pipeline:
python rag.py

You should see output like:

Ingested 8 chunks from 2 documents.

The exact number may vary slightly depending on how whitespace is handled, but you should see chunks created from both documents.

Seeing an error instead of the chunk count?

  • If you see ModuleNotFoundError: No module named 'chromadb', your virtual environment may not be activated. Run .venv\Scripts\activate and try again.
  • If you see NameError: name 'chunk_text' is not defined, make sure chunk_text is defined before ingest_documents in the file. Python reads top to bottom.
  • If you see IndentationError, check that all functions are at the module level (no extra indentation) and the if __name__ block is at the very bottom of the file.

Need help? Why is my rag.py failing when I try to run the ingestion pipeline?

✔️ Awesome, I've got everything!

Great. Double check you have saved rag.py and you see the ingested chunks message in your terminal.

ⓧ I'd like to double check the full code

Scroll up and confirm your rag.py contains each of these sections in order:

  • Imports: os, chromadb, and Anthropic.
  • DOCUMENTS dictionary with two entries: employee_handbook.txt and product_guide.txt.
  • CHUNK_SIZE and CHUNK_OVERLAP constants set to 500 and 50.
  • chunk_text function with the sliding window loop.
  • ingest_documents function that chunks, assigns IDs and metadata, and calls collection.add().
  • if __name__ == "__main__" block that creates the client, collection, ingests, and prints the count.

Your knowledge base is loaded and ready. Next up, you'll wire this retrieval system into Claude so it can answer questions using your actual documents instead of making things up.

Wire Up Retrieval and Generation

In the last step, you chunked your documents and stored them as vectors in ChromaDB. That gives you a searchable knowledge base, but no way to actually use it.

Now you need to close the loop. You'll search ChromaDB for the most relevant chunks, pass them to Claude as context, and see what happens when the model has real information to draw from.

In this step, get ready to:
  • Write a retrieve function to search ChromaDB by similarity.
  • Write a generate_answer function to send retrieved context to Claude.
  • Test the full pipeline with in-scope and out-of-scope questions.
Search your knowledge base

Your document chunks are stored as vectors in ChromaDB. To search them, you pass in a plain-text question. ChromaDB converts it to a vector and finds the chunks closest in meaning using similarity search.

  • Add the retrieve function to rag.py by copying this code below the ingest_documents function:
def retrieve(collection, query, n_results=3):
    """Find the most relevant chunks for a query."""
    results = collection.query(
        query_texts=[query],
        n_results=n_results,
    )
    return [
        {"text": doc, "source": meta["source"], "distance": dist}
        for doc, meta, dist in zip(
            results["documents"][0],
            results["metadatas"][0],
            results["distances"][0],
        )
    ]

What does this code do?

  • collection.query() sends your question text to ChromaDB, which converts it to a vector and finds the 3 closest chunks by distance.
  • The list comprehension packages each result into a dictionary with the chunk text, its source filename, and a distance score (lower means more relevant).
  • Save rag.py.
  • Test retrieval by running the script in interactive mode:
python -i rag.py

What does the -i flag do?

The -i flag tells Python to run the script first, then drop you into an interactive prompt (>>>) where all the variables from the script are still available. You can test your functions without writing throwaway code in the file.

After the chunk count prints, you'll see a >>> prompt. Type these two lines to test a query:

results = retrieve(collection, "remote work policy")
print(results[0]["source"])

You should see employee_handbook.txt printed. ChromaDB found the right document for your "remote work" query.

  • Type exit() to leave the interactive prompt.

Seeing a NameError?

  • If you see NameError: name 'retrieve' is not defined, make sure the retrieve function is placed ABOVE the if __name__ == "__main__": block, not inside it.
  • If you see NameError: name 'collection' is not defined, make sure you ran python -i rag.py (with the -i flag), not just python rag.py.
  • Still stuck? Help me debug a NameError when testing my retrieve function in interactive mode.
Wire retrieval to Claude

Retrieval is working. Now you need to format the returned chunks as context and send them to Claude alongside the user's question so it can generate an answer grounded in your actual documents.

  • Add the generate_answer function below retrieve in rag.py:
def generate_answer(query, context_chunks):
    """Generate a grounded answer using Claude with retrieved context."""
    client = Anthropic()

    # Format each chunk with its source label
    context = "\n\n---\n\n".join(
        [f"[Source: {c['source']}]\n{c['text']}" for c in context_chunks]
    )

    message = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        system="Answer the user's question based on the provided context.",
        messages=[
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}",
            }
        ],
    )

    # Extract the text from Claude's response
    for block in message.content:
        if block.type == "text":
            return block.text
    return ""

What does this code do?

  • The function joins retrieved chunks into a single context string, separated by --- dividers. Each chunk is labeled with its source filename so Claude can see where the information came from.
  • The system prompt is deliberately simple right now: "Answer the user's question based on the provided context." It tells Claude to use the context but does not yet enforce citations or honest refusal.
  • The user message combines the formatted context and the question into one prompt.
  • Save rag.py.
Run the full pipeline

You have retrieval and generation as separate functions. The last piece is a function that ties them together and prints the results.

  • Add the ask function below generate_answer in rag.py:
def ask(collection, query):
    """Full RAG pipeline: retrieve then generate."""
    chunks = retrieve(collection, query)

    if not chunks:
        print(f"\nQuestion: {query}")
        print("No relevant documents found.")
        return

    answer = generate_answer(query, chunks)

    # Print the retrieved chunks and final answer
    print(f"\nQuestion: {query}")
    print(f"\nRetrieved {len(chunks)} chunks:")
    for c in chunks:
        print(
            f"  - [{c['source']}] (distance: {c['distance']:.4f}) "
            f"{c['text'][:80]}..."
        )
    print(f"\nAnswer:\n{answer}")
    print("-" * 60)
    return answer

What does this code do?

  • ask is the full RAG pipeline in one function. It calls retrieve to get the closest chunks, then passes them to generate_answer to get Claude's response.
  • It prints the retrieved chunks with their source and distance score so you can see exactly what information Claude is working with. Then it prints the final answer.

Now update the __main__ block to call ask with three different questions.

  • In the __main__ block, find this line:
    print(f"Ingested {num_chunks} chunks from {len(DOCUMENTS)} documents.\n")
  • Add these three lines directly below it, so the bottom of your file looks like this:
    print(f"Ingested {num_chunks} chunks from {len(DOCUMENTS)} documents.\n")

    ask(collection, "What is the remote work policy?")
    ask(collection, "How much does the MoonBeam Team Plan cost?")
    ask(collection, "What is the company's policy on stock options?")
  • Save rag.py.

Before you run this, think about the three questions. The first two ask about information that exists in your documents. The third asks about stock options, which is not mentioned anywhere. Do you think Claude will handle all three the same way?

  • Run the full pipeline:
python rag.py

You should see three rounds of output. For each question, the script prints the retrieved chunks and Claude's answer.

What should I see?

  • "What is the remote work policy?" pulls chunks from employee_handbook.txt and Claude answers correctly (3 days per week, core hours 10 AM to 3 PM).
  • "How much does the MoonBeam Team Plan cost?" pulls chunks from product_guide.txt and Claude answers correctly ($199/month).
  • "What is the company's policy on stock options?" is the interesting one. ChromaDB still returns chunks (it always returns the closest matches, even if they're not relevant). Claude reads those irrelevant chunks and fabricates a stock options policy that does not exist in your documents.

The in-scope questions prove that retrieval works. Claude is answering from your actual documents instead of making things up. That is the core value of RAG.

The stock options question reveals a gap. Your basic system prompt tells Claude to use the context, but it does not tell Claude what to do when the context does not contain the answer. Claude fills the silence with a confident fabrication. This is the problem the next step will fix.

Seeing an API error?

  • If you see an authentication error, check that your ANTHROPIC_API_KEY environment variable is still set. Virtual environment terminals can lose environment variables if you closed and reopened them.
  • If you see a syntax error, make sure the retrieve, generate_answer, and ask functions are all at the top level of the file (not indented inside another function or the __main__ block).
  • Still stuck? Help me debug an error when running my RAG pipeline.

✔️ Awesome, I've got everything!

Double check you've saved rag.py and that all three questions produce output when you run the script.

ⓧ I'd like to double check the full code

Your rag.py should contain these sections in order. Scroll up and confirm each one matches the code you added:

  • Imports: os, chromadb, and Anthropic.
  • DOCUMENTS: dictionary with employee_handbook.txt and product_guide.txt.
  • Constants: CHUNK_SIZE = 500 and CHUNK_OVERLAP = 50.
  • chunk_text function.
  • ingest_documents function.
  • retrieve function (added in this step).
  • generate_answer function (added in this step).
  • ask function (added in this step).
  • __main__ block with ingestion, chunk count print, and three ask() calls.

Your RAG pipeline is working. Claude now answers from your actual documents instead of inventing information. In the next step, you'll bulletproof the pipeline so Claude cites its sources and honestly refuses questions it cannot answer.

Bulletproof Your Pipeline

In the last step, you wired retrieval into generation and watched Claude correctly answer in-scope questions using your documents. One problem remains: when you asked about stock options, Claude still fabricated an answer by stitching together irrelevant chunks.

The fix is surprisingly simple. A stronger system prompt tells Claude exactly how to behave when the context falls short. In this step, you will upgrade the prompt to demand citations and honest refusal, then confirm how your overlapping chunks protect facts at boundaries.

In this step, get ready to:
  • Upgrade the system prompt to require source citations and honest refusal.
  • Understand how overlapping chunks preserve facts at boundaries.
  • Run the pipeline and compare cited answers to the fabricated ones from the previous step.
Upgrade the system prompt

The current system prompt tells Claude to use the provided context, but it says nothing about what to do when the context does not contain the answer. Claude defaults to being helpful, which means it fills in gaps with fabricated information.

  • In rag.py, scroll down to the generate_answer function.
  • Find this line inside client.messages.create():
        system="Answer the user's question based on the provided context.",
  • Replace that entire system= line with the following improved prompt:
        system=(
            "You answer questions based only on the provided context. "
            "If the context does not contain enough information to answer "
            "the question, say: 'I don't have enough information to answer "
            "that question based on the available documents.' "
            "Always cite which source document your answer comes from "
            "using [Source: filename] format."
        ),

What do these three rules do?

  • "based only on the provided context" tells Claude to ignore its pre-trained knowledge entirely. Only the retrieved chunks count.
  • The explicit refusal phrase gives Claude a specific script to follow when the context falls short. Without it, Claude defaults to filling in the gap.
  • The citation requirement forces Claude to back every claim with a [Source: filename] tag. If it cannot cite a source, it cannot make the claim.
  • Save rag.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved rag.py before moving on.

ⓧ I'd like to double check the full code

Your rag.py file is too long to show in full here. Scroll through the file and confirm these sections are present, in order:

  • The imports (os, chromadb, Anthropic) and the DOCUMENTS dictionary.
  • CHUNK_SIZE = 500 and CHUNK_OVERLAP = 50 constants.
  • chunk_text() with the overlap logic (start = end - overlap).
  • ingest_documents(), retrieve(), and generate_answer() with the new multi-line system= prompt.
  • ask() and the __main__ block with three test queries.

Seeing a syntax error?

  • Make sure the parentheses around the multi-line string are balanced. The opening ( goes right after system= and the closing ), must include the trailing comma.
  • Check that you replaced the entire old system= line. Leaving part of the old string behind creates invalid syntax.

Why is my system prompt change causing an error?

Understand how overlapping chunks work

The system prompt is one half of the improvement. The other half is already in your code: the overlap logic in chunk_text().

Why do chunks need to overlap?

When you split a document into 500-character chunks with no overlap, a fact that lands right at a chunk boundary gets sliced in half. The first chunk has the beginning of the sentence and the second chunk has the end. Neither chunk contains the complete fact, so retrieval misses it.

Your chunk_text() function uses start = end - overlap to advance the window by chunk_size - overlap (450 characters) instead of the full 500. This means consecutive chunks share 50 characters of boundary text, so facts near the edges appear in at least one complete chunk.

Run the improved pipeline

Before you run the pipeline, think about this: the stock options question retrieves chunks, but none of them mention stock options. What do you expect Claude to do now with the new system prompt?

  • Run the pipeline by entering this command:
python rag.py

You should see three answers. The first two (remote work policy and MoonBeam Team Plan pricing) now include [Source: employee_handbook.txt] or [Source: product_guide.txt] citations. The third question about stock options now gets an honest refusal instead of a fabricated policy.

What changed between Step 4 and Step 5?

The model is the same. The documents are the same. The retrieval logic is the same. The entire difference is in how you instructed Claude to handle the context. Prompt engineering turned a confident fabricator into a reliable, source-citing assistant.

Still seeing fabricated answers for stock options?

  • Make sure you saved rag.py after editing the system prompt. Running the script without saving uses the old prompt.
  • Check that the old system= line is fully replaced. If both the old and new prompts are present, Python uses whichever appears last (or throws a syntax error).
  • LLM responses are non-deterministic. Occasionally Claude may still attempt an answer. Run the script again. If the refusal consistently fails, the prompt string may be malformed.

Why is Claude still fabricating answers after my prompt change?

Secret mission

Add Source Filtering

Your knowledge base has multiple documents, but sometimes you only want answers from one. Build an ask_filtered function that scopes retrieval to a specific document, so HR questions only search the employee handbook and product questions only search the product guide.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally with no ongoing costs or running services. The Claude API usage you incurred during the project (a few cents) has already been billed. ChromaDB ran in-memory, so no data persists after your script finishes.

Resources you used:

  • The rag-from-scratch project folder on your Desktop (contains no_rag.py, rag.py, and the .venv virtual environment).
  • Python 3.13 (only if you installed it for this project).

Keep everything running

No action needed. Choose this if you want to keep experimenting with your RAG pipeline or extend it with new documents.

  • Your project files stay in the rag-from-scratch folder on your Desktop.
  • The ANTHROPIC_API_KEY environment variable was session-scoped. It disappears when you close your terminal. You will need to set it again with set next time you run the scripts.
  • There are no running services or recurring charges to worry about.

Pause - I'll come back to this later

There are no running processes to shut down. ChromaDB ran in-memory, so its data is already gone after each script run.

  • If your virtual environment is still active, deactivate it by running:
deactivate
  • Your project folder and virtual environment stay on your Desktop for when you come back.
  • When you return, reactivate the virtual environment and reset the API key before running your scripts.

Delete - I don't want to use this again

Remove all project files and start fresh. This cannot be undone.

  • If your virtual environment is still active, deactivate it first by running:
deactivate
  • Navigate out of the project folder by running:
cd ~/Desktop
  • Delete the project folder by running the command for your operating system:

macOS

rm -rf rag-from-scratch

Windows

rmdir /s /q rag-from-scratch
  • Confirm the folder is gone by running:
ls

You should no longer see rag-from-scratch in the output.

  • If you installed Python 3.13 specifically for this project and want to remove it:
  • On Windows, open Settings.
  • Select Apps, then Installed apps.
  • Search for Python 3.13, click the three-dot menu, and select Uninstall.

Keep Python 3.13 if you plan to use ChromaDB again

ChromaDB has compatibility issues with Python 3.14. If you plan to work with ChromaDB in future projects, keeping Python 3.13 installed saves you the reinstall next time.

Nice Work!

Nice Work!

You just built a complete RAG pipeline from scratch in Python, with no framework abstractions hiding the mechanics. You saw the hallucination problem firsthand and built the exact solution that production AI knowledge systems rely on.

You've learned how to:

  • Experienced the LLM hallucination problem firsthand by asking Claude about a fictional company, then fixed it by wiring retrieval-augmented generation into the pipeline to ground answers in real documents.
  • Chunked documents with overlapping boundaries, stored them as vector embeddings in ChromaDB, and retrieved the most relevant chunks using similarity search.
  • Wired retrieval into Claude's context and improved the pipeline with prompt engineering that enforces source citations and honest refusal for out-of-scope questions.
  • Secret Mission: Added metadata-based source filtering to scope queries to specific documents using ChromaDB's where filter.

Ready to quiz yourself?