Build a RAG Document Q&A Chatbot

Build a RAG pipeline that answers questions about your documents with citations.

Introduction

30 Second Summary

You ask an AI chatbot about a report you just read. It gives you a confident, detailed answer that turns out to be completely made up.

In this project, you will build a document Q&A chatbot powered by Retrieval-Augmented Generation (RAG). You will feed your own documents into a vector database so the chatbot can pull the exact passages it needs to answer your questions with cited sources.

What You'll Build

You will paste a document into a sidebar, ask a question in the chat, and watch Claude stream back a grounded answer that cites the exact passages it pulled from your text.

By the end of this project, you'll have:

  • A working RAG pipeline that chunks your documents, embeds them into ChromaDB, and retrieves the most relevant passages by semantic similarity whenever you ask a question.
  • A Streamlit chat interface where Claude streams responses in real time and remembers your conversation across multiple questions.
  • Numbered source citations on every answer and a visible retrieval panel showing which document chunks the search returned, so you can verify every claim against the original text.
  • An optional Secret Mission to push your skills further.

Are there any prerequisites?

This project includes setup instructions for every tool you need. You will need an Anthropic API key with credits to power Claude's responses. The total API cost for completing this project is typically under $0.50.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Python Environment

Your RAG chatbot is a Python application. Before you can write any code, you need Python itself and a few packages that handle the AI responses, the document search, and the chat interface.

In this step, you'll create a dedicated project folder, set up an isolated Python environment, and install everything the chatbot needs to run.

In this step, get ready to:
  • Confirm Python 3.10 or higher is installed on your machine.
  • Create a project directory and a virtual environment.
  • Install anthropic, chromadb, and streamlit.
Create your project directory and virtual environment

Python 3.10 or higher is required for this project because Streamlit depends on it.

  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check your installed Python version by running this command:
python3 --version

✔️ I see version 3.10 or higher

Your Python version meets the minimum requirement. You're ready to continue.

ⓧ I see an older version

Your current Python version is too old for this project. You need 3.10 or higher.

  • Download the latest Python from the official Python downloads page.
  • Run the installer, following the default prompts.
  • Close and reopen your terminal after installation.
  • Run python3 --version again to confirm you now see 3.10 or higher.

macOS ships with an older Python

macOS often includes an older system Python. After installing a newer version, your terminal may still default to the old one. If python3 --version still shows an old version, try running the full path (e.g. /usr/local/bin/python3) or use a version manager like pyenv to switch your default.

ⓧ Command not found

Python is not installed on your machine yet.

  • Download Python 3.10+ from the official Python downloads page.
  • Run the installer, following the default prompts.
  • Close and reopen your terminal after installation.
  • Run python3 --version to confirm you see 3.10 or higher.

Windows users

On Windows, the command may be python --version instead of python3 --version. If python3 is not recognized after installing, try python instead.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-chatbot
cd rag-chatbot
  • Create and activate a Python virtual environment by running these commands:

macOS

python3 -m venv .venv
source .venv/bin/activate

Windows

python -m venv .venv
.venv\Scripts\activate

Your terminal prompt now starts with (.venv). This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python.

This prevents version conflicts between projects and makes cleanup simple. Delete the folder and everything goes away.

Not seeing (.venv) in your prompt?

  • Make sure you ran the activation command for your operating system. macOS uses source while Windows uses the Scripts\activate path.
  • On Windows PowerShell, if you see a "running scripts is disabled" error, run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser first, then try activating again.
  • Help me activate my Python virtual environment.
Install dependencies and set your API key

Your chatbot needs three Python packages. Each one handles a different piece of the pipeline.

  • Install all three packages by running this command:
pip install anthropic chromadb streamlit

What do these packages do?

  • anthropic connects to the Claude AI model through the Anthropic API.
  • chromadb is a vector database that stores and searches document embeddings locally using ChromaDB.
  • streamlit provides ready-made chat UI components so you can build the interface without writing any frontend code.

You should see pip downloading and installing a list of packages. This may take a minute because chromadb has many dependencies.

Seeing installation errors?

  • Confirm your virtual environment is active by checking for (.venv) in your prompt. If it is missing, go back and run the activation command.
  • If pip is not recognized, try python3 -m pip install anthropic chromadb streamlit instead.
  • Help me fix my pip install errors.

You also need an Anthropic API key so your code can send requests to Claude. The SDK reads this key automatically from an environment variable.

  • If you don't already have an Anthropic API key, create one in the Claude Console API keys page.
  • Copy the key value, then set it in your terminal by running this command:

macOS

export ANTHROPIC_API_KEY="your-api-key-here"

Windows

set ANTHROPIC_API_KEY=your-api-key-here
  • Replace your-api-key-here with the actual key you copied from the Claude Console.

Your API key stays in this terminal session

The export command (or set on Windows) sets the key only for your current terminal session. If you close the terminal, you will need to set it again.

The Anthropic SDK reads this key automatically when your code runs. You will not need to paste it anywhere in your code.

Verify the installation

Everything should be ready. Before you run the next command, what do you expect to see if all three packages installed correctly?

  • Test that all three packages import successfully by running this command:
python3 -c "import anthropic; import chromadb; import streamlit; print('All packages installed!')"

Windows users

On Windows, use python instead of python3 in the command above.

You should see All packages installed! printed in your terminal. This confirms Python can find and load all three packages inside your virtual environment.

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is active. You should see (.venv) at the start of your terminal prompt.
  • If the virtual environment is active but a package is missing, run pip install anthropic chromadb streamlit again.
  • I'm getting a ModuleNotFoundError when importing packages.

Your Python environment is locked and loaded. Next up, you'll build your first chat interface with Streamlit.

Build a Minimal Chat Interface

Your Python environment is ready with all three packages installed. Now it is time to build something you can see.

Before wiring up any AI, you need a working chat interface to iterate on. Streamlit has built-in chat components that make this fast.

In this step, get ready to:
  • Create app.py with Streamlit chat components and mock echo responses.
  • Run the app and verify that conversation history persists.
Set up the Streamlit app skeleton

Every Streamlit app starts with page configuration and state management. You will set up the page title and initialize a list to hold the conversation history.

  • Create a new file called app.py inside your rag-chatbot folder and add the following code:
import streamlit as st

# Configure the page layout and title
st.set_page_config(page_title="RAG Document Q&A", page_icon="\ud83d\udcc4", layout="wide")
st.title("RAG Document Q&A Chatbot")

# Initialize conversation history in session state
if "messages" not in st.session_state:
    st.session_state.messages = []

What does this code do?

st.set_page_config() sets the browser tab title and switches the app to a wide layout.

st.session_state is Streamlit's built-in storage that persists data across reruns. Streamlit reruns your entire script every time the user interacts with the app, so without session_state, your chat history would vanish after every message.

  • Save app.py.
  • Launch the app by running this command:
streamlit run app.py

What will Streamlit do when it starts?

Streamlit takes over your terminal and keeps running until you stop it with Ctrl+C. It also opens a browser tab automatically pointing to localhost:8501.

If this is your first time running Streamlit, it may ask for your email in the terminal. You can press Enter to skip this and continue.

You should see a page with the title "RAG Document Q&A Chatbot" and nothing else yet.

Don't see anything in the browser?

  • If the browser did not open automatically, go to http://localhost:8501 manually.
  • Make sure your virtual environment is active. You should see (.venv) at the start of your terminal prompt.
  • Check that app.py is saved inside the rag-chatbot folder, not a parent directory.

help me debug why Streamlit is not loading in my browser

Add the chat display and input logic

The page shows a title, but there is no way to chat yet. You need two things: a loop that renders past messages and an input box where the user types new ones.

  • In app.py, add the following code below the session state block to render the conversation history:
# Render all previous messages from session state
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

What does this loop do?

st.chat_message() renders a chat bubble styled for the given role ("user" or "assistant"). This loop replays every past message on each rerun, keeping the full conversation visible.

Right now the messages list is empty, so this loop does nothing yet. The next chunk creates messages for it to display.

  • Still in app.py, add the chat input and echo response logic below the display loop:
# Accept user input and generate a mock echo response
if prompt := st.chat_input("Ask a question about your documents"):
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.markdown(prompt)

    # Echo the user's message back as a mock assistant reply
    response = f"Echo: {prompt}"
    with st.chat_message("assistant"):
        st.markdown(response)
    st.session_state.messages.append({"role": "assistant", "content": response})

What does this code do?

st.chat_input() places a text input bar at the bottom of the page. When the user submits a message, the walrus operator (:=) captures it into prompt.

The user's message and a mock echo reply are both appended to st.session_state.messages. On the next rerun, the display loop above renders them both as chat bubbles.

  • Save app.py. Streamlit auto-reloads when you save.

You should now see a chat input bar at the bottom of the page. Type a message and you will see it echoed back as an assistant reply.

Chat input not appearing?

  • If Streamlit shows a "Rerun" button at the top of the page, click it to reload your changes.
  • Double-check that the st.chat_input line is at the top level of your script, not indented inside the for loop above it.
  • Make sure you saved the file after adding both new code blocks.

help me debug why my chat input is not showing up in Streamlit

✔️ Awesome, I've got everything!

Great. Double-check you have saved app.py and your echo chat is working.

ⓧ I'd like to double check the full code

Here is the complete app.py so far. Compare it against yours line by line.

import streamlit as st

st.set_page_config(page_title="RAG Document Q&A", page_icon="\ud83d\udcc4", layout="wide")
st.title("RAG Document Q&A Chatbot")

if "messages" not in st.session_state:
    st.session_state.messages = []

for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

if prompt := st.chat_input("Ask a question about your documents"):
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.markdown(prompt)

    response = f"Echo: {prompt}"
    with st.chat_message("assistant"):
        st.markdown(response)
    st.session_state.messages.append({"role": "assistant", "content": response})
Test the conversation flow

Before you type your next message, think about what will happen to the messages already on screen. Will they stay visible or disappear when you send a new one?

  • Type a few more messages into the chat input and press Enter after each one.
  • Confirm that every message you send stays on screen alongside its echo reply.

Each message pair (your input and the echo) stays on screen because st.session_state.messages persists across reruns and the display loop replays them all.

Your chat skeleton is alive. Next up, you will replace those echo replies with real AI responses from Claude.

Connect Claude and Hit the Context Limit

Your chat UI works. You can type messages and see them echoed back in real time.

Now it is time to plug in a real brain. In this step, you will connect the Anthropic API so Claude answers your questions with streaming responses. But here is the catch: Claude only knows what is in its training data. When you ask it about your own documents, it will either make things up or admit it has no idea. That gap is the entire reason RAG exists.

In this step, get ready to:
  • Replace the echo logic with real Claude API calls using streaming.
  • Add a system prompt that tells Claude it is a document Q&A assistant.
  • Hit the context limit by asking Claude about a document it has never seen.
Wire up the Anthropic SDK

The echo responses served their purpose. Time to swap them out for real AI.

  • Switch back to app.py in your editor.
  • Find the first line import streamlit as st and add this import directly below it:
import anthropic
  • Scroll down past the if "messages" not in st.session_state: block. Add this line directly below it, outside the if block (no indentation):
# Create the Anthropic client (reads ANTHROPIC_API_KEY automatically)
client = anthropic.Anthropic()

What does this set up?

The anthropic import gives you access to the Anthropic Python SDK. Calling anthropic.Anthropic() creates a client that automatically reads your ANTHROPIC_API_KEY environment variable. No need to pass the key in code.

Replace the echo logic with Claude streaming

Right now, every message you type gets echoed back as a string. You will replace that entire block with a real Claude API call that streams its response word by word.

  • In app.py, find these four lines inside the if prompt block:
    response = f"Echo: {prompt}"
    with st.chat_message("assistant"):
        st.markdown(response)
    st.session_state.messages.append({"role": "assistant", "content": response})
  • Delete those four lines and replace them with this code:
    # Tell Claude what role to play
    system_prompt = "You are a helpful document Q&A assistant."

    # Build conversation history from the last 10 messages
    conversation_history = [
        {"role": m["role"], "content": m["content"]}
        for m in st.session_state.messages[-10:]
    ]

    with st.chat_message("assistant"):
        # Define a generator that streams Claude's response
        def stream_response():
            with client.messages.stream(
                model="claude-sonnet-5",
                max_tokens=1024,
                system=system_prompt,
                messages=conversation_history,
            ) as stream:
                for text in stream.text_stream:
                    yield text

        # Display the streaming response and capture the full text
        response = st.write_stream(stream_response())

    st.session_state.messages.append({"role": "assistant", "content": response})

What does this code do?

The system_prompt tells Claude it is a document Q&A assistant. This shapes how Claude frames its answers.

The conversation_history list comprehension grabs the last 10 messages from session state and formats them as Claude expects. This gives Claude context about what the user has already asked.

Inside stream_response(), the client.messages.stream() context manager opens a streaming connection to the Anthropic API. The stream.text_stream iterator yields text chunks as Claude generates them. Streamlit's st.write_stream renders each chunk with a typewriter effect and returns the full response as a string.

  • Save app.py.
  • Return to your browser at localhost:8501. Streamlit should detect the change and prompt you to rerun. Click Rerun (or select Always rerun).
  • Type a general knowledge question like What is the capital of France? and press Enter.

You should see Claude's response stream in word by word. No more echo. Real AI.

Seeing an API key error?

Make sure your ANTHROPIC_API_KEY environment variable is set in the same terminal session where you ran streamlit run app.py. If you opened a new terminal, you need to re-export the key and re-activate your virtual environment.

Help me fix my Anthropic API key error in Streamlit

Hit the context limit

Claude answered your general question perfectly. But this chatbot is supposed to answer questions about your documents. What happens when you ask about a document Claude has never seen?

  • Before you try this, predict: what do you think Claude will say if you ask it about a specific document it has never been given?
  • Type this into the chat:
What does my project proposal say about the timeline for Phase 2?

Claude either invents an answer that sounds plausible but is completely fabricated, or it tells you it does not have access to your project proposal. Both outcomes prove the same point: Claude can only work with what is in its training data or the current conversation. It has no way to access your documents.

Why is this the key problem?

This is the core limitation that the entire RAG pipeline solves. An LLM on its own has no access to private data. It will either hallucinate a plausible-sounding answer or refuse to answer.

In the next step, you will build a retrieval pipeline that feeds your actual document content into Claude's prompt. That way, Claude answers from your data instead of guessing.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py.

ⓧ I'd like to double check the full code

Scroll up and confirm your app.py contains these sections in order:

  • The import streamlit as st and import anthropic imports at the top.
  • The st.set_page_config and st.title lines.
  • The session state initialization block.
  • The client = anthropic.Anthropic() line.
  • The for message loop that displays chat history.
  • The if prompt block containing system_prompt, conversation_history, the stream_response() function, and st.write_stream.

You have a working Claude-powered chatbot with streaming responses. But you have also seen its biggest limitation firsthand: it cannot answer questions about documents it has never seen. In the next step, you will solve that by building a RAG pipeline with ChromaDB.

Build the RAG Pipeline with ChromaDB

In the last step, you saw Claude's limitation firsthand. Without access to your documents, it either made things up or admitted it could not help.

This is the core problem that Retrieval-Augmented Generation solves. Instead of hoping the model knows the answer, you feed it the relevant passages from your own documents right before it generates a response.

In this step, you will build the full RAG pipeline. Documents go in, get split into small chunks, get embedded into a ChromaDB vector database, and get retrieved by semantic similarity whenever the user asks a question.

In this step, get ready to:
  • Build a chunking function and document upload sidebar that processes text into ChromaDB.
  • Wire up semantic retrieval so Claude receives relevant document passages before generating answers.
  • Display the retrieved context alongside each answer so you can verify what Claude saw.
Set up ChromaDB and the chunking function

Before you can store or retrieve any documents, you need to initialize a vector database and give your app a way to split long text into smaller, searchable pieces.

  • In app.py, find this line near the top of your file:
import anthropic
  • Add import chromadb on the next line, so your imports look like this:
import anthropic
import chromadb
  • Next, find the existing session state block that initializes messages:
if "messages" not in st.session_state:
    st.session_state.messages = []
  • Add the following ChromaDB initialization block directly below it:
# Initialize ChromaDB vector database in session state
if "chroma_client" not in st.session_state:
    st.session_state.chroma_client = chromadb.Client()
    st.session_state.collection = st.session_state.chroma_client.get_or_create_collection(
        name="documents"
    )
    st.session_state.chunk_count = 0

What does this code do?

chromadb.Client() starts an in-memory vector database. No server, no config files. It lives entirely inside your Python process.

get_or_create_collection(name="documents") creates a named collection to hold your document chunks and their vector embeddings. If the collection already exists (because Streamlit re-ran the script), it reuses it.

chunk_count tracks how many chunks have been stored so far, which gives each chunk a unique ID.

  • Save app.py and check that your Streamlit app still loads at localhost:8501 without errors.

Seeing an import error?

Make sure your virtual environment is activated. If you see ModuleNotFoundError: No module named 'chromadb', run pip install chromadb again inside your active virtual environment.

Help me fix this ChromaDB import error.

Now add the function that splits a long document into smaller, overlapping chunks. This is what makes retrieval work. Instead of feeding Claude the entire document, you break it into passages small enough for the embedding model to represent meaningfully.

  • In app.py, add the chunk_text function below the client = anthropic.Anthropic() line by pasting the following code:
def chunk_text(text, chunk_size=500, overlap=100):
    # Split text into overlapping chunks for embedding
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        if chunk.strip():
            chunks.append(chunk.strip())
        start += chunk_size - overlap
    return chunks

Why overlapping chunks?

Each chunk is ~500 characters long, but the window slides forward by only 400 characters (500 minus the 100-character overlap). This means the end of one chunk appears again at the start of the next.

Overlap prevents important sentences from being split across two chunks and lost during retrieval. If a key fact spans the boundary between two chunks, the overlap ensures at least one chunk captures it in full.

  • Save app.py.
Build the document upload sidebar

You have the chunking function, but no way for a user to provide a document yet. The sidebar gives the user a text area to paste their document and a button to process it into the vector database.

  • In app.py, find the for message in st.session_state.messages: loop. Add the entire sidebar block directly above that loop:
with st.sidebar:
    st.header("Document Upload")
    doc_text = st.text_area(
        "Paste your document text here:",
        height=300,
        placeholder="Paste the text of the document you want to ask questions about...",
    )
    if st.button("Process Document", type="primary"):
        if doc_text.strip():
            # Chunk the text and store in ChromaDB
            chunks = chunk_text(doc_text)
            ids = [
                f"chunk_{st.session_state.chunk_count + i}"
                for i in range(len(chunks))
            ]
            st.session_state.collection.add(
                documents=chunks,
                ids=ids,
            )
            st.session_state.chunk_count += len(chunks)
            st.success(f"Processed {len(chunks)} chunks into the vector database!")
        else:
            st.warning("Please paste some text first.")

    st.divider()
    chunk_total = st.session_state.collection.count()
    st.caption(f"Total chunks in database: {chunk_total}")

What does this code do?

st.text_area creates a large text input in the sidebar where the user pastes their document.

When the user clicks Process Document, the code calls chunk_text() to split the text, generates a unique ID for each chunk, and stores them in ChromaDB using collection.add(). ChromaDB handles vectorization automatically using its built-in embedding model (Sentence Transformers all-MiniLM-L6-v2).

The caption at the bottom shows how many chunks are currently stored, so you always know how much context is available.

  • Save app.py and look at your Streamlit app in the browser.
  • You should see a Document Upload sidebar on the left with a text area and a Process Document button.
  • Paste any text you have handy into the text area (a paragraph from an article, a project description, meeting notes). Click Process Document.
  • You should see a green success message showing how many chunks were stored.

First-time model download

The first time you click Process Document, ChromaDB downloads its default embedding model (all-MiniLM-L6-v2) automatically. This is a one-time download that may take a moment. Subsequent runs use the cached model.

Don't see the sidebar?

Make sure the with st.sidebar: block is placed above the for message in st.session_state.messages: loop, not inside it.

If the sidebar appears but the button does nothing, check that your chunk_text function is defined above the sidebar block. Python needs the function to exist before it is called.

Help me fix my document upload sidebar.

Wire up semantic retrieval into the chat flow

Documents are stored, but Claude still cannot see them. You need a retrieval function that searches the vector database for the most relevant chunks, then injects those chunks into Claude's prompt.

  • In app.py, add the retrieve_context function directly below the chunk_text function by pasting the following code:
def retrieve_context(query, n_results=3):
    # Search ChromaDB for chunks most similar to the query
    collection = st.session_state.collection
    if collection.count() == 0:
        return [], []
    actual_n = min(n_results, collection.count())
    results = collection.query(
        query_texts=[query],
        n_results=actual_n,
    )
    documents = results["documents"][0] if results["documents"] else []
    distances = results["distances"][0] if results["distances"] else []
    return documents, distances

What does this code do?

collection.query() takes the user's question, converts it into a vector embedding, and finds the chunks whose embeddings are closest in meaning. This is semantic search. It matches by meaning, not by exact keywords.

The function returns two lists: the actual chunk text (documents) and their distances from the query (distances). Lower distance means higher similarity.

Now update the chat handler to call this function and build a context-aware system prompt. Claude will receive the retrieved chunks as part of its instructions.

  • Inside the if prompt := block, find this line:
    system_prompt = "You are a helpful document Q&A assistant."
  • Replace that single line with the following block. This calls retrieve_context and builds a dynamic system prompt based on whether any chunks were found:
    # Retrieve relevant chunks from the vector database
    context_chunks, context_distances = retrieve_context(prompt)

    # Build a dynamic system prompt with retrieved context
    system_prompt = "You are a helpful document Q&A assistant."
    if context_chunks:
        context_text = "\n\n".join(
            [f"[{i + 1}] {chunk}" for i, chunk in enumerate(context_chunks)]
        )
        system_prompt += (
            "\n\nUse the following document excerpts to answer the user's question. "
            "Cite your sources using numbered references like [1], [2], etc. "
            "If the provided context does not contain enough information to answer, "
            "say so clearly. Do not make up information beyond what is in the context."
            f"\n\n--- DOCUMENT CONTEXT ---\n{context_text}\n--- END CONTEXT ---"
        )
    else:
        system_prompt += (
            "\n\nNo documents have been uploaded yet. Let the user know they should "
            "paste a document in the sidebar before asking questions. You can still "
            "answer general questions, but clarify you are not referencing any uploaded documents."
        )

What does this code do?

Before Claude generates a response, the code queries ChromaDB for the top 3 most relevant chunks. It numbers them [1], [2], [3] and injects them into the system prompt between --- DOCUMENT CONTEXT --- markers.

The system prompt also tells Claude to cite which numbered chunks it used. If no documents have been uploaded yet, Claude is told to say so instead of guessing.

  • Save app.py.
  • If you already processed a document earlier, go back to the chat and ask a question about it. What do you think will happen?

Claude should now give a grounded answer that references specific details from your document. Compare this to the hallucinated or refused response from Step 3.

Still getting generic answers?

Make sure you replaced the original single-line system_prompt = ... and did not accidentally leave both the old and new lines in place. There should be only one system_prompt assignment inside the if prompt block.

Also confirm you processed a document in the sidebar first. The caption at the bottom of the sidebar should show a chunk count greater than 0.

Help me debug my RAG retrieval.

Display retrieved context for each answer

The chatbot now answers from your documents, but you cannot see which chunks it received. Adding a visible "Retrieved Context" expander lets you inspect exactly what the vector search returned and how similar each chunk was to your question.

  • In app.py, find the message history loop. Locate this line inside it:
        st.markdown(message["content"])
  • Add the following code directly below that line, still inside the with st.chat_message(message["role"]): block:
        # Show retrieved context for historical messages
        if "context_chunks" in message and message["context_chunks"]:
            with st.expander("Retrieved Context"):
                for i, (chunk, dist) in enumerate(
                    zip(message["context_chunks"], message["context_distances"])
                ):
                    st.markdown(f"**[{i + 1}]** (distance: {dist:.4f})")
                    st.text(chunk[:200] + "..." if len(chunk) > 200 else chunk)
                    st.divider()

What does this code do?

For each past assistant message, this checks if context chunks were saved alongside it. If they were, a collapsible "Retrieved Context" expander shows each chunk with its distance score. Lower distance means the chunk was more relevant to the question.

Now add the same expander for the current response, and update how messages are saved so context data persists in chat history.

  • Scroll down to the bottom of the if prompt block. Find this line:
    st.session_state.messages.append({"role": "assistant", "content": response})
  • Replace that line with the following block. This adds the expander for the current response and stores the context data in the message history:
    # Show retrieved context for the current response
    if context_chunks:
        with st.expander("Retrieved Context"):
            for i, (chunk, dist) in enumerate(
                zip(context_chunks, context_distances)
            ):
                st.markdown(f"**[{i + 1}]** (distance: {dist:.4f})")
                st.text(chunk[:200] + "..." if len(chunk) > 200 else chunk)
                st.divider()

    # Save message with context data for history
    st.session_state.messages.append(
        {
            "role": "assistant",
            "content": response,
            "context_chunks": context_chunks,
            "context_distances": context_distances,
        }
    )

What does this code do?

The expander block renders the same chunk-and-distance display for the response that was just generated. This lets you immediately inspect what ChromaDB returned.

The updated messages.append now stores context_chunks and context_distances alongside each assistant message. When Streamlit re-renders the chat history, the earlier loop code can display the context for every past answer.

  • Save app.py.

Before you test, think about what you expect to see. When you ask a question about your uploaded document, will there be a "Retrieved Context" expander? What will the distance values tell you?

  • If your app restarted and lost the previous document, paste your text into the sidebar again and click Process Document.
  • Ask a question about the document content in the chat.
  • Below the assistant's answer, click the Retrieved Context expander. You should see the top 3 chunks with their distance scores.

The chunks with the lowest distance values are the ones most semantically similar to your question. Claude used these passages to generate its answer.

No Retrieved Context expander?

Make sure you replaced the old st.session_state.messages.append(...) line with the new version that includes context_chunks and context_distances. The expander only appears when context data is saved with the message.

If you see the expander but it is empty, confirm that you have a processed document by checking the chunk count caption in the sidebar.

Help me debug my Retrieved Context display.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py.

ⓧ I'd like to double check the full code

Your app.py is now over 100 lines, so scroll through and confirm each section is present, in this order:

  • Imports: streamlit, anthropic, chromadb.
  • Page config and title.
  • Session state: messages list, then chroma_client, collection, and chunk_count.
  • client = anthropic.Anthropic().
  • chunk_text() function.
  • retrieve_context() function.
  • with st.sidebar: block with text area, button, divider, and caption.
  • for message in st.session_state.messages: loop with context_chunks expander inside.
  • if prompt := block with retrieve_context(prompt) call, dynamic system_prompt, streaming response, expander, and updated messages.append.

Your RAG pipeline is working end to end. Documents go in, get chunked and embedded, and Claude pulls the most relevant passages to answer your questions. Next up, you will add source citations and conversation memory to make this a production-quality system.

Add Source Citations and Conversation Memory

Your RAG chatbot pulls relevant context from uploaded documents and uses it to ground Claude's answers. That is a huge leap from the hallucinating chatbot in Step 3.

Two things take it from prototype to production-quality. First, structured citations that let you trace every claim back to its source chunk. Second, conversation memory so Claude can handle follow-up questions without losing the thread.

In this step, get ready to:
  • Verify that source citations map numbered references to the retrieved document chunks.
  • Test multi-turn conversation memory by asking follow-up questions.
Verify source citations in your answers

When someone asks your chatbot a question, how do they know the answer actually came from the document? Without citations, the user has to take the response on faith.

Your system prompt already instructs Claude to format citations as numbered references like [1], [2] that correspond to the retrieved chunks. Let's trace exactly how this works in your code.

  • In app.py, find the if context_chunks: block inside the chat input handler. Look at this section of the system prompt:
context_text = "\n\n".join(
    [f"[{i + 1}] {chunk}" for i, chunk in enumerate(context_chunks)]
)
system_prompt += (
    "\n\nUse the following document excerpts to answer the user's question. "
    "Cite your sources using numbered references like [1], [2], etc. "
    "If the provided context does not contain enough information to answer, "
    "say so clearly. Do not make up information beyond what is in the context."
    f"\n\n--- DOCUMENT CONTEXT ---\n{context_text}\n--- END CONTEXT ---"
)

How do the citations connect to the chunks?

Each retrieved chunk gets a numbered label like [1], [2], [3] in the context_text string. Claude sees these numbered chunks inside the system prompt and is told to cite them by number in its response.

The "Retrieved Context" expander below each answer shows the raw chunks and their distance scores. This lets you verify that Claude's [1] reference actually points to the chunk labeled [1] in the expander.

Let's test this. Make sure you have a document loaded from Step 4. If you restarted the app, the in-memory ChromaDB database resets, so you will need to paste and process a document again.

  • Return to your Streamlit app at localhost:8501.
  • If you need to reload a document, paste text into the sidebar and click Process Document.
  • Ask a specific question about something in your document. For example, if you pasted a project proposal, type something like "What are the main objectives?" in the chat input.

Claude's response should include numbered references like [1] and [2] inline. Click the Retrieved Context expander below the answer to see the actual chunks those numbers refer to.

Not seeing numbered citations in the response?

Make sure you processed a document before asking. If the "Total chunks in database" counter in the sidebar shows 0, paste and process a document first.

Claude sometimes paraphrases without explicit numbers. Ask a more specific question that targets a particular section. The model follows citation instructions more reliably when the question has a narrow scope.

Help me debug why my RAG chatbot isn't showing numbered citations in its responses.

Test conversation memory across multiple turns

A real Q&A session involves follow-up questions. "Tell me more about that second point" only makes sense if the chatbot remembers what it just said.

  • Find the conversation_history variable inside the chat input handler:
# Build conversation history from stored messages
conversation_history = [
    {"role": m["role"], "content": m["content"]}
    for m in st.session_state.messages[-10:]
]

How does conversation memory work here?

Every message (user and assistant) is stored in st.session_state.messages. Before each API call, the code extracts the last 10 messages and passes them as the messages parameter to Claude.

The [-10:] slice caps the history to avoid exceeding Claude's token limits. This keeps API costs predictable while giving Claude enough context for natural follow-ups.

Before you test this, think about what should happen. If you ask a follow-up like "Tell me more about that second point," do you think Claude will know what "that second point" refers to?

  • In the same chat session at localhost:8501, ask a follow-up question that references your previous answer. For example: "Tell me more about that second point" or "Can you explain that in simpler terms?"

Claude responds with a relevant follow-up that builds on its previous answer. It knows what "that" and "second point" refer to because it can see the full conversation history.

Why cap at 10 messages?

Claude Sonnet 5 supports up to 1M input tokens, so 10 messages is well within limits. The cap is a practical safeguard. In a long session with many document queries, the combined context (conversation history plus retrieved chunks plus system prompt) could grow large. Capping keeps API costs predictable and response times fast.

Claude not remembering previous messages?

If you refreshed the browser page between questions, st.session_state resets and the conversation history is cleared. Ask both questions in the same session without refreshing.

If you restarted the Streamlit server, the in-memory ChromaDB database also resets. You will need to re-paste and process your document before testing follow-ups.

Help me debug why Claude is not remembering my conversation history.

Double check the full code

Your app.py now has every feature in place: document ingestion, chunking, vector retrieval, streaming Claude responses, numbered citations, and conversation memory.

✔️ Awesome, I've got everything!

Great. Make sure you have saved app.py.

ⓧ I'd like to double check the full code

Here is the complete app.py. Scroll through and confirm each section is present in order:

  • Imports and page config at the top.
  • Session state initialization for messages, chroma_client, collection, and chunk_count.
  • The chunk_text() and retrieve_context() helper functions.
  • The sidebar with document upload and processing.
  • Message history replay with the Retrieved Context expander.
  • The chat input handler with retrieve_context(), the system prompt with citation instructions, conversation_history capped at 10, and stream_response().

Your RAG chatbot is complete. It ingests documents, retrieves relevant context by semantic similarity, streams cited answers from Claude, and remembers what you talked about across turns. That is the full Retrieval-Augmented Generation loop, built from scratch.

Secret mission

Multi-Document Support with Metadata Filtering

Your chatbot handles one document at a time. Real RAG systems manage entire libraries of documents and let users target their questions to specific sources. Can you upgrade your pipeline to support multiple named documents with filtered retrieval?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs locally with pay-per-use API charges, so there are no ongoing costs when the Streamlit app is not running.

Resources you used:

  • The rag-chatbot project directory on your Desktop, including app.py and the .venv virtual environment.
  • Anthropic API key, exported as the ANTHROPIC_API_KEY environment variable (billed per token during active use).

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue experimenting with your RAG chatbot.

  • Your project files and virtual environment stay on disk inside the rag-chatbot folder on your Desktop.
  • ChromaDB data is in-memory only. It resets each time you restart the app, so there is nothing persisted to disk.
  • The ANTHROPIC_API_KEY environment variable only lasts for your current terminal session. Anthropic API charges are pay-per-use with no ongoing cost when the app is not running.

Pause - I'll come back to this later

Shut down the running Streamlit server to free up your terminal. Your project files stay on disk so you can pick up where you left off.

  • Switch back to the terminal where Streamlit is running and press Ctrl+C to stop the server.
  • No ongoing charges. Your code and virtual environment remain in the rag-chatbot folder on your Desktop.
  • To resume later, open your terminal, navigate to the project folder, activate your virtual environment, re-export your API key, and run streamlit run app.py.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your code, virtual environment, and installed packages.

  • If the Streamlit server is still running, press Ctrl+C in the terminal to stop it.
  • If your virtual environment is still active, deactivate it by running:
deactivate
  • Delete the project directory. On macOS/Linux, run:
rm -rf ~/Desktop/rag-chatbot

On Windows, run:

rmdir /s /q %USERPROFILE%\Desktop\rag-chatbot
  • Optionally, revoke your API key in the Anthropic Console if you no longer need it. This prevents any future usage on that key.

Nice Work!

Nice Work!

You just built a production-style RAG chatbot from scratch. Your app takes raw documents, chunks and embeds them into a vector database, and lets Claude generate grounded, cited answers in real time.

You've learned how to:

  • Build a complete RAG pipeline from scratch, covering document chunking, vector embedding with ChromaDB, semantic retrieval, and grounded answer generation with the Anthropic API.
  • Experience the core RAG problem firsthand by watching Claude hallucinate without document context, then fix it with retrieval-augmented generation that cites numbered sources back to the original passages.
  • Create a polished Streamlit chat interface with streaming responses and multi-turn conversation memory so the chatbot remembers what you discussed earlier in the conversation.
  • Secret Mission: Add multi-document support with named documents and metadata-filtered vector search so you can ask questions about specific sources by name.

Ready to quiz yourself?