Build an AI Knowledge Assistant

Build a RAG pipeline that answers questions from your PDF documents.

Introduction

30 Second Summary

Ever asked an AI chatbot a question and got a confident answer that turned out to be completely wrong? The problem is that general-purpose AI models pull from training data that might be outdated, incomplete, or irrelevant to your specific documents.

In this project, you will build a Retrieval-Augmented Generation (RAG) knowledge assistant that answers questions using only information from your PDF documents. Every answer cites its source, and the system refuses to make things up when your documents don't contain the answer.

What You'll Build

A Python CLI application that ingests PDF documents, splits them into chunks, embeds them into a local ChromaDB vector store, and answers questions using Claude Haiku 4.5.

By the end of this project, you'll have:

  • A PDF ingestion pipeline that extracts text, splits it into chunks, and generates vector embeddings for semantic search.
  • A local vector database storing your document embeddings, queryable by meaning instead of keywords.
  • A question-answering interface that cites exact sources (filename + page number) and refuses to answer when documents lack relevant information.
  • Secret Mission: Add multi-document filtering so users can scope questions to specific documents.

Are there any prerequisites?

You should be comfortable reading and understanding basic Python code. You'll also need an Anthropic API key (free credits are available on signup).

Before We Start

Before We Start...

Set Up Your Environment

Before you can build a RAG pipeline, you need a working Python environment with the right libraries installed.

This step gets your machine ready by installing Python, setting up a virtual environment, connecting to the Anthropic API, and creating the project folder structure.

In this step, get ready to:
  • Install Python 3.10+ and all project dependencies.
  • Set and verify your Anthropic API key.
  • Create the project directory structure with a sample PDF.
Install Python and project dependencies

This project requires Python 3.10 or higher because both PyMuPDF and sentence-transformers depend on it.

  • Press Ctrl+Alt+T to open a terminal window.
  • Check your Python version by running this command:
python3 --version

✔️ I see version 3.10 or higher

Your Python version is ready. Move on to the next instruction below.

ⓧ I see an older version

Many Linux distributions ship with an older Python version by default. You need to install Python 3.10+ alongside it using pyenv.

  • Install pyenv by running:
curl -fsSL https://pyenv.run | bash
  • Follow the on-screen instructions to add pyenv to your shell. Then restart your terminal.
  • Install Python 3.10 by running:
pyenv install 3.10
pyenv global 3.10
  • Close and reopen your terminal, then run python3 --version again to confirm you see 3.10 or higher.

ⓧ Command not found

Python is not installed on your system. Install it using pyenv.

  • Install pyenv by running:
curl -fsSL https://pyenv.run | bash
  • Follow the on-screen instructions to add pyenv to your shell. Then restart your terminal.
  • Install Python 3.10 by running:
pyenv install 3.10
pyenv global 3.10
  • Close and reopen your terminal, then run python3 --version again to confirm you see 3.10 or higher.

Now you will create a virtual environment to keep this project's packages isolated from your system Python.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir pdf-rag
cd pdf-rag
  • Create and activate a virtual environment by running:
python3 -m venv venv
source venv/bin/activate

What does activating a virtual environment do?

When you activate a virtual environment, your terminal uses a project-specific copy of Python and pip. Any packages you install go into the venv/ folder instead of your system-wide Python installation.

You should see (venv) at the beginning of your terminal prompt. This confirms the virtual environment is active.

  • Install all project dependencies by running:
pip install pymupdf==1.27.2.3 chromadb==1.5.9 sentence-transformers==5.6.0 anthropic==0.109.2

What are these packages?

  • pymupdf extracts text from PDF files with page-level precision.
  • chromadb is a local vector database for storing and searching document embeddings.
  • sentence-transformers converts text into numerical vectors (embeddings) for similarity search.
  • anthropic is the official Python SDK for calling Claude models.

This installation downloads several large model files and may take a few minutes. You should see a progress bar and a final success message.

Installation failing?

Make sure you see (venv) in your terminal prompt. If you do not, re-run source venv/bin/activate from inside the pdf-rag folder.

If you see errors about missing build tools, install them with sudo apt install build-essential and try the pip install command again.

help me troubleshoot my pip install errors for this RAG project

Set your Anthropic API key

The Anthropic Python SDK reads your API key from an environment variable called ANTHROPIC_API_KEY.

  • Set the environment variable by running this command, replacing the placeholder with your actual key:
export ANTHROPIC_API_KEY="your-api-key-here"

Where do I get an Anthropic API key?

Sign up at console.anthropic.com and create a new API key from the dashboard. New accounts receive free credits that are more than enough for this project.

Keep your key private. Never commit it to version control or share it in screenshots.

  • Verify the key works by running this test command:
python3 -c "from anthropic import Anthropic; client = Anthropic(); msg = client.messages.create(model='claude-haiku-4-5', max_tokens=50, messages=[{'role': 'user', 'content': 'Say hello in one word.'}]); print(msg.content[0].text)"
  • You should see a short greeting printed in your terminal (something like "Hello").

Seeing an authentication error?

Double-check that you copied your full API key correctly. Run echo $ANTHROPIC_API_KEY to confirm the variable is set in your current terminal session.

If you opened a new terminal tab, you need to run the export command again because environment variables are session-specific.

help me fix my Anthropic API key authentication error

Create the project directory structure

Your RAG project needs two subdirectories: one for your source PDFs, and one where ChromaDB will store its vector database files.

  • Create the docs/ and db/ subdirectories by running:
mkdir docs db
  • Place a sample PDF in the docs/ directory. You can use any PDF you have on hand. For example, download a free PDF from the internet and move it into docs/.
  • Confirm your project structure by running:
ls
  • You should see db, docs, and venv listed in the output.
  • Confirm your PDF is in place by running:
ls docs/
  • You should see the filename of the PDF you placed there.

Use any PDF you like

The more interesting the PDF content, the more fun your questions will be later. Try a textbook chapter, a product manual, or a research paper.

Don't see your PDF?

Make sure you are inside the pdf-rag folder when running ls docs/. Run pwd to confirm your current directory is ~/Desktop/pdf-rag.

help me find my project folder and place a PDF in the docs directory

Your environment is ready. Next up, you will write the ingestion script that extracts text from your PDF and splits it into searchable chunks.

Extract and Chunk PDF Text

Your environment is ready and your sample PDF is waiting in the docs/ folder. Before your AI assistant can answer questions about a document, it needs to read the document's text and break it into small, searchable pieces.

In this step, you will write ingest.py to extract text from PDFs page by page using PyMuPDF, then split that text into overlapping chunks. Overlapping chunks ensure that sentences sitting at a boundary are never cut in half and lost.

In this step, get ready to:
  • Write a function that extracts text from PDFs while preserving page numbers.
  • Write a chunking function that splits text into overlapping segments.
  • Test both functions on your sample PDF and verify the output.
Write the PDF text extraction function

The first job in any RAG pipeline is turning raw documents into plain text. PyMuPDF can open a PDF file and give you the text content of each page, which is exactly what you need to know which page an answer came from later.

  • Create a new file called ingest.py in your pdf-rag/ folder by running:
touch ingest.py
  • Open ingest.py in your code editor and add the following imports and extraction function:
import os
import pymupdf
from sentence_transformers import SentenceTransformer
import chromadb


def extract_text_from_pdf(pdf_path):
    """Extract text from a PDF file, returning a list of (page_num, text) tuples."""
    doc = pymupdf.open(pdf_path)
    pages = []
    for page_num, page in enumerate(doc, start=1):
        text = page.get_text()
        if text.strip():
            pages.append((page_num, text))
    doc.close()
    return pages

What does this code do?

The imports bring in four libraries you installed earlier. You will use pymupdf and os in this step. The SentenceTransformer and chromadb imports are for the next step.

The extract_text_from_pdf() function opens a PDF with pymupdf.open(), then loops through every page. For each page, page.get_text() returns the plain text content. It skips empty pages and records the 1-based page number alongside the text, so you can cite exact sources later.

  • Save ingest.py.
  • Confirm the file exists by running:
ls ingest.py

You should see ingest.py listed in the output.

Don't see the file?

Make sure your terminal is inside the pdf-rag/ folder. Run pwd to check your current directory, then cd ~/Desktop/pdf-rag if needed.

help me figure out why my ingest.py file isn't showing up

Add the text chunking function

A full page of text is too long for a language model to search through efficiently. Splitting text into smaller chunks (around 500 characters each) lets the system find the most relevant snippet without flooding the model with irrelevant content.

  • In ingest.py, add the following function below extract_text_from_pdf():
def chunk_text(text, chunk_size=500, overlap=100):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

Why overlapping chunks?

Each chunk is 500 characters long, but the next chunk starts only 400 characters after the previous one (500 minus 100 overlap). This means 100 characters of shared text appear in both neighbouring chunks.

The overlap prevents important sentences from being split across two chunks where neither chunk has the full context. When a user asks a question later, the overlap gives the retrieval system a better chance of finding a complete, meaningful answer.

  • Save ingest.py.

You won't see a visible change yet. The next substep tests both functions together.

Test extraction and chunking on your sample PDF

Time to verify that both functions work correctly on a real PDF. You will run a quick test in your terminal that imports the functions, extracts text from your sample PDF, chunks the first page, and prints the results.

  • Run this test command in your terminal (replace the PDF filename with your actual file):
python3 << 'EOF'
from ingest import extract_text_from_pdf, chunk_text

# Extract text from the sample PDF
pages = extract_text_from_pdf("docs/[[PDF_FILENAME="your-pdf-filename.pdf"]]")
print(f"Pages extracted: {len(pages)}")

# Show metadata for the first two pages
for page_num, text in pages[:2]:
    chunks = chunk_text(text)
    print(f"  Page {page_num}: {len(chunks)} chunks")
    print(f"  First chunk preview: {chunks[0][:80]}...")

print("\nChunking works! Each chunk has source page metadata.")
EOF

What should I see?

You should see output showing how many pages were extracted, how many chunks each page produced, and a preview of the first chunk's text. The exact numbers depend on your PDF, but a typical single-page document might produce 3-10 chunks.

The key thing to confirm: the page numbers start at 1 (not 0), and the chunk preview shows actual text from your document.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is still activated. Your terminal prompt should show the environment name in parentheses. If not, activate it again before re-running the test.

If you see a FileNotFoundError, double-check the PDF filename matches exactly what is in your docs/ folder (including the file extension and capitalisation). Run ls docs/ to see the exact filename.

help me debug why my ingest.py test is failing

✔️ Awesome, I've got everything!

Great. Double check you have saved ingest.py and your test output shows pages and chunks.

ⓧ I'd like to double check the full code

Here is the complete ingest.py as it should look at the end of this step:

import os
import pymupdf
from sentence_transformers import SentenceTransformer
import chromadb


def extract_text_from_pdf(pdf_path):
    """Extract text from a PDF file, returning a list of (page_num, text) tuples."""
    doc = pymupdf.open(pdf_path)
    pages = []
    for page_num, page in enumerate(doc, start=1):
        text = page.get_text()
        if text.strip():
            pages.append((page_num, text))
    doc.close()
    return pages


def chunk_text(text, chunk_size=500, overlap=100):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

Your PDF extraction and chunking pipeline is working. Next up, you will take these chunks, convert them into vector embeddings, and store them in ChromaDB so your assistant can search them by meaning.

Embed and Store in ChromaDB

In the previous step, you wrote functions that extract text from PDFs and split it into chunks with metadata. Now you have the raw ingredients for your knowledge base.

The next challenge is making those chunks searchable by meaning, not just keywords. You'll convert each chunk into a numerical vector (an embedding) and store it in a vector database that can find the most relevant chunks for any question.

In this step, get ready to:
  • Load the all-MiniLM-L6-v2 embedding model and generate vectors for all chunks.
  • Store embeddings in a persistent ChromaDB collection with source metadata.
  • Run a sample similarity query to verify retrieval works.
Write the main function to process PDFs and generate embeddings

Your ingest.py already has the helper functions for extracting and chunking text. Now you need a main() function that orchestrates the full pipeline: discover PDFs, chunk them, and convert chunks into vector embeddings.

  • In ingest.py, add the following code below the chunk_text function to build the main processing pipeline:
def main():
    docs_dir = "docs"
    db_dir = "db"

    # Scan the docs/ directory for all PDF files
    pdf_files = [f for f in os.listdir(docs_dir) if f.lower().endswith(".pdf")]
    if not pdf_files:
        print("No PDF files found in docs/ directory.")
        return

    print(f"Found {len(pdf_files)} PDF file(s). Processing...")

    # Collect all chunks, metadata, and unique IDs across all PDFs
    all_chunks = []
    all_metadatas = []
    all_ids = []

    for pdf_file in pdf_files:
        pdf_path = os.path.join(docs_dir, pdf_file)
        pages = extract_text_from_pdf(pdf_path)
        print(f"  {pdf_file}: {len(pages)} pages extracted")

        for page_num, text in pages:
            chunks = chunk_text(text)
            for i, chunk in enumerate(chunks):
                # ID format: filename_pPageNum_cChunkIndex
                chunk_id = f"{pdf_file}_p{page_num}_c{i}"
                all_chunks.append(chunk)
                all_metadatas.append({"source": pdf_file, "page": page_num})
                all_ids.append(chunk_id)

    print(f"\nTotal chunks: {len(all_chunks)}")
    print("Generating embeddings...")

    # Load the embedding model and convert all chunks to vectors
    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    embeddings = model.encode(all_chunks, show_progress_bar=True)

What does this code do?

The first section scans your docs/ directory for PDF files, then loops through each one. For every PDF, it calls your existing extract_text_from_pdf() and chunk_text() functions, collecting all chunks along with their metadata (source filename and page number).

Each chunk gets a unique ID like report.pdf_p3_c2 so you can trace any result back to its exact location. The SentenceTransformer model then converts every chunk into a 384-dimensional vector that captures its meaning.

What is a vector embedding?

A vector embedding is a list of numbers that represents the meaning of a piece of text. Sentences with similar meanings produce vectors that are close together in mathematical space.

The all-MiniLM-L6-v2 model runs locally on your CPU and produces these vectors at roughly 2000 sentences per second. No API calls, no cost.

Store the embeddings in ChromaDB

With embeddings generated, you need somewhere to store and query them. ChromaDB is a vector database that persists your data to disk and lets you search by similarity.

  • Still in ingest.py, add the following code directly below the embeddings = model.encode(all_chunks, show_progress_bar=True) line, inside the main() function:
    print("Storing in ChromaDB...")

    # Create a persistent database client that saves to disk
    client = chromadb.PersistentClient(path=db_dir)
    collection = client.get_or_create_collection(name="pdf_knowledge_base")

    # Upsert in batches of 100 to handle large document sets
    batch_size = 100
    for i in range(0, len(all_chunks), batch_size):
        end = min(i + batch_size, len(all_chunks))
        collection.upsert(
            ids=all_ids[i:end],
            documents=all_chunks[i:end],
            embeddings=embeddings[i:end].tolist(),
            metadatas=all_metadatas[i:end],
        )

    print(f"Done! {len(all_chunks)} chunks stored in {db_dir}/")
    print(f"Collection '{collection.name}' has {collection.count()} documents.")


if __name__ == "__main__":
    main()

What does this code do?

chromadb.PersistentClient(path=db_dir) creates a database that saves to your db/ folder. The data survives between runs, so you only need to ingest once.

get_or_create_collection either retrieves an existing collection or creates a new one. Using upsert instead of add means re-running the script updates chunks instead of duplicating them.

Batching into groups of 100 keeps memory usage stable when processing large document collections.

  • Save ingest.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before running it.

ⓧ I'd like to double check the full code

import os
import pymupdf
from sentence_transformers import SentenceTransformer
import chromadb


def extract_text_from_pdf(pdf_path):
    """Extract text from a PDF file, returning a list of (page_num, text) tuples."""
    doc = pymupdf.open(pdf_path)
    pages = []
    for page_num, page in enumerate(doc, start=1):
        text = page.get_text()
        if text.strip():
            pages.append((page_num, text))
    doc.close()
    return pages


def chunk_text(text, chunk_size=500, overlap=100):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks


def main():
    docs_dir = "docs"
    db_dir = "db"

    pdf_files = [f for f in os.listdir(docs_dir) if f.lower().endswith(".pdf")]
    if not pdf_files:
        print("No PDF files found in docs/ directory.")
        return

    print(f"Found {len(pdf_files)} PDF file(s). Processing...")

    all_chunks = []
    all_metadatas = []
    all_ids = []

    for pdf_file in pdf_files:
        pdf_path = os.path.join(docs_dir, pdf_file)
        pages = extract_text_from_pdf(pdf_path)
        print(f"  {pdf_file}: {len(pages)} pages extracted")

        for page_num, text in pages:
            chunks = chunk_text(text)
            for i, chunk in enumerate(chunks):
                chunk_id = f"{pdf_file}_p{page_num}_c{i}"
                all_chunks.append(chunk)
                all_metadatas.append({"source": pdf_file, "page": page_num})
                all_ids.append(chunk_id)

    print(f"\nTotal chunks: {len(all_chunks)}")
    print("Generating embeddings...")

    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    embeddings = model.encode(all_chunks, show_progress_bar=True)

    print("Storing in ChromaDB...")

    client = chromadb.PersistentClient(path=db_dir)
    collection = client.get_or_create_collection(name="pdf_knowledge_base")

    batch_size = 100
    for i in range(0, len(all_chunks), batch_size):
        end = min(i + batch_size, len(all_chunks))
        collection.upsert(
            ids=all_ids[i:end],
            documents=all_chunks[i:end],
            embeddings=embeddings[i:end].tolist(),
            metadatas=all_metadatas[i:end],
        )

    print(f"Done! {len(all_chunks)} chunks stored in {db_dir}/")
    print(f"Collection '{collection.name}' has {collection.count()} documents.")


if __name__ == "__main__":
    main()

Seeing indentation errors?

Make sure all the code inside main() is indented by exactly 4 spaces. The if __name__ == "__main__": block should be at the top level (no indentation), not inside main().

help me fix indentation errors in my ingest.py file

Run the ingestion and test retrieval

Time to run the full pipeline and verify that your vector database can find relevant chunks by meaning.

  • Run the ingestion script by executing this command from your pdf-rag directory:
python3 ingest.py
  • You should see output showing the number of pages extracted, total chunks created, a progress bar for embedding generation, and a final confirmation like Done! X chunks stored in db/.

First run takes a moment

The first time you run this, the sentence-transformers library downloads the all-MiniLM-L6-v2 model (about 90 MB). Subsequent runs skip the download and load it from cache.

  • Now test that semantic search works by running this quick verification script:
python3 -c "
from sentence_transformers import SentenceTransformer
import chromadb

model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
client = chromadb.PersistentClient(path='db')
collection = client.get_collection(name='pdf_knowledge_base')

query = 'What is this document about?'
query_embedding = model.encode([query])[0].tolist()
results = collection.query(query_embeddings=[query_embedding], n_results=3)

for doc, meta in zip(results['documents'][0], results['metadatas'][0]):
    print('[' + meta['source'] + ', p.' + str(meta['page']) + '] ' + doc[:80] + '...')
"
  • You should see 3 results printed, each showing the source filename, page number, and a preview of the chunk text.

What does this test do?

This script embeds the question "What is this document about?" using the same model, then queries ChromaDB for the 3 most similar chunks. The results prove that your vector database is working and that semantic search returns relevant content with metadata intact.

Getting zero results or a collection error?

  • If you see a collection-not-found error, make sure you ran python3 ingest.py first and it completed without errors.
  • If you see zero documents in the collection, check that your docs/ folder contains at least one PDF file with actual text content (not just images).
  • Make sure you're running the commands from inside the pdf-rag directory where db/ and docs/ exist.

help me debug why my ChromaDB query returns no results

Your vector database is loaded and searchable. Next up, you'll build the question-answering interface that retrieves relevant chunks and sends them to Claude for a grounded answer.

Build the Question-Answering Interface

Your vector database is loaded with document chunks and their embeddings. Now it's time to build the interface that actually answers questions.

In this step, you'll create ask.py. This script takes a question, finds the most relevant chunks using semantic search, then passes those chunks to Claude as context so it can generate a grounded answer.

In this step, get ready to:
  • Write the retrieval helper that formats document chunks for Claude.
  • Connect to your embedding model, ChromaDB, and the Anthropic API.
  • Build an interactive question-answering loop in your terminal.
Write the imports and context formatter

The question-answering script needs to embed questions, search the vector database, and call Claude. Each of these requires a dedicated library.

  • Create a new file called ask.py in your pdf-rag folder (the same directory where ingest.py lives).
  • Add the following imports and system prompt to the top of ask.py:
import argparse
from sentence_transformers import SentenceTransformer
import chromadb
from anthropic import Anthropic


# Instructions that tell Claude how to behave when answering questions
SYSTEM_PROMPT = """You are a knowledge assistant that answers questions using ONLY the provided context from PDF documents.

Rules:
1. Answer ONLY based on the provided context. Do not use any outside knowledge.
2. After each claim, cite the source in the format [filename, p.X].
3. If the context does not contain enough information to answer the question, say: "The documents do not contain information about this topic."
4. Be concise and direct in your answers."""

What does this code do?

The imports bring in four libraries: argparse for command-line flags, SentenceTransformer for embedding questions, chromadb for querying your vector store, and Anthropic for calling Claude.

The SYSTEM_PROMPT is a set of rules Claude must follow. It forces Claude to answer only from the document context you provide, cite sources in a specific format, and explicitly refuse when it doesn't have enough information.

What is a system prompt?

A system prompt is a special instruction block that shapes how a language model behaves across all its responses. Unlike user messages, the system prompt stays constant for the entire conversation.

In RAG systems, the system prompt is where you enforce grounding rules. Without it, the model might hallucinate answers from its training data instead of sticking to your documents.

Now add the build_context() function directly below the SYSTEM_PROMPT constant:

  • Add this function to ask.py below SYSTEM_PROMPT:
def build_context(results):
    """Format retrieved chunks into context for the prompt."""
    context_parts = []
    for i, (doc, metadata) in enumerate(zip(results["documents"][0], results["metadatas"][0])):
        source = metadata["source"]
        page = metadata["page"]
        context_parts.append(f"[Source: {source}, Page {page}]\n{doc}")
    return "\n\n---\n\n".join(context_parts)

What does this function do?

When ChromaDB returns search results, they come as raw lists of documents and metadata. The build_context() function formats these results into a readable block of text that Claude can parse.

Each chunk gets a header showing its source filename and page number, followed by the chunk text. Chunks are separated by --- dividers so Claude can clearly distinguish between different pieces of evidence.

  • Save ask.py.

You won't see output yet. The next chunks add the logic that actually runs these helpers.

Seeing a syntax error?

Check that the triple-quoted string for SYSTEM_PROMPT has matching """ at both the start and the end. A missing closing set of triple quotes causes Python to treat the rest of your file as part of the string.

help me fix a syntax error in my ask.py file

Set up the main function

The main function is where you connect to all three services: the embedding model, ChromaDB, and the Anthropic API. It also sets up a command-line flag you'll use later.

  • Add the main() function below build_context() in ask.py:
def main():
    # Set up command-line argument parsing
    parser = argparse.ArgumentParser(description="Ask questions about your PDF documents.")
    parser.add_argument("--source", type=str, default=None, help="Filter to a specific PDF filename.")
    args = parser.parse_args()

    db_dir = "db"

    # Load the same embedding model used during ingestion
    print("Loading embedding model...")
    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

    # Connect to the existing ChromaDB collection
    client = chromadb.PersistentClient(path=db_dir)
    collection = client.get_collection(name="pdf_knowledge_base")

    # Initialize the Anthropic client (reads ANTHROPIC_API_KEY from environment)
    anthropic_client = Anthropic()

    print(f"\nKnowledge base: {collection.count()} chunks loaded.")
    if args.source:
        print(f"Filtering to: {args.source}")
    print("Ask a question (type 'quit' to exit):\n")

What does this code do?

The argparse block sets up a --source flag that will let users filter results to a specific PDF. The flag is defined here but its filtering logic comes later.

The function loads the same all-MiniLM-L6-v2 model you used in ingest.py. Questions must be embedded with the same model that embedded the chunks, or the similarity search won't work.

Notice get_collection() instead of get_or_create_collection(). The collection must already exist from running ingest.py. If it doesn't exist, this will raise an error, which is the correct behavior. You don't want the Q&A script to silently create an empty database.

  • Save ask.py.

You won't see a change yet. The next chunk adds the interactive loop that uses all of these connections.

Not sure if your API key is set?

The Anthropic() client reads the ANTHROPIC_API_KEY environment variable automatically. If you closed your terminal since Step 1, you may need to re-export it.

help me check if my Anthropic API key environment variable is set correctly

Add the conversational loop

This is where the entire RAG pipeline comes together. Each question goes through embedding, retrieval, prompt construction, and generation in a single pass.

  • Add the following while loop inside the main() function, directly below the last print statement, then add the entry point at the bottom of the file:
    while True:
        question = input("You: ").strip()
        if not question:
            continue
        if question.lower() == "quit":
            print("Goodbye!")
            break

        # Embed the question using the same model as the documents
        question_embedding = model.encode([question])[0].tolist()

        # Query ChromaDB for the 5 most similar chunks
        query_params = {
            "query_embeddings": [question_embedding],
            "n_results": 5,
        }

        results = collection.query(**query_params)

        # Format retrieved chunks into readable context
        context = build_context(results)

        # Construct the user message with context and question
        user_message = f"""Context from documents:

{context}

---

Question: {question}"""

        # Send the grounded prompt to Claude
        message = anthropic_client.messages.create(
            model="claude-haiku-4-5",
            max_tokens=1024,
            system=SYSTEM_PROMPT,
            messages=[{"role": "user", "content": user_message}],
        )

        answer = message.content[0].text
        print(f"\nAssistant: {answer}\n")


if __name__ == "__main__":
    main()

What does this code do?

The loop reads a question from the terminal, then runs the full RAG pipeline on each iteration:

  • model.encode([question]) converts the question into a 384-dimensional vector, the same space as your document chunks.
  • collection.query() finds the 5 chunks whose embeddings are closest to the question embedding. This is semantic search.
  • The user_message f-string places the retrieved context above the question, so Claude sees evidence before the query.
  • anthropic_client.messages.create() sends the system prompt plus user message to claude-haiku-4-5 and returns the generated answer.

Why Haiku instead of Sonnet?

Claude Haiku 4.5 costs $1 per million input tokens and $5 per million output tokens. For a RAG grounding task where the answer exists in the provided context, Haiku is more than capable.

The entire tutorial uses roughly 10-50K tokens total, costing under $0.05 USD.

  • Save ask.py.
  • Test your assistant by running this command:
python3 ask.py

You should see the model load, then a prompt saying Ask a question (type 'quit' to exit): followed by You: waiting for your input.

  • Type a question related to your PDF content and press Enter.
  • You should see an answer with citations in the format [filename, p.X] after each claim.
  • Type quit to exit the loop.

Seeing an authentication error?

If you see a message about an invalid API key or missing authentication, your ANTHROPIC_API_KEY environment variable is not set in this terminal session. Re-export it before running the script again.

If you see a collection-not-found error, make sure you ran python3 ingest.py first. The ask.py script expects the pdf_knowledge_base collection to already exist in db/.

help me debug why ask.py is failing to connect to ChromaDB or the Anthropic API

✔️ Awesome, I've got everything!

Great. Double check you've saved ask.py and that your assistant answered with citations.

ⓧ I'd like to double check the full code

Here's the complete ask.py file up to this point:

import argparse
from sentence_transformers import SentenceTransformer
import chromadb
from anthropic import Anthropic


SYSTEM_PROMPT = """You are a knowledge assistant that answers questions using ONLY the provided context from PDF documents.

Rules:
1. Answer ONLY based on the provided context. Do not use any outside knowledge.
2. After each claim, cite the source in the format [filename, p.X].
3. If the context does not contain enough information to answer the question, say: "The documents do not contain information about this topic."
4. Be concise and direct in your answers."""


def build_context(results):
    """Format retrieved chunks into context for the prompt."""
    context_parts = []
    for i, (doc, metadata) in enumerate(zip(results["documents"][0], results["metadatas"][0])):
        source = metadata["source"]
        page = metadata["page"]
        context_parts.append(f"[Source: {source}, Page {page}]\n{doc}")
    return "\n\n---\n\n".join(context_parts)


def main():
    parser = argparse.ArgumentParser(description="Ask questions about your PDF documents.")
    parser.add_argument("--source", type=str, default=None, help="Filter to a specific PDF filename.")
    args = parser.parse_args()

    db_dir = "db"

    print("Loading embedding model...")
    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

    client = chromadb.PersistentClient(path=db_dir)
    collection = client.get_collection(name="pdf_knowledge_base")

    anthropic_client = Anthropic()

    print(f"\nKnowledge base: {collection.count()} chunks loaded.")
    if args.source:
        print(f"Filtering to: {args.source}")
    print("Ask a question (type 'quit' to exit):\n")

    while True:
        question = input("You: ").strip()
        if not question:
            continue
        if question.lower() == "quit":
            print("Goodbye!")
            break

        question_embedding = model.encode([question])[0].tolist()

        query_params = {
            "query_embeddings": [question_embedding],
            "n_results": 5,
        }

        results = collection.query(**query_params)

        context = build_context(results)

        user_message = f"""Context from documents:

{context}

---

Question: {question}"""

        message = anthropic_client.messages.create(
            model="claude-haiku-4-5",
            max_tokens=1024,
            system=SYSTEM_PROMPT,
            messages=[{"role": "user", "content": user_message}],
        )

        answer = message.content[0].text
        print(f"\nAssistant: {answer}\n")


if __name__ == "__main__":
    main()

Your RAG assistant is live. Next up, you'll test its refusal behavior and make sure citations are formatted consistently across different types of questions.

Add Refusal and Source Citation

Your assistant can answer questions from the vector store. But how do you know you can trust those answers? Without explicit guardrails, any language model will happily fabricate plausible-sounding information when context is missing.

In this step, you'll walk through the refusal and citation mechanisms in your system prompt, wire up the document source filter, and test both behaviors with real questions.

In this step, get ready to:
  • Review the refusal and citation instructions in your system prompt.
  • Wire the --source flag to filter queries by document.
  • Test grounding and refusal behavior with answerable and off-topic questions.
Review the refusal and citation rules

Your SYSTEM_PROMPT in ask.py already contains the two rules that make this assistant trustworthy. Let's look at what they do.

  • In ask.py, find the SYSTEM_PROMPT variable at the top of the file.
  • Locate Rule 3:
3. If the context does not contain enough information to answer the question, say: "The documents do not contain information about this topic."

Why does explicit refusal matter?

Language models generate confident-sounding text even when they lack information. Without this rule, the model would fill gaps with fabricated facts. By giving Claude an exact refusal phrase, you make it obvious to users when the answer comes from your documents versus thin air.

This is a core RAG design pattern called grounding. The model is grounded to your documents and nothing else.

  • Now locate Rule 2:
2. After each claim, cite the source in the format [filename, p.X].

How do citations work end-to-end?

The build_context() function formats each retrieved chunk with a [Source: filename, Page N] header. Claude reads these headers and uses them to produce inline citations like [source.pdf, p.3] in its answers.

This gives the user a clear audit trail back to the original document and page number for every claim.

Wire the source filter into the query

Your ask.py already accepts a --source flag via argparse. Right now, the flag is parsed but never used in the actual query. You'll add two lines that pass a ChromaDB metadata filter when the flag is provided.

  • In ask.py, scroll down to the while True loop and find this block:
        query_params = {
            "query_embeddings": [question_embedding],
            "n_results": 5,
        }

        results = collection.query(**query_params)
  • Add the following two lines between the query_params dictionary and the results = collection.query(**query_params) line, so it looks like this:
        query_params = {
            "query_embeddings": [question_embedding],
            "n_results": 5,
        }
        if args.source:
            query_params["where"] = {"source": args.source}

        results = collection.query(**query_params)

What does this filter do?

ChromaDB's where parameter filters results by metadata. When args.source is set, only chunks whose "source" metadata matches the given filename are returned.

This means the user can scope answers to a specific PDF file. Without the flag, all documents are searched as before.

  • Save ask.py.

Seeing an indentation error?

The if args.source: line must be indented to the same level as the query_params dictionary above it (8 spaces, inside the while True loop). The query_params["where"] assignment line needs one additional level of indentation (12 spaces).

help me fix the indentation in my ask.py where filter

Test refusal and citation behavior

Time to prove both behaviors work. You'll ask a question your documents can answer, and then ask something completely off-topic.

  • Run your assistant by entering this command:
python ask.py
  • Wait for the model to load and the prompt to appear.
  • Type a question that your sample PDF can answer. For example, if your PDF is about machine learning, type something like:
What is the main topic discussed in the document?
  • Check the response. You should see an answer with inline citations in the format [filename.pdf, p.X] after each claim.

What should I see?

A grounded answer that directly references content from your PDF. Each factual claim includes a citation like [your-file.pdf, p.2] pointing back to the exact page the information came from.

  • Now test the refusal behavior. Type a question that has nothing to do with your documents:
What is the capital of Mars?
  • The assistant should respond with: "The documents do not contain information about this topic."
  • Type quit to exit the assistant.

Not seeing the refusal message?

If Claude answers the off-topic question instead of refusing, double-check that SYSTEM_PROMPT in your ask.py contains Rule 3 with the exact refusal phrase. Also verify that system=SYSTEM_PROMPT is passed in the anthropic_client.messages.create() call.

my assistant isn't refusing off-topic questions

✔️ Awesome, I've got everything!

Great. Double check you've saved ask.py with the source filter in place.

ⓧ I'd like to double check the full code

Here is the complete ask.py at this point:

import argparse
from sentence_transformers import SentenceTransformer
import chromadb
from anthropic import Anthropic


SYSTEM_PROMPT = """You are a knowledge assistant that answers questions using ONLY the provided context from PDF documents.

Rules:
1. Answer ONLY based on the provided context. Do not use any outside knowledge.
2. After each claim, cite the source in the format [filename, p.X].
3. If the context does not contain enough information to answer the question, say: "The documents do not contain information about this topic."
4. Be concise and direct in your answers."""


def build_context(results):
    """Format retrieved chunks into context for the prompt."""
    context_parts = []
    for i, (doc, metadata) in enumerate(zip(results["documents"][0], results["metadatas"][0])):
        source = metadata["source"]
        page = metadata["page"]
        context_parts.append(f"[Source: {source}, Page {page}]\n{doc}")
    return "\n\n---\n\n".join(context_parts)


def main():
    parser = argparse.ArgumentParser(description="Ask questions about your PDF documents.")
    parser.add_argument("--source", type=str, default=None, help="Filter to a specific PDF filename.")
    args = parser.parse_args()

    db_dir = "db"

    print("Loading embedding model...")
    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

    client = chromadb.PersistentClient(path=db_dir)
    collection = client.get_collection(name="pdf_knowledge_base")

    anthropic_client = Anthropic()

    print(f"\nKnowledge base: {collection.count()} chunks loaded.")
    if args.source:
        print(f"Filtering to: {args.source}")
    print("Ask a question (type 'quit' to exit):\n")

    while True:
        question = input("You: ").strip()
        if not question:
            continue
        if question.lower() == "quit":
            print("Goodbye!")
            break

        question_embedding = model.encode([question])[0].tolist()

        query_params = {
            "query_embeddings": [question_embedding],
            "n_results": 5,
        }
        if args.source:
            query_params["where"] = {"source": args.source}

        results = collection.query(**query_params)

        context = build_context(results)

        user_message = f"""Context from documents:

{context}

---

Question: {question}"""

        message = anthropic_client.messages.create(
            model="claude-haiku-4-5",
            max_tokens=1024,
            system=SYSTEM_PROMPT,
            messages=[{"role": "user", "content": user_message}],
        )

        answer = message.content[0].text
        print(f"\nAssistant: {answer}\n")


if __name__ == "__main__":
    main()

Your assistant now only speaks from evidence and proves it with citations. Next up, you'll push this further in the Secret Mission by filtering answers to specific documents.

Secret mission

Multi-Document Filtering

Your RAG assistant queries all documents at once. But what if a user only wants answers from one specific PDF? In this mission, you'll test the multi-document filtering feature that scopes retrieval to a single source file.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs. The only expense was the Anthropic API calls you already made (approximately $0.01-0.05).

Resources you used:

  • The pdf-rag/ project directory (Python scripts, virtual environment, ChromaDB database in db/, and PDF documents in docs/).
  • The ANTHROPIC_API_KEY environment variable set in your terminal.
  • Your Anthropic API key (managed on Anthropic's platform).

Keep everything running

No action needed. Choose this if you're still actively building or plan to add more PDFs to your knowledge base.

  • Your ChromaDB data persists on disk in the db/ directory. It takes no memory or CPU when your scripts aren't running.
  • Your API key remains available for future use. No charges are incurred unless you run ask.py again.
  • You can add new PDFs to docs/ and re-run ingest.py any time to expand your knowledge base.

Pause - I'll come back to this later

There is nothing to pause. All resources are local files on your disk that consume no CPU or memory when idle.

  • If your virtual environment is still active, deactivate it by running:
deactivate
  • Unset the environment variable for this terminal session by running:
unset ANTHROPIC_API_KEY

Your project files, ChromaDB database, and PDFs remain on disk. You can pick up right where you left off by reactivating the virtual environment and re-exporting your API key.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your code, virtual environment, ChromaDB database, and PDF documents.

  • If your virtual environment is still active, deactivate it first by running:
deactivate
  • Unset the environment variable by running:
unset ANTHROPIC_API_KEY
  • Delete the entire project directory by running:
rm -rf ~/Desktop/pdf-rag
  • If you no longer need your Anthropic API key, log in to your Anthropic account and revoke it from your API keys page. This prevents any future usage on your account.

Nice Work!

Nice Work!

You just built a fully functional RAG knowledge assistant from scratch. Your system ingests PDFs, embeds them into a vector database, and answers questions with grounded citations.

You've learned how to:

  • Build a complete RAG pipeline that extracts text from PDFs with PyMuPDF, chunks it with overlap, generates vector embeddings, and stores them for retrieval.
  • Store and query document embeddings in a persistent ChromaDB vector database using semantic search to find the most relevant chunks for any question.
  • Connect Claude Haiku 4.5 via the Anthropic Python SDK with a system prompt that enforces citation grounding and refusal behavior for unanswerable questions.
  • Secret Mission: Added multi-document metadata filtering so users can scope questions to a specific PDF using ChromaDB's where parameter.

Ready to quiz yourself?