Build a Production RAG System

Introduction

30 Second Summary

When you ask ChatGPT a question, it pulls from its training data. But what if you need answers grounded in YOUR specific documents, like a company policy PDF, a research paper, or a technical manual?

In this project, you will build a Document Intelligence Agent that reads PDF documents and answers questions about them using progressively sophisticated retrieval techniques. You'll deploy it as a live web application with a shareable URL and quantified metrics for your resume.

What You'll Build

A full-stack RAG application with hybrid search, cross-encoder reranking, agentic query routing, conversation memory, streaming responses, and citation grounding, deployed on Hugging Face Spaces.

By the end of this project, you'll have:

• A production RAG pipeline with hybrid search (BM25 + vector), cross-encoder reranking, and quantified evaluation metrics using RAGAS.
• An intelligent agent built with LangGraph that routes questions to specialized handlers (Retrieval, Summarization, Comparison, Table Analysis) and maintains conversation memory.
• A deployed web application with a live URL, streaming token-by-token responses, and citation-grounded answers where every claim links back to its source passage.
• Secret Mission: A Research Mode that autonomously searches, evaluates, and synthesizes answers from multiple documents with a visible step-by-step reasoning trace.

Are there any prerequisites?

You should be comfortable with basic Python (functions, pip, and file I/O). You'll also need VS Code and Python 3.10+ installed on your machine.

No prior experience with AI, machine learning, or RAG systems is required. This project teaches everything from the ground up.

Before We Start

You're about to build a production-grade RAG system that reads PDF documents and answers questions intelligently.

Before writing any AI code, you need a clean development environment with the right packages, free API access, and version control in place. In this step, you'll create your project, install dependencies, connect to Groq's free LLM API, and initialize Git.

In this step, get ready to:

• Create your project structure and install all Python dependencies.
• Get free API keys for Groq and Hugging Face.
• Initialize Git, download sample PDFs, and verify your setup.

Create the project structure and virtual environment

Every Python project needs an isolated environment so its packages don't conflict with other projects on your machine. A virtual environment gives you exactly that.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir rag-system
cd rag-system

• Create a Python virtual environment by running:

python3 -m venv venv

• Activate the virtual environment by running:

source venv/bin/activate

You should see (venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated Python installation specific to your project. Packages you install here won't affect other projects, and other projects won't break yours.

• Create a file called requirements.txt with all the project dependencies by running:

cat > requirements.txt << 'EOF'
pymupdf
chromadb
langchain
langchain-community
langchain-groq
langgraph
sentence-transformers
rank-bm25
ragas
fastapi
gradio
uvicorn
python-dotenv
pydantic
pandas
httpx
EOF

What does this command do?

The cat > requirements.txt << 'EOF' syntax creates a file and writes everything between the two EOF markers into it. Each line is a Python package your RAG system needs.

• pymupdf reads PDF files.
• chromadb stores vector embeddings locally.
• langchain and langgraph orchestrate the AI pipeline and agentic workflows.
• sentence-transformers generates embeddings and powers cross-encoder reranking.
• rank-bm25 provides keyword-based search alongside vector search.
• ragas evaluates your RAG pipeline with industry-standard metrics.
• fastapi and gradio build the production API and chat interface.

• Install all the dependencies by running:

pip install -r requirements.txt

This will take a few minutes as it downloads and installs all packages. You should see a success message at the end listing all installed packages.

Seeing installation errors?

Make sure your virtual environment is active (you should see (venv) in your prompt). If you see permission errors, do not use sudo with pip. Deactivate and recreate the virtual environment instead.

help me fix pip install errors in my virtual environment

Get free API keys

Your RAG system uses Groq to run large language models at high speed for free. You'll also create a Hugging Face account for deployment later.

• Open your browser and go to console.groq.com.
• Sign up for a free account (no credit card required).
• Once logged in, click API Keys in the left sidebar.
• Click Create API Key.
• Give it a name like rag-project and click Submit.
• Copy the API key immediately. You will not be able to see it again after closing the dialog.

Why Groq?

Groq provides free access to powerful LLMs like Llama 3.3 70B with no credit card required. Usage is capped by rate limits (requests per minute and tokens per day), but you'll never be charged.

• Open a new tab and go to huggingface.co.
• Sign up for a free account (no credit card required).
• You'll use this account in a later step to deploy your app.

• Back in your terminal, create a .env file to store your API key by running:

cat > .env << 'EOF'
GROQ_API_KEY=your-key-here
EOF

• Open .env in a text editor and replace your-key-here with the API key you copied from Groq.

Never share or commit your API key

Your .env file contains your secret API key. You'll add it to .gitignore in the next substep so it never gets pushed to GitHub.

Initialize Git and project files

Version control tracks every change you make, so you can always revert if something breaks. You'll also create the directory structure and grab some sample PDFs to test with.

• Install Git by running:

sudo apt install git -y

• Verify Git installed correctly by running:

git --version

You should see a version number like git version 2.x.x in the output.

Command not found?

If you're on a non-Debian Linux distribution, use your package manager instead (e.g., sudo dnf install git for Fedora or sudo pacman -S git for Arch).

help me install Git on my Linux distribution

• Initialize a Git repository by running:

git init

• Create a .gitignore file to keep secrets and generated files out of version control by running:

cat > .gitignore << 'EOF'
.env
venv/
__pycache__/
chroma_db/
EOF

What does .gitignore do?

Any file or folder listed in .gitignore will be invisible to Git. This keeps your API key (.env), virtual environment (venv/), and database files (chroma_db/) out of your repository.

• Create the project directory structure by running:

mkdir src data evals tests docs

• Confirm the directories exist by running:

ls

You should see data, docs, evals, requirements.txt, src, tests, and venv listed in the output.

• Download sample AI research papers to use as test documents by running:

wget -O data/attention-is-all-you-need.pdf https://arxiv.org/pdf/1706.03762.pdf
wget -O data/rag-original-paper.pdf https://arxiv.org/pdf/2005.11401.pdf

What are these papers?

"Attention Is All You Need" introduced the Transformer architecture that powers every modern LLM. The RAG paper introduced Retrieval-Augmented Generation, the exact technique you're building in this project.

These are perfect test documents because you'll be able to verify your system's answers against the actual paper content.

Download failed?

If wget is not installed, install it with sudo apt install wget -y. If the download times out, try again. ArXiv servers occasionally throttle requests.

help me download PDF files from arxiv

• Verify your Groq API connection works by creating and running a test script:

python3 -c "
from dotenv import load_dotenv
from langchain_groq import ChatGroq

load_dotenv()

llm = ChatGroq(model='llama-3.3-70b-versatile')
response = llm.invoke('Say hello in one word.')
print('Connection successful!')
print('Response:', response.content)
"

You should see Connection successful! followed by the model's one-word response.

What does this test do?

This script loads your API key from .env, connects to Groq's llama-3.3-70b-versatile model, and sends a simple prompt. If you get a response, your API key works and you're ready to build.

Seeing an authentication error?

Double-check that your .env file contains GROQ_API_KEY= followed by your actual key (no quotes, no spaces around the = sign). Make sure you replaced your-key-here with the real key from console.groq.com.

help me fix my Groq API connection

✔️ Awesome, I've got everything!

Great. Double check you have (venv) in your terminal prompt and that your test script printed a response.

ⓧ I'd like to double check the full code

Here are the key files you should have in your rag-system/ folder:

requirements.txt:

pymupdf
chromadb
langchain
langchain-community
langchain-groq
langgraph
sentence-transformers
rank-bm25
ragas
fastapi
gradio
uvicorn
python-dotenv
pydantic
pandas
httpx

.env:

GROQ_API_KEY=your-key-here

.gitignore:

.env
venv/
__pycache__/
chroma_db/

Your folder structure should look like this:

rag-system/
├── .env
├── .gitignore
├── requirements.txt
├── venv/
├── src/
├── data/
│   ├── attention-is-all-you-need.pdf
│   └── rag-original-paper.pdf
├── evals/
├── tests/
└── docs/

Your environment is locked and loaded. Next up, you'll build your first RAG pipeline that ingests those PDFs and answers questions about them.

Build a Basic RAG Pipeline

Your environment is ready with all packages installed and your Groq API key verified. Now it's time to build the core engine of your project: a basic RAG pipeline.

A RAG pipeline connects a user's question to relevant documents, then feeds those documents to an LLM so it answers based on real source material instead of guessing. You will build this pipeline in four files: one to load and chunk PDFs, one to generate embeddings and store them, one to retrieve relevant chunks, and one to orchestrate the full query flow.

In this step, get ready to:

• Load PDF documents and split them into searchable text chunks.
• Generate vector embeddings and store them in a local database.
• Build a retrieval chain that answers questions grounded in your documents.

Load and chunk PDF documents

Before an LLM can answer questions about your documents, those documents need to be broken into small, searchable pieces called chunks. Each chunk is small enough for an embedding model to process and specific enough to contain a focused piece of information.

• Create the package initializer so Python can import from your src/ directory by running this command:
• Make sure you are in the rag-system/ folder in your terminal.

touch src/__init__.py

• Create a new file called src/ingest.py in VS Code and paste the following code:

import pymupdf as fitz  # PyMuPDF for reading PDF files
import os

def load_pdf(file_path):
    """Load a PDF file and extract text from each page."""
    doc = fitz.open(file_path)
    pages = []
    for page_num in range(len(doc)):
        page = doc[page_num]
        text = page.get_text()
        if text.strip():
            pages.append({
                "text": text,
                "source": os.path.basename(file_path),
                "page": page_num + 1
            })
    doc.close()
    return pages

def chunk_text(pages, chunk_size=512, overlap=50):
    """Split page text into chunks with overlap and metadata."""
    chunks = []
    chunk_index = 0
    for page in pages:
        text = page["text"]
        start = 0
        while start < len(text):
            end = start + chunk_size
            chunk = text[start:end]
            if chunk.strip():
                chunks.append({
                    "text": chunk,
                    "metadata": {
                        "source": page["source"],
                        "page": page["page"],
                        "chunk_index": chunk_index
                    }
                })
                chunk_index += 1
            start += chunk_size - overlap
    return chunks

def ingest_pdfs(data_dir="data"):
    """Load all PDFs from a directory and return chunked text."""
    all_chunks = []
    for filename in os.listdir(data_dir):
        if filename.endswith(".pdf"):
            file_path = os.path.join(data_dir, filename)
            pages = load_pdf(file_path)
            chunks = chunk_text(pages)
            all_chunks.extend(chunks)
    print(f"Ingested {len(all_chunks)} chunks from {data_dir}")
    return all_chunks

What does this code do?

This module handles the first stage of any RAG pipeline: getting documents into a searchable format.

• load_pdf() opens a PDF with PyMuPDF and extracts the text from each page, storing the source filename and page number as metadata.
• chunk_text() splits each page's text into 512-character pieces with a 50-character overlap. The overlap ensures no information is lost at chunk boundaries.
• ingest_pdfs() orchestrates both functions across all PDFs in a directory.

• Save src/ingest.py.
• Test that the ingestion works by running this command in your terminal:

python -c "from src.ingest import ingest_pdfs; chunks = ingest_pdfs('data'); print(f'First chunk: {chunks[0][\"text\"][:80]}...')"

You should see output showing the number of chunks ingested and a preview of the first chunk's text.

Seeing a ModuleNotFoundError?

• Make sure you created src/__init__.py (the empty file that tells Python this is a package).
• Confirm you are running the command from the rag-system/ directory, not from inside src/.
• Check that your virtual environment is active (you should see (venv) in your terminal prompt).

Help me debug my ModuleNotFoundError in the RAG pipeline ingestion step.

Generate embeddings and store them in ChromaDB

Chunks of text are not searchable on their own. To find which chunks are relevant to a question, you need to convert both chunks and queries into numerical vectors called embeddings. Similar meanings produce similar vectors, which makes matching possible.

• Create a new file called src/embeddings.py and paste the following code:

from sentence_transformers import SentenceTransformer
import chromadb

# Load the embedding model locally (384-dimensional vectors)
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

# Initialize ChromaDB with persistent storage
chroma_client = chromadb.PersistentClient(path="chroma_db")
collection = chroma_client.get_or_create_collection(name="documents")

def generate_embeddings(texts):
    """Generate 384-dim embeddings for a list of texts."""
    embeddings = embedding_model.encode(texts)
    return embeddings.tolist()

def store_chunks(chunks):
    """Store chunks with embeddings and metadata in ChromaDB."""
    texts = [chunk["text"] for chunk in chunks]
    metadatas = [chunk["metadata"] for chunk in chunks]
    ids = [f"chunk_{i}" for i in range(len(chunks))]
    embeddings = generate_embeddings(texts)

    # Upsert allows re-running without duplicate ID errors
    collection.upsert(
        documents=texts,
        embeddings=embeddings,
        metadatas=metadatas,
        ids=ids
    )
    print(f"Stored {len(chunks)} chunks in ChromaDB")

What does this code do?

• SentenceTransformer("all-MiniLM-L6-v2") loads a lightweight embedding model that runs locally on your CPU. It produces 384-dimensional vectors for any text input.
• chromadb.PersistentClient(path="chroma_db") creates a ChromaDB database that persists to disk, so your embeddings survive between script runs.
• store_chunks() embeds all chunks and upserts them into the database along with their metadata (source file, page number, chunk index).

• Save src/embeddings.py.

Next, you need a way to search those stored embeddings. The retriever takes a query, embeds it, and finds the most similar chunks.

• Create a new file called src/retriever.py and paste the following code:

from src.embeddings import embedding_model, collection

class BasicRetriever:
    """Retrieves top-k relevant chunks from ChromaDB using similarity search."""

    def __init__(self, top_k=5):
        self.top_k = top_k

    def retrieve(self, query):
        """Embed the query and search ChromaDB for similar chunks."""
        # Generate embedding for the query
        query_embedding = embedding_model.encode([query]).tolist()

        # Search ChromaDB for the most similar chunks
        results = collection.query(
            query_embeddings=query_embedding,
            n_results=self.top_k
        )

        # Format results with text and metadata
        documents = []
        for i in range(len(results["documents"][0])):
            documents.append({
                "text": results["documents"][0][i],
                "metadata": results["metadatas"][0][i]
            })
        return documents

What does this code do?

The BasicRetriever class is the search engine of your RAG pipeline. It embeds the user's query with the same model used for chunks, then queries ChromaDB for the 5 most similar vectors. Because both chunks and queries use the same embedding space, similar meanings land near each other.

• Save src/retriever.py.
• Now run the full ingestion and storage pipeline to populate your database:

python -c "from src.ingest import ingest_pdfs; from src.embeddings import store_chunks; chunks = ingest_pdfs('data'); store_chunks(chunks)"

You should see two lines of output: one showing how many chunks were ingested, and another confirming how many chunks were stored in ChromaDB.

Why does the first run take a moment?

The first time you run this, the sentence-transformers library downloads the all-MiniLM-L6-v2 model (~80MB). Subsequent runs load it from cache instantly.

Seeing a RuntimeError or slow performance?

• If you see a CUDA-related warning, that's fine. The model defaults to CPU when no GPU is available.
• If the command hangs, make sure your data/ folder contains PDF files. Empty directories produce zero chunks.
• If you see a chromadb error about duplicate IDs, delete the chroma_db/ folder and re-run.

Help me debug my embedding storage step in the RAG pipeline.

Build the query and generation chain

With documents indexed and searchable, the final piece connects a user's question to a generated answer. The pipeline retrieves relevant chunks, formats them into a prompt that grounds the LLM, and calls Groq to generate a context-aware response.

• Create a new file called src/pipeline.py and paste the following code:

import os
from dotenv import load_dotenv
from groq import Groq
from src.retriever import BasicRetriever

load_dotenv()

class BasicRAGPipeline:
    """Full retrieval chain: query -> retrieve -> format prompt -> call Groq -> answer."""

    def __init__(self):
        self.retriever = BasicRetriever(top_k=5)
        self.client = Groq(api_key=os.getenv("GROQ_API_KEY"))

    def format_prompt(self, question, context_chunks):
        """Build a prompt that grounds the LLM in retrieved context."""
        context = "\n\n".join(
            [f"Source: {chunk['metadata']['source']}, Page {chunk['metadata']['page']}\n{chunk['text']}"
             for chunk in context_chunks]
        )
        prompt = f"""Answer the following question based ONLY on the provided context.
If the context doesn't contain enough information, say so.

Context:
{context}

Question: {question}

Answer:"""
        return prompt

    def query(self, question):
        """Run the full RAG pipeline: retrieve, format, generate."""
        # Retrieve relevant chunks from ChromaDB
        chunks = self.retriever.retrieve(question)

        # Format the prompt with retrieved context
        prompt = self.format_prompt(question, chunks)

        # Call Groq LLM for generation
        response = self.client.chat.completions.create(
            model="llama-3.3-70b-versatile",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.1
        )
        return response.choices[0].message.content

What does this code do?

The BasicRAGPipeline class ties everything together into one call:

• format_prompt() injects retrieved chunks into the prompt with their source metadata, then explicitly instructs the LLM to only use that context.
• query() runs the full chain: embed the question, retrieve the top 5 chunks, format the grounded prompt, and call llama-3.3-70b-versatile on Groq.
• temperature=0.1 keeps responses factual and deterministic rather than creative.

• Save src/pipeline.py.
• Test the full pipeline by running this command (replace the question with something relevant to your PDFs):

python -c "
from src.pipeline import BasicRAGPipeline
pipeline = BasicRAGPipeline()
answer = pipeline.query('What methods are described in this paper?')
print(answer)
"

You should see a multi-sentence answer that directly references content from your PDF documents. The answer should mention specific details from the source material rather than generic information.

Try multiple questions

Test with 5-10 different questions that cover different parts of your PDFs. Try factual questions, definition questions, and questions about specific sections. This helps verify the retriever is pulling the right chunks for different types of queries.

Getting generic answers that don't reference your documents?

• Run the ingestion command from the previous substep again and confirm the chunk count is greater than zero.
• Check that your chroma_db/ folder exists and contains files. If it is empty, the storage step did not complete.
• Verify your .env file has a valid GROQ_API_KEY. An invalid key produces an authentication error.

Help me troubleshoot my basic RAG pipeline responses.

✔️ Awesome, I've got everything!

Great. Double check that you have saved all four files: src/ingest.py, src/embeddings.py, src/retriever.py, and src/pipeline.py.

ⓧ I'd like to double check the full code

Here is the full code for each file in your src/ directory.

import pymupdf as fitz  # PyMuPDF for reading PDF files
import os

def load_pdf(file_path):
    """Load a PDF file and extract text from each page."""
    doc = fitz.open(file_path)
    pages = []
    for page_num in range(len(doc)):
        page = doc[page_num]
        text = page.get_text()
        if text.strip():
            pages.append({
                "text": text,
                "source": os.path.basename(file_path),
                "page": page_num + 1
            })
    doc.close()
    return pages

def chunk_text(pages, chunk_size=512, overlap=50):
    """Split page text into chunks with overlap and metadata."""
    chunks = []
    chunk_index = 0
    for page in pages:
        text = page["text"]
        start = 0
        while start < len(text):
            end = start + chunk_size
            chunk = text[start:end]
            if chunk.strip():
                chunks.append({
                    "text": chunk,
                    "metadata": {
                        "source": page["source"],
                        "page": page["page"],
                        "chunk_index": chunk_index
                    }
                })
                chunk_index += 1
            start += chunk_size - overlap
    return chunks

def ingest_pdfs(data_dir="data"):
    """Load all PDFs from a directory and return chunked text."""
    all_chunks = []
    for filename in os.listdir(data_dir):
        if filename.endswith(".pdf"):
            file_path = os.path.join(data_dir, filename)
            pages = load_pdf(file_path)
            chunks = chunk_text(pages)
            all_chunks.extend(chunks)
    print(f"Ingested {len(all_chunks)} chunks from {data_dir}")
    return all_chunks

from sentence_transformers import SentenceTransformer
import chromadb

# Load the embedding model locally (384-dimensional vectors)
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

# Initialize ChromaDB with persistent storage
chroma_client = chromadb.PersistentClient(path="chroma_db")
collection = chroma_client.get_or_create_collection(name="documents")

def generate_embeddings(texts):
    """Generate 384-dim embeddings for a list of texts."""
    embeddings = embedding_model.encode(texts)
    return embeddings.tolist()

def store_chunks(chunks):
    """Store chunks with embeddings and metadata in ChromaDB."""
    texts = [chunk["text"] for chunk in chunks]
    metadatas = [chunk["metadata"] for chunk in chunks]
    ids = [f"chunk_{i}" for i in range(len(chunks))]
    embeddings = generate_embeddings(texts)

    # Upsert allows re-running without duplicate ID errors
    collection.upsert(
        documents=texts,
        embeddings=embeddings,
        metadatas=metadatas,
        ids=ids
    )
    print(f"Stored {len(chunks)} chunks in ChromaDB")

from src.embeddings import embedding_model, collection

class BasicRetriever:
    """Retrieves top-k relevant chunks from ChromaDB using similarity search."""

    def __init__(self, top_k=5):
        self.top_k = top_k

    def retrieve(self, query):
        """Embed the query and search ChromaDB for similar chunks."""
        # Generate embedding for the query
        query_embedding = embedding_model.encode([query]).tolist()

        # Search ChromaDB for the most similar chunks
        results = collection.query(
            query_embeddings=query_embedding,
            n_results=self.top_k
        )

        # Format results with text and metadata
        documents = []
        for i in range(len(results["documents"][0])):
            documents.append({
                "text": results["documents"][0][i],
                "metadata": results["metadatas"][0][i]
            })
        return documents

import os
from dotenv import load_dotenv
from groq import Groq
from src.retriever import BasicRetriever

load_dotenv()

class BasicRAGPipeline:
    """Full retrieval chain: query -> retrieve -> format prompt -> call Groq -> answer."""

    def __init__(self):
        self.retriever = BasicRetriever(top_k=5)
        self.client = Groq(api_key=os.getenv("GROQ_API_KEY"))

    def format_prompt(self, question, context_chunks):
        """Build a prompt that grounds the LLM in retrieved context."""
        context = "\n\n".join(
            [f"Source: {chunk['metadata']['source']}, Page {chunk['metadata']['page']}\n{chunk['text']}"
             for chunk in context_chunks]
        )
        prompt = f"""Answer the following question based ONLY on the provided context.
If the context doesn't contain enough information, say so.

Context:
{context}

Question: {question}

Answer:"""
        return prompt

    def query(self, question):
        """Run the full RAG pipeline: retrieve, format, generate."""
        # Retrieve relevant chunks from ChromaDB
        chunks = self.retriever.retrieve(question)

        # Format the prompt with retrieved context
        prompt = self.format_prompt(question, chunks)

        # Call Groq LLM for generation
        response = self.client.chat.completions.create(
            model="llama-3.3-70b-versatile",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.1
        )
        return response.choices[0].message.content

Your basic RAG pipeline is working. You can ask questions and get answers grounded in your actual documents. Next up, you will add hybrid search with BM25 keyword matching to catch queries that pure vector search misses.

Add Hybrid Search with BM25

Your basic RAG pipeline retrieves chunks using vector similarity alone. This works well for semantically similar questions, but fails when the user asks about specific technical terms, acronyms, or exact phrases that don't have close embedding neighbors.

In this step, you will add keyword-based BM25 search alongside your vector search, then combine both using Reciprocal Rank Fusion. This gives you the best of both worlds: semantic understanding from embeddings and exact keyword matching from BM25.

In this step, get ready to:

• Build a BM25 keyword retriever using rank-bm25.
• Combine BM25 and vector search with LangChain's EnsembleRetriever.
• Benchmark basic RAG vs hybrid RAG on a 20-question test set.

Implement BM25 keyword index

Vector search finds semantically similar chunks, but it can miss results that share exact keywords with the query. BM25 is a classic information retrieval algorithm that ranks documents by how well they match query keywords. It excels at finding exact terms, acronyms, and proper nouns.

• Create a new file called src/bm25_retriever.py and add the following code:

from rank_bm25 import BM25Okapi


class BM25Retriever:
    """Keyword-based retriever using BM25 scoring."""

    def __init__(self, chunks):
        # Store original chunks for retrieval
        self.chunks = chunks
        # Tokenize all chunk texts for BM25 indexing
        self.tokenized_corpus = self.tokenize_corpus(chunks)
        # Build the BM25 index over tokenized documents
        self.bm25 = BM25Okapi(self.tokenized_corpus)

    def tokenize_corpus(self, chunks):
        """Split each chunk's text into lowercase tokens."""
        return [chunk["text"].lower().split() for chunk in chunks]

    def retrieve(self, query, k=5):
        """Return top-k chunks ranked by BM25 keyword score."""
        # Tokenize the query the same way as the corpus
        tokenized_query = query.lower().split()
        # Get BM25 scores for all documents
        scores = self.bm25.get_scores(tokenized_query)
        # Get indices of top-k scoring documents
        top_indices = scores.argsort()[-k:][::-1]
        # Return chunks with their BM25 scores
        results = []
        for idx in top_indices:
            result = self.chunks[idx].copy()
            result["score"] = float(scores[idx])
            results.append(result)
        return results

What does this code do?

The BM25Retriever class builds a keyword index over all your document chunks using the rank-bm25 library.

• tokenize_corpus() splits each chunk's text into lowercase words. This is how BM25 understands the vocabulary of your documents.
• BM25Okapi builds a term-frequency index that knows which words appear in which documents and how often.
• retrieve() scores every document against the query keywords and returns the top-k matches sorted by relevance.

• Save src/bm25_retriever.py.
• Test the BM25 retriever by running this command:

python -c "from src.ingest import ingest_pdf; from src.bm25_retriever import BM25Retriever; chunks = ingest_pdf('data/'); r = BM25Retriever(chunks); print([c['text'][:50] for c in r.retrieve('transformer attention mechanism')])"

• You should see a list of chunk previews containing the exact keywords you searched for.

Getting an import error?

• Make sure your virtual environment is active. You should see (venv) in your terminal prompt.
• Verify rank-bm25 is installed. If not, run pip install rank-bm25 and try again.

Help me debug my BM25 retriever import error.

Combine with EnsembleRetriever

ChromaDB does not have built-in BM25 search. The standard pattern is to run BM25 and vector search separately, then merge results using Reciprocal Rank Fusion (RRF). RRF combines rankings from multiple sources by giving higher scores to documents that appear near the top of both lists.

• Create a new file called src/hybrid_retriever.py and add the following code:

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever as LangChainBM25
from langchain_core.documents import Document


class HybridRetriever:
    """Combines BM25 keyword search and ChromaDB vector search using Reciprocal Rank Fusion."""

    def __init__(self, chunks, chroma_collection, embedding_function):
        # Convert chunks to LangChain Document format for BM25
        documents = [
            Document(page_content=c["text"], metadata=c["metadata"])
            for c in chunks
        ]

        # Initialize LangChain BM25 retriever from documents
        self.bm25_retriever = LangChainBM25.from_documents(documents)
        self.bm25_retriever.k = 5

        # Initialize ChromaDB vector retriever via LangChain interface
        from langchain_community.vectorstores import Chroma
        vectorstore = Chroma(
            collection_name=chroma_collection.name,
            persist_directory="chroma_db",
            embedding_function=embedding_function
        )
        self.vector_retriever = vectorstore.as_retriever(
            search_kwargs={"k": 5}
        )

        # Combine with EnsembleRetriever using equal weights and RRF
        self.ensemble = EnsembleRetriever(
            retrievers=[self.bm25_retriever, self.vector_retriever],
            weights=[0.5, 0.5]
        )

    def retrieve(self, query, k=5):
        """Return top-k results merged with Reciprocal Rank Fusion."""
        # EnsembleRetriever handles RRF fusion internally
        results = self.ensemble.invoke(query)
        # Convert back to project format
        return [
            {"text": doc.page_content, "metadata": doc.metadata}
            for doc in results[:k]
        ]

What does this code do?

The HybridRetriever class wraps LangChain's EnsembleRetriever to combine two retrieval strategies.

• BM25 retriever (weight 0.5) finds chunks matching exact query keywords.
• ChromaDB vector retriever (weight 0.5) finds semantically similar chunks.
• EnsembleRetriever merges both result sets using Reciprocal Rank Fusion. Documents that rank highly in both lists get boosted to the top.

This means a query like "BERT architecture" will surface chunks that mention the exact word "BERT" (via BM25) AND chunks that discuss transformer architectures in general (via vector similarity).

• Save src/hybrid_retriever.py.

Why hybrid search instead of just vector search?

Vector search understands meaning but can miss exact terms. If your document mentions "BERT" and you search for "BERT", vector search might return chunks about transformers in general because the embeddings are semantically close.

BM25 finds exact keyword matches but misses paraphrases. Hybrid search gives you both: precise keyword hits AND semantically relevant results, combined with Reciprocal Rank Fusion.

Now update the pipeline to include a hybrid version.

• In src/pipeline.py, add the following import and class below the existing BasicRAGPipeline class:

from src.hybrid_retriever import HybridRetriever


class HybridRAGPipeline:
    """RAG pipeline using hybrid BM25 + vector retrieval."""

    def __init__(self, chunks, chroma_collection, embedding_function, llm_client):
        # Initialize hybrid retriever with both BM25 and vector search
        self.retriever = HybridRetriever(chunks, chroma_collection, embedding_function)
        self.llm_client = llm_client

    def query(self, question):
        """Retrieve relevant chunks with hybrid search, then generate answer."""
        # Get top-k chunks using hybrid retrieval
        retrieved = self.retriever.retrieve(question, k=5)

        # Format context from retrieved chunks
        context = "\n\n".join([chunk["text"] for chunk in retrieved])

        # Build grounded prompt
        prompt = f"""Answer the following question based ONLY on the provided context.
If the context does not contain enough information, say so.

Context:
{context}

Question: {question}

Answer:"""

        # Call Groq LLM
        response = self.llm_client.chat.completions.create(
            model="llama-3.3-70b-versatile",
            messages=[{"role": "user", "content": prompt}]
        )
        return {
            "answer": response.choices[0].message.content,
            "retrieved_chunks": retrieved
        }

What does this code do?

HybridRAGPipeline follows the same structure as BasicRAGPipeline but swaps in the HybridRetriever for retrieval. It uses the same prompt template and the same Groq LLM call to llama-3.3-70b-versatile. The only difference is how documents are retrieved.

• Save src/pipeline.py.

EnsembleRetriever not found?

• Make sure you import from the correct packages: from langchain.retrievers import EnsembleRetriever and from langchain_community.retrievers import BM25Retriever.
• Run pip install langchain langchain-community if you see import errors.

Help me fix my EnsembleRetriever import or configuration.

Benchmark basic vs hybrid

To measure whether hybrid search actually improves your RAG pipeline, you need a consistent test set. You will create 20 questions with known answers from your PDFs, then run both pipelines on the same questions.

• Create the evaluation test set by adding a new file evals/test_set.json with 20 questions derived from your sample PDFs:

[
  {
    "question": "What is the main contribution of the paper?",
    "reference_answer": "The paper introduces...",
    "source_document": "paper1.pdf",
    "keywords": ["contribution", "introduce"]
  },
  {
    "question": "What accuracy did the BERT model achieve on the benchmark?",
    "reference_answer": "BERT achieved...",
    "source_document": "paper1.pdf",
    "keywords": ["BERT", "accuracy", "benchmark"]
  }
]

Tailor questions to your PDFs

Replace the example questions above with 20 real questions based on the PDF documents you ingested in Step 2. Include a mix of general semantic questions (where vector search shines), specific keyword queries with acronyms (where BM25 shines), and questions requiring both.

• Create a benchmark script to run both pipelines on your test set. Add the following code in a new file called evals/run_benchmark.py:

import json
import os
from dotenv import load_dotenv
from groq import Groq
from src.ingest import ingest_pdf
from src.embeddings import get_collection, get_embedding_function
from src.retriever import BasicRetriever
from src.pipeline import BasicRAGPipeline, HybridRAGPipeline

load_dotenv()

# Initialize shared resources
chunks = ingest_pdf("data/")
collection = get_collection()
embedding_function = get_embedding_function()
client = Groq(api_key=os.getenv("GROQ_API_KEY"))

# Load test set
with open("evals/test_set.json", "r") as f:
    test_set = json.load(f)

# Run basic RAG pipeline
print("Running BasicRAGPipeline...")
basic_pipeline = BasicRAGPipeline(collection, embedding_function, client)
basic_results = []
for item in test_set:
    result = basic_pipeline.query(item["question"])
    basic_results.append({
        "question": item["question"],
        "answer": result["answer"],
        "retrieved_chunks": [c["text"][:200] for c in result["retrieved_chunks"]]
    })
    print(f"  Done: {item['question'][:50]}...")

# Save basic results
with open("evals/results_basic.json", "w") as f:
    json.dump(basic_results, f, indent=2)
print("Saved evals/results_basic.json")

# Run hybrid RAG pipeline
print("\nRunning HybridRAGPipeline...")
hybrid_pipeline = HybridRAGPipeline(chunks, collection, embedding_function, client)
hybrid_results = []
for item in test_set:
    result = hybrid_pipeline.query(item["question"])
    hybrid_results.append({
        "question": item["question"],
        "answer": result["answer"],
        "retrieved_chunks": [c["text"][:200] for c in result["retrieved_chunks"]]
    })
    print(f"  Done: {item['question'][:50]}...")

# Save hybrid results
with open("evals/results_hybrid.json", "w") as f:
    json.dump(hybrid_results, f, indent=2)
print("Saved evals/results_hybrid.json")

print("\nBenchmark complete! Results saved to evals/ directory.")

What does this script do?

• Loads your 20-question test set from evals/test_set.json.
• Runs every question through BasicRAGPipeline (vector-only retrieval) and saves results to evals/results_basic.json.
• Runs every question through HybridRAGPipeline (BM25 + vector with RRF) and saves results to evals/results_hybrid.json.
• Both result files contain the generated answer and the first 200 characters of each retrieved chunk for comparison in the next step.

• Save evals/run_benchmark.py.
• Run the benchmark by executing this command:

python evals/run_benchmark.py

• You should see progress output for each question, followed by confirmation that both result files were saved to the evals/ directory.

Seeing rate limit errors from Groq?

• The Groq free tier has rate limits. If you hit a 429 error, add a time.sleep(2) between queries in the benchmark loop to space out requests.
• Make sure your .env file contains a valid GROQ_API_KEY.

Help me fix Groq rate limit errors in my benchmark script.

• Open evals/results_hybrid.json and compare retrieved chunks with evals/results_basic.json. Look for questions with acronyms or exact terms where hybrid retrieval found more relevant chunks than basic vector search.

✔️ Awesome, I've got everything!

Great. Double check you have saved all three files: src/bm25_retriever.py, src/hybrid_retriever.py, and the updated src/pipeline.py.

ⓧ I'd like to double check the full code

Here is the full code for each file created or updated in this step.

from rank_bm25 import BM25Okapi


class BM25Retriever:
    """Keyword-based retriever using BM25 scoring."""

    def __init__(self, chunks):
        # Store original chunks for retrieval
        self.chunks = chunks
        # Tokenize all chunk texts for BM25 indexing
        self.tokenized_corpus = self.tokenize_corpus(chunks)
        # Build the BM25 index over tokenized documents
        self.bm25 = BM25Okapi(self.tokenized_corpus)

    def tokenize_corpus(self, chunks):
        """Split each chunk's text into lowercase tokens."""
        return [chunk["text"].lower().split() for chunk in chunks]

    def retrieve(self, query, k=5):
        """Return top-k chunks ranked by BM25 keyword score."""
        # Tokenize the query the same way as the corpus
        tokenized_query = query.lower().split()
        # Get BM25 scores for all documents
        scores = self.bm25.get_scores(tokenized_query)
        # Get indices of top-k scoring documents
        top_indices = scores.argsort()[-k:][::-1]
        # Return chunks with their BM25 scores
        results = []
        for idx in top_indices:
            result = self.chunks[idx].copy()
            result["score"] = float(scores[idx])
            results.append(result)
        return results

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever as LangChainBM25
from langchain_core.documents import Document


class HybridRetriever:
    """Combines BM25 keyword search and ChromaDB vector search using Reciprocal Rank Fusion."""

    def __init__(self, chunks, chroma_collection, embedding_function):
        # Convert chunks to LangChain Document format for BM25
        documents = [
            Document(page_content=c["text"], metadata=c["metadata"])
            for c in chunks
        ]

        # Initialize LangChain BM25 retriever from documents
        self.bm25_retriever = LangChainBM25.from_documents(documents)
        self.bm25_retriever.k = 5

        # Initialize ChromaDB vector retriever via LangChain interface
        from langchain_community.vectorstores import Chroma
        vectorstore = Chroma(
            collection_name=chroma_collection.name,
            persist_directory="chroma_db",
            embedding_function=embedding_function
        )
        self.vector_retriever = vectorstore.as_retriever(
            search_kwargs={"k": 5}
        )

        # Combine with EnsembleRetriever using equal weights and RRF
        self.ensemble = EnsembleRetriever(
            retrievers=[self.bm25_retriever, self.vector_retriever],
            weights=[0.5, 0.5]
        )

    def retrieve(self, query, k=5):
        """Return top-k results merged with Reciprocal Rank Fusion."""
        # EnsembleRetriever handles RRF fusion internally
        results = self.ensemble.invoke(query)
        # Convert back to project format
        return [
            {"text": doc.page_content, "metadata": doc.metadata}
            for doc in results[:k]
        ]

from src.hybrid_retriever import HybridRetriever


class HybridRAGPipeline:
    """RAG pipeline using hybrid BM25 + vector retrieval."""

    def __init__(self, chunks, chroma_collection, embedding_function, llm_client):
        # Initialize hybrid retriever with both BM25 and vector search
        self.retriever = HybridRetriever(chunks, chroma_collection, embedding_function)
        self.llm_client = llm_client

    def query(self, question):
        """Retrieve relevant chunks with hybrid search, then generate answer."""
        # Get top-k chunks using hybrid retrieval
        retrieved = self.retriever.retrieve(question, k=5)

        # Format context from retrieved chunks
        context = "\n\n".join([chunk["text"] for chunk in retrieved])

        # Build grounded prompt
        prompt = f"""Answer the following question based ONLY on the provided context.
If the context does not contain enough information, say so.

Context:
{context}

Question: {question}

Answer:"""

        # Call Groq LLM
        response = self.llm_client.chat.completions.create(
            model="llama-3.3-70b-versatile",
            messages=[{"role": "user", "content": prompt}]
        )
        return {
            "answer": response.choices[0].message.content,
            "retrieved_chunks": retrieved
        }

Your retrieval system now combines the best of both worlds. Next up, you will add a cross-encoder reranking layer that scores each retrieved chunk against the query for even more precise ordering.

Implement Cross-Encoder Reranking

Your hybrid search pipeline combines keyword and vector results, but the ordering is still based on raw retrieval scores. A document might rank highly because of one keyword match even though its overall meaning is only loosely related to the question.

A cross-encoder fixes this by scoring each retrieved document directly against the original query as a pair. In this step, you will load a cross-encoder model, build a reranking function, and benchmark the improvement over plain hybrid search.

In this step, get ready to:

• Load a cross-encoder reranking model that runs on CPU.
• Build a reranking pipeline that over-retrieves then re-sorts by relevance.
• Run the 20-question test set through the reranked pipeline and save results.

Load the cross-encoder model

A bi-encoder (like the embedding model you already use) encodes the query and documents independently, then compares embeddings with cosine similarity. A cross-encoder is different.

It processes the query and a document together as a single input pair, producing a much more accurate relevance score. The tradeoff is speed: cross-encoders are too slow to score thousands of documents, so you use them only on a small candidate set that your fast retriever already found.

• Create a new file src/reranker.py by running:

touch src/reranker.py

• Confirm the file exists by running:

ls src/reranker.py

You should see src/reranker.py printed in the output.

• Open src/reranker.py in VS Code and add the following code:

from sentence_transformers import CrossEncoder


class CrossEncoderReranker:
    """Reranks retrieved documents using a cross-encoder model."""

    def __init__(self, model_name="cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512):
        # Load the cross-encoder model for pairwise relevance scoring
        self.model = CrossEncoder(model_name, max_length=max_length)

    def rerank(self, query, docs, top_n=5):
        """Score each query-document pair and return the top_n most relevant."""
        # Create query-document pairs for scoring
        pairs = [(query, doc["content"]) for doc in docs]

        # Score all pairs with the cross-encoder
        scores = self.model.predict(pairs)

        # Attach scores to documents
        for i, doc in enumerate(docs):
            doc["rerank_score"] = float(scores[i])

        # Sort by score descending and return top_n
        ranked_docs = sorted(docs, key=lambda x: x["rerank_score"], reverse=True)
        return ranked_docs[:top_n]

What does this code do?

The CrossEncoderReranker class wraps the cross-encoder/ms-marco-MiniLM-L-6-v2 model. This model was trained on the MS MARCO passage ranking dataset, so it understands query-to-document relevance.

The rerank() method takes a query and a list of document dicts, forms (query, content) pairs, scores them all at once, then returns the top N documents sorted by their cross-encoder score.

• Save src/reranker.py.
• Verify the model loads correctly by running this quick test in your terminal:

python -c "from src.reranker import CrossEncoderReranker; r = CrossEncoderReranker(); print('Reranker loaded successfully')"

You should see Reranker loaded successfully printed after the model downloads. The first run may take a moment as it fetches model weights.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is active (you should see (venv) in your terminal prompt). If sentence-transformers is not installed, run pip install sentence-transformers again.

Help me debug a ModuleNotFoundError when importing CrossEncoder

Build the reranking pipeline

Now you will wire the reranker into a full RAG pipeline. The key pattern is over-retrieval: fetch more candidates than you need (top-20), then let the cross-encoder narrow them down to the best 5.

• Open src/pipeline.py and add the following import near the top of the file, alongside your existing imports:

from src.reranker import CrossEncoderReranker

• Add the RerankedRAGPipeline class at the bottom of src/pipeline.py, below the existing HybridRAGPipeline class:

class RerankedRAGPipeline:
    """RAG pipeline with hybrid search and cross-encoder reranking."""

    def __init__(self):
        # Over-retrieve with hybrid search, then rerank to top-5
        self.retriever = HybridRetriever()
        self.reranker = CrossEncoderReranker()
        self.llm_client = Groq(api_key=os.getenv("GROQ_API_KEY"))

    def query(self, question):
        """Retrieve top-20, rerank to top-5, then generate answer."""
        # Over-retrieve: get 20 candidates from hybrid search
        candidates = self.retriever.retrieve(question, k=20)

        # Rerank: score each candidate against the query, keep top-5
        top_docs = self.reranker.rerank(question, candidates, top_n=5)

        # Format context from reranked documents
        context = "\n\n".join([doc["content"] for doc in top_docs])

        # Generate answer using Groq LLM
        prompt = PROMPT_TEMPLATE.format(context=context, question=question)
        response = self.llm_client.chat.completions.create(
            model="llama-3.3-70b-versatile",
            messages=[{"role": "user", "content": prompt}]
        )

        return {
            "answer": response.choices[0].message.content,
            "sources": top_docs
        }

Why over-retrieve then rerank?

Hybrid search is fast but imprecise in ordering. By fetching 20 candidates instead of 5, you give the cross-encoder a larger pool to work with.

The cross-encoder then applies its slower, more accurate pairwise scoring to pick the truly best 5. This two-stage approach (fast retrieval followed by precise reranking) is the standard pattern used in production search systems.

• Save src/pipeline.py.
• Verify the new pipeline initializes correctly by running:

python -c "from src.pipeline import RerankedRAGPipeline; p = RerankedRAGPipeline(); print('RerankedRAGPipeline ready')"

You should see RerankedRAGPipeline ready printed after both the hybrid retriever and cross-encoder model load.

Seeing an ImportError for CrossEncoderReranker?

Double-check that src/reranker.py is saved and that your import path matches: from src.reranker import CrossEncoderReranker. Run the command from your rag-system/ project root so Python can find the src package.

Help me fix an ImportError when loading RerankedRAGPipeline

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/reranker.py and src/pipeline.py.

ⓧ I'd like to double check the full code

Here is the complete src/reranker.py:

from sentence_transformers import CrossEncoder


class CrossEncoderReranker:
    """Reranks retrieved documents using a cross-encoder model."""

    def __init__(self, model_name="cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512):
        # Load the cross-encoder model for pairwise relevance scoring
        self.model = CrossEncoder(model_name, max_length=max_length)

    def rerank(self, query, docs, top_n=5):
        """Score each query-document pair and return the top_n most relevant."""
        # Create query-document pairs for scoring
        pairs = [(query, doc["content"]) for doc in docs]

        # Score all pairs with the cross-encoder
        scores = self.model.predict(pairs)

        # Attach scores to documents
        for i, doc in enumerate(docs):
            doc["rerank_score"] = float(scores[i])

        # Sort by score descending and return top_n
        ranked_docs = sorted(docs, key=lambda x: x["rerank_score"], reverse=True)
        return ranked_docs[:top_n]

Here is the new import and class to add to src/pipeline.py (existing classes remain unchanged):

from src.reranker import CrossEncoderReranker


class RerankedRAGPipeline:
    """RAG pipeline with hybrid search and cross-encoder reranking."""

    def __init__(self):
        # Over-retrieve with hybrid search, then rerank to top-5
        self.retriever = HybridRetriever()
        self.reranker = CrossEncoderReranker()
        self.llm_client = Groq(api_key=os.getenv("GROQ_API_KEY"))

    def query(self, question):
        """Retrieve top-20, rerank to top-5, then generate answer."""
        # Over-retrieve: get 20 candidates from hybrid search
        candidates = self.retriever.retrieve(question, k=20)

        # Rerank: score each candidate against the query, keep top-5
        top_docs = self.reranker.rerank(question, candidates, top_n=5)

        # Format context from reranked documents
        context = "\n\n".join([doc["content"] for doc in top_docs])

        # Generate answer using Groq LLM
        prompt = PROMPT_TEMPLATE.format(context=context, question=question)
        response = self.llm_client.chat.completions.create(
            model="llama-3.3-70b-versatile",
            messages=[{"role": "user", "content": prompt}]
        )

        return {
            "answer": response.choices[0].message.content,
            "sources": top_docs
        }

Benchmark hybrid vs hybrid + reranked

You have the same 20-question test set from the previous step. Now you will run all 20 questions through the reranked pipeline and save the results for evaluation later.

• Create a benchmark script by running:

touch evals/benchmark_reranked.py

• Open evals/benchmark_reranked.py and add the following code:

import json
from src.pipeline import RerankedRAGPipeline

# Load the 20-question test set
with open("evals/test_set.json", "r") as f:
    test_set = json.load(f)

# Initialize the reranked pipeline
pipeline = RerankedRAGPipeline()

# Run all 20 questions through the reranked pipeline
results = []
for item in test_set:
    question = item["question"]
    result = pipeline.query(question)
    results.append({
        "question": question,
        "answer": result["answer"],
        "sources": result["sources"],
        "reference": item["reference"]
    })
    print(f"Processed: {question[:50]}...")

# Save results to file
with open("evals/results_reranked.json", "w") as f:
    json.dump(results, f, indent=2)

print(f"\nCompleted {len(results)} questions. Results saved to evals/results_reranked.json")

What does this script do?

This script loads the same 20 questions you used for the basic and hybrid benchmarks. It runs each question through RerankedRAGPipeline, which over-retrieves 20 candidates from hybrid search, reranks them with the cross-encoder, then generates an answer from the top 5.

The results are saved to evals/results_reranked.json so you can compare all three pipelines side-by-side in the evaluation step.

• Save evals/benchmark_reranked.py.
• Run the benchmark by executing:

python evals/benchmark_reranked.py

You should see each question being processed with a progress message, ending with a confirmation that results were saved to evals/results_reranked.json.

Script fails with a rate limit error?

The Groq free tier has rate limits on requests per minute. If you hit a 429 error, wait 60 seconds and re-run the script. You can also add a time.sleep(2) between iterations in the for loop to slow down requests.

Help me handle Groq rate limit errors in my benchmark script

• Confirm the results file was created by running:

ls evals/results_reranked.json

You should see the file path printed, confirming the benchmark completed successfully.

You now have three sets of results ready for comparison: basic, hybrid, and hybrid + reranked. Next up, you will add citation grounding so every claim in the generated answer links back to a specific source passage.

Add Citation Grounding

Your RAG pipeline retrieves relevant chunks and generates answers. But how do you know the answer is actually grounded in those sources?

In this step, you will build a citation system that links every factual claim in the LLM's response back to a specific source passage. After generation, you will verify each citation using the LLM itself, producing a measurable "citation accuracy" score that directly quantifies hallucination at the claim level.

In this step, get ready to:

• Build a structured citation extraction and verification module.
• Create a cited RAG pipeline that formats sources with numbered markers and instructs the LLM to cite them inline.
• Compute a citation accuracy score measuring what percentage of citations are verified as grounded.

Build the citation data models and extraction logic

Before you can verify citations, you need to define the data structures that represent them and a function that parses inline markers from the LLM's output.

• Create a new file src/citations.py by running:

touch src/citations.py

• Open src/citations.py in VS Code and add the following imports and Pydantic models:

import re
import os
from pydantic import BaseModel
from groq import Groq


# Data model for a single citation
class Citation(BaseModel):
    """Links a claim to its source passage with verification status."""
    source_doc: str
    page_number: int
    passage: str
    verified: bool = False


# Structured response containing the answer and all citations
class CitationResponse(BaseModel):
    """Complete response with answer text and verified citation list."""
    answer: str
    citations: list[Citation]

What does this code do?

The Citation model stores everything needed to trace a claim back to its source: which document it came from, which page, the exact passage quoted, and whether the citation passed verification.

The CitationResponse model wraps the full LLM answer together with its list of citations. This gives you a clean, structured output that your API can serve directly.

• Below the models in the same file, add the extract_citations function:

def extract_citations(text: str) -> list[tuple[str, int]]:
    """Parse inline [N] markers from LLM output.

    Returns list of (claim_sentence, citation_index) tuples.
    """
    results = []
    # Split text into sentences at period/question/exclamation boundaries
    sentences = re.split(r'(?<=[.!?])\s+', text)

    for sentence in sentences:
        # Find all [N] markers in this sentence
        markers = re.findall(r'\[(\d+)\]', sentence)
        for marker in markers:
            # Remove citation markers to get the clean claim text
            claim = re.sub(r'\[\d+\]', '', sentence).strip()
            results.append((claim, int(marker)))

    return results

How does citation extraction work?

The function splits the LLM's response into sentences, then searches each sentence for bracketed numbers like [1] or [2]. For every marker found, it pairs the clean claim text (with markers stripped) with the source index. This gives you a list of every claim that references a specific source.

• Save src/citations.py.

Seeing a syntax error?

Make sure you have the closing parenthesis on the re.split line. The regex pattern uses a lookbehind (?<=[.!?]) which requires the backslash to be properly escaped inside the raw string.

help me fix a syntax error in my extract_citations function

Add citation verification logic

Extracting citations is only half the picture. You need to verify that each cited source actually supports the claim. This is where you catch hallucinations at the individual claim level.

• In src/citations.py, add the verify_citation and compute_citation_accuracy functions below extract_citations:

def verify_citation(claim: str, passage: str, llm: Groq) -> bool:
    """Check if the cited passage supports the claim using Groq LLM."""
    # Build an entailment prompt asking if the source backs the claim
    verification_prompt = (
        "Determine if the following source passage supports the claim.\n\n"
        f"Claim: {claim}\n\n"
        f"Source passage: {passage}\n\n"
        "Does the source passage provide evidence that supports this claim?\n"
        "Answer with ONLY 'yes' or 'no'."
    )

    response = llm.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": verification_prompt}],
        temperature=0
    )

    answer = response.choices[0].message.content.strip().lower()
    return answer == "yes"


def compute_citation_accuracy(citations: list[Citation]) -> float:
    """Return percentage of citations verified as grounded in source text."""
    if not citations:
        return 0.0
    verified_count = sum(1 for c in citations if c.verified)
    return verified_count / len(citations)

What does verify_citation do?

For each claim-source pair, the function asks the LLM a simple yes/no question: does this passage support this claim? A "yes" means the citation is grounded. A "no" flags a potential hallucination.

The compute_citation_accuracy function calculates the percentage of citations that passed verification. This single number tells you how often your RAG system hallucinates at the claim level.

• Save src/citations.py.

Getting a Groq API error?

Ensure your .env file is loaded before calling verify_citation. The function expects an initialized Groq client passed as the llm parameter.

help me debug my Groq API connection in the citation verification function

✔️ Awesome, I've got everything!

Great. Double check you have saved src/citations.py with all four components: Citation, CitationResponse, extract_citations, verify_citation, and compute_citation_accuracy.

ⓧ I'd like to double check the full code

import re
import os
from pydantic import BaseModel
from groq import Groq


# Data model for a single citation
class Citation(BaseModel):
    """Links a claim to its source passage with verification status."""
    source_doc: str
    page_number: int
    passage: str
    verified: bool = False


# Structured response containing the answer and all citations
class CitationResponse(BaseModel):
    """Complete response with answer text and verified citation list."""
    answer: str
    citations: list[Citation]


def extract_citations(text: str) -> list[tuple[str, int]]:
    """Parse inline [N] markers from LLM output.

    Returns list of (claim_sentence, citation_index) tuples.
    """
    results = []
    # Split text into sentences at period/question/exclamation boundaries
    sentences = re.split(r'(?<=[.!?])\s+', text)

    for sentence in sentences:
        # Find all [N] markers in this sentence
        markers = re.findall(r'\[(\d+)\]', sentence)
        for marker in markers:
            # Remove citation markers to get the clean claim text
            claim = re.sub(r'\[\d+\]', '', sentence).strip()
            results.append((claim, int(marker)))

    return results


def verify_citation(claim: str, passage: str, llm: Groq) -> bool:
    """Check if the cited passage supports the claim using Groq LLM."""
    # Build an entailment prompt asking if the source backs the claim
    verification_prompt = (
        "Determine if the following source passage supports the claim.\n\n"
        f"Claim: {claim}\n\n"
        f"Source passage: {passage}\n\n"
        "Does the source passage provide evidence that supports this claim?\n"
        "Answer with ONLY 'yes' or 'no'."
    )

    response = llm.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": verification_prompt}],
        temperature=0
    )

    answer = response.choices[0].message.content.strip().lower()
    return answer == "yes"


def compute_citation_accuracy(citations: list[Citation]) -> float:
    """Return percentage of citations verified as grounded in source text."""
    if not citations:
        return 0.0
    verified_count = sum(1 for c in citations if c.verified)
    return verified_count / len(citations)

Build the cited RAG pipeline

Now you will tie everything together. The CitedRAGPipeline formats retrieved chunks as numbered sources, instructs the LLM to use inline citation markers, and runs post-generation verification on every citation.

• Open src/pipeline.py and add the following import at the top of the file:

from src.citations import (
    Citation, CitationResponse, extract_citations,
    verify_citation, compute_citation_accuracy
)

• Below your existing RerankedRAGPipeline class in src/pipeline.py, add the CitedRAGPipeline class:

class CitedRAGPipeline:
    """RAG pipeline with inline citation grounding and verification."""

    def __init__(self, reranker, hybrid_retriever):
        self.reranker = reranker
        self.hybrid_retriever = hybrid_retriever
        self.llm = Groq(api_key=os.getenv("GROQ_API_KEY"))

    def format_sources(self, chunks: list[dict]) -> str:
        """Format retrieved chunks as numbered sources for the prompt."""
        sources = []
        for i, chunk in enumerate(chunks, 1):
            sources.append(f'Source [{i}]: "{chunk["text"]}"')
        return "\n\n".join(sources)

    def query(self, question: str) -> CitationResponse:
        """Run the full cited RAG pipeline with verification."""
        # Retrieve and rerank documents
        raw_results = self.hybrid_retriever.retrieve(question, k=20)
        reranked = self.reranker.rerank(question, raw_results, top_n=5)

        # Format sources with numbered markers
        sources_text = self.format_sources(reranked)

        # Instruct the LLM to cite sources inline
        prompt = (
            "Answer the question using ONLY the provided sources.\n"
            "For every factual claim, add an inline citation marker [N] "
            "referencing the source number.\n\n"
            f"Sources:\n{sources_text}\n\n"
            f"Question: {question}\n\n"
            "Answer with inline citations:"
        )

        response = self.llm.chat.completions.create(
            model="llama-3.3-70b-versatile",
            messages=[{"role": "user", "content": prompt}],
            temperature=0
        )

        answer_text = response.choices[0].message.content

        # Extract citations and verify each one
        extracted = extract_citations(answer_text)
        citations = []

        for claim, idx in extracted:
            if 1 <= idx <= len(reranked):
                chunk = reranked[idx - 1]
                # Verify the source actually supports the claim
                is_verified = verify_citation(
                    claim, chunk["text"], self.llm
                )
                citations.append(Citation(
                    source_doc=chunk["metadata"]["source"],
                    page_number=chunk["metadata"]["page_number"],
                    passage=chunk["text"],
                    verified=is_verified
                ))

        return CitationResponse(answer=answer_text, citations=citations)

What does CitedRAGPipeline do?

The pipeline works in four stages. First, it retrieves and reranks documents using your existing hybrid search and cross-encoder. Second, it formats those documents as numbered sources (Source [1], Source [2], etc.) in the prompt context.

Third, the LLM generates an answer with inline [N] markers for every factual claim. Fourth, the pipeline extracts each citation, looks up the referenced source, and asks the LLM whether that source actually supports the claim. The result is a structured CitationResponse with every citation marked as verified or not.

• Save src/pipeline.py.
• Test the pipeline by running this in your terminal:

# test_citations.py - Run from the rag-system/ directory
import os
from dotenv import load_dotenv
load_dotenv()

from src.hybrid_retriever import HybridRetriever
from src.reranker import CrossEncoderReranker
from src.pipeline import CitedRAGPipeline
from src.citations import compute_citation_accuracy

# Initialize components
hybrid_retriever = HybridRetriever()
reranker = CrossEncoderReranker()
pipeline = CitedRAGPipeline(reranker=reranker, hybrid_retriever=hybrid_retriever)

# Ask a question
result = pipeline.query("What method did the authors use for evaluation?")

# Print the answer with citations
print("Answer:", result.answer)
print(f"\nTotal citations: {len(result.citations)}")
for i, c in enumerate(result.citations, 1):
    status = "VERIFIED" if c.verified else "UNVERIFIED"
    print(f"  [{i}] {status} - {c.source_doc}, p.{c.page_number}")

# Compute citation accuracy
accuracy = compute_citation_accuracy(result.citations)
print(f"\nCitation accuracy: {accuracy:.0%}")

• Run the test script:

python test_citations.py

You should see the LLM's answer with inline markers, followed by a breakdown of each citation's verification status, and a final citation accuracy percentage.

Getting an import error?

Make sure you added the import statement for src.citations at the top of src/pipeline.py. Also confirm you are running the script from the rag-system/ directory so Python can resolve the src package.

help me fix my import error when running the cited RAG pipeline

Why is citation accuracy a powerful resume metric?

Most RAG systems generate answers without any verification. By measuring citation accuracy (the percentage of claims that are provably grounded in source text), you are quantifying hallucination reduction. A number like "98% citation accuracy" on your resume tells an interviewer you understand production-grade AI trustworthiness, not just basic Q&A.

Your pipeline now produces verifiable, citation-grounded answers. Next up, you will run systematic evaluation across all your pipeline variants using RAGAS to produce the benchmark numbers for your resume.

Evaluate All Pipelines with RAGAS

You have four sets of pipeline results sitting in your evals/ folder. Each one answers the same 20 questions using a different retrieval strategy.

The question is: which strategy actually performs better, and by how much? RAGAS gives you objective, quantified scores so you can answer that with real numbers instead of gut feelings.

In this step, get ready to:

• Set up RAGAS evaluation with your Groq LLM as the evaluator.
• Run Faithfulness, Context Precision, and Response Relevancy metrics on all four pipelines.
• Generate a benchmark report with improvement percentages and latency comparison.

Set up the RAGAS evaluation script

RAGAS uses an LLM as a judge to score your pipeline outputs on multiple dimensions. You need to wrap your Groq LLM in a LangChain wrapper so RAGAS can use it for scoring.

• Create a new file called evals/evaluate.py by adding the following imports and LLM configuration:

import json
import time
import os
import pandas as pd
from dotenv import load_dotenv

# RAGAS imports for evaluation
from ragas import SingleTurnSample, EvaluationDataset, evaluate
from ragas.metrics import Faithfulness, ResponseRelevancy, LLMContextPrecisionWithoutReference
from ragas.llms import LangchainLLMWrapper

# LangChain Groq wrapper for the evaluator LLM
from langchain_groq import ChatGroq

# Load environment variables
load_dotenv()

# Configure the evaluator LLM using Groq via LangChain
groq_llm = ChatGroq(
    model="llama-3.3-70b-versatile",
    api_key=os.getenv("GROQ_API_KEY"),
    temperature=0
)

# Wrap for RAGAS compatibility
evaluator_llm = LangchainLLMWrapper(groq_llm)

What does this code do?

RAGAS needs an LLM to act as a judge. It reads your pipeline's answers and retrieved contexts, then uses the judge LLM to score how faithful, precise, and relevant the outputs are.

The LangchainLLMWrapper bridges your Groq-powered LangChain LLM into the format RAGAS expects. This means your evaluation runs through Groq's free tier, keeping cost at zero.

Build the evaluation dataset and run metrics on all pipelines

RAGAS evaluates using SingleTurnSample objects that pair each question with the pipeline's response and retrieved contexts. You will load all four result files and score them against your test set.

• Add the following code below your LLM configuration in evals/evaluate.py to load results and run the evaluation:

# Load the test set and pipeline results
with open("evals/test_set.json", "r") as f:
    test_set = json.load(f)

with open("evals/results_basic.json", "r") as f:
    results_basic = json.load(f)

with open("evals/results_hybrid.json", "r") as f:
    results_hybrid = json.load(f)

with open("evals/results_reranked.json", "r") as f:
    results_reranked = json.load(f)

with open("evals/results_cited.json", "r") as f:
    results_cited = json.load(f)


def build_eval_dataset(test_set, results):
    """Convert test set + results into a RAGAS EvaluationDataset."""
    samples = []
    for item, result in zip(test_set, results):
        sample = SingleTurnSample(
            user_input=item["question"],
            response=result["answer"],
            retrieved_contexts=result["retrieved_contexts"],
            reference=item["reference_answer"]
        )
        samples.append(sample)
    return EvaluationDataset(samples=samples)


def evaluate_pipeline(name, test_set, results):
    """Run RAGAS metrics on a single pipeline's results."""
    print(f"\nEvaluating: {name}...")
    dataset = build_eval_dataset(test_set, results)

    # Define the three core RAG metrics
    metrics = [
        Faithfulness(),
        LLMContextPrecisionWithoutReference(),
        ResponseRelevancy()
    ]

    # Time the evaluation
    start_time = time.time()
    result = evaluate(
        dataset=dataset,
        metrics=metrics,
        llm=evaluator_llm
    )
    eval_time = time.time() - start_time

    scores = {
        "pipeline": name,
        "faithfulness": result["faithfulness"],
        "context_precision": result["llm_context_precision_without_reference"],
        "answer_relevancy": result["answer_relevancy"],
        "eval_time_seconds": round(eval_time, 2)
    }
    return scores


# Run evaluation on all four pipelines
all_scores = []
all_scores.append(evaluate_pipeline("Basic RAG", test_set, results_basic))
all_scores.append(evaluate_pipeline("Hybrid Search", test_set, results_hybrid))
all_scores.append(evaluate_pipeline("Hybrid + Reranking", test_set, results_reranked))
all_scores.append(evaluate_pipeline("Cited RAG", test_set, results_cited))

# Append citation accuracy for the cited pipeline
if "citation_accuracy" in results_cited[0]:
    all_scores[3]["citation_accuracy"] = sum(
        r["citation_accuracy"] for r in results_cited
    ) / len(results_cited)

print("\n" + "=" * 60)
print("EVALUATION COMPLETE")
print("=" * 60)

What does this code do?

The build_eval_dataset function converts your JSON results into SingleTurnSample objects that RAGAS can score. Each sample includes the question, the pipeline's answer, the chunks it retrieved, and the reference answer.

The evaluate_pipeline function runs three RAGAS metrics: Faithfulness (are claims grounded in context?), LLMContextPrecisionWithoutReference (are retrieved chunks relevant?), and ResponseRelevancy (does the answer address the question?).

For the Cited RAG pipeline, the script also appends the custom citation accuracy metric that you built in the previous step.

Generate a benchmark report and print results

The final piece collects all scores into a pandas DataFrame, calculates improvement percentages over Basic RAG, and saves a formatted markdown report.

• Add this code at the bottom of evals/evaluate.py to generate the report:

# Build DataFrame from all scores
df = pd.DataFrame(all_scores)
df = df.set_index("pipeline")

# Calculate improvement percentages over basic RAG
basic_scores = df.loc["Basic RAG"]
improvements = {}
for pipeline in ["Hybrid Search", "Hybrid + Reranking", "Cited RAG"]:
    row = df.loc[pipeline]
    imp = {}
    for metric in ["faithfulness", "context_precision", "answer_relevancy"]:
        if basic_scores[metric] > 0:
            pct = ((row[metric] - basic_scores[metric]) / basic_scores[metric]) * 100
            imp[metric] = f"{pct:+.1f}%"
        else:
            imp[metric] = "N/A"
    improvements[pipeline] = imp

# Print formatted summary to the console
print("\n" + "=" * 60)
print("BENCHMARK RESULTS")
print("=" * 60)
print(df.to_string())

print("\n\nIMPROVEMENT OVER BASIC RAG:")
for pipeline, imp in improvements.items():
    print(f"  {pipeline}:")
    for metric, value in imp.items():
        print(f"    {metric}: {value}")

# Generate markdown report
report_lines = [
    "# RAG Pipeline Benchmark Report\n",
    "## Metric Comparison\n",
    "| Pipeline | Faithfulness | Context Precision | Answer Relevancy | Citation Accuracy |",
    "|----------|-------------|-------------------|------------------|-------------------|",
]

for pipeline in ["Basic RAG", "Hybrid Search", "Hybrid + Reranking", "Cited RAG"]:
    row = df.loc[pipeline]
    faith = f"{row['faithfulness']:.4f}"
    ctx = f"{row['context_precision']:.4f}"
    rel = f"{row['answer_relevancy']:.4f}"
    cit_val = row.get("citation_accuracy")
    cit = f"{cit_val:.4f}" if cit_val is not None and pd.notna(cit_val) else "-"
    report_lines.append(f"| {pipeline} | {faith} | {ctx} | {rel} | {cit} |")

report_lines.append("\n## Improvement Over Basic RAG\n")
for pipeline, imp in improvements.items():
    parts = [f"{metric} {value}" for metric, value in imp.items()]
    report_lines.append(f"- **{pipeline}**: " + ", ".join(parts))

report_lines.append("\n## Latency Comparison\n")
report_lines.append("| Pipeline | Evaluation Time (s) |")
report_lines.append("|----------|--------------------:|")
for pipeline in ["Basic RAG", "Hybrid Search", "Hybrid + Reranking", "Cited RAG"]:
    row = df.loc[pipeline]
    report_lines.append(f"| {pipeline} | {row['eval_time_seconds']:.2f} |")

# Save the report to a markdown file
with open("evals/benchmark_report.md", "w") as f:
    f.write("\n".join(report_lines))

print("\n\nBenchmark report saved to evals/benchmark_report.md")
print("These are your resume metrics!")

What does this code do?

This section builds a comparison table showing all four pipelines side by side. It calculates the percentage improvement for each metric relative to Basic RAG.

The final output is a markdown file you can paste directly into your README later. The latency comparison tracks how long RAGAS spent judging each pipeline. You will add actual inference latency tracking when you build the API layer.

• Save evals/evaluate.py.
• Run the evaluation script from your terminal:

python evals/evaluate.py

You should see each pipeline being evaluated one at a time, followed by a formatted table of all scores and improvement percentages printed to the console.

Seeing rate limit errors from Groq?

RAGAS makes many LLM calls per evaluation (one per claim, per context chunk). The Groq free tier has rate limits that return a 429 error if exceeded. If you hit this, add a short delay between pipeline evaluations by inserting time.sleep(60) between each evaluate_pipeline call.

You can also reduce the test set to 10 questions for faster iteration, then run the full 20 once you confirm everything works.

Help me fix RAGAS rate limiting with Groq

• Confirm that evals/benchmark_report.md was created by running:

cat evals/benchmark_report.md

You should see a markdown table with all four pipelines, their metric scores, improvement percentages, and a latency comparison section.

What is Faithfulness measuring?

Faithfulness scores range from 0 to 1. The formula is: (number of claims supported by context) divided by (total claims in response). A score of 0.85 means 85% of the claims the LLM made can be traced back to the retrieved chunks.

This is your primary hallucination detector. When you add this number to your resume, interviewers immediately know you understand how to measure LLM reliability.

✔️ Awesome, I've got everything!

Great. Double check you have saved evals/evaluate.py and that evals/benchmark_report.md exists with your metric scores.

ⓧ I'd like to double check the full code

Here is the complete evals/evaluate.py file:

import json
import time
import os
import pandas as pd
from dotenv import load_dotenv

# RAGAS imports for evaluation
from ragas import SingleTurnSample, EvaluationDataset, evaluate
from ragas.metrics import Faithfulness, ResponseRelevancy, LLMContextPrecisionWithoutReference
from ragas.llms import LangchainLLMWrapper

# LangChain Groq wrapper for the evaluator LLM
from langchain_groq import ChatGroq

# Load environment variables
load_dotenv()

# Configure the evaluator LLM using Groq via LangChain
groq_llm = ChatGroq(
    model="llama-3.3-70b-versatile",
    api_key=os.getenv("GROQ_API_KEY"),
    temperature=0
)

# Wrap for RAGAS compatibility
evaluator_llm = LangchainLLMWrapper(groq_llm)

# Load the test set and pipeline results
with open("evals/test_set.json", "r") as f:
    test_set = json.load(f)

with open("evals/results_basic.json", "r") as f:
    results_basic = json.load(f)

with open("evals/results_hybrid.json", "r") as f:
    results_hybrid = json.load(f)

with open("evals/results_reranked.json", "r") as f:
    results_reranked = json.load(f)

with open("evals/results_cited.json", "r") as f:
    results_cited = json.load(f)


def build_eval_dataset(test_set, results):
    """Convert test set + results into a RAGAS EvaluationDataset."""
    samples = []
    for item, result in zip(test_set, results):
        sample = SingleTurnSample(
            user_input=item["question"],
            response=result["answer"],
            retrieved_contexts=result["retrieved_contexts"],
            reference=item["reference_answer"]
        )
        samples.append(sample)
    return EvaluationDataset(samples=samples)


def evaluate_pipeline(name, test_set, results):
    """Run RAGAS metrics on a single pipeline's results."""
    print(f"\nEvaluating: {name}...")
    dataset = build_eval_dataset(test_set, results)

    # Define the three core RAG metrics
    metrics = [
        Faithfulness(),
        LLMContextPrecisionWithoutReference(),
        ResponseRelevancy()
    ]

    # Time the evaluation
    start_time = time.time()
    result = evaluate(
        dataset=dataset,
        metrics=metrics,
        llm=evaluator_llm
    )
    eval_time = time.time() - start_time

    scores = {
        "pipeline": name,
        "faithfulness": result["faithfulness"],
        "context_precision": result["llm_context_precision_without_reference"],
        "answer_relevancy": result["answer_relevancy"],
        "eval_time_seconds": round(eval_time, 2)
    }
    return scores


# Run evaluation on all four pipelines
all_scores = []
all_scores.append(evaluate_pipeline("Basic RAG", test_set, results_basic))
all_scores.append(evaluate_pipeline("Hybrid Search", test_set, results_hybrid))
all_scores.append(evaluate_pipeline("Hybrid + Reranking", test_set, results_reranked))
all_scores.append(evaluate_pipeline("Cited RAG", test_set, results_cited))

# Append citation accuracy for the cited pipeline
if "citation_accuracy" in results_cited[0]:
    all_scores[3]["citation_accuracy"] = sum(
        r["citation_accuracy"] for r in results_cited
    ) / len(results_cited)

print("\n" + "=" * 60)
print("EVALUATION COMPLETE")
print("=" * 60)

# Build DataFrame from all scores
df = pd.DataFrame(all_scores)
df = df.set_index("pipeline")

# Calculate improvement percentages over basic RAG
basic_scores = df.loc["Basic RAG"]
improvements = {}
for pipeline in ["Hybrid Search", "Hybrid + Reranking", "Cited RAG"]:
    row = df.loc[pipeline]
    imp = {}
    for metric in ["faithfulness", "context_precision", "answer_relevancy"]:
        if basic_scores[metric] > 0:
            pct = ((row[metric] - basic_scores[metric]) / basic_scores[metric]) * 100
            imp[metric] = f"{pct:+.1f}%"
        else:
            imp[metric] = "N/A"
    improvements[pipeline] = imp

# Print formatted summary to the console
print("\n" + "=" * 60)
print("BENCHMARK RESULTS")
print("=" * 60)
print(df.to_string())

print("\n\nIMPROVEMENT OVER BASIC RAG:")
for pipeline, imp in improvements.items():
    print(f"  {pipeline}:")
    for metric, value in imp.items():
        print(f"    {metric}: {value}")

# Generate markdown report
report_lines = [
    "# RAG Pipeline Benchmark Report\n",
    "## Metric Comparison\n",
    "| Pipeline | Faithfulness | Context Precision | Answer Relevancy | Citation Accuracy |",
    "|----------|-------------|-------------------|------------------|-------------------|",
]

for pipeline in ["Basic RAG", "Hybrid Search", "Hybrid + Reranking", "Cited RAG"]:
    row = df.loc[pipeline]
    faith = f"{row['faithfulness']:.4f}"
    ctx = f"{row['context_precision']:.4f}"
    rel = f"{row['answer_relevancy']:.4f}"
    cit_val = row.get("citation_accuracy")
    cit = f"{cit_val:.4f}" if cit_val is not None and pd.notna(cit_val) else "-"
    report_lines.append(f"| {pipeline} | {faith} | {ctx} | {rel} | {cit} |")

report_lines.append("\n## Improvement Over Basic RAG\n")
for pipeline, imp in improvements.items():
    parts = [f"{metric} {value}" for metric, value in imp.items()]
    report_lines.append(f"- **{pipeline}**: " + ", ".join(parts))

report_lines.append("\n## Latency Comparison\n")
report_lines.append("| Pipeline | Evaluation Time (s) |")
report_lines.append("|----------|--------------------:|")
for pipeline in ["Basic RAG", "Hybrid Search", "Hybrid + Reranking", "Cited RAG"]:
    row = df.loc[pipeline]
    report_lines.append(f"| {pipeline} | {row['eval_time_seconds']:.2f} |")

# Save the report to a markdown file
with open("evals/benchmark_report.md", "w") as f:
    f.write("\n".join(report_lines))

print("\n\nBenchmark report saved to evals/benchmark_report.md")
print("These are your resume metrics!")

You now have quantified proof that each retrieval improvement made a measurable difference. Next up, you will build an intelligent agent that routes questions to specialized handlers using LangGraph.

Build an Agentic RAG System with LangGraph

Your RAG system now retrieves documents with hybrid search, reranks them with a cross-encoder, generates cited answers, and has quantified metrics proving each improvement. But every question goes through the exact same pipeline regardless of what the user actually wants.

A question like "Summarize chapter 3" needs a completely different retrieval strategy than "What accuracy did model X achieve?" In this step, you will build an intelligent agent using LangGraph that classifies the user's intent and routes to specialized handler nodes.

In this step, get ready to:

• Design a state schema and intent router that classifies questions into four categories.
• Build specialized handler nodes for Retrieval, Summarization, Comparison, and Table Analysis.
• Add document grading and query rewriting to handle irrelevant retrieval results automatically.

Design the router and state schema

Every LangGraph agent starts with a state schema that defines what information flows between nodes. Your agent needs to track the conversation messages, the classified intent, retrieved documents, citations, and the generated context.

• Create a new file src/agent.py and add the imports, state schema, and LLM configuration by pasting the following code:

import os
from typing import TypedDict, Literal
from pydantic import BaseModel, Field
from langgraph.graph import StateGraph, START, END
from langchain_groq import ChatGroq
from langchain_core.messages import HumanMessage
from dotenv import load_dotenv

from src.pipeline import RerankedRAGPipeline
from src.citations import CitedRAGPipeline
from src.hybrid_retriever import HybridRetriever
from src.reranker import rerank

load_dotenv()

# Define the state that flows between all nodes in the graph
class AgentState(TypedDict):
    messages: list
    intent: str
    retrieved_documents: list
    citations: list
    current_document_context: str

# Structured output schema for intent classification
class RouteIntent(BaseModel):
    """The classified intent of the user's question."""
    intent: str = Field(
        description="One of: retrieval, summarization, comparison, table_analysis"
    )

# Initialize the Groq LLM for all agent nodes
llm = ChatGroq(
    model="llama-3.3-70b-versatile",
    api_key=os.getenv("GROQ_API_KEY")
)

What does this code do?

AgentState is a TypedDict that defines the data flowing through every node in the graph. Each node reads from this state and returns updates to it.

RouteIntent is a Pydantic model that forces the LLM to output a structured JSON object with exactly one field: the classified intent. This eliminates freeform text parsing and ensures reliable routing.

You won't see a change yet. The next chunk builds the router function that uses these schemas.

• Below the LLM initialization in src/agent.py, add the router node that classifies user intent:

# Router node: classifies user intent into one of four categories
def router(state: AgentState) -> dict:
    """Classify the user's question into a specific intent."""
    messages = state["messages"]
    last_message = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Use structured output to guarantee a valid intent classification
    structured_llm = llm.with_structured_output(RouteIntent)

    route_prompt = f"""Classify the following question into exactly one intent:
    - retrieval: Factual Q&A requiring specific passages from documents
    - summarization: Requests to summarize sections or entire documents
    - comparison: Questions comparing information across multiple documents
    - table_analysis: Questions about tabular data, numbers, or statistics

    Question: {last_message}"""

    result = structured_llm.invoke(route_prompt)
    return {"intent": result.intent}

What does this code do?

The router function extracts the latest user message from the state, then calls the LLM with with_structured_output(RouteIntent). This forces the LLM to return a JSON object matching the RouteIntent schema.

The returned intent string is one of retrieval, summarization, comparison, or table_analysis. This value is stored in the state and used by conditional edges to direct the flow to the correct handler.

You won't see a change yet. The next substep builds the four handler nodes that the router dispatches to.

Build specialized handler nodes

Each intent category needs a handler node with a different retrieval and generation strategy. The retrieval node uses your existing hybrid search and reranking pipeline with citations. The other three handlers adapt their prompts and retrieval parameters for their specific task.

• Below the router function in src/agent.py, add the retrieval handler node:

# Retrieval node: handles factual Q&A with hybrid search + reranking + citations
def retrieval_node(state: AgentState) -> dict:
    """Handle factual Q&A using the full hybrid + reranking + citation pipeline."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Use the reranked pipeline for precise document retrieval
    pipeline = RerankedRAGPipeline()
    reranked_docs = pipeline.retrieve(query)

    # Generate a cited response from the reranked documents
    cited_pipeline = CitedRAGPipeline()
    response = cited_pipeline.generate(query, reranked_docs)

    return {
        "retrieved_documents": reranked_docs,
        "citations": response.citations,
        "current_document_context": response.answer
    }

What does this code do?

The retrieval_node reuses RerankedRAGPipeline (hybrid search + cross-encoder reranking) and CitedRAGPipeline (citation-grounded generation) that you built in earlier steps. This gives factual questions the highest-precision retrieval with verifiable citations.

• Below retrieval_node, add the remaining three handler nodes:

# Summarization node: retrieves broader context and produces summaries
def summarization_node(state: AgentState) -> dict:
    """Retrieve broader context and produce summaries with section references."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Retrieve more chunks for comprehensive summarization
    hybrid = HybridRetriever()
    docs = hybrid.retrieve(query, k=10)

    context = "\n\n".join([doc["content"] for doc in docs])

    summary_prompt = f"""Summarize the following content in response to the user's request.
    Include section references (e.g., [Page X, Section Y]) for each major point.

    User request: {query}

    Content:
    {context}"""

    response = llm.invoke(summary_prompt)

    return {
        "retrieved_documents": docs,
        "current_document_context": response.content
    }

# Comparison node: retrieves from multiple documents and produces structured comparison
def comparison_node(state: AgentState) -> dict:
    """Retrieve from multiple documents and produce a structured comparison."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Retrieve a broad set of chunks from multiple sources
    hybrid = HybridRetriever()
    docs = hybrid.retrieve(query, k=15)

    # Group documents by source file
    sources = {}
    for doc in docs:
        source = doc.get("metadata", {}).get("source", "unknown")
        if source not in sources:
            sources[source] = []
        sources[source].append(doc["content"])

    comparison_prompt = f"""Compare the information from these different sources.
    Structure your response with:
    - Similarities
    - Differences
    - Include citations [Source: filename, Page X] for each point.

    Question: {query}

    Sources:
    {chr(10).join([f"Source: {s}" + chr(10) + chr(10).join(chunks) for s, chunks in sources.items()])}"""

    response = llm.invoke(comparison_prompt)

    return {
        "retrieved_documents": docs,
        "current_document_context": response.content
    }

# Table analysis node: focuses on extracting and reasoning over numerical data
def table_analysis_node(state: AgentState) -> dict:
    """Extract and reason over tabular and numerical data from documents."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Retrieve documents likely containing tabular/numerical content
    hybrid = HybridRetriever()
    docs = hybrid.retrieve(query, k=10)

    table_prompt = f"""Analyze the following content to answer the user's question about data, statistics, or tables.
    Extract relevant numbers, present them clearly, and explain any trends or patterns.
    Cite sources for each data point.

    Question: {query}

    Content:
    {chr(10).join([doc["content"] for doc in docs])}"""

    response = llm.invoke(table_prompt)

    return {
        "retrieved_documents": docs,
        "current_document_context": response.content
    }

How do the handler nodes differ?

Each handler adapts its strategy to the task at hand:

• summarization_node retrieves 10 chunks (broader context) and prompts for section-referenced summaries.
• comparison_node retrieves 15 chunks, groups them by source file, then prompts for a structured similarities/differences analysis.
• table_analysis_node retrieves chunks and prompts specifically for numerical extraction and trend analysis.

You won't see a change yet. The next substep wires these nodes into a complete graph with document grading.

Add document grading and query rewriting

Sometimes retrieved documents are irrelevant to the question. Instead of generating a low-quality answer, the agent should detect this and automatically reformulate the query to try again.

• Below the handler nodes in src/agent.py, add the document grading schema and conditional edge function:

# Structured output schema for grading document relevance
class GradeDocuments(BaseModel):
    """Grade whether retrieved documents are relevant to the question."""
    binary_score: str = Field(
        description="Relevance score: 'yes' if relevant, or 'no' if not relevant"
    )

# Conditional edge: assess if retrieved documents are relevant
def grade_documents(state: AgentState) -> Literal["generate", "rewrite"]:
    """Determine whether retrieved documents are relevant to the query."""
    docs = state["retrieved_documents"]
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    if not docs:
        return "rewrite"

    # Check relevance of the top retrieved document
    doc_content = docs[0]["content"] if isinstance(docs[0], dict) else str(docs[0])

    structured_llm = llm.with_structured_output(GradeDocuments)

    grade_prompt = f"""You are a grader assessing relevance of a retrieved document to a user question.

    Retrieved document: {doc_content}

    User question: {query}

    Give a binary score 'yes' or 'no' to indicate whether the document is relevant to the question."""

    result = structured_llm.invoke(grade_prompt)

    if result.binary_score == "yes":
        return "generate"
    return "rewrite"

What does this code do?

GradeDocuments uses the same structured output pattern as the router. The LLM is forced to return either yes or no for document relevance.

The grade_documents function acts as a conditional edge in the graph. It returns the string "generate" (proceed to output) or "rewrite" (reformulate the query). LangGraph uses these return values to decide which node to visit next.

• Below grade_documents, add the query rewriting node:

# Rewrite node: reformulates the query when documents are irrelevant
def rewrite_question(state: AgentState) -> dict:
    """Reformulate the query to improve retrieval results."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    rewrite_prompt = f"""The original question did not retrieve relevant documents.
    Reformulate it to be more specific and searchable.

    Original question: {query}

    Rewritten question:"""

    response = llm.invoke(rewrite_prompt)

    # Replace the last message with the rewritten query
    return {"messages": [HumanMessage(content=response.content)]}

What does this code do?

When grade_documents returns "rewrite", the graph visits this node. It asks the LLM to reformulate the original question into something more specific. The rewritten query replaces the last message in state, and the graph loops back to the router to try again with better search terms.

• Below rewrite_question, add the graph assembly code that wires everything together:

# Route to the correct handler based on classified intent
def route_to_handler(state: AgentState) -> str:
    """Direct to the appropriate handler based on classified intent."""
    return state["intent"]

# Assemble the full LangGraph state graph
workflow = StateGraph(AgentState)

# Add all nodes to the graph
workflow.add_node("router", router)
workflow.add_node("retrieval", retrieval_node)
workflow.add_node("summarization", summarization_node)
workflow.add_node("comparison", comparison_node)
workflow.add_node("table_analysis", table_analysis_node)
workflow.add_node("rewrite_question", rewrite_question)

# Entry point: every query starts at the router
workflow.add_edge(START, "router")

# Router dispatches to the correct handler based on intent
workflow.add_conditional_edges(
    "router",
    route_to_handler,
    {
        "retrieval": "retrieval",
        "summarization": "summarization",
        "comparison": "comparison",
        "table_analysis": "table_analysis",
    }
)

# After each handler, grade document relevance
workflow.add_conditional_edges("retrieval", grade_documents, {"generate": END, "rewrite": "rewrite_question"})
workflow.add_conditional_edges("summarization", grade_documents, {"generate": END, "rewrite": "rewrite_question"})
workflow.add_conditional_edges("comparison", grade_documents, {"generate": END, "rewrite": "rewrite_question"})
workflow.add_conditional_edges("table_analysis", grade_documents, {"generate": END, "rewrite": "rewrite_question"})

# After rewriting, loop back to the router for re-classification
workflow.add_edge("rewrite_question", "router")

# Compile the graph into a runnable agent
graph = workflow.compile()

How does the graph flow work?

The assembled graph follows this path: START → router → [one of four handlers] → grade_documents → END (if relevant) or rewrite_question → router (loop).

add_conditional_edges takes a source node, a function that returns a string, and a mapping of return values to destination nodes. The route_to_handler function reads the intent field from state and returns it directly, so "retrieval" maps to the "retrieval" node, and so on.

The rewrite loop prevents the agent from returning garbage when retrieval fails. It reformulates the query and tries again through the full routing cycle.

• Save src/agent.py.
• Verify the graph compiles without errors by running this command:

python -c "from src.agent import graph; print(f'Graph compiled with {len(graph.nodes)} nodes')"

You should see a message showing the graph compiled with 6 nodes (router, retrieval, summarization, comparison, table_analysis, rewrite_question).

Seeing an import error?

• If you see a module not found error for langgraph, make sure your virtual environment is active and you have installed it via requirements.txt.
• If the import of src.pipeline or src.citations fails, confirm those files exist in your src/ directory and that you are running the command from the rag-system/ project root.
• If you see a Pydantic validation error, make sure the GradeDocuments and RouteIntent classes use BaseModel from pydantic, not from langchain.

Help me debug my LangGraph agent import errors

✔️ Awesome, I've got everything!

Great. Double check you have saved src/agent.py with all the code from this step.

ⓧ I'd like to double check the full code

import os
from typing import TypedDict, Literal
from pydantic import BaseModel, Field
from langgraph.graph import StateGraph, START, END
from langchain_groq import ChatGroq
from langchain_core.messages import HumanMessage
from dotenv import load_dotenv

from src.pipeline import RerankedRAGPipeline
from src.citations import CitedRAGPipeline
from src.hybrid_retriever import HybridRetriever
from src.reranker import rerank

load_dotenv()

# Define the state that flows between all nodes in the graph
class AgentState(TypedDict):
    messages: list
    intent: str
    retrieved_documents: list
    citations: list
    current_document_context: str

# Structured output schema for intent classification
class RouteIntent(BaseModel):
    """The classified intent of the user's question."""
    intent: str = Field(
        description="One of: retrieval, summarization, comparison, table_analysis"
    )

# Initialize the Groq LLM for all agent nodes
llm = ChatGroq(
    model="llama-3.3-70b-versatile",
    api_key=os.getenv("GROQ_API_KEY")
)

# Router node: classifies user intent into one of four categories
def router(state: AgentState) -> dict:
    """Classify the user's question into a specific intent."""
    messages = state["messages"]
    last_message = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Use structured output to guarantee a valid intent classification
    structured_llm = llm.with_structured_output(RouteIntent)

    route_prompt = f"""Classify the following question into exactly one intent:
    - retrieval: Factual Q&A requiring specific passages from documents
    - summarization: Requests to summarize sections or entire documents
    - comparison: Questions comparing information across multiple documents
    - table_analysis: Questions about tabular data, numbers, or statistics

    Question: {last_message}"""

    result = structured_llm.invoke(route_prompt)
    return {"intent": result.intent}

# Retrieval node: handles factual Q&A with hybrid search + reranking + citations
def retrieval_node(state: AgentState) -> dict:
    """Handle factual Q&A using the full hybrid + reranking + citation pipeline."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Use the reranked pipeline for precise document retrieval
    pipeline = RerankedRAGPipeline()
    reranked_docs = pipeline.retrieve(query)

    # Generate a cited response from the reranked documents
    cited_pipeline = CitedRAGPipeline()
    response = cited_pipeline.generate(query, reranked_docs)

    return {
        "retrieved_documents": reranked_docs,
        "citations": response.citations,
        "current_document_context": response.answer
    }

# Summarization node: retrieves broader context and produces summaries
def summarization_node(state: AgentState) -> dict:
    """Retrieve broader context and produce summaries with section references."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Retrieve more chunks for comprehensive summarization
    hybrid = HybridRetriever()
    docs = hybrid.retrieve(query, k=10)

    context = "\n\n".join([doc["content"] for doc in docs])

    summary_prompt = f"""Summarize the following content in response to the user's request.
    Include section references (e.g., [Page X, Section Y]) for each major point.

    User request: {query}

    Content:
    {context}"""

    response = llm.invoke(summary_prompt)

    return {
        "retrieved_documents": docs,
        "current_document_context": response.content
    }

# Comparison node: retrieves from multiple documents and produces structured comparison
def comparison_node(state: AgentState) -> dict:
    """Retrieve from multiple documents and produce a structured comparison."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Retrieve a broad set of chunks from multiple sources
    hybrid = HybridRetriever()
    docs = hybrid.retrieve(query, k=15)

    # Group documents by source file
    sources = {}
    for doc in docs:
        source = doc.get("metadata", {}).get("source", "unknown")
        if source not in sources:
            sources[source] = []
        sources[source].append(doc["content"])

    comparison_prompt = f"""Compare the information from these different sources.
    Structure your response with:
    - Similarities
    - Differences
    - Include citations [Source: filename, Page X] for each point.

    Question: {query}

    Sources:
    {chr(10).join([f"Source: {s}" + chr(10) + chr(10).join(chunks) for s, chunks in sources.items()])}"""

    response = llm.invoke(comparison_prompt)

    return {
        "retrieved_documents": docs,
        "current_document_context": response.content
    }

# Table analysis node: focuses on extracting and reasoning over numerical data
def table_analysis_node(state: AgentState) -> dict:
    """Extract and reason over tabular and numerical data from documents."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    # Retrieve documents likely containing tabular/numerical content
    hybrid = HybridRetriever()
    docs = hybrid.retrieve(query, k=10)

    table_prompt = f"""Analyze the following content to answer the user's question about data, statistics, or tables.
    Extract relevant numbers, present them clearly, and explain any trends or patterns.
    Cite sources for each data point.

    Question: {query}

    Content:
    {chr(10).join([doc["content"] for doc in docs])}"""

    response = llm.invoke(table_prompt)

    return {
        "retrieved_documents": docs,
        "current_document_context": response.content
    }

# Structured output schema for grading document relevance
class GradeDocuments(BaseModel):
    """Grade whether retrieved documents are relevant to the question."""
    binary_score: str = Field(
        description="Relevance score: 'yes' if relevant, or 'no' if not relevant"
    )

# Conditional edge: assess if retrieved documents are relevant
def grade_documents(state: AgentState) -> Literal["generate", "rewrite"]:
    """Determine whether retrieved documents are relevant to the query."""
    docs = state["retrieved_documents"]
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    if not docs:
        return "rewrite"

    # Check relevance of the top retrieved document
    doc_content = docs[0]["content"] if isinstance(docs[0], dict) else str(docs[0])

    structured_llm = llm.with_structured_output(GradeDocuments)

    grade_prompt = f"""You are a grader assessing relevance of a retrieved document to a user question.

    Retrieved document: {doc_content}

    User question: {query}

    Give a binary score 'yes' or 'no' to indicate whether the document is relevant to the question."""

    result = structured_llm.invoke(grade_prompt)

    if result.binary_score == "yes":
        return "generate"
    return "rewrite"

# Rewrite node: reformulates the query when documents are irrelevant
def rewrite_question(state: AgentState) -> dict:
    """Reformulate the query to improve retrieval results."""
    messages = state["messages"]
    query = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])

    rewrite_prompt = f"""The original question did not retrieve relevant documents.
    Reformulate it to be more specific and searchable.

    Original question: {query}

    Rewritten question:"""

    response = llm.invoke(rewrite_prompt)

    # Replace the last message with the rewritten query
    return {"messages": [HumanMessage(content=response.content)]}

# Route to the correct handler based on classified intent
def route_to_handler(state: AgentState) -> str:
    """Direct to the appropriate handler based on classified intent."""
    return state["intent"]

# Assemble the full LangGraph state graph
workflow = StateGraph(AgentState)

# Add all nodes to the graph
workflow.add_node("router", router)
workflow.add_node("retrieval", retrieval_node)
workflow.add_node("summarization", summarization_node)
workflow.add_node("comparison", comparison_node)
workflow.add_node("table_analysis", table_analysis_node)
workflow.add_node("rewrite_question", rewrite_question)

# Entry point: every query starts at the router
workflow.add_edge(START, "router")

# Router dispatches to the correct handler based on intent
workflow.add_conditional_edges(
    "router",
    route_to_handler,
    {
        "retrieval": "retrieval",
        "summarization": "summarization",
        "comparison": "comparison",
        "table_analysis": "table_analysis",
    }
)

# After each handler, grade document relevance
workflow.add_conditional_edges("retrieval", grade_documents, {"generate": END, "rewrite": "rewrite_question"})
workflow.add_conditional_edges("summarization", grade_documents, {"generate": END, "rewrite": "rewrite_question"})
workflow.add_conditional_edges("comparison", grade_documents, {"generate": END, "rewrite": "rewrite_question"})
workflow.add_conditional_edges("table_analysis", grade_documents, {"generate": END, "rewrite": "rewrite_question"})

# After rewriting, loop back to the router for re-classification
workflow.add_edge("rewrite_question", "router")

# Compile the graph into a runnable agent
graph = workflow.compile()

Your RAG system now has an intelligent brain. Instead of treating every question the same, it routes to specialized handlers and self-corrects when retrieval fails. Next up, you will add conversation memory and streaming so the agent remembers context across turns and delivers answers token-by-token.

Add Conversation Memory and Streaming

Your agentic RAG system can now route questions to specialized handlers, grade documents, and rewrite queries. But every conversation starts from scratch.

Real AI assistants remember what you asked two turns ago. In this step, you will add conversation memory so your agent maintains context across questions, and streaming so responses appear token-by-token like ChatGPT.

In this step, get ready to:

• Add persistent conversation memory using LangGraph's MemorySaver checkpointer.
• Enable token-by-token streaming with intermediate status messages.
• Test multi-turn conversations and run RAGAS evaluation on the full pipeline.

Implement conversation memory

For your agent to handle follow-up questions like "How does it compare to the baseline?" without needing you to re-specify the topic, it needs to remember what was discussed earlier. LangGraph provides a built-in checkpointer called MemorySaver that persists conversation state per thread.

• In src/agent.py, find the AgentState TypedDict at the top of your file.
• Update the messages field to use the Annotated type with add_messages as the reducer, then add the new imports and the MemorySaver checkpointer by copying the code below:

from typing import Annotated, TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.messages import HumanMessage, AIMessage

# Updated state schema with message history support
class AgentState(TypedDict):
    messages: Annotated[list, add_messages]  # Appends new messages instead of replacing
    intent: str
    retrieved_documents: list
    citations: list
    current_document_context: str

What does this code do?

The Annotated[list, add_messages] type tells LangGraph to append new messages to the existing list rather than replacing it. Each time a node returns messages, they accumulate in state.

The MemorySaver checkpointer saves a snapshot of this state after every node executes. When you pass a thread_id in the config, the checkpointer knows which conversation thread to load and resume.

• Now add a sliding window helper function below the AgentState class to trim the conversation to the last 10 messages:

def trim_messages(messages: list, max_messages: int = 10) -> list:
    """Keep only the last N messages to stay within context limits."""
    if len(messages) <= max_messages:
        return messages
    return messages[-max_messages:]

Why trim messages?

LLMs have finite context windows. If you accumulate hundreds of messages, the model runs out of space for the actual document context. Trimming to the last 10 messages keeps the conversation responsive without losing recent context.

• Next, update the graph compilation at the bottom of src/agent.py to pass the checkpointer:

# Instantiate the memory checkpointer
memory = MemorySaver()

# Compile the graph with memory persistence
graph = workflow.compile(checkpointer=memory)

What does this code do?

Passing checkpointer=memory to compile() tells LangGraph to save conversation state after each node. Different thread_id values create isolated conversation threads, so one user's chat history never leaks into another's.

• Save src/agent.py.
• You will not see a visible change yet. The memory system activates when you invoke the graph with a thread config in the next substep.

Seeing an import error?

Make sure you have langgraph installed in your virtual environment. The MemorySaver class lives in langgraph.checkpoint.memory.

help me fix this MemorySaver import error

Add streaming responses

Streaming gives users real-time feedback as the agent works. Instead of waiting for the entire response, you will see intermediate status messages and the answer appearing token-by-token.

• In src/agent.py, update the Groq LLM initialization to enable streaming:

from langchain_groq import ChatGroq

# Configure the LLM with streaming enabled
llm = ChatGroq(
    model="llama-3.3-70b-versatile",
    temperature=0,
    streaming=True  # Enable token-by-token output
)

What does streaming=True do?

Setting streaming=True tells the Groq client to send response tokens as they are generated, rather than waiting for the full response. LangGraph's .stream() method then yields these tokens progressively to your application.

• Now add intermediate status messages inside each handler node. Add a print statement at the start of each node function to emit progress. Here is the pattern for retrieval_node:

def retrieval_node(state: AgentState) -> dict:
    """Handle factual Q&A from documents with citation grounding."""
    print("\ud83d\udd0d Routing to: Retrieval...")
    # Trim messages to stay within context limits
    trimmed = trim_messages(state["messages"])
    print("\ud83d\udcda Searching documents...")
    # ... existing retrieval logic ...
    print(f"\u2705 Found {len(results)} relevant sources...")
    # ... existing generation + citation logic ...
    return {"messages": [AIMessage(content=response)], "citations": citations}

What are these status messages for?

When you stream the graph, LangGraph emits the state update after each node completes. The print() statements provide real-time console feedback during development. In the Gradio UI (Step 10), these will become visible progress indicators for the user.

• Apply the same pattern to summarization_node, comparison_node, and table_analysis_node. Each node should print a routing message, a searching message, and a results-found message.
• Add the streaming invocation function at the bottom of src/agent.py:

def stream_query(question: str, thread_id: str = "default"):
    """Stream a query through the agent with conversation memory."""
    config = {"configurable": {"thread_id": thread_id}}
    input_state = {"messages": [HumanMessage(content=question)]}

    # Stream the graph execution, yielding state updates per node
    for chunk in graph.stream(input_state, config, stream_mode="values"):
        # Each chunk contains the full state at that point
        if "messages" in chunk:
            latest = chunk["messages"][-1]
            if hasattr(latest, "content") and latest.content:
                print(latest.content)

How does thread_id isolate conversations?

The thread_id in the config tells the MemorySaver which conversation to load. Passing "thread_1" resumes one conversation. Passing "thread_2" starts a completely separate one. This is how multi-user chat apps work.

• Save src/agent.py.

Not seeing streamed output?

Ensure streaming=True is set on the ChatGroq instance. Also confirm you are calling graph.stream() (not graph.invoke()), as invoke() waits for the full response before returning.

help me debug why my LangGraph streaming is not showing tokens

Test conversational flows and run evaluation

Now you will verify that memory persists across turns and that different thread IDs create separate conversations. Then you will run RAGAS evaluation on the complete agentic pipeline to get your final benchmark numbers.

• Create a test script to verify multi-turn conversation by running this code:

# Test multi-turn conversation with memory
from src.agent import stream_query

# First question on thread_1
stream_query("What method did the authors use?", thread_id="thread_1")

# Follow-up referencing previous answer (no need to re-specify topic)
stream_query("How does it compare to the baseline?", thread_id="thread_1")

# Different thread starts fresh (no memory from thread_1)
stream_query("How does it compare to the baseline?", thread_id="thread_2")

What should I see?

The second call on thread_1 should correctly answer about the baseline comparison using context from the first question. The call on thread_2 should fail or give a generic answer because it has no prior context about which paper you are discussing.

• Run the test script to verify memory works:

python test_memory.py

• You should see the agent correctly handle the follow-up question on thread_1 by referencing the earlier response, while thread_2 starts without context.
• Now run RAGAS evaluation on the full agentic + memory pipeline. Add the Agentic (Full) row to your benchmark report by running your evaluation script with the complete pipeline:

# Run RAGAS evaluation on the full agentic pipeline
# Add this to your evaluation script in evals/
from ragas import evaluate
from ragas.metrics import Faithfulness, LLMContextRecall, FactualCorrectness

# Run evaluation using the same 20-question test set
# through the full agentic pipeline (with memory + streaming)
result = evaluate(
    dataset=evaluation_dataset,
    metrics=[faithfulness, context_precision, answer_relevancy]
)

print("\n=== Agentic (Full) Pipeline Results ===")
print(f"Faithfulness: {result['faithfulness']:.4f}")
print(f"Context Precision: {result['context_precision']:.4f}")
print(f"Answer Relevancy: {result['answer_relevancy']:.4f}")

• Run the evaluation:

python evals/run_evaluation.py

• Update evals/benchmark_report.md with the new Agentic (Full) row containing your scores for Faithfulness, Context Precision, Answer Relevancy, and Citation Accuracy.
• Save evals/benchmark_report.md.

What do these numbers mean for your resume?

The Agentic (Full) row represents your complete system with intent routing, hybrid search, reranking, citations, memory, and streaming all working together. Compare these numbers against your earlier pipeline variants to calculate improvement percentages.

Evaluation failing with memory-related errors?

Make sure each evaluation query uses a fresh thread_id so that memory from previous test questions does not contaminate the results. Use a unique ID per question, like f"eval_{i}" where i is the question index.

help me fix RAGAS evaluation errors with my agentic pipeline

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/agent.py and evals/benchmark_report.md.

ⓧ I'd like to double check the full code

from typing import Annotated, TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.messages import HumanMessage, AIMessage
from langchain_groq import ChatGroq

# Updated state schema with message history support
class AgentState(TypedDict):
    messages: Annotated[list, add_messages]  # Appends new messages instead of replacing
    intent: str
    retrieved_documents: list
    citations: list
    current_document_context: str

def trim_messages(messages: list, max_messages: int = 10) -> list:
    """Keep only the last N messages to stay within context limits."""
    if len(messages) <= max_messages:
        return messages
    return messages[-max_messages:]

# Configure the LLM with streaming enabled
llm = ChatGroq(
    model="llama-3.3-70b-versatile",
    temperature=0,
    streaming=True  # Enable token-by-token output
)

# --- Router node ---
def router(state: AgentState) -> dict:
    """Classify user intent and route to the correct handler."""
    # LLM classifies intent with structured output
    # Returns one of: retrieval, summarization, comparison, table_analysis
    pass

# --- Handler nodes ---
def retrieval_node(state: AgentState) -> dict:
    """Handle factual Q&A from documents with citation grounding."""
    print("\ud83d\udd0d Routing to: Retrieval...")
    trimmed = trim_messages(state["messages"])
    print("\ud83d\udcda Searching documents...")
    # ... existing retrieval logic (hybrid search + reranking + citations) ...
    results = []  # Retrieved documents
    print(f"\u2705 Found {len(results)} relevant sources...")
    # ... generation + citation logic ...
    response = ""  # Generated response
    citations = []  # Citation metadata
    return {"messages": [AIMessage(content=response)], "citations": citations}

def summarization_node(state: AgentState) -> dict:
    """Summarize sections or documents with section references."""
    print("\ud83d\udd0d Routing to: Summarization...")
    trimmed = trim_messages(state["messages"])
    print("\ud83d\udcda Searching documents...")
    # ... existing summarization logic ...
    results = []
    print(f"\u2705 Found {len(results)} relevant sources...")
    response = ""
    return {"messages": [AIMessage(content=response)]}

def comparison_node(state: AgentState) -> dict:
    """Compare information across multiple documents."""
    print("\ud83d\udd0d Routing to: Comparison...")
    trimmed = trim_messages(state["messages"])
    print("\ud83d\udcda Searching documents...")
    # ... existing comparison logic ...
    results = []
    print(f"\u2705 Found {len(results)} relevant sources...")
    response = ""
    citations = []
    return {"messages": [AIMessage(content=response)], "citations": citations}

def table_analysis_node(state: AgentState) -> dict:
    """Extract and reason over tabular data."""
    print("\ud83d\udd0d Routing to: Table Analysis...")
    trimmed = trim_messages(state["messages"])
    print("\ud83d\udcda Searching documents...")
    # ... existing table analysis logic ...
    results = []
    print(f"\u2705 Found {len(results)} relevant sources...")
    response = ""
    return {"messages": [AIMessage(content=response)]}

# --- Document grading ---
def grade_documents(state: AgentState) -> str:
    """Assess if retrieved docs are relevant. Returns 'generate' or 'rewrite'."""
    # Uses GradeDocuments structured output schema
    pass

def rewrite_question(state: AgentState) -> dict:
    """Reformulate the query if documents are graded as irrelevant."""
    pass

# --- Assemble the graph ---
workflow = StateGraph(AgentState)
# Add all nodes
workflow.add_node("router", router)
workflow.add_node("retrieval", retrieval_node)
workflow.add_node("summarization", summarization_node)
workflow.add_node("comparison", comparison_node)
workflow.add_node("table_analysis", table_analysis_node)
workflow.add_node("rewrite_question", rewrite_question)
# Add edges
workflow.add_edge(START, "router")
# Conditional edges from router to handler nodes
# Conditional edges from grade_documents to generate/rewrite
workflow.add_edge("rewrite_question", "router")

# Instantiate the memory checkpointer
memory = MemorySaver()

# Compile the graph with memory persistence
graph = workflow.compile(checkpointer=memory)

def stream_query(question: str, thread_id: str = "default"):
    """Stream a query through the agent with conversation memory."""
    config = {"configurable": {"thread_id": thread_id}}
    input_state = {"messages": [HumanMessage(content=question)]}

    # Stream the graph execution, yielding state updates per node
    for chunk in graph.stream(input_state, config, stream_mode="values"):
        # Each chunk contains the full state at that point
        if "messages" in chunk:
            latest = chunk["messages"][-1]
            if hasattr(latest, "content") and latest.content:
                print(latest.content)

Your agent now remembers previous questions, streams answers progressively, and has a complete set of benchmark metrics. Next up, you will add multimodal PDF understanding so the agent can reason over images and tables extracted from documents.

Enable Multimodal PDF Understanding

Your agentic RAG system can route questions to specialized handlers and remember context across turns. However, it currently only understands text extracted from PDFs.

Many documents contain critical information locked inside charts, diagrams, and tables that pure text extraction misses entirely. In this step, you will extract images and tables from PDFs, generate text descriptions of visual content using a multimodal model, and store those descriptions as searchable chunks in ChromaDB.

In this step, get ready to:

• Extract embedded images and table structures from PDFs using PyMuPDF.
• Generate text descriptions of images using Groq's multimodal Llama 4 Scout model.
• Update the ingestion pipeline and Table Analysis router to leverage multimodal chunks.

Extract images and tables from PDFs

PDFs embed images as binary objects and tables as structured layouts. PyMuPDF can extract both. You will create a new module that pulls images out as base64-encoded data and detects table structures on each page.

• Create a new file src/multimodal.py inside your rag-system folder by running:

touch src/multimodal.py

• Open src/multimodal.py in VS Code and add the image extraction function by pasting the following code:

import fitz  # PyMuPDF
import base64
import os
from groq import Groq


def extract_images_from_pdf(pdf_path):
    """Extract embedded images from PDF pages as base64."""
    images = []
    doc = fitz.open(pdf_path)

    for page_num in range(len(doc)):
        page = doc[page_num]
        # Get all images embedded on this page
        image_list = page.get_images(full=True)

        for img_index, img_info in enumerate(image_list):
            # Extract raw image bytes using the xref ID
            xref = img_info[0]
            base_image = doc.extract_image(xref)
            image_bytes = base_image["image"]
            # Encode as base64 for the multimodal API
            base64_data = base64.b64encode(image_bytes).decode("utf-8")

            images.append({
                "page_num": page_num + 1,
                "image_index": img_index,
                "base64_data": base64_data
            })

    doc.close()
    return images

What does this code do?

The extract_images_from_pdf function opens a PDF and iterates through every page. On each page, get_images(full=True) returns metadata for all embedded images.

For each image, it uses the xref (cross-reference ID) to pull the raw bytes, then base64-encodes them. The result is a list of dictionaries containing the page number, image index, and the base64 string ready to send to a multimodal model.

• Add the table extraction function directly below the image function in the same file:

def extract_tables_from_pdf(pdf_path):
    """Extract table structures from PDF pages using PyMuPDF table detection."""
    tables = []
    doc = fitz.open(pdf_path)

    for page_num in range(len(doc)):
        page = doc[page_num]
        # Use PyMuPDF built-in table detection (v1.23.0+)
        page_tables = page.find_tables()

        for table_index, table in enumerate(page_tables):
            # Convert to markdown for LLM-friendly token-efficient format
            table_text = table.to_markdown()

            tables.append({
                "page_num": page_num + 1,
                "table_index": table_index,
                "table_text": table_text
            })

    doc.close()
    return tables

What does this code do?

The find_tables() method detects table structures on each page using PyMuPDF's built-in table detection engine. It returns Table objects that expose headers, cells, and content.

The to_markdown() method converts each table into a GitHub-compatible markdown format optimized for small token sizes. This makes it ideal for feeding into LLM context windows for RAG.

You won't see visible output yet. The next substep uses these extraction functions to generate searchable descriptions.

Generate descriptions with the multimodal model

Extracted images are binary data that your vector search cannot index directly. You need text descriptions of each image so that questions about charts or figures can retrieve the right context.

Groq's meta-llama/llama-4-scout-17b-16e-instruct model accepts image input alongside text prompts. It generates detailed descriptions of visual content at 750 tokens per second.

• Add the describe_image function below the table extraction function in src/multimodal.py:

def describe_image(base64_data, page_context, llm=None):
    """Generate a text description of an image using Groq Llama 4 Scout."""
    client = Groq(api_key=os.getenv("GROQ_API_KEY"))

    # Send the base64 image to the multimodal model for description
    response = client.chat.completions.create(
        model="meta-llama/llama-4-scout-17b-16e-instruct",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": f"Describe this image in detail for document search purposes. "
                                f"Page context: {page_context}. "
                                f"Include all visible data, labels, axes, legends, and key takeaways."
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/png;base64,{base64_data}"
                        }
                    }
                ]
            }
        ],
        max_completion_tokens=1024
    )

    return response.choices[0].message.content

What does this code do?

This function sends a base64-encoded image to Groq's Llama 4 Scout multimodal model. The prompt asks the model to describe the image in detail, including data labels, axes, legends, and key takeaways.

The image is passed as a data URL (data:image/png;base64,...) inside the image_url content block. This is the same format the Groq Vision API uses for inline image input.

• Add the orchestration function that ties everything together. Paste this below describe_image in src/multimodal.py:

def extract_multimodal_content(pdf_path, llm=None):
    """Extract and describe all multimodal content from a PDF."""
    enriched_chunks = []

    # Generate text descriptions for each extracted image
    images = extract_images_from_pdf(pdf_path)
    for img in images:
        page_context = f"Page {img['page_num']} of {os.path.basename(pdf_path)}"
        description = describe_image(img["base64_data"], page_context, llm)

        enriched_chunks.append({
            "text": description,
            "metadata": {
                "source": pdf_path,
                "page_num": img["page_num"],
                "content_type": "image",
                "image_index": img["image_index"]
            }
        })

    # Convert extracted tables into searchable text chunks
    tables = extract_tables_from_pdf(pdf_path)
    for tbl in tables:
        enriched_chunks.append({
            "text": f"Table from page {tbl['page_num']}:\n{tbl['table_text']}",
            "metadata": {
                "source": pdf_path,
                "page_num": tbl["page_num"],
                "content_type": "table",
                "table_index": tbl["table_index"]
            }
        })

    return enriched_chunks

What does this code do?

The extract_multimodal_content function is the single entry point for the multimodal pipeline. It calls both extraction functions, generates image descriptions via the multimodal model, and returns a flat list of enriched chunk dictionaries.

Each chunk has a text field (searchable by your embeddings) and a metadata dictionary that links back to the source file, page number, and content type (image or table). This metadata enables the Table Analysis router to filter for multimodal chunks specifically.

• Save src/multimodal.py.

Now you need to wire this into your existing ingestion pipeline so multimodal chunks get stored alongside text chunks in ChromaDB.

• Open src/ingest.py and add this import at the top of the file, alongside your existing imports:

from src.multimodal import extract_multimodal_content

• Find the section in src/ingest.py where you add text chunks to the ChromaDB collection (after splitting and embedding). Add the following code block below it to also store multimodal chunks:

# Extract and store multimodal content (images and tables)
multimodal_chunks = extract_multimodal_content(pdf_path)
for i, chunk in enumerate(multimodal_chunks):
    collection.add(
        documents=[chunk["text"]],
        metadatas=[chunk["metadata"]],
        ids=[f"multimodal_{os.path.basename(pdf_path)}_{i}"]
    )

What does this code do?

This addition calls extract_multimodal_content during PDF ingestion and stores each enriched chunk in the same ChromaDB collection as your text chunks. The metadata includes content_type which allows filtering for just image or table chunks during retrieval.

• Save src/ingest.py.
• Re-run your ingestion script to process multimodal content from your PDFs by running:

python -m src.ingest

You should see the script process your PDF files and store additional multimodal chunks. The output indicates how many images and tables were extracted and described.

Not seeing multimodal chunks?

If the script runs but reports 0 images or tables, your test PDF may not contain embedded images or structured tables. Try using a research paper PDF that contains figures and result tables.

If you see a rate limit error from Groq, wait a minute and re-run. The free tier has request-per-minute limits that reset quickly.

help me debug multimodal extraction errors

Test multimodal queries and update the benchmark

The multimodal chunks are now stored in ChromaDB. You need to update the table_analysis_node in your agent to specifically query for these chunks using a metadata filter. This ensures table and chart questions retrieve the multimodal descriptions.

• Open src/agent.py and find the table_analysis_node function.
• Update the retrieval call inside table_analysis_node to filter for multimodal chunks using a metadata where clause:

# Query ChromaDB for multimodal chunks (images and tables)
multimodal_results = collection.query(
    query_texts=[state["messages"][-1].content],
    n_results=5,
    where={"content_type": {"$in": ["image", "table"]}}
)

What does this code do?

The where clause filters ChromaDB results to only return chunks with content_type set to "image" or "table". This means the Table Analysis handler retrieves image descriptions and table markdown specifically, rather than generic text passages.

• Save src/agent.py.
• Test the multimodal pipeline with a question that requires chart or table understanding by running:

# test_multimodal.py
from src.agent import graph
from langchain_core.messages import HumanMessage

# Test with a table/chart question
config = {"configurable": {"thread_id": "multimodal-test"}}
response = graph.invoke(
    {"messages": [HumanMessage(content="What were the accuracy scores reported in the paper?")]},
    config=config
)

print(response["messages"][-1].content)

• Run the test script:

python test_multimodal.py

You should see the agent route the question to the Table Analysis handler, retrieve multimodal chunks describing tables or figures, and generate an answer grounded in visual content from the PDF.

• Add the multimodal test results to your benchmark report. Open evals/benchmark_report.md and append a new section:

## Multimodal Understanding Results

| Query Type | Routed To | Retrieved Multimodal Chunks | Answer Quality |
|------------|-----------|----------------------------|----------------|
| Table data question | Table Analysis | Yes (table markdown) | Accurate |
| Chart interpretation | Table Analysis | Yes (image description) | Accurate |
| Text-only question | Retrieval | No (text chunks only) | Accurate |

• Save evals/benchmark_report.md.

Agent not routing to Table Analysis?

If questions about tables route to the Retrieval handler instead of Table Analysis, make your question more explicit about tabular data. Phrases like "What were the numerical results" or "Show me the comparison table" trigger the Table Analysis intent more reliably.

help me debug my agent's routing for table questions

✔️ Awesome, I've got everything!

Great. Double check you have saved src/multimodal.py, src/ingest.py, and src/agent.py.

ⓧ I'd like to double check the full code

Here is the complete src/multimodal.py file:

import fitz  # PyMuPDF
import base64
import os
from groq import Groq


def extract_images_from_pdf(pdf_path):
    """Extract embedded images from PDF pages as base64."""
    images = []
    doc = fitz.open(pdf_path)

    for page_num in range(len(doc)):
        page = doc[page_num]
        # Get all images embedded on this page
        image_list = page.get_images(full=True)

        for img_index, img_info in enumerate(image_list):
            # Extract raw image bytes using the xref ID
            xref = img_info[0]
            base_image = doc.extract_image(xref)
            image_bytes = base_image["image"]
            # Encode as base64 for the multimodal API
            base64_data = base64.b64encode(image_bytes).decode("utf-8")

            images.append({
                "page_num": page_num + 1,
                "image_index": img_index,
                "base64_data": base64_data
            })

    doc.close()
    return images


def extract_tables_from_pdf(pdf_path):
    """Extract table structures from PDF pages using PyMuPDF table detection."""
    tables = []
    doc = fitz.open(pdf_path)

    for page_num in range(len(doc)):
        page = doc[page_num]
        # Use PyMuPDF built-in table detection (v1.23.0+)
        page_tables = page.find_tables()

        for table_index, table in enumerate(page_tables):
            # Convert to markdown for LLM-friendly token-efficient format
            table_text = table.to_markdown()

            tables.append({
                "page_num": page_num + 1,
                "table_index": table_index,
                "table_text": table_text
            })

    doc.close()
    return tables


def describe_image(base64_data, page_context, llm=None):
    """Generate a text description of an image using Groq Llama 4 Scout."""
    client = Groq(api_key=os.getenv("GROQ_API_KEY"))

    # Send the base64 image to the multimodal model for description
    response = client.chat.completions.create(
        model="meta-llama/llama-4-scout-17b-16e-instruct",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": f"Describe this image in detail for document search purposes. "
                                f"Page context: {page_context}. "
                                f"Include all visible data, labels, axes, legends, and key takeaways."
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/png;base64,{base64_data}"
                        }
                    }
                ]
            }
        ],
        max_completion_tokens=1024
    )

    return response.choices[0].message.content


def extract_multimodal_content(pdf_path, llm=None):
    """Extract and describe all multimodal content from a PDF."""
    enriched_chunks = []

    # Generate text descriptions for each extracted image
    images = extract_images_from_pdf(pdf_path)
    for img in images:
        page_context = f"Page {img['page_num']} of {os.path.basename(pdf_path)}"
        description = describe_image(img["base64_data"], page_context, llm)

        enriched_chunks.append({
            "text": description,
            "metadata": {
                "source": pdf_path,
                "page_num": img["page_num"],
                "content_type": "image",
                "image_index": img["image_index"]
            }
        })

    # Convert extracted tables into searchable text chunks
    tables = extract_tables_from_pdf(pdf_path)
    for tbl in tables:
        enriched_chunks.append({
            "text": f"Table from page {tbl['page_num']}:\n{tbl['table_text']}",
            "metadata": {
                "source": pdf_path,
                "page_num": tbl["page_num"],
                "content_type": "table",
                "table_index": tbl["table_index"]
            }
        })

    return enriched_chunks

Your RAG system now understands visual content from PDFs. Next up, you will wrap the entire system in a production-ready FastAPI backend with a polished Gradio chat interface.

Build the Production API and Gradio UI

Your RAG agent can route questions, remember conversations, and stream answers. Now you need to wrap it in a production-ready interface so other people can actually use it.

In this step, you will build two layers on top of your agent. First, a FastAPI backend with REST endpoints for document ingestion, querying, and evaluation. Second, a Gradio chat interface with streaming responses, file upload, and a metrics dashboard.

In this step, get ready to:

• Build a FastAPI backend with endpoints for ingestion, querying, evaluation, and health checks.
• Build a Gradio chat UI with streaming, file upload, citation cards, and a metrics tab.
• Test the full flow from PDF upload to streamed answer with citations.

Build FastAPI endpoints

The API layer exposes your agent as a service. Any frontend or external application can call these endpoints to ingest documents and ask questions.

• Create the API file by running:

touch src/api.py

• Open src/api.py in VS Code and add the imports and Pydantic models:

import os
import json
import logging
import tempfile
from typing import AsyncGenerator

from fastapi import FastAPI, UploadFile, WebSocket, WebSocketDisconnect
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from dotenv import load_dotenv

from src.ingest import ingest_pdf
from src.agent import build_graph
from src.pipeline import run_pipeline

# Load environment variables
load_dotenv()

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Initialize FastAPI app
app = FastAPI(title="RAG System API")

# Build the agent graph once at startup
graph = build_graph()


# --- Pydantic models ---
class QueryRequest(BaseModel):
    """Request body for the /query endpoint."""
    question: str
    conversation_id: str = "default"


class QueryResponse(BaseModel):
    """Response body for the /query endpoint."""
    answer: str
    citations: list[dict]
    intent: str

What does this code do?

You define the FastAPI app and two Pydantic models that validate incoming requests and outgoing responses.

The QueryRequest model accepts a question and an optional conversation ID for memory. The QueryResponse model structures the answer with citations and the detected intent.

• Below the models, add the four API endpoints:

# --- Endpoints ---

@app.get("/health")
async def health_check():
    """Health check endpoint for monitoring."""
    return {"status": "ok"}


@app.post("/ingest")
async def ingest_document(file: UploadFile):
    """Accept a PDF upload, process and index it."""
    logger.info(f"Ingesting file: {file.filename}")

    # Save uploaded file to a temp location
    with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as tmp:
        content = await file.read()
        tmp.write(content)
        tmp_path = tmp.name

    try:
        # Run the ingestion pipeline
        chunk_count = ingest_pdf(tmp_path)
        logger.info(f"Ingested {chunk_count} chunks from {file.filename}")
        return {"filename": file.filename, "chunks_indexed": chunk_count}
    finally:
        os.unlink(tmp_path)


@app.post("/query")
async def query_documents(request: QueryRequest):
    """Accept a question and return a streamed answer with citations."""
    logger.info(f"Query: {request.question} (thread: {request.conversation_id})")

    async def generate() -> AsyncGenerator[str, None]:
        """Stream tokens from the agent."""
        config = {"configurable": {"thread_id": request.conversation_id}}
        full_response = ""
        intent = ""
        citations = []

        for event in graph.stream(
            {"messages": [{"role": "user", "content": request.question}]},
            config=config,
            stream_mode="updates"
        ):
            # Extract streamed tokens and metadata
            for node_name, node_output in event.items():
                if "intent" in node_output:
                    intent = node_output["intent"]
                if "messages" in node_output:
                    for msg in node_output["messages"]:
                        if hasattr(msg, "content"):
                            token = msg.content
                            full_response += token
                            yield token
                if "citations" in node_output:
                    citations = node_output["citations"]

        # Yield final metadata as JSON
        yield "\n\n" + json.dumps({
            "_meta": {"intent": intent, "citations": citations}
        })

    return StreamingResponse(generate(), media_type="text/plain")


@app.get("/evaluate")
async def evaluate_pipeline():
    """Run RAGAS evaluation on recent queries and return metrics."""
    logger.info("Running RAGAS evaluation")
    try:
        metrics = run_pipeline("evaluate")
        return {"metrics": metrics}
    except Exception as e:
        logger.error(f"Evaluation failed: {e}")
        return {"error": str(e)}

What does this code do?

• `/health` returns a simple status check for monitoring tools.
• `/ingest` accepts a PDF file upload, saves it temporarily, runs the ingestion pipeline, and returns the number of chunks indexed.
• `/query` streams the agent's response token-by-token using FastAPI's `StreamingResponse`. It appends citation metadata at the end.
• `/evaluate` triggers RAGAS evaluation and returns the metrics dictionary.

• Finally, add the WebSocket endpoint for real-time streaming to the frontend:

# --- WebSocket for real-time streaming ---

@app.websocket("/ws/query")
async def websocket_query(websocket: WebSocket):
    """Stream tokens to the frontend via WebSocket."""
    await websocket.accept()
    logger.info("WebSocket connection established")

    try:
        while True:
            # Receive query as JSON
            data = await websocket.receive_text()
            request_data = json.loads(data)
            question = request_data.get("question", "")
            conversation_id = request_data.get("conversation_id", "default")

            config = {"configurable": {"thread_id": conversation_id}}

            # Stream agent response
            for event in graph.stream(
                {"messages": [{"role": "user", "content": question}]},
                config=config,
                stream_mode="updates"
            ):
                for node_name, node_output in event.items():
                    if "messages" in node_output:
                        for msg in node_output["messages"]:
                            if hasattr(msg, "content"):
                                await websocket.send_text(msg.content)

            # Signal end of response
            await websocket.send_text("[END]")

    except WebSocketDisconnect:
        logger.info("WebSocket disconnected")

What does this code do?

The WebSocket endpoint at /ws/query accepts a persistent connection. When the client sends a JSON message with a question, the server streams tokens back in real-time. A [END] signal tells the client the response is complete.

• Save src/api.py.
• Test that the API starts correctly by running:

uvicorn src.api:app --reload --port 8000

You should see output confirming the server is running on port 8000.

Server won't start?

• Make sure uvicorn is in your requirements.txt and installed in your virtual environment.
• Check that your .env file contains a valid GROQ_API_KEY value.
• If you see import errors, make sure you are running the command from the rag-system/ directory.

Having trouble?

Build the Gradio chat UI

Gradio gives you a polished chat interface with minimal code. You will build a tabbed layout with a chat panel, file upload sidebar, and a metrics dashboard.

• Create the Gradio app file by running:

touch src/app.py

• Open src/app.py and add the imports and helper functions:

import gradio as gr
import pandas as pd
from dotenv import load_dotenv

from src.agent import build_graph
from src.ingest import ingest_pdf

# Load environment variables
load_dotenv()

# Build the agent graph
graph = build_graph()

# Intent emoji mapping for router decisions
INTENT_EMOJIS = {
    "retrieval": "\U0001F50D",      # magnifying glass
    "summarization": "\U0001F4DD",  # memo
    "comparison": "\u2696\uFE0F",    # balance scale
    "table_analysis": "\U0001F4CA"  # bar chart
}


def process_upload(file):
    """Handle PDF file upload and ingestion."""
    if file is None:
        return "No file uploaded."
    chunk_count = ingest_pdf(file.name)
    return f"Indexed {chunk_count} chunks from {file.name.split('/')[-1]}"

What does this code do?

You import Gradio and your existing agent modules. The INTENT_EMOJIS dictionary maps each router path to a visual indicator so users see which mode the agent chose. The process_upload function handles file uploads by calling the same ingestion pipeline the API uses.

• Below the helpers, add the chat function that streams agent responses:

def chat(message, history):
    """Chat function for Gradio ChatInterface with streaming."""
    config = {"configurable": {"thread_id": "gradio-session"}}
    full_response = ""
    intent = ""
    citations = []

    # Stream the agent response
    for event in graph.stream(
        {"messages": [{"role": "user", "content": message}]},
        config=config,
        stream_mode="updates"
    ):
        for node_name, node_output in event.items():
            # Capture intent from router
            if "intent" in node_output:
                intent = node_output["intent"]
            # Stream message content
            if "messages" in node_output:
                for msg in node_output["messages"]:
                    if hasattr(msg, "content") and msg.content:
                        full_response += msg.content
                        # Prefix with intent emoji on first token
                        emoji = INTENT_EMOJIS.get(intent, "")
                        prefix = f"{emoji} Mode: {intent.title()}\n\n" if intent else ""
                        yield prefix + full_response
            # Capture citations
            if "citations" in node_output:
                citations = node_output["citations"]

    # Append citation cards at the end
    if citations:
        citation_text = "\n\n---\n**Citations:**\n"
        for i, cite in enumerate(citations, 1):
            source = cite.get("source", "Unknown")
            page = cite.get("page_number", "?")
            passage = cite.get("passage", "")[:150]
            verified = "Verified" if cite.get("verified", False) else "Unverified"
            citation_text += f"\n[{i}] **{source}** (p.{page}) - {verified}\n> {passage}...\n"
        emoji = INTENT_EMOJIS.get(intent, "")
        prefix = f"{emoji} Mode: {intent.title()}\n\n" if intent else ""
        yield prefix + full_response + citation_text

What does this code do?

The chat function is a Python generator. It yields partial responses as the agent streams tokens, giving users real-time feedback like ChatGPT.

After streaming completes, it appends formatted citation cards showing the document name, page number, passage, and verification status for each source.

• Now add the main Gradio Blocks layout with tabs for chat and metrics:

def load_benchmark_data():
    """Load benchmark report as a DataFrame for the metrics tab."""
    try:
        data = {
            "Pipeline": ["Basic RAG", "Hybrid Search", "Hybrid + Reranking", "Agentic (Full)"],
            "Faithfulness": ["-", "-", "-", "-"],
            "Context Precision": ["-", "-", "-", "-"],
            "Answer Relevancy": ["-", "-", "-", "-"],
            "Citation Accuracy": ["-", "-", "-", "-"]
        }
        # Try to load actual results from benchmark report
        if pd.io.common.file_exists("evals/benchmark_report.md"):
            # Parse markdown table if it exists
            pass
        return pd.DataFrame(data)
    except Exception:
        return pd.DataFrame()


# --- Build the Gradio interface ---
with gr.Blocks(title="Document Intelligence Agent") as demo:
    gr.Markdown("# Document Intelligence Agent")
    gr.Markdown("Upload PDFs and ask questions with intelligent routing, citations, and streaming.")

    with gr.Tab("Chat"):
        with gr.Row():
            # Sidebar for file upload
            with gr.Column(scale=1):
                gr.Markdown("### Upload Documents")
                file_upload = gr.File(
                    label="Upload PDF",
                    file_types=[".pdf"],
                    type="filepath"
                )
                upload_status = gr.Textbox(
                    label="Status",
                    interactive=False
                )
                file_upload.upload(
                    fn=process_upload,
                    inputs=file_upload,
                    outputs=upload_status
                )

            # Main chat area
            with gr.Column(scale=3):
                gr.ChatInterface(
                    fn=chat,
                    title="Ask Your Documents",
                    examples=[
                        "What methods were used in this paper?",
                        "Summarize the key findings.",
                        "Compare the approaches in the uploaded documents.",
                        "What were the accuracy scores?"
                    ]
                )

    with gr.Tab("Metrics"):
        gr.Markdown("### RAG Pipeline Benchmark Results")
        benchmark_table = gr.DataFrame(
            value=load_benchmark_data(),
            label="Pipeline Comparison"
        )
        gr.Markdown("*Run the /evaluate endpoint or re-run benchmarks to populate with real scores.*")


if __name__ == "__main__":
    demo.launch(server_name="0.0.0.0", server_port=7860)

What does this code do?

• The gr.Blocks layout creates a tabbed app. The **Chat** tab has a sidebar for PDF uploads and a main area with the streaming chat interface.
• The gr.File component triggers process_upload when a user uploads a PDF. The status textbox shows how many chunks were indexed.
• The **Metrics** tab displays a DataFrame with your RAGAS benchmark scores for each pipeline variant.
• The gr.ChatInterface wraps your chat generator function and handles streaming display automatically.

• Save src/app.py.
• Launch the Gradio app by running:

python src/app.py

You should see output showing the Gradio app running on port 7860. Open the URL in your browser to see the chat interface.

Gradio not launching?

• Make sure gradio and pandas are installed in your virtual environment.
• If you see a port conflict, stop the FastAPI server first or change the port in demo.launch().
• Check that your src/agent.py builds the graph without errors by importing it in a Python shell.

Still stuck?

Add production features and test the full flow

With both files created, you need to verify the complete user journey works end to end. This confirms that uploading a PDF, asking a question, receiving a streamed answer with citations, and asking a follow-up question all function correctly.

• Make sure the Gradio app is running. Open it in your browser at http://localhost:7860.
• In the sidebar, click the file upload area and upload one of the sample PDFs from your data/ folder.
• Wait for the status box to show a message like "Indexed 42 chunks from paper.pdf".
• In the chat box, type a question like "What methods were used in this paper?" and press Enter.
• Watch the response stream in token by token. You should see the router mode indicator at the top followed by the answer and citation cards at the bottom.
• Ask a follow-up question like "How does it compare to the baseline?" to verify conversation memory.
• Click the **Metrics** tab to confirm the benchmark table renders.

What should I see?

Each response starts with a mode indicator (a magnifying glass for retrieval, a memo for summarization, scales for comparison, or a bar chart for table analysis). The answer streams in progressively. After the answer completes, citation cards appear below with the document name, page number, and quoted passage.

Follow-up questions should reference context from the previous answer without you needing to re-state the topic.

Not seeing citations?

• Make sure your agent's state includes a citations field that gets populated by the generation node.
• Verify that you uploaded a PDF before asking the question. Without indexed documents, the agent has no sources to cite.
• Check the terminal running the Gradio app for any error messages from the agent.

Need help?

✔️ Awesome, I've got everything!

Great. Make sure both src/api.py and src/app.py are saved.

ⓧ I'd like to double check the full code

Here is the complete src/api.py:

import os
import json
import logging
import tempfile
from typing import AsyncGenerator

from fastapi import FastAPI, UploadFile, WebSocket, WebSocketDisconnect
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from dotenv import load_dotenv

from src.ingest import ingest_pdf
from src.agent import build_graph
from src.pipeline import run_pipeline

# Load environment variables
load_dotenv()

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Initialize FastAPI app
app = FastAPI(title="RAG System API")

# Build the agent graph once at startup
graph = build_graph()


# --- Pydantic models ---
class QueryRequest(BaseModel):
    """Request body for the /query endpoint."""
    question: str
    conversation_id: str = "default"


class QueryResponse(BaseModel):
    """Response body for the /query endpoint."""
    answer: str
    citations: list[dict]
    intent: str


# --- Endpoints ---

@app.get("/health")
async def health_check():
    """Health check endpoint for monitoring."""
    return {"status": "ok"}


@app.post("/ingest")
async def ingest_document(file: UploadFile):
    """Accept a PDF upload, process and index it."""
    logger.info(f"Ingesting file: {file.filename}")

    # Save uploaded file to a temp location
    with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as tmp:
        content = await file.read()
        tmp.write(content)
        tmp_path = tmp.name

    try:
        # Run the ingestion pipeline
        chunk_count = ingest_pdf(tmp_path)
        logger.info(f"Ingested {chunk_count} chunks from {file.filename}")
        return {"filename": file.filename, "chunks_indexed": chunk_count}
    finally:
        os.unlink(tmp_path)


@app.post("/query")
async def query_documents(request: QueryRequest):
    """Accept a question and return a streamed answer with citations."""
    logger.info(f"Query: {request.question} (thread: {request.conversation_id})")

    async def generate() -> AsyncGenerator[str, None]:
        """Stream tokens from the agent."""
        config = {"configurable": {"thread_id": request.conversation_id}}
        full_response = ""
        intent = ""
        citations = []

        for event in graph.stream(
            {"messages": [{"role": "user", "content": request.question}]},
            config=config,
            stream_mode="updates"
        ):
            # Extract streamed tokens and metadata
            for node_name, node_output in event.items():
                if "intent" in node_output:
                    intent = node_output["intent"]
                if "messages" in node_output:
                    for msg in node_output["messages"]:
                        if hasattr(msg, "content"):
                            token = msg.content
                            full_response += token
                            yield token
                if "citations" in node_output:
                    citations = node_output["citations"]

        # Yield final metadata as JSON
        yield "\n\n" + json.dumps({
            "_meta": {"intent": intent, "citations": citations}
        })

    return StreamingResponse(generate(), media_type="text/plain")


@app.get("/evaluate")
async def evaluate_pipeline():
    """Run RAGAS evaluation on recent queries and return metrics."""
    logger.info("Running RAGAS evaluation")
    try:
        metrics = run_pipeline("evaluate")
        return {"metrics": metrics}
    except Exception as e:
        logger.error(f"Evaluation failed: {e}")
        return {"error": str(e)}


# --- WebSocket for real-time streaming ---

@app.websocket("/ws/query")
async def websocket_query(websocket: WebSocket):
    """Stream tokens to the frontend via WebSocket."""
    await websocket.accept()
    logger.info("WebSocket connection established")

    try:
        while True:
            # Receive query as JSON
            data = await websocket.receive_text()
            request_data = json.loads(data)
            question = request_data.get("question", "")
            conversation_id = request_data.get("conversation_id", "default")

            config = {"configurable": {"thread_id": conversation_id}}

            # Stream agent response
            for event in graph.stream(
                {"messages": [{"role": "user", "content": question}]},
                config=config,
                stream_mode="updates"
            ):
                for node_name, node_output in event.items():
                    if "messages" in node_output:
                        for msg in node_output["messages"]:
                            if hasattr(msg, "content"):
                                await websocket.send_text(msg.content)

            # Signal end of response
            await websocket.send_text("[END]")

    except WebSocketDisconnect:
        logger.info("WebSocket disconnected")

And here is the complete src/app.py:

import gradio as gr
import pandas as pd
from dotenv import load_dotenv

from src.agent import build_graph
from src.ingest import ingest_pdf

# Load environment variables
load_dotenv()

# Build the agent graph
graph = build_graph()

# Intent emoji mapping for router decisions
INTENT_EMOJIS = {
    "retrieval": "\U0001F50D",      # magnifying glass
    "summarization": "\U0001F4DD",  # memo
    "comparison": "\u2696\uFE0F",    # balance scale
    "table_analysis": "\U0001F4CA"  # bar chart
}


def process_upload(file):
    """Handle PDF file upload and ingestion."""
    if file is None:
        return "No file uploaded."
    chunk_count = ingest_pdf(file.name)
    return f"Indexed {chunk_count} chunks from {file.name.split('/')[-1]}"


def chat(message, history):
    """Chat function for Gradio ChatInterface with streaming."""
    config = {"configurable": {"thread_id": "gradio-session"}}
    full_response = ""
    intent = ""
    citations = []

    # Stream the agent response
    for event in graph.stream(
        {"messages": [{"role": "user", "content": message}]},
        config=config,
        stream_mode="updates"
    ):
        for node_name, node_output in event.items():
            # Capture intent from router
            if "intent" in node_output:
                intent = node_output["intent"]
            # Stream message content
            if "messages" in node_output:
                for msg in node_output["messages"]:
                    if hasattr(msg, "content") and msg.content:
                        full_response += msg.content
                        # Prefix with intent emoji on first token
                        emoji = INTENT_EMOJIS.get(intent, "")
                        prefix = f"{emoji} Mode: {intent.title()}\n\n" if intent else ""
                        yield prefix + full_response
            # Capture citations
            if "citations" in node_output:
                citations = node_output["citations"]

    # Append citation cards at the end
    if citations:
        citation_text = "\n\n---\n**Citations:**\n"
        for i, cite in enumerate(citations, 1):
            source = cite.get("source", "Unknown")
            page = cite.get("page_number", "?")
            passage = cite.get("passage", "")[:150]
            verified = "Verified" if cite.get("verified", False) else "Unverified"
            citation_text += f"\n[{i}] **{source}** (p.{page}) - {verified}\n> {passage}...\n"
        emoji = INTENT_EMOJIS.get(intent, "")
        prefix = f"{emoji} Mode: {intent.title()}\n\n" if intent else ""
        yield prefix + full_response + citation_text


def load_benchmark_data():
    """Load benchmark report as a DataFrame for the metrics tab."""
    try:
        data = {
            "Pipeline": ["Basic RAG", "Hybrid Search", "Hybrid + Reranking", "Agentic (Full)"],
            "Faithfulness": ["-", "-", "-", "-"],
            "Context Precision": ["-", "-", "-", "-"],
            "Answer Relevancy": ["-", "-", "-", "-"],
            "Citation Accuracy": ["-", "-", "-", "-"]
        }
        # Try to load actual results from benchmark report
        if pd.io.common.file_exists("evals/benchmark_report.md"):
            # Parse markdown table if it exists
            pass
        return pd.DataFrame(data)
    except Exception:
        return pd.DataFrame()


# --- Build the Gradio interface ---
with gr.Blocks(title="Document Intelligence Agent") as demo:
    gr.Markdown("# Document Intelligence Agent")
    gr.Markdown("Upload PDFs and ask questions with intelligent routing, citations, and streaming.")

    with gr.Tab("Chat"):
        with gr.Row():
            # Sidebar for file upload
            with gr.Column(scale=1):
                gr.Markdown("### Upload Documents")
                file_upload = gr.File(
                    label="Upload PDF",
                    file_types=[".pdf"],
                    type="filepath"
                )
                upload_status = gr.Textbox(
                    label="Status",
                    interactive=False
                )
                file_upload.upload(
                    fn=process_upload,
                    inputs=file_upload,
                    outputs=upload_status
                )

            # Main chat area
            with gr.Column(scale=3):
                gr.ChatInterface(
                    fn=chat,
                    title="Ask Your Documents",
                    examples=[
                        "What methods were used in this paper?",
                        "Summarize the key findings.",
                        "Compare the approaches in the uploaded documents.",
                        "What were the accuracy scores?"
                    ]
                )

    with gr.Tab("Metrics"):
        gr.Markdown("### RAG Pipeline Benchmark Results")
        benchmark_table = gr.DataFrame(
            value=load_benchmark_data(),
            label="Pipeline Comparison"
        )
        gr.Markdown("*Run the /evaluate endpoint or re-run benchmarks to populate with real scores.*")


if __name__ == "__main__":
    demo.launch(server_name="0.0.0.0", server_port=7860)

Your RAG system now has a complete production interface. Users can upload documents, ask questions, and receive streamed answers with verifiable citations. Next, you will create a professional GitHub README to showcase this project.

Create a Professional GitHub README

Your RAG system is fully built and evaluated, but code alone does not land interviews. A professional README transforms your repository from "some Python files" into a portfolio piece that recruiters and hiring managers immediately understand.

In this step, you will create architecture diagrams that visualize how the entire system fits together, document your benchmark results in a format that screams "production-grade," and assemble a README that tells the full story of what you built.

In this step, get ready to:

• Create architecture, retrieval flow, and routing diagrams using Mermaid.
• Write a professional README with benchmarks, tech stack, and live demo link.
• Commit the complete documentation to Git.

Create architecture and flow diagrams

Diagrams give readers an instant mental model of your system. You will create three diagrams using Mermaid syntax and export them as PNG images.

• Create the docs/ directory by running this command:

mkdir -p docs

Mermaid is a text-based diagramming tool. You write diagram code, paste it into the Mermaid Live Editor, and export the rendered image as a PNG file.

• Open the Mermaid Live Editor in your browser:

# Open this URL in your browser
# https://mermaid.live

• Copy the following system architecture diagram code and paste it into the Code panel on the left side of the editor:

graph LR
    A[PDF Documents] --> B[Document Ingestion]
    B --> C[Text Chunking]
    C --> D[Embedding\nall-MiniLM-L6-v2]
    D --> E[(ChromaDB)]
    C --> F[BM25 Index]

    G[User Question] --> H[Hybrid Search]
    E --> H
    F --> H
    H --> I[Cross-Encoder\nReranking]
    I --> J[LangGraph Agent]
    J --> K[Groq LLM\nLlama 3.3 70B]
    K --> L[Citation\nVerification]
    L --> M[Response with\nInline Citations]

What does this diagram show?

This is a left-to-right flowchart showing the complete data pipeline. PDF documents flow through ingestion, chunking, and embedding into ChromaDB.

User questions go through hybrid search (combining BM25 keyword matching with vector similarity), then cross-encoder reranking, before reaching the LangGraph agent and Groq LLM. The final output includes verified inline citations.

• In the Mermaid Live Editor, click Actions in the top menu.
• Click export PNG to download the rendered diagram.
• Save the downloaded file as architecture.png in your docs/ folder.

• Clear the editor and paste this retrieval flow diagram:

graph TD
    A[User Question] --> B[Intent Router]
    B --> C{Classified Intent}
    C -->|Retrieval| D[Hybrid Search + Rerank]
    C -->|Summarization| E[Broad Context Retrieval]
    C -->|Comparison| F[Multi-Document Search]
    C -->|Table Analysis| G[Table-Focused Extraction]

    D --> H[Grade Documents]
    E --> H
    F --> H
    G --> H

    H -->|Relevant| I[Generate Answer]
    H -->|Irrelevant| J[Rewrite Query]
    J --> D

    I --> K[Citation Verification]
    K --> L[Response with Sources]

• Export this diagram as PNG and save it as retrieval_flow.png in your docs/ folder.

• Clear the editor once more and paste this query routing diagram:

graph TD
    A[User Question] --> B[Intent Router]

    B --> C["Retrieval\n'What method did they use?'\n'What accuracy did model X achieve?'"]
    B --> D["Summarization\n'Summarize chapter 3'\n'Give me an overview of the results'"]
    B --> E["Comparison\n'How do Paper A and B differ?'\n'Compare the two approaches'"]
    B --> F["Table Analysis\n'What were the accuracy scores?'\n'Show me the performance metrics'"]

    C --> G[Hybrid Search + Rerank + Citations]
    D --> H[Broad Context + Concise Summary]
    E --> I[Multi-Source + Structured Comparison]
    F --> J[Table Extraction + Numerical Reasoning]

• Export this as PNG and save it as routing_diagram.png in your docs/ folder.

• Confirm all three diagram files exist by running:

ls docs/

You should see architecture.png, retrieval_flow.png, and routing_diagram.png listed in the output.

Diagrams not exporting?

Make sure there are no syntax errors in the Mermaid code. The Live Editor shows a red error message below the Code panel if something is wrong.

If the export button does not respond, try a different browser (Chrome works best with the Mermaid Live Editor).

help me export my Mermaid diagrams as PNG files

Write the README with evaluation results and benchmarks

A great README tells the story of your project in under 30 seconds of scrolling. Recruiters scan for: what it does, what tech you used, what results you got, and whether they can try it live.

• Create README.md at the root of your rag-system/ folder and paste the following content:

# Document Intelligence Agent

![Python](https://img.shields.io/badge/Python-3.10%2B-blue)
![License](https://img.shields.io/badge/License-MIT-green)
![Deployment](https://img.shields.io/badge/Deployed-Hugging%20Face%20Spaces-yellow)
![Demo](https://img.shields.io/badge/Live%20Demo-Click%20Here-brightgreen)

> A production-grade RAG system that ingests PDF documents and answers questions using hybrid search, cross-encoder reranking, agentic query routing, citation grounding, conversation memory, and streaming responses.

**[Live Demo on Hugging Face Spaces](https://huggingface.co/spaces/YOUR_USERNAME/rag-system)** | [Architecture](#architecture) | [Benchmarks](#evaluation-results) | [Quick Start](#quick-start)

---

## Features

- **Hybrid Search**: Combines BM25 keyword matching with dense vector retrieval using Reciprocal Rank Fusion
- **Cross-Encoder Reranking**: Re-scores retrieved passages with `cross-encoder/ms-marco-MiniLM-L-6-v2` for precision ordering
- **Citation Grounding**: Every claim includes inline citations verified against source passages
- **Agentic Query Routing**: LangGraph state graph routes questions to specialized handlers (Retrieval, Summarization, Comparison, Table Analysis)
- **Conversation Memory**: Maintains context across multi-turn interactions with sliding window
- **Streaming Responses**: Token-by-token output with real-time status updates
- **Multimodal PDF Understanding**: Extracts and reasons over images and tables from documents
- **RAGAS Evaluation**: Quantified metrics (Faithfulness, Context Precision, Answer Relevancy, Citation Accuracy)

---

## Tech Stack

| Component | Technology |
|-----------|------------|
| LLM | Groq (Llama 3.3 70B Versatile) |
| Embeddings | sentence-transformers (all-MiniLM-L6-v2, 384-dim) |
| Vector Store | ChromaDB (persistent, local) |
| Keyword Search | rank-bm25 |
| Reranker | cross-encoder/ms-marco-MiniLM-L-6-v2 |
| Agent Framework | LangGraph (StateGraph, conditional edges) |
| Evaluation | RAGAS (Faithfulness, Context Precision, Relevancy) |
| Backend | FastAPI (REST + WebSocket) |
| Frontend | Gradio (ChatInterface, streaming) |
| PDF Parsing | PyMuPDF (text, images, tables) |
| Deployment | Hugging Face Spaces (Docker SDK) |

---

## Quick Start

```bash
# Clone the repository
git clone https://github.com/YOUR_USERNAME/rag-system.git
cd rag-system

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Set up environment variables
cp .env.example .env
# Add your GROQ_API_KEY to .env

# Run the application
python -m src.app
```

The Gradio interface will open at `http://localhost:7860`.

---

## Project Structure

```
rag-system/
├── src/
│   ├── ingest.py            # PDF loading and text chunking
│   ├── embeddings.py        # Sentence-transformer embedding generation
│   ├── retriever.py         # ChromaDB vector retriever
│   ├── bm25_retriever.py    # BM25 keyword search index
│   ├── hybrid_retriever.py  # EnsembleRetriever with RRF
│   ├── reranker.py          # Cross-encoder reranking pipeline
│   ├── citations.py         # Citation extraction and verification
│   ├── pipeline.py          # End-to-end retrieval pipeline
│   ├── agent.py             # LangGraph agentic state graph
│   ├── multimodal.py        # Image/table extraction from PDFs
│   ├── api.py               # FastAPI REST + WebSocket endpoints
│   └── app.py               # Gradio chat interface
├── evals/
│   └── benchmark_report.md  # RAGAS evaluation results
├── docs/
│   ├── architecture.png     # System architecture diagram
│   ├── retrieval_flow.png   # Retrieval pipeline flow
│   └── routing_diagram.png  # Intent routing paths
├── data/                    # PDF documents for testing
├── tests/                   # Unit and integration tests
├── requirements.txt
├── Dockerfile
├── .env.example
└── README.md
```

---

## How It Works

The system processes documents through three stages:

1. **Ingestion**: PDFs are loaded with PyMuPDF, split into overlapping chunks (512 chars, 50 overlap), and embedded with all-MiniLM-L6-v2 into ChromaDB. A parallel BM25 index is built for keyword search.

2. **Retrieval**: User questions pass through an intent router that classifies them into one of four paths. Hybrid search (BM25 + vector) retrieves candidate passages, then a cross-encoder reranks them for precision.

3. **Generation**: The LangGraph agent grades retrieved documents for relevance, rewrites the query if needed, and generates a cited response. Every factual claim includes an inline citation verified against the source passage.

---

## Architecture

![System Architecture](docs/architecture.png)

### Retrieval Flow

![Retrieval Flow](docs/retrieval_flow.png)

### Query Routing

![Query Routing](docs/routing_diagram.png)

---

## Evaluation Results

### RAGAS Benchmark Comparison

| Pipeline | Faithfulness | Context Precision | Answer Relevancy | Citation Accuracy |
|----------|:---:|:---:|:---:|:---:|
| Basic RAG | 0.72 | 0.65 | 0.78 | - |
| Hybrid Search | 0.79 | 0.81 | 0.82 | - |
| Hybrid + Reranking | 0.85 | 0.89 | 0.86 | 0.94 |
| Agentic (Full System) | 0.88 | 0.91 | 0.89 | 0.96 |

**Key Improvements:**
- Context Precision: **+40%** (Basic → Agentic)
- Faithfulness: **+22%** (Basic → Agentic)
- Citation Accuracy: **96%** of inline citations verified as grounded in source text

### Latency Comparison

| Pipeline | Avg Response Time |
|----------|:---:|
| Basic RAG | ~1.2s |
| Hybrid Search | ~1.8s |
| Hybrid + Reranking | ~2.4s |
| Agentic (Full System) | ~3.1s |

*Measured on Groq API with llama-3.3-70b-versatile. Streaming provides first-token latency of ~0.3s.*

---

## Screenshots

### Chat Interface
![Chat Interface](docs/screenshots/chat_interface.png)

### Citation Cards
![Citation Cards](docs/screenshots/citation_cards.png)

### Metrics Dashboard
![Metrics Dashboard](docs/screenshots/metrics_dashboard.png)

### Router in Action
![Router Decision](docs/screenshots/router_action.png)

---

## Demo

![App Demo](docs/demo.gif)

*Upload a PDF → Ask a question → See streamed answer with verifiable citations → Ask follow-up questions*

---

## Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

---

## License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.

What makes this README stand out?

The README leads with a live demo link and badges, so anyone landing on the repo immediately sees it is deployed and working.

The benchmark table with quantified improvement percentages proves you can measure and optimize your systems. Interviewers love seeing concrete numbers over vague claims like "improved performance."

The architecture diagrams show you think in systems, not just individual scripts. The tech stack table maps every component to a specific tool, proving you understand the full stack.

Replace placeholder values with your real metrics

The benchmark numbers in this README (0.72, 0.79, etc.) are placeholders. After running your RAGAS evaluations in Step 6, replace them with the actual scores from your evals/benchmark_report.md file.

• Save README.md.

Add screenshots, GIF placeholders, and commit

Your README references screenshot files and a demo GIF. You will create the directory structure for these assets and then commit everything to Git.

• Create the screenshots directory by running:

mkdir -p docs/screenshots

When should I capture the actual screenshots?

You will take the real screenshots after deploying the app in the next step (Step 12). For now, the README references these image paths as placeholders. Once you have the app running live, use your screenshot tool to capture the chat interface, citation cards, metrics dashboard, and router decisions.

For the demo GIF, use a screen recording tool (like the built-in screen recorder on macOS or a free tool like ScreenToGif on Windows) to capture the full flow: upload a PDF, ask a question, and see the streamed answer with citations.

• Create a LICENSE file for the MIT license by running:

cat > LICENSE << 'EOF'
MIT License

Copyright (c) 2026

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
EOF

• Create a .env.example file so new contributors know which environment variables are needed:

cat > .env.example << 'EOF'
GROQ_API_KEY=your-api-key-here
EOF

Why a .env.example file?

Your .gitignore excludes .env from version control (which is correct for keeping secrets safe). The .env.example file shows anyone cloning the repo exactly which variables they need to set, without revealing actual keys.

• Stage and commit all documentation files by running these commands:

git add docs/ README.md LICENSE .env.example
git commit -m "Add professional README with architecture diagrams and benchmarks"

• Verify the commit was recorded by running:

git log --oneline -1

You should see your commit message Add professional README with architecture diagrams and benchmarks as the most recent entry.

Git says nothing to commit?

Make sure you saved all files before running git add. Check which files are untracked with git status and verify the docs/ folder contains your PNG files.

If you forgot to export the PNG files from the Mermaid Live Editor, go back and re-export them before running git add.

help me troubleshoot why Git is not detecting my new files

✔️ Awesome, I've got everything!

Great. Double check you have saved all your files and that git log shows the documentation commit.

ⓧ I'd like to double check the full README

# Document Intelligence Agent

![Python](https://img.shields.io/badge/Python-3.10%2B-blue)
![License](https://img.shields.io/badge/License-MIT-green)
![Deployment](https://img.shields.io/badge/Deployed-Hugging%20Face%20Spaces-yellow)
![Demo](https://img.shields.io/badge/Live%20Demo-Click%20Here-brightgreen)

> A production-grade RAG system that ingests PDF documents and answers questions using hybrid search, cross-encoder reranking, agentic query routing, citation grounding, conversation memory, and streaming responses.

**[Live Demo on Hugging Face Spaces](https://huggingface.co/spaces/YOUR_USERNAME/rag-system)** | [Architecture](#architecture) | [Benchmarks](#evaluation-results) | [Quick Start](#quick-start)

---

## Features

- **Hybrid Search**: Combines BM25 keyword matching with dense vector retrieval using Reciprocal Rank Fusion
- **Cross-Encoder Reranking**: Re-scores retrieved passages with `cross-encoder/ms-marco-MiniLM-L-6-v2` for precision ordering
- **Citation Grounding**: Every claim includes inline citations verified against source passages
- **Agentic Query Routing**: LangGraph state graph routes questions to specialized handlers (Retrieval, Summarization, Comparison, Table Analysis)
- **Conversation Memory**: Maintains context across multi-turn interactions with sliding window
- **Streaming Responses**: Token-by-token output with real-time status updates
- **Multimodal PDF Understanding**: Extracts and reasons over images and tables from documents
- **RAGAS Evaluation**: Quantified metrics (Faithfulness, Context Precision, Answer Relevancy, Citation Accuracy)

---

## Tech Stack

| Component | Technology |
|-----------|------------|
| LLM | Groq (Llama 3.3 70B Versatile) |
| Embeddings | sentence-transformers (all-MiniLM-L6-v2, 384-dim) |
| Vector Store | ChromaDB (persistent, local) |
| Keyword Search | rank-bm25 |
| Reranker | cross-encoder/ms-marco-MiniLM-L-6-v2 |
| Agent Framework | LangGraph (StateGraph, conditional edges) |
| Evaluation | RAGAS (Faithfulness, Context Precision, Relevancy) |
| Backend | FastAPI (REST + WebSocket) |
| Frontend | Gradio (ChatInterface, streaming) |
| PDF Parsing | PyMuPDF (text, images, tables) |
| Deployment | Hugging Face Spaces (Docker SDK) |

---

## Quick Start

```bash
# Clone the repository
git clone https://github.com/YOUR_USERNAME/rag-system.git
cd rag-system

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Set up environment variables
cp .env.example .env
# Add your GROQ_API_KEY to .env

# Run the application
python -m src.app
```

The Gradio interface will open at `http://localhost:7860`.

---

## Project Structure

```
rag-system/
├── src/
│   ├── ingest.py            # PDF loading and text chunking
│   ├── embeddings.py        # Sentence-transformer embedding generation
│   ├── retriever.py         # ChromaDB vector retriever
│   ├── bm25_retriever.py    # BM25 keyword search index
│   ├── hybrid_retriever.py  # EnsembleRetriever with RRF
│   ├── reranker.py          # Cross-encoder reranking pipeline
│   ├── citations.py         # Citation extraction and verification
│   ├── pipeline.py          # End-to-end retrieval pipeline
│   ├── agent.py             # LangGraph agentic state graph
│   ├── multimodal.py        # Image/table extraction from PDFs
│   ├── api.py               # FastAPI REST + WebSocket endpoints
│   └── app.py               # Gradio chat interface
├── evals/
│   └── benchmark_report.md  # RAGAS evaluation results
├── docs/
│   ├── architecture.png     # System architecture diagram
│   ├── retrieval_flow.png   # Retrieval pipeline flow
│   └── routing_diagram.png  # Intent routing paths
├── data/                    # PDF documents for testing
├── tests/                   # Unit and integration tests
├── requirements.txt
├── Dockerfile
├── .env.example
└── README.md
```

---

## How It Works

The system processes documents through three stages:

1. **Ingestion**: PDFs are loaded with PyMuPDF, split into overlapping chunks (512 chars, 50 overlap), and embedded with all-MiniLM-L6-v2 into ChromaDB. A parallel BM25 index is built for keyword search.

2. **Retrieval**: User questions pass through an intent router that classifies them into one of four paths. Hybrid search (BM25 + vector) retrieves candidate passages, then a cross-encoder reranks them for precision.

3. **Generation**: The LangGraph agent grades retrieved documents for relevance, rewrites the query if needed, and generates a cited response. Every factual claim includes an inline citation verified against the source passage.

---

## Architecture

![System Architecture](docs/architecture.png)

### Retrieval Flow

![Retrieval Flow](docs/retrieval_flow.png)

### Query Routing

![Query Routing](docs/routing_diagram.png)

---

## Evaluation Results

### RAGAS Benchmark Comparison

| Pipeline | Faithfulness | Context Precision | Answer Relevancy | Citation Accuracy |
|----------|:---:|:---:|:---:|:---:|
| Basic RAG | 0.72 | 0.65 | 0.78 | - |
| Hybrid Search | 0.79 | 0.81 | 0.82 | - |
| Hybrid + Reranking | 0.85 | 0.89 | 0.86 | 0.94 |
| Agentic (Full System) | 0.88 | 0.91 | 0.89 | 0.96 |

**Key Improvements:**
- Context Precision: **+40%** (Basic → Agentic)
- Faithfulness: **+22%** (Basic → Agentic)
- Citation Accuracy: **96%** of inline citations verified as grounded in source text

### Latency Comparison

| Pipeline | Avg Response Time |
|----------|:---:|
| Basic RAG | ~1.2s |
| Hybrid Search | ~1.8s |
| Hybrid + Reranking | ~2.4s |
| Agentic (Full System) | ~3.1s |

*Measured on Groq API with llama-3.3-70b-versatile. Streaming provides first-token latency of ~0.3s.*

---

## Screenshots

### Chat Interface
![Chat Interface](docs/screenshots/chat_interface.png)

### Citation Cards
![Citation Cards](docs/screenshots/citation_cards.png)

### Metrics Dashboard
![Metrics Dashboard](docs/screenshots/metrics_dashboard.png)

### Router in Action
![Router Decision](docs/screenshots/router_action.png)

---

## Demo

![App Demo](docs/demo.gif)

*Upload a PDF → Ask a question → See streamed answer with verifiable citations → Ask follow-up questions*

---

## Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

---

## License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.

Your repository now tells a complete story. Next up, you will deploy the application to Hugging Face Spaces so the live demo link in your README actually works.

Deploy to Hugging Face Spaces

Your RAG system works locally with all four router paths, citation grounding, conversation memory, and streaming responses. Now it needs a home on the internet.

In this step, you will containerize your application with Docker and deploy it to Hugging Face Spaces for free. This gives you a live, shareable URL that anyone can visit to see your system in action.

In this step, get ready to:

• Create a production Dockerfile and finalize your dependencies.
• Deploy your application to Hugging Face Spaces with a Docker SDK Space.
• Test the live deployment and update your README with the demo URL.

Prepare for deployment

Hugging Face Spaces runs your application inside a Docker container. You need a Dockerfile that tells the platform how to build and start your app.

• Create a file called Dockerfile in the root of your rag-system/ folder by running:

touch Dockerfile

• Open Dockerfile in VS Code and add the following:

# Use Python 3.10 slim as the base image
FROM python:3.10-slim

# Create a non-root user (HF Spaces runs as uid 1000)
RUN useradd -m -u 1000 user

# Set working directory
WORKDIR /app

# Copy and install dependencies first (Docker layer caching)
COPY --chown=user requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code
COPY --chown=user . .

# Create a persistent directory for ChromaDB data
RUN mkdir -p /app/chroma_db && chown user:user /app/chroma_db

# Switch to non-root user
USER user

# Set environment variables for the user
ENV HOME=/home/user \
    PATH=/home/user/.local/bin:$PATH

# Expose port 7860 (required by Hugging Face Spaces)
EXPOSE 7860

# Start the application with uvicorn
CMD ["python", "-m", "uvicorn", "src.api:app", "--host", "0.0.0.0", "--port", "7860"]

What does this Dockerfile do?

The Dockerfile builds your app image in layers for efficiency. It installs dependencies first so they get cached between code changes, then copies your source.

Hugging Face Spaces requires the container to run as user ID 1000. The EXPOSE 7860 line matches the default port that Spaces expects for Docker SDK apps.

The CMD starts your FastAPI server with uvicorn, which also mounts your Gradio interface.

• Now finalize your requirements.txt with all production dependencies. Open requirements.txt and make sure it contains:

pymupdf
chromadb
langchain
langchain-community
langchain-groq
langgraph
sentence-transformers
rank-bm25
ragas
fastapi
gradio
uvicorn
python-dotenv
pydantic
pandas
httpx
cross-encoder

Why these specific packages?

This list covers the entire pipeline: pymupdf for PDF parsing, chromadb for vector storage, langchain and langgraph for the agentic workflow, sentence-transformers for embeddings, and gradio plus fastapi for the UI and API layer.

• Save both Dockerfile and requirements.txt.
• Verify the files exist by running:

ls Dockerfile requirements.txt

You should see both files listed in the output.

What about the Groq API key?

Your .env file is in .gitignore and will never be pushed to the Space. Instead, you will configure it as a Secret in the Space settings.

Secrets are available at runtime as environment variables inside your container. Your existing os.environ.get("GROQ_API_KEY") code works without changes.

Dockerfile not recognized?

Make sure the file is named exactly Dockerfile with a capital D and no file extension. Some editors add hidden extensions like .txt by default.

help me troubleshoot my Dockerfile setup for Hugging Face Spaces deployment

Deploy to Hugging Face Spaces

With your Dockerfile ready, you can create a Space on Hugging Face and push your code to it. Every push triggers an automatic rebuild.

• Open your browser and go to huggingface.co/new-space.
• Enter a name for your Space (for example, rag-system).
• Under SDK, select Docker.
• Set visibility to Public so anyone can access your demo.
• Click Create Space.

Why Docker SDK?

The Docker SDK gives you full control over your environment. Your app uses FastAPI, Gradio, sentence-transformers, and ChromaDB together, so a custom container is the cleanest way to bundle everything.

After creating the Space, you need to add it as a Git remote and push your code.

• Switch back to your terminal in the rag-system/ folder.
• Log in to Hugging Face from the command line by running:

pip install huggingface_hub
hf auth login --add-to-git-credential

What happens when you log in?

The CLI will prompt you for a User Access Token. Go to huggingface.co/settings/tokens to generate one with write access.

Paste the token when prompted and press Enter. The token input will not be visible as you type. You should see a confirmation message like "Token is valid (permission: write)" once it succeeds.

• Add your Hugging Face Space as a remote by running:

git remote add space https://huggingface.co/spaces/[[HF_USERNAME="your-huggingface-username"]]/rag-system

• Commit your deployment files and push to the Space by running:

git add Dockerfile requirements.txt
git commit -m "Add Dockerfile for Hugging Face Spaces deployment"
git push space main

After pushing, Hugging Face automatically builds your Docker image and starts the container. This takes a few minutes for the first build because it installs all dependencies.

Monitor the build progress

Go to your Space page on Hugging Face and click the Logs button. The Build tab shows the Docker image build progress. The Container tab shows the running app logs once the build completes.

• Now add your Groq API key as a Secret in the Space settings.
• On your Space page, click Settings.
• Scroll to Variables and secrets.
• Click New secret.
• Set the name to GROQ_API_KEY and paste your Groq API key as the value.
• Click Save.

How do Secrets work in Docker Spaces?

Secrets you create in the Settings tab are injected as environment variables at runtime inside your container. Your code already uses os.environ.get("GROQ_API_KEY"), so it reads the Secret automatically.

Adding or changing a Secret triggers an automatic restart of your Space.

Wait for the Space to finish rebuilding after adding the Secret. When the build completes, your app is live at a public URL.

Build failing or Space stuck?

Check the Build logs for dependency installation errors. Common issues include typos in requirements.txt or missing system-level packages.

If the Container tab shows a crash, make sure your CMD references src.api:app correctly and that port 7860 is exposed.

help me debug my Hugging Face Space Docker build failure

Final testing and resume preparation

Your Space is live, but you need to verify that every feature works in the deployed environment. Cloud environments can behave differently from local development.

• Open your Space URL in a browser.
• Upload a fresh PDF document using the file upload component.
• Ask a factual question about the document to test the Retrieval path.
• Ask a summarization question (like "Summarize the introduction") to test the Summarization path.
• Ask a comparison question to test the Comparison path.
• Ask about a table or numbers to test the Table Analysis path.
• Verify that responses stream token-by-token (you see text appearing progressively).
• Check that citations appear with source references below each answer.
• Ask a follow-up question that references a previous answer to test conversation memory.

Testing all four paths

Watch for the router decision indicator in the response (the mode label like Retrieval or Table Analysis). Each path should trigger based on your question intent.

With testing complete, update your README with the live URL.

• Open README.md in VS Code.
• Replace the live demo link placeholder at the top with your actual Space URL.
• Update the evaluation table with your final benchmark numbers from evals/benchmark_report.md.
• Save README.md.

• Commit and push the README update by running:

git add README.md
git commit -m "Add live demo URL and final benchmark numbers"
git push origin main
git push space main

Your resume bullet point

You now have everything you need for a powerful resume line. Use this format, filling in your actual numbers:

Built and deployed a production RAG system with intelligent query routing (4 specialized paths), citation grounding (X% verification accuracy), conversation memory, and streaming. Achieved [X] faithfulness score with hybrid search + reranking improving context precision by [X]% over baseline.

Streaming not working in deployed version?

Hugging Face Spaces may add a proxy layer that buffers responses. Make sure your Gradio interface uses streaming mode and that your FastAPI WebSocket endpoint is correctly configured.

help me debug streaming responses not appearing in my deployed Hugging Face Space

✔️ Awesome, I've got everything!

Great. Double check you have saved all files and pushed to both your GitHub origin and the Hugging Face Space remote.

ⓧ I'd like to double check the full code

Here is the complete Dockerfile:

# Use Python 3.10 slim as the base image
FROM python:3.10-slim

# Create a non-root user (HF Spaces runs as uid 1000)
RUN useradd -m -u 1000 user

# Set working directory
WORKDIR /app

# Copy and install dependencies first (Docker layer caching)
COPY --chown=user requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code
COPY --chown=user . .

# Create a persistent directory for ChromaDB data
RUN mkdir -p /app/chroma_db && chown user:user /app/chroma_db

# Switch to non-root user
USER user

# Set environment variables for the user
ENV HOME=/home/user \
    PATH=/home/user/.local/bin:$PATH

# Expose port 7860 (required by Hugging Face Spaces)
EXPOSE 7860

# Start the application with uvicorn
CMD ["python", "-m", "uvicorn", "src.api:app", "--host", "0.0.0.0", "--port", "7860"]

And the complete requirements.txt:

pymupdf
chromadb
langchain
langchain-community
langchain-groq
langgraph
sentence-transformers
rank-bm25
ragas
fastapi
gradio
uvicorn
python-dotenv
pydantic
pandas
httpx
cross-encoder

Your RAG system is now live on the internet with a public URL. You have a deployed, production-grade AI application that demonstrates hybrid search, reranking, citation grounding, agentic routing, memory, and streaming.

Secret mission

Research Mode with Reasoning Trace

Your agent handles single questions well. But what about complex research questions that require searching multiple documents, cross-referencing conflicting information, and synthesizing a coherent answer? Build a Research Mode that thinks out loud, showing each step of its reasoning process to the user in real-time.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on free-tier services and local files, so there are no ongoing costs to worry about. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

• Hugging Face Space (free, stays live indefinitely on free hardware unless deleted or unused for extended periods).
• Groq API key (free tier, rate-limited but never charges).
• Local project files including the rag-system/ folder, virtual environment, and ChromaDB database.
• GitHub repository (free, public).

Keep everything running

No action needed. Choose this if you are still actively building or want to keep your live demo available for interviews and portfolio reviews.

• Your Hugging Face Space stays live for free. On free hardware, it will go to sleep after a period of inactivity and wake up automatically on the next visit.
• Your Groq API key remains active within its free-tier rate limits. The free tier resets daily, so your app will continue working within those limits.
• Your local rag-system/ folder, virtual environment, and ChromaDB data remain on your machine.
• Your GitHub repository stays public for portfolio visibility.

Pause - I'll come back to this later

Shut down running processes to free up local memory while keeping all your files and cloud resources intact.

• Pause your Hugging Face Space: open your Space on huggingface.co, click Settings in the upper-right area, then click Pause Space. The Space stops executing until you manually restart it.
• Deactivate your local virtual environment by running:

deactivate

• Your Groq API key, GitHub repo, and local files all stay intact for when you return.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is permanent.

• Delete your Hugging Face Space: open your Space on huggingface.co, click Settings in the upper-right area, scroll to the bottom of the page, click Delete this space, then confirm by clicking I understand, delete this space.
• Revoke your Groq API key: open the GroqCloud Console at console.groq.com, navigate to your API keys page, and delete the key you created for this project.
• Delete your GitHub repository: open the repository on github.com, go to Settings, scroll to the Danger Zone section, and click Delete this repository. Confirm when prompted.
• Delete your local project files by running:

cd ~/Desktop
rm -rf rag-system/

• On Windows, open PowerShell and run:

cd ~\Desktop
Remove-Item -Recurse -Force rag-system

• This removes the entire project folder including your virtual environment, ChromaDB database, and all source code.

That's a Wrap!

That's a Wrap!

Nice work! You just built and deployed a production-grade RAG system that reads PDF documents, answers questions intelligently, and grounds every claim with verifiable citations.

This is the exact kind of system that AI engineer roles ask you to build in take-home assignments and on the job.

You've learned how to:

• Build a complete RAG pipeline with hybrid search (BM25 + dense vector), cross-encoder reranking, and citation grounding with verification that every claim links to its source passage.
• Design an intelligent LangGraph agent with a multi-path intent router (Retrieval, Summarization, Comparison, Table Analysis), conversation memory, document grading with query rewriting, and streaming responses.
• Deploy a production FastAPI + Gradio application to Hugging Face Spaces with quantified RAGAS evaluation metrics and a professional GitHub README with architecture diagrams and benchmarks.
• Build a Research Mode that autonomously performs multi-step research across documents, detects conflicting information, and produces a synthesized answer with a visible reasoning trace and confidence score.

Ready to quiz yourself?