Build a RAG Chatbot with FastAPI

Build a fully local RAG chatbot that answers questions from your PDF documents.

Introduction

30 Second Summary

AI chatbots are everywhere, but most either hallucinate answers or can only repeat what they were trained on. The ones that actually answer from your own documents use a technique called Retrieval-Augmented Generation.

In this project, you will build a fully local RAG chatbot that reads a PDF, searches it for relevant sections, and generates accurate answers using an AI model running on your machine.

What You'll Build

You type a question into a browser-based chat widget, the system searches your PDF knowledge base using vector similarity, retrieves the most relevant sections, and returns a grounded answer in real time.

By the end of this project, you'll have:

  • A browser-based chat interface served by FastAPI where you can type questions and see AI-generated answers drawn directly from your PDF.
  • A RAG pipeline that extracts PDF text, splits it into context-aware chunks with document and section headers prepended, embeds them as vectors, and retrieves the most relevant ones for any question.
  • A fully local AI system running IBM Granite 3.3 and nomic-embed-text through Ollama with zero API costs and zero internet dependency.
  • Secret Mission: Add source citations that show users exactly which PDF sections powered each answer.

Are there any prerequisites?

You need Python 3.10 or higher, Ollama installed and running, and VS Code on your machine. Step 1 walks you through verifying each one.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters to you.

Set Up Your Development Environment

To build a RAG chatbot that runs entirely on your machine, you need a few tools working together: Python for the server code, Ollama for running AI models locally, and several Python packages for the web framework and vector search.

This step gets everything installed and verified so you can focus purely on building in the steps that follow.

In this step, get ready to:
  • Verify Python 3.11 or higher is installed.
  • Install Ollama and pull the Granite 3.3 and nomic-embed-text models.
  • Create a project folder with a virtual environment and install all dependencies.
Verify Python 3.11 or higher is installed
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Check your Python version by running:
python3 --version

On Windows?

If python3 is not recognized, try python --version or py --version instead.

✔️ I see version 3.11 or higher

You're all set. Move on to the next substep.

ⓧ I see an older version

Your system has Python installed, but it is older than 3.11. You need 3.11+ for the type syntax and packages this project uses.

  • Download the latest Python 3.13 installer from python.org/downloads.
  • Run the installer. On Windows, check Add python.exe to PATH before clicking Install Now.
  • Close and reopen your terminal, then run python3 --version again to confirm the new version.

macOS still showing the old version?

macOS ships with an older Python by default. After installing a newer version, you may need to use python3.13 explicitly, or use pyenv to manage versions. See the official Python docs for macOS-specific guidance.

ⓧ Command not found

Python is not installed yet. Follow these steps to install it:

  • Download the latest Python 3.13 installer from python.org/downloads.
  • Run the installer. On Windows, check Add python.exe to PATH before clicking Install Now.
  • Close and reopen your terminal, then run python3 --version to confirm installation.
Install Ollama and pull the AI models

Ollama lets you run large language models locally with a single command. You need two models: Granite 3.3 for generating answers, and nomic-embed-text for converting text into vectors.

  • Download and install Ollama from ollama.com/download.
  • Follow the installer prompts for your operating system.
  • Verify Ollama is installed by running:
ollama --version

✔️ I see a version number

Ollama is installed. Continue below to pull the models.

ⓧ Command not found

The installer may not have added Ollama to your PATH. Close and reopen your terminal, then try again. If it still fails, visit ollama.com/download and re-run the installer.

  • Pull the Granite 3.3 chat model (1.5GB download) by running:
ollama pull granite3.3:2b

You should see a progress bar followed by success once the download completes.

  • Pull the embedding model (274MB download) by running:
ollama pull nomic-embed-text

You should see another success message once the download finishes.

  • Confirm both models are available by running:
ollama list

You should see both granite3.3:2b and nomic-embed-text:latest listed in the output.

Model pull failing or timing out?

  • Make sure Ollama is running. On macOS, check for the Ollama icon in your menu bar. On Windows, look for it in the system tray.
  • If you see a connection error, try restarting Ollama by quitting and reopening the app.
  • help me get Ollama running and pulling models
Create the project folder and install dependencies
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-chatbot
cd rag-chatbot
  • Create a Python virtual environment by running:
python3 -m venv .venv

What is a virtual environment?

A virtual environment is an isolated Python installation for this project. Packages you install here won't interfere with other projects on your machine.

  • Activate the virtual environment by running:

macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate

Linux

source .venv/bin/activate

You should see (.venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

  • Install all the project dependencies by running:
pip install "fastapi[standard]" ollama pypdf reportlab faiss-cpu numpy

You should see a series of download progress bars followed by Successfully installed and a list of all installed packages.

What are these packages?

  • fastapi[standard] is the web framework for your API. The [standard] extra includes uvicorn, the server that runs your app.
  • ollama is the Python client for talking to your local Ollama models.
  • pypdf extracts text from PDF files.
  • reportlab generates a sample PDF for testing.
  • faiss-cpu and numpy power the vector search that finds relevant document chunks.

🙋‍♀️ Pip install failing?

  • Make sure your virtual environment is active (you see (.venv) in your prompt). If not, re-run the activation command above.
  • If you see permission errors, do not use sudo. Deactivate and recreate the virtual environment instead.
  • help me fix this pip install error
  • Verify everything is working together by running a quick import check:
python -c "import fastapi; import ollama; import pypdf; import faiss; print('All packages installed successfully')"

You should see All packages installed successfully printed with no errors.

Your environment is ready. Next up, you'll build a basic chat endpoint and see firsthand why RAG matters when the LLM hallucinates without access to your documents.

Build a Basic Chat Endpoint

Your environment is set up with Python, Ollama, and all the packages you need. Now it's time to write actual code.

Before layering in retrieval and vector search, you need a working web server that talks to your local LLM. You will also see, firsthand, why RAG matters: when you ask the LLM a question about your PDF, it will make up an answer because it has no access to the document.

In this step, get ready to:
  • Create a FastAPI app with a /chat endpoint that sends questions to Granite 3.3.
  • Generate a sample knowledge base PDF with ReportLab.
  • Observe the LLM hallucinate without retrieval context.
Create the FastAPI chat endpoint

Your main.py file is the heart of the server. It defines the request shape, response shape, and a single route that forwards questions to Granite.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select your project folder on the Desktop, then click Open.
  • Create a new file called main.py in the project root by clicking the New File icon in the VS Code Explorer sidebar.
  • Add the imports, app instance, and request/response models by pasting this code:
from pathlib import Path

import ollama
from fastapi import FastAPI
from fastapi.responses import HTMLResponse
from pydantic import BaseModel

app = FastAPI()

# Model and path configuration
CHAT_MODEL = "granite3.3:2b"
EMBED_MODEL = "nomic-embed-text"
PDF_PATH = Path("data/knowledge.pdf")


# Define the shape of incoming chat requests
class ChatRequest(BaseModel):
    question: str


# Define the shape of outgoing chat responses
class ChatResponse(BaseModel):
    answer: str
    sources: list[str]

What does this code do?

CHAT_MODEL and EMBED_MODEL store the Ollama model names so you can change them in one place later.

ChatRequest uses Pydantic to validate that every incoming POST body has a question field. ChatResponse defines the JSON shape the endpoint returns: an answer string and a sources list (empty for now).

  • Below the models, add the chat endpoint that forwards questions to Granite:
@app.post("/chat", response_model=ChatResponse)
def chat(request: ChatRequest):
    # Send the question directly to Granite with no context
    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[
            {"role": "user", "content": request.question},
        ],
    )
    # Return the raw LLM answer with no sources
    return ChatResponse(
        answer=response.message.content,
        sources=[],
    )

What does this endpoint do?

The @app.post("/chat") decorator registers a POST route. When a request arrives, FastAPI validates the body against ChatRequest, then calls your function.

Inside, ollama.chat() sends the question directly to Granite 3.3 with no additional context. The response text lives at response.message.content. Sources are empty because there is no retrieval yet.

  • Save main.py.
  • Open the integrated terminal in VS Code by pressing Ctrl+` (backtick key, same on macOS and Windows).
  • Make sure your virtual environment is active (you should see (.venv) in your terminal prompt).
  • Start the FastAPI dev server by running:
uvicorn main:app --reload

You should see output including Uvicorn running on http://127.0.0.1:8000 confirming the server started.

Server not starting?

  • If you see ModuleNotFoundError, your virtual environment may not be active. Run source .venv/bin/activate (macOS/Linux) or .venv\Scripts\activate (Windows) and try again.
  • If port 8000 is already in use, stop whatever is using it or add --port 8001 to the command.

Ask for help: My uvicorn server won't start and I see an error in the terminal.

Create the sample knowledge base PDF

You need a real document for the chatbot to answer questions about. This script generates a Company Insurance FAQ with clear section headings and detailed answers.

  • Stop the server by pressing Ctrl+C in the terminal.
  • Create the rag package folder by running:
mkdir rag
  • Create the empty __init__.py file inside rag by right-clicking the rag folder in the VS Code Explorer sidebar, clicking New File, and naming it __init__.py.
  • Confirm the folder and file exist by running:
ls rag

You should see __init__.py listed.

  • Create another new file inside the rag folder called make_sample_pdf.py (right-click rag, click New File).
  • Paste the imports and the first FAQ entry into rag/make_sample_pdf.py:
from pathlib import Path
from reportlab.lib.pagesizes import letter
from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer
from reportlab.lib.styles import getSampleStyleSheet


def create_sample_pdf():
    # Create the output directory and configure the PDF
    output_path = Path("data/knowledge.pdf")
    output_path.parent.mkdir(parents=True, exist_ok=True)

    doc = SimpleDocTemplate(str(output_path), pagesize=letter)
    styles = getSampleStyleSheet()
    story = []

    # Each tuple is (text_content, reportlab_style_name)
    content = [
        ("Company Insurance FAQ", "Title"),
        ("How do I file a claim?", "Heading2"),
        (
            "To file a claim, log into your account at portal.example.com and navigate to "
            "the Claims section. Click 'New Claim' and fill out the required information "
            "including the date of incident, description of damages, and any supporting "
            "documentation such as photos or police reports. Claims are typically processed "
            "within 5-7 business days. You will receive email updates on your claim status.",
            "Normal",
        ),

What does this code do?

ReportLab builds PDFs programmatically. SimpleDocTemplate handles page layout, and each paragraph gets a named style (Title, Heading2, Normal).

The content list stores tuples of text and style name so you can loop through them to build the PDF. The first entry sets the document title, followed by question headings and their answers.

  • Directly below the code you just pasted (still inside the content list), add the remaining three FAQ sections:
        ("What is covered under the standard plan?", "Heading2"),
        (
            "The standard plan covers property damage, liability protection up to $500,000, "
            "medical payments up to $10,000 per person, and comprehensive coverage for theft "
            "and natural disasters. Deductibles start at $500 for property claims and $250 "
            "for medical claims. The plan does not cover intentional damage, normal wear and "
            "tear, or pre-existing conditions reported after policy activation.",
            "Normal",
        ),
        ("How do I cancel my policy?", "Heading2"),
        (
            "To cancel your policy, contact our support team at support@example.com or call "
            "1-800-555-0199 during business hours (Monday-Friday, 8am-6pm EST). You can also "
            "submit a cancellation request through your online account. Please note that "
            "cancellations require 30 days notice. Any prepaid premiums for the remaining "
            "period will be refunded within 10 business days after the cancellation date.",
            "Normal",
        ),
        ("What is the claims appeal process?", "Heading2"),
        (
            "If your claim is denied, you have 60 days to file an appeal. Submit your appeal "
            "in writing to appeals@example.com with your claim number, the reason you believe "
            "the denial was incorrect, and any additional supporting evidence. Appeals are "
            "reviewed by a senior claims adjuster within 15 business days. You will receive "
            "a written decision by email and postal mail.",
            "Normal",
        ),
    ]
  • Below the closing ] of the content list, add the build logic and the script entry point:
    # Build each paragraph with its style and add vertical spacing
    for text, style_name in content:
        story.append(Paragraph(text, styles[style_name]))
        story.append(Spacer(1, 12))

    doc.build(story)
    print(f"Sample PDF created at: {output_path}")


if __name__ == "__main__":
    create_sample_pdf()
  • Save rag/make_sample_pdf.py.
  • Generate the PDF by running:
python rag/make_sample_pdf.py

You should see Sample PDF created at: data/knowledge.pdf printed in the terminal.

  • Confirm the file exists by running:
ls data

You should see knowledge.pdf listed.

Script failing with an import error?

  • Make sure reportlab is installed. Run pip install reportlab and try again.
  • If you see an indentation error, check that the content list entries line up consistently. Python is strict about whitespace.

Ask for help: My make_sample_pdf.py script is failing when I run it.

✔️ Awesome, I've got everything!

Great. Double check you saved rag/make_sample_pdf.py and that data/knowledge.pdf exists on disk.

ⓧ I'd like to double check the full code

Scroll up and confirm your rag/make_sample_pdf.py contains all three code chunks in order:

  • Chunk 1: imports, create_sample_pdf() function start, document setup, and the first FAQ entry (How do I file a claim?).
  • Chunk 2: three remaining FAQ entries (standard plan, cancel policy, claims appeal) closing the content list.
  • Chunk 3: the for loop that builds paragraphs, doc.build(story), the print statement, and the if __name__ block.
Test the chat endpoint and observe the hallucination

You now have a PDF knowledge base and a chat endpoint. Time to see how the LLM responds when it has no access to your document.

  • Restart the FastAPI server by running:
uvicorn main:app --reload
  • Open your browser and navigate to http://localhost:8000/docs.

You should see the FastAPI interactive documentation page (Swagger UI) with your /chat endpoint listed.

Before you test, think about this: you have a PDF about insurance claims, and you are about to ask Granite a question about that PDF. Do you think the model will give an accurate answer pulled from the PDF, or something else entirely?

  • Click the POST /chat row to expand it.
  • Click Try it out.
  • Replace the request body with:
{
  "question": "How do I file a claim?"
}
  • Click Execute.

Granite returns an answer, but it is generic or completely made up. It might mention steps that sound plausible but do not match the specific details in your PDF (like portal.example.com or the 5-7 business day processing timeline).

Why is the answer wrong?

Granite 3.3 was never trained on your Company Insurance FAQ. Without retrieval, the model can only guess from its general training data. This is called hallucination.

RAG solves this by retrieving relevant passages from your documents and injecting them into the prompt before the model generates an answer. That gap between what the model knows and what your document says is exactly what the rest of this project fills.

✔️ Awesome, I've got everything!

Great. Double check you saved main.py and that your server is still running.

ⓧ I'd like to double check the full code

Scroll up and confirm your main.py contains both code chunks in order:

  • Chunk 1: imports, app = FastAPI(), model constants, ChatRequest, and ChatResponse classes.
  • Chunk 2: the @app.post("/chat") endpoint that calls ollama.chat() and returns a ChatResponse.

You have proven that an LLM alone cannot answer questions from specific documents. Next, you will build the retrieval pipeline that extracts, chunks, and embeds your PDF so the model can answer from real data.

Build the RAG Pipeline

In the last step you saw the problem firsthand. When you asked the LLM about your PDF, it hallucinated a generic answer because it has zero access to the document.

Now you build the retrieval system that fixes this. The core insight is that chunks must be self-contained and meaningful in isolation. Naive character splitting cuts sentences in half and loses section context. Instead, you will apply the Contextual Chunk Headers (CCH) pattern: split on sentence boundaries and prepend both the document title and section heading so each chunk carries its own provenance when retrieved.

In this step, get ready to:
  • Build a PDF text extractor and a context-aware chunker that produces self-contained chunks.
  • Create an embedding and vector storage layer with FAISS.
  • Add /ingest and /search endpoints to your FastAPI server.
Build the PDF text extractor and context-aware chunker

The retrieval pipeline starts by extracting raw text from the PDF, then splitting it into meaningful chunks. You will build two modules: one for extraction and one for chunking.

  • In VS Code, create a new file called pdf_to_text.py inside the rag folder.
  • Paste the following code into the file:
from pathlib import Path
from pypdf import PdfReader


def extract_text_from_pdf(pdf_path: Path) -> str:
    reader = PdfReader(pdf_path)
    text = ""
    for page in reader.pages:
        page_text = page.extract_text()
        if page_text:
            text += page_text + "\n"
    return text

What does this code do?

  • PdfReader from pypdf opens the PDF and exposes each page.
  • extract_text() pulls the text content from each page and concatenates it all into one string with newline separators.
  • Save rag/pdf_to_text.py.
  • Test the extraction by running this command in your terminal:
python -c "from rag.pdf_to_text import extract_text_from_pdf; from pathlib import Path; print(extract_text_from_pdf(Path('data/knowledge.pdf'))[:200])"

You should see the first 200 characters of your FAQ text printed, starting with the document title.

Seeing a ModuleNotFoundError?

  • Make sure you are running the command from your project root folder (the one containing the rag folder and data folder).
  • Confirm that rag/__init__.py exists. Python needs this file to treat rag as a package.
  • help me debug this ModuleNotFoundError

Now for the chunker. This is where the Contextual Chunk Headers pattern comes in. Each chunk will be prefixed with the document title and section heading so it carries its own context when retrieved later.

  • Create a new file called chunking.py inside the rag folder.
  • Paste this first section of code at the top of the file. It starts the main chunk_text_with_context function and handles parsing the text into sections:
import re


def chunk_text_with_context(
    text: str, max_chunk_size: int = 400, doc_title: str = ""
) -> list[str]:
    lines = text.split("\n")

    if not doc_title:
        doc_title = detect_document_title(lines)

    sections = []
    current_heading = "General"
    current_body_lines = []

    for line in lines:
        stripped = line.strip()
        if not stripped:
            continue

        if is_heading(stripped):
            if current_body_lines:
                sections.append((current_heading, " ".join(current_body_lines)))
                current_body_lines = []
            current_heading = stripped
        else:
            current_body_lines.append(stripped)

    if current_body_lines:
        sections.append((current_heading, " ".join(current_body_lines)))

What does this section-parsing code do?

  • It walks through each line of the extracted PDF text, detecting headings versus body text.
  • When it finds a heading, it saves whatever body text it has collected so far as a section (paired with the previous heading).
  • The result is a list of (heading, body) tuples representing each FAQ section.
  • Directly below the code you just pasted (still inside the same function, maintaining the indentation), add the sentence-chunking logic with the CCH prefix:
    chunks = []
    for heading, body in sections:
        sentences = split_into_sentences(body)
        current_chunk_sentences = []
        current_size = 0
        prefix = f"[Document: {doc_title} | Section: {heading}] "
        prefix_len = len(prefix)

        for sentence in sentences:
            sentence_len = len(sentence)
            if current_size + sentence_len + prefix_len > max_chunk_size and current_chunk_sentences:
                chunk_text = prefix + " ".join(current_chunk_sentences)
                chunks.append(chunk_text)
                overlap_sentence = current_chunk_sentences[-1]
                current_chunk_sentences = [overlap_sentence]
                current_size = len(overlap_sentence)
            current_chunk_sentences.append(sentence)
            current_size += sentence_len

        if current_chunk_sentences:
            chunk_text = prefix + " ".join(current_chunk_sentences)
            chunks.append(chunk_text)

    return chunks

What does the CCH chunking logic do?

  • For each section, it splits the body into sentences and groups them into chunks that stay under max_chunk_size characters.
  • Every chunk gets a prefix like [Document: Company Insurance FAQ | Section: How do I file a claim?] so it is self-contained when retrieved later.
  • The last sentence of each chunk overlaps into the next chunk to preserve continuity at boundaries.
  • Below the main function (no more indentation), add the three helper functions that the main function relies on:
def detect_document_title(lines: list[str]) -> str:
    for line in lines:
        stripped = line.strip()
        if stripped and len(stripped) <= 80:
            return stripped
    return "Untitled Document"


def is_heading(line: str) -> bool:
    if len(line) > 100:
        return False
    if line.endswith("?"):
        return True
    if line[0].isupper() and not line.endswith("."):
        words = line.split()
        if len(words) <= 10:
            return True
    return False


def split_into_sentences(text: str) -> list[str]:
    sentences = re.split(r"(?<=[.!?])\s+", text)
    return [s.strip() for s in sentences if s.strip()]

What do the helper functions do?

  • detect_document_title grabs the first short line (likely the document title) as the title for the CCH prefix.
  • is_heading identifies lines that look like section headers: short lines ending in ? or capitalized lines without a period.
  • split_into_sentences uses a regex to split on sentence-ending punctuation followed by whitespace, keeping sentences intact.
  • Save rag/chunking.py.
  • Test the full chunking pipeline by running:
python -c "from rag.pdf_to_text import extract_text_from_pdf; from rag.chunking import chunk_text_with_context; from pathlib import Path; text = extract_text_from_pdf(Path('data/knowledge.pdf')); chunks = chunk_text_with_context(text); print(f'{len(chunks)} chunks created'); print(); print(chunks[0])"

You should see something like 6 chunks created followed by the first chunk starting with a CCH prefix like [Document: Company Insurance FAQ | Section: How do I file a claim?].

Seeing 0 chunks or a NameError?

  • If you see a NameError, double check that the helper functions (detect_document_title, is_heading, split_into_sentences) are defined below the main function with no indentation.
  • If you see 0 chunks, confirm the import re line is at the very top of the file.
  • help me debug my chunking.py file

✔️ Awesome, I've got everything!

Great. Double check you have saved both rag/pdf_to_text.py and rag/chunking.py.

ⓧ I'd like to double check the full code

Your rag/pdf_to_text.py should contain the single extract_text_from_pdf function you pasted earlier.

Your rag/chunking.py should contain, in order from top to bottom:

  • The import re line.
  • The chunk_text_with_context function (section-parsing loop followed by the sentence-chunking loop ending with return chunks).
  • The three helper functions: detect_document_title, is_heading, split_into_sentences.

Scroll up and confirm each block is present in that order.

Build the embedding and vector storage layer

With chunks ready, you need to convert them into numerical vectors so you can search by meaning. You will embed each chunk using nomic-embed-text and store the vectors in a FAISS index for fast similarity search.

  • Create a new file called embed_store.py inside the rag folder.
  • Paste this code that handles embedding chunks and building the FAISS index:
import faiss
import numpy as np
import ollama

index: faiss.IndexFlatIP | None = None
chunks_store: list[str] = []


def embed_texts(chunks: list[str], model: str) -> None:
    global index, chunks_store

    response = ollama.embed(model=model, input=chunks)
    embeddings = np.array(response["embeddings"], dtype=np.float32)

    faiss.normalize_L2(embeddings)

    dimension = embeddings.shape[1]
    index = faiss.IndexFlatIP(dimension)
    index.add(embeddings)

    chunks_store = chunks

What does this code do?

  • ollama.embed() sends all chunks to nomic-embed-text in one batch call, returning 768-dimensional vectors.
  • faiss.normalize_L2() normalizes the vectors so that inner-product search is equivalent to cosine similarity.
  • IndexFlatIP is an exact-search index using inner product. For small document sets this is fast and avoids training complexity.
  • Directly below the embed_texts function, add the search function:
def search_index(query: str, model: str, top_k: int = 3) -> list[str]:
    global index, chunks_store

    if index is None or len(chunks_store) == 0:
        return []

    response = ollama.embed(model=model, input=[query])
    query_embedding = np.array(response["embeddings"], dtype=np.float32)

    faiss.normalize_L2(query_embedding)

    distances, indices = index.search(query_embedding, top_k)

    results = []
    for idx in indices[0]:
        if idx < len(chunks_store):
            results.append(chunks_store[idx])

    return results

What does search_index do?

  • It embeds the user's query into the same 768-dimensional space using nomic-embed-text.
  • It searches the FAISS index for the top_k most similar vectors (default 3) and returns the original chunk text for each match.
  • Save rag/embed_store.py.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file with both functions.

ⓧ I'd like to double check the full code

Your rag/embed_store.py should contain, in order from top to bottom:

  • The three imports (faiss, numpy, ollama) and the two module-level variables (index and chunks_store).
  • The embed_texts function.
  • The search_index function.

Scroll up and confirm each block is present in that order.

Add the /ingest and /search endpoints

The RAG modules are ready. Now you expose them through your FastAPI server so you can ingest PDFs and search chunks via HTTP.

  • Switch back to main.py in VS Code.
  • Find these existing import lines near the top of the file:
from rag.chunking import chunk_text_with_context
from rag.pdf_to_text import extract_text_from_pdf
from rag.embed_store import embed_texts, search_index
  • Add the three lines above directly below the from pydantic import BaseModel line, with a blank line separating them:
from pydantic import BaseModel

from rag.chunking import chunk_text_with_context
from rag.pdf_to_text import extract_text_from_pdf
from rag.embed_store import embed_texts, search_index
  • Now add the two new endpoints. Find the ChatResponse class definition and paste the following code directly below it (above the existing /chat endpoint):
@app.post("/ingest")
def ingest():
    text = extract_text_from_pdf(PDF_PATH)
    chunks = chunk_text_with_context(text, max_chunk_size=400)
    embed_texts(chunks, model=EMBED_MODEL)
    return {"status": "ok", "chunks_ingested": len(chunks)}


@app.post("/search")
def search(request: ChatRequest):
    results = search_index(request.question, model=EMBED_MODEL, top_k=3)
    return {"results": results}

What do these endpoints do?

  • /ingest runs the full pipeline: reads the PDF, chunks it with CCH headers, embeds the chunks, and stores them in the FAISS index.
  • /search takes a question, embeds it, and returns the top-3 most similar chunks from the index.
  • Save main.py. Your server should auto-reload since you started it with --reload.

Server showing an ImportError on reload?

  • Make sure the rag/__init__.py file exists (it can be empty).
  • Check that the three new import lines reference the exact filenames: rag.chunking, rag.pdf_to_text, and rag.embed_store.
  • help me fix this ImportError in my FastAPI app

Before you test, what do you think the /ingest response will contain? How many chunks do you expect from a 4-section FAQ?

  • Open http://localhost:8000/docs in your browser.
  • Find the POST /ingest endpoint, click Try it out, then click Execute.

You should see a JSON response like:

{"status": "ok", "chunks_ingested": 6}

The exact chunk count depends on how the text splits across sentence boundaries. A number between 4 and 8 is normal for this FAQ.

  • Now find the POST /search endpoint, click Try it out, and enter this JSON body:
{"question": "How do I file a claim?"}
  • Click Execute.

You should see a results array with 3 chunks, each starting with a CCH prefix like [Document: Company Insurance FAQ | Section: How do I file a claim?] followed by complete sentences about filing claims.

Why do the chunks start with document and section labels?

This is the Contextual Chunk Headers pattern in action. When the LLM receives these chunks as context in the next step, it knows exactly which section of which document the information came from. Without the prefix, a chunk like "Claims are typically processed within 5-7 business days" has no provenance.

🙋‍♀️ Search returning empty results?

  • Make sure you called /ingest first. The FAISS index is in-memory and starts empty.
  • Check that the /ingest response showed a non-zero chunks_ingested count.
  • Confirm Ollama is still running. The embed call needs nomic-embed-text to be available.
  • help me debug why /search returns empty results

✔️ Awesome, I've got everything!

Great. Confirm you have saved all files and that both /ingest and /search are working.

ⓧ I'd like to double check the full code

Your main.py should contain, in order from top to bottom:

  • Imports from pathlib, ollama, fastapi, pydantic, and the three rag module imports.
  • The app, CHAT_MODEL, EMBED_MODEL, and PDF_PATH constants.
  • The ChatRequest and ChatResponse Pydantic models.
  • The /ingest endpoint, then the /search endpoint, then the /chat endpoint.

Scroll up and confirm each section is present. The /chat endpoint should still be the same as before (no RAG context injected yet).

Your retrieval system is fully operational. You can ingest a PDF and search it by meaning. Next up, you will wire this retrieval into the /chat endpoint so Granite answers from your actual document instead of hallucinating.

Wire Retrieval into Chat

You now have two separate systems that work independently. The /chat endpoint talks to Granite but has no knowledge of your PDF. The /search endpoint finds relevant chunks but does nothing with them.

In this step, you connect the two. The /chat endpoint will search for relevant chunks first, then pass them as context to Granite so it answers from your PDF instead of hallucinating.

In this step, get ready to:
  • Search FAISS for relevant chunks before generating a response.
  • Build a RAG prompt that constrains Granite to answer from context only.
  • Verify the LLM now gives accurate, PDF-grounded answers.
Add retrieval to the chat endpoint

Right now the chat function sends the user's question straight to Granite with no context. You need to insert a retrieval step that finds relevant chunks and formats them before the LLM sees the question.

  • In main.py, find the existing chat function.
  • Find these lines inside the function body:
    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[
            {"role": "user", "content": request.question},
        ],
    )
    return ChatResponse(
        answer=response.message.content,
        sources=[],
    )
  • Delete all of these lines inside the function body. Replace them with the following retrieval and formatting code:
    # Search for relevant chunks from the knowledge base
    results = search_index(request.question, model=EMBED_MODEL, top_k=3)

    # Return early if no documents have been ingested
    if not results:
        return ChatResponse(
            answer="No documents have been ingested yet. Please ingest a PDF first.",
            sources=[],
        )

    # Format retrieved chunks as numbered context passages
    context_parts = []
    for i, chunk in enumerate(results, 1):
        context_parts.append(f"[{i}] {chunk}")

    context = "\n\n".join(context_parts)

What does this code do?

This is the "retrieval" half of Retrieval-Augmented Generation. Before the LLM sees the question, you search your vector index for the 3 most relevant chunks.

  • search_index() embeds the user's question and finds the closest chunks in FAISS.
  • The guard clause handles the case where no PDF has been ingested yet.
  • Each chunk is numbered [1], [2], [3] so the model can reference specific passages in its answer.
Craft the RAG prompt template

The formatted context is useless unless you tell the model how to use it. A well-crafted system prompt constrains Granite to answer only from the provided passages and admit when it does not have enough information.

  • Directly below the context = "\n\n".join(context_parts) line you just added, paste the following system prompt and chat call:
    # Build a system prompt that grounds the model in retrieved context
    system_prompt = (
        "You are a helpful assistant that answers questions based only on the provided context. "
        "If the context does not contain enough information to answer the question, say "
        "'I don't have enough information to answer that question.' "
        "Do not make up information. Be concise and direct.\n\n"
        f"Context:\n{context}"
    )

    # Send the system prompt and user question to Granite
    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": request.question},
        ],
    )

    # Return the answer along with the source chunks
    return ChatResponse(
        answer=response.message.content,
        sources=results,
    )

What does this code do?

This is the "generation" half of RAG. The system prompt tells Granite exactly how to behave with the retrieved context.

  • The system message injects the numbered chunks so Granite reads them before the user's question.
  • The instruction to say "I don't have enough information" prevents hallucination when the PDF does not cover the topic.
  • sources=results passes the raw retrieved chunks back in the response so you can show users where the answer came from.

💡 Why a system prompt instead of stuffing context into the user message?

The system role sets persistent instructions that shape how the model interprets every user message. Placing the context there keeps the user message clean and makes it easy to swap prompting strategies later without touching the user input.

  • Save main.py.

Your server auto-reloads thanks to the --reload flag.

✔️ Awesome, I've got everything!

Great. Confirm you saved the file and your terminal shows the server reloaded without errors.

ⓧ I'd like to double check the full code

Your chat function in main.py should now contain these sections in order:

  • The search_index() call and if not results: guard clause.
  • The context_parts loop that formats chunks as [1], [2], [3].
  • The system_prompt string with the context injected at the end.
  • The ollama.chat() call with both a system and user message.
  • The return statement with sources=results.

Scroll up and confirm each section matches what you pasted. The rest of main.py (imports, constants, ChatRequest, ChatResponse, /ingest, and /search) should be unchanged.

Server showing an error after save?

  • Check indentation. Every line inside the chat function must be indented by 4 spaces.
  • Make sure you deleted the old response = ollama.chat(...) and old return ChatResponse(...) lines completely. Duplicate returns cause a syntax error.
  • Confirm the f-string uses a lowercase f before the opening quote in f"Context:\n{context}".

Why is my server crashing after I edited the chat function?

Test the RAG-powered response

In Step 2, you asked "How do I file a claim?" and Granite gave a generic or incorrect answer because it had no access to your PDF. Now the same question will go through retrieval first.

  • Make sure you have already called /ingest in this server session. If you restarted the server since Step 3, re-ingest by sending a POST to /ingest in the Swagger UI first.
  • Open http://localhost:8000/docs in your browser.
  • Expand the POST /chat section and click Try it out.

Before you hit Execute, predict: will the answer be accurate this time, or will Granite still hallucinate?

  • Enter this request body and click Execute:
{
  "question": "How do I file a claim?"
}

This time the response includes a precise answer pulled from your PDF. You should see details about logging into your account at portal.example.com, navigating to the Claims section, and the 5-7 business day processing time.

The sources array in the response now contains the actual chunks that were retrieved, each starting with a Contextual Chunk Header like [Document: Company Insurance FAQ | Section: How do I file a claim?].

What just happened?

This is the full RAG loop in action. Your question was embedded, matched against the FAISS index to find the most relevant chunks, those chunks were injected into the system prompt, and Granite generated an answer grounded in that context.

The same question that hallucinated in Step 2 now returns a factual, verifiable answer. That contrast is the core value proposition of RAG.

🙋‍♀️ Still getting a generic answer?

  • Check whether sources in the response is an empty array. If so, you need to call POST /ingest first to load the PDF into FAISS.
  • Verify your system prompt includes the f"Context:\n{context}" line. Without the f-string prefix, the context variable is not interpolated.
  • Confirm the messages list has two entries: one with "role": "system" and one with "role": "user". If you only have the user message, Granite never sees the context.

My /chat endpoint still returns a generic answer even after ingesting the PDF. How do I debug this?

Your chatbot now answers from real data instead of making things up. Next up, you will give it a proper browser-based chat interface so anyone can interact with it.

Add the Chat Widget

Your RAG pipeline works perfectly through the API, but a terminal-only interface is not a chatbot anyone would want to use.

In this step, you will build a browser-based chat widget served directly by FastAPI. Anyone can open a URL, type a question, and get a PDF-grounded answer in a clean chat interface.

In this step, get ready to:
  • Create an HTML chat interface with embedded CSS and JavaScript.
  • Wire the JavaScript to POST questions to your API and display responses as chat bubbles.
  • Configure FastAPI to serve the chat widget at the root URL and test the full flow.
Create the chat interface HTML and CSS

The chat widget is a single HTML file with embedded styles and JavaScript. FastAPI will serve it directly with no separate frontend server needed.

  • Create the static folder for your frontend file by running this command:
mkdir static
  • Confirm the folder exists by running:
ls

You should see static listed alongside rag, data, and main.py.

  • Create a new file static/index.html in VS Code and paste the following HTML head section with all the CSS styles:
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>RAG Chatbot</title>
    <style>
        * { margin: 0; padding: 0; box-sizing: border-box; }
        body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #f5f5f5; height: 100vh; display: flex; align-items: center; justify-content: center; }
        .chat-container { width: 100%; max-width: 600px; height: 80vh; background: white; border-radius: 12px; box-shadow: 0 4px 24px rgba(0,0,0,0.1); display: flex; flex-direction: column; overflow: hidden; }
        .chat-header { padding: 16px 20px; background: #1a1a2e; color: white; font-size: 16px; font-weight: 600; }
        .chat-messages { flex: 1; overflow-y: auto; padding: 20px; display: flex; flex-direction: column; gap: 12px; }
        .message { max-width: 80%; padding: 10px 14px; border-radius: 12px; font-size: 14px; line-height: 1.5; word-wrap: break-word; }
        .message.user { align-self: flex-end; background: #1a1a2e; color: white; border-bottom-right-radius: 4px; }
        .message.assistant { align-self: flex-start; background: #e8e8e8; color: #333; border-bottom-left-radius: 4px; }
        .message .sources { margin-top: 8px; padding-top: 8px; border-top: 1px solid #ccc; font-size: 12px; color: #666; }
        .message .sources summary { cursor: pointer; font-weight: 600; margin-bottom: 4px; }
        .message .sources p { margin: 4px 0; padding: 4px 8px; background: #f9f9f9; border-radius: 4px; border-left: 3px solid #1a1a2e; }
        .chat-input { display: flex; padding: 16px; border-top: 1px solid #eee; gap: 8px; }
        .chat-input input { flex: 1; padding: 10px 14px; border: 1px solid #ddd; border-radius: 8px; font-size: 14px; outline: none; }
        .chat-input input:focus { border-color: #1a1a2e; }
        .chat-input button { padding: 10px 20px; background: #1a1a2e; color: white; border: none; border-radius: 8px; font-size: 14px; cursor: pointer; }
        .chat-input button:disabled { opacity: 0.5; cursor: not-allowed; }
        .typing { align-self: flex-start; color: #999; font-size: 13px; font-style: italic; }
    </style>
</head>

What does this CSS do?

  • .chat-container creates a centered card with a fixed height, rounded corners, and a shadow.
  • .message.user and .message.assistant style the chat bubbles. User messages align right with a dark background. Assistant messages align left with a light background.
  • .sources styles the collapsible section that will display retrieved chunks below each answer.
  • .typing styles the "Thinking..." indicator that appears while waiting for the AI response.
  • Directly below the </head> tag, paste the chat container body markup and the script initialization:
<body>
    <div class="chat-container">
        <div class="chat-header">RAG Chatbot - Ask about your documents</div>
        <div class="chat-messages" id="messages">
            <div class="message assistant">Hello! I can answer questions about the ingested PDF documents. Ask me anything.</div>
        </div>
        <div class="chat-input">
            <input type="text" id="input" placeholder="Ask a question..." autocomplete="off">
            <button id="send" onclick="sendMessage()">Send</button>
        </div>
    </div>

    <script>
        const messagesEl = document.getElementById('messages');
        const inputEl = document.getElementById('input');
        const sendBtn = document.getElementById('send');

        inputEl.addEventListener('keydown', (e) => {
            if (e.key === 'Enter' && !sendBtn.disabled) sendMessage();
        });

What does this markup do?

  • The chat-messages div is the scrollable area where message bubbles appear.
  • The chat-input div holds the text input and send button at the bottom.
  • The three const declarations cache DOM references so the JavaScript can manipulate the UI without repeated lookups.
  • The event listener lets users press Enter to send a message instead of clicking the button.
  • Save static/index.html.
Wire the JavaScript to POST questions and display responses

The chat widget needs two functions. One sends the user's question to your /chat endpoint and handles the response. The other renders messages as styled chat bubbles with an optional sources section.

  • In static/index.html, directly below the event listener you just added, paste the sendMessage function:

        async function sendMessage() {
            const question = inputEl.value.trim();
            if (!question) return;

            addMessage(question, 'user');
            inputEl.value = '';
            sendBtn.disabled = true;

            const typingEl = document.createElement('div');
            typingEl.className = 'typing';
            typingEl.textContent = 'Thinking...';
            messagesEl.appendChild(typingEl);
            messagesEl.scrollTop = messagesEl.scrollHeight;

            try {
                const res = await fetch('/chat', {
                    method: 'POST',
                    headers: { 'Content-Type': 'application/json' },
                    body: JSON.stringify({ question }),
                });
                const data = await res.json();
                typingEl.remove();
                addMessage(data.answer, 'assistant', data.sources);
            } catch (err) {
                typingEl.remove();
                addMessage('Error: Could not reach the server.', 'assistant');
            }

            sendBtn.disabled = false;
            inputEl.focus();
        }

What does sendMessage do?

  • It grabs the user's text, immediately renders it as a user bubble via addMessage(question, 'user'), and clears the input.
  • A "Thinking..." indicator appears while the request is in flight so the user knows the AI is working.
  • It POSTs to /chat with the question as JSON. On success, it passes both the answer and the sources array to addMessage.
  • The catch block handles network errors gracefully instead of silently failing.
  • Directly below the sendMessage function, paste the addMessage function and the closing tags:

        function addMessage(text, role, sources) {
            const div = document.createElement('div');
            div.className = `message ${role}`;
            div.textContent = text;

            if (sources && sources.length > 0) {
                const sourcesDiv = document.createElement('div');
                sourcesDiv.className = 'sources';
                const details = document.createElement('details');
                const summary = document.createElement('summary');
                summary.textContent = `Sources (${sources.length})`;
                details.appendChild(summary);
                sources.forEach((s) => {
                    const p = document.createElement('p');
                    p.textContent = s.substring(0, 200) + (s.length > 200 ? '...' : '');
                    details.appendChild(p);
                });
                sourcesDiv.appendChild(details);
                div.appendChild(sourcesDiv);
            }

            messagesEl.appendChild(div);
            messagesEl.scrollTop = messagesEl.scrollHeight;
        }
    </script>
</body>
</html>

What does addMessage do?

  • It creates a new div with the appropriate class (message user or message assistant) so CSS handles the alignment and colors.
  • When sources are present (assistant messages from RAG), it builds a collapsible <details> element showing the retrieved chunks. Each chunk is truncated to 200 characters for readability.
  • The scroll position auto-advances so the latest message is always visible.
  • Save static/index.html.

Seeing a syntax error in your HTML?

  • Make sure the closing </script> tag appears after the addMessage function, followed by </body> and </html>.
  • Check that backtick template literals (`message ${role}`) use backticks, not single quotes.
  • Verify the </head> tag closes before <body> opens.

Why is my index.html not valid HTML?

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before continuing.

ⓧ I'd like to double check the full code

Scroll up and confirm your static/index.html contains these sections in order:

  • The HTML head with the full CSS style block (25 CSS rules).
  • The body with the chat container div, messages area, and input row.
  • A script tag containing three const declarations, one event listener, the sendMessage function, and the addMessage function.
  • Closing </script>, </body>, and </html> tags.
Serve the chat widget from FastAPI and test end-to-end

FastAPI needs a route that reads your HTML file and sends it to the browser. Since the HTMLResponse import already exists in main.py, you only need to add the endpoint function.

  • In main.py, scroll to the bottom of the file (below the chat function).
  • Add this new endpoint that serves your HTML at the root URL:
# Serve the chat widget at the root URL
@app.get("/", response_class=HTMLResponse)
def serve_ui():
    html_path = Path("static/index.html")
    return html_path.read_text()

What does this endpoint do?

The @app.get("/") decorator tells FastAPI to handle GET requests to the root URL. It reads static/index.html and returns it with the correct HTML content type so the browser renders it as a page instead of downloading raw text.

  • Save main.py. The running server will auto-reload.

Server showing an error after saving?

  • Check that the serve_ui function is at the module level (not indented inside another function).
  • Confirm static/index.html exists. If the file is missing, the server will crash with a FileNotFoundError when you visit the root URL.

Why is my FastAPI server crashing after I added the serve_ui endpoint?

The server restarted, which cleared the in-memory FAISS index. You need to re-ingest the PDF before testing the chat.

  • Re-ingest the PDF by running this command in a new terminal:
curl -X POST http://localhost:8000/ingest

You should see a JSON response like {"status":"ok","chunks_ingested":...} confirming the index is rebuilt.

Now for the moment of truth. Before you open the browser, what do you expect to see at the root URL? A raw JSON response, or something else?

  • Open your browser and navigate to http://localhost:8000.

You should see a centered chat widget with a dark header reading "RAG Chatbot - Ask about your documents", a welcome message from the assistant, and an input field at the bottom.

  • Type How do I file a claim? into the input field and press Enter.

Your question appears as a dark bubble on the right. A "Thinking..." indicator shows while Granite processes the request. After a few seconds, an accurate answer appears in a light bubble on the left, describing the portal login and claims process from your PDF.

  • Click the Sources toggle below the assistant's answer.

A collapsible section expands, showing the text chunks that were retrieved from your PDF to generate the answer. Each chunk starts with a document/section header prefix from the CCH pattern you built in Step 3.

Chat widget not loading or showing errors?

  • If you see a blank page, open DevTools (F12) and check the Console tab for JavaScript errors. A missing closing brace or mismatched quote in your HTML will prevent the script from running.
  • If the page loads but you get "Error: Could not reach the server" when you send a message, confirm your FastAPI server is still running in its terminal. Check that it shows no import errors.
  • If the answer comes back as "No documents have been ingested yet", you need to call POST /ingest again. The in-memory index resets each time the server reloads.

Why is my RAG chat widget not working in the browser?

You now have a fully working RAG chatbot running in your browser. Anyone can open the URL, type a question, and get answers grounded in your PDF. The entire system runs locally with zero cost and zero internet dependency.

Secret mission

Add Source Citations

Your chatbot gives accurate answers, but users have no way to verify where those answers come from. Can you add source citations that show exactly which PDF sections powered each response, so users can trust and fact-check the AI?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder on your Desktop (source code, virtual environment, PDF data, static files).
  • Ollama models: granite3.3:2b (chat) and nomic-embed-text (embeddings).

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with the chatbot.

  • Your project folder stays on your Desktop, ready to open in VS Code.
  • Ollama models stay on disk and auto-unload from RAM after 5 minutes of inactivity.
  • To restart the chatbot later, activate your virtual environment and run uvicorn main:app --reload from your project folder.

Pause - I'll come back to this later

Shut down running processes to free up RAM. Your files and models stay on disk so you can resume later.

  • Press Ctrl+C in the terminal running uvicorn to stop the FastAPI server.
  • Stop the Granite model to free RAM by running:
ollama stop granite3.3:2b
  • Your project folder, virtual environment, and Ollama models all remain on disk for next time.

Delete - I don't want to use this again

Remove all project resources and reclaim disk space. This cannot be undone.

  • Press Ctrl+C in the terminal running uvicorn to stop the FastAPI server.
  • Remove both Ollama models to reclaim approximately 1.8GB of disk space by running:
ollama rm granite3.3:2b
ollama rm nomic-embed-text
  • Delete your project folder from the Desktop. On macOS or Linux, run:
cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]
  • On Windows, run:
cd ~\Desktop
rmdir /s /q [[PROJECT_FOLDER="your project folder name"]]
  • This removes all source code, the virtual environment, the generated PDF, and static files.

Nice Work!

Nice Work!

You just built a fully local AI chatbot that answers questions from your own PDF documents. From first hallucination to grounded, citation-backed answers, you experienced the full RAG journey.

Here's what you learned:

  • Built a complete RAG pipeline from scratch: PDF extraction, context-engineered chunking with the Contextual Chunk Headers pattern, embedding generation, and vector similarity search with FAISS.
  • Experienced firsthand why RAG exists by watching the LLM hallucinate without context, then produce accurate, PDF-grounded answers once retrieval was wired in.
  • Integrated two Ollama models locally (Granite 3.3 for chat and nomic-embed-text for embeddings) into a FastAPI backend serving both a JSON API and a browser-based chat interface at zero cost.
  • Added source citations so users can verify AI answers against the original PDF sections that powered each response.

Ready to quiz yourself?