Chat with Your PDFs Using RAG

Build a local RAG app that answers questions about any PDF using LangChain and Ollama.

Introduction

30 Second Summary

We all have PDFs filled with information we need. Instead of scrolling through pages searching for an answer, imagine asking a question and getting an instant, accurate response pulled directly from the document.

In this project, you will build a local AI-powered chatbot that reads any PDF and answers your questions about it. You will create a complete Retrieval-Augmented Generation (RAG) pipeline using LangChain, Ollama, and ChromaDB.

What You'll Build

A command-line Python application that ingests PDFs, stores their content as vector embeddings in a local database, and uses a local LLM to answer questions grounded in the document's actual text.

By the end of this project, you'll have:

  • A complete RAG pipeline that loads, chunks, embeds, stores, retrieves, and generates answers from document content.
  • A persistent ChromaDB vector store that indexes your PDF for fast similarity search across sessions.
  • An interactive chat loop that answers questions using a local Llama 3.1 model via Ollama.
  • Secret Mission: Add conversation memory so follow-up questions understand context from previous answers.

Are there any prerequisites?

You should be comfortable writing Python scripts and familiar with the basics of how LLM APIs work. No prior experience with RAG, LangChain, or vector databases is needed.

Set Up Your Environment

Before you can chat with a PDF, you need a local AI stack that can understand documents and generate answers. This step sets up three layers: a Python environment, the libraries that orchestrate your RAG pipeline, and the AI models that power it.

You already have Python 3.14 on your machine, but ChromaDB (the vector database you will use) currently crashes on Python 3.14. You will install Python 3.13 specifically for this project and keep everything isolated in a virtual environment.

In this step, get ready to:
  • Install Python 3.13 and create a virtual environment.
  • Install the Python packages your RAG app needs.
  • Pull the Ollama models for embedding and chat.
Install Python 3.13 and create a virtual environment

Python 3.13 is the version that works reliably with all the libraries in this project. You will install it alongside your existing Python 3.14 and use it only inside this project's virtual environment.

  • In VS Code, open a terminal by pressing Ctrl+` (macOS) or Ctrl+` (Windows).
  • Check if Python 3.13 is already installed by running this command:
python3.13 --version

✔️ I see Python 3.13.x

Python 3.13 is already on your machine. Skip ahead to creating the project folder below.

ⓧ Command not found

Python 3.13 is not installed yet. Install it with Homebrew by running this command:

brew install python@3.13

This download takes a minute or two. Once it finishes, verify the install by running:

python3.13 --version

You should now see a version number starting with 3.13.

Still not seeing Python 3.13?

If Homebrew says the formula is already installed but the command is not found, you may need to restart your terminal. Close the terminal panel and reopen it with Ctrl+`.

help me troubleshoot my Python 3.13 installation on macOS with Homebrew

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir pdf-rag-app
cd pdf-rag-app
  • Create a virtual environment using Python 3.13 by running this command:
python3.13 -m venv venv

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python, so different projects can use different versions of the same library without conflicts.

  • Now open the pdf-rag-app folder in VS Code by pressing Cmd+K then Cmd+O (macOS) or Ctrl+K then Ctrl+O (Windows).
  • Navigate to Desktop, select the pdf-rag-app folder, and click Open.
  • Open a fresh terminal in the new window by pressing Ctrl+`.
  • Activate your virtual environment by running this command:
source venv/bin/activate
  • Verify the environment is working by running:
python --version

You should see Python 3.13.x printed in the terminal. Your prompt should also show (venv) at the beginning, confirming the virtual environment is active.

Seeing Python 3.14 instead of 3.13?

If python --version shows 3.14, the virtual environment was likely created with the wrong Python. Delete it and recreate it:

rm -rf venv
python3.13 -m venv venv
source venv/bin/activate

help me fix my virtual environment so it uses Python 3.13 instead of 3.14

Install Python dependencies

Your RAG app uses LangChain to orchestrate the entire pipeline, from loading PDFs to generating answers. You also need a few companion packages for Ollama integration, ChromaDB storage, and PDF parsing.

  • Install all the project dependencies by running this command:
pip install langchain langchain-ollama langchain-chroma langchain-community langchain-text-splitters pypdf

What do these packages do?

  • langchain is the core framework that connects your document loader, vector store, and LLM into a single chain.
  • langchain-ollama provides the ChatOllama and OllamaEmbeddings classes for local model inference.
  • langchain-chroma integrates ChromaDB as your vector store.
  • langchain-community includes community-maintained document loaders like PyPDFLoader.
  • langchain-text-splitters handles splitting documents into chunks.
  • pypdf is the PDF parsing engine that PyPDFLoader uses under the hood.

The install may take a minute as pip downloads all the packages and their dependencies. Once it finishes, you should see a Successfully installed message listing all the packages.

Seeing a permissions error?

Make sure your (venv) prefix is visible in your terminal prompt. If it is missing, reactivate the environment:

source venv/bin/activate

help me fix pip install errors in my virtual environment

Pull Ollama models

Your app needs two models: one to convert text into vector embeddings, and one to generate answers from retrieved context. Both run locally through Ollama.

  • Pull the embedding model by running this command:
ollama pull nomic-embed-text

This downloads the nomic-embed-text model (~274MB). It converts text into numerical vectors that capture meaning.

  • Pull the chat model by running this command:
ollama pull llama3.1:8b

Why these two models?

nomic-embed-text is a lightweight embedding model (137M parameters) that excels at RAG tasks. It is the most popular embedding model on Ollama with over 75 million pulls.

llama3.1:8b is a capable chat model that fits comfortably in 16GB of RAM alongside the embedding model. It provides high-quality answers while staying fully local.

The llama3.1:8b model is approximately 5GB. This download may take several minutes depending on your internet connection.

  • Confirm both models are downloaded by running:
ollama list

You should see both nomic-embed-text and llama3.1:8b in the output list.

Model download stuck or failing?

Make sure Ollama is running. You can check by opening a new terminal tab and running ollama serve. If it says the address is already in use, Ollama is running and the pull should work.

help me troubleshoot Ollama model downloads

Your local AI stack is fully set up. Next, you will load a PDF and split it into chunks that your embedding model can process.

Load and Chunk a PDF

Your environment is ready with all dependencies installed and both Ollama models pulled. Now it's time to write the code that reads a PDF and prepares it for AI-powered question answering.

The first step in any RAG pipeline is getting your data into a format the LLM can work with. A PDF might have dozens of pages, but an LLM can only process limited context at once. You'll solve this by loading the PDF and slicing it into smaller, overlapping chunks.

In this step, get ready to:
  • Load a PDF into Python as Document objects.
  • Split the document into overlapping text chunks.
  • Verify the output by inspecting chunk content and metadata.
Load a PDF document

LangChain provides a PyPDFLoader class that reads each page of a PDF as a separate Document object. Each Document has two attributes: page_content (the extracted text) and metadata (a dictionary with the source file path and page number).

  • In VS Code, create a new file called rag_app.py in your project folder.
  • Add the following imports to the top of rag_app.py:
import warnings
warnings.filterwarnings("ignore")

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

What does this code do?

  • warnings.filterwarnings("ignore") suppresses noisy deprecation warnings from LangChain's dependencies so your terminal output stays clean.
  • PyPDFLoader is LangChain's built-in PDF reader. It uses the pypdf library under the hood to extract text page by page.
  • RecursiveCharacterTextSplitter is the tool you'll use to break those pages into smaller chunks. You'll configure it in the next substep.

Seeing ModuleNotFoundError?

This means your virtual environment isn't active. Check your terminal prompt for the (venv) prefix. If it's missing, activate it again from your project folder.

help me fix a ModuleNotFoundError for langchain imports

Split the document into chunks

RecursiveCharacterTextSplitter breaks text at natural boundaries (paragraphs, sentences, words) to keep chunks readable. Overlapping chunks ensure that context near a split boundary isn't lost.

  • Below your imports in rag_app.py, add the following function:
def load_and_chunk_pdf(pdf_path):
    """Load a PDF and split it into chunks."""
    # Load each page as a separate Document object
    loader = PyPDFLoader(pdf_path)
    documents = loader.load()
    print(f"Loaded {len(documents)} pages from {pdf_path}")

    # Split pages into smaller overlapping chunks
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
    )
    chunks = text_splitter.split_documents(documents)
    print(f"Split into {len(chunks)} chunks")
    return chunks

What does this function do?

  • PyPDFLoader(pdf_path) creates a loader for the given file path. Calling .load() reads the PDF and returns a list of Document objects, one per page.
  • chunk_size=1000 means each chunk targets roughly 1000 characters. This keeps chunks small enough for the embedding model to process effectively.
  • chunk_overlap=200 means consecutive chunks share 200 characters of overlap. This prevents important sentences from being cut in half at chunk boundaries.
  • split_documents() applies the splitter to all loaded pages and returns a new list of smaller Document objects, each with the original page metadata preserved.
  • Save rag_app.py.

Seeing a syntax error?

Double-check that the f" prefix is present on both print statements. Without it, the curly braces won't be interpreted as expressions. Also confirm the function body is indented with 4 spaces, not tabs.

help me fix a syntax error in my load_and_chunk_pdf function

✔️ Awesome, I've got everything!

Great. Double check you've saved rag_app.py before moving on.

ⓧ I'd like to double check the full code

Here's what your complete rag_app.py should look like so far:

import warnings
warnings.filterwarnings("ignore")

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter


def load_and_chunk_pdf(pdf_path):
    """Load a PDF and split it into chunks."""
    loader = PyPDFLoader(pdf_path)
    documents = loader.load()
    print(f"Loaded {len(documents)} pages from {pdf_path}")

    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
    )
    chunks = text_splitter.split_documents(documents)
    print(f"Split into {len(chunks)} chunks")
    return chunks
Verify the chunking output

Before building the rest of the pipeline, confirm that your function correctly loads pages and produces chunks with metadata intact.

  • Find any PDF file on your computer that you'd like to chat with. It can be a report, a paper, or any document with a few pages of text.
  • Note the full file path to the PDF (e.g. ~/Desktop/report.pdf).
  • Test your function by running this command in your terminal:
python -c "from rag_app import load_and_chunk_pdf; chunks = load_and_chunk_pdf([[PDFPATH='path/to/your/file.pdf']]); print(chunks[0].page_content[:200]); print(chunks[0].metadata)"

What should I see?

You should see output similar to this:

  • A line showing how many pages were loaded (e.g. Loaded 12 pages from ...).
  • A line showing the chunk count (e.g. Split into 47 chunks).
  • The first 200 characters of the first chunk's text content.
  • A metadata dictionary containing source (your file path) and page (the page number, starting from 0).

Seeing FileNotFoundError?

Make sure the PDF path you provided is correct and includes the file extension. If your path contains spaces, wrap it in quotes. Try dragging the PDF file directly into your terminal to auto-fill the path.

help me fix a FileNotFoundError when loading my PDF

Your PDF is loaded, chunked, and ready for the next stage. Next up, you'll turn these text chunks into vector embeddings and store them in a searchable database.

Create Embeddings and Store in ChromaDB

Your PDF is split into manageable chunks, but raw text alone isn't searchable by meaning. To find the right passages for a question, you need to convert each chunk into a numerical representation that captures its semantic content.

In this step, you'll use Ollama's nomic-embed-text model to generate vector embeddings for your chunks. Then you'll store those embeddings in a ChromaDB vector database that persists on disk.

In this step, get ready to:
  • Initialize the Ollama embedding model.
  • Create a persistent ChromaDB vector store from your PDF chunks.
  • Test similarity search to verify relevant chunks are returned.
Initialize the Ollama embedding model

An embedding model converts text into a list of numbers (a vector) that represents its meaning. Texts with similar meanings produce vectors that are close together in space.

What are vector embeddings?

Think of embeddings as coordinates on a map of meaning. The sentence "dogs love treats" and "puppies enjoy snacks" would land near each other on this map because they mean similar things.

The nomic-embed-text model is lightweight (137M parameters) and runs easily alongside your llama3.1:8b chat model. When your RAG app searches for relevant chunks, it compares the embedding of your question against the embeddings of all stored chunks to find the closest matches.

Create a ChromaDB vector store from the chunks

ChromaDB is a vector database that stores your embeddings on disk. Once your chunks are embedded and stored, you can query them across sessions without re-processing the PDF.

  • In rag_app.py, find the existing import lines at the top of the file. Add these two new imports directly below the RecursiveCharacterTextSplitter import:
from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma
  • Now add the create_vector_store function below your existing load_and_chunk_pdf function by copying this code:
def create_vector_store(chunks):
    """Create or load a ChromaDB vector store."""
    embeddings = OllamaEmbeddings(model="nomic-embed-text")

    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory="./chroma_db",
    )
    print("Vector store created and persisted to ./chroma_db/")
    return vector_store

What does this code do?

This function takes your document chunks and turns them into a searchable vector database.

  • OllamaEmbeddings(model="nomic-embed-text") connects to Ollama's local embedding model to generate vectors for each chunk.
  • Chroma.from_documents() takes your chunks, generates embeddings for each one, and stores them in a ChromaDB collection.
  • persist_directory="./chroma_db" tells ChromaDB to save the database to disk so it survives between runs.
  • Save rag_app.py.

Seeing a red underline on the imports?

Make sure your virtual environment is active in VS Code. Check the bottom-left corner of VS Code for the Python interpreter. It should show your rag_env environment, not the system Python.

If the packages aren't recognized, confirm they installed correctly by running pip list | grep langchain in your terminal. You should see langchain-ollama and langchain-chroma in the output.

Help me fix my import errors in rag_app.py

Test similarity search

Before moving on, you need to verify that the vector store actually returns relevant chunks when queried. You'll add temporary test code to the bottom of your file, run it, and then remove it.

  • Add this temporary test code at the very bottom of rag_app.py, below the create_vector_store function:
# Temporary test code - remove after verifying
if __name__ == "__main__":
    pdf_path = input("Enter the path to your PDF: ").strip()

    # Load and chunk the PDF
    chunks = load_and_chunk_pdf(pdf_path)

    # Create the vector store
    vector_store = create_vector_store(chunks)

    # Test similarity search
    results = vector_store.similarity_search("What is this document about?", k=4)
    print(f"\nFound {len(results)} relevant chunks:")
    for i, doc in enumerate(results):
        print(f"\n--- Chunk {i+1} (page {doc.metadata.get('page', 'N/A')}) ---")
        print(doc.page_content[:200])

What does this test code do?

  • It loads and chunks your PDF using the function from the previous step.
  • It creates the vector store, embedding all chunks with nomic-embed-text.
  • similarity_search() converts your query into an embedding, then finds the 4 closest chunks by vector distance.
  • It prints the first 200 characters of each result so you can verify they relate to your query.
  • Save rag_app.py.
  • Run the test by executing this command in your terminal:
python rag_app.py
  • When prompted, enter the path to your PDF file (the same one you used in the previous step).

The embedding process may take a moment depending on the size of your PDF. You should see output showing your chunks being loaded, the vector store being created, and then 4 relevant chunks printed with their page numbers and content previews.

What should I see?

Your terminal should display something like:

Loaded 12 pages from your-file.pdf
Split into 47 chunks
Vector store created and persisted to ./chroma_db/

Found 4 relevant chunks:

--- Chunk 1 (page 0) ---
[First 200 characters of relevant text...]

The exact numbers depend on your PDF. The key thing to verify is that all 4 chunks contain text related to your query.

Seeing a connection error?

Make sure Ollama is running. Open a separate terminal window and run ollama list to confirm it responds. If Ollama isn't running, start it with ollama serve in a new terminal tab.

If you see an error about the model not being found, re-pull it by running ollama pull nomic-embed-text in your terminal.

Help me debug my vector store creation

  • Confirm that a chroma_db directory was created in your project folder by running:
ls chroma_db

You should see files inside this directory. This is your persistent vector database stored on disk.

  • Now remove the temporary test code from the bottom of rag_app.py. Delete everything from the # Temporary test code comment down to the end of the file.
  • Save rag_app.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file with the test code removed.

ⓧ I'd like to double check the full code

Here's what your rag_app.py should look like at this point:

import warnings
warnings.filterwarnings("ignore")

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma


def load_and_chunk_pdf(pdf_path):
    """Load a PDF and split it into chunks."""
    loader = PyPDFLoader(pdf_path)
    documents = loader.load()
    print(f"Loaded {len(documents)} pages from {pdf_path}")

    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
    )
    chunks = text_splitter.split_documents(documents)
    print(f"Split into {len(chunks)} chunks")
    return chunks


def create_vector_store(chunks):
    """Create or load a ChromaDB vector store."""
    embeddings = OllamaEmbeddings(model="nomic-embed-text")

    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory="./chroma_db",
    )
    print("Vector store created and persisted to ./chroma_db/")
    return vector_store

Your chunks are now embedded and stored in a persistent vector database. Next up, you'll build the full RAG chain that retrieves relevant chunks and generates grounded answers using your local LLM.

Build the RAG Chain and Chat Loop

In the previous step, you stored your PDF chunks as vector embeddings in ChromaDB. Now it's time to connect that vector store to a local LLM so your app can retrieve relevant chunks and generate grounded answers.

This is where the "generation" half of Retrieval-Augmented Generation comes alive. You'll build a LangChain chain that takes a user question, fetches the most relevant document chunks, and passes them to Llama 3.1 with instructions to answer based on that context.

In this step, get ready to:
  • Connect the Ollama chat model and retriever into a RAG chain.
  • Write a prompt template that injects retrieved document context.
  • Build an interactive chat loop for question answering.
Set up the ChatOllama LLM and retriever

Your RAG chain needs access to LangChain's chat model wrapper, prompt templates, and chain-building utilities. You'll add these imports alongside the ones already in your file.

  • In rag_app.py, find this import line near the top of your file:
from langchain_ollama import OllamaEmbeddings
  • Replace it with this updated version that also imports ChatOllama:
from langchain_ollama import OllamaEmbeddings, ChatOllama
  • Add these three new import lines directly below the from langchain_chroma import Chroma line:
from langchain_core.prompts import ChatPromptTemplate
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain

What do these imports do?

  • ChatOllama is LangChain's wrapper for Ollama chat models. It sends your prompt to the local llama3.1:8b model and returns the response.
  • ChatPromptTemplate lets you define a structured prompt with system and human message slots.
  • create_retrieval_chain wires the retriever and the LLM together so that relevant chunks are fetched before the LLM generates an answer.
  • create_stuff_documents_chain takes the retrieved documents and "stuffs" them into the prompt's context placeholder.

You won't see a change yet. The next chunk uses these imports to build the RAG chain function.

Create the retrieval QA prompt and chain

The retrieval chain needs two pieces: a retriever that fetches relevant chunks, and a prompt that tells the LLM how to use them. You'll wrap both into a single function.

  • Add the following create_rag_chain function below your existing create_vector_store function in rag_app.py:
def create_rag_chain(vector_store):
    """Build the RAG chain with retriever and LLM."""
    llm = ChatOllama(model="llama3.1:8b")
    retriever = vector_store.as_retriever(search_kwargs={"k": 4})

    system_prompt = (
        "You are a helpful assistant that answers questions based on the "
        "provided context from a PDF document. If the answer is not in the "
        "context, say you don't have enough information to answer.\n\n"
        "Context:\n{context}"
    )

    prompt = ChatPromptTemplate.from_messages([
        ("system", system_prompt),
        ("human", "{input}"),
    ])

    question_answer_chain = create_stuff_documents_chain(llm, prompt)
    rag_chain = create_retrieval_chain(retriever, question_answer_chain)
    return rag_chain

What does this function do?

  • ChatOllama(model="llama3.1:8b") initializes the local Llama 3.1 model as the generation engine.
  • vector_store.as_retriever(search_kwargs={"k": 4}) creates a retriever that returns the 4 most similar chunks for each query.
  • The system_prompt tells the LLM to only answer from the provided context. The {context} placeholder is where LangChain injects the retrieved chunks at runtime.
  • create_stuff_documents_chain combines the LLM and prompt into a chain that formats documents into the context slot.
  • create_retrieval_chain connects the retriever to that document chain, creating the full retrieve-then-generate pipeline.

Why tell the LLM to say it does not know?

Without this instruction, the LLM might hallucinate answers that sound plausible but are not supported by your PDF. Grounding the model in retrieved context is the core benefit of RAG over plain LLM prompting.

You won't see a change yet. The next chunk adds the main() function that calls everything together.

Build an interactive chat loop

The final piece ties all your functions together and gives you an interactive terminal interface. You'll ask a question, the chain retrieves relevant chunks, and the LLM generates a grounded answer.

  • Add the following main() function and entry point at the bottom of rag_app.py, below create_rag_chain:
def main():
    pdf_path = input("Enter the path to your PDF: ").strip()

    print("\nLoading and processing PDF...")
    chunks = load_and_chunk_pdf(pdf_path)

    print("\nCreating vector store (this may take a moment)...")
    vector_store = create_vector_store(chunks)

    print("\nBuilding RAG chain...")
    rag_chain = create_rag_chain(vector_store)

    print("\nReady! Ask questions about your PDF. Type 'quit' to exit.\n")

    while True:
        question = input("You: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not question:
            continue

        response = rag_chain.invoke({"input": question})
        print(f"\nAssistant: {response['answer']}\n")


if __name__ == "__main__":
    main()

What does this code do?

  • main() prompts for a PDF path, then runs the full pipeline: load, chunk, embed, store, and build the chain.
  • The while True loop accepts questions until the user types quit, exit, or q.
  • rag_chain.invoke({"input": question}) sends the question through the retrieval chain. LangChain fetches the top 4 chunks, injects them into the prompt, and returns the LLM's answer.
  • response['answer'] extracts just the generated text from the chain's response dictionary.
  • if __name__ == "__main__" ensures the script only runs when executed directly, not when imported.
  • Save rag_app.py.
  • Run the script by entering this command in your terminal:
python rag_app.py
  • When prompted, enter the path to the same PDF you used in the previous step.
  • Wait for the "Ready!" message, then type a question about the PDF content.

You should see a grounded answer pulled from the PDF's content. Type quit to exit the chat loop.

First response is slow?

The first question may take 10-30 seconds as Ollama loads the llama3.1:8b model into memory. Subsequent answers are much faster because the model stays loaded.

Not getting an answer?

  • Make sure Ollama is running. Open a separate terminal and run ollama list to confirm both models are available.
  • Check that your PDF path is correct. The script will error if the file does not exist at that location.
  • If you see a connection error, Ollama's server may not be running. Start it with ollama serve in a separate terminal window.

Help me debug my RAG chain

✔️ Awesome, I've got everything!

Great. Double check you have saved rag_app.py and that your chat loop responds to questions.

ⓧ I'd like to double check the full code

import warnings
warnings.filterwarnings("ignore")

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain


def load_and_chunk_pdf(pdf_path):
    """Load a PDF and split it into chunks."""
    loader = PyPDFLoader(pdf_path)
    documents = loader.load()
    print(f"Loaded {len(documents)} pages from {pdf_path}")

    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
    )
    chunks = text_splitter.split_documents(documents)
    print(f"Split into {len(chunks)} chunks")
    return chunks


def create_vector_store(chunks):
    """Create or load a ChromaDB vector store."""
    embeddings = OllamaEmbeddings(model="nomic-embed-text")

    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory="./chroma_db",
    )
    print("Vector store created and persisted to ./chroma_db/")
    return vector_store


def create_rag_chain(vector_store):
    """Build the RAG chain with retriever and LLM."""
    llm = ChatOllama(model="llama3.1:8b")
    retriever = vector_store.as_retriever(search_kwargs={"k": 4})

    system_prompt = (
        "You are a helpful assistant that answers questions based on the "
        "provided context from a PDF document. If the answer is not in the "
        "context, say you don't have enough information to answer.\n\n"
        "Context:\n{context}"
    )

    prompt = ChatPromptTemplate.from_messages([
        ("system", system_prompt),
        ("human", "{input}"),
    ])

    question_answer_chain = create_stuff_documents_chain(llm, prompt)
    rag_chain = create_retrieval_chain(retriever, question_answer_chain)
    return rag_chain


def main():
    pdf_path = input("Enter the path to your PDF: ").strip()

    print("\nLoading and processing PDF...")
    chunks = load_and_chunk_pdf(pdf_path)

    print("\nCreating vector store (this may take a moment)...")
    vector_store = create_vector_store(chunks)

    print("\nBuilding RAG chain...")
    rag_chain = create_rag_chain(vector_store)

    print("\nReady! Ask questions about your PDF. Type 'quit' to exit.\n")

    while True:
        question = input("You: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not question:
            continue

        response = rag_chain.invoke({"input": question})
        print(f"\nAssistant: {response['answer']}\n")


if __name__ == "__main__":
    main()

Your RAG app is fully functional. You can now chat with any PDF using a completely local, private AI pipeline.

Secret mission

Add Conversation Memory

Your RAG chatbot answers questions well, but each question exists in isolation. Ask "tell me more about that" and the chatbot has no idea what "that" refers to. In this secret mission, you will add conversation memory so follow-up questions understand context from previous answers.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama models (llama3.1:8b and nomic-embed-text, approximately 5.3GB total disk space).
  • ChromaDB vector store (./chroma_db/ directory).
  • Python 3.13 virtual environment and project files.
  • Python 3.13 installed via Homebrew.

Keep everything running

No action needed. Choose this if you plan to keep chatting with PDFs or want to index new documents later.

  • Ollama models stay cached on disk and load instantly when you run the app again.
  • The ./chroma_db/ directory persists your indexed PDF so you can query it without re-processing.
  • Your virtual environment stays ready to activate with no reinstallation needed.
  • Total disk usage is approximately 5.3GB for the models plus a small amount for the vector store and project files.

Pause - I'll come back to this later

Remove the Ollama models to free up approximately 5.3GB of disk space. Keep your project files and vector store so you can resume quickly.

  • Remove the Ollama models by running these commands in your terminal:
ollama rm llama3.1:8b
ollama rm nomic-embed-text
  • Your ./chroma_db/ directory and rag_app.py stay intact.
  • When you come back, re-pull the models with ollama pull llama3.1:8b and ollama pull nomic-embed-text.

Delete - I don't want to use this again

Remove all project resources and start fresh. This reclaims all disk space used by the project.

  • Remove the Ollama models by running:
ollama rm llama3.1:8b
ollama rm nomic-embed-text
  • Delete the ChromaDB vector store and project files. Navigate to your project directory and run:
cd ~/Desktop
rm -rf chroma_db/
rm rag_app.py
  • If you no longer need your virtual environment, delete it:
rm -rf venv/
  • If you installed Python 3.13 only for this project and no longer need it, uninstall it via Homebrew:
brew uninstall python@3.13

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully local PDF chatbot powered by a complete RAG pipeline, all running on your own machine with zero API costs.

You've learned how to:

  • Built a complete RAG pipeline from scratch: PDF loading, text chunking, vector embedding, storage, retrieval, and generation.
  • Used LangChain to orchestrate document Q&A with clean, modular code and Ollama to run everything locally for full privacy.
  • Stored embeddings in a persistent ChromaDB vector store that you can reuse across sessions without re-processing your PDF.
  • (Secret Mission) Added conversation memory for multi-turn contextual Q&A, so follow-up questions understand context from previous answers.

Ready to quiz yourself?