Build a Local PDF Q&A Tool with Ollama

Build a fully local RAG pipeline that answers questions about any PDF.

Introduction

30 Second Summary

You have PDFs full of important information, but reading them cover to cover every time you need an answer is painful. AI chatbots can help, but they send your private documents to someone else's server.

In this project, you will build a local PDF Q&A tool that runs entirely on your Mac. You give it any PDF, ask questions in plain English, and get accurate answers powered by a large language model running on your own machine.

What You'll Build

A terminal-based Python script that implements a complete RAG pipeline from scratch, using Ollama to run both a chat model and an embedding model locally.

By the end of this project, you'll have:

  • Ollama running locally with a chat model (llama3.2) and an embedding model (nomic-embed-text).
  • A complete RAG pipeline built from scratch with no frameworks, covering PDF extraction, text chunking, embedding generation, cosine similarity search, and grounded generation.
  • A working Q&A loop that answers questions about any PDF using only your document's content.
  • Secret Mission: Add streaming responses and source transparency so you can see exactly which chunks the LLM is using.

Are there any prerequisites?

You should be comfortable with basic Python concepts like functions, loops, and lists. No prior AI or machine learning experience is needed.

Set Up Your Environment

Before you can ask questions about a PDF, you need the tools that make it possible. Ollama runs large language models locally on your Mac, and a few Python packages handle PDF reading and math.

In this step, you will install Ollama with two models, set up a Python project folder, and get everything ready so the next step is pure coding.

In this step, get ready to:
  • Install Ollama and pull two AI models (a chat model and an embedding model).
  • Create a Python virtual environment and install project packages.
  • Create the project file and add a sample PDF.
Install Ollama and pull the models

Ollama is a lightweight tool that downloads and runs open-source LLMs on your machine. You will install it, then pull two models: one for chatting and one for creating embeddings.

  • Open your browser and go to ollama.com/download.
  • Click Download for macOS.
  • Open the downloaded Ollama.dmg file and drag Ollama into your Applications folder.
  • Open Ollama from your Applications folder (double-click it).

What happens on first launch?

When Ollama opens for the first time, it asks for permission to install a command-line tool. Click Allow so the ollama command becomes available in your Terminal.

Ollama then runs as a background service. You will see a small llama icon in your menu bar.

  • Press Cmd+Space to open Spotlight.
  • Type Terminal and press Enter to open it.
  • Verify Ollama installed correctly by running this command:
ollama --version

✔️ I see a version number

Ollama is installed and ready. Time to pull your models.

ⓧ Command not found

The CLI did not install correctly. Try these fixes:

  • Make sure you opened the Ollama app at least once and clicked Allow when prompted to install the command-line tool.
  • Close and reopen Terminal so it picks up the new path.
  • If neither works, reinstall by running this in Terminal:
curl -fsSL https://ollama.com/install.sh | sh
  • Pull the chat model (llama3.2, 2GB) by running this command:
ollama pull llama3.2

What is llama3.2?

This is Meta's Llama 3.2 model with 3 billion parameters. It has a 128K context window and runs comfortably on Apple Silicon. This model handles the "thinking" part of your Q&A tool.

  • Pull the embedding model (nomic-embed-text, 274MB) by running this command:
ollama pull nomic-embed-text

What is an embedding model?

An embedding model converts text into a list of numbers (a vector) that captures its meaning. Later, you will use these vectors to find which chunks of your PDF are most relevant to a question.

  • Confirm both models downloaded successfully by running:
ollama list
  • You should see both llama3.2 and nomic-embed-text in the output.

Don't see both models?

If a model pull failed, check your internet connection and try the ollama pull command again. Large model downloads can time out on slow connections.

help me troubleshoot why my Ollama model pull failed

Create your project folder and install packages

Now you need a dedicated folder for your project and a Python virtual environment to keep your packages isolated.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir pdf-qa
cd pdf-qa
  • Create a Python virtual environment and activate it by running these commands:
python3 -m venv venv
source venv/bin/activate

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python, so different projects never conflict with each other.

You will know it is active when you see (venv) at the start of your terminal prompt.

  • Install the three packages your project needs by running:
pip install ollama pypdf numpy

What do these packages do?

  • ollama lets Python talk to the Ollama models running on your machine.
  • pypdf reads PDF files and extracts their text content.
  • numpy provides fast math operations for comparing vectors (needed for search).
  • Verify all packages installed correctly by running:
python3 -c "import ollama, pypdf, numpy; print('Ready')"
  • You should see Ready printed in the terminal.

Seeing an import error?

Make sure your virtual environment is active (you should see (venv) in your prompt). If not, run source venv/bin/activate again before retrying the install.

help me fix my Python package import error for ollama, pypdf, or numpy

Create the project file and add a sample PDF

You need two things in your project folder: the Python script you will build throughout this project, and a PDF to test with.

  • Create the empty project file by running:
touch pdf_qa.py
  • Confirm the file was created by running:
ls
  • You should see pdf_qa.py and venv in the output.
  • Find or download any multi-page PDF (a research paper, report, or ebook chapter works well).
  • Move or copy that PDF file into your pdf-qa folder on your Desktop.

Pick a PDF you care about

The more interesting your PDF, the more fun it is to test your Q&A tool. A paper from your field, a company report, or study notes all work perfectly.

  • Open pdf_qa.py in VS Code by running:
code pdf_qa.py

code command not found?

If the code command is not recognized, open VS Code manually. Press Cmd+Shift+P inside VS Code, type Shell Command: Install 'code' command in PATH, and select it. Then try the command again.

help me set up the VS Code code command in my terminal

Your environment is ready. Next up, you will talk to the LLM directly to see what it knows (and does not know) about your PDF.

Chat with the LLM Directly

Your environment is set up and Ollama is running with both models pulled. Before building the full RAG pipeline, you need to see what happens when you ask an LLM about a document it has never seen.

In this step, you'll chat with the LLM directly to discover a critical limitation. This limitation is the exact problem that RAG solves.

In this step, get ready to:
  • Test the LLM with a general knowledge question.
  • Ask the LLM about your specific PDF and observe the gap.
Write a test chat script

Large language models have impressive general knowledge, but they only know what was in their training data. Your PDF was never part of that training.

  • In pdf_qa.py (open in VS Code), add the following imports, configuration, and a test chat call:
import ollama
import numpy as np
from pypdf import PdfReader

# Configuration
LLM_MODEL = "llama3.2"
EMBEDDING_MODEL = "nomic-embed-text"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50

# Temporary test: ask the LLM a general question
response = ollama.chat(
    model=LLM_MODEL,
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
print(response.message.content)

What does this code do?

The first three lines import the libraries you installed in Step 1. The configuration section stores model names and chunking parameters as constants so they're easy to change later.

The test section calls ollama.chat() with your llama3.2 model and a simple question. The response comes back as an object, and response.message.content holds the text answer.

  • Save pdf_qa.py.
  • Run the script in your terminal:
python3 pdf_qa.py

You should see a multi-sentence explanation of why the sky is blue, generated entirely by the local LLM.

Not seeing a response?

If you see a connection error, Ollama may not be running. Open a new terminal tab and run ollama list to check that Ollama is active and both models appear.

If you see a model-not-found error, double check that you pulled llama3.2 (not llama3 or another variant).

help me fix a connection or model error when calling ollama.chat() in Python

Ask about your PDF's content

The LLM handled a general knowledge question easily. Now try something only someone who has read your specific PDF would know.

  • In pdf_qa.py, find this line:
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
  • Replace "Why is the sky blue?" with a specific question about your PDF's content (for example, a question about a name, date, or finding mentioned in the document). The line should now look something like this:
    messages=[{"role": "user", "content": "What were the key findings in section 3?"}],
  • Save pdf_qa.py and run it again:
python3 pdf_qa.py

This time, the LLM either makes something up (a hallucination) or admits it does not have that information. It has no way to access your PDF.

Why can't the LLM answer?

LLMs only know what was in their training data. Your PDF was never part of that data, so the model is guessing blindly.

This is the core problem RAG solves. Instead of hoping the model already knows your content, you retrieve the relevant passages and inject them directly into the prompt.

  • Now remove the temporary test code from pdf_qa.py. Delete everything below the CHUNK_OVERLAP = 50 line, so your file contains only the imports and configuration.
  • Save pdf_qa.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file with just the imports and configuration constants.

ⓧ I'd like to double check the full code

import ollama
import numpy as np
from pypdf import PdfReader

# Configuration
LLM_MODEL = "llama3.2"
EMBEDDING_MODEL = "nomic-embed-text"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50

You've seen the gap firsthand. The LLM is capable, but blind to your documents. Next up, you'll extract text from your PDF and split it into searchable chunks.

Extract and Chunk the PDF

In the last step, you saw the LLM's blind spot. It can chat about general knowledge, but it has zero awareness of your PDF's content. To fix this, you need to get the text out of the PDF and into a format Python can search through.

In this step, you'll write two functions. One extracts raw text from a PDF file. The other splits that text into small, overlapping chunks that can be individually searched later.

In this step, get ready to:
  • Extract all text from a PDF into a single string.
  • Split the text into overlapping chunks of 500 characters.
  • Verify the extraction and chunking work correctly on your sample PDF.
Write the text extraction function

PDFs store text in a structured binary format, not as plain readable characters. The pypdf library handles the parsing so you can pull out readable text page by page.

  • In pdf_qa.py, add the following function below your configuration constants:
def extract_text(pdf_path):
    """Extract all text from a PDF file."""
    reader = PdfReader(pdf_path)
    text = ""
    for page in reader.pages:
        page_text = page.extract_text()
        if page_text:
            text += page_text
    return text

What does this code do?

  • PdfReader(pdf_path) opens and parses the PDF file.
  • reader.pages gives you a list of every page in the document.
  • page.extract_text() pulls readable text from a single page. Some pages may return nothing (like image-only pages), so the if page_text check skips those.
  • The function concatenates all page text into one long string and returns it.
  • Now add these temporary test lines at the very bottom of your file to verify extraction works:
# Temporary test code (remove later)
text = extract_text("[[PDFFILE=\"your-pdf-filename.pdf\"]]") 
print(f"Extracted {len(text)} characters")
  • Save pdf_qa.py.
  • Run your script by entering this command:
python3 pdf_qa.py

You should see a line like Extracted 23847 characters (your number will differ depending on your PDF's length).

Seeing 0 characters or an error?

  • If you see a FileNotFoundError, double-check that your PDF filename matches exactly (including the .pdf extension) and that the PDF is inside your pdf-qa folder.
  • If you see 0 characters, your PDF might be image-based (scanned). Try a different PDF that contains selectable text.
  • Make sure your virtual environment is still active. You should see (venv) at the start of your terminal prompt.

Still stuck? Why is my PDF extraction returning 0 characters or throwing an error?

Write the text chunking function

Feeding the entire document to an embedding model at once loses granularity. Splitting into small chunks means each chunk can be independently matched to a question later.

Why overlapping chunks?

If you split text at exact boundaries, a sentence that spans two chunks gets cut in half. Neither half captures the full meaning. Overlap (50 characters in this project) means the end of one chunk repeats at the start of the next, so no information is lost at the seam.

  • In pdf_qa.py, add the following function below extract_text():
def split_text(text, chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

What does this code do?

  • The function uses a sliding window approach. It starts at position 0 and grabs 500 characters (CHUNK_SIZE).
  • Instead of jumping forward a full 500 characters, it only advances 450 (500 minus the 50-character CHUNK_OVERLAP). This creates the overlap between consecutive chunks.
  • The loop continues until start passes the end of the text, ensuring no characters are missed.
  • Now update the temporary test code at the bottom of your file to also test chunking. Replace the existing test lines with:
# Temporary test code (remove later)
text = extract_text("[[PDFFILE=\"your-pdf-filename.pdf\"]]") 
print(f"Extracted {len(text)} characters")

chunks = split_text(text)
print(f"Created {len(chunks)} chunks")
print(f"\nFirst chunk:\n{chunks[0]}")
  • Save pdf_qa.py.
  • Run the script again:
python3 pdf_qa.py

You should see three things: the character count, the chunk count, and the first chunk's readable text from your PDF.

Seeing unexpected chunk counts?

  • If you see 0 chunks, the text extraction likely returned an empty string. Go back and confirm the extraction step prints a non-zero character count first.
  • If the first chunk looks garbled, your PDF might use unusual encoding. Try a different PDF with standard text.

Need help? My chunks aren't being created correctly from my PDF text.

Clean up the temporary test code

The test code confirmed everything works. You'll build a proper interactive loop in a later step, so the temporary lines can go now.

  • In pdf_qa.py, delete all the lines below the split_text() function (everything from # Temporary test code onwards).
  • Save pdf_qa.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file with only the imports, configuration, and two functions remaining.

ⓧ I'd like to double check the full code

import ollama
import numpy as np
from pypdf import PdfReader

# Configuration
LLM_MODEL = "llama3.2"
EMBEDDING_MODEL = "nomic-embed-text"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50


def extract_text(pdf_path):
    """Extract all text from a PDF file."""
    reader = PdfReader(pdf_path)
    text = ""
    for page in reader.pages:
        page_text = page.extract_text()
        if page_text:
            text += page_text
    return text


def split_text(text, chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

Your PDF is now extracted and split into searchable pieces. Next up, you'll turn those text chunks into numerical vectors that can be compared mathematically to any question.

Embed and Search

Your PDF is now split into searchable chunks. The next challenge is figuring out which chunks are relevant to any given question.

Text embeddings solve this. An embedding converts text into a list of numbers (a vector) that captures its meaning. Similar meanings produce similar vectors. In this step, you will convert every chunk into an embedding, then build a search function that finds the closest matches to any question.

In this step, get ready to:
  • Write functions to generate text embeddings using Ollama.
  • Build a cosine similarity function to compare vectors.
  • Create a search function that retrieves the top 3 most relevant chunks for any question.
Write the embedding functions

The nomic-embed-text model you pulled earlier converts text into a vector of numbers. Two pieces of text with similar meaning produce vectors that point in similar directions.

  • In pdf_qa.py, find the split_text() function. Add the following two functions below it:
def get_embeddings(texts):
    """Generate embeddings for a list of texts using Ollama."""
    response = ollama.embed(model=EMBEDDING_MODEL, input=texts)
    return response.embeddings


def get_embedding(text):
    """Generate an embedding for a single text."""
    response = ollama.embed(model=EMBEDDING_MODEL, input=text)
    return response.embeddings[0]

What does this code do?

These two functions are wrappers around Ollama's embedding API.

  • get_embeddings() takes a list of texts and returns all their embeddings at once (batch mode). This is faster than embedding one at a time.
  • get_embedding() is a convenience wrapper for a single piece of text. It returns just the first (and only) embedding from the response.
  • Both use the EMBEDDING_MODEL constant (nomic-embed-text) you defined at the top of the file.
  • Save pdf_qa.py.
  • You won't see a change yet. The next functions will use these embeddings to search your chunks.
Write the similarity and search functions

To find which chunks are most relevant to a question, you need to measure how similar two vectors are. Cosine similarity does exactly that. It returns a value between -1 and 1, where 1 means identical meaning.

  • Below the embedding functions you just wrote, add the cosine_similarity() function:
def cosine_similarity(vec_a, vec_b):
    """Calculate cosine similarity between two vectors."""
    a = np.array(vec_a)
    b = np.array(vec_b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

What does this code do?

This function converts both vectors into numpy arrays, then calculates the cosine of the angle between them. A higher score means the two texts are more semantically similar.

The formula divides the dot product of the two vectors by the product of their magnitudes. You do not need to memorize this formula. The important thing is that it gives you a reliable similarity score.

  • Now add the find_relevant_chunks() function directly below cosine_similarity():
def find_relevant_chunks(question, chunks, embeddings, top_k=3):
    """Find the most relevant chunks for a question."""
    question_embedding = get_embedding(question)
    scores = []
    for i, emb in enumerate(embeddings):
        score = cosine_similarity(question_embedding, emb)
        scores.append((score, i))
    scores.sort(reverse=True)
    return [chunks[i] for _, i in scores[:top_k]]

What does this code do?

This function ties everything together into a search pipeline.

  • It embeds the question using get_embedding() so it becomes a vector in the same space as the chunks.
  • It compares the question vector against every chunk embedding using cosine_similarity().
  • It sorts by score (highest first) and returns the top 3 chunks.
  • Save pdf_qa.py.
  • You won't see a change yet. The next substep tests the full search pipeline.
Test the search pipeline

Time to verify that your search returns relevant chunks. You will add temporary test code to the bottom of the file, run it, and then remove it.

  • At the very bottom of pdf_qa.py, add this temporary test code:
# --- Temporary test code (remove after verifying) ---
pdf_path = [[PDFNAME="your-sample.pdf"]]
text = extract_text(pdf_path)
chunks = split_text(text)
print(f"Created {len(chunks)} chunks")

print("Generating embeddings...")
embeddings = get_embeddings(chunks)
print(f"Generated {len(embeddings)} embeddings")

question = "What is this document about?"
relevant = find_relevant_chunks(question, chunks, embeddings)
print(f"\nTop 3 chunks for: '{question}'\n")
for i, chunk in enumerate(relevant, 1):
    print(f"--- Chunk {i} ---")
    print(chunk[:200])
    print()

What does this code do?

This test code runs the full search pipeline end-to-end. It extracts text from your PDF, splits it into chunks, generates embeddings for all chunks, then searches for the 3 most relevant chunks matching your question.

The first 200 characters of each chunk are printed so you can quickly check whether the results make sense.

  • Save pdf_qa.py.
  • Run the script in your terminal:
python3 pdf_qa.py
  • You should see a chunk count, then "Generating embeddings..." (this may take 10-30 seconds the first time), followed by 3 printed chunks that relate to the general topic of your PDF.

Not seeing relevant results?

  • Make sure Ollama is running. Open a separate terminal window and run ollama list to confirm both models are available.
  • Check that your PDF path in the test code matches the actual filename (including the .pdf extension).
  • If you see an import error, confirm your virtual environment is still active. You should see (venv) at the start of your terminal prompt.

Still stuck? Help me debug why my embedding search isn't returning relevant chunks.

  • Once you have confirmed the 3 chunks look relevant, delete the temporary test code block (everything below the find_relevant_chunks() function).
  • Save pdf_qa.py again.

✔️ Awesome, I've got everything!

Great. Double check you have saved pdf_qa.py and removed the temporary test code at the bottom.

ⓧ I'd like to double check the full code

Here is what pdf_qa.py should look like at this point (no test code at the bottom):

import ollama
import numpy as np
from pypdf import PdfReader

# Configuration
LLM_MODEL = "llama3.2"
EMBEDDING_MODEL = "nomic-embed-text"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50


def extract_text(pdf_path):
    """Extract all text from a PDF file."""
    reader = PdfReader(pdf_path)
    text = ""
    for page in reader.pages:
        page_text = page.extract_text()
        if page_text:
            text += page_text
    return text


def split_text(text, chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks


def get_embeddings(texts):
    """Generate embeddings for a list of texts using Ollama."""
    response = ollama.embed(model=EMBEDDING_MODEL, input=texts)
    return response.embeddings


def get_embedding(text):
    """Generate an embedding for a single text."""
    response = ollama.embed(model=EMBEDDING_MODEL, input=text)
    return response.embeddings[0]


def cosine_similarity(vec_a, vec_b):
    """Calculate cosine similarity between two vectors."""
    a = np.array(vec_a)
    b = np.array(vec_b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))


def find_relevant_chunks(question, chunks, embeddings, top_k=3):
    """Find the most relevant chunks for a question."""
    question_embedding = get_embedding(question)
    scores = []
    for i, emb in enumerate(embeddings):
        score = cosine_similarity(question_embedding, emb)
        scores.append((score, i))
    scores.sort(reverse=True)
    return [chunks[i] for _, i in scores[:top_k]]

Your search pipeline is working. You can now find the most relevant passages in any document for any question. Next up, you will connect this retrieval system to the LLM so it can generate answers grounded in your actual PDF content.

Build the Complete RAG Q&A Loop

You have a working search engine that finds the most relevant chunks for any question. But search alone does not produce answers. The final piece of a RAG pipeline is feeding those retrieved chunks to an LLM as context, then asking it to generate an answer grounded in that context.

In this step, you will wire retrieval into generation and wrap it all in an interactive loop. By the end, you will have a fully working PDF Q&A tool you can point at any document.

In this step, get ready to:
  • Write the ask() function that retrieves context and generates grounded answers.
  • Write a main() function with a Q&A loop.
  • Run the complete pipeline on your PDF and verify answers are accurate.
Write the ask() function

The ask() function is the core of your RAG pipeline. It retrieves the most relevant chunks, combines them into a single context block, then constructs a prompt that instructs the LLM to answer only from the provided context.

  • In pdf_qa.py, scroll to the bottom of your file (below the find_relevant_chunks() function) and add the following code:
def ask(question, chunks, embeddings):
    """Ask a question and get an answer grounded in the document."""
    relevant_chunks = find_relevant_chunks(question, chunks, embeddings)
    context = "\n\n".join(relevant_chunks)

    prompt = (
        "You are a helpful assistant. Use ONLY the following context to "
        "answer the question. If the answer is not in the context, say "
        "'I don't have enough information to answer that.'\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\n\n"
        "Answer:"
    )

    response = ollama.chat(
        model=LLM_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.message.content

What does this code do?

This function ties retrieval to generation in three steps:

  • find_relevant_chunks() finds the top 3 chunks most similar to the question.
  • The prompt wraps those chunks in explicit instructions telling the LLM to use ONLY the context provided. If the answer is not in the chunks, the LLM says it does not have enough information instead of hallucinating.
  • ollama.chat() sends the prompt to llama3.2 and returns the generated answer.

You will not see a change yet. The next chunk (main()) uses this function to power the interactive loop.

Write the main() function

The main() function orchestrates the entire pipeline. It extracts text, creates chunks, generates embeddings, then enters a loop where the user asks questions and receives grounded answers.

  • Below the ask() function you just added, paste the following code at the bottom of pdf_qa.py:
def main():
    pdf_path = input("Enter the path to your PDF: ").strip()

    print("\nExtracting text from PDF...")
    text = extract_text(pdf_path)
    print(f"Extracted {len(text)} characters")

    print("Splitting into chunks...")
    chunks = split_text(text)
    print(f"Created {len(chunks)} chunks")

    print("Generating embeddings (this may take a moment)...")
    embeddings = get_embeddings(chunks)
    print(f"Generated {len(embeddings)} embeddings")

    print("\nReady! Ask questions about your PDF (type 'quit' to exit)\n")

    while True:
        question = input("You: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not question:
            continue

        answer = ask(question, chunks, embeddings)
        print(f"\nAssistant: {answer}\n")


if __name__ == "__main__":
    main()

What does this code do?

The main() function runs the full pipeline in sequence:

  • It asks for the PDF path, extracts all text, splits it into chunks, and generates embeddings for every chunk.
  • It enters a while True loop that reads user input, calls ask() with the question, and prints the answer.
  • Typing quit, exit, or q breaks the loop and ends the script.

The if __name__ == "__main__" guard ensures main() only runs when you execute the script directly, not when importing it as a module.

  • Save pdf_qa.py.

Seeing a syntax error on save?

  • Check that the if __name__ line is at the very bottom of the file with no indentation.
  • Verify the main() function's while True block is properly indented (4 spaces inside the function, 8 spaces inside the loop).
  • Make sure you have not accidentally placed main() inside the ask() function. There should be a blank line separating them.

Why is my script throwing a syntax error?

Run the complete pipeline

Your script is complete. Time to run it end-to-end and verify that RAG produces accurate, document-grounded answers.

  • Run your script by entering this command in Terminal:
python3 pdf_qa.py
  • When prompted, type the path to your sample PDF. If it is in the same folder, just type the filename (e.g. sample.pdf).
  • Wait for the script to process your PDF. You should see output showing the character count, chunk count, and embedding count.
  • Ask 2-3 questions about specific content in your PDF. Verify the answers reference real information from the document.
  • Ask a question about something NOT in your PDF. The LLM should respond with "I don't have enough information to answer that."
  • Type quit to exit when done.

What should I see?

After entering your PDF path, you should see something like:

Extracting text from PDF...
Extracted 24531 characters
Splitting into chunks...
Created 55 chunks
Generating embeddings (this may take a moment)...
Generated 55 embeddings

Ready! Ask questions about your PDF (type 'quit' to exit)

Your numbers will differ based on your PDF's length. The key sign of success is that answers reference actual content from your document instead of generic or made-up information.

Getting incorrect or hallucinated answers?

  • If the LLM answers with information not in your PDF, check that your prompt includes the instruction to use ONLY the context. Copy the prompt string exactly from the code above.
  • If you see a connection error, make sure Ollama is still running. Open a new Terminal tab and run ollama list to check.
  • If embedding generation is very slow, that is normal for the first run. The nomic-embed-text model needs to load into memory once, then subsequent calls are faster.

Why is my RAG pipeline returning inaccurate answers?

✔️ Awesome, I've got everything!

Great. Double check you have saved pdf_qa.py and that your Q&A loop exits cleanly when you type quit.

ⓧ I'd like to double check the full code

Here is the complete pdf_qa.py file. Compare yours against it:

import ollama
import numpy as np
from pypdf import PdfReader

# Configuration
LLM_MODEL = "llama3.2"
EMBEDDING_MODEL = "nomic-embed-text"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50


def extract_text(pdf_path):
    """Extract all text from a PDF file."""
    reader = PdfReader(pdf_path)
    text = ""
    for page in reader.pages:
        page_text = page.extract_text()
        if page_text:
            text += page_text
    return text


def split_text(text, chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks


def get_embeddings(texts):
    """Generate embeddings for a list of texts using Ollama."""
    response = ollama.embed(model=EMBEDDING_MODEL, input=texts)
    return response.embeddings


def get_embedding(text):
    """Generate an embedding for a single text."""
    response = ollama.embed(model=EMBEDDING_MODEL, input=text)
    return response.embeddings[0]


def cosine_similarity(vec_a, vec_b):
    """Calculate cosine similarity between two vectors."""
    a = np.array(vec_a)
    b = np.array(vec_b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))


def find_relevant_chunks(question, chunks, embeddings, top_k=3):
    """Find the most relevant chunks for a question."""
    question_embedding = get_embedding(question)
    scores = []
    for i, emb in enumerate(embeddings):
        score = cosine_similarity(question_embedding, emb)
        scores.append((score, i))
    scores.sort(reverse=True)
    return [chunks[i] for _, i in scores[:top_k]]


def ask(question, chunks, embeddings):
    """Ask a question and get an answer grounded in the document."""
    relevant_chunks = find_relevant_chunks(question, chunks, embeddings)
    context = "\n\n".join(relevant_chunks)

    prompt = (
        "You are a helpful assistant. Use ONLY the following context to "
        "answer the question. If the answer is not in the context, say "
        "'I don't have enough information to answer that.'\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\n\n"
        "Answer:"
    )

    response = ollama.chat(
        model=LLM_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.message.content


def main():
    pdf_path = input("Enter the path to your PDF: ").strip()

    print("\nExtracting text from PDF...")
    text = extract_text(pdf_path)
    print(f"Extracted {len(text)} characters")

    print("Splitting into chunks...")
    chunks = split_text(text)
    print(f"Created {len(chunks)} chunks")

    print("Generating embeddings (this may take a moment)...")
    embeddings = get_embeddings(chunks)
    print(f"Generated {len(embeddings)} embeddings")

    print("\nReady! Ask questions about your PDF (type 'quit' to exit)\n")

    while True:
        question = input("You: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        if not question:
            continue

        answer = ask(question, chunks, embeddings)
        print(f"\nAssistant: {answer}\n")


if __name__ == "__main__":
    main()

Your PDF Q&A tool is fully functional. You have built a complete RAG pipeline from scratch. Next up, you can extend it with streaming output and source transparency in the Secret Mission.

Secret mission

Add Streaming and Source Transparency

Your Q&A tool works, but it feels like talking to a wall. You ask a question, wait in silence, then get a block of text dumped at once. Real AI tools stream responses word-by-word and show their sources. Can you upgrade your pipeline to do both?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama application and models (llama3.2 and nomic-embed-text).
  • Python virtual environment (venv folder inside pdf-qa).
  • Project folder (pdf-qa on your Desktop).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your PDF Q&A tool or want to try it on other documents.

  • Ollama will continue running as a background service on your Mac. Models stay cached on disk for fast loading.
  • Your pdf-qa folder and virtual environment remain on your Desktop.

Pause - I'll come back to this later

Free up memory by quitting Ollama while keeping your files intact. You can restart Ollama any time and pick up where you left off.

  • Quit Ollama by clicking the Ollama icon in your menu bar and selecting Quit Ollama.
  • Deactivate your virtual environment by running:
deactivate
  • Your project folder, virtual environment, and downloaded models all stay on disk. To resume later, reopen Ollama, activate your venv, and run python3 pdf_qa.py.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to fully clean up.

  • Remove the downloaded models by running these commands in Terminal:
ollama rm llama3.2
ollama rm nomic-embed-text
  • Delete the project folder from your Desktop by running:
rm -rf ~/Desktop/pdf-qa
  • To fully uninstall Ollama from your Mac, quit the Ollama app first (click the menu bar icon and select Quit Ollama), then run these commands:
sudo rm -rf /Applications/Ollama.app
sudo rm /usr/local/bin/ollama
rm -rf ~/.ollama

What do these commands remove?

  • ollama rm removes individual models from Ollama's local storage.
  • rm -rf ~/Desktop/pdf-qa deletes your project folder, virtual environment, and the script.
  • The final three commands remove the Ollama application, CLI binary, and all cached model data from your system.

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully local PDF Q&A tool powered by a RAG pipeline you wrote from scratch.

Every piece of the system runs on your machine. No API keys, no cloud services, no data leaving your laptop.

You've learned how to:

  • Build a complete RAG pipeline from scratch: PDF extraction, text chunking, embedding generation, cosine similarity search, and grounded LLM generation.
  • Run both a chat model (llama3.2) and an embedding model (nomic-embed-text) locally via Ollama.
  • Understand why RAG is needed: LLMs cannot answer questions about documents they have not seen, so you retrieve relevant context and ground the response in real content.
  • Secret Mission: Added streaming output and source citations for transparency into how RAG selects context.

Ready to quiz yourself?