Build a Local PDF Q&A System

Build a private RAG system that answers questions about your PDFs locally.

Introduction

30 Second Summary

You have folders full of PDFs sitting on your computer or network drive. Manuals, research papers, notes. When you need a specific detail, you end up scrolling through pages hoping to spot the right paragraph.

In this project, you will build a command-line tool that turns any folder of PDFs into a private, searchable knowledge base powered by a local AI model running entirely on your machine.

What You'll Build

You run a single Python script, point it at a folder of PDFs, and ask questions in plain English. The local LLM reads your documents and gives you grounded answers pulled directly from the actual content.

By the end of this project, you'll have:

A local RAG pipeline that reads your PDFs, creates vector embeddings, and builds a searchable knowledge base. No cloud APIs, no internet required after setup.
A persistent vector index that saves to disk so your PDFs only need processing once. Re-launching the tool is instant.
An interactive chat loop where you ask follow-up questions and explore your documents conversationally, right from the terminal.
Secret Mission: Add source citations that show exactly which PDF and page number each answer came from.

Are there any prerequisites?

No prior experience with AI or Python is required. This guide walks you through installing every tool from scratch, including Ollama for running local AI models and LlamaIndex for building the RAG pipeline.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. This is the moment to commit to the goal so the hands-on work ahead has clear direction.

Install Ollama and Pull Models

Before you can build a document Q&A system, you need a way to run AI models on your machine. Ollama is a local LLM runtime that downloads and serves AI models with a single command.

In this step, you'll install Ollama on Ubuntu, then download two models: a chat model for generating answers and an embedding model for understanding document content.

In this step, get ready to:

Install Ollama and verify it's running.
Pull the llama3.1:8b chat model and nomic-embed-text embedding model.
Smoke test the LLM to confirm it generates responses.

Install Ollama on Ubuntu

Ollama's official install script handles everything: downloading the binary, creating a system service, and starting it in the background.

Open a terminal in VS Code by pressing Ctrl+`.
Install Ollama by running this command:

curl -fsSL https://ollama.com/install.sh | sh

What does this command do?

This downloads Ollama's official install script and runs it. The script installs the Ollama binary, creates a dedicated system user, sets up a background service, and starts it automatically.

The script will prompt you for your sudo password so it can install system-wide files. Enter your password when you see the prompt.

The install takes a minute. When it finishes, you'll see a message ending with something like "Install complete."

Verify Ollama is installed by running:

ollama --version

✔️ I see a version number

You should see output like ollama version is X.Y.Z. Ollama is installed and running. Move on to the next substep.

ⓧ Command not found

The install script may not have completed successfully. Try these fixes:

Close your terminal and open a new one (press Ctrl+` twice), then run ollama --version again.
If it still fails, re-run the install script to retry the download.

If you're still stuck, help me troubleshoot my Ollama installation on Ubuntu.

Pull the chat and embedding models

You need two models: one that generates human-like text answers, and one that converts text into numerical vectors for searching. Both run locally through Ollama.

Download the chat model (approximately 4.7 GB) by running:

ollama pull llama3.1:8b

What is this model?

Llama 3.1 8B is Meta's open-source language model with 8 billion parameters. It generates text responses and will power the Q&A answers in your tool. The download is approximately 4.7 GB.

This download may take a few minutes depending on your internet speed. You'll see a progress bar as it downloads.

Once that finishes, download the embedding model (approximately 300 MB) by running:

ollama pull nomic-embed-text

Why a separate embedding model?

The chat model generates answers, but it cannot search through your documents efficiently. The embedding model (nomic-embed-text) converts text chunks into vector embeddings so your system can find the most relevant passages to answer a question.

Confirm both models are available by running:

ollama list

You should see both llama3.1:8b and nomic-embed-text listed in the output.

Model not showing up?

Make sure the pull command finished without errors. If you see a network timeout, re-run the ollama pull command. Ollama resumes interrupted downloads.
If ollama list shows nothing at all, the Ollama service may have stopped. Restart it with sudo systemctl restart ollama and try pulling again.

Still stuck? Help me troubleshoot why my Ollama models aren't showing up.

Run a quick smoke test

Both models are downloaded. Before moving on, confirm the chat model can actually generate a response on your hardware.

Before you run this next command, what do you think will happen? Will you get an instant response, or will it take a moment? Why?

Test the chat model by running:

ollama run llama3.1:8b "Say hello"

After 10-30 seconds (the model runs on CPU which takes a moment), you'll see a friendly text response from the LLM. The exact wording varies each time.

Why does it take so long?

Your system uses an AMD Radeon Vega integrated GPU, which does not support GPU acceleration for Ollama. The model runs entirely on your CPU. With 32 GB RAM, it runs comfortably but takes 10-30 seconds per response. This is perfectly normal for CPU inference.

No response after a minute?

Check that the Ollama service is still running with sudo systemctl status ollama. It should show "active (running)".
If it shows "inactive", restart it with sudo systemctl restart ollama and try the command again.

Still not working? Help me get Ollama responding on my Ubuntu system.

Ollama is installed, both models are loaded, and your local LLM is generating responses. Next up, you'll write your first Python script to talk to the LLM programmatically.

Set Up Python Environment and Get Your First LLM Response

You have Ollama running and two models ready to go. Now you need an isolated Python environment where your project's dependencies live without affecting anything else on your system.

In this step, you'll create a project folder, install the LlamaIndex packages that connect Python to Ollama, and write a short script that proves your local LLM can respond to Python code.

In this step, get ready to:

Create a project folder with a Python virtual environment and install LlamaIndex packages.
Write a Python script that talks to your local Llama 3.1 model.
Run the script and see a real LLM response in your terminal.

Create your project folder and Python environment

A virtual environment is an isolated copy of Python where you install packages without affecting the rest of your system. Every Python project should have its own.

Move to your Desktop and create the project folder by running these commands in your terminal:

cd ~/Desktop
mkdir pdf-qa
cd pdf-qa

Confirm you're in the new folder by running:

pwd

You should see /home/your-username/Desktop/pdf-qa printed.

Create and activate a virtual environment by running these commands:

python3 -m venv venv
source venv/bin/activate

Your terminal prompt now shows (venv) at the beginning. This confirms all pip installs will go into this isolated environment.

Don't see (venv) in your prompt?

Make sure you ran source venv/bin/activate (not just python3 -m venv venv). The first command creates the environment. The second activates it.

If you see an error about ensurepip not being available, install the venv module with sudo apt install python3-venv and try again.

Help me activate my Python virtual environment.

Install the LlamaIndex packages by running:

pip install llama-index-core llama-index-llms-ollama llama-index-embeddings-ollama llama-index-readers-file

This takes a minute or two to download. When it finishes, you'll see a Successfully installed message listing all the packages.

What are these packages?

llama-index-core is the main LlamaIndex library for building RAG pipelines.
llama-index-llms-ollama lets LlamaIndex talk to your local Ollama models.
llama-index-embeddings-ollama provides the embedding connection for turning text into vectors.
llama-index-readers-file adds PDF reading support so LlamaIndex can extract text from your documents.

Seeing a connection error or timeout?

Make sure you have an active internet connection. The packages are downloaded from PyPI.

If you see a permissions error, confirm your virtual environment is active ((venv) in your prompt). Never use sudo pip install inside a virtual environment.

Help me install LlamaIndex packages in my virtual environment.

Write your first LLM script

With your packages installed, you can now write a Python script that sends a prompt to your local Llama 3.1 model and prints the response.

Open the pdf-qa folder in VS Code by running this command in your terminal:

code .

What does code . do?

The code command launches VS Code from the terminal. The dot (.) means "open the current folder." This opens your pdf-qa project in the VS Code Explorer sidebar.

In VS Code, hover over the **Explorer** sidebar on the left and click the **New File...** button.
Type main.py and press **Enter**.
Paste the following code into main.py:

from llama_index.llms.ollama import Ollama

# Create an LLM instance pointing at your local Ollama model
llm = Ollama(model="llama3.1:8b", request_timeout=120.0)

# Send a prompt and capture the response
response = llm.complete("Say something interesting about AI.")
print(response)

What does this code do?

from llama_index.llms.ollama import Ollama imports the class that connects Python to your local Ollama server.
Ollama(model="llama3.1:8b", request_timeout=120.0) creates an LLM instance. The timeout is set to 120 seconds because CPU inference can be slow on the first run.
llm.complete(...) sends your prompt to the model and returns the generated text.

Save main.py by pressing **Ctrl+S**.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

Here is the complete main.py file. Confirm yours matches:

from llama_index.llms.ollama import Ollama

llm = Ollama(model="llama3.1:8b", request_timeout=120.0)
response = llm.complete("Say something interesting about AI.")
print(response)

Run the script and see your LLM respond

This is the moment you see your local LLM generate a response from a Python script. No cloud. No API key. Just your machine.

Before you run this, think about what you expect to see. Will the response be instant, or will it take a moment? Why?
Switch back to your terminal (make sure you still see (venv) in the prompt) and run:

python main.py

After 10-30 seconds, you'll see a text response from Llama 3.1 printed in your terminal. The first run is slower because Ollama loads the model into memory.

Why does it take so long?

Your machine is running an 8-billion-parameter neural network entirely on CPU. With 32 GB of RAM, the model fits comfortably in memory, but generating each token requires billions of calculations. Subsequent queries are faster because the model stays loaded.

Seeing a connection error?

If you see a message about failing to connect to localhost:11434, your Ollama service may have stopped. Restart it by running sudo systemctl start ollama in a separate terminal.

If you see a timeout error, increase the request_timeout value in your script (try 240.0). First-run model loading can be slower on some machines.

If you see ModuleNotFoundError, make sure your virtual environment is active ((venv) in your prompt). Run source venv/bin/activate first.

Help me get my first LlamaIndex script running.

Your local LLM is responding to Python code. Next up, you'll feed it your actual PDF documents and ask questions about their content.

Load PDFs and Ask Your First Question

In the previous step, you proved that your local LLM can generate text from its training data. That is impressive, but it cannot answer questions about your documents.

In this step, you will feed your PDFs to the LLM through a Retrieval-Augmented Generation (RAG) pipeline. Instead of answering from memory, the LLM will search your documents for relevant passages and ground its answer in that content.

In this step, get ready to:

Add PDF files to a data folder for the pipeline to read.
Configure LlamaIndex with your Ollama models and build a vector index from your documents.
Ask a question and get an answer grounded in your PDF content.

Add PDF files to your project

The RAG pipeline needs source documents to search through. You will create a data/ folder inside your project and drop in a few PDFs.

Create the data folder inside your pdf-qa project folder by running this command:

mkdir data

Copy 2-3 PDF files into the data/ folder. You can use any PDFs you have on hand. Manuals, reports, research papers, or e-books all work.
Confirm your PDFs are in place by running:

ls data/

You should see your PDF filenames listed in the output.

Use small PDFs to start

For your first run, pick PDFs that are under 50 pages. Larger documents work fine but take longer to process on CPU. You can always swap in bigger files later.

What if my PDFs are on a NAS?

If your documents live on a mounted network drive, you can point the script at that mount path instead of ./data. You will change the path in the code in the next substep.

Configure the RAG pipeline and build the vector index

Your current main.py calls the LLM directly with a hardcoded prompt. You need to replace that with a full RAG pipeline that loads your PDFs, creates vector embeddings for each chunk, and stores them in a searchable index.

Open main.py in VS Code and replace its entire contents with the following code:

from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding

# --- Configuration ---
DATA_DIR = "./data"
OLLAMA_MODEL = "llama3.1:8b"
EMBED_MODEL = "nomic-embed-text"

# --- Set up LLM and embedding model ---
Settings.llm = Ollama(model=OLLAMA_MODEL, request_timeout=120.0)
Settings.embed_model = OllamaEmbedding(model_name=EMBED_MODEL)

documents = SimpleDirectoryReader(DATA_DIR).load_data()
print(f"Loaded {len(documents)} document chunks. Building index...")
index = VectorStoreIndex.from_documents(documents)

What does this code do?

DATA_DIR tells the reader where your PDFs live. Change this to your NAS mount path if needed.
Settings.llm and Settings.embed_model configure LlamaIndex to use your local Ollama models for both chat and embedding.
SimpleDirectoryReader reads every PDF in your folder and splits them into text chunks with page-level metadata attached.
VectorStoreIndex.from_documents converts each chunk into a vector embedding (a numeric fingerprint capturing meaning) and stores it in a searchable in-memory index.

Save main.py.
Run the script to confirm your PDFs load and the index builds:

python main.py

You should see output like:

Loaded 42 document chunks. Building index...

The exact number depends on how many pages your PDFs have. Building the index takes 10-30 seconds on CPU as each chunk is embedded via nomic-embed-text.

Seeing an error about no files found?

Make sure your data/ folder contains at least one .pdf file. Run ls data/ to double-check.
If you pointed DATA_DIR at a NAS path, confirm the mount is accessible from your terminal.

Why are my PDFs not loading?

Ask a question about your PDF content

The index is built. Now you can create a query engine that finds the most relevant chunks from your documents and sends them to the LLM along with your question.

Add the following code to the bottom of main.py, below the line that says index = VectorStoreIndex.from_documents(documents):

query_engine = index.as_query_engine(similarity_top_k=3)

question = "What is this document about?"
response = query_engine.query(question)
print(f"Answer: {response}")

What does this code do?

as_query_engine(similarity_top_k=3) creates an engine that retrieves the 3 most semantically similar chunks to your question before sending them to the LLM.
The LLM reads those chunks as context and generates an answer grounded in your actual document content, not its training data.

Save main.py.

Before you run this, what do you expect the answer to contain? Think about what your PDFs are about.

Run the script:

python main.py

You should see the loaded chunks message, a pause while the LLM thinks, then an answer that references information actually contained in your PDFs.

Answer seems unrelated to your PDFs?

Try asking a more specific question that matches content you know is in one of your documents. Replace the question string with something like "What does section 2 discuss?" and re-run.
If the script hangs for over 2 minutes, Ollama may not be running. Open a new terminal and run ollama list to check.

Why is the LLM ignoring my document content?

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py and confirmed the answer references your PDF content.

ⓧ I'd like to double check the full code

Here is the complete main.py at the end of this step:

from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding

# --- Configuration ---
DATA_DIR = "./data"
OLLAMA_MODEL = "llama3.1:8b"
EMBED_MODEL = "nomic-embed-text"

# --- Set up LLM and embedding model ---
Settings.llm = Ollama(model=OLLAMA_MODEL, request_timeout=120.0)
Settings.embed_model = OllamaEmbedding(model_name=EMBED_MODEL)

documents = SimpleDirectoryReader(DATA_DIR).load_data()
print(f"Loaded {len(documents)} document chunks. Building index...")
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=3)

question = "What is this document about?"
response = query_engine.query(question)
print(f"Answer: {response}")

Your local RAG pipeline is working. The LLM now grounds its answers in your actual documents instead of hallucinating. Next up, you will add persistence so your PDFs only need processing once.

Add Persistence So You Only Process PDFs Once

Your PDF Q&A tool works, but there is a problem hiding in plain sight. Every time you run the script, it re-reads every PDF, splits them into chunks, and computes vector embeddings from scratch.

With a handful of test PDFs that takes 30 seconds or so. Point it at a large folder on your NAS and you could be waiting many minutes each time. In this step, you will add persistence so the vector index saves to disk after the first run and loads instantly on every run after that.

In this step, get ready to:

Refactor your flat script into reusable functions that build or load the index.
Save the vector index to a local ./storage directory using StorageContext.
Add a startup check that loads from disk when the index already exists.

Save your vector index to disk

Right now your script runs everything at the top level. You will restructure it into functions and add a persistence layer that writes the index to disk after building it.

In VS Code, open main.py and select all the existing code (Ctrl+A on Linux).
Delete everything and replace it with the following imports, configuration, and build_index() function:

from pathlib import Path

from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex, StorageContext, load_index_from_storage
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding

# --- Configuration ---
DATA_DIR = "./data"
STORAGE_DIR = "./storage"
OLLAMA_MODEL = "llama3.1:8b"
EMBED_MODEL = "nomic-embed-text"

# --- Set up LLM and embedding model ---
Settings.llm = Ollama(model=OLLAMA_MODEL, request_timeout=120.0)
Settings.embed_model = OllamaEmbedding(model_name=EMBED_MODEL)

def build_index():
    """Load PDFs and build a new vector index."""
    print(f"Loading documents from: {DATA_DIR}")
    documents = SimpleDirectoryReader(DATA_DIR).load_data()
    print(f"Loaded {len(documents)} document chunks. Building index...")
    index = VectorStoreIndex.from_documents(documents)
    index.storage_context.persist(persist_dir=STORAGE_DIR)
    print(f"Index saved to: {STORAGE_DIR}")
    return index

What changed from your previous code?

Two new imports: Path from Python's standard library (for checking if a folder exists), and StorageContext plus load_index_from_storage from LlamaIndex (for saving and loading the index).
A new STORAGE_DIR constant tells the script where to save the index on disk.
The flat document-loading code now lives inside a build_index() function. The key new line is index.storage_context.persist(persist_dir=STORAGE_DIR) which writes the entire vector index to the ./storage folder.

Add the load function and main entry point

Now you need a function to reload the index from disk, and a main() function that decides which path to take at startup.

Below your build_index() function, add the load_index() function, the main() function, and the entry point:

def load_index():
    """Load an existing index from disk."""
    print(f"Loading existing index from: {STORAGE_DIR}")
    storage_context = StorageContext.from_defaults(persist_dir=STORAGE_DIR)
    index = load_index_from_storage(storage_context)
    print("Index loaded successfully.")
    return index

def main():
    if Path(STORAGE_DIR).exists():
        index = load_index()
    else:
        index = build_index()

    query_engine = index.as_query_engine(similarity_top_k=3)
    question = "What is this document about?"
    response = query_engine.query(question)
    print(f"Answer: {response}")

if __name__ == "__main__":
    main()

What does this code do?

load_index() reconstructs the vector index from the files saved in ./storage without re-reading or re-embedding any PDFs.
main() checks if ./storage already exists. If it does, the index loads from disk. If not, it builds a fresh index from your PDFs and persists it.
if __name__ == "__main__" is a standard Python pattern that runs main() only when you execute the script directly.

Save main.py.
Run the script by executing this command in your terminal:

python main.py

This is the first run, so it will process your PDFs from scratch. You should see output like:

Loading documents from: ./data
Loaded 12 document chunks. Building index...
Index saved to: ./storage
Answer: ...

Confirm the ./storage directory was created by running:

ls storage

You should see several JSON files inside (these are your persisted vector index).

Not seeing the storage folder?

Make sure you saved main.py before running it. Unsaved changes mean the old flat script ran instead.
Check that your ./data folder still contains PDF files. If it is empty, the script has nothing to index.
If you see a connection error, make sure Ollama is still running (ollama list should show your models).

Help me debug why my storage folder was not created after running main.py.

Test both paths: build and load

Your first run proved the build-and-save path works. Now test the load-from-disk path to confirm your persistence actually saves time.

Before you run the script again, predict: will it re-process your PDFs, or load from disk?

Run the script again:

python main.py

This time you should see it load from disk almost instantly:

Loading existing index from: ./storage
Index loaded successfully.
Answer: ...

The script skipped the slow PDF reading and embedding step entirely. It loaded your pre-built index in under 2 seconds and still produced the same answer.

Why is this so much faster?

The expensive work in a RAG pipeline is computing vector embeddings for every chunk of text. That requires sending each chunk through the embedding model one by one.

Persistence saves the finished embeddings to disk as JSON files. On subsequent runs, the script just reads those files back into memory. No model inference needed.

Still seeing 'Building index' on the second run?

Check that STORAGE_DIR is set to "./storage" (with the dot-slash prefix). A mismatch means Path(STORAGE_DIR).exists() returns False.
Confirm you are running python main.py from inside your project folder (the one containing the storage directory).

Help me figure out why my script is rebuilding the index instead of loading from storage.

✔️ Awesome, I've got everything!

Great. Make sure you have saved main.py and that running it twice in a row shows the fast load path on the second run.

ⓧ I'd like to double check the full code

Your main.py should contain these sections in order from top to bottom:

Imports: Path from pathlib, then Settings, SimpleDirectoryReader, VectorStoreIndex, StorageContext, load_index_from_storage from llama_index.core, plus Ollama and OllamaEmbedding.
Configuration block with DATA_DIR, STORAGE_DIR, OLLAMA_MODEL, and EMBED_MODEL.
Settings.llm and Settings.embed_model assignments.
build_index() function that loads PDFs, builds the index, persists it to STORAGE_DIR, and returns the index.
load_index() function that creates a StorageContext from STORAGE_DIR and returns the loaded index.
main() function that checks Path(STORAGE_DIR).exists() and calls either load_index() or build_index(), then queries the index.
if __name__ == "__main__": main() at the very bottom.

Your knowledge base now persists between sessions. Next up, you will wrap the query logic in an interactive chat loop so you can ask follow-up questions without restarting the script each time.

Build an Interactive Chat Loop

Your PDF knowledge base is persistent and loads instantly. But every time you want to ask a question, you have to edit main.py, change the hardcoded string, and re-run the script.

In this step, you will replace that single-shot query with an interactive chat loop. The loop reads questions from your terminal, sends them to the RAG pipeline, and prints grounded answers until you decide to quit.

In this step, get ready to:

Replace the hardcoded query with a while True loop that reads user input.
Add a quit command and a startup message showing which folder is indexed.
Add error handling so the program does not crash on empty input or connection issues.

Replace the single query with an interactive loop

Right now your main() function runs one hardcoded question and exits. You will replace everything inside main() with code that loads the index, prints a startup banner, and waits for your input in a loop.

In VS Code, open main.py.
First, add a helpful comment to your DATA_DIR line so you remember you can change it later. Find this line:

DATA_DIR = "./data"

Update it to include the comment:

DATA_DIR = "./data"  # Change this to your NAS mount path if needed

Now find your main() function and delete everything inside it. Replace the function body with the following setup code:

def main():
    # Load or build the index
    if Path(STORAGE_DIR).exists():
        index = load_index()
    else:
        index = build_index()

    # Create query engine
    query_engine = index.as_query_engine(similarity_top_k=3)

    # Interactive loop
    print("\n--- PDF Q&A Ready ---")
    print(f"Documents indexed from: {DATA_DIR}")
    print("Type your question and press Enter. Type 'quit' to exit.\n")

What does this code do?

The persistence logic stays the same. If ./storage exists, load from disk. Otherwise, build a fresh index.
The three print() lines are your startup banner. They tell you which folder was indexed and how to exit.
similarity_top_k=3 means each query retrieves the 3 most relevant document chunks for context.

Add the interactive loop with error handling

The loop needs to handle several edge cases gracefully. It should catch keyboard interrupts, skip empty input, and not crash if Ollama is temporarily unreachable.

Directly below the print statements you just added (still inside main()), add the while loop:

    while True:
        try:
            question = input("You: ").strip()
        except (KeyboardInterrupt, EOFError):
            print("\nGoodbye!")
            break

        if not question:
            continue
        if question.lower() in ("quit", "exit"):
            print("Goodbye!")
            break

        print("Thinking...\n")
        try:
            response = query_engine.query(question)
            print(f"Answer: {response}\n")
        except Exception as e:
            print(f"Error: {e}\n")

What does this loop do?

input("You: ").strip() waits for the user to type a question and strips whitespace.
The outer try/except catches KeyboardInterrupt (Ctrl+C) and EOFError so the program exits cleanly instead of printing a traceback.
if not question: continue skips empty input so the user can press Enter without triggering a query.
The quit and exit keywords give the user a typed way to leave the loop.
The inner try/except Exception catches any query errors (like Ollama being stopped) and prints the message instead of crashing.

Save main.py.

Seeing indentation errors?

The while True: block must be indented one level inside main(), at the same level as the print statements above it. Make sure you did not accidentally nest it inside the if/else block.

Help me fix my indentation error

Test the interactive chat

Before you run this, what do you expect to see? A single answer and exit, or a prompt that waits for you to type?

Run your script:

python main.py

You should see the startup banner followed by a You: prompt waiting for your input. The index loads from disk in under 2 seconds since it was persisted in the previous step.

Type a question about your PDFs and press Enter.
Wait for the answer (10-30 seconds on CPU).
Ask a second question to confirm the loop continues.
Press Enter with no text to confirm it does not crash.
Type quit and press Enter to exit.

You should see Goodbye! printed and the program exits cleanly back to your terminal prompt.

Not getting any response?

Make sure Ollama is still running. In a separate terminal, run ollama list to confirm the models are available.
If you stopped Ollama earlier, restart it with sudo systemctl start ollama.
Check that your ./storage directory still exists from the previous step. If you deleted it, the script will rebuild the index (which takes longer but still works).

Help me debug why I'm not getting responses

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py with all your changes.

ⓧ I'd like to double check the full code

Your main.py is over 30 lines, so confirm these sections are present in order:

Imports (Path, LlamaIndex core, Ollama, OllamaEmbedding).
Configuration block (DATA_DIR, STORAGE_DIR, OLLAMA_MODEL, EMBED_MODEL) with the NAS comment on DATA_DIR.
Settings.llm and Settings.embed_model assignments.
build_index() function (load, index, persist).
load_index() function (load from storage).
main() function with persistence check, query engine creation, startup prints, and the while True loop.
if __name__ == "__main__": main() at the bottom.

Your local PDF Q&A tool is fully interactive. You can point it at any folder, ask as many questions as you want, and exit cleanly. Next up, the Secret Mission adds source citations so you can verify exactly where each answer comes from.

Secret mission

Add Source Citations

Your Q&A tool gives answers, but how do you know which PDF those answers came from? Add source citations that show exactly which file and page number each answer references. This is the difference between a toy demo and a tool you actually trust.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

Ollama installation and downloaded models (approximately 5 GB disk space for llama3.1:8b and nomic-embed-text).
Python virtual environment and LlamaIndex packages inside your project folder.
The ./storage directory containing your persisted vector index.
The data/ folder containing your sample PDFs.

Keep everything running

No action needed. Choose this if you plan to keep using your PDF Q&A tool or want to add more documents later.

Ollama runs as a background service and uses minimal resources when idle.
Your vector index persists in the ./storage directory and loads instantly on next run.
Your virtual environment stays on disk but consumes no memory unless activated.

Pause - I'll come back to this later

Stop the Ollama background service to free up any memory it uses while idle. Your files and index stay on disk.

Stop the Ollama service by running:

sudo systemctl stop ollama

When you are ready to come back, restart it by running:

sudo systemctl start ollama

Your project folder, virtual environment, and vector index all remain on disk ready to use.

Delete - I don't want to use this again

Remove all project resources and Ollama from your system entirely.

Delete your project folder by running:

cd ~/Desktop
rm -rf pdf-qa

Stop and disable the Ollama service, then remove its systemd service file by running:

sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service

Remove the Ollama binary by running:

sudo rm $(which ollama)

Remove the downloaded models and the Ollama service user and group by running:

sudo rm -r /usr/share/ollama
sudo userdel ollama
sudo groupdel ollama

That's a Wrap!

Nice work! You've built a fully local PDF Q&A system that reads your documents, builds a searchable knowledge base, and answers questions using a local LLM.

You've learned how to:

Installed Ollama and ran a local 8-billion-parameter LLM entirely on your machine with no cloud APIs or internet required.
Built a RAG pipeline that loads PDFs, chunks them, creates vector embeddings, and stores them for semantic search.
Created a persistent vector index so your PDFs only need processing once and subsequent runs load instantly.
Built an interactive chat loop for conversational document exploration you can point at any folder.
(Secret Mission) Added source citations showing exactly which PDF and page number each answer comes from.

Ready to quiz yourself?