Private Document Q&A Tool with Ollama

Build an offline RAG pipeline that answers questions about your documents.

Introduction

30 Second Summary

Every time you paste a company report, medical record, or private note into ChatGPT, that data leaves your machine and lands on someone else's server. For anyone working with sensitive documents, that is a dealbreaker.

In this project, you will build a private document Q&A tool that runs entirely on your own computer. You will drop files into a folder, ask questions in your terminal, and get accurate, cited answers from a local AI that never sends a single byte to the internet.

What You'll Build

Picture this: you drop a handful of PDFs and text files into a folder, run one command, then start asking questions about them in a chat loop. The AI pulls the exact passage that answers your question, tells you which file it came from, and does it all without touching the internet.

By the end of this project, you'll have:

  • A fully offline AI chatbot running on your own machine. Open your terminal, type a question, and get a response from a local LLM powered by Ollama.
  • A RAG pipeline that reads your documents and answers questions grounded in their contents. Every answer cites the source file it came from, so you can verify it yourself.
  • A reusable, persistent search index saved to disk. Restart your computer, come back days later, and query your documents instantly without re-processing them.
  • Secret Mission: extend the tool to answer questions about codebases by loading .py and .js source files, turning it into a private coding assistant.

Are there any prerequisites?

You need Python 3.10+ installed on a Windows 10+ machine with at least 8 GB of RAM. Basic Python knowledge (variables, functions, loops) is helpful but not required. Everything runs on your CPU, so no GPU is needed.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

You're going to build a fully offline document Q&A tool that runs on your own machine. No data leaves your computer, no API keys needed, no cloud bills. Take a moment to think about why that matters for you and the kind of documents you'd want to query privately.

Set Up Ollama and Pull Your Models

Every AI tool you use online sends your data to someone else's server. To keep your documents private, you need an AI engine that runs entirely on your own machine.

That engine is Ollama. It runs large language models directly on your hardware with zero cloud dependency. In this step, you'll install Ollama, download two AI models, and verify everything works before writing any code.

In this step, get ready to:
  • Install Ollama on your Windows machine.
  • Download the Qwen 3.5 chat model and the all-minilm embedding model.
  • Test the chat model by having a live conversation in your terminal.
Install Ollama on Windows

Ollama is a lightweight tool that runs AI models locally. You install it once, and it runs quietly in the background, ready for any application on your machine to connect to it.

  • Open your web browser and go to the Ollama download page.
  • Click Download for Windows to download OllamaSetup.exe.
  • Double-click OllamaSetup.exe to run the installer.
  • Follow the installer prompts to complete the installation.

What does the installer do?

The installer places Ollama in your home directory and adds it to your PATH. No administrator rights are needed. After installation, Ollama runs as a background service and serves a local API on http://localhost:11434.

You will also see an Ollama icon appear in your system tray (bottom-right of your taskbar). This means Ollama is running in the background and ready to accept commands.

Now you need to open PowerShell to start using the Ollama command line.

  • Press the Windows key to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Verify Ollama is installed by running this command:
ollama -v

✔️ I see a version number

Ollama is installed and ready. Keep this PowerShell window open for the next substep.

ⓧ Command not recognized

PowerShell does not recognize the ollama command yet. This usually means the PATH update from the installer has not taken effect in your current terminal session.

  • Close this PowerShell window completely.
  • Open a brand-new PowerShell window using the Windows key search method above.
  • Run ollama -v again in the new window.

If it still does not work, try restarting your computer and opening a fresh PowerShell window. The installer needs a new terminal session to register the ollama command.

Pull the chat and embedding models

Your project needs two AI models. The first is a chat model that generates human-like responses. The second is an embedding model that converts text into numerical vectors for search.

  • Download the Qwen 3.5 chat model by running this command:
ollama pull qwen3.5:4b

What is this model?

Qwen 3.5 is a 4-billion parameter chat model. At ~3.4 GB, it fits comfortably in 8 GB of RAM running on CPU. This is the model that will read your documents and answer your questions.

The download can take a few minutes depending on your internet speed. You will see a progress bar in the terminal.

Once the download finishes, you should see a success message confirming the model was pulled.

  • Download the all-minilm embedding model by running:
ollama pull all-minilm

What are embeddings?

An embedding model converts text into arrays of numbers called vectors. Words with similar meanings produce similar vectors. This is how the system will search your documents for relevant information later.

At just 23 million parameters, all-minilm is tiny and downloads in seconds. It leaves plenty of RAM headroom for the chat model.

  • Confirm both models are downloaded by running:
ollama ls

You should see both qwen3.5:4b and all-minilm listed in the output.

Don't see both models listed?

  • If a model is missing, re-run the ollama pull command for that model. A network interruption during download can cause an incomplete pull.
  • Make sure you typed the model name exactly as shown, including the colon and size tag for qwen3.5:4b.

Help me troubleshoot my Ollama model download

Why Ollama over other local AI tools?

Ollama runs as a background service and exposes a local API that Python libraries can connect to directly. Alternatives like LM Studio are GUI-focused, while GPT4All has a narrower model library.

For building a Python pipeline, Ollama is the purpose-built choice. It also powers tools like OpenClaw and Claude Code.

Test the chat model

Both models are downloaded. Time to prove the chat model actually works on your machine. You are about to have a conversation with an AI running entirely on your own hardware, with no internet required.

Before you run this command, think about what you expect to happen. Will the model respond instantly? Will it be slow? Will it work at all on CPU-only hardware?

  • Start an interactive chat session with the Qwen 3.5 model by running:
ollama run qwen3.5:4b

You should see a >>> prompt appear, waiting for your input.

  • Type Hello and press Enter.

The model generates a response token by token directly in your terminal. Every word is produced locally on your CPU. No data leaves your machine.

Why is the response slow?

On CPU-only hardware with 8 GB of RAM, the model generates text at roughly 20-30 tokens per second. That is noticeably slower than cloud-hosted AI, but the tradeoff is complete privacy. Nothing is sent to the internet.

  • Type /bye and press Enter to exit the chat session.

You just had a conversation with an AI that ran entirely on your own computer. That confirms Ollama is working and the Qwen 3.5 model responds on your hardware.

Not seeing a response?

  • Make sure Ollama is still running in the background. Check for the Ollama icon in your system tray (bottom-right of the taskbar).
  • If the model seems stuck, give it up to 30 seconds for the first response. Loading the model into memory takes time on the first run.
  • If you see an error about insufficient memory, close other applications to free up RAM. The model needs roughly 4 GB of free memory.

Help me troubleshoot my Ollama chat session

Ollama is running and your AI models are ready. Next up, you'll connect to this local AI from Python and build your first offline chatbot in under 10 lines of code.

Chat with Your Local AI from Python

Ollama is running and your models are downloaded. Now it's time to talk to your local AI from Python code instead of the Ollama terminal.

In this step, you'll build a working offline chatbot in under 10 lines of code. This proves the Python-to-Ollama connection works before you add document intelligence in later steps.

In this step, get ready to:
  • Create a Python project folder with a virtual environment.
  • Build a chatbot script that talks to your local Qwen model.
  • Test the chatbot and spot its limitation with private data.
Create your project folder and virtual environment

Before writing any Python, you need a dedicated project folder and an isolated environment so your packages don't collide with anything else on your machine.

  • Move to your Desktop in PowerShell by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir doc-qa
cd doc-qa
  • Create a Python virtual environment and activate it by running these commands:
python -m venv venv
venv\Scripts\activate

Your terminal prompt now shows (venv) at the beginning. This means every pip install goes into the project's own sandbox.

What is a virtual environment?

A virtual environment is an isolated copy of Python for one project. Packages you install inside it stay separate from your system Python, so different projects can use different versions without conflicts.

Don't see (venv) in your prompt?

Make sure you typed venv\Scripts\activate with a backslash, not a forward slash. On Windows, the activation script lives inside Scripts\, not bin/.

If you see a red error about execution policies, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned and then try activating again.

Still stuck? Help me activate my Python virtual environment on Windows.

  • Install the LangChain Ollama package by running:
pip install langchain-ollama

You should see download progress bars followed by a line that starts with Successfully installed.

Why LangChain instead of the raw Ollama API?

Ollama exposes a REST API on localhost:11434 that you could call directly with Python's requests library. LangChain wraps that API and adds document loaders, text splitters, vector store connectors, and chain composition out of the box.

Building the full RAG pipeline from raw API calls would take 3-4x more code and teach plumbing instead of RAG concepts. LangChain lets you focus on the interesting parts.

Install failed?

Make sure your virtual environment is active. Your prompt should show (venv). If it does not, run venv\Scripts\activate again before retrying the install.

If you see a network error, check your internet connection. pip downloads packages from PyPI and needs internet access for this one-time install.

Need more help? Help me fix my pip install for langchain-ollama on Windows.

Write the chat script

LangChain's ChatOllama class connects Python to the Qwen model running inside Ollama. You'll wrap it in a loop that reads your input and prints the model's response.

  • Create a new file called chat.py in your doc-qa folder and paste in the following code:
# Connect to the local Qwen model through LangChain's Ollama wrapper
from langchain_ollama import ChatOllama

# Initialize the model with temperature=0 for consistent responses
llm = ChatOllama(model="qwen3.5:4b", temperature=0)

print("Offline AI Chat (type 'quit' to exit)")
print("-" * 40)

# Loop until the user types 'quit'
while True:
    question = input("\nYou: ")
    if question.lower() == "quit":
        break

    # Send the question to the local model and print the response
    response = llm.invoke(question)
    print(f"\nAI: {response.content}")

What does this code do?

  • ChatOllama(model="qwen3.5:4b", temperature=0) creates a connection to the Qwen model running inside Ollama on your machine. Setting temperature=0 makes the model give consistent, deterministic answers instead of random variations.
  • llm.invoke(question) sends your text to the local model and waits for the full response. No data leaves your machine.
  • response.content holds the model's text reply. The while True loop keeps the conversation going until you type quit.
  • Save chat.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete chat.py file. Compare it against yours line by line.

from langchain_ollama import ChatOllama

llm = ChatOllama(model="qwen3.5:4b", temperature=0)

print("Offline AI Chat (type 'quit' to exit)")
print("-" * 40)

while True:
    question = input("\nYou: ")
    if question.lower() == "quit":
        break

    response = llm.invoke(question)
    print(f"\nAI: {response.content}")
Run the chatbot and spot its blind spot

Your chatbot script is ready. Before you run it, think about what you expect to happen when you type a question.

  • Run the chatbot by entering this command:
python chat.py

You should see the banner Offline AI Chat (type 'quit' to exit) followed by a You: prompt waiting for your input.

  • Type a general knowledge question and press Enter. For example, try asking "What is the capital of France?".

The model takes a few seconds to generate its answer entirely on your CPU. No internet required. You should see a correct response appear after the AI: label.

First response is slow

The very first question takes longer because Ollama loads the model into memory. Follow-up questions in the same session are noticeably faster.

  • Now try asking the chatbot a specific question about a document on your computer. For example: "What are the key findings in my Q2 report?" or "Summarize the notes from my Monday meeting."

The model either makes something up or admits it does not know. It has no access to your files. This is the core limitation you'll fix with RAG in the next steps.

  • Type quit and press Enter to exit the chatbot.

Seeing a connection error?

If you see httpx.ConnectError: [Errno 61] Connection refused, Ollama is not running in the background. Look for the Ollama icon in your system tray (bottom-right of your taskbar). If it is missing, search for Ollama in your Start menu and open it.

If you see an ImportError for langchain_ollama, your virtual environment may not be active. Check that your prompt starts with (venv) and re-run pip install langchain-ollama if needed.

Something else going wrong? Help me debug my Python chat script that connects to Ollama.

You have a working offline chatbot running entirely on your machine. The model is smart, but it is blind to your files. Next up, you'll give it something specific to fail on, and then build the pipeline that fixes it.

Ask About Your Documents and Watch It Fail

In the last step, you built an offline chatbot that talks to a local AI model directly from your terminal. You asked it general knowledge questions and it responded without touching the internet.

But your chatbot has a blind spot. It has zero access to the files sitting on your machine. In this step, you'll gather some real documents, ask the model about them, and confirm exactly where it falls short.

In this step, get ready to:
  • Create a docs folder with your own text or PDF files.
  • Ask the chatbot about your documents and confirm it cannot answer correctly.
Create a docs folder with your files

To test whether the model can answer questions about your data, you need real documents with content you know well. That way, you can judge whether the model's answer is accurate or made up.

  • Create a docs folder inside your doc-qa project by running this command:
mkdir docs
  • Press the Windows key to open your search bar.
  • Type File Explorer and press Enter to open it.
  • Navigate to the doc-qa\docs folder on your Desktop.
  • Drag 3-5 of your own .txt or .pdf files into the folder. Pick files with specific content you can verify later.

No documents handy?

You can create a quick test file. Open Notepad, write a few paragraphs about a specific topic (meeting notes, a recipe, project plans), and save it as sample-notes.txt in the docs folder. The more specific the content, the easier it is to spot a wrong answer.

  • Confirm the files are in the folder by running:
ls docs

You should see your files listed in the output.

Ask the chatbot about your documents

Your documents are ready. Before you run the chatbot, think about this: the model was trained on public internet data, not your personal files. Do you think it will give you an accurate answer about your document's contents?

  • Start the chatbot by running:
python chat.py
  • Ask a specific question about one of the documents you just added. For example, if you added meeting notes, type something like What were the action items from the Q2 meeting?.
  • Read the model's response carefully and compare it to what the document actually says.

The model either makes something up, gives a vague generic answer, or tells you it does not have access to your files. This is the expected result.

Why can't the model answer?

The model only knows what was in its training data. Your personal documents were never part of that training, so it has no information to draw from.

This is the core problem that RAG (Retrieval-Augmented Generation) solves. Instead of retraining the model on your files, RAG retrieves relevant pieces of your documents and feeds them to the model at the moment you ask a question.

Seeing an error instead of a response?

  • If you see a ModuleNotFoundError, your virtual environment may have been deactivated. Run venv\Scripts\activate first, then try again.
  • If you see a connection error, Ollama might not be running. Check your system tray for the Ollama icon and make sure the service is active.
  • Still stuck? Help me debug an error when running python chat.py with Ollama.
  • Type quit to exit the chatbot.

The model is smart but completely blind to your data. In the next step, you'll build a pipeline that loads your documents, converts them into searchable chunks, and gives the model access to real answers.

Build the RAG Pipeline

In the last step, you asked the chatbot a specific question about your documents and it came up empty. The model is capable, but it is completely blind to your files.

RAG fixes this. It loads your documents, splits them into searchable chunks, converts each chunk into a numerical vector (called an embedding), and stores everything in a fast-searchable index. When you ask a question, the system finds the most relevant chunks and feeds them to the model as context.

In this step, get ready to:
  • Install the remaining Python packages for document loading, text splitting, and vector search.
  • Create an ingestion script that loads your documents, chunks them, embeds them, and saves the index to disk.
  • Run the pipeline and verify the index was created.
Install the remaining packages

The langchain-ollama package you installed earlier handles the connection to your local model. Now you need four more packages that handle the other pieces of the RAG pipeline: loading documents, splitting text, creating embeddings, and searching them.

  • Install all four packages by running this command in your terminal (make sure your virtual environment is still active):
pip install langchain-community langchain-text-splitters faiss-cpu pypdf

What are these packages?

  • langchain-community provides document loaders (PyPDFLoader, TextLoader) and the FAISS vector store wrapper.
  • langchain-text-splitters splits documents into overlapping chunks that fit into the model's context window.
  • faiss-cpu is Facebook's fast similarity-search library, optimised for CPU.
  • pypdf lets Python read and extract text from PDF files.
  • Wait for the installation to finish. You should see Successfully installed followed by the package names at the end of the output.

Install failing or hanging?

  • Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt. If not, run venv\Scripts\activate from inside your doc-qa folder.
  • If faiss-cpu specifically fails, try running pip install faiss-cpu on its own to isolate the error.
  • Help me troubleshoot a pip install failure for langchain-community, langchain-text-splitters, faiss-cpu, or pypdf on Windows.
Create the ingestion script

The ingestion script is the core of your RAG pipeline. It reads every file in your docs folder, breaks each one into small overlapping chunks, converts those chunks into numerical vectors, and saves everything into a searchable index on disk.

  • Create a new file called ingest.py in your doc-qa folder (the same folder where chat.py lives).
  • Start by adding the imports, path constants, and the document-loading section of the ingestion function:
import os
from langchain_community.document_loaders import PyPDFLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import FAISS

# Paths to the documents folder and the output index
DOCS_DIR = "./docs"
FAISS_INDEX = "./faiss_index"

def ingest():
    documents = []

    # Loop through every file in the docs folder
    for filename in os.listdir(DOCS_DIR):
        filepath = os.path.join(DOCS_DIR, filename)

        # Load PDFs using PyPDFLoader (one document per page)
        if filename.endswith(".pdf"):
            loader = PyPDFLoader(filepath)
            documents.extend(loader.load())
        # Load plain text files with explicit UTF-8 encoding
        elif filename.endswith(".txt"):
            loader = TextLoader(filepath, encoding="utf-8")
            documents.extend(loader.load())

    print(f"Loaded {len(documents)} pages from {DOCS_DIR}")

What does this code do?

  • DOCS_DIR and FAISS_INDEX are constants that point to where your documents live and where the searchable index will be saved.
  • The for loop walks through every file in docs/ and picks the right loader based on file extension. PDFs get PyPDFLoader (which creates one document per page). Text files get TextLoader.
  • loader.load() returns a list of LangChain Document objects. Each one holds the text content plus metadata (like the source filename).
  • Save ingest.py. You cannot run it yet because the function is not complete. The next chunk adds the processing logic.
  • Still inside ingest.py, add this code directly below the print(f"Loaded {len(documents)} pages from {DOCS_DIR}") line, still inside the ingest() function:
    # Split documents into overlapping chunks for better search
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=50,
        separators=["\n\n", "\n", " ", ""],
    )
    chunks = splitter.split_documents(documents)
    print(f"Split into {len(chunks)} chunks")

    # Convert chunks into numerical vectors using the local embedding model
    embeddings = OllamaEmbeddings(model="all-minilm")

    # Build the FAISS index and save it to disk
    vectorstore = FAISS.from_documents(chunks, embeddings)
    vectorstore.save_local(FAISS_INDEX)
    print(f"FAISS index saved to {FAISS_INDEX}")

if __name__ == "__main__":
    ingest()

What does this code do?

  • RecursiveCharacterTextSplitter breaks each document into chunks of roughly 500 characters with a 50-character overlap. The overlap means neighbouring chunks share a few sentences, so the search does not lose context at chunk boundaries.
  • OllamaEmbeddings(model="all-minilm") connects to the all-minilm embedding model you pulled earlier. It converts each text chunk into a vector of numbers that captures its meaning.
  • FAISS.from_documents() builds the searchable vector index from all the chunks. save_local() writes the index to a folder on disk so you can reload it later without re-embedding.
  • The if __name__ == "__main__": guard means the ingest() function only runs when you execute the script directly.

Why overlapping chunks?

Imagine cutting a paragraph exactly at the 500-character mark. The sentence that straddles the cut loses its meaning in both halves. The 50-character overlap keeps a few shared sentences on both sides of each split, so the search can still find relevant matches even when a key sentence spans two chunks.

  • Save ingest.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved ingest.py and that both code chunks are present, in order.

ⓧ I'd like to double check the full code

Scroll up and confirm both code chunks are present in ingest.py, in this order:

  • Chunk 1: The five imports, two path constants, and the ingest() function definition with the document-loading for loop, ending with the "Loaded" print statement.
  • Chunk 2: The RecursiveCharacterTextSplitter setup, OllamaEmbeddings, FAISS.from_documents(), save_local(), and the if __name__ guard at the bottom.

Make sure the splitting, embedding, and saving code is indented inside the ingest() function. The if __name__ guard should be at the top level (no indentation).

Run the pipeline and verify the index

The script is ready. Before you run it, think about what you expect to see. How many pages do your documents have in total? The script will print that number, then the number of chunks it created, then confirm the index was saved.

  • Run the ingestion script by entering this command:
python ingest.py

You should see three lines of output. The first reports how many pages were loaded from your docs folder. The second shows how many chunks those pages were split into. The third confirms the FAISS index was saved.

What should I see?

The exact numbers depend on your documents, but a typical run with 3-5 files looks something like this:

Loaded 12 pages from ./docs
Split into 47 chunks
FAISS index saved to ./faiss_index

The embedding step may take a minute or two on CPU. That is normal. The all-minilm model is small but still needs to process every chunk.

  • Confirm the index folder was created by running:
ls faiss_index
  • You should see index.faiss and index.pkl listed inside the folder. These two files are your searchable vector index.

Seeing an error when running ingest.py?

  • If you see an import error, make sure all four packages installed successfully. Re-run the pip install command from earlier in this step.
  • If you see "Loaded 0 pages", check that your docs folder contains .pdf or .txt files. Other file types (like .docx) are not supported by this script.
  • Make sure Ollama is still running in the background. The embedding step needs the all-minilm model. If Ollama stopped, restart it before running the script again.
  • Help me debug an error when running my ingest.py RAG pipeline script with LangChain, FAISS, and Ollama.

Your documents are now indexed and searchable. In the next step, you will wire this index into a query pipeline that retrieves the most relevant chunks, feeds them to the Qwen model as context, and returns grounded answers with source citations.

Query Your Documents with Cited Answers

Your documents are now stored as searchable vectors in the FAISS index. The hard work of turning raw text into machine-searchable numbers is done.

Now you need a way to actually query that index. In this step, you'll build a script that finds the most relevant chunks for your question, feeds them as context to the local Qwen model, and prints a grounded answer with source citations.

In this step, get ready to:
  • Create a query script that retrieves relevant document chunks and generates cited answers.
  • Test the full RAG pipeline by asking questions about your documents.
  • Add a new document and confirm the pipeline picks it up automatically.
Create the query script

The query script has three jobs: load the index you saved in the last step, find the chunks most relevant to each question, and feed those chunks to the Qwen model as context so it generates a grounded answer instead of guessing.

  • Create a new file called query.py inside your doc-qa folder (the same folder as ingest.py).
  • Add the imports, the index path constant, and two helper functions by pasting this code:
from langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain_community.vectorstores import FAISS
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

# Path to the FAISS index saved by ingest.py
FAISS_INDEX = "./faiss_index"

# Join retrieved chunks into a single context string for the prompt
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# Extract unique source file names from retrieved documents
def get_sources(docs):
    sources = set()
    for doc in docs:
        source = doc.metadata.get("source", "unknown")
        sources.add(source)
    return list(sources)

What does this code do?

The imports bring in the same OllamaEmbeddings and ChatOllama classes you used before, plus LangChain's chain-building tools for composing the retrieval and generation pipeline.

  • format_docs() combines the retrieved text chunks into one string that gets injected into the prompt as context.
  • get_sources() collects the file names from each retrieved chunk's metadata so you can see which documents the answer came from.
  • Save query.py.
  • Confirm the imports resolve correctly by running this command in your terminal:
python query.py

You should see no error output. If the command finishes silently and returns you to the prompt, your imports are correct.

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt.
  • Check that langchain-core is installed. It should have been installed automatically as a dependency of langchain-ollama.

Help me debug a ModuleNotFoundError when running query.py

Now you'll add the main() function. This function loads the saved FAISS index, sets up the retriever, builds a prompt that forces the model to use only retrieved context, and wires everything into a single chain.

  • In query.py, add the main() function below the get_sources() function by pasting this code:
def main():
    # Load the FAISS index from disk
    embeddings = OllamaEmbeddings(model="all-minilm")
    vectorstore = FAISS.load_local(
        FAISS_INDEX,
        embeddings,
        allow_dangerous_deserialization=True,
    )

    # Create a retriever that fetches the 4 most relevant chunks
    retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

    # Connect to the same local Qwen model for generation
    llm = ChatOllama(model="qwen3.5:4b", temperature=0)

    # Build a prompt that grounds answers on retrieved context
    prompt = ChatPromptTemplate.from_messages([
        ("system",
         "You are a helpful assistant. Answer the question using only the "
         "context below. If the answer is not in the context, say you do not "
         "know. Cite which document the information comes from.\n\n"
         "Context:\n{context}"),
        ("human", "{question}"),
    ])

    # Chain: retrieve -> fill prompt -> generate -> parse output
    rag_chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )

What does this code do?

  • FAISS.load_local() reloads the vector index you saved in the previous step, so you never need to re-embed your documents just to query them.
  • as_retriever(search_kwargs={"k": 4}) turns the vector store into a retriever that returns the 4 chunks most similar to your question.
  • The system prompt tells the model to answer using only the retrieved context. This is what prevents hallucination.
  • The rag_chain pipes the question through retrieval, prompt filling, model generation, and output parsing in a single call.

Why allow_dangerous_deserialization?

LangChain requires this flag when loading a FAISS index from disk because the index file uses Python's pickle format, which can execute arbitrary code. This is safe here because you built the index yourself in the previous step. Never load an index you received from an untrusted source.

The setup is ready. The last piece is the interactive loop that takes your questions, retrieves relevant chunks, generates answers, and prints the source citations.

  • Still inside the main() function, add the interactive query loop and the entry point below the rag_chain by pasting this code:
    # Interactive query loop
    print("Document Q&A (type 'quit' to exit)")
    print("-" * 40)

    while True:
        question = input("\nYou: ")
        if question.lower() == "quit":
            break

        # Retrieve relevant chunks and extract source file names
        retrieved_docs = retriever.invoke(question)
        sources = get_sources(retrieved_docs)

        # Generate the grounded answer
        answer = rag_chain.invoke(question)

        print(f"\nAI: {answer}")
        print(f"\nSources: {', '.join(sources)}")

if __name__ == "__main__":
    main()

What does this code do?

  • For each question, the loop first calls retriever.invoke() to find the 4 most relevant chunks, then passes the question through the full RAG chain to generate a grounded answer.
  • The sources are printed separately so you can verify exactly which files the answer drew from.
  • if __name__ == "__main__" ensures main() only runs when you execute the script directly, not when it is imported.
  • Save query.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved query.py before running it in the next substep.

ⓧ I'd like to double check the full code

Your query.py should contain these sections in order. Scroll up and confirm each one is present.

  • Five import lines and the FAISS_INDEX constant.
  • The format_docs() helper function.
  • The get_sources() helper function.
  • The main() function with index loading, retriever, LLM, prompt, and rag_chain setup.
  • The interactive while True query loop inside main().
  • The if __name__ == "__main__": main() entry point at the bottom.
Ask your documents a question

Remember the question you asked in Step 3 that the chatbot couldn't answer? This is the moment you find out whether the RAG pipeline fixes that gap.

Before you run the script, think about what you expect to see. Will the answer be accurate this time? Will the model cite which document the information came from?

  • Run the query script by running this command:
python query.py
  • Type the same question that failed in Step 3 and press Enter.

This time, the answer is accurate and grounded in your actual document. Below the answer, you'll see a Sources: line listing exactly which files the answer drew from.

Why is this answer better than Step 3?

In Step 3, the model had no access to your files. It could only guess based on its training data. Now, the retriever searches your FAISS index for the 4 most relevant chunks, and the system prompt tells the model to answer using only that context. The model's knowledge is grounded in your documents, not its imagination.

  • Ask 2-3 more questions about different documents to see the retrieval and source citations in action.
  • Try a question about something that is not in your documents. The model should tell you it does not know rather than making something up.
  • Type quit to exit the query loop when you're done.

Getting slow or empty responses?

  • The first query may take 30-60 seconds on CPU while the model loads into memory. Subsequent queries should be faster.
  • If the answer is empty or just repeats your question, check that your docs/ folder actually has files in it and that you ran python ingest.py after adding them.
  • Make sure Ollama is still running in the background. If you restarted your computer since the last step, Ollama may need to be reopened.

Help me debug slow or empty responses from my RAG query script

Add a new document and re-index

A useful document Q&A tool needs to handle new files without rebuilding everything from scratch. Your pipeline already supports this because the ingestion step scans the entire docs/ folder every time it runs.

  • Add a new text or PDF file to your docs/ folder. Pick something with specific facts you can verify in a question.
  • Re-run the ingestion script to rebuild the index with the new document by running this command:
python ingest.py

You should see an updated count in the output. The number of loaded pages and chunks should be higher than before because the new document is now included.

  • Run the query script again by running this command:
python query.py
  • Ask a question about the new document you just added.

The answer references your new document and cites it in the sources. The pipeline picked it up without any code changes.

  • Type quit to exit.

New document not showing up in answers?

  • Make sure you ran python ingest.py after adding the new file. The FAISS index only updates when you re-run ingestion.
  • Check that the new file has a .txt or .pdf extension. The ingestion script only loads those two file types.
  • Verify the new file is inside the docs/ folder, not in the project root or a subfolder inside docs/.

Help me figure out why my new document isn't being picked up by the RAG pipeline

Extra for Experts: Why does re-indexing rebuild everything?

The current ingestion script re-embeds all documents every time. For a small collection, this is fast enough. In production, you would track which files have changed and only embed the new or modified ones. FAISS supports merging indexes with merge_from() for incremental updates.

Your full RAG pipeline is working end-to-end. You can drop any text or PDF into a folder, index it, and query it with accurate, cited answers. Next up, you'll take this further in the Secret Mission by teaching the tool to read and answer questions about code.

Secret mission

Extend the Tool to Answer Questions About Code

Your document Q&A tool reads PDFs and text files. But what if it could also read your Python and JavaScript source code? In this secret mission, you'll add language-aware chunking that respects function and class boundaries, turning your tool into a private coding assistant.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine, so there are no cloud services or ongoing costs to worry about. Decide whether to keep your resources as they are, free up disk space by removing the models, or delete everything entirely.

Resources you used:

  • Ollama installation and two downloaded models (qwen3.5:4b at ~3.4 GB and all-minilm at ~23 MB) stored in ~\.ollama\models.
  • Project folder doc-qa on your Desktop, containing your Python scripts, virtual environment, FAISS index, and documents.

Keep everything running

No action needed. Choose this if you plan to keep querying your documents or want to experiment with new files.

  • Ollama runs as a background service but uses no CPU or RAM when idle.
  • The downloaded models consume about 3.5 GB of disk space total.
  • Your FAISS index persists on disk for instant re-querying whenever you need it.

Pause - I'll come back to this later

Remove the models to free about 3.5 GB of disk space while keeping Ollama installed and your project files intact. You can re-pull the models anytime.

  • Remove the chat model by running this command in PowerShell:
ollama rm qwen3.5:4b
  • Remove the embedding model by running:
ollama rm all-minilm
  • When you are ready to come back, re-pull the models with ollama pull qwen3.5:4b and ollama pull all-minilm.

Delete - I don't want to use this again

Remove all project resources, models, and the Ollama installation entirely.

Save your personal files first

Your docs folder contains your personal documents. Copy anything you want to keep out of doc-qa\docs before deleting the project folder.

  • Remove the Ollama models by running these commands in PowerShell:
ollama rm qwen3.5:4b
ollama rm all-minilm
  • Quit Ollama by right-clicking the Ollama icon in your system tray (bottom-right of your taskbar) and selecting Quit.
  • Open Settings on your computer by pressing the Windows key and typing Settings.
  • Go to Apps, then Installed apps.
  • Find Ollama in the list and click Uninstall.
  • Delete the leftover .ollama folder (which stores model data) by running:
Remove-Item -Recurse -Force ~\.ollama
  • Delete the project folder by running:
Remove-Item -Recurse -Force ~\Desktop\doc-qa

Nice Work!

Nice Work!

Nice work! You just built a fully offline document Q&A tool powered by a local LLM running on your own machine.

You've learned how to:

  • Install Ollama and run a local LLM entirely offline on 8GB RAM with zero cloud dependency.
  • Build a complete RAG pipeline covering document loading, chunking, embedding, vector storage with FAISS, retrieval, and grounded generation.
  • Query your own private documents and get accurate answers with source citations, proving the system works on real data.
  • Secret Mission: Extend the tool to read and answer questions about codebases using language-aware chunking, turning it into a private coding assistant.

Ready to quiz yourself?