Local RAG Research Assistant

Build a local AI assistant that answers questions from your PDF research papers.

Introduction

30 Second Summary

Imagine having 97 research papers on your computer and needing to find a specific fact buried somewhere in them. Searching manually through each one could take hours.

In this project, you will build a local AI research assistant that has "read" all your papers and can answer questions about them with exact citations. Everything runs on your own machine, completely free.

What You'll Build

A terminal-based research assistant that processes your PDF collection into searchable vector embeddings and uses a local Qwen AI model to answer questions grounded in your documents.

By the end of this project, you'll have:

  • An ingestion pipeline that converts 97 PDFs into a persistent vector database you only need to build once.
  • A conversational chat interface with memory that answers questions using only your research papers.
  • Source attribution that tells you exactly which paper and page each answer comes from.
  • Secret Mission: A search mode that finds all relevant passages across your papers without generating an answer, giving you a reading list with paper names and page numbers.

Are there any prerequisites?

This project assumes you have Ollama installed with the qwen2.5 model already pulled, VS Code installed, and basic familiarity with running terminal commands.

You also need a folder of PDFs you want to search through. The project is written around 97 research papers, but any collection of PDFs will work.

Set Up Your Python Environment

Your 97 research papers hold a wealth of knowledge, but searching through each PDF manually is painfully slow. To build an AI assistant that can answer questions from all of them, you first need a clean Python environment with the right tools installed.

In this step, you will set up an isolated Python project with all the libraries needed for PDF parsing, vector storage, and AI orchestration. You will also pull a dedicated embedding model that turns text into searchable vectors.

In this step, get ready to:
  • Create a project folder and Python virtual environment.
  • Install the Python packages for PDF parsing, vector storage, and LangChain orchestration.
  • Pull the nomic-embed-text embedding model via Ollama.
Create a project folder and Python virtual environment

Before installing any packages, you need to confirm that Python 3.10 or higher is available on your machine. The PDF parsing library (pymupdf4llm) requires Python 3.10+ to run.

  • In VS Code, open the integrated terminal by pressing Ctrl+` (macOS) or Ctrl+` (Windows).
  • Check your Python version by running this command:
python3 --version

✔️ I see version 3.10 or higher

Your Python version is compatible. You are ready to create your project folder.

ⓧ I see an older version

macOS often includes an older Python version via Xcode Command Line Tools. You need to install a newer version alongside it.

  • Visit python.org/downloads and download the latest Python 3 installer for your operating system.
  • Run the downloaded installer and follow the prompts.
  • After installation completes, close and reopen your terminal.
  • Run python3 --version again to confirm the new version is 3.10 or higher.

Why does macOS show an old version?

macOS includes a system Python managed by Apple's Xcode tools. Installing from python.org adds a separate, newer Python that takes priority in your terminal path. Both installations coexist safely.

ⓧ Command not found

Python is not installed on your machine yet. You need to install it from the official source.

  • Visit python.org/downloads and download the latest Python 3 installer for your operating system.
  • Run the downloaded installer and follow the prompts.
  • After installation completes, close and reopen your terminal.
  • Run python3 --version again to confirm it shows 3.10 or higher.

Now you will create a dedicated project folder and a virtual environment inside it to keep your dependencies isolated from the rest of your system.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir research-rag
cd research-rag
  • Create a Python virtual environment and activate it by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python3 -m venv venv
venv\Scripts\activate

What is a virtual environment?

A virtual environment is an isolated Python workspace. Packages you install inside it stay contained in this project folder. This prevents version conflicts with other Python projects on your machine.

You will know the virtual environment is active when you see (venv) at the start of your terminal prompt.

  • Confirm you see (venv) at the beginning of your terminal prompt.

Don't see (venv) in your prompt?

  • Make sure you ran the source venv/bin/activate command (macOS) or venv\Scripts\activate (Windows) after creating the environment.
  • If you see a permissions error on Windows, try running PowerShell as Administrator and entering Set-ExecutionPolicy RemoteSigned first.

Help me activate my Python virtual environment.

Install Python dependencies

Your RAG pipeline needs several libraries working together. PyMuPDF4LLM extracts text from PDFs (including tables). LangChain orchestrates the retrieval and generation chain. ChromaDB stores your document chunks as searchable vector embeddings.

  • Install all required packages by running this command:
pip install pymupdf4llm langchain langchain-core langchain-text-splitters langchain-chroma langchain-ollama chromadb

What are these packages?

  • pymupdf4llm converts PDF pages to Markdown, preserving tables and reading order.
  • langchain, langchain-core, and langchain-text-splitters provide the orchestration framework and text chunking tools.
  • langchain-chroma connects LangChain to the ChromaDB vector database.
  • langchain-ollama connects LangChain to your local Ollama models.
  • chromadb is the vector database that stores and retrieves your document embeddings.

The installation may take a minute or two as it downloads all dependencies. You should see a success message for each package.

Seeing installation errors?

  • Make sure (venv) appears in your prompt. If it does not, your virtual environment is not active and packages will try to install globally.
  • If you see a build error mentioning missing compilers, ensure Xcode Command Line Tools are installed by running xcode-select --install in a separate terminal window.

Help me fix my pip install errors for the RAG project dependencies.

Pull the embedding model and verify Ollama

Your RAG pipeline needs two models from Ollama. You already have Qwen 2.5 for generating answers. Now you need nomic-embed-text, a small (274MB) model purpose-built for creating vector embeddings that power semantic search.

  • Pull the embedding model by running this command:
ollama pull nomic-embed-text

This downloads approximately 274MB. Wait for the download to complete before continuing.

  • Verify both models are available by running:
ollama list
  • Confirm you see both nomic-embed-text and qwen2.5 in the output list.

Why two separate models?

Each model has a different job. The nomic-embed-text model converts text into 768-dimensional number arrays (vectors) optimized for finding similar passages. The qwen2.5 model reads those passages and generates human-readable answers.

Using a dedicated embedding model is faster and more accurate for retrieval than repurposing a chat model for both tasks. With 8GB of RAM, keeping the embedding model small (274MB) leaves plenty of room for the chat model (~2.4GB).

Ollama command not found?

  • Make sure Ollama is running. Open the Ollama application if it is not active in your menu bar.
  • If the ollama command is not recognized, try opening a new terminal tab. The path may not have been set in your current session.

Help me get the ollama command working in my terminal.

Your environment is ready. You have an isolated Python workspace with all the right tools, plus two local AI models standing by. Next up, you will feed your 97 PDFs through an ingestion pipeline that turns them into a searchable vector database.

Ingest Your PDFs into the Vector Store

Your Python environment is ready and both AI models are loaded. Now it's time to make your 97 research papers searchable by an AI.

In this step, you'll build an ingestion pipeline that reads every PDF, extracts text (including tables), splits it into searchable chunks, and stores them as vector embeddings in a local database. Once this runs, you never need to process the PDFs again.

In this step, get ready to:
  • Set up the project structure and copy your PDFs into it.
  • Write an ingestion script that converts PDFs into embedded chunks.
  • Run the ingestion to populate your vector store.
Set up the project structure and copy your PDFs

Your ingestion script needs a known location to find PDFs. You'll create a papers/ directory inside your project and copy all 97 research papers into it.

  • Create the papers directory inside your project by running this command:
mkdir papers
  • Copy your entire papers folder (including subfolders) into the papers/ directory by running:
cp -r [[PAPERS_PATH="path/to/your/papers/folder"]]/* ./papers/

What does this command do?

The cp -r command recursively copies all files and subfolders from your source folder into the project's papers/ directory. The /* at the end means "everything inside this folder."

  • Verify your PDFs are in place by running:
ls papers/ | head -10

You should see a list of PDF filenames and/or subfolder names from your collection.

Don't see any files?

Make sure the source path in your cp command points to the folder that actually contains your PDFs. You can check by running ls on your source path first to confirm the files are there.

help me find the right path to my papers folder

Write the ingestion script

The ingestion script is the heart of your RAG pipeline. It reads every PDF, converts each page to Markdown (preserving tables), splits the text into overlapping chunks, and embeds them into a ChromaDB vector store.

  • Create a new file called ingest.py in your research-rag folder and add these imports and configuration constants:
import os
import pymupdf4llm
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma

DOCS_DIR = "./papers"
CHROMA_DIR = "./chroma_db"
COLLECTION_NAME = "research_papers"
EMBED_MODEL = "nomic-embed-text"
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200

What does this code do?

  • pymupdf4llm is the PDF parser that converts documents to Markdown, preserving tables and multi-column layouts.
  • The LangChain imports provide the document model, text splitter, embeddings wrapper, and vector store interface.
  • CHUNK_SIZE = 1000 means each text chunk is about 1000 characters. This is a good balance for research papers with dense text.
  • CHUNK_OVERLAP = 200 means adjacent chunks share 200 characters at their boundaries. This prevents important sentences from being split across chunks.

You won't see a change yet. The next chunk adds the PDF loading function.

  • Below the constants in ingest.py, add the load_pdfs() function:

def load_pdfs(docs_dir):
    """Load all PDFs from directory and subdirectories, convert to per-page Markdown."""
    documents = []
    pdf_files = []

    for root, dirs, files in os.walk(docs_dir):
        for file in files:
            if file.lower().endswith(".pdf"):
                pdf_files.append(os.path.join(root, file))

    print(f"Found {len(pdf_files)} PDF files")

    for i, pdf_path in enumerate(pdf_files, 1):
        filename = os.path.basename(pdf_path)
        print(f"[{i}/{len(pdf_files)}] Processing: {filename}")
        try:
            page_chunks = pymupdf4llm.to_markdown(pdf_path, page_chunks=True)
            for chunk in page_chunks:
                page_num = chunk["metadata"].get("page_number", 0)
                text = chunk["text"].strip()
                if not text:
                    continue
                doc = Document(
                    page_content=text,
                    metadata={
                        "source": pdf_path,
                        "filename": filename,
                        "page": page_num + 1,
                    },
                )
                documents.append(doc)
        except Exception as e:
            print(f"  Error: {e}")

    return documents

What does this function do?

  • os.walk() traverses all subdirectories to find every PDF file, no matter how deeply nested.
  • pymupdf4llm.to_markdown(pdf_path, page_chunks=True) converts each PDF into Markdown text split by page. Tables get converted to GitHub-flavored Markdown with pipe characters.
  • Each page becomes a LangChain Document with metadata tracking the source filename and page number. This metadata powers source citations later.
  • The try/except block catches corrupted PDFs gracefully so one bad file doesn't crash the entire ingestion.

Why PyMuPDF4LLM over simpler PDF parsers?

Research papers often have tables, multi-column layouts, and complex formatting. Basic PDF extractors scramble multi-column text and lose table structure entirely.

PyMuPDF4LLM performs layout analysis and reconstructs reading order. It also converts tables into structured Markdown, so your embeddings capture tabular data accurately.

You won't see a change yet. The next chunk adds the main function that ties everything together.

  • Below load_pdfs(), add the main() function and entry point:

def main():
    print("=" * 50)
    print("Research Paper Ingestion")
    print("=" * 50)

    print(f"\nLoading PDFs from: {DOCS_DIR}")
    documents = load_pdfs(DOCS_DIR)
    print(f"\nLoaded {len(documents)} page documents from PDFs")

    print("\nSplitting into chunks...")
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=CHUNK_SIZE,
        chunk_overlap=CHUNK_OVERLAP,
    )
    chunks = splitter.split_documents(documents)
    print(f"Created {len(chunks)} chunks")

    print("\nEmbedding and storing in ChromaDB...")
    print("(This may take several minutes for 97 PDFs)")
    embeddings = OllamaEmbeddings(model=EMBED_MODEL)
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=CHROMA_DIR,
        collection_name=COLLECTION_NAME,
    )

    print(f"\nDone! Vector store saved to: {CHROMA_DIR}")
    print(f"Total chunks indexed: {len(chunks)}")
    print("You can now run chat.py to ask questions.")


if __name__ == "__main__":
    main()

What does this function do?

  • RecursiveCharacterTextSplitter takes the full-page documents and splits them into smaller, overlapping chunks. Metadata from each parent page propagates to all its child chunks.
  • OllamaEmbeddings(model=EMBED_MODEL) creates an embedding function that uses your local nomic-embed-text model to convert text into 768-dimensional vectors.
  • Chroma.from_documents() embeds every chunk and stores the vectors in a persistent database on disk. You only need to run this once.
  • if __name__ == "__main__": ensures the script only runs when you execute it directly, not when it's imported elsewhere.
  • Save ingest.py.

Seeing a syntax error when you save?

Check that all your f-strings use curly braces correctly (e.g. f"Found {len(pdf_files)} PDF files"). Also confirm that the indentation inside load_pdfs() uses consistent spaces throughout.

help me fix a syntax error in my ingest.py file

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on to running the ingestion.

ⓧ I'd like to double check the full code

Here's the complete ingest.py for comparison:

import os
import pymupdf4llm
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma

DOCS_DIR = "./papers"
CHROMA_DIR = "./chroma_db"
COLLECTION_NAME = "research_papers"
EMBED_MODEL = "nomic-embed-text"
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200


def load_pdfs(docs_dir):
    """Load all PDFs from directory and subdirectories, convert to per-page Markdown."""
    documents = []
    pdf_files = []

    for root, dirs, files in os.walk(docs_dir):
        for file in files:
            if file.lower().endswith(".pdf"):
                pdf_files.append(os.path.join(root, file))

    print(f"Found {len(pdf_files)} PDF files")

    for i, pdf_path in enumerate(pdf_files, 1):
        filename = os.path.basename(pdf_path)
        print(f"[{i}/{len(pdf_files)}] Processing: {filename}")
        try:
            page_chunks = pymupdf4llm.to_markdown(pdf_path, page_chunks=True)
            for chunk in page_chunks:
                page_num = chunk["metadata"].get("page_number", 0)
                text = chunk["text"].strip()
                if not text:
                    continue
                doc = Document(
                    page_content=text,
                    metadata={
                        "source": pdf_path,
                        "filename": filename,
                        "page": page_num + 1,
                    },
                )
                documents.append(doc)
        except Exception as e:
            print(f"  Error: {e}")

    return documents


def main():
    print("=" * 50)
    print("Research Paper Ingestion")
    print("=" * 50)

    print(f"\nLoading PDFs from: {DOCS_DIR}")
    documents = load_pdfs(DOCS_DIR)
    print(f"\nLoaded {len(documents)} page documents from PDFs")

    print("\nSplitting into chunks...")
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=CHUNK_SIZE,
        chunk_overlap=CHUNK_OVERLAP,
    )
    chunks = splitter.split_documents(documents)
    print(f"Created {len(chunks)} chunks")

    print("\nEmbedding and storing in ChromaDB...")
    print("(This may take several minutes for 97 PDFs)")
    embeddings = OllamaEmbeddings(model=EMBED_MODEL)
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=CHROMA_DIR,
        collection_name=COLLECTION_NAME,
    )

    print(f"\nDone! Vector store saved to: {CHROMA_DIR}")
    print(f"Total chunks indexed: {len(chunks)}")
    print("You can now run chat.py to ask questions.")


if __name__ == "__main__":
    main()
Run the ingestion

Your script is ready. Running it will process all 97 PDFs, embed every chunk, and save the results to a persistent database on disk.

This is a one-time operation. Once the vector store is built, your chat script can load it instantly without re-processing the PDFs.

  • Run the ingestion script by executing this command:
python3 ingest.py

How long does this take?

Expect 5-15 minutes depending on the total page count across your 97 PDFs. You'll see progress updates as each file is processed.

The script uses nomic-embed-text for embeddings, which is lightweight at 274MB. Ollama manages model loading automatically, so your 8GB of RAM can handle this comfortably.

  • Wait for the script to finish. You should see output ending with lines like:
Done! Vector store saved to: ./chroma_db
Total chunks indexed: <number>
You can now run chat.py to ask questions.
  • Confirm the vector store was created on disk by running:
ls chroma_db/

You should see files and directories inside chroma_db/. This is your persistent vector store containing all embedded chunks from your 97 research papers.

Script crashes or hangs?

  • If you see a connection error mentioning Ollama, make sure Ollama is running. Open a new terminal window and run ollama list to check.
  • If the script errors on a specific PDF, it will print the filename and continue processing the rest. A few skipped files won't affect your results.
  • If you see a "No module named" error, confirm your virtual environment is still active. You should see (venv) at the start of your terminal prompt.

help me debug my ingestion script

Your 97 research papers are now searchable by AI. The vector store persists on disk, so you never need to run ingestion again unless you add new papers. Next up, you'll build the conversational chat interface that queries this store and generates grounded answers with citations.

Build the Conversational RAG Chain

Your vector store is loaded and ready to answer questions. Now you need a way to actually ask those questions and get useful, grounded answers back.

In this step, you will build the conversational interface that ties everything together. You will connect your ChromaDB vector store to the Qwen model through LangChain, add conversation memory so the assistant remembers follow-up questions, and design a grounding prompt that prevents hallucination.

In this step, get ready to:
  • Create a chat script that loads your vector store and connects it to Qwen.
  • Add conversation memory and source citation formatting.
  • Design a grounding prompt that keeps answers faithful to your papers.
Write the chat script with retrieval

The chat script needs to reconnect to your persisted vector store, create a retriever that pulls the most relevant chunks for each question, and wire everything to the Qwen language model.

  • Create a new file called chat.py in your research-rag folder by running:
touch chat.py
  • Open chat.py in VS Code and add the imports, constants, and beginning of the main() function by pasting the following code:
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

CHROMA_DIR = "./chroma_db"
COLLECTION_NAME = "research_papers"
EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "qwen2.5:3b"
RETRIEVAL_K = 4
MAX_HISTORY = 3


def main():
    # Load the persisted vector store from disk
    print("Loading vector store...")
    embeddings = OllamaEmbeddings(model=EMBED_MODEL)
    vector_store = Chroma(
        persist_directory=CHROMA_DIR,
        embedding_function=embeddings,
        collection_name=COLLECTION_NAME,
    )
    # Create a retriever that returns the top 4 most relevant chunks
    retriever = vector_store.as_retriever(search_kwargs={"k": RETRIEVAL_K})

    # Connect to the Qwen model with low temperature for factual answers
    llm = ChatOllama(model=CHAT_MODEL, temperature=0.1, num_ctx=4096)

What does this code do?

This sets up the three core components of the RAG pipeline:

  • OllamaEmbeddings uses the same nomic-embed-text model you used during ingestion so it can find matching vectors.
  • Chroma() loads the existing database from disk rather than creating a new one. No re-ingestion needed.
  • ChatOllama connects to Qwen 2.5 with temperature=0.1 (keeping answers factual) and num_ctx=4096 (enough room for retrieved context plus conversation history).

You won't see a change yet. The next chunks will add the formatting helpers and interactive loop that make this runnable.

Why a low temperature?

Temperature controls how creative the model's responses are. For a research assistant, you want factual, consistent answers grounded in your papers. A temperature of 0.1 keeps the model conservative and focused on the provided context.

Add conversation memory and source citations

When you ask a follow-up question like "Can you elaborate on that?", the assistant needs memory of what "that" refers to. You also want every answer to cite which paper and page it came from.

  • In chat.py, add the following two helper functions between the MAX_HISTORY = 3 constant and the def main(): line:
def format_docs(docs):
    """Format retrieved documents with source attribution."""
    formatted = []
    for doc in docs:
        filename = doc.metadata.get("filename", "Unknown")
        page = doc.metadata.get("page", "?")
        formatted.append(f"[Source: {filename}, Page {page}]\n{doc.page_content}")
    return "\n\n---\n\n".join(formatted)


def format_history(history):
    """Format conversation history for the prompt."""
    if not history:
        return "No previous conversation."
    lines = []
    for question, answer in history[-MAX_HISTORY:]:
        lines.append(f"User: {question}")
        lines.append(f"Assistant: {answer}")
    return "\n".join(lines)

What do these functions do?

  • format_docs() takes the raw retrieved chunks and prepends each one with its source filename and page number. When the model reads these, it can cite exactly where information came from.
  • format_history() keeps only the last 3 exchanges (MAX_HISTORY) to avoid overflowing the context window. This gives the model enough memory for follow-ups without consuming too many tokens.

You won't see a change yet. The next chunk completes main() with the prompt and interactive loop.

Design the grounding prompt and interactive loop

The grounding prompt is what prevents your assistant from making things up. It explicitly tells the model to answer ONLY from the retrieved context and to admit when it does not have enough information.

  • In chat.py, add the following code directly below the llm = ChatOllama(model=CHAT_MODEL, temperature=0.1, num_ctx=4096) line, still inside main():
    # Define the grounding prompt template
    template = """You are a research assistant. Answer the question based ONLY on the following context from research papers. If the context does not contain enough information to answer, say "I don't have enough information in the papers to answer this."

Always mention which source paper(s) your answer comes from.

Conversation History:
{history}

Context from Research Papers:
{context}

Question: {question}

Answer:"""

    prompt = ChatPromptTemplate.from_template(template)
    # Build the chain: prompt -> model -> parse output as string
    chain = prompt | llm | StrOutputParser()

    # Track conversation history as (question, answer) tuples
    history = []

    print("\n" + "=" * 50)
    print("Research Paper Assistant")
    print("=" * 50)
    print("Ask questions about your papers. Type 'quit' to exit.\n")

    # Interactive loop - keeps running until user types quit
    while True:
        question = input("You: ").strip()

        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break

        if not question:
            continue

        print("\nSearching papers...")
        docs = retriever.invoke(question)
        context = format_docs(docs)
        history_text = format_history(history)

        print("Generating answer...\n")
        response = chain.invoke({
            "context": context,
            "history": history_text,
            "question": question,
        })

        print(f"Assistant: {response}\n")
        history.append((question, response))


if __name__ == "__main__":
    main()

What does this code do?

  • The template string is a prompt with three placeholders: {history}, {context}, and {question}. LangChain fills these in at runtime.
  • The chain = prompt | llm | StrOutputParser() line uses LangChain's pipe operator to compose the full pipeline: fill the template, send to the model, parse the output as a string.
  • The while True loop keeps the conversation going. Each iteration retrieves fresh context for the new question, formats the history, invokes the chain, and appends the exchange to memory.

Why include history directly in the prompt?

LangChain has memory abstractions, but they add complexity and some are deprecated. Including the last 3 exchanges directly in the prompt template is transparent and easy to debug. You can see exactly what the model receives.

  • Save chat.py.
  • Test that the script starts correctly by running:
python3 chat.py
  • You should see "Loading vector store..." followed by the "Research Paper Assistant" banner and a You: input prompt.
  • Type quit and press **Enter** to exit for now. You will test it properly in the next step.

Not seeing the You: prompt?

  • If you see an indentation error, check that all the code inside main() is indented with exactly 4 spaces. The template, prompt, chain, and while True should all be at one indent level inside main().
  • If you see a connection error, make sure Ollama is still running. Open a new terminal and run ollama list to confirm.
  • If you see a collection not found error, make sure the chroma_db/ folder exists in your research-rag directory. If it is missing, re-run python3 ingest.py from Step 2.

Help me debug my chat.py script - it's not starting correctly.

✔️ Awesome, I've got everything!

Great. Double check you have saved chat.py and confirmed the You: prompt appears.

ⓧ I'd like to double check the full code

Here is the complete chat.py file. Compare yours against it:

from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

CHROMA_DIR = "./chroma_db"
COLLECTION_NAME = "research_papers"
EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "qwen2.5:3b"
RETRIEVAL_K = 4
MAX_HISTORY = 3


def format_docs(docs):
    """Format retrieved documents with source attribution."""
    formatted = []
    for doc in docs:
        filename = doc.metadata.get("filename", "Unknown")
        page = doc.metadata.get("page", "?")
        formatted.append(f"[Source: {filename}, Page {page}]\n{doc.page_content}")
    return "\n\n---\n\n".join(formatted)


def format_history(history):
    """Format conversation history for the prompt."""
    if not history:
        return "No previous conversation."
    lines = []
    for question, answer in history[-MAX_HISTORY:]:
        lines.append(f"User: {question}")
        lines.append(f"Assistant: {answer}")
    return "\n".join(lines)


def main():
    print("Loading vector store...")
    embeddings = OllamaEmbeddings(model=EMBED_MODEL)
    vector_store = Chroma(
        persist_directory=CHROMA_DIR,
        embedding_function=embeddings,
        collection_name=COLLECTION_NAME,
    )
    retriever = vector_store.as_retriever(search_kwargs={"k": RETRIEVAL_K})

    llm = ChatOllama(model=CHAT_MODEL, temperature=0.1, num_ctx=4096)

    template = """You are a research assistant. Answer the question based ONLY on the following context from research papers. If the context does not contain enough information to answer, say "I don't have enough information in the papers to answer this."

Always mention which source paper(s) your answer comes from.

Conversation History:
{history}

Context from Research Papers:
{context}

Question: {question}

Answer:"""

    prompt = ChatPromptTemplate.from_template(template)
    chain = prompt | llm | StrOutputParser()

    history = []

    print("\n" + "=" * 50)
    print("Research Paper Assistant")
    print("=" * 50)
    print("Ask questions about your papers. Type 'quit' to exit.\n")

    while True:
        question = input("You: ").strip()

        if question.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break

        if not question:
            continue

        print("\nSearching papers...")
        docs = retriever.invoke(question)
        context = format_docs(docs)
        history_text = format_history(history)

        print("Generating answer...\n")
        response = chain.invoke({
            "context": context,
            "history": history_text,
            "question": question,
        })

        print(f"Assistant: {response}\n")
        history.append((question, response))


if __name__ == "__main__":
    main()

Your research assistant is assembled and ready to answer questions. Next up, you will put it through its paces by testing factual retrieval, follow-up memory, and hallucination resistance.

Test Your Research Assistant

Your RAG pipeline is fully assembled. You have an ingestion script that processed 97 PDFs into a vector store, and a chat script that retrieves relevant passages and generates answers.

Now it's time to verify that the system actually works as intended. A good RAG assistant needs to pass three tests: it answers factual questions with correct citations, it remembers context across follow-up questions, and it refuses to make things up when the answer isn't in your papers.

In this step, get ready to:
  • Ask a factual question and verify source citations.
  • Test conversation memory with follow-up questions.
  • Test grounding by asking something not in your papers.
Ask a factual question

The best way to verify your assistant is to ask it something you already know the answer to from one of your papers. This lets you check both the accuracy of the response and whether the citation points to the right source.

  • Start the chat interface by running this command:
python3 chat.py

You should see the assistant print "Loading vector store..." followed by the "Research Paper Assistant" banner and a You: prompt.

  • Think of a fact, definition, or finding you remember from one specific paper in your collection.
  • Type your question at the You: prompt and press Enter.

The assistant will print "Searching papers..." while it retrieves relevant chunks from your vector store, then "Generating answer..." while Qwen produces the response.

  • Check the response for two things: the answer content matches what you know from the paper, and the citation mentions the correct filename and page number.

What should a good response look like?

A well-grounded response will reference specific papers by name in its answer text. The citations come from the metadata your ingestion script attached to each chunk (the filename and page number stored in ChromaDB).

If the answer feels vague or doesn't cite sources, the retriever may have pulled less relevant chunks. Try rephrasing your question with more specific keywords that appear in the paper.

Getting a slow or empty response?

If the response takes more than 30 seconds, Ollama may be loading the Qwen model into memory for the first time. The first query is always slower.

If you get an empty response or a connection error, make sure Ollama is running in the background. Open a new terminal and run ollama list to confirm it is active.

Help me debug my research assistant

Test conversation memory with follow-ups

A real research conversation doesn't stop at one question. You often want to drill deeper into an answer or ask the assistant to compare what different papers say. Your chat script stores the last 3 exchanges in its history buffer, so it should understand follow-up questions that reference previous answers.

  • Without exiting the chat, type a follow-up question that references the previous answer.
  • Try something like "Can you elaborate on that?" or "What does another paper say about this topic?" or "How does that compare to other approaches?"
  • Press Enter and wait for the response.

The assistant should understand what "that" or "this topic" refers to based on the conversation history. Its response should build on the previous exchange rather than starting from scratch.

How does conversation memory work here?

The format_history() function takes the last 3 question-answer pairs from the history list and includes them in the prompt sent to Qwen. This gives the model enough context to resolve pronouns like "that" or "it" from earlier in the conversation.

The limit of 3 exchanges keeps the prompt within the 4096-token context window while still enabling natural multi-turn conversation.

Follow-up answer ignoring context?

If the assistant treats your follow-up as a brand new question, it may not have enough context from the retrieved chunks to connect the two exchanges. Try a more explicit follow-up that repeats a keyword from the first answer.

With the qwen2.5:3b model, very short follow-ups like "Why?" may not carry enough signal. A slightly longer question like "Why is that approach recommended in the paper?" works better.

Help me improve my assistant's follow-up handling

Test grounding by asking something not in your papers

The most important safety check for a RAG system is making sure it doesn't hallucinate. Your system prompt explicitly tells the model to say "I don't have enough information in the papers to answer this" when the retrieved context doesn't cover the question.

  • Ask a question about a topic you know is NOT covered by any of your 97 papers.
  • Try something completely unrelated to your research area, like a question about cooking, sports, or a different academic field.
  • Press Enter and observe the response.

The assistant should respond with something along the lines of "I don't have enough information in the papers" rather than generating a plausible-sounding but unsupported answer.

Why is grounding so important?

Without grounding, a language model will confidently generate answers from its pre-training data. This is dangerous in a research context because you can't distinguish between a fact from your papers and something the model made up.

The system prompt in your chat.py acts as a guardrail. It tells the model to treat the retrieved context as the only valid source of information. This is what makes RAG trustworthy compared to a plain chatbot.

  • Type quit to exit the chat interface when you're done testing.

Assistant still generating answers for unrelated topics?

Smaller models like qwen2.5:3b sometimes follow the retrieved context too aggressively, finding tenuous connections between your papers and unrelated questions. If this happens, try an even more off-topic question (like asking about a recipe or a sports score).

If the model consistently ignores the grounding instruction, double check that the template variable in chat.py includes the exact phrase: "Answer the question based ONLY on the following context from research papers."

Help me fix my RAG grounding prompt

Secret mission

Add a Search Mode

Sometimes you don't need an AI-generated answer. You just want to know which papers discuss a topic and where. In this secret mission, you'll add a /search command that bypasses the LLM entirely and returns a ranked reading list of the most relevant passages across all 97 papers.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The research-rag/ project folder (contains ingest.py, chat.py, and your papers/ folder).
  • The chroma_db/ vector store directory (persistent ChromaDB database).
  • The Python virtual environment (venv/ folder inside your project).
  • Ollama models: nomic-embed-text (274MB) and qwen2.5:3b (~2.4GB).

Keep everything running

No action needed. Choose this if you're still actively exploring your research papers or want to keep asking questions.

  • Your chroma_db/ vector store persists on disk. You can re-run chat.py anytime without re-ingesting.
  • The Ollama models remain downloaded and ready to use for other projects too.
  • Your virtual environment stays inside the research-rag/ folder and only activates when you explicitly run source venv/bin/activate.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and database stay intact for next time.

  • If your chat script is still running, type quit to exit it.
  • Deactivate your Python virtual environment by running:
deactivate
  • Everything else (your project folder, vector store, and Ollama models) stays on disk and costs nothing while idle.
  • When you come back, navigate to research-rag/, run source venv/bin/activate, and start chat.py again.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is permanent.

  • If your virtual environment is still active, deactivate it first by running:
deactivate
  • Delete the entire project folder (this removes your scripts, virtual environment, and ChromaDB data) by running:
cd ~/Desktop
rm -rf research-rag

What about your papers?

If your papers/ folder inside research-rag/ was a copy of your PDFs, deleting the project folder removes that copy. If you symlinked or moved your original PDFs into the project, make sure you move them back to a safe location before deleting.

  • Remove the Ollama embedding model by running:
ollama rm nomic-embed-text
  • If you no longer need the Qwen chat model for other projects, remove it too by running:
ollama rm qwen2.5:3b
  • Confirm both models are removed by running:
ollama list

You should no longer see nomic-embed-text or qwen2.5:3b in the output.

Nice Work!

Nice Work!

Nice work! You just built a fully local, private research assistant that can search through 97 PDFs and answer your questions with specific citations.

Here's what you learned:

  • Built a complete local RAG pipeline that ingests 97 research PDFs with table support using PyMuPDF4LLM.
  • Created a persistent vector database using ChromaDB that survives restarts and only needs to be built once.
  • Built a conversational chat interface with conversation memory and source attribution using Qwen 2.5 and LangChain.
  • Secret Mission: Added a search mode for finding all relevant passages across papers without LLM generation.

Ready to quiz yourself?