Build a Local RAG Knowledge Base

Build a local RAG pipeline that answers questions from your own documents.

Introduction

30 Second Summary

Imagine you could ask a question about your own notes, documents, or research and get an instant, accurate answer with the exact source cited. That is what AI-powered knowledge bases do, and companies are hiring engineers to build them at record pace.

In this project, you will build a local RAG (Retrieval-Augmented Generation) knowledge base that ingests your documents, stores them in a vector database, and answers questions using only the information you provide.

What You'll Build

A Python-based knowledge base assistant that chunks text files, embeds them locally, and generates grounded answers with source citations.

By the end of this project, you'll have:

  • A document ingestion pipeline that chunks and embeds text files into a local ChromaDB vector store.
  • A semantic search system that finds the most relevant passages for any natural language question.
  • A grounded answer generator powered by a local LLM through Ollama that cites its sources.
  • Secret Mission: Add a relevance scoring system with confidence indicators that warns when the retrieved context may not contain the answer.

Are there any prerequisites?

You should be comfortable with Python basics like functions, loops, and file I/O. No prior experience with AI, embeddings, or vector databases is needed.

Install Ollama and Pull Models

Your RAG pipeline needs two AI capabilities: turning text into numerical vectors (called embeddings), and generating natural language answers from context.

Ollama is a tool that lets you run AI models locally on your own machine, completely free. In this step, you'll install Ollama and download two models that power your knowledge base.

In this step, get ready to:
  • Install Ollama on your machine.
  • Pull the embedding model (nomic-embed-text).
  • Pull the generation model (llama3.2:3b).
Download and install Ollama

Ollama runs AI models locally on your computer. It handles all the complexity of loading model weights and serving inference requests through a simple CLI.

  • Open your browser and go to https://ollama.com/download/windows.
  • Click the download button to get the OllamaSetup.exe installer.
  • Run the installer. It does not require Administrator rights and installs in your home directory by default.
  • Wait for the installation to complete. Ollama will start running in the background automatically.

What happens after installing?

After installing, Ollama runs as a background service on your machine. The ollama command becomes available in PowerShell, Command Prompt, or any terminal. It also serves a local API on http://localhost:11434.

Now verify that Ollama installed correctly.

  • In VS Code, open the integrated terminal by pressing Ctrl+` (backtick key, usually above Tab).
  • Check that Ollama is accessible by running this command:
ollama -v

✔️ I see a version number

You should see output like ollama version is X.X.X. Ollama is installed and running. Move on to pulling models.

ⓧ 'ollama' is not recognized

If your terminal does not recognize the ollama command, try these fixes:

  • Close and reopen your terminal (or restart VS Code) so it picks up the new PATH.
  • Make sure the Ollama installer finished completely. Re-run OllamaSetup.exe if needed.
  • If you installed Ollama to a custom directory, ensure that directory is in your system PATH.

After fixing, run ollama -v again to confirm you see a version number.

Pull the AI models

Your RAG pipeline uses two separate models. One creates embeddings (numerical representations of text) and the other generates human-readable answers.

  • Download the embedding model by running this command:
ollama pull nomic-embed-text

What is nomic-embed-text?

This is a 768-dimension embedding model. It converts text into numerical vectors that capture meaning. Similar sentences produce similar vectors, which is how your knowledge base will find relevant documents later.

The download is approximately 274 MB. You'll see a progress bar as it downloads.

  • Once that finishes, download the text generation model by running:
ollama pull llama3.2:3b

What is llama3.2:3b?

This is a 3-billion parameter LLM (Large Language Model) that generates text responses. The :3b tag means it's the 3-billion parameter variant, which is small enough to run well on a CPU.

The download is approximately 2 GB. This one takes a few minutes depending on your internet speed.

  • After both downloads complete, verify that both models are available by running:
ollama list
  • You should see both nomic-embed-text and llama3.2:3b listed in a table with their NAME, ID, SIZE, and MODIFIED columns.

Not seeing both models in the list?

  • Make sure each ollama pull command finished without errors. Look for a "success" message at the end of each download.
  • If a download was interrupted, re-run the same ollama pull command. It will resume from where it left off.
  • Ensure you have at least 4 GB of free disk space for both models combined.

Still stuck? Help me troubleshoot why my Ollama models aren't showing up in ollama list.

Both AI models are ready to go on your machine. Next up, you'll set up your Python project and install the packages that let your code talk to these models.

Set Up Your Python Project

Ollama is running with both models pulled and ready to go. Now you need a Python project that can talk to those models.

In this step, you'll create a dedicated project folder with its own isolated Python environment, install the two packages your RAG pipeline needs, and add some sample documents for the knowledge base to ingest later.

In this step, get ready to:
  • Create a project folder and Python virtual environment.
  • Install the chromadb and ollama Python packages.
  • Create sample documents for your knowledge base.
Create your project folder and virtual environment

Before creating the project, confirm that you have Python 3.10 or higher installed. The packages this project uses require at least that version.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.10 or higher

You're all set. Continue to the next instruction below.

ⓧ I see an older version

You need Python 3.10 or higher for this project. Download the latest version from python.org/downloads and run the installer.

On macOS, after installing a newer version, use python3.12 (or whichever version you installed) instead of python3 in subsequent commands. The system default python3 may still point to the older version.

On Windows, the installer updates the default python command automatically. Re-run the version check after installing to confirm.

ⓧ Command not found

Python is not installed on your system. Download it from python.org/downloads and run the installer.

On Windows, make sure to check the box that says Add Python to PATH during installation. After the installer finishes, close and reopen PowerShell, then re-run the version check.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-knowledge-base
cd rag-knowledge-base
  • Create a Python virtual environment inside your project folder by running:

macOS

python3 -m venv venv

Windows

python -m venv venv

What is a virtual environment?

A virtual environment is an isolated copy of Python that lives inside your project folder. Packages you install here stay separate from your system Python, so different projects can use different versions of the same library without conflicts.

  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate
  • Confirm the activation worked by checking your terminal prompt. You should see (venv) at the beginning of your command line.

Don't see (venv) in your prompt?

On Windows PowerShell, you may see a permissions error. Run this command first, then try the activate command again:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

On macOS, make sure you used source before the path. Without it, the command will fail silently.

Help me troubleshoot this

Install the Python dependencies

Your RAG pipeline needs two packages. ChromaDB is a vector database that stores and searches your document embeddings. The ollama Python package lets your code talk to the Ollama models you pulled in the previous step.

  • Install both packages by running this command:
pip install chromadb ollama

What does this command do?

pip install chromadb ollama downloads and installs both the ChromaDB vector database library and the Ollama Python client into your virtual environment. Because you activated the venv first, these packages are isolated to this project.

  • Wait for the installation to finish. This may take a minute or two as ChromaDB has several dependencies.
  • Verify the installation succeeded by running:
pip list | grep -i chromadb
  • You should see chromadb listed with a version number in the output.

Installation failed?

Make sure you see (venv) in your prompt before running pip install. If you closed and reopened the terminal, you need to activate the virtual environment again.

If you see a permissions error, do NOT use sudo. Instead, re-activate your venv and try the install again.

Help me debug my pip install error

Create your sample knowledge base documents

Your RAG pipeline needs documents to ingest. You'll create three text files about a topic of your choice.

  • Create the docs folder inside your project by running:
mkdir docs
  • Confirm the folder exists by running:
ls
  • You should see docs and venv listed in the output.

Now open VS Code so you can create the document files with a proper editor.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the rag-knowledge-base folder, and click Open.

You should see the docs and venv folders in the VS Code Explorer sidebar on the left.

  • In the Explorer sidebar, right-click the docs folder.
  • Click New File.
  • Name the file solar_energy.txt and press Enter.
  • Paste or type the following content into the file:
Solar energy is generated by converting sunlight into electricity using photovoltaic (PV) cells. These cells are made from semiconductor materials, most commonly silicon. When sunlight hits a PV cell, it knocks electrons loose from their atoms, creating an electrical current.

Modern solar panels typically convert 15-22% of sunlight into usable electricity. The efficiency depends on factors like panel angle, temperature, and cloud cover. Solar farms can generate electricity at utility scale, while rooftop panels serve individual homes and businesses.

Solar energy produces no greenhouse gas emissions during operation. The main environmental cost comes from manufacturing the panels, which is offset within 1-3 years of operation. Solar panels typically last 25-30 years with minimal maintenance.
  • Save the file by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Right-click the docs folder again and create a new file named wind_energy.txt.
  • Paste or type the following content:
Wind energy captures the kinetic energy of moving air using turbines. Modern wind turbines have three blades mounted on a tower, typically 80-120 meters tall. As wind turns the blades, a generator inside the nacelle converts the rotational energy into electricity.

Onshore wind farms are the most common installation type and can generate electricity at costs competitive with fossil fuels. Offshore wind farms, built in oceans or large lakes, benefit from stronger and more consistent winds but cost more to install and maintain.

Wind energy is one of the fastest-growing renewable sources worldwide. A single modern turbine can power approximately 1,500 homes per year. The main challenges include intermittency, noise concerns, and impact on bird populations.
  • Save the file.
  • Create one more file in the docs folder named battery_storage.txt.
  • Paste or type the following content:
Battery storage systems store excess electricity generated by renewable sources for use when generation is low. Lithium-ion batteries are the dominant technology for grid-scale storage due to their high energy density and declining costs.

Grid-scale battery installations can respond to demand changes in milliseconds, making them ideal for stabilizing power grids that rely on intermittent sources like solar and wind. A typical utility-scale battery can store enough energy to power thousands of homes for several hours.

The cost of lithium-ion batteries has fallen by approximately 90% since 2010. Emerging alternatives include solid-state batteries, flow batteries, and compressed air storage. Each technology offers different tradeoffs between cost, capacity, lifespan, and response time.
  • Save the file.

Use your own topic

The example files above use renewable energy, but you can write about anything you like. Pick a topic you find interesting and create three files covering different sub-topics.

Aim for at least 400 characters per file so the chunking algorithm has enough text to work with. The more distinct information each file covers, the better the RAG retrieval will demonstrate source attribution.

  • Confirm all three files appear in the Explorer sidebar under the docs folder.

Don't see your files in the docs folder?

Make sure you right-clicked the docs folder specifically, not the root rag-knowledge-base folder. If the files ended up in the wrong place, drag them into the docs folder in the sidebar.

Help me move my files to the right place

Your project workspace is ready with an isolated Python environment, the two packages you need, and sample documents waiting to be ingested. Next up, you'll write the ingestion pipeline that chunks these documents and stores them as vector embeddings in ChromaDB.

Build the Document Ingestion Pipeline

Your project environment is ready with all the packages installed and sample documents waiting in the docs/ folder. Now it's time to write the code that turns those raw text files into something a language model can search through.

The ingestion pipeline has three jobs: load your documents, split them into small overlapping chunks, and embed each chunk into a vector that ChromaDB can store and search. This is the core of any RAG system.

In this step, get ready to:
  • Write a document loader and text chunking function.
  • Write an embedding and indexing function using Ollama's nomic-embed-text model.
  • Store embedded chunks in a persistent ChromaDB collection.
Set up imports and load your documents

Every ingestion pipeline starts by reading files off disk. You need a function that scans the docs/ folder, reads each text file, and returns their contents in a structured format.

  • In VS Code, create a new file called ingest.py in your project root (the same folder that contains docs/).
  • Add the imports and configuration constants by pasting this code:
import os
import ollama
import chromadb

DOCS_DIR = "./docs"
CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "knowledge_base"
EMBED_MODEL = "nomic-embed-text"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50

What do these constants do?

  • DOCS_DIR tells the script where to find your text files.
  • CHROMA_PATH is where ChromaDB saves its database files on disk.
  • CHUNK_SIZE and CHUNK_OVERLAP control how documents get split. Each chunk is 500 characters, with 50 characters of overlap between neighbours so context isn't lost at boundaries.
  • Save ingest.py.

You won't see a visible change yet. The next chunk adds the function that actually reads your files.

  • Below the constants in ingest.py, add the document loading function:
def load_documents(docs_dir):
    """Load all .txt and .md files from the docs directory."""
    documents = []
    for filename in sorted(os.listdir(docs_dir)):
        if filename.endswith((".txt", ".md")):
            filepath = os.path.join(docs_dir, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read().strip()
            if text:
                documents.append({"filename": filename, "content": text})
    return documents

What does this function do?

load_documents() scans the folder you point it at, finds every .txt and .md file, reads the content, and returns a list of dictionaries. Each dictionary holds the filename (for source citation later) and the full text content.

The sorted() call ensures files are always processed in alphabetical order. This makes your ingestion results reproducible.

  • Save ingest.py.
  • Check that VS Code shows no red underlines in the file.

Seeing a red underline on the imports?

  • Make sure your virtual environment is selected as the Python interpreter in VS Code. Click the Python version in the bottom status bar and choose the interpreter inside your project's virtual environment folder.
  • Confirm chromadb and ollama are installed by checking for them in your terminal with pip list.

Help me fix import errors in my Python file

Write the text chunking function

Large language models have limited context windows. Feeding an entire document in one go wastes space and dilutes relevance. Chunking splits each document into smaller pieces so the retriever can pinpoint exactly the right passage for a question.

  • Below load_documents() in ingest.py, add the chunking function:
def chunk_text(text, chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

Why overlapping chunks?

Imagine a sentence that falls right at the boundary between two chunks. Without overlap, you'd lose the full context of that sentence in both chunks.

The 50-character overlap means each chunk shares a small window with its neighbour. This ensures no important sentence fragment is orphaned at a boundary.

  • Save ingest.py.
  • VS Code should still show no red underlines.

Seeing an indentation error?

  • Make sure the function body uses consistent indentation (4 spaces per level). Mixing tabs and spaces causes Python to reject the file.
  • Confirm there is a blank line separating load_documents() from chunk_text().

Help me fix an indentation error in my Python code

Build the embedding and storage function

The build_index() function ties everything together. It loads your documents, chunks them, converts each chunk into a numeric embedding using Ollama's nomic-embed-text model, and stores the results in a persistent vector database.

  • Below chunk_text() in ingest.py, add the build_index() function and the entry point:
def build_index():
    """Embed all documents and store in ChromaDB."""
    client = chromadb.PersistentClient(path=CHROMA_PATH)

    # Delete existing collection to rebuild fresh
    try:
        client.delete_collection(COLLECTION_NAME)
    except ValueError:
        pass

    collection = client.create_collection(
        name=COLLECTION_NAME,
        metadata={"hnsw:space": "cosine"},
    )

    documents = load_documents(DOCS_DIR)
    if not documents:
        print(f"No documents found in {DOCS_DIR}/")
        return

    all_ids = []
    all_embeddings = []
    all_documents = []
    all_metadatas = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        print(f"  {doc['filename']}: {len(chunks)} chunks")

        for i, chunk in enumerate(chunks):
            response = ollama.embed(model=EMBED_MODEL, input=chunk)
            embedding = response["embeddings"][0]

            chunk_id = f"{doc['filename']}::chunk{i}"
            all_ids.append(chunk_id)
            all_embeddings.append(embedding)
            all_documents.append(chunk)
            all_metadatas.append({
                "source": doc["filename"],
                "chunk_index": i,
            })

    collection.add(
        ids=all_ids,
        embeddings=all_embeddings,
        documents=all_documents,
        metadatas=all_metadatas,
    )

    print(f"\nIndexed {len(all_ids)} chunks into '{COLLECTION_NAME}'.")


if __name__ == "__main__":
    print("Ingesting documents...")
    build_index()
    print("Done!")

What does this function do?

  • chromadb.PersistentClient(path=CHROMA_PATH) creates a local database that saves everything to disk. Your embeddings survive between script runs.
  • The try/except ValueError block deletes any existing collection so you always get a clean rebuild. ChromaDB raises a ValueError if the collection doesn't exist yet, so we catch and ignore that.
  • create_collection() with "hnsw:space": "cosine" tells ChromaDB to measure similarity using cosine distance. This is the standard metric for text embeddings.
  • ollama.embed(model=EMBED_MODEL, input=chunk) sends each chunk to Ollama's local embedding model and returns a 768-dimension vector.
  • collection.add() stores all the IDs, embeddings, raw text, and metadata in one batch insert.

Why delete and recreate the collection each time?

This pattern gives you an idempotent ingestion script. Every time you run it, you get a clean collection that perfectly reflects the current state of your docs/ folder. If you edited or added documents, just rerun the script and the index stays in sync.

  • Save ingest.py.
  • Confirm the file has no red underlines in VS Code. You should see the full file with imports at the top, three functions in the middle, and the if __name__ block at the bottom.

Seeing errors in the file?

  • If you see a red underline on ollama.embed, your ollama package may not be installed in the active virtual environment. Run pip install ollama in your terminal.
  • If you see a syntax error near f" strings, make sure you're using Python 3.10 or higher. F-strings with nested quotes require Python 3.12+, but the f-strings in this file use simple expressions that work in 3.10+.
  • If you accidentally indented the if __name__ block inside build_index(), move it back to the leftmost column. It should be at the same indent level as the function definitions.

Help me fix errors in my ingest.py file

✔️ Awesome, I've got everything!

Great. Double check you've saved ingest.py and that VS Code shows no errors.

ⓧ I'd like to double check the full code

Here is the complete ingest.py file. Compare yours against this:

import os
import ollama
import chromadb

DOCS_DIR = "./docs"
CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "knowledge_base"
EMBED_MODEL = "nomic-embed-text"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50


def load_documents(docs_dir):
    """Load all .txt and .md files from the docs directory."""
    documents = []
    for filename in sorted(os.listdir(docs_dir)):
        if filename.endswith((".txt", ".md")):
            filepath = os.path.join(docs_dir, filename)
            with open(filepath, "r", encoding="utf-8") as f:
                text = f.read().strip()
            if text:
                documents.append({"filename": filename, "content": text})
    return documents


def chunk_text(text, chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks


def build_index():
    """Embed all documents and store in ChromaDB."""
    client = chromadb.PersistentClient(path=CHROMA_PATH)

    # Delete existing collection to rebuild fresh
    try:
        client.delete_collection(COLLECTION_NAME)
    except ValueError:
        pass

    collection = client.create_collection(
        name=COLLECTION_NAME,
        metadata={"hnsw:space": "cosine"},
    )

    documents = load_documents(DOCS_DIR)
    if not documents:
        print(f"No documents found in {DOCS_DIR}/")
        return

    all_ids = []
    all_embeddings = []
    all_documents = []
    all_metadatas = []

    for doc in documents:
        chunks = chunk_text(doc["content"])
        print(f"  {doc['filename']}: {len(chunks)} chunks")

        for i, chunk in enumerate(chunks):
            response = ollama.embed(model=EMBED_MODEL, input=chunk)
            embedding = response["embeddings"][0]

            chunk_id = f"{doc['filename']}::chunk{i}"
            all_ids.append(chunk_id)
            all_embeddings.append(embedding)
            all_documents.append(chunk)
            all_metadatas.append({
                "source": doc["filename"],
                "chunk_index": i,
            })

    collection.add(
        ids=all_ids,
        embeddings=all_embeddings,
        documents=all_documents,
        metadatas=all_metadatas,
    )

    print(f"\nIndexed {len(all_ids)} chunks into '{COLLECTION_NAME}'.")


if __name__ == "__main__":
    print("Ingesting documents...")
    build_index()
    print("Done!")

Your ingestion pipeline is fully written. Next up, you'll build the query system that searches these embedded chunks and generates answers with source citations.

Build the Query and Retrieval System

Your ingestion pipeline is ready to embed documents and store them in a vector database. Now you need a way to actually ask questions and get grounded answers back.

In this step, you'll build the retrieval side of RAG. This means embedding the user's question, finding the most relevant chunks from ChromaDB, and feeding those chunks to a local LLM with instructions to cite its sources.

In this step, get ready to:
  • Write a retrieval function that searches your vector database by meaning.
  • Build a system prompt that constrains the LLM to answer only from context.
  • Create an interactive query loop for asking questions.
Write the query function

The query function is the core of retrieval. It takes a natural language question, converts it into an embedding using the same model that embedded your documents, then searches ChromaDB for the closest matching chunks.

  • In VS Code, create a new file called query.py in your project root (the same folder where ingest.py lives).
  • Add the following imports and configuration constants at the top of query.py:
import ollama
import chromadb

CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "knowledge_base"
EMBED_MODEL = "nomic-embed-text"
LLM_MODEL = "llama3.2:3b"
TOP_K = 3

What do these constants do?

These mirror the same paths and model names you used in ingest.py. Both scripts need to agree on where the database lives (CHROMA_PATH) and which collection to use (COLLECTION_NAME).

TOP_K = 3 means each query retrieves the 3 most relevant chunks. This is enough context for a focused answer without overwhelming the LLM.

  • Below the constants, add the retrieve function:
def retrieve(question, collection, n_results=TOP_K):
    """Embed the question and retrieve relevant chunks from ChromaDB."""
    response = ollama.embed(model=EMBED_MODEL, input=question)
    query_embedding = response["embeddings"][0]

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
    )

    return results

What does this code do?

ollama.embed() converts the user's question into a vector using the same nomic-embed-text model that embedded your documents. This is critical. Questions and documents must share the same embedding space for similarity search to work.

collection.query() then finds the n_results chunks whose vectors are closest to the question vector. ChromaDB handles the math behind the scenes using cosine similarity.

  • Save query.py.

Seeing a red underline on ollama or chromadb?

Make sure your virtual environment is activated. If VS Code shows import errors, click the Python interpreter in the bottom status bar and select the one inside your project's venv folder.

Help me fix Python import errors in VS Code

Build a system prompt and generation function

Retrieval alone just gives you raw text chunks. You need an LLM to synthesize those chunks into a coherent answer. The system prompt is what keeps the LLM honest by forcing it to cite sources and admit when it doesn't know.

  • In query.py, find the line TOP_K = 3. Add the following system prompt below it (above the retrieve function):
SYSTEM_PROMPT = """You are a helpful assistant that answers questions using ONLY the provided context.

Rules:
- Answer based solely on the context below.
- For every claim, cite the source using [Source: filename].
- If the context does not contain enough information to answer, say exactly: "I don't have enough information in my documents to answer that question."
- Keep answers concise and factual.
"""

Why is this system prompt so strict?

Without constraints, LLMs will happily make up answers. This prompt forces two guardrails: the model must cite its sources with [Source: filename], and it must refuse to answer if the context doesn't contain relevant information.

These constraints are what make RAG trustworthy in production. The model becomes a "reader" of your documents, not a "guesser" from its training data.

  • Below the retrieve function, add the generate_answer function:
def generate_answer(question, results):
    """Generate an answer using retrieved context and the local LLM."""
    documents = results["documents"][0]
    metadatas = results["metadatas"][0]

    # Build context string with source labels
    context_parts = []
    for i, (doc, meta) in enumerate(zip(documents, metadatas)):
        context_parts.append(f"[{i+1}] Source: {meta['source']}\n{doc}")

    context = "\n\n---\n\n".join(context_parts)

    user_prompt = f"""Context:
{context}

Question: {question}"""

    response = ollama.chat(
        model=LLM_MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_prompt},
        ],
    )

    return response["message"]["content"]

What does this code do?

The function first unpacks the retrieved documents and their metadata (source filenames). It builds a numbered context string where each chunk is labeled with its source file.

Then it constructs a user prompt that includes the full context and the original question. Finally, ollama.chat() sends both the system prompt and user prompt to llama3.2:3b, which generates a grounded answer.

  • Save query.py.

Seeing a syntax error in the f-string?

Triple-quoted f-strings (f"""...""") are sensitive to indentation. Make sure the closing """ of user_prompt is on its own line with no extra spaces before it.

Help me fix a syntax error in my generate_answer function

Create an interactive loop for asking questions

The retrieval and generation functions are ready. Now you need a way to connect them into a conversational interface where you can keep asking questions until you type quit.

  • Below the generate_answer function in query.py, add the main function and entry point:
def main():
    """Interactive query loop."""
    client = chromadb.PersistentClient(path=CHROMA_PATH)

    try:
        collection = client.get_collection(COLLECTION_NAME)
    except ValueError:
        print("No knowledge base found. Run ingest.py first!")
        return

    print("RAG Knowledge Base Ready!")
    print(f"Collection has {collection.count()} chunks indexed.")
    print("Type 'quit' to exit.\n")

    while True:
        question = input("Ask a question: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            break
        if not question:
            continue

        print("\nSearching knowledge base...")
        results = retrieve(question, collection)

        print("Generating answer...\n")
        answer = generate_answer(question, results)

        print(f"Answer: {answer}")
        print("\n" + "=" * 60 + "\n")


if __name__ == "__main__":
    main()

What does this code do?

main() first connects to the existing ChromaDB database and loads the knowledge_base collection. If the collection doesn't exist (because you haven't run ingest.py yet), it prints a helpful error and exits.

The while True loop continuously asks for input, retrieves relevant chunks, generates an answer, and prints it. Typing quit, exit, or q breaks out of the loop.

  • Save query.py.
  • You won't run this file yet. In the next step, you'll run ingest.py first to populate the database, then use query.py to ask questions.

Not sure if your code is correct?

Double check your full file against the reference below. Common mistakes include mismatched indentation, missing commas in the messages list, or forgetting the if __name__ block at the bottom.

Help me check my query.py for errors

✔️ Awesome, I've got everything!

Great. Double check you've saved query.py and move on to the next step.

ⓧ I'd like to double check the full code

Here's the complete query.py file. Compare yours against this:

import ollama
import chromadb

CHROMA_PATH = "./chroma_db"
COLLECTION_NAME = "knowledge_base"
EMBED_MODEL = "nomic-embed-text"
LLM_MODEL = "llama3.2:3b"
TOP_K = 3

SYSTEM_PROMPT = """You are a helpful assistant that answers questions using ONLY the provided context.

Rules:
- Answer based solely on the context below.
- For every claim, cite the source using [Source: filename].
- If the context does not contain enough information to answer, say exactly: "I don't have enough information in my documents to answer that question."
- Keep answers concise and factual.
"""


def retrieve(question, collection, n_results=TOP_K):
    """Embed the question and retrieve relevant chunks from ChromaDB."""
    response = ollama.embed(model=EMBED_MODEL, input=question)
    query_embedding = response["embeddings"][0]

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
    )

    return results


def generate_answer(question, results):
    """Generate an answer using retrieved context and the local LLM."""
    documents = results["documents"][0]
    metadatas = results["metadatas"][0]

    # Build context string with source labels
    context_parts = []
    for i, (doc, meta) in enumerate(zip(documents, metadatas)):
        context_parts.append(f"[{i+1}] Source: {meta['source']}\n{doc}")

    context = "\n\n---\n\n".join(context_parts)

    user_prompt = f"""Context:
{context}

Question: {question}"""

    response = ollama.chat(
        model=LLM_MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_prompt},
        ],
    )

    return response["message"]["content"]


def main():
    """Interactive query loop."""
    client = chromadb.PersistentClient(path=CHROMA_PATH)

    try:
        collection = client.get_collection(COLLECTION_NAME)
    except ValueError:
        print("No knowledge base found. Run ingest.py first!")
        return

    print("RAG Knowledge Base Ready!")
    print(f"Collection has {collection.count()} chunks indexed.")
    print("Type 'quit' to exit.\n")

    while True:
        question = input("Ask a question: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            break
        if not question:
            continue

        print("\nSearching knowledge base...")
        results = retrieve(question, collection)

        print("Generating answer...\n")
        answer = generate_answer(question, results)

        print(f"Answer: {answer}")
        print("\n" + "=" * 60 + "\n")


if __name__ == "__main__":
    main()

Your query system is complete. In the next step, you'll run the full pipeline end-to-end: ingest your documents, ask questions, and verify the answers are grounded in your text files.

Test Your RAG Pipeline End-to-End

Your ingestion and query scripts are complete. Now it's time to run them together and see the full RAG pipeline in action.

You'll feed your documents into ChromaDB, ask questions to verify the answers come from your actual files, and then deliberately ask something your documents don't cover. That last test proves your guardrails are working.

In this step, get ready to:
  • Run document ingestion to build the vector store.
  • Query the knowledge base and verify grounded answers with source citations.
  • Test an out-of-scope question to confirm the guardrails work.
Run the ingestion script

The ingestion script reads every file in your docs/ folder, splits the text into chunks, embeds each chunk using nomic-embed-text, and stores everything in a persistent ChromaDB database.

  • Run the ingestion script by entering this command in your terminal:
python ingest.py

This will take a few seconds. Each document chunk gets sent to Ollama for embedding, so you'll see progress printed as each file is processed.

  • You should see output listing each filename with its chunk count, a total count of indexed chunks, and Done! at the end.
  • Confirm the database was created by running:
ls

You should see a chroma_db folder in the output alongside ingest.py, query.py, and docs. That folder holds your persistent vector store.

Seeing an error?

If you see a connection error mentioning localhost:11434, Ollama is not running. Open a new terminal and run ollama list to check. If Ollama is not started, launch it from your system tray or start menu.

If you see No documents found in ./docs/, make sure your docs/ folder contains at least one .txt or .md file with text content.

help me debug my ingest.py script

Ask a question grounded in your documents

Now that your documents are embedded and stored, you can query them. The query script embeds your question, searches ChromaDB for the most relevant chunks, and passes those chunks to the LLM as context.

  • Start the interactive query loop by running:
python query.py
  • You should see RAG Knowledge Base Ready! followed by the number of chunks indexed.
  • At the Ask a question: prompt, type a question that your documents should be able to answer. Pick something specific to the content you wrote in your docs/ files.

What should I see?

After a few seconds of processing, the model prints an answer that draws directly from your document content. Each claim in the answer includes a [Source: filename] citation pointing back to the original file.

The generation step may take 10-30 seconds depending on your CPU speed. This is normal for local inference without a GPU.

Answer doesn't cite sources?

Local LLMs don't always follow instructions perfectly. If your answer is missing [Source: filename] tags, try asking a more specific question that targets content clearly present in one of your files. The model should cite sources more reliably with direct factual questions.

help me get source citations working in my RAG answers

Test with an out-of-scope question

A well-built RAG system knows when it doesn't have the answer. Your system prompt instructs the model to refuse gracefully when the retrieved context doesn't cover the question.

  • At the Ask a question: prompt, type a question completely unrelated to your documents. For example, if your documents are about cooking, ask about quantum physics.

The model should respond with: I don't have enough information in my documents to answer that question. This proves the model is constrained to your document context and won't hallucinate answers.

Why does this matter?

In production RAG systems, hallucination is the biggest risk. If the model invents answers instead of admitting ignorance, users lose trust. Testing the refusal case is just as important as testing the success case.

The secret mission builds on this concept by adding confidence scores, so you can quantify how confident the system is in its answer.

  • Type quit and press Enter to exit the interactive loop.

Model answers the out-of-scope question anyway?

Smaller models sometimes ignore system prompt instructions. Try making your out-of-scope question more obviously unrelated to your documents. If the model still answers, it means the retrieved chunks happened to contain a few words that the model latched onto.

This is a known limitation of small local models. The system prompt instructs the refusal behavior, but llama3.2:3b won't follow it 100% of the time. A larger model or a stricter prompt would improve reliability.

help me make the model refuse out-of-scope questions

Your RAG pipeline is fully operational. You've proven it can retrieve relevant context, generate grounded answers with citations, and refuse to answer when it doesn't have the information. Next up, the Secret Mission takes this further by adding confidence scoring so you can see exactly how certain the system is about each answer.

Secret mission

Add Relevance Scoring and Confidence Indicators

Your RAG pipeline answers questions, but how do you know when it's guessing? Real production systems need evaluation awareness. In this secret mission, you'll add confidence scoring that tells you exactly how relevant your retrieved sources are and warns you when the knowledge base might not have the answer.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama models (nomic-embed-text and llama3.2:3b) stored locally (~2-4 GB disk space).
  • Project folder containing ingest.py, query.py, docs/, chroma_db/, and your Python virtual environment.

Keep everything running

No action needed. Choose this if you're still actively experimenting with your RAG knowledge base or planning to add more documents.

  • Ollama stays installed and running in the background. Your pulled models remain available for any future project.
  • The chroma_db/ database persists on disk. You can re-run query.py anytime without re-ingesting.
  • Your virtual environment and Python packages stay ready to use.

Pause - I'll come back to this later

Stop Ollama from running in the background to free up system resources. Your models and project files stay on disk.

  • Open Task Manager by pressing Ctrl+Shift+Esc.
  • Click the Startup apps tab.
  • Search for ollama in the list.
  • Click Disable to stop Ollama from launching at startup.

What does this do?

This prevents Ollama from auto-starting when you boot your computer. Your models and database stay on disk, so when you're ready to return, re-enable the startup entry and everything works again.

Delete - I don't want to use this again

Remove all project resources and reclaim your disk space. This is permanent.

  • Remove the Ollama models by running these commands in your terminal:
ollama rm nomic-embed-text
ollama rm llama3.2:3b
  • Delete your project folder. On macOS or Linux, run:
cd ~/Desktop
rm -rf rag-knowledge-base
  • On Windows, run this in PowerShell:
cd ~\Desktop
Remove-Item -Recurse -Force rag-knowledge-base

What about Ollama itself?

Removing the models frees the bulk of disk space (~2-4 GB). If you also want to uninstall the Ollama application, go to Settings > Apps > Installed apps, find Ollama, and click Uninstall.

Nice Work!

Nice Work!

You just built a production-grade RAG pipeline entirely from scratch, running 100% locally on your own machine. This is the exact architecture behind enterprise AI assistants, internal documentation tools, and customer support bots.

You've learned how to:

  • Build a complete document ingestion pipeline that chunks text, generates embeddings with Ollama, and stores vectors in a persistent ChromaDB collection.
  • Create a semantic search system that retrieves relevant document chunks based on meaning, not just keyword matching.
  • Implement grounded generation with source citations using a local LLM, ensuring answers stay faithful to your documents.
  • Added a relevance scoring system with visual confidence indicators that warns when retrieved context may not contain the answer.

Ready to quiz yourself?