Local RAG Chatbot with LlamaIndex

Build a fully local RAG chatbot that answers questions from your own documents.

Introduction

30 Second Summary

You have probably asked an AI chatbot a question about your own documents and gotten a vague, generic answer. That is because the AI has no access to your actual files.

In this project, you will build a fully local RAG chatbot that reads your documents, converts them into searchable embeddings, and answers questions grounded in your own data.

What You'll Build

You will have a conversational chatbot running entirely on your machine that you can ask about your documents and get accurate, sourced answers back in seconds.

By the end of this project, you'll have:

A working RAG chatbot where you ask a question about your documents and get a specific, grounded answer pulled directly from your files.
A conversational memory system where you ask follow-up questions like "tell me more about that" and the chatbot understands the context from your previous messages.
A persistent vector database that saves your document embeddings to disk, so you can close the program, reopen it later, and pick up instantly without re-processing.
Secret Mission: Add metadata filtering so you can scope queries to specific document categories and get more precise answers.

Are there any prerequisites?

You should have basic Python familiarity and some experience working with LLMs or APIs. You do not need any prior experience with LlamaIndex or RAG.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Ollama and the Python Environment

Before you can build a RAG chatbot, you need two things: a local LLM that can generate answers, and the Python libraries that orchestrate the retrieval pipeline.

In this step, you will install Ollama as your local model runtime, pull the two models your chatbot needs, and set up a Python virtual environment with all the LlamaIndex packages.

In this step, get ready to:

Install Ollama and pull the generation and embedding models.
Create a Python virtual environment and install all required packages.
Verify everything is wired up correctly.

Install Ollama and pull the models

Ollama is a lightweight runtime that lets you run large language models locally. Without it, LlamaIndex has no model to call for embeddings or text generation.

Install Ollama by opening a terminal and running the command for your operating system:

macOS

Open your terminal and install Ollama by running:

curl -fsSL https://ollama.com/install.sh | sh

This downloads and runs the official Ollama installer. It requires macOS 14 Sonoma or later.

Windows

Open PowerShell and install Ollama by running:

irm https://ollama.com/install.ps1 | iex

This downloads and runs the official Ollama installer for Windows. It requires Windows 10 22H2 or newer.

What does this command do?

The install script downloads the Ollama binary and sets it up as a background service on your machine. Once installed, Ollama serves a local API at http://localhost:11434 that LlamaIndex will connect to.

Confirm Ollama installed successfully by running:

ollama --version

You should see a version number printed. If you see a version, Ollama is ready.

Command not found?

On Windows, close and reopen PowerShell after the installer finishes. On macOS, you may need to open a new terminal window for the PATH to update.

help me troubleshoot Ollama not being recognized after installation

Now you need to download the two models your chatbot will use. The first is the language model that generates answers. The second is the embedding model that converts text into vectors.

Pull the generation model (~2GB download) by running:

ollama pull llama3.2:3b

This downloads the Llama 3.2 3-billion parameter model. It will take a few minutes depending on your internet speed.

Pull the embedding model (~274MB download) by running:

ollama pull nomic-embed-text

Confirm both models are available by running:

ollama list

You should see both llama3.2:3b and nomic-embed-text in the output table.

Why two models?

RAG pipelines need two separate AI capabilities. The embedding model (nomic-embed-text) converts your documents into numerical vectors for search. The generation model (llama3.2:3b) reads the retrieved chunks and writes a human-readable answer.

Pull seems stuck or fails?

Make sure Ollama is running in the background. On Windows, check your system tray for the Ollama icon. On macOS, check the menu bar. If neither appears, run ollama serve in a separate terminal window.

help me fix Ollama model pull failing or hanging

Create a Python virtual environment and install packages

A virtual environment keeps your project's packages isolated from other Python projects on your machine. This prevents version conflicts between different projects.

In your terminal, navigate to your project folder and create a virtual environment by running:

python -m venv venv

Activate the virtual environment:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

Your terminal prompt should now show (venv) at the beginning, confirming the environment is active.

Install all the required packages by running:

pip install llama-index-core llama-index-llms-ollama llama-index-embeddings-ollama llama-index-vector-stores-chroma llama-index-readers-file chromadb

What are these packages?

llama-index-core is the main LlamaIndex engine that orchestrates the RAG pipeline.
llama-index-llms-ollama and llama-index-embeddings-ollama connect LlamaIndex to your local Ollama models.
llama-index-vector-stores-chroma integrates ChromaDB as the vector database for storing embeddings.
llama-index-readers-file lets LlamaIndex read documents from local files.
chromadb is the vector database itself that stores and searches your document embeddings.

The install may take a minute or two. You will see progress bars as each package downloads.

Seeing a permissions error or install failure?

Make sure you see (venv) in your terminal prompt. If not, the virtual environment is not active. Re-run the activate command above. Also confirm you are using Python 3.10 or higher, as LlamaIndex requires it.

help me fix pip install errors in my virtual environment

Verify everything works together

Before you write any chatbot code, confirm that Ollama and all Python packages are correctly installed and can talk to each other.

First, confirm both models are still available by running:

ollama list

You should see llama3.2:3b and nomic-embed-text listed.

Before you run the next command, what do you think will happen if LlamaIndex can successfully import its Ollama integration?
Test that Python can import the LlamaIndex packages by running:

python -c "from llama_index.llms.ollama import Ollama; print('Ready!')"

You should see Ready! printed to the terminal. This confirms that all packages are installed and LlamaIndex can find the Ollama integration.

Seeing a ModuleNotFoundError?

This means a package did not install correctly. Make sure your virtual environment is active (you should see (venv) in your prompt), then re-run the pip install command from the previous substep.

help me fix ModuleNotFoundError for llama_index packages

Your local AI stack is ready. You have a generation model, an embedding model, and all the Python libraries wired up. Next up, you will load documents and build your first RAG query.

Load Documents and Build Your First RAG Query

Your environment is ready with Ollama running and all the Python packages installed. Now it's time to do the thing that makes RAG special: grounding an LLM in YOUR data.

Without RAG, an LLM can only answer from its training data. It has no idea what your company's products are, what your refund policy says, or what happened at last week's meeting. In this step, you'll load real documents into a searchable index and ask your first grounded question.

In this step, get ready to:

Create sample documents that give the chatbot a knowledge base to search.
Write a RAG pipeline that loads, embeds, and queries your documents.
Run your first grounded query and see a specific answer pulled from your files.

Create sample documents

A RAG chatbot needs documents to search through. You'll create three text files about a fictional company called TechFlow Solutions, covering its overview, products, and support policies.

Create a data folder inside your project directory by running this command:

mkdir data

Confirm the folder exists by running:

ls

You should see data listed in the output.

In VS Code, right-click the data folder in the Explorer sidebar and select New File.
Name it company_overview.txt and paste the following content:

TechFlow Solutions - Company Overview

TechFlow Solutions is a mid-size software company founded in 2019, headquartered in Austin, Texas. We specialize in building workflow automation tools for small and medium businesses.

Our mission is to eliminate repetitive manual work so teams can focus on creative problem-solving. We currently serve over 2,000 companies across 15 countries.

TechFlow was founded by Maria Chen and David Park, who previously worked together at a large enterprise software company. They saw that small businesses needed the same automation capabilities as large enterprises but couldn't afford the complexity or cost.

The company has grown from 5 employees to 120 in four years. We raised a Series B round of $25 million in 2022 and became profitable in 2023.

Our core values are: simplicity first, customer obsession, and transparent pricing. We believe software should be easy to use without reading a manual.

Right-click the data folder again and create a second file named product_features.txt with this content:

TechFlow Product Guide

TechFlow offers three main products:

1. FlowBuilder - A visual workflow automation tool that lets users create multi-step automations using a drag-and-drop interface. No coding required. Supports triggers from email, calendar, CRM, and 50+ integrations. Pricing starts at $29/month per user.

2. FlowAnalytics - A dashboard and reporting tool that tracks workflow performance, bottlenecks, and time savings. Includes pre-built templates for common metrics like "time saved per week" and "error rate reduction." Included free with FlowBuilder.

3. FlowAPI - A developer-focused product that exposes TechFlow's automation engine as a REST API. Teams can embed automation capabilities directly into their own applications. Pricing is usage-based starting at $0.01 per workflow execution.

Key features across all products:
- Real-time collaboration (multiple users editing simultaneously)
- Version history with one-click rollback
- SOC 2 Type II certified security
- 99.9% uptime SLA on all paid plans
- 24/7 email support, live chat during business hours (9am-6pm CT)

Create a third file named support_policy.txt in the data folder with this content:

TechFlow Support and Refund Policy

Support Channels:
- Email support: support@techflow.io (response within 4 business hours)
- Live chat: Available Monday-Friday, 9am-6pm Central Time
- Knowledge base: docs.techflow.io (self-service, 200+ articles)
- Community forum: community.techflow.io

Refund Policy:
All TechFlow products come with a 30-day money-back guarantee. If you are not satisfied within the first 30 days of your subscription, contact support for a full refund. No questions asked.

After 30 days, refunds are handled on a case-by-case basis. Annual subscriptions can be refunded pro-rata within the first 90 days.

Cancellation:
You can cancel your subscription at any time from your account settings. Your access continues until the end of your current billing period. We do not charge cancellation fees.

Data Export:
Upon cancellation, you have 60 days to export your workflows and data. After 60 days, all data is permanently deleted per our data retention policy.

Escalation:
If you are not satisfied with the support you receive, ask to escalate to a Team Lead. Response time for escalations is within 2 business hours.

Save all three files.
Confirm all documents are in place by running:

ls data/

You should see company_overview.txt, product_features.txt, and support_policy.txt listed.

Don't see all three files?

Make sure you created each file inside the data folder, not in the project root. Check the Explorer sidebar in VS Code to verify each file is nested under data/.

help me figure out why my data files aren't showing up

Write the RAG pipeline script

Now you'll write the Python script that ties everything together: configuring Ollama as both the LLM and embedding model, loading your documents, building a vector index, and querying it.

In VS Code, create a new file in your project root called rag_chatbot.py.
Add the imports and model configuration by pasting this code:

import sys
import shutil
import chromadb
from llama_index.core import SimpleDirectoryReader, StorageContext, VectorStoreIndex, Settings
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama
from llama_index.vector_stores.chroma import ChromaVectorStore


def main():
    # Configure the local LLM
    llm = Ollama(
        model="llama3.2:3b",
        base_url="http://localhost:11434",
        temperature=0.0,
        request_timeout=120.0,
        context_window=4096,
    )

    # Configure the local embedding model
    embed_model = OllamaEmbedding(
        model_name="nomic-embed-text",
        base_url="http://localhost:11434",
    )

    # Set global defaults so all LlamaIndex components use these
    Settings.llm = llm
    Settings.embed_model = embed_model

What does this code do?

The imports pull in LlamaIndex components for reading files, building indexes, and connecting to Ollama.
The Ollama() call configures your local LLM (llama3.2:3b) for generating answers. The temperature=0.0 setting makes responses deterministic and factual.
The OllamaEmbedding() call configures the embedding model (nomic-embed-text) that converts text into vector numbers for similarity search.
The Settings object registers both models globally so every LlamaIndex component knows which models to use.

Now add the document loading, indexing, and query logic below the Settings lines:

    # Load all text files from the data folder
    documents = SimpleDirectoryReader("data").load_data()
    index = VectorStoreIndex.from_documents(documents)

    # Query the index with a test question
    query_engine = index.as_query_engine()
    response = query_engine.query("What products does TechFlow offer?")
    print(response)


if __name__ == "__main__":
    main()

What does this code do?

SimpleDirectoryReader("data").load_data() reads every file in your data/ folder and converts them into Document objects that LlamaIndex can process.
VectorStoreIndex.from_documents() chunks each document into smaller pieces, embeds them into vectors using nomic-embed-text, and stores them in a searchable index.
as_query_engine() creates a query engine that retrieves the most relevant chunks for a question, then feeds them to the LLM as context for generating an answer.

Save rag_chatbot.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved rag_chatbot.py and move on to running it.

ⓧ I'd like to double check the full code

Your rag_chatbot.py should contain these sections in order:

Imports (sys, shutil, chromadb, and four llama_index packages).
A main() function that creates an Ollama LLM and an OllamaEmbedding model, then sets both on Settings.
Document loading with SimpleDirectoryReader("data").load_data() followed by VectorStoreIndex.from_documents(documents).
A query engine that asks "What products does TechFlow offer?" and prints the response.
An if __name__ == "__main__": block that calls main().

Scroll up and confirm each section is present in your file.

Run your first RAG query

Everything is wired up. Your script will read the three text files, chunk them, embed the chunks into vectors, find the most relevant pieces for your question, and feed them to the LLM.

Before you run the script, predict: will the answer mention specific TechFlow products by name, or will it give a generic response about software companies?
Run the script by entering this command:

python rag_chatbot.py

The first run takes 10-30 seconds because Ollama is embedding all your document chunks and then generating a response.

You should see a detailed answer mentioning TechFlow's specific products: FlowBuilder, FlowAnalytics, and FlowAPI, along with details like pricing and features pulled directly from your documents.

What just happened behind the scenes?

LlamaIndex performed the full RAG pipeline in one command. It split your documents into chunks, sent each chunk to nomic-embed-text to create vector embeddings, stored those vectors in memory, then found the chunks most similar to your question and passed them to llama3.2:3b as context for generating the answer.

The LLM never "memorized" TechFlow's products. It only knows about them because the relevant text was retrieved and injected into the prompt. That's the power of retrieval-augmented generation.

Seeing a connection error or timeout?

Make sure Ollama is still running. Check the Ollama icon in your system tray. If it's not there, restart Ollama from your Start menu.

If you see a timeout after 120 seconds, the model may still be loading into memory. Wait a moment and try running the script again.

help me fix my Ollama connection error when running rag_chatbot.py

Your chatbot just answered a question using only the knowledge in your documents. Next up, you'll discover what happens when you try to ask a follow-up question and hit the memory wall.

Add Conversational Memory with a Chat Engine

Your RAG chatbot answers standalone questions about TechFlow's documents. Real conversations involve follow-ups where each question builds on the previous answer.

Let's test how the current query engine handles multi-turn interaction, then upgrade to an approach that supports real conversation.

In this step, get ready to:

Test the query engine's handling of follow-up questions.
Switch to a chat engine with conversational memory.
Build an interactive chat loop for multi-turn conversation.

Test the query engine with a follow-up question

Your script currently asks one question and prints one answer. Let's add a second question that references the first answer and see what happens.

In rag_chatbot.py, find the print(response) line at the bottom of your main() function.
Add these lines directly below it:

    # Ask a follow-up that references the first answer
    response2 = query_engine.query("Tell me more about the pricing for the first one")
    print(response2)

What does this test?

The first query asks about TechFlow's products. The second query says "the first one" without naming anything specific. A human would know "the first one" means FlowBuilder from the previous answer. Let's see if the query engine makes that connection.

Save rag_chatbot.py.

Before you run this, do you think the second answer will make sense as a follow-up to the first? Will the query engine know what "the first one" refers to?

Run your script to see both answers by running this command:

python rag_chatbot.py

The first answer lists TechFlow's products correctly. The second answer is incoherent. It either gives a generic response, talks about something unrelated, or says it doesn't know what "the first one" means.

That's the expected result. The query engine treats every question as completely independent. It has zero memory of what came before.

Why does this happen?

A query engine is stateless. Each call to .query() performs a fresh retrieval against the vector index with no context about previous questions or answers.

For "Tell me more about the first one" to work, the engine would need to remember that "the first one" refers to FlowBuilder from the previous answer. That requires conversational memory.

Replace the query engine with a chat engine

LlamaIndex's chat engine solves this by maintaining conversation history. It reformulates each new question using past context before performing retrieval.

In rag_chatbot.py, find everything from query_engine = index.as_query_engine() down to the last print(response2) line.

    query_engine = index.as_query_engine()
    response = query_engine.query("What products does TechFlow offer?")
    print(response)
    # Ask a follow-up that references the first answer
    response2 = query_engine.query("Tell me more about the pricing for the first one")
    print(response2)

Delete all of those lines and replace them with the following code:

    # Create a conversational chat engine with context retrieval
    chat_engine = index.as_chat_engine(chat_mode="context")

    # Interactive chat loop
    print("\nRAG Chatbot ready! Ask questions about your documents.")
    print("Commands: 'quit' to exit, 'reset' to clear chat history.\n")

    while True:
        user_input = input("You: ").strip()
        if not user_input:
            continue
        if user_input.lower() in ("quit", "exit"):
            print("Goodbye!")
            break
        if user_input.lower() == "reset":
            chat_engine.reset()
            print("Chat history cleared.\n")
            continue

        response = chat_engine.chat(user_input)
        print(f"\nAssistant: {response}\n")

What does this code do?

as_chat_engine(chat_mode="context") creates a chat engine that retrieves relevant document chunks for each message while maintaining conversation history across turns.
The while True loop keeps the conversation going until the user types quit or exit.
chat_engine.reset() clears the conversation history so you can start fresh without restarting the script.
.chat() replaces .query(). It sends the message plus past conversation context to the LLM, so follow-up questions like "tell me more about that" resolve correctly.

Save rag_chatbot.py.

Seeing a syntax error after saving?

Make sure you deleted ALL the old query engine lines (including the response2 lines you added earlier). Leftover code will cause errors.
Check that the new code is indented with 4 spaces inside the main() function. Python is sensitive to indentation.
Confirm the escaped characters in the print statements match exactly: \n for newlines and f"..." for the f-string.

Why is my chat engine code raising an error?

✔️ Awesome, I've got everything!

Great. Make sure you've saved rag_chatbot.py before moving on.

ⓧ I'd like to double check the full code

Your rag_chatbot.py should contain these sections in order. Scroll up and confirm each is present:

Import block at the top (sys, shutil, chromadb, and the LlamaIndex packages).
def main(): with the Ollama LLM configuration (llama3.2:3b).
OllamaEmbedding configuration (nomic-embed-text).
Settings.llm and Settings.embed_model assignments.
SimpleDirectoryReader("data").load_data() and VectorStoreIndex.from_documents(documents).
chat_engine = index.as_chat_engine(chat_mode="context") (no more as_query_engine()).
The interactive while True loop with quit, exit, and reset commands.
if __name__ == "__main__": at the very bottom.

Test multi-turn conversation

Now let's prove the chat engine solves the memory problem you saw earlier.

Before you run this, what do you expect will happen when you ask "How long does that take?" after asking about the refund policy?

Run your chatbot by running this command:

python rag_chatbot.py

You should see the prompt:

RAG Chatbot ready! Ask questions about your documents.
Commands: 'quit' to exit, 'reset' to clear chat history.

You:

Type this as your first question:

What is TechFlow's refund policy?

The chatbot should respond with details about the 30-day money-back guarantee from your support policy document.

Now type this follow-up question:

How long does that take?

The chatbot correctly references the refund timeline from the first answer. It knows "that" refers to the refund process because the chat engine passes conversation history to the LLM along with the retrieved context.

Try the reset command, then ask "How long does that take?" again without any prior context.

After resetting, the chatbot can no longer resolve "that" because the conversation history is gone. This confirms the memory is working as expected.

Type quit to exit the chatbot.

Getting incoherent follow-up answers?

Make sure you're using as_chat_engine(chat_mode="context") not as_query_engine(). The query engine has no memory.
Check that you're calling .chat() not .query() on the chat engine. The method name matters.
If the chatbot takes a very long time (over 2 minutes), ensure Ollama is still running in the background. The model needs to be loaded in memory.

Why is my chat engine not remembering conversation context?

Your chatbot now holds a real conversation. It retrieves relevant documents and remembers what you discussed. There is one problem left: every time you run the script, it re-embeds all your documents from scratch. Next up, you'll add persistent storage so the index survives between sessions.

Persist Your Index with ChromaDB

Your chatbot works, but there is a hidden cost. Every time you run the script, LlamaIndex re-embeds all your documents from scratch. That takes 30+ seconds and wastes compute on documents that have not changed.

ChromaDB's PersistentClient solves this by saving embeddings to disk. Subsequent runs detect the existing data and skip straight to answering questions, loading in under 2 seconds.

In this step, get ready to:

Replace in-memory storage with ChromaDB's persistent storage.
Add a rebuild flag so you can re-index when documents change.
Test that your index survives a restart and loads instantly.

Switch from in-memory to persistent storage

Right now your script builds a fresh index every time it runs. You will replace those two lines with ChromaDB logic that checks whether embeddings already exist on disk.

In rag_chatbot.py, find these two lines inside main() (right after the Settings assignments):

    documents = SimpleDirectoryReader("data").load_data()
    index = VectorStoreIndex.from_documents(documents)

Delete those two lines and replace them with this code:

    # ChromaDB persistent storage setup
    PERSIST_DIR = "./chroma_db"
    COLLECTION_NAME = "my_documents"

    # Connect to ChromaDB with persistence
    chroma_client = chromadb.PersistentClient(path=PERSIST_DIR)
    chroma_collection = chroma_client.get_or_create_collection(COLLECTION_NAME)
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

    # Build or load the index
    if chroma_collection.count() == 0:
        print("Building new index from documents...")
        documents = SimpleDirectoryReader("data").load_data()
        storage_context = StorageContext.from_defaults(vector_store=vector_store)
        index = VectorStoreIndex.from_documents(
            documents,
            storage_context=storage_context,
            embed_model=embed_model,
        )
        print(f"Done! Indexed {len(documents)} documents ({chroma_collection.count()} chunks).")
    else:
        print(f"Found existing index ({chroma_collection.count()} chunks). Loading...")
        index = VectorStoreIndex.from_vector_store(
            vector_store,
            embed_model=embed_model,
        )

What does this code do?

This code creates a persistent pipeline that remembers your embeddings across runs.

chromadb.PersistentClient(path=PERSIST_DIR) stores all vector data in a chroma_db/ folder on disk instead of in memory.
get_or_create_collection() either opens an existing collection or creates a new one.
chroma_collection.count() == 0 detects whether embeddings already exist. If yes, it loads them directly. If no, it builds from scratch.
VectorStoreIndex.from_vector_store() loads the existing index without re-embedding anything.

Save rag_chatbot.py.
Run the script to confirm persistent storage works:

python rag_chatbot.py

You should see Building new index from documents... followed by a count of indexed documents and chunks. The chatbot then starts as normal.

Type quit to exit the chatbot.
Confirm the database folder was created by running:

ls chroma_db

You should see files inside the chroma_db/ folder. These are your persisted vector embeddings.

Seeing an error about ChromaDB?

Make sure Ollama is still running in the background. The embedding model needs to be accessible at http://localhost:11434.
Check that your data/ folder still contains the three text files from the previous step.
Verify that the two old lines (documents = SimpleDirectoryReader... and index = VectorStoreIndex.from_documents(documents)) are fully deleted, not just commented out.

Help me debug a ChromaDB persistence error in my LlamaIndex RAG chatbot.

Add a rebuild flag

Persistence is great until you add or change documents. You need a way to force a re-index when your source files change.

In rag_chatbot.py, find the very first line inside main(). It currently starts with:

    llm = Ollama(

Add these two lines above it (so they become the first thing inside main()):

    # Check for rebuild flag
    rebuild = "--rebuild" in sys.argv

Now scroll down to the PERSIST_DIR and COLLECTION_NAME lines you just added. Find this line:

    # Connect to ChromaDB with persistence

Add this block directly above that comment (between COLLECTION_NAME and the ChromaDB connection):

    # Handle rebuild flag
    if rebuild and shutil.os.path.exists(PERSIST_DIR):
        print("Rebuilding index: removing existing database...")
        shutil.rmtree(PERSIST_DIR)

What does this code do?

The --rebuild flag gives you manual control over re-indexing.

rebuild = "--rebuild" in sys.argv checks if you passed the flag when running the script.
shutil.rmtree(PERSIST_DIR) deletes the entire chroma_db/ folder, forcing a fresh build on the next line's count() == 0 check.

Save rag_chatbot.py.
Test the rebuild flag by running:

python rag_chatbot.py --rebuild

You should see Rebuilding index: removing existing database... followed by Building new index from documents... as it re-indexes everything.

Type quit to exit the chatbot.

Not seeing the rebuild message?

Make sure --rebuild is after the filename: python rag_chatbot.py --rebuild, not before it.
Check that the rebuild = "--rebuild" in sys.argv line is inside main() (indented with 4 spaces).

Help me fix my --rebuild flag in my RAG chatbot script.

Test persistence across restarts

The real payoff of persistence is speed. Your first run took 30+ seconds to embed everything. A second run should skip all that work entirely.

Before you run the next command, predict: will the script print "Building new index..." or "Loading existing index..." this time?
Run the script again without the rebuild flag:

python rag_chatbot.py

You should see Found existing index (X chunks). Loading... and the chatbot should be ready almost instantly. No 30-second wait.

Ask a question to confirm the loaded index still works:

Type: What products does TechFlow offer? and confirm you get a grounded answer mentioning FlowBuilder, FlowAnalytics, and FlowAPI.

Type quit to exit.

Why is loading so much faster?

When you built the index, LlamaIndex sent every document chunk to the nomic-embed-text model in Ollama, received 768-dimensional vectors back, and stored them in ChromaDB. That embedding work is the expensive part.

On the second run, chroma_collection.count() returns a non-zero number, so the script skips embedding entirely and just loads the pre-computed vectors from disk.

Still seeing "Building new index" on the second run?

Make sure you did NOT pass --rebuild in the command.
Check that chroma_db/ exists in your project folder by running ls. If it is missing, your PERSIST_DIR path might have a typo.
Confirm the else: branch uses VectorStoreIndex.from_vector_store(), not from_documents().

Help me figure out why my ChromaDB index isn't persisting between runs.

✔️ Awesome, I've got everything!

Great. Double check you have saved rag_chatbot.py and that the chroma_db/ folder exists in your project directory.

ⓧ I'd like to double check the full code

Your rag_chatbot.py should contain these sections in order. Scroll through your file and confirm each is present:

Imports (sys, shutil, chromadb, LlamaIndex modules).
def main(): starting with rebuild = "--rebuild" in sys.argv.
LLM and embedding model configuration (Ollama and OllamaEmbedding).
Settings.llm and Settings.embed_model assignments.
PERSIST_DIR and COLLECTION_NAME constants.
Rebuild flag handler (if rebuild and shutil.os.path.exists(PERSIST_DIR)).
ChromaDB connection (PersistentClient, get_or_create_collection, ChromaVectorStore).
Build-or-load logic (if chroma_collection.count() == 0 branch with from_documents, else branch with from_vector_store).
Chat engine and interactive loop (as_chat_engine, quit, reset commands).
if __name__ == "__main__": main() at the bottom.

Your chatbot now remembers its work across sessions. No more waiting 30 seconds every time you start it up. Next, you will take on the Secret Mission and add metadata filtering so you can scope queries to specific document categories.

Secret mission

Filter Queries by Document Category

Your chatbot searches all documents for every query. In a real knowledge base with hundreds of files, that means product questions pull in support policies and company history, diluting the answer quality. Add metadata filtering so you can scope queries to specific document categories and get sharper, more relevant results.

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

chroma_db/ folder containing your persistent vector embeddings (local files, no ongoing cost).
Ollama models: llama3.2:3b (~2GB) and nomic-embed-text (~274MB) stored in your home directory.
Python virtual environment and installed packages in your project folder.
data/ folder with your sample text documents.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your RAG chatbot or adding new documents.

Everything persists locally on your machine at no cost.
Ollama models remain available for other projects.
Your chroma_db/ index loads instantly next time you run the chatbot.

Pause - I'll come back to this later

Shut down running processes to free up RAM when you are not actively using the chatbot.

Quit the Ollama tray application by right-clicking its icon in your system tray and selecting Quit.
This frees the RAM that Ollama uses when idle. Your models and embeddings stay on disk.
To resume later, relaunch Ollama from your Start menu and run python rag_chatbot.py again.

Delete - I don't want to use this again

Remove all project resources and reclaim disk space. Follow each step below.

Remove the Ollama models by running these commands in your terminal:

ollama rm llama3.2:3b
ollama rm nomic-embed-text

This reclaims approximately 2.3GB of disk space.

Delete the chroma_db/ folder to remove your stored embeddings. Run this from inside your project folder:

rm -rf chroma_db

Delete the entire project folder from your Desktop:

cd ~/Desktop
rm -rf rag-chatbot

Uninstall Ollama from your machine.
Open Settings on your computer.
Select Apps, then Add or remove programs.
Find Ollama in the list and click Uninstall.
Follow the prompts to complete the removal.

Nice Work!

You just built a fully local RAG chatbot that answers questions grounded in your own documents. No API keys, no cloud costs, no data leaving your machine.

You've learned how to:

Build a retrieval-augmented generation pipeline using LlamaIndex, Ollama, and ChromaDB that runs entirely on your local machine.
Solve the stateless retrieval problem by switching from a query engine to a conversational chat engine that maintains context across follow-up questions.
Implement persistent vector storage so your document embeddings survive across sessions and load instantly on subsequent runs.
Secret Mission: Added metadata-based filtering to scope queries to specific document categories, making retrieval more precise and relevant.

Ready to quiz yourself?