Chat With Your Docs Using Local AI

Build a RAG pipeline that answers questions grounded in your own documents.

Introduction

30 Second Summary

Every AI chatbot that lets you "chat with your files" uses the same trick behind the scenes. It finds the most relevant pieces of your documents first, then feeds them to an AI model so it can answer your question with actual facts instead of guessing.

In this project, you will build that exact system from scratch. You'll create a RAG (Retrieval-Augmented Generation) pipeline that ingests documents, stores them as vectors, retrieves relevant context, and generates grounded answers using a local AI model.

What You'll Build

A Python-based "Chat With Your Docs" system that chunks documents, embeds them into a ChromaDB vector store, and answers natural-language questions using a local Llama 3.2 model running through Ollama.

By the end of this project, you'll have:

  • A document ingestion pipeline that loads, chunks, and embeds text files into a vector database.
  • A local LLM (Llama 3.2) running on your machine via Ollama with zero API keys or costs.
  • A RAG query function that retrieves relevant chunks using semantic similarity and generates answers grounded in your documents.
  • Secret Mission: Add a Cross-Encoder re-ranking step that improves retrieval precision for production-quality results.

Are there any prerequisites?

You should be comfortable with Python basics like functions, loops, and installing libraries. No prior experience with AI, machine learning, or vector databases is needed.

Set Up Your Project Environment

Before you can build a RAG system, you need a dedicated workspace with the right tools installed.

In this step, you'll create a project folder, set up a Python virtual environment, and install the four libraries that power your pipeline: ChromaDB for vector storage, sentence-transformers for embeddings, ollama for talking to your local LLM, and ipykernel for running Jupyter notebooks in VS Code.

In this step, get ready to:
  • Create a project folder and open it in VS Code.
  • Set up a Python virtual environment and install all dependencies.
  • Register a Jupyter kernel and create your notebook file.
Create a new project folder and open it in VS Code

Your RAG project needs its own dedicated folder to keep all the code, documents, and vector database files organized in one place.

  • Open your terminal in VS Code by pressing Ctrl+` (backtick key, usually above Tab).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-project
cd rag-project
  • Open this folder in VS Code by running:
code .

What does this command do?

The code . command tells VS Code to open the current directory as a workspace. The dot (.) means "this folder".

  • A new VS Code window opens with rag-project shown in the Explorer sidebar on the left.
  • If VS Code asks whether you trust the authors of this folder, click Yes, I trust the authors.
  • Confirm the folder exists by running:
ls

You should see an empty listing since the folder is brand new.

code command not found?

If your terminal says the code command is not recognized, open VS Code manually, then press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Shell Command: Install, and select the option to install it in your PATH.

Alternatively, open VS Code, click File, then Open Folder, and select the rag-project folder on your Desktop.

Need more help? Help me fix the code command in my terminal

Create a virtual environment and install dependencies

A virtual environment gives your project its own isolated set of Python packages. This prevents conflicts between projects that might need different library versions.

  • Open a new terminal in VS Code by pressing Ctrl+` (this terminal should already be inside your rag-project folder).
  • Create a virtual environment by running this command:
python -m venv venv

What does this command do?

This creates a folder called venv inside your project. It contains a copy of Python and a clean space for installing packages without affecting the rest of your system.

  • Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms you're working inside the virtual environment.

Don't see (venv) in your prompt?

On Windows, you may need to allow PowerShell to run scripts. Run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned and then try the activate command again.

On macOS, make sure you used source before the path. Without it, the activation won't stick in your current session.

Still stuck? Help me activate my Python virtual environment

  • Install the four project dependencies by running this command:
pip install chromadb sentence-transformers ollama ipykernel

What are these packages?

  • chromadb is a vector database that stores your document embeddings and handles similarity search.
  • sentence-transformers provides pre-trained models that convert text into numerical vectors (embeddings).
  • ollama is a Python client that talks to the Ollama app running your local LLM.
  • ipykernel lets VS Code run your virtual environment as a Jupyter notebook kernel.

This installation may take a few minutes. sentence-transformers in particular downloads several sub-dependencies including PyTorch.

When it finishes, you should see a success message like "Successfully installed chromadb-... sentence-transformers-... ollama-... ipykernel-..." at the bottom of the output.

Installation failed or timed out?

Make sure your (venv) is still active in your terminal prompt. If it disappeared, re-run the activate command before trying pip install again.

If you see a permissions error, make sure you're using the virtual environment's pip (not your system pip). You can verify with which pip (macOS) or where pip (Windows). It should point to a path inside your venv folder.

Need more help? Help me fix my pip install error

Register the kernel and create your notebook

VS Code needs to know about your virtual environment before it can run Jupyter notebooks with the packages you just installed. Registering a kernel creates that link.

  • Register your virtual environment as a Jupyter kernel by running:
python -m ipykernel install --user --name=rag-project --display-name "Python (rag-project)"

What does this command do?

This registers your venv as a kernel named Python (rag-project). When you open a notebook in VS Code, you'll select this kernel to run cells with all your installed packages available.

You should see output like "Installed kernelspec rag-project" confirming the registration succeeded.

  • Create your notebook file by running:

macOS

touch rag_pipeline.ipynb

Windows

type nul > rag_pipeline.ipynb
  • Confirm the file was created by running:
ls

You should see rag_pipeline.ipynb and the venv folder listed in the output.

  • Open the notebook by clicking rag_pipeline.ipynb in the VS Code Explorer sidebar on the left.
  • VS Code will open it as a Jupyter notebook. In the top right corner of the notebook, click Select Kernel.
  • Select Python Environments, then choose Python (rag-project) from the list.

Don't see Python (rag-project) in the kernel list?

Make sure you have the Jupyter extension installed in VS Code. Press Ctrl+Shift+X (Windows) or Cmd+Shift+X (macOS) to open Extensions, search for "Jupyter", and install it.

After installing the extension, close and reopen the notebook file. The kernel should now appear in the list.

Still stuck? Help me find my Jupyter kernel in VS Code

Your workspace is ready. Next up, you'll install Ollama and download the Llama 3.2 model that will power your RAG system's answers.

Install Ollama and Pull the Model

Your Python environment is ready with all the libraries installed. Now you need the actual LLM that will generate answers from your documents.

In this step, you will install Ollama, a tool that runs large language models locally on your machine. You will pull the Llama 3.2 model and verify it responds to prompts from Python.

In this step, get ready to:
  • Install Ollama on Windows and confirm it is running.
  • Download the Llama 3.2 3B model to your machine.
  • Test the model from Python to confirm the connection works.
Download and install Ollama

Ollama is a lightweight tool that downloads, manages, and serves LLMs on your local machine. It runs as a background service and exposes a local API that your Python code can talk to.

  • Open a new PowerShell terminal (not the VS Code terminal where your venv is active).
  • Install Ollama by running this command:
irm https://ollama.com/install.ps1 | iex

What does this command do?

This downloads and runs the official Ollama installer script for Windows. It installs Ollama into your user directory, so no Administrator rights are needed.

  • Wait for the install to complete. The script will download the binary and set up the background service.
  • Verify that Ollama installed successfully by running:
ollama -v

You should see output like ollama version is followed by a version number.

Not seeing a version number?

  • Close and reopen your PowerShell window. The installer adds Ollama to your PATH, but existing terminals may not pick up the change.
  • Make sure you have at least 4GB of free disk space. Ollama needs this for the binary and model storage.
  • Check that you are running Windows 10 22H2 or newer.

Help me troubleshoot the Ollama installation on Windows.

Pull the Llama 3.2 3B model

Ollama is installed, but it does not come with any models out of the box. You need to download one. Llama 3.2 3B is a compact model that runs comfortably on most machines.

  • Download the Llama 3.2 model by running this command in your PowerShell terminal:
ollama pull llama3.2

What is this downloading?

This downloads the Llama 3.2 3B parameter model. The download is about 2.0GB and uses a quantized format (Q4_K_M) that keeps quality high while fitting in less RAM.

The model files are stored in %HOMEPATH%\.ollama on your machine.

  • Wait for the download to complete. You will see a progress bar in the terminal.

Once the progress bar reaches 100% and you see a success message, the model is ready to use.

Download stuck or failed?

  • Check your internet connection. The model is 2.0GB, so a stable connection is needed.
  • If the download stalled, run the same ollama pull llama3.2 command again. It resumes from where it left off.
  • Make sure you have at least 4GB of free disk space for the model files.

Help me fix an Ollama model download issue on Windows.

Test the model from Python

The model is downloaded, but you need to confirm your Python code can actually talk to it. The ollama Python library communicates with the Ollama background service running on your machine.

  • Return to VS Code and your rag_pipeline.ipynb notebook.
  • Make sure the notebook kernel is set to your rag-project virtual environment. Click the kernel selector in the top-right of the notebook and choose the rag-project kernel.
  • In the first cell, add the following imports:
# Cell 1: Imports and Setup
import os
import glob
from sentence_transformers import SentenceTransformer
import chromadb
from ollama import chat

What do these imports do?

These five imports bring in the libraries you will use throughout the project.

  • os and glob handle file paths and finding documents on disk.
  • SentenceTransformer generates vector embeddings from text.
  • chromadb is the vector database that stores and searches those embeddings.
  • chat from the ollama library sends prompts to your local Llama 3.2 model.
  • Run Cell 1 by pressing Shift+Enter.
  • Click the + Code button below Cell 1 to create a new cell.
  • In the new Cell 2, add this test code to verify the connection:
# Quick test: send a message to Llama 3.2
response = chat(model='llama3.2', messages=[{'role': 'user', 'content': 'Hello!'}])
print(response.message.content)
  • Run Cell 2 by pressing Shift+Enter.

You should see a text response from the model (something like a greeting). The first run may take 10-20 seconds while Ollama loads the model into memory.

Seeing a connection error?

  • Make sure the Ollama background service is running. Open a PowerShell window and run ollama -v to check.
  • If you see a "could not connect" message, start the service manually by running ollama serve in a separate terminal window.
  • Make sure you selected the correct kernel. The ollama Python package must be installed in the kernel's environment.

Help me fix a Python connection error to my local Ollama instance.

  • Once you see a successful response, delete Cell 2 (the test cell) by clicking inside it and pressing Dd or right-clicking and selecting Delete Cell.
  • Save rag_pipeline.ipynb with Ctrl+S.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook with just the imports in Cell 1.

ⓧ I'd like to double check the full code

Cell 1 of your notebook should look exactly like this:

# Cell 1: Imports and Setup
import os
import glob
from sentence_transformers import SentenceTransformer
import chromadb
from ollama import chat

Your local LLM is ready to generate answers. Next up, you will create the documents that your RAG system will search over and learn how to split them into chunks.

Load and Chunk Your Documents

Llama 3.2 is running on your machine and ready to generate text. Now it's time to give it something specific to be smart about.

On its own, an LLM draws only on training data. To ground it in YOUR documents, you need to break those documents into searchable chunks that you can retrieve at query time.

In this step, get ready to:
  • Create a knowledge base of sample documents in a docs/ folder.
  • Write functions to load documents and split them into overlapping chunks.
  • Verify your chunks are splitting correctly.
Create your sample documents

Your RAG system needs documents to search over. You'll create three markdown files about Python topics that will serve as your knowledge base.

  • Create a docs folder inside your rag-project directory by running this command in your VS Code terminal:
mkdir docs
  • In the VS Code Explorer sidebar, right-click the new docs folder.
  • Click New File and name it python_basics.md.
  • Paste the following content into python_basics.md:
# Python Basics

## Variables and Data Types

Python supports several built-in data types. Integers (int) are whole numbers like 42 or -7. Floating point numbers (float) represent decimals like 3.14. Strings (str) are sequences of characters enclosed in quotes. Booleans (bool) are either True or False.

Variables in Python don't need type declarations. You simply assign a value: `x = 42`. Python uses dynamic typing, meaning the type is determined at runtime. You can check a variable's type with `type(x)`.

## Lists and Dictionaries

Lists are ordered, mutable collections created with square brackets: `fruits = ["apple", "banana", "cherry"]`. You can access elements by index (starting at 0), slice them, and modify them in place.

Dictionaries map keys to values: `person = {"name": "Alice", "age": 30}`. Keys must be immutable (strings, numbers, tuples). Access values with `person["name"]` or the safer `person.get("name")`.

## Functions

Functions are defined with the `def` keyword. They can accept parameters and return values. Default parameters provide fallback values when arguments aren't supplied.

```python
def greet(name, greeting="Hello"):
    return f"{greeting}, {name}!"
```

Python supports *args for variable positional arguments and **kwargs for variable keyword arguments, making functions highly flexible.

## Error Handling

Python uses try/except blocks for error handling. The `try` block contains code that might raise an exception. The `except` block handles specific exception types. The `finally` block runs regardless of whether an exception occurred.

Common exceptions include TypeError (wrong type), ValueError (wrong value), KeyError (missing dictionary key), and IndexError (list index out of range).
  • Right-click the docs folder again and click New File.
  • Name this file python_advanced.md.
  • Paste the following content into python_advanced.md:
# Advanced Python Concepts

## Decorators

Decorators are functions that modify other functions. They use the @syntax placed above a function definition. A decorator takes a function as input, wraps it with additional behavior, and returns the modified function.

Common use cases include logging, timing, authentication, and caching. The `@functools.lru_cache` decorator memoizes function results for repeated calls with the same arguments.

## Generators

Generators are functions that use `yield` instead of `return`. They produce values lazily one at a time, making them memory-efficient for large sequences. A generator function returns a generator object that implements the iterator protocol.

Generator expressions provide a concise syntax similar to list comprehensions but with parentheses: `squares = (x**2 for x in range(1000000))`. This creates values on demand rather than storing them all in memory.

## Context Managers

Context managers handle resource setup and cleanup using the `with` statement. The most common example is file handling: `with open("file.txt") as f:` ensures the file is properly closed even if an error occurs.

You can create custom context managers using the `@contextmanager` decorator from contextlib, or by implementing `__enter__` and `__exit__` methods in a class.

## Concurrency

Python offers multiple approaches to concurrency. Threading is suitable for I/O-bound tasks (network requests, file operations). Multiprocessing bypasses the GIL for CPU-bound tasks by using separate processes. Asyncio provides cooperative multitasking with async/await syntax for handling many concurrent I/O operations efficiently.

The Global Interpreter Lock (GIL) prevents multiple threads from executing Python bytecode simultaneously, which is why threading doesn't speed up CPU-bound work in CPython.
  • Right-click the docs folder one more time and click New File.
  • Name this file python_data_science.md.
  • Paste the following content into python_data_science.md:
# Python for Data Science

## NumPy

NumPy is the foundation of scientific computing in Python. It provides ndarray, a fast N-dimensional array object. NumPy arrays are homogeneous (all elements same type) and support vectorized operations that are much faster than Python loops.

Key operations include array creation (`np.array`, `np.zeros`, `np.arange`), reshaping, broadcasting (automatic shape alignment for arithmetic), and linear algebra (`np.dot`, `np.linalg`).

## Pandas

Pandas provides DataFrame and Series objects for structured data manipulation. A DataFrame is a 2D labeled table with columns of potentially different types. Series is a 1D labeled array.

Common operations include reading data (`pd.read_csv`), filtering (`df[df["age"] > 30]`), grouping (`df.groupby("category").mean()`), merging (`pd.merge`), and handling missing data (`df.fillna`, `df.dropna`).

## Matplotlib and Visualization

Matplotlib is Python's primary plotting library. The pyplot interface provides MATLAB-like plotting. Common plot types include line plots (`plt.plot`), scatter plots (`plt.scatter`), bar charts (`plt.bar`), and histograms (`plt.hist`).

Seaborn builds on matplotlib with a higher-level interface for statistical visualization. It provides attractive default styles and works directly with pandas DataFrames.

## Scikit-learn

Scikit-learn provides machine learning algorithms with a consistent API. The typical workflow is: create a model, fit it on training data, and predict on new data. All estimators follow the fit/predict pattern.

Key modules include classification (LogisticRegression, RandomForest), regression (LinearRegression, SVR), clustering (KMeans, DBSCAN), preprocessing (StandardScaler, OneHotEncoder), and model selection (train_test_split, cross_val_score).
  • Save all three files.
  • Confirm the files exist by running:
ls docs

You should see python_advanced.md, python_basics.md, and python_data_science.md listed in the output.

Why these documents?

These three files cover Python basics, advanced concepts, and data science libraries. They give your RAG system a realistic knowledge base with enough variety to test whether retrieval finds the right content for different questions.

Don't see your files listed?

Make sure you created the files inside the docs folder, not in the root rag-project folder. Check the Explorer sidebar to confirm the three .md files appear nested under docs.

If you see an empty folder, try right-clicking the docs folder and selecting Reveal in File Explorer to double check.

my docs/ folder is empty or the files aren't showing up

Write the document loader and chunking function

With your documents ready, you need two functions. The first loads all markdown files from a folder into memory. The second splits each document into overlapping chunks small enough for the embedding model to process.

Why chunk documents?

Embedding models work best on short passages, not entire documents. Splitting into ~500 character chunks means each piece is focused on one idea, making retrieval more precise.

The 50-character overlap ensures you don't lose context at chunk boundaries. If a sentence gets split between two chunks, the overlap keeps enough of it in both pieces to preserve meaning.

  • In your rag_pipeline.ipynb notebook, click below Cell 1 to create a new cell.
  • Add the document loading function by pasting this code into Cell 2:
# Cell 2: Document Loading
def load_documents(docs_folder):
    """Load all markdown files from a folder."""
    documents = []
    filepaths = glob.glob(os.path.join(docs_folder, "*.md"))
    for filepath in filepaths:
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
            documents.append({
                "content": content,
                "source": os.path.basename(filepath)
            })
    print(f"Loaded {len(documents)} documents")
    return documents

docs = load_documents("docs")

What does this code do?

  • glob.glob(os.path.join(docs_folder, "*.md")) finds every file ending in .md inside the folder you specify.
  • For each file, it reads the full text content and stores it alongside the filename as a dictionary.
  • docs = load_documents("docs") runs the function immediately, loading your three Python documentation files.
  • Run Cell 2 by pressing Shift+Enter.

You should see Loaded 3 documents printed below the cell.

Seeing "Loaded 0 documents"?

This means the function can't find your markdown files. Make sure your docs folder is in the same directory as your notebook file (rag-project).

Also check that your files end with .md (not .txt or .md.txt). Windows sometimes hides file extensions, which can cause naming mistakes.

my load_documents function returns 0 documents

  • Create another new cell below Cell 2.
  • Add the chunking function by pasting this code into Cell 3:
# Cell 3: Chunking
def chunk_document(document, chunk_size=500, overlap=50):
    """Split a document into overlapping chunks."""
    text = document["content"]
    source = document["source"]
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk_text = text[start:end]
        chunks.append({
            "text": chunk_text,
            "source": source,
            "chunk_index": len(chunks)
        })
        start += chunk_size - overlap
    return chunks

all_chunks = []
for doc in docs:
    all_chunks.extend(chunk_document(doc))

print(f"Total chunks: {len(all_chunks)}")
print(f"\nSample chunk from '{all_chunks[0]['source']}' (chunk {all_chunks[0]['chunk_index']}):")
print(all_chunks[0]["text"][:200] + "...")

What does this code do?

  • chunk_document takes a single document and slides a 500-character window across its text, advancing 450 characters each time (500 minus 50 overlap).
  • Each chunk stores its text, the source filename, and its position index so you can trace results back to their origin.
  • The loop runs chunk_document on every loaded document and collects all chunks into one flat list.
  • The print statements show the total chunk count and a preview of the first chunk so you can confirm it worked.
  • Run Cell 3 by pressing Shift+Enter.

You should see a total chunk count and a 200-character preview of the first chunk printed below the cell.

Seeing a NameError for docs?

This means Cell 2 hasn't been run yet, so the docs variable doesn't exist. Go back and run Cell 2 first (click on it and press Shift+Enter), then run Cell 3 again.

I'm getting a NameError when running my chunking cell

Test the chunking output

Cell 3's output tells you whether your chunking is working correctly. The numbers should make sense given the size of your documents and your chunk settings.

  • Check the Total chunks number in the output. With three documents of moderate length, you should see roughly 15-25 chunks depending on exact file sizes.
  • Read the sample chunk preview. It should show recognizable text from one of your markdown files, cut off at 200 characters.
  • Confirm the source filename and chunk index are printed correctly.

How do I know the chunks are good?

A good chunk is long enough to carry meaning but short enough for the embedding model to focus. At 500 characters with 50-character overlap, each chunk holds roughly 1-2 paragraphs of content.

If your total count seems too high, your documents might have extra whitespace. If it seems too low, double check that all three files have content saved.

✔️ Awesome, I've got everything!

Great. Double check you've saved both your notebook and all three markdown files.

ⓧ I'd like to double check the full code

Here is the full notebook code up to this point:

# Cell 1: Imports and Setup
import os
import glob
from sentence_transformers import SentenceTransformer
import chromadb
from ollama import chat

# Cell 2: Document Loading
def load_documents(docs_folder):
    """Load all markdown files from a folder."""
    documents = []
    filepaths = glob.glob(os.path.join(docs_folder, "*.md"))
    for filepath in filepaths:
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
            documents.append({
                "content": content,
                "source": os.path.basename(filepath)
            })
    print(f"Loaded {len(documents)} documents")
    return documents

docs = load_documents("docs")

# Cell 3: Chunking
def chunk_document(document, chunk_size=500, overlap=50):
    """Split a document into overlapping chunks."""
    text = document["content"]
    source = document["source"]
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk_text = text[start:end]
        chunks.append({
            "text": chunk_text,
            "source": source,
            "chunk_index": len(chunks)
        })
        start += chunk_size - overlap
    return chunks

all_chunks = []
for doc in docs:
    all_chunks.extend(chunk_document(doc))

print(f"Total chunks: {len(all_chunks)}")
print(f"\nSample chunk from '{all_chunks[0]['source']}' (chunk {all_chunks[0]['chunk_index']}):")
print(all_chunks[0]["text"][:200] + "...")

Your documents are loaded and chunked into searchable pieces. Next up, you'll turn these chunks into vector embeddings and store them in ChromaDB for fast semantic retrieval.

Embed and Store Chunks in ChromaDB

Your documents are chunked into bite-sized pieces. But text alone is not something a computer can compare for meaning.

To find which chunks are relevant to a question, you need to convert each chunk into a numerical representation called a vector embedding. These embeddings capture semantic meaning so that similar concepts land close together in vector space. In this step, you will generate embeddings and store them in a ChromaDB vector database for fast retrieval.

In this step, get ready to:
  • Load an embedding model and generate vectors for all your chunks.
  • Store embeddings in a persistent ChromaDB collection.
  • Test retrieval by querying the collection with a sample question.
Load the embedding model and generate vectors

An embedding model turns text into a fixed-length array of numbers. Words and sentences with similar meaning produce vectors that are close together. The model you will use is all-MiniLM-L6-v2, a lightweight model that produces 384-dimensional vectors.

  • Create a new cell below Cell 3 in your notebook. This will be Cell 4.
  • Load the embedding model and generate embeddings for every chunk by pasting the following code:
# Cell 4: Embedding
print("Loading embedding model...")
embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

chunk_texts = [chunk["text"] for chunk in all_chunks]
print(f"Generating embeddings for {len(chunk_texts)} chunks...")
embeddings = embedding_model.encode(chunk_texts, show_progress_bar=True)
print(f"Embedding dimension: {embeddings.shape[1]}")

What does this code do?

This cell handles the conversion from human-readable text to machine-comparable numbers.

  • SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") downloads and loads the embedding model. The first run downloads ~80MB of model weights.
  • chunk_texts extracts just the text string from each chunk dictionary so the model can process them as a batch.
  • embedding_model.encode(...) converts all chunks into a NumPy array of shape (num_chunks, 384). Each row is a 384-dimensional vector representing one chunk's meaning.
  • Run Cell 4 by clicking the play button or pressing Shift+Enter.

You should see the progress bar fill up and a final line printing the embedding dimension (384).

Seeing a download error or timeout?

  • The first run downloads the model from Hugging Face. Make sure you have an internet connection.
  • If you see a memory error, close other heavy applications (browsers, Docker, Ollama if not needed yet) and try again.

Help me debug my embedding model download error.

Store embeddings in ChromaDB

Now that you have vectors, you need somewhere to store and search them efficiently. ChromaDB is a vector database that persists your embeddings to disk and lets you query them by similarity.

  • Create a new cell below Cell 4. This will be Cell 5.
  • Create a persistent ChromaDB client, set up a collection, and add all chunks by pasting this code:
# Cell 5: Store in ChromaDB
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(
    name="python_docs",
    metadata={"hnsw:space": "cosine"}
)

collection.add(
    ids=[f"chunk_{i}" for i in range(len(all_chunks))],
    embeddings=embeddings.tolist(),
    documents=chunk_texts,
    metadatas=[{"source": chunk["source"], "chunk_index": chunk["chunk_index"]} for chunk in all_chunks]
)
print(f"Stored {collection.count()} chunks in ChromaDB")

What does this code do?

  • PersistentClient(path="./chroma_db") creates a database that saves to a chroma_db/ folder on disk. Your data survives notebook restarts.
  • get_or_create_collection creates a collection named python_docs configured to use cosine similarity for comparing vectors. Cosine measures angle between vectors, ignoring magnitude.
  • collection.add(...) inserts your chunks with unique IDs (chunk_0, chunk_1, etc.), their embedding vectors, the original text, and metadata (source file and chunk position).

Why cosine similarity?

Cosine similarity measures how similar two vectors are by their direction, not their length. Two chunks about the same topic will point in similar directions even if one is longer. This makes cosine the standard choice for comparing text embeddings.

  • Run Cell 5.

You should see a message like Stored 18 chunks in ChromaDB (the exact number depends on your documents).

Getting a duplicate ID error?

If you run Cell 5 more than once, ChromaDB rejects duplicate IDs. Restart your kernel (**Kernel > Restart**), then re-run all cells from Cell 1 through Cell 5.

Help me fix a ChromaDB duplicate ID error in my notebook.

Test retrieval with a sample query

The whole point of storing embeddings is to retrieve the most relevant chunks for a question. You can test this directly using ChromaDB's query method.

  • Create a new cell below Cell 5 and test retrieval by pasting the following code:
# Test retrieval with a sample question
test_question = "What are decorators in Python?"
test_embedding = embedding_model.encode([test_question])

results = collection.query(
    query_embeddings=test_embedding.tolist(),
    n_results=3
)

print(f"Question: {test_question}\n")
for i, (doc, meta, dist) in enumerate(zip(results["documents"][0], results["metadatas"][0], results["distances"][0])):
    print(f"Result {i+1} (distance: {dist:.4f}) from {meta['source']}:")
    print(f"  {doc[:150]}...")
    print()

What does this code do?

  • embedding_model.encode([test_question]) converts your question into the same 384-dimensional vector space as your chunks.
  • collection.query(...) finds the 3 chunks whose embeddings are closest to your question's embedding using cosine distance.
  • The results contain documents (chunk text), metadatas (source file and index), and distances (how far each result is from the query). Lower distance means higher relevance.
  • Run the cell.

You should see 3 results printed with their distances and source files. The top result should come from python_advanced.md since that file contains the decorator content.

Results look irrelevant?

  • Check that Cell 4 ran successfully (the embedding model loaded and printed dimension 384).
  • Make sure Cell 5 stored all chunks (the count should match your total from Cell 3).

Help me understand why my ChromaDB retrieval results seem wrong.

Why not delete this test cell?

Keep this cell for now. In the next step, you will build a proper retrieve function that wraps this same logic. Having the test here confirms your database works before you add more abstraction on top.

✔️ Awesome, I've got everything!

Great. Make sure you have saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here is the complete notebook code through Cell 5 (the persisted cells for this step):

# Cell 1: Imports and Setup
import os
import glob
from sentence_transformers import SentenceTransformer
import chromadb
from ollama import chat

# Cell 2: Document Loading
def load_documents(docs_folder):
    """Load all markdown files from a folder."""
    documents = []
    filepaths = glob.glob(os.path.join(docs_folder, "*.md"))
    for filepath in filepaths:
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
            documents.append({
                "content": content,
                "source": os.path.basename(filepath)
            })
    print(f"Loaded {len(documents)} documents")
    return documents

docs = load_documents("docs")

# Cell 3: Chunking
def chunk_document(document, chunk_size=500, overlap=50):
    """Split a document into overlapping chunks."""
    text = document["content"]
    source = document["source"]
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk_text = text[start:end]
        chunks.append({
            "text": chunk_text,
            "source": source,
            "chunk_index": len(chunks)
        })
        start += chunk_size - overlap
    return chunks

all_chunks = []
for doc in docs:
    all_chunks.extend(chunk_document(doc))

print(f"Total chunks: {len(all_chunks)}")
print(f"\nSample chunk from '{all_chunks[0]['source']}' (chunk {all_chunks[0]['chunk_index']}):")
print(all_chunks[0]["text"][:200] + "...")

# Cell 4: Embedding
print("Loading embedding model...")
embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

chunk_texts = [chunk["text"] for chunk in all_chunks]
print(f"Generating embeddings for {len(chunk_texts)} chunks...")
embeddings = embedding_model.encode(chunk_texts, show_progress_bar=True)
print(f"Embedding dimension: {embeddings.shape[1]}")

# Cell 5: Store in ChromaDB
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(
    name="python_docs",
    metadata={"hnsw:space": "cosine"}
)

collection.add(
    ids=[f"chunk_{i}" for i in range(len(all_chunks))],
    embeddings=embeddings.tolist(),
    documents=chunk_texts,
    metadatas=[{"source": chunk["source"], "chunk_index": chunk["chunk_index"]} for chunk in all_chunks]
)
print(f"Stored {collection.count()} chunks in ChromaDB")

Your documents are now searchable by meaning. In the next step, you will build a full RAG pipeline that retrieves these chunks and feeds them to Llama 3.2 to generate grounded answers.

Build the RAG Pipeline

Your documents are now chunked, embedded, and stored in ChromaDB. That means you have a searchable knowledge base ready to answer questions.

Now you need the actual pipeline that ties everything together. When a user asks a question, the system needs to find the most relevant chunks, inject them into a prompt, and let the LLM generate a grounded answer. This is the heart of RAG.

In this step, get ready to:
  • Write a retrieval function that finds the most relevant chunks for any question.
  • Write a generation function that feeds retrieved context to Llama 3.2 and returns a grounded answer.
  • Demonstrate the difference between RAG-powered answers and raw LLM answers.
Write the retrieval function

The retrieval function is the bridge between a user's question and your stored knowledge. It embeds the question into the same vector space as your chunks, then asks ChromaDB to find the closest matches.

  • Create a new cell in your notebook (Cell 6) and add the retrieval function by pasting this code:
# Cell 6: Retrieval Function
def retrieve(question, n_results=3):
    """Retrieve the most relevant chunks for a question."""
    question_embedding = embedding_model.encode([question])
    results = collection.query(
        query_embeddings=question_embedding.tolist(),
        n_results=n_results
    )
    return results["documents"][0], results["metadatas"][0], results["distances"][0]

# Test retrieval
question = "What are decorators in Python?"
docs_retrieved, metadatas, distances = retrieve(question)
print(f"Question: {question}\n")
for i, (doc, meta, dist) in enumerate(zip(docs_retrieved, metadatas, distances)):
    print(f"Result {i+1} (distance: {dist:.4f}) from {meta['source']}:")
    print(f"  {doc[:150]}...")
    print()

What does this code do?

The retrieve function takes a question string and converts it into an embedding using the same model that embedded your chunks. This is critical because the question must live in the same vector space as the stored documents for similarity comparison to work.

It then queries ChromaDB with collection.query() to find the top n_results closest chunks. The function returns three things: the chunk text, the metadata (source file and chunk index), and the cosine distances (lower = more similar).

  • Run the cell by pressing Shift+Enter.
  • You should see 3 results printed, each showing a distance score and a snippet from your documents about decorators.

Seeing a NameError?

If you see an error about embedding_model or collection not being defined, make sure you ran Cells 4 and 5 first. Each cell depends on variables created by earlier cells.

help me debug a NameError in my retrieval function

Write the RAG generation function

Retrieval alone is not enough. You need a function that takes the retrieved chunks, packages them into a prompt, and sends that prompt to Llama 3.2 so the model generates an answer grounded in your documents.

  • Create a new cell (Cell 7) and add the RAG generation function by pasting this code:
# Cell 7: RAG Generation
def generate_answer(question, n_results=3):
    """Generate an answer using RAG: retrieve context, then generate."""
    # Retrieve relevant chunks
    retrieved_docs, metadatas, distances = retrieve(question, n_results)

    # Build context from retrieved chunks
    context = "\n\n---\n\n".join(retrieved_docs)

    # Construct the prompt
    messages = [
        {
            "role": "system",
            "content": (
                "You are a helpful assistant. Answer the user's question based ONLY on the "
                "provided context. If the context doesn't contain enough information to answer, "
                "say so. Do not make up information."
            )
        },
        {
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {question}"
        }
    ]

    # Generate with Ollama
    response = chat(model="llama3.2", messages=messages)
    return response.message.content, retrieved_docs, metadatas

What does this code do?

This function performs the full RAG cycle. First, it calls your retrieve function to get the top-k relevant chunks. Then it joins them with separators into a single context string.

The system prompt is the key to grounding. It instructs the LLM to answer ONLY from the provided context and to admit when it does not have enough information. This prevents hallucination.

Finally, it sends the messages to Ollama using the chat function and returns the generated answer alongside the source documents.

Why is prompt engineering important here?

Without the system prompt constraint, the LLM would freely mix its training data with the retrieved context. By explicitly saying "Answer based ONLY on the provided context," you force the model to stay grounded in your documents.

This is the difference between a knowledge assistant that cites real sources and one that confidently makes things up.

  • Run the cell by pressing Shift+Enter.
  • This cell defines the function without producing output. You will use it in the next cell.
Demonstrate RAG vs raw LLM answers

The real payoff of RAG is visible when you compare answers with and without retrieved context. Without context, the LLM draws from its general training data and may hallucinate or give generic answers. With RAG, it stays anchored to your specific documents.

  • Create a new cell (Cell 8) and add the comparison code by pasting this:
# Cell 8: Compare RAG vs No Context
def generate_without_context(question):
    """Generate an answer WITHOUT any retrieved context."""
    messages = [
        {
            "role": "system",
            "content": "You are a helpful assistant. Answer the user's question."
        },
        {
            "role": "user",
            "content": question
        }
    ]
    response = chat(model="llama3.2", messages=messages)
    return response.message.content

# Demonstrate the difference
question = "What approaches to concurrency does Python offer and what is the GIL?"
print(f"Question: {question}")
print("\n" + "="*60)
print("ANSWER WITHOUT CONTEXT (raw LLM):")
print("="*60)
print(generate_without_context(question))
print("\n" + "="*60)
print("ANSWER WITH RAG (grounded in documents):")
print("="*60)
answer, sources, meta = generate_answer(question)
print(answer)
print(f"\nSources used: {[m['source'] for m in meta]}")

What does this code do?

The generate_without_context function asks the same LLM the same question, but with no retrieved documents. It uses a generic system prompt with no grounding instructions.

The comparison prints both answers side by side so you can see the difference. The RAG answer cites specific details from your python_advanced.md document. The raw LLM answer draws from general training data and may include details not in your knowledge base.

  • Run the cell by pressing Shift+Enter.
  • Wait for both answers to generate. This takes a few seconds as Llama 3.2 processes each prompt.
  • You should see two clearly separated answers. The RAG answer references threading, multiprocessing, asyncio, and the GIL as described in your documents. The sources line should show python_advanced.md.

Getting a connection error from Ollama?

If Ollama is not responding, it may have stopped running in the background. Open a new terminal and run ollama serve to restart it. Then re-run Cell 8.

If the response is very slow (more than 60 seconds), close other heavy applications to free up RAM. Llama 3.2 needs about 3-4 GB of memory to run smoothly.

help me fix an Ollama connection error in my RAG pipeline

✔️ Awesome, I've got everything!

Great. Make sure your notebook is saved before moving on.

ⓧ I'd like to double check the full code

Here is the complete notebook up to this point. Compare your cells against this:

# Cell 1: Imports and Setup
import os
import glob
from sentence_transformers import SentenceTransformer
import chromadb
from ollama import chat

# Cell 2: Document Loading
def load_documents(docs_folder):
    """Load all markdown files from a folder."""
    documents = []
    filepaths = glob.glob(os.path.join(docs_folder, "*.md"))
    for filepath in filepaths:
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
            documents.append({
                "content": content,
                "source": os.path.basename(filepath)
            })
    print(f"Loaded {len(documents)} documents")
    return documents

docs = load_documents("docs")

# Cell 3: Chunking
def chunk_document(document, chunk_size=500, overlap=50):
    """Split a document into overlapping chunks."""
    text = document["content"]
    source = document["source"]
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk_text = text[start:end]
        chunks.append({
            "text": chunk_text,
            "source": source,
            "chunk_index": len(chunks)
        })
        start += chunk_size - overlap
    return chunks

all_chunks = []
for doc in docs:
    all_chunks.extend(chunk_document(doc))

print(f"Total chunks: {len(all_chunks)}")
print(f"\nSample chunk from '{all_chunks[0]['source']}' (chunk {all_chunks[0]['chunk_index']}):")
print(all_chunks[0]["text"][:200] + "...")

# Cell 4: Embedding
print("Loading embedding model...")
embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

chunk_texts = [chunk["text"] for chunk in all_chunks]
print(f"Generating embeddings for {len(chunk_texts)} chunks...")
embeddings = embedding_model.encode(chunk_texts, show_progress_bar=True)
print(f"Embedding dimension: {embeddings.shape[1]}")

# Cell 5: Store in ChromaDB
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(
    name="python_docs",
    metadata={"hnsw:space": "cosine"}
)

collection.add(
    ids=[f"chunk_{i}" for i in range(len(all_chunks))],
    embeddings=embeddings.tolist(),
    documents=chunk_texts,
    metadatas=[{"source": chunk["source"], "chunk_index": chunk["chunk_index"]} for chunk in all_chunks]
)
print(f"Stored {collection.count()} chunks in ChromaDB")

# Cell 6: Retrieval Function
def retrieve(question, n_results=3):
    """Retrieve the most relevant chunks for a question."""
    question_embedding = embedding_model.encode([question])
    results = collection.query(
        query_embeddings=question_embedding.tolist(),
        n_results=n_results
    )
    return results["documents"][0], results["metadatas"][0], results["distances"][0]

# Test retrieval
question = "What are decorators in Python?"
docs_retrieved, metadatas, distances = retrieve(question)
print(f"Question: {question}\n")
for i, (doc, meta, dist) in enumerate(zip(docs_retrieved, metadatas, distances)):
    print(f"Result {i+1} (distance: {dist:.4f}) from {meta['source']}:")
    print(f"  {doc[:150]}...")
    print()

# Cell 7: RAG Generation
def generate_answer(question, n_results=3):
    """Generate an answer using RAG: retrieve context, then generate."""
    # Retrieve relevant chunks
    retrieved_docs, metadatas, distances = retrieve(question, n_results)

    # Build context from retrieved chunks
    context = "\n\n---\n\n".join(retrieved_docs)

    # Construct the prompt
    messages = [
        {
            "role": "system",
            "content": (
                "You are a helpful assistant. Answer the user's question based ONLY on the "
                "provided context. If the context doesn't contain enough information to answer, "
                "say so. Do not make up information."
            )
        },
        {
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {question}"
        }
    ]

    # Generate with Ollama
    response = chat(model="llama3.2", messages=messages)
    return response.message.content, retrieved_docs, metadatas

# Cell 8: Compare RAG vs No Context
def generate_without_context(question):
    """Generate an answer WITHOUT any retrieved context."""
    messages = [
        {
            "role": "system",
            "content": "You are a helpful assistant. Answer the user's question."
        },
        {
            "role": "user",
            "content": question
        }
    ]
    response = chat(model="llama3.2", messages=messages)
    return response.message.content

# Demonstrate the difference
question = "What approaches to concurrency does Python offer and what is the GIL?"
print(f"Question: {question}")
print("\n" + "="*60)
print("ANSWER WITHOUT CONTEXT (raw LLM):")
print("="*60)
print(generate_without_context(question))
print("\n" + "="*60)
print("ANSWER WITH RAG (grounded in documents):")
print("="*60)
answer, sources, meta = generate_answer(question)
print(answer)
print(f"\nSources used: {[m['source'] for m in meta]}")

Your RAG pipeline is fully functional. You can now ask any question and get an answer grounded in your documents. Next up, you will stress-test this system with different questions and tune the parameters to improve answer quality.

Test Your RAG System

In the last step, you proved that RAG eliminates hallucination. The side-by-side comparison showed how retrieved context keeps the LLM grounded in your documents instead of making things up.

A single question isn't enough to trust a system, though. You need to stress-test it across different topics and then experiment with the knobs that control answer quality.

In this step, get ready to:
  • Ask multiple questions and evaluate whether retrieved chunks are relevant and answers stay grounded.
  • Experiment with top_k, chunk size, and system prompt parameters to see how they affect answer quality.
Run a batch of questions through your pipeline

A good test covers different topics in your document set. Your three markdown files cover Python basics, advanced concepts, and data science. Asking questions that span all three files tells you whether retrieval is pulling from the right source.

  • Create a new cell at the bottom of rag_pipeline.ipynb and paste the following code:
# Cell 9: Interactive Testing
questions = [
    "How do generators work and why are they memory efficient?",
    "What is the difference between a list and a dictionary?",
    "How does scikit-learn's API work?",
]

for q in questions:
    print(f"\nQ: {q}")
    print("-" * 40)
    answer, sources, meta = generate_answer(q)
    print(f"A: {answer}")
    print(f"Sources: {[m['source'] for m in meta]}")
    print()

What does this code do?

This cell loops through a list of questions that target different documents in your collection.

  • "How do generators work..." targets python_advanced.md.
  • "What is the difference between a list and a dictionary?" targets python_basics.md.
  • "How does scikit-learn's API work?" targets python_data_science.md.
  • The Sources: line at the end of each answer shows which files the retriever pulled from, so you can verify the right document was found.
  • Run the cell.
  • For each answer, check two things: does the Sources list name the file you'd expect, and does the answer stay within what your documents actually say?

What should I look for?

A well-grounded answer only states facts present in the retrieved chunks. If the LLM adds information that isn't in your documents, that's a sign the system prompt isn't constraining it enough or the retriever is pulling irrelevant chunks.

The answers will take a few seconds each since Ollama is running inference locally. This is normal for a CPU-only setup.

Answers taking too long or not appearing?

  • Make sure Ollama is still running. Open a terminal and check with ollama list. If it returns nothing, start it again.
  • Close other heavy applications. Llama 3.2 needs around 3-4GB of RAM when loaded.
  • Make sure you ran all previous cells (Cells 1-8) first. The generate_answer function depends on the embedding model and ChromaDB collection being loaded.

Still stuck? Ask for help.

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook before moving on.

ⓧ I'd like to double check the full code

# Cell 1: Imports and Setup
import os
import glob
from sentence_transformers import SentenceTransformer
import chromadb
from ollama import chat

# Cell 2: Document Loading
def load_documents(docs_folder):
    """Load all markdown files from a folder."""
    documents = []
    filepaths = glob.glob(os.path.join(docs_folder, "*.md"))
    for filepath in filepaths:
        with open(filepath, "r", encoding="utf-8") as f:
            content = f.read()
            documents.append({
                "content": content,
                "source": os.path.basename(filepath)
            })
    print(f"Loaded {len(documents)} documents")
    return documents

docs = load_documents("docs")

# Cell 3: Chunking
def chunk_document(document, chunk_size=500, overlap=50):
    """Split a document into overlapping chunks."""
    text = document["content"]
    source = document["source"]
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk_text = text[start:end]
        chunks.append({
            "text": chunk_text,
            "source": source,
            "chunk_index": len(chunks)
        })
        start += chunk_size - overlap
    return chunks

all_chunks = []
for doc in docs:
    all_chunks.extend(chunk_document(doc))

print(f"Total chunks: {len(all_chunks)}")
print(f"\nSample chunk from '{all_chunks[0]['source']}' (chunk {all_chunks[0]['chunk_index']}):")
print(all_chunks[0]["text"][:200] + "...")

# Cell 4: Embedding
print("Loading embedding model...")
embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

chunk_texts = [chunk["text"] for chunk in all_chunks]
print(f"Generating embeddings for {len(chunk_texts)} chunks...")
embeddings = embedding_model.encode(chunk_texts, show_progress_bar=True)
print(f"Embedding dimension: {embeddings.shape[1]}")

# Cell 5: Store in ChromaDB
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(
    name="python_docs",
    metadata={"hnsw:space": "cosine"}
)

collection.add(
    ids=[f"chunk_{i}" for i in range(len(all_chunks))],
    embeddings=embeddings.tolist(),
    documents=chunk_texts,
    metadatas=[{"source": chunk["source"], "chunk_index": chunk["chunk_index"]} for chunk in all_chunks]
)
print(f"Stored {collection.count()} chunks in ChromaDB")

# Cell 6: Retrieval Function
def retrieve(question, n_results=3):
    """Retrieve the most relevant chunks for a question."""
    question_embedding = embedding_model.encode([question])
    results = collection.query(
        query_embeddings=question_embedding.tolist(),
        n_results=n_results
    )
    return results["documents"][0], results["metadatas"][0], results["distances"][0]

# Test retrieval
question = "What are decorators in Python?"
docs_retrieved, metadatas, distances = retrieve(question)
print(f"Question: {question}\n")
for i, (doc, meta, dist) in enumerate(zip(docs_retrieved, metadatas, distances)):
    print(f"Result {i+1} (distance: {dist:.4f}) from {meta['source']}:")
    print(f"  {doc[:150]}...")
    print()

# Cell 7: RAG Generation
def generate_answer(question, n_results=3):
    """Generate an answer using RAG: retrieve context, then generate."""
    retrieved_docs, metadatas, distances = retrieve(question, n_results)
    context = "\n\n---\n\n".join(retrieved_docs)
    messages = [
        {
            "role": "system",
            "content": (
                "You are a helpful assistant. Answer the user's question based ONLY on the "
                "provided context. If the context doesn't contain enough information to answer, "
                "say so. Do not make up information."
            )
        },
        {
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {question}"
        }
    ]
    response = chat(model="llama3.2", messages=messages)
    return response.message.content, retrieved_docs, metadatas

# Cell 8: Compare RAG vs No Context
def generate_without_context(question):
    """Generate an answer WITHOUT any retrieved context."""
    messages = [
        {
            "role": "system",
            "content": "You are a helpful assistant. Answer the user's question."
        },
        {
            "role": "user",
            "content": question
        }
    ]
    response = chat(model="llama3.2", messages=messages)
    return response.message.content

question = "What approaches to concurrency does Python offer and what is the GIL?"
print(f"Question: {question}")
print("\n" + "="*60)
print("ANSWER WITHOUT CONTEXT (raw LLM):")
print("="*60)
print(generate_without_context(question))
print("\n" + "="*60)
print("ANSWER WITH RAG (grounded in documents):")
print("="*60)
answer, sources, meta = generate_answer(question)
print(answer)
print(f"\nSources used: {[m['source'] for m in meta]}")

# Cell 9: Interactive Testing
questions = [
    "How do generators work and why are they memory efficient?",
    "What is the difference between a list and a dictionary?",
    "How does scikit-learn's API work?",
]

for q in questions:
    print(f"\nQ: {q}")
    print("-" * 40)
    answer, sources, meta = generate_answer(q)
    print(f"A: {answer}")
    print(f"Sources: {[m['source'] for m in meta]}")
    print()
Experiment with retrieval parameters

Three parameters have the biggest impact on answer quality: how many chunks you retrieve, how large those chunks are, and what instructions you give the LLM in the system prompt.

Try each experiment below, re-run the affected cells, and observe how the answers change.

Experiment 1: Change top_k (number of retrieved chunks)
  • In Cell 9, find the line that calls generate_answer:
answer, sources, meta = generate_answer(q)
  • Change it to pass n_results=5 so the line looks like this:
answer, sources, meta = generate_answer(q, n_results=5)
  • Re-run Cell 9 and compare the answers and source lists to what you saw before.

What does changing top_k do?

Retrieving more chunks gives the LLM more context to work with. This can improve answers when information is spread across multiple chunks. But it can also introduce noise if the extra chunks are only loosely related, diluting the relevant context.

Experiment 2: Change chunk size
  • Go back to Cell 3 and find the function signature:
def chunk_document(document, chunk_size=500, overlap=50):
  • Change chunk_size=500 to chunk_size=300 so the line looks like this:
def chunk_document(document, chunk_size=300, overlap=50):
  • Re-run Cells 3, 4, and 5 in order. This re-chunks, re-embeds, and re-stores everything in ChromaDB.
  • Then re-run Cell 9 and notice how the answers differ with smaller chunks.

What does changing chunk size do?

Smaller chunks are more focused. Each retrieved result is tightly scoped to one idea. Larger chunks carry more surrounding context but may include irrelevant content that confuses the LLM.

There is no perfect chunk size. It depends on how your documents are structured and what types of questions you expect.

Experiment 3: Change the system prompt
  • Go back to Cell 7 and find the system message inside generate_answer:
"content": (
    "You are a helpful assistant. Answer the user's question based ONLY on the "
    "provided context. If the context doesn't contain enough information to answer, "
    "say so. Do not make up information."
)
  • Try replacing it with a more specific instruction. For example, tell the model to respond in bullet points and cite the source filename:
"content": (
    "You are a helpful assistant. Answer the user's question based ONLY on the "
    "provided context. Format your answer as bullet points. At the end, list which "
    "source file(s) you used. If the context doesn't contain enough information, say so."
)
  • Re-run Cells 7 and 9 to see how the output format changes.

What does changing the system prompt do?

The system prompt controls HOW the LLM uses the context, not WHAT context it receives. You can shape output format, strictness, tone, and citation behavior without touching any retrieval logic.

Reset your parameters
  • Set chunk_size back to 500 in Cell 3.
  • Restore the original system prompt in Cell 7.
  • Remove n_results=5 from Cell 9 so the call goes back to:
answer, sources, meta = generate_answer(q)
  • Re-run Cells 3, 4, 5, 7, and 9 to bring everything back to the default state.
  • Save your notebook.

These are the three levers of RAG quality

In production RAG systems, teams spend most of their tuning time on exactly these three areas: chunking strategy, retrieval depth (top_k), and prompt engineering. You now have hands-on intuition for how each one affects the final answer.

Secret mission

Add Cross-Encoder Re-Ranking

Your RAG pipeline retrieves chunks based on raw vector similarity. But vector similarity is a rough approximation. What if you could add a second, more precise scoring step that re-evaluates each candidate chunk against your exact question?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs or cloud bills to worry about. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The rag-project folder (contains your virtual environment, notebook, docs, and ChromaDB database).
  • Ollama with the Llama 3.2 model (2.0GB downloaded to your machine).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your RAG pipeline or want to add more documents to your knowledge base.

  • Ollama continues running in the background and will restart when you log in.
  • Your ChromaDB database persists in the chroma_db/ folder, ready for queries at any time.
  • The Llama 3.2 model stays downloaded at ~2.0GB in your home directory.

Pause - I'll come back to this later

Shut down Ollama to free up memory while keeping all your files intact for later.

  • Right-click the Ollama icon in your system tray (bottom-right corner of your taskbar) and select Quit.
  • To stop Ollama from starting automatically, open Task Manager, go to the Startup apps tab, find ollama, and click Disable.
  • Deactivate your virtual environment by running:
deactivate

When you want to resume, start Ollama from the Start menu and reactivate your venv.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees up disk space but cannot be undone.

  • Remove the Llama 3.2 model by running this in your terminal:
ollama rm llama3.2
  • Uninstall Ollama through Windows Settings. Open Settings, select Apps, then Installed apps. Search for Ollama and click Uninstall.
  • Deactivate your virtual environment first, then delete the project folder:
deactivate
cd ~
Remove-Item -Recurse -Force rag-project
  • Confirm the folder is gone by running:
ls rag-project

You should see an error indicating the path does not exist. This confirms everything is cleaned up.

Nice Work!

Nice Work!

You just built a complete Retrieval-Augmented Generation system from scratch, running entirely on your machine with zero API costs.

You've learned how to:

  • Build a document ingestion pipeline that loads, chunks, and embeds text into a ChromaDB vector store with metadata and persistence.
  • Run a local LLM (Llama 3.2) via Ollama and generate grounded answers by augmenting prompts with retrieved context from semantic similarity search.
  • Demonstrate how RAG eliminates hallucination by comparing raw LLM responses against context-grounded answers sourced from your own documents.
  • Added Cross-Encoder re-ranking for production-quality retrieval precision.

Ready to quiz yourself?