Build a RAG System with LangChain

Build a local RAG pipeline that answers questions grounded in your own documents.

Introduction

30 Second Summary

Every time you ask an AI chatbot a question about your company's internal docs or a niche technical topic, it either makes something up or gives a generic answer. That's because the model doesn't have access to your specific data.

In this project, you will build a local RAG (Retrieval-Augmented Generation) system that feeds your own documents to an LLM so it can give accurate, grounded answers instead of hallucinating.

What You'll Build

A Python-based documentation assistant that loads web pages, splits them into searchable chunks, stores them as vector embeddings, and retrieves relevant context to generate accurate answers using a local LLM.

By the end of this project, you'll have:

  • A working RAG pipeline that loads, chunks, embeds, and queries documents entirely on your machine.
  • Hands-on understanding of embeddings, vector databases, and similarity search by actually generating and querying them.
  • A simple evaluation harness that measures retrieval quality, the skill that separates production AI engineers from demo builders.
  • Secret Mission: Add hybrid search and source citations for more accurate, trustworthy answers.

Are there any prerequisites?

You should have basic Python familiarity (writing functions, using pip) and be comfortable running commands in the terminal. No prior AI or machine learning experience is needed.

Set Up Your Local AI Environment

Your RAG system needs three things to work: a local LLM to generate answers, an embedding model to turn text into numerical vectors, and Python libraries to orchestrate the pipeline.

In this step, you'll install Ollama (a tool that serves LLMs on your machine), pull two models, and set up a Python environment with all the packages your project needs.

In this step, get ready to:
  • Install Ollama and pull two models (an LLM and an embedding model).
  • Create a Python virtual environment and install all required packages.
  • Verify everything works by running a test script.
Install Ollama and pull the models

Ollama lets you run LLMs locally without needing an API key or internet connection. It manages model downloads and serves them over a local API that your Python code will talk to.

  • Install Ollama by running this command:
curl -fsSL https://ollama.com/install.sh | sh

What does this command do?

This downloads and runs the official Ollama install script. It installs the ollama binary, creates a systemd service, and starts Ollama automatically in the background.

  • Confirm Ollama installed successfully by running:
ollama -v

You should see a version number printed in your terminal.

Don't see a version number?

Make sure the install script completed without errors. If you see a permissions error, try running the install command with sudo prepended. If your terminal says the command is not found, close and reopen your terminal to reload your PATH.

Help me troubleshoot my Ollama installation on Linux.

  • Pull the llama3.2 model (a 3B parameter LLM that runs on CPU) by running:
ollama pull llama3.2

This downloads approximately 2GB. It may take a few minutes depending on your internet speed.

  • Pull the nomic-embed-text embedding model by running:
ollama pull nomic-embed-text
  • Verify both models are available by running:
ollama list

You should see both llama3.2 and nomic-embed-text in the output table.

Why two models?

Your RAG system uses each model for a different job. The nomic-embed-text model converts text into numerical vectors (embeddings) for similarity search. The llama3.2 model reads retrieved context and generates human-readable answers.

Pull command hanging or failing?

Check that the Ollama service is running with systemctl status ollama. If the service is inactive, start it with sudo systemctl start ollama. Network timeouts can happen on slow connections. Try the pull command again.

Help me fix my Ollama model pull.

Create a Python virtual environment and install packages

Your project needs Python 3.10 or higher because the langchain package requires it.

  • Check your Python version by running:
python3 --version

✔️ I see version 3.10 or higher

Your Python version is ready. Continue to the next instruction below.

ⓧ I see an older version

You need Python 3.10+ for this project. On many Linux distributions, you can install a newer version alongside the system default.

  • Install Python 3.10+ using your package manager. On Ubuntu/Debian, run:
sudo apt update
sudo apt install python3.10 python3.10-venv

After installing, use python3.10 instead of python3 in the commands that follow.

ⓧ Command not found

Python is not installed on your system. Install it using your package manager.

  • On Ubuntu/Debian, install Python by running:
sudo apt update
sudo apt install python3.10 python3.10-venv
  • Verify the installation by running:
python3.10 --version

Use python3.10 instead of python3 in the commands that follow.

  • Navigate to your project directory and create a virtual environment by running:
python3 -m venv rag_env
  • Activate the virtual environment by running:
source rag_env/bin/activate

You should see (rag_env) appear at the start of your terminal prompt.

What is a virtual environment?

A virtual environment is an isolated Python installation. Packages you install inside it stay separate from your system Python. This prevents version conflicts between projects.

  • Install all required packages by running:
pip install langchain langchain-text-splitters chromadb langchain-chroma langchain-ollama beautifulsoup4 requests

This installs seven packages. It may take a minute or two as some packages have compiled dependencies.

What are these packages?

  • langchain is the orchestration framework that connects your LLM, embeddings, and vector store together.
  • langchain-text-splitters provides tools to break documents into chunks.
  • chromadb is a local vector database for storing and searching embeddings.
  • langchain-chroma and langchain-ollama are integration packages that let LangChain talk to ChromaDB and Ollama.
  • beautifulsoup4 and requests handle web page fetching and HTML parsing.

Seeing install errors?

Make sure your virtual environment is active (you should see (rag_env) in your prompt). If you see a build error for chromadb, you may need to install build tools with sudo apt install build-essential.

Help me fix my pip install errors.

Verify the setup with a test script

Time to confirm that Python can talk to Ollama. You'll write a short script that sends a prompt to the local LLM and prints the response.

  • Create a file called setup_test.py in your project directory and add the following code:
from langchain_ollama import OllamaLLM

# Connect to the local Ollama instance using llama3.2
llm = OllamaLLM(model="llama3.2")
# Send a test prompt and print the response
response = llm.invoke("Say hello in one sentence.")
print(response)

What does this code do?

  • OllamaLLM(model="llama3.2") creates a connection to your local Ollama server and specifies which model to use.
  • llm.invoke(...) sends the prompt string to the model and waits for a response.
  • The response is a plain string containing the model's output.
  • Save setup_test.py and run it:
python3 setup_test.py

You should see a one-sentence greeting printed in your terminal. The first run may take 10-20 seconds as Ollama loads the model into memory.

Not getting a response?

If you see a connection error, make sure Ollama is running with systemctl status ollama. If you see a model not found error, confirm llama3.2 appears when you run ollama list.

Help me debug my setup_test.py script.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

from langchain_ollama import OllamaLLM

llm = OllamaLLM(model="llama3.2")
response = llm.invoke("Say hello in one sentence.")
print(response)

Your local AI environment is fully operational. Next up, you'll load a real document, break it into chunks, and prepare it for retrieval.

Load and Chunk Documents

Your local AI environment is working. You sent a test prompt and got a response back from Ollama.

Now it is time to give your LLM something useful to read. In this step, you will fetch a web page, clean it up, and split it into small overlapping pieces called chunks. These chunks are the raw material your RAG system will search through when answering questions.

In this step, get ready to:
  • Write a document loader that fetches and cleans a web page.
  • Split the document into overlapping chunks using a text splitter.
  • Run the loader and splitter to see your chunks in the terminal.
Write the document loader

Your RAG system needs source material. You will write a function that downloads a web page, strips out navigation and formatting clutter, and returns the raw text as a LangChain Document object.

  • Create a new file called rag.py in the same directory as setup_test.py.
  • Add the following imports at the top of rag.py:
import requests
import bs4
import chromadb
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings, OllamaLLM
from langchain_chroma import Chroma

What are these imports for?

  • requests fetches web pages over HTTP.
  • bs4 (BeautifulSoup) parses HTML so you can extract clean text.
  • Document is LangChain's standard container for text plus metadata.
  • RecursiveCharacterTextSplitter splits long text into smaller chunks.
  • The remaining imports (OllamaEmbeddings, OllamaLLM, Chroma) are for later steps. Adding them now keeps the file organized.

You won't see output yet. The next chunk uses these imports to build the loader function.

  • Below the imports, add a section comment and the load_web_page function:
# --- Step 2: Load and Chunk Documents ---

def load_web_page(url: str) -> list[Document]:
    # Fetch the page content
    response = requests.get(url)
    response.raise_for_status()
    # Parse HTML and remove non-content elements
    soup = bs4.BeautifulSoup(response.text, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "header"]):
        tag.decompose()
    # Extract clean text
    text = soup.get_text(separator="\n", strip=True)
    return [Document(page_content=text, metadata={"source": url})]

What does this function do?

  • requests.get(url) downloads the web page's raw HTML.
  • raise_for_status() throws an error if the download failed (404, timeout, etc.).
  • decompose() permanently removes navigation, footer, header, script, and style tags from the parsed HTML. This leaves only the article text.
  • The function returns a list containing one Document object with the cleaned text and a metadata field recording where it came from.
  • Save rag.py.

You won't see a visible change yet. The next function will use this loader and produce testable output.

Seeing a red underline on the imports?

Make sure your virtual environment is activated. Your terminal prompt should show the environment name in parentheses. If it does not, reactivate it before continuing.

help me fix import errors in my Python virtual environment

Split documents into chunks

A full web page can be thousands of characters long. Feeding all of that to an LLM would drown the relevant details in noise. Instead, you split the text into smaller overlapping pieces so your system can retrieve just the parts that matter.

  • Below the load_web_page function in rag.py, add the chunk_documents function:
def chunk_documents(docs: list[Document]) -> list[Document]:
    # Configure the text splitter
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        add_start_index=True,
    )
    # Split all documents into chunks
    all_splits = text_splitter.split_documents(docs)
    print(f"Split into {len(all_splits)} chunks.")
    return all_splits

What does this function do?

  • RecursiveCharacterTextSplitter breaks text at natural boundaries (paragraphs, sentences, words) to keep chunks readable.
  • chunk_size=1000 sets the maximum chunk length to 1000 characters.
  • chunk_overlap=200 makes each chunk share 200 characters with its neighbor. This prevents meaning from being lost at the boundary between two chunks.
  • add_start_index=True records each chunk's character position in the original text, which is useful for tracing answers back to their source.
  • Save rag.py.
  • Test the loader and chunker together by running this command in your terminal:
python -c "
from rag import load_web_page, chunk_documents
docs = load_web_page('https://lilianweng.github.io/posts/2023-06-23-agent/')
print(f'Loaded {len(docs)} document(s).')
chunks = chunk_documents(docs)
print(f'First chunk preview: {chunks[0].page_content[:200]}')
print(f'Second chunk preview: {chunks[1].page_content[:200]}')
"

You should see output reporting 40-80 chunks (the exact number depends on the page length) followed by a preview of the first two chunks.

Seeing a connection error?

The script fetches a live web page. Make sure your machine has internet access. If the URL is unreachable, check that you copied it exactly as shown.

If you see an ImportError, confirm your virtual environment is active and all packages were installed in Step 1.

help me debug why my rag.py script fails to run

Understand why chunking parameters matter

The two numbers you chose (1000 and 200) are not arbitrary. They control how precisely your system can retrieve relevant information later.

How do chunk size and overlap affect retrieval?

Chunk size controls the tradeoff between precision and context. Chunks that are too large bring in irrelevant noise alongside the answer. Chunks that are too small might cut off a sentence mid-thought, losing the information you need.

Overlap is your insurance policy against unlucky splits. With 200 characters of overlap, any sentence that falls near a boundary appears in both neighboring chunks. This means your retrieval system can still find it regardless of which chunk it searches.

1000 characters with 200-character overlap is a solid starting point. Production systems tune these values based on their documents and query patterns.

what chunk size and overlap should I use for different document types

✔️ Awesome, I've got everything!

Great. Double check you have saved rag.py and that your test command printed chunk previews.

ⓧ I'd like to double check the full code

import requests
import bs4
import chromadb
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings, OllamaLLM
from langchain_chroma import Chroma


# --- Step 2: Load and Chunk Documents ---

def load_web_page(url: str) -> list[Document]:
    response = requests.get(url)
    response.raise_for_status()
    soup = bs4.BeautifulSoup(response.text, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "header"]):
        tag.decompose()
    text = soup.get_text(separator="\n", strip=True)
    return [Document(page_content=text, metadata={"source": url})]


def chunk_documents(docs: list[Document]) -> list[Document]:
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        add_start_index=True,
    )
    all_splits = text_splitter.split_documents(docs)
    print(f"Split into {len(all_splits)} chunks.")
    return all_splits

Your documents are loaded and sliced into retrieval-ready pieces. Next up, you will turn these text chunks into numerical vectors and store them in a vector database for lightning-fast similarity search.

Create Embeddings and Store in a Vector Database

You now have a collection of text chunks ready to go. The problem is, computers can't search text the way humans read it. To find relevant chunks for a question, you need to convert text into numbers that capture meaning.

This is where embeddings come in. An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. Similar ideas produce similar vectors, which means you can find relevant content by finding the closest vectors to your question. In this step, you'll generate embeddings for all your chunks and store them in ChromaDB, a local vector database.

In this step, get ready to:
  • Initialize the embedding model and verify it produces vectors.
  • Store all your document chunks and their embeddings in a persistent vector database.
  • Test similarity search to confirm retrieval works.
Initialize the embedding model and test it

Before building the full storage pipeline, you should confirm that the embedding model is working and understand what its output looks like.

  • Test the embedding model by running this Python snippet in your terminal:
python3 -c "
from langchain_ollama import OllamaEmbeddings

# Initialize the embedding model
embeddings = OllamaEmbeddings(model='nomic-embed-text')

# Generate a test embedding for a short sentence
test_vector = embeddings.embed_query('What is a vector database?')
print(f'Embedding dimensions: {len(test_vector)}')
print(f'First 5 values: {test_vector[:5]}')
"

What does this output show?

The OllamaEmbeddings class connects to your local Ollama service and uses the nomic-embed-text model to convert text into vectors.

You should see Embedding dimensions: 768 followed by 5 floating-point numbers. Each chunk of text gets transformed into a list of 768 numbers that capture its semantic meaning. Two texts about similar topics will produce vectors that are close together in this 768-dimensional space.

  • Confirm you see 768 dimensions and a list of decimal numbers in the output.

Not seeing any output?

Make sure Ollama is still running. You can check with ollama -v in a separate terminal.

If you see a connection error, restart the Ollama service and try again.

help me debug why my Ollama embedding model isn't responding

Write the vector store functions

Now that you know the embedding model works, you need two functions: one to create a vector store from your chunks, and one to reload an existing store from disk.

  • In rag.py, find the end of the chunk_documents function. Add the following section comment and create_vector_store function below it:
# --- Step 3: Create Embeddings and Store in Vector DB ---

def create_vector_store(chunks: list[Document], persist_directory: str = "./chroma_db") -> Chroma:
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=persist_directory,
        collection_name="rag_tutorial",
    )
    print(f"Stored {len(chunks)} chunks in vector database.")
    return vector_store

What does this code do?

  • OllamaEmbeddings(model="nomic-embed-text") creates the embedding model that will convert each chunk into a 768-dimensional vector.
  • Chroma.from_documents() does the heavy lifting in one call. It takes your chunks, generates an embedding for each one, and stores both the text and the vector in ChromaDB.
  • persist_directory tells ChromaDB to save the database to disk at ./chroma_db so your embeddings survive between script runs.
  • collection_name gives your set of embeddings a label inside the database. This lets you store multiple collections in one ChromaDB instance.
  • Add the load_vector_store function directly below create_vector_store:
def load_vector_store(persist_directory: str = "./chroma_db") -> Chroma:
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vector_store = Chroma(
        persist_directory=persist_directory,
        embedding_function=embeddings,
        collection_name="rag_tutorial",
    )
    return vector_store

Why do you need a separate load function?

create_vector_store generates embeddings for every chunk from scratch. That takes time (especially on CPU). Once the embeddings are saved to disk, load_vector_store reopens the existing database without re-computing anything. This is how you'll use the vector store in future steps.

  • Save rag.py.

Seeing a syntax error on save?

Check that the load_vector_store function is at the same indentation level as create_vector_store (both should be top-level functions, not nested inside anything).

Make sure you have a blank line between the two functions.

help me fix a syntax error in my rag.py vector store functions

✔️ Awesome, I've got everything!

Great. Double check you've saved rag.py with both new functions.

ⓧ I'd like to double check the full code

Here is the complete rag.py file so far:

import requests
import bs4
import chromadb
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings, OllamaLLM
from langchain_chroma import Chroma


# --- Step 2: Load and Chunk Documents ---

def load_web_page(url: str) -> list[Document]:
    response = requests.get(url)
    response.raise_for_status()
    soup = bs4.BeautifulSoup(response.text, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "header"]):
        tag.decompose()
    text = soup.get_text(separator="\n", strip=True)
    return [Document(page_content=text, metadata={"source": url})]


def chunk_documents(docs: list[Document]) -> list[Document]:
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        add_start_index=True,
    )
    all_splits = text_splitter.split_documents(docs)
    print(f"Split into {len(all_splits)} chunks.")
    return all_splits


# --- Step 3: Create Embeddings and Store in Vector DB ---

def create_vector_store(chunks: list[Document], persist_directory: str = "./chroma_db") -> Chroma:
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=persist_directory,
        collection_name="rag_tutorial",
    )
    print(f"Stored {len(chunks)} chunks in vector database.")
    return vector_store


def load_vector_store(persist_directory: str = "./chroma_db") -> Chroma:
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vector_store = Chroma(
        persist_directory=persist_directory,
        embedding_function=embeddings,
        collection_name="rag_tutorial",
    )
    return vector_store
Test retrieval with a similarity search

Your functions are written. Now run the full pipeline: load the document, chunk it, store the embeddings, and then query the vector store to confirm retrieval works.

  • Run this test script in your terminal to build the vector store and perform a similarity search:
python3 -c "
from rag import load_web_page, chunk_documents, create_vector_store

# Load and chunk the document
url = 'https://lilianweng.github.io/posts/2023-06-23-agent/'
docs = load_web_page(url)
chunks = chunk_documents(docs)

# Store in vector database
vector_store = create_vector_store(chunks)

# Test similarity search
results = vector_store.similarity_search('What are the components of an LLM-powered agent?', k=3)
print(f'\nRetrieved {len(results)} chunks')
for i, doc in enumerate(results):
    print(f'\n--- Chunk {i+1} (first 200 chars) ---')
    print(doc.page_content[:200])
"

What is happening here?

  • The script loads the web page, splits it into chunks, then calls your create_vector_store function to embed and store everything in ChromaDB.
  • similarity_search() converts your query into a vector, then finds the 3 chunks whose vectors are closest to it. The k=3 parameter controls how many results to return.
  • The results should be chunks that discuss LLM agent components (planning, memory, tools). If they are, your embeddings are capturing meaning correctly.
  • Confirm that you see output showing the number of chunks stored and 3 retrieved chunks that are relevant to the query about LLM agent components.

This may take 1-2 minutes on CPU since it needs to generate an embedding for every chunk.

Seeing a timeout or connection error?

Embedding all chunks puts sustained load on Ollama. If the process hangs or times out, make sure no other Ollama requests are running in parallel.

If you see a ModuleNotFoundError, confirm your virtual environment is activated. You should see its name in parentheses in your terminal prompt.

help me debug why my vector store creation is failing

What is a vector database?

A vector database is a database optimized for storing and searching vectors (lists of numbers). Traditional databases find rows by exact matching (WHERE name = 'Alice'). Vector databases find rows by similarity. You ask "what's closest to this vector?" and it returns the nearest neighbors.

This is what makes RAG possible. Instead of searching for exact keywords in your documents, you search for meaning. A question about "agent components" finds chunks about "planning, memory, and tools" even if those exact words aren't in the query.

Your documents are now embedded and searchable. Next up, you'll connect this retrieval to the LLM so it can generate answers grounded in the context it finds.

Build the RAG Chain

You now have a vector database full of document chunks, each stored as a numerical embedding. The missing piece is connecting retrieval to generation.

When a user asks a question, a RAG system finds the most relevant chunks from the database and feeds them to the LLM alongside the question. This gives the model grounded context instead of forcing it to guess from memory.

In this step, get ready to:
  • Write a retrieval function that searches the vector store for relevant chunks.
  • Write a generation function that sends retrieved context to the LLM with instructions.
  • Combine both into a single query function and test with multiple questions.
Write the retrieve_context function

The retrieval function is the first half of RAG. It takes a user question, performs a similarity search against the vector store, and returns the most relevant chunks formatted as a string.

  • In rag.py, scroll to the bottom of the file (below the load_vector_store function).
  • Add the following section comment and function:
# --- Step 4: Build the RAG Chain ---

def retrieve_context(vector_store: Chroma, query: str, k: int = 3) -> str:
    results = vector_store.similarity_search(query, k=k)
    context = "\n\n---\n\n".join(
        f"[Chunk {i+1}] {doc.page_content}"
        for i, doc in enumerate(results)
    )
    return context

What does this code do?

similarity_search(query, k=k) converts your query into an embedding and finds the 3 closest chunks in the vector database.

The results are formatted as a single string with each chunk labeled [Chunk 1], [Chunk 2], etc., separated by --- dividers. This makes it easy for the LLM to distinguish between different pieces of context.

  • Save rag.py.
  • You won't see a change yet. The next function will use this retrieved context to generate an answer.
Write the generate_answer function

The generation function is the second half of RAG. It constructs a prompt that combines a system instruction, the retrieved context, and the user's question, then sends it to the LLM.

  • Below the retrieve_context function in rag.py, add the following function:
def generate_answer(llm: OllamaLLM, query: str, context: str) -> str:
    prompt = f"""You are a helpful assistant. Answer the question based ONLY on the provided context.
If the context does not contain enough information to answer, say "I don't have enough information to answer that."

Context:
{context}

Question: {query}

Answer:"""
    return llm.invoke(prompt)

What does this code do?

The prompt has three parts: a system instruction telling the LLM to only use the provided context, the retrieved context itself, and the user's question.

The instruction to say "I don't have enough information" is critical. Without it, the LLM will happily hallucinate answers even when the context contains nothing relevant. This is what makes RAG trustworthy.

Why restrict the LLM to context only?

An unrestricted LLM uses its training data to answer questions. That data can be outdated, wrong, or made up. By forcing the model to answer only from retrieved context, you guarantee the answer is grounded in your actual documents.

This is the core value proposition of RAG: accurate answers from your data, not the model's memory.

  • Save rag.py.
  • You won't see a change yet. The next function combines retrieval and generation into a single call.
Combine both and test with questions

Now you will wire retrieval and generation together into one function, then test it to see the difference RAG makes.

  • Below the generate_answer function in rag.py, add this function:
def rag_query(vector_store: Chroma, llm: OllamaLLM, query: str) -> str:
    context = retrieve_context(vector_store, query)
    answer = generate_answer(llm, query, context)
    return answer

What does this code do?

rag_query is the clean interface to your entire RAG pipeline. Give it a vector store, an LLM, and a question. It retrieves relevant context and generates a grounded answer in one call.

  • Save rag.py.
  • Test your RAG pipeline by running this command in your terminal:
python3 << 'EOF'
from rag import load_vector_store, rag_query
from langchain_ollama import OllamaLLM

vs = load_vector_store()
llm = OllamaLLM(model="llama3.2")

# Test 1: Topic covered in the document
print("Q1: What are the components of an LLM-powered agent?")
print(rag_query(vs, llm, "What are the components of an LLM-powered agent?"))
print("\n" + "="*50 + "\n")

# Test 2: Another document topic
print("Q2: What is task decomposition?")
print(rag_query(vs, llm, "What is task decomposition?"))
print("\n" + "="*50 + "\n")

# Test 3: Topic NOT in the document (should say it doesn't know)
print("Q3: What is the capital of France?")
print(rag_query(vs, llm, "What is the capital of France?"))
print("\n" + "="*50 + "\n")

# Compare: same question WITHOUT RAG (raw LLM)
print("--- Same question WITHOUT RAG (raw LLM) ---")
print("Q: What are the components of an LLM-powered agent?")
print(llm.invoke("What are the components of an LLM-powered agent?"))
EOF

What should I see?

Questions 1 and 2 should produce detailed, accurate answers grounded in the document. Question 3 (about France) should produce a response like "I don't have enough information to answer that" because the document does not cover that topic.

The final comparison shows the same question answered by the raw LLM without any context. Notice how the raw LLM gives a generic, possibly inaccurate answer, while the RAG version cites specific details from the source document.

Getting an import error?

Make sure your virtual environment is activated. If you see ModuleNotFoundError, your terminal session may have lost the virtual environment.

Confirm you are running the command from the same directory where rag.py lives. The import statement needs to find the file in the current directory.

If the LLM takes a long time to respond, that is normal. The llama3.2 model running on CPU can take 30-60 seconds per answer.

help me debug my RAG query test

✔️ Awesome, I've got everything!

Great. Double check you have saved rag.py with all three new functions.

ⓧ I'd like to double check the full code

Here is the full rag.py file up to this point:

import requests
import bs4
import chromadb
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings, OllamaLLM
from langchain_chroma import Chroma


# --- Step 2: Load and Chunk Documents ---

def load_web_page(url: str) -> list[Document]:
    response = requests.get(url)
    response.raise_for_status()
    soup = bs4.BeautifulSoup(response.text, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "header"]):
        tag.decompose()
    text = soup.get_text(separator="\n", strip=True)
    return [Document(page_content=text, metadata={"source": url})]


def chunk_documents(docs: list[Document]) -> list[Document]:
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        add_start_index=True,
    )
    all_splits = text_splitter.split_documents(docs)
    print(f"Split into {len(all_splits)} chunks.")
    return all_splits


# --- Step 3: Create Embeddings and Store in Vector DB ---

def create_vector_store(chunks: list[Document], persist_directory: str = "./chroma_db") -> Chroma:
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=persist_directory,
        collection_name="rag_tutorial",
    )
    print(f"Stored {len(chunks)} chunks in vector database.")
    return vector_store


def load_vector_store(persist_directory: str = "./chroma_db") -> Chroma:
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vector_store = Chroma(
        persist_directory=persist_directory,
        embedding_function=embeddings,
        collection_name="rag_tutorial",
    )
    return vector_store


# --- Step 4: Build the RAG Chain ---

def retrieve_context(vector_store: Chroma, query: str, k: int = 3) -> str:
    results = vector_store.similarity_search(query, k=k)
    context = "\n\n---\n\n".join(
        f"[Chunk {i+1}] {doc.page_content}"
        for i, doc in enumerate(results)
    )
    return context


def generate_answer(llm: OllamaLLM, query: str, context: str) -> str:
    prompt = f"""You are a helpful assistant. Answer the question based ONLY on the provided context.
If the context does not contain enough information to answer, say "I don't have enough information to answer that."

Context:
{context}

Question: {query}

Answer:"""
    return llm.invoke(prompt)


def rag_query(vector_store: Chroma, llm: OllamaLLM, query: str) -> str:
    context = retrieve_context(vector_store, query)
    answer = generate_answer(llm, query, context)
    return answer

Your RAG pipeline is working. Next up, you will build an evaluation harness that measures how well your retrieval is performing. This is the skill that separates demo builders from production AI engineers.

Evaluate Retrieval Quality

Your RAG pipeline can retrieve context and generate answers. But how do you know the retrieval is actually working well?

In this step, you will build a simple evaluation harness that measures retrieval quality. This is the skill that separates production AI engineers from demo builders.

In this step, get ready to:
  • Write an evaluation function that checks retrieval relevance.
  • Create a golden dataset of test questions with expected keywords.
  • Run the full pipeline end-to-end and measure your retrieval hit rate.
Write the evaluation function

A retrieval hit rate answers one question: did the vector store return chunks that actually contain the information needed to answer? You check this by comparing retrieved text against known keywords from the source document.

  • In rag.py, scroll to the bottom of the file (after the rag_query function).
  • Add the evaluation function by copying this code below the existing functions:
# --- Step 5: Evaluate Retrieval Quality ---

def evaluate_retrieval(vector_store: Chroma, test_cases: list[dict]) -> float:
    hits = 0
    for case in test_cases:
        query = case["question"]
        expected_keywords = case["expected_keywords"]
        results = vector_store.similarity_search(query, k=3)
        retrieved_text = " ".join(doc.page_content for doc in results).lower()
        if any(kw.lower() in retrieved_text for kw in expected_keywords):
            hits += 1
            print(f"  HIT: '{query}'")
        else:
            print(f"  MISS: '{query}'")
    hit_rate = hits / len(test_cases) * 100
    print(f"\nRetrieval hit rate: {hit_rate:.0f}% ({hits}/{len(test_cases)})")
    return hit_rate

What does this code do?

  • For each test case, it runs a similarity search (k=3) and joins the retrieved chunk text into a single lowercase string.
  • It checks whether any of the expected keywords appear in that retrieved text. If at least one keyword is found, it counts as a HIT.
  • The hit rate is the percentage of questions where retrieval found relevant content. This single number tells you whether your chunking and embedding strategy is working.
  • You won't see a change yet. The next chunk adds the main script block that calls this function.
Create the golden dataset and run the full pipeline

A golden dataset is a set of question-answer pairs where you already know the correct answer from reading the source document. You compare retrieval results against these known answers to measure quality.

  • Below the evaluate_retrieval function, add the main script block that ties the entire pipeline together:
# --- Main ---

if __name__ == "__main__":
    url = "https://lilianweng.github.io/posts/2023-06-23-agent/"
    print("Loading document...")
    docs = load_web_page(url)
    print(f"Loaded {len(docs)} document(s) from {url}")

    print("\nChunking...")
    chunks = chunk_documents(docs)

    print("\nCreating vector store...")
    vector_store = create_vector_store(chunks)

    print("\nTesting retrieval...")
    test_query = "What are the components of an LLM-powered agent?"
    context = retrieve_context(vector_store, test_query)
    print(f"Query: {test_query}")
    print(f"Retrieved context (first 300 chars): {context[:300]}...")

    print("\nGenerating answer with RAG...")
    llm = OllamaLLM(model="llama3.2")
    answer = rag_query(vector_store, llm, test_query)
    print(f"\nAnswer: {answer}")

    print("\n--- Evaluation ---")
    test_cases = [
        {
            "question": "What are the components of an LLM-powered agent?",
            "expected_keywords": ["planning", "memory", "tool"],
        },
        {
            "question": "What is chain of thought prompting?",
            "expected_keywords": ["step", "reasoning", "think"],
        },
        {
            "question": "What is task decomposition?",
            "expected_keywords": ["subgoal", "decompos", "smaller"],
        },
        {
            "question": "What types of memory are discussed?",
            "expected_keywords": ["short-term", "long-term"],
        },
        {
            "question": "What tools can agents use?",
            "expected_keywords": ["api", "search", "calculator"],
        },
    ]
    evaluate_retrieval(vector_store, test_cases)

What does this main block do?

This block runs the entire RAG pipeline end-to-end when you execute the script directly.

  • It loads the web page, chunks it, and stores embeddings in ChromaDB.
  • It tests a single retrieval query and generates one RAG answer so you can inspect the output.
  • It defines 5 test cases (the golden dataset) and runs the evaluation. Each test case has a question and a list of keywords you expect to find in the retrieved chunks.
  • Save rag.py.
  • Run the full pipeline by executing this command in your terminal:
python rag.py
  • You should see output that includes the document loading, chunking, a test retrieval, a generated answer, and finally the evaluation results showing HIT or MISS for each question followed by a retrieval hit rate percentage.

What should I see?

The script prints progress as it runs. At the end, you will see something like:

--- Evaluation ---
  HIT: 'What are the components of an LLM-powered agent?'
  HIT: 'What is chain of thought prompting?'
  HIT: 'What is task decomposition?'
  HIT: 'What types of memory are discussed?'
  HIT: 'What tools can agents use?'

Retrieval hit rate: 100% (5/5)

Your exact hit rate may vary. A rate of 60-100% means your chunking and embedding strategy is working well. Below 60% indicates you may need to adjust chunk size or try a different embedding model.

Script taking too long or failing?

  • The script fetches a live web page, generates embeddings for all chunks, and calls the LLM. On CPU this can take 2-5 minutes total. Be patient.
  • If you see a connection error, make sure Ollama is still running. Check with ollama -v in a separate terminal.
  • If you see a requests error about the URL, confirm you have internet access and the URL is reachable in your browser.

Still stuck? Why is my rag.py script failing when I run it?

✔️ Awesome, I've got everything!

Great. Double check you have saved rag.py and that your terminal shows the retrieval hit rate output.

ⓧ I'd like to double check the full code

Here is the complete rag.py file as it should look at the end of this step:

import requests
import bs4
import chromadb
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings, OllamaLLM
from langchain_chroma import Chroma


# --- Step 2: Load and Chunk Documents ---

def load_web_page(url: str) -> list[Document]:
    response = requests.get(url)
    response.raise_for_status()
    soup = bs4.BeautifulSoup(response.text, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "header"]):
        tag.decompose()
    text = soup.get_text(separator="\n", strip=True)
    return [Document(page_content=text, metadata={"source": url})]


def chunk_documents(docs: list[Document]) -> list[Document]:
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        add_start_index=True,
    )
    all_splits = text_splitter.split_documents(docs)
    print(f"Split into {len(all_splits)} chunks.")
    return all_splits


# --- Step 3: Create Embeddings and Store in Vector DB ---

def create_vector_store(chunks: list[Document], persist_directory: str = "./chroma_db") -> Chroma:
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=persist_directory,
        collection_name="rag_tutorial",
    )
    print(f"Stored {len(chunks)} chunks in vector database.")
    return vector_store


def load_vector_store(persist_directory: str = "./chroma_db") -> Chroma:
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vector_store = Chroma(
        persist_directory=persist_directory,
        embedding_function=embeddings,
        collection_name="rag_tutorial",
    )
    return vector_store


# --- Step 4: Build the RAG Chain ---

def retrieve_context(vector_store: Chroma, query: str, k: int = 3) -> str:
    results = vector_store.similarity_search(query, k=k)
    context = "\n\n---\n\n".join(
        f"[Chunk {i+1}] {doc.page_content}"
        for i, doc in enumerate(results)
    )
    return context


def generate_answer(llm: OllamaLLM, query: str, context: str) -> str:
    prompt = f"""You are a helpful assistant. Answer the question based ONLY on the provided context.
If the context does not contain enough information to answer, say "I don't have enough information to answer that."

Context:
{context}

Question: {query}

Answer:"""
    return llm.invoke(prompt)


def rag_query(vector_store: Chroma, llm: OllamaLLM, query: str) -> str:
    context = retrieve_context(vector_store, query)
    answer = generate_answer(llm, query, context)
    return answer


# --- Step 5: Evaluate Retrieval Quality ---

def evaluate_retrieval(vector_store: Chroma, test_cases: list[dict]) -> float:
    hits = 0
    for case in test_cases:
        query = case["question"]
        expected_keywords = case["expected_keywords"]
        results = vector_store.similarity_search(query, k=3)
        retrieved_text = " ".join(doc.page_content for doc in results).lower()
        if any(kw.lower() in retrieved_text for kw in expected_keywords):
            hits += 1
            print(f"  HIT: '{query}'")
        else:
            print(f"  MISS: '{query}'")
    hit_rate = hits / len(test_cases) * 100
    print(f"\nRetrieval hit rate: {hit_rate:.0f}% ({hits}/{len(test_cases)})")
    return hit_rate


# --- Main ---

if __name__ == "__main__":
    url = "https://lilianweng.github.io/posts/2023-06-23-agent/"
    print("Loading document...")
    docs = load_web_page(url)
    print(f"Loaded {len(docs)} document(s) from {url}")

    print("\nChunking...")
    chunks = chunk_documents(docs)

    print("\nCreating vector store...")
    vector_store = create_vector_store(chunks)

    print("\nTesting retrieval...")
    test_query = "What are the components of an LLM-powered agent?"
    context = retrieve_context(vector_store, test_query)
    print(f"Query: {test_query}")
    print(f"Retrieved context (first 300 chars): {context[:300]}...")

    print("\nGenerating answer with RAG...")
    llm = OllamaLLM(model="llama3.2")
    answer = rag_query(vector_store, llm, test_query)
    print(f"\nAnswer: {answer}")

    print("\n--- Evaluation ---")
    test_cases = [
        {
            "question": "What are the components of an LLM-powered agent?",
            "expected_keywords": ["planning", "memory", "tool"],
        },
        {
            "question": "What is chain of thought prompting?",
            "expected_keywords": ["step", "reasoning", "think"],
        },
        {
            "question": "What is task decomposition?",
            "expected_keywords": ["subgoal", "decompos", "smaller"],
        },
        {
            "question": "What types of memory are discussed?",
            "expected_keywords": ["short-term", "long-term"],
        },
        {
            "question": "What tools can agents use?",
            "expected_keywords": ["api", "search", "calculator"],
        },
    ]
    evaluate_retrieval(vector_store, test_cases)

Secret mission

Add Hybrid Search and Source Citations

Your RAG system uses pure vector similarity search. That works well for semantic matches, but it misses exact keywords like product codes, function names, and technical terms. Production RAG systems combine vector search with keyword matching to get the best of both worlds. In this mission, you'll implement hybrid search and add source citations so your answers are both more accurate and more trustworthy.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama service and models (llama3.2, nomic-embed-text).
  • Python virtual environment with installed packages.
  • Project files (setup_test.py, rag.py, rag_hybrid.py).
  • ChromaDB data directories (chroma_db, chroma_db_hybrid).

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with different documents and queries.

  • Your Ollama service stays running in the background and serves models on demand.
  • Your ChromaDB data persists on disk so you can query it again without re-embedding.
  • Your virtual environment stays ready to activate whenever you return.

Pause - I'll come back to this later

Shut down the Ollama service to free up system resources. Your files and models stay on disk for when you return.

  • Stop the Ollama service by running:
  • Deactivate your Python virtual environment by running:
sudo systemctl stop ollama

This stops the background service. Your models remain downloaded and ready.

deactivate

When you come back, start Ollama again with:

sudo systemctl start ollama

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below.

  • Remove the downloaded Ollama models by running:
ollama rm llama3.2
ollama rm nomic-embed-text
  • Uninstall the Ollama service and binary by running:
sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm $(which ollama)
sudo rm -r /usr/share/ollama
sudo userdel ollama
sudo groupdel ollama
  • Deactivate your virtual environment (if active) and delete the project folder by running:
deactivate
cd ~/Desktop
rm -rf rag-project

This removes your Python files, virtual environment, and both ChromaDB data directories.

Nice Work!

Nice Work!

You just built a fully local RAG system from scratch. Everything runs on your machine with no API keys, no cloud costs, and no data leaving your network.

You've learned how to:

  • Built a complete RAG pipeline from scratch, covering document loading, chunking, embedding, vector storage, retrieval, and generation.
  • Learned how embeddings and vector databases work by generating vectors with nomic-embed-text and querying them in ChromaDB.
  • Built an evaluation harness that measures retrieval hit rate, the skill that separates production AI engineers from demo builders.
  • Secret Mission: implemented hybrid search combining vector similarity with BM25 keyword matching, and added source citations for more accurate, trustworthy answers.

Ready to quiz yourself?