Build a RAG Pipeline with a Local LLM

Build a local RAG pipeline with ChromaDB, Ollama, and Langfuse tracing.

Introduction

30 Second Summary

When you ask a chatbot a question about your company's handbook and it gives you a specific, accurate answer, it's not magic. There's a system working behind the scenes that finds the right information and feeds it to the AI before it responds.

In this project, you will build that exact system from scratch on your own machine. Your pipeline will retrieve relevant information from a document collection, generate answers using a local language model, and trace every step for full observability.

What You'll Build

A local question-answering system that embeds documents into a ChromaDB vector store, retrieves relevant chunks for any question, and generates grounded answers using Llama 3.1 running via Ollama.

By the end of this project, you'll have:

  • A Python script that ingests documents into a ChromaDB vector store using local embeddings.
  • A RAG pipeline that retrieves context and generates answers via a local LLM, with every call traced in Langfuse.
  • A comparison framework showing how different prompts and models affect answer quality, latency, and token usage.
  • Secret Mission: Build a mini-evaluation pipeline that scores prompt variants and compares them in Langfuse.

Are there any prerequisites?

None. This guide includes full setup for all required tools, including Ollama, ChromaDB, and Langfuse. You just need a computer with at least 16GB of RAM to run the local language model.

Set Up Ollama and Pull a Model

Your RAG pipeline needs a local Large Language Model to generate answers. The model is the brain that reads retrieved context and produces human-readable responses.

In this step, you'll install Ollama (a tool that runs LLMs locally) and download Llama 3.1 8B. This gives your pipeline a powerful language model without needing any cloud API keys or paid services.

In this step, get ready to:
  • Install Ollama on Windows.
  • Download the Llama 3.1 8B model.
  • Confirm the OpenAI-compatible API is responding.
Install Ollama on Windows

Ollama is a lightweight tool that downloads, manages, and serves LLMs locally. It exposes an OpenAI-compatible API so other tools can talk to your local models the same way they would talk to cloud-hosted models.

  • Press the Windows key to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Install Ollama by running this command:
irm https://ollama.com/install.ps1 | iex

What does this command do?

This downloads and runs Ollama's official install script. The script downloads the installer, verifies its signature, installs Ollama silently, and updates your session PATH so the ollama command works immediately.

You'll see messages like "Downloading Ollama..." and "Install complete. Run 'ollama' from the command line." when it finishes.

  • Verify Ollama installed successfully by running:
ollama --version

You should see a version number in the output (for example, ollama version is 0.x.x).

✔️ I see a version number

Ollama is installed and ready. Move on to the next substep.

ⓧ Command not found

PowerShell cannot find the ollama command. Try these fixes:

  • Close PowerShell completely and reopen it. The installer updates your PATH, but an already-open session may not pick it up.
  • If reopening does not help, restart your computer and try ollama --version again.
  • If the issue persists, download and run the installer manually from ollama.com/download.
Pull the Llama 3.1 model and test it

Ollama is installed, but it has no models yet. You need to download a model before it can generate anything.

  • Download the Llama 3.1 8B model by running:
ollama pull llama3.1

What is happening during the download?

Ollama downloads the model weights (~4.7GB) and stores them at C:\Users\%username%\.ollama\models. This is a one-time download. Future runs load the model from disk.

The 8B in the name means 8 billion parameters. More parameters generally mean better answers but slower inference and more memory usage.

The download may take several minutes depending on your internet speed. Wait for it to complete before continuing.

  • Test that the model works by running:
ollama run llama3.1 "What is RAG in AI?"

You should see the model generate a multi-sentence response explaining Retrieval-Augmented Generation. The first run may take 10-20 seconds as the model loads into memory.

What just happened?

You asked a local LLM a question and it responded entirely on your machine. No data left your computer. No API key was needed. This is the same model your RAG pipeline will use to generate answers from retrieved context.

  • Press Ctrl+D or type /bye to exit the interactive session if prompted.

Model download stuck or failing?

  • Check your internet connection. The download is ~4.7GB and requires a stable connection.
  • If your terminal seems frozen during the pull, give it a minute. Large downloads sometimes pause between layers.
  • If you see a timeout error, run ollama pull llama3.1 again. The download resumes from where it left off.

Still stuck? Help me troubleshoot why ollama pull llama3.1 is failing on Windows.

Verify the OpenAI-compatible API

Your RAG pipeline will communicate with Ollama through its OpenAI-compatible API endpoint. This means any tool designed for the OpenAI API can talk to your local model without modification.

  • Open a new PowerShell window (press the Windows key, type PowerShell, and press Enter).
  • Confirm the API is responding by running:
curl http://localhost:11434/v1/models

You should see a JSON response listing llama3.1 as an available model. This confirms the API is active at http://localhost:11434/v1/.

Why does this matter?

The /v1/models endpoint follows the OpenAI API standard. In later steps, you'll point the Langfuse OpenAI wrapper at this URL. Langfuse will trace every call automatically because it thinks it's talking to OpenAI, but it's actually hitting your local Ollama server.

Seeing a connection refused error?

  • Ollama starts automatically after installation, but if you see an error, the background service may not be running. Try running ollama serve in a separate PowerShell window to start it manually.
  • Make sure you opened a new PowerShell window for the curl command. The model may still be loaded in the first window's session.

Still having trouble? Help me get the Ollama API responding at localhost:11434.

Your local LLM is installed, tested, and serving an API. Next up, you'll install the Python packages and connect Langfuse so every call to this model is automatically traced.

Install Python Dependencies and Configure Langfuse

Your local LLM is up and running. Now you need the supporting cast: a Python environment with the right packages, and an observability platform to watch what your pipeline is doing under the hood.

In this step, you'll create a clean project workspace, install the libraries that power your RAG pipeline, and connect to Langfuse Cloud so every LLM call is automatically traced.

In this step, get ready to:
  • Create a project directory with a Python virtual environment.
  • Install the RAG pipeline dependencies.
  • Connect the project to Langfuse Cloud for observability.
Create your project directory and install dependencies

Your RAG pipeline needs its own isolated workspace so its packages don't interfere with anything else on your machine. A virtual environment gives you exactly that.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-pipeline
cd rag-pipeline
  • Confirm the folder exists by running:
ls
  • You should see the empty rag-pipeline folder listed on your Desktop.
  • Create a Python virtual environment by running:
python -m venv venv

What is a virtual environment?

A virtual environment is an isolated Python installation that keeps this project's packages separate from your system Python. This prevents version conflicts between projects.

  • Activate the virtual environment by running:
.\venv\Scripts\Activate
  • You should see (venv) appear at the start of your terminal prompt. This confirms you're working inside the virtual environment.
  • Install the project dependencies by running:
pip install langfuse chromadb openai python-dotenv

What do these packages do?

  • langfuse traces every LLM call so you can see prompts, responses, latency, and token usage in a dashboard.
  • chromadb is a vector database that stores and searches document embeddings locally.
  • openai provides the chat completions API format that Ollama is compatible with.
  • python-dotenv loads environment variables from a .env file into Python.
  • You should see Successfully installed messages for all four packages and their dependencies.

Installation failed?

  • Make sure (venv) appears in your terminal prompt. If it doesn't, run .\venv\Scripts\Activate again before retrying the install.
  • If you see a permissions error, make sure you're not running PowerShell as Administrator. A regular user session is what you want.
  • ChromaDB requires Python 3.9 or higher. Run python --version to check your version.

Help me troubleshoot my pip install failing in this RAG pipeline project

Sign up for Langfuse Cloud and create a project

Langfuse is the observability layer for your pipeline. It captures every LLM call your code makes so you can inspect prompts, responses, latency, and token counts without adding print statements everywhere.

  • Open your browser and go to https://cloud.langfuse.com.
  • Click Sign Up and create an account. The Hobby plan is free and does not require a credit card.
  • Once signed in, create a new project. Name it something descriptive like rag-pipeline.
  • Navigate to your project's Settings, then open the API Keys section.
  • Copy both your Public Key (starts with pk-lf-) and Secret Key (starts with sk-lf-). Save them somewhere you can paste from in the next substep.

What is Langfuse?

Langfuse is an observability platform purpose-built for LLM applications. It automatically records every call your pipeline makes, including the full prompt, the model's response, how long it took, and how many tokens were consumed.

The free Hobby plan gives you 50k observation units per month. That is more than enough for this project.

Can't find your API keys?

Langfuse automatically generates API keys when you create a project. If you don't see them, navigate to your project name in the sidebar, then go to Settings and look for the API Keys section.

Help me find my Langfuse API keys in the dashboard

Create your .env file and verify the connection

Your Python code needs those API keys to communicate with Langfuse. Storing them in a .env file keeps secrets out of your source code.

  • Create a new file called .env in your rag-pipeline folder with the following content, replacing the placeholder values with your actual keys:
LANGFUSE_PUBLIC_KEY=pk-lf-your-key-here
LANGFUSE_SECRET_KEY=sk-lf-your-key-here
LANGFUSE_BASE_URL=https://cloud.langfuse.com

What do these variables do?

  • LANGFUSE_PUBLIC_KEY identifies your project to the Langfuse API.
  • LANGFUSE_SECRET_KEY authenticates your requests. Keep this private.
  • LANGFUSE_BASE_URL points to the Langfuse Cloud EU region where your data is stored.
  • Create a test script called test_connection.py in your rag-pipeline folder to verify that Langfuse can read your keys:
"""Test Langfuse connection."""

from dotenv import load_dotenv
from langfuse import get_client

# Load environment variables from .env file
load_dotenv()

# Initialize Langfuse client
langfuse = get_client()

# Check authentication
if langfuse.auth_check():
    print("Langfuse connected successfully.")
else:
    print("Langfuse authentication failed. Check your keys.")

What does this code do?

  • load_dotenv() reads your .env file and loads the keys into environment variables that the Langfuse SDK can find.
  • get_client() initializes the Langfuse client using those environment variables automatically.
  • auth_check() sends a test request to Langfuse Cloud to confirm your keys are valid.
  • Run the test script by running:
python test_connection.py
  • You should see Langfuse connected successfully. printed in your terminal.

Seeing 'authentication failed'?

  • Double-check that your .env file is in the same folder where you're running the script (your rag-pipeline folder).
  • Make sure there are no extra spaces around the = signs in your .env file. The format is KEY=value with no spaces.
  • Verify you copied the full key from Langfuse. Public keys start with pk-lf- and secret keys start with sk-lf-.

Help me debug my Langfuse auth_check() failing

✔️ Awesome, I've got everything!

Great. Make sure you've saved both .env and test_connection.py.

ⓧ I'd like to double check the full code

Here is the complete .env file:

LANGFUSE_PUBLIC_KEY=pk-lf-your-key-here
LANGFUSE_SECRET_KEY=sk-lf-your-key-here
LANGFUSE_BASE_URL=https://cloud.langfuse.com

And here is the complete test_connection.py file:

"""Test Langfuse connection."""

from dotenv import load_dotenv
from langfuse import get_client

# Load environment variables from .env file
load_dotenv()

# Initialize Langfuse client
langfuse = get_client()

# Check authentication
if langfuse.auth_check():
    print("Langfuse connected successfully.")
else:
    print("Langfuse authentication failed. Check your keys.")

Your project workspace is set up and Langfuse is listening for traces. Next, you'll build the knowledge base and load it into a vector database.

Ingest Documents into ChromaDB

Your environment is connected to Langfuse and Ollama is ready to generate text. Now you need something for your RAG pipeline to retrieve from.

In this step, you will create a small knowledge base of text documents and store them in ChromaDB. ChromaDB converts each document into a numerical representation called an embedding, then indexes those embeddings so you can search by meaning instead of exact keywords.

In this step, get ready to:
  • Create a knowledge base of Kubernetes text chunks stored as a Python list.
  • Ingest documents into a ChromaDB vector store with metadata.
  • Query the vector store and verify it returns relevant results.
Create your knowledge base documents

A RAG pipeline needs a collection of text to retrieve from. You will create a Python file that holds 8 short text chunks about Kubernetes fundamentals.

  • Create a new file called knowledge_base.py in your project directory.
  • Paste the following code into knowledge_base.py:
"""Knowledge base documents for the RAG pipeline."""

DOCUMENTS = [
    {
        "id": "k8s-1",
        "text": "Kubernetes is an open-source container orchestration platform that automates deploying, scaling, and managing containerized applications. It was originally designed by Google and is now maintained by the Cloud Native Computing Foundation (CNCF).",
        "metadata": {"source": "k8s-docs", "topic": "overview"},
    },
    {
        "id": "k8s-2",
        "text": "A Pod is the smallest deployable unit in Kubernetes. A Pod represents a single instance of a running process in your cluster and can contain one or more containers that share storage and network resources.",
        "metadata": {"source": "k8s-docs", "topic": "pods"},
    },
    {
        "id": "k8s-3",
        "text": "A Deployment provides declarative updates to Pods. You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate. Deployments manage ReplicaSets which in turn manage Pods.",
        "metadata": {"source": "k8s-docs", "topic": "deployments"},
    },
    {
        "id": "k8s-4",
        "text": "A Service in Kubernetes is an abstraction that defines a logical set of Pods and a policy by which to access them. Services enable loose coupling between dependent Pods. A Service is defined using YAML or JSON, like all Kubernetes objects.",
        "metadata": {"source": "k8s-docs", "topic": "services"},
    },
    {
        "id": "k8s-5",
        "text": "Namespaces provide a mechanism for isolating groups of resources within a single cluster. Names of resources need to be unique within a namespace, but not across namespaces. Namespaces are intended for use in environments with many users spread across multiple teams or projects.",
        "metadata": {"source": "k8s-docs", "topic": "namespaces"},
    },
    {
        "id": "k8s-6",
        "text": "kubectl is the command-line tool for interacting with a Kubernetes cluster. It communicates with the Kubernetes API server. Common commands include kubectl get pods, kubectl apply -f, kubectl describe, and kubectl logs.",
        "metadata": {"source": "k8s-docs", "topic": "kubectl"},
    },
    {
        "id": "k8s-7",
        "text": "A ConfigMap is an API object used to store non-confidential data in key-value pairs. Pods can consume ConfigMaps as environment variables, command-line arguments, or as configuration files in a volume. ConfigMaps allow you to decouple configuration from container images.",
        "metadata": {"source": "k8s-docs", "topic": "configmaps"},
    },
    {
        "id": "k8s-8",
        "text": "Horizontal Pod Autoscaler automatically scales the number of Pods in a Deployment or ReplicaSet based on observed CPU utilization or other custom metrics. The HPA controller periodically checks metrics and adjusts the replica count to match the target utilization.",
        "metadata": {"source": "k8s-docs", "topic": "autoscaling"},
    },
]

What does this code do?

DOCUMENTS is a list of 8 dictionaries, each representing one "chunk" of knowledge about Kubernetes.

Each chunk has three fields: an id for unique identification, a text field with the actual content, and a metadata dictionary with the source and topic. The metadata lets you filter results later by topic or source.

  • Save knowledge_base.py.
  • Confirm the file exists by running:
ls knowledge_base.py

You should see knowledge_base.py listed in the output.

File not found?

Make sure you saved the file inside your project directory (the same folder where your .env file lives). If you are in the wrong directory, use cd to navigate to your project folder before running ls.

help me find my project directory

Write the ingestion script

Now you need a script that takes those documents, converts them into embeddings, and stores them in a searchable index. This is the "ingest" step of a RAG pipeline.

  • Create a new file called ingest.py in your project directory.
  • Add the imports and the create_vector_store() function by pasting this code into ingest.py:
"""Ingest documents into ChromaDB vector store."""

import chromadb
from knowledge_base import DOCUMENTS


def create_vector_store():
    """Create and populate the ChromaDB collection."""
    client = chromadb.Client()

    collection = client.create_collection(
        name="knowledge_base",
        metadata={"hnsw:space": "cosine"},
    )

    collection.add(
        ids=[doc["id"] for doc in DOCUMENTS],
        documents=[doc["text"] for doc in DOCUMENTS],
        metadatas=[doc["metadata"] for doc in DOCUMENTS],
    )

    print(f"Ingested {collection.count()} documents into 'knowledge_base' collection.")
    return client, collection

What does this code do?

chromadb.Client() creates an in-memory vector database. No server or external process needed.

create_collection() sets up a named collection with cosine similarity as the distance metric. Cosine similarity measures how similar two vectors are in direction, regardless of their magnitude.

collection.add() passes the raw text to ChromaDB, which automatically converts each chunk into a 384-dimensional vector using the all-MiniLM-L6-v2 embedding model running locally.

  • Now add the test_query() function and the main execution block below your create_vector_store() function:

def test_query(collection, query_text):
    """Test a query against the collection."""
    results = collection.query(
        query_texts=[query_text],
        n_results=3,
    )

    print(f"\nQuery: {query_text}")
    print("-" * 50)
    for i, (doc, distance) in enumerate(
        zip(results["documents"][0], results["distances"][0])
    ):
        print(f"  Result {i + 1} (distance: {distance:.4f}):")
        print(f"    {doc[:100]}...")
    return results


if __name__ == "__main__":
    client, collection = create_vector_store()
    test_query(collection, "How do I scale my application in Kubernetes?")
    test_query(collection, "What is a Pod?")

What does this code do?

test_query() takes a plain-text question and searches the collection for the 3 most similar chunks. ChromaDB converts your question into a vector using the same embedding model, then finds the closest matches by cosine distance.

The if __name__ == "__main__" block runs two test queries when you execute the file directly. Lower distance scores mean higher similarity.

  • Save ingest.py.

Seeing a syntax error?

Make sure you pasted the second block below the first one in the same file, not inside or overlapping the create_vector_store() function. The test_query function should be at the same indentation level as create_vector_store.

help me fix a syntax error in ingest.py

✔️ Awesome, I've got everything!

Great. Double check you have saved both knowledge_base.py and ingest.py.

ⓧ I'd like to double check the full code

Here is the complete ingest.py file:

"""Ingest documents into ChromaDB vector store."""

import chromadb
from knowledge_base import DOCUMENTS


def create_vector_store():
    """Create and populate the ChromaDB collection."""
    client = chromadb.Client()

    collection = client.create_collection(
        name="knowledge_base",
        metadata={"hnsw:space": "cosine"},
    )

    collection.add(
        ids=[doc["id"] for doc in DOCUMENTS],
        documents=[doc["text"] for doc in DOCUMENTS],
        metadatas=[doc["metadata"] for doc in DOCUMENTS],
    )

    print(f"Ingested {collection.count()} documents into 'knowledge_base' collection.")
    return client, collection


def test_query(collection, query_text):
    """Test a query against the collection."""
    results = collection.query(
        query_texts=[query_text],
        n_results=3,
    )

    print(f"\nQuery: {query_text}")
    print("-" * 50)
    for i, (doc, distance) in enumerate(
        zip(results["documents"][0], results["distances"][0])
    ):
        print(f"  Result {i + 1} (distance: {distance:.4f}):")
        print(f"    {doc[:100]}...")
    return results


if __name__ == "__main__":
    client, collection = create_vector_store()
    test_query(collection, "How do I scale my application in Kubernetes?")
    test_query(collection, "What is a Pod?")
Run the ingestion script and inspect results

Time to see your vector store in action. Running the script will ingest all 8 documents and search them with two test questions.

  • Run the ingestion script by executing this command:
python ingest.py

First run downloads the embedding model

The first time you run this, ChromaDB automatically downloads the all-MiniLM-L6-v2 embedding model (about 80MB). This is a one-time download that gets cached locally. Subsequent runs will be instant.

You should see output showing 8 documents ingested, followed by query results with distance scores. Each result shows the most relevant chunk and a distance score below 1.0.

What just happened under the hood?

Each text chunk was converted into a 384-dimensional vector (a list of 384 numbers) by the all-MiniLM-L6-v2 model running locally on your machine. These vectors capture the meaning of the text, not just the words.

When you queried "How do I scale my application?", ChromaDB converted your question into a vector using the same model. It then used cosine similarity to find the stored vectors closest in direction to your question vector. The autoscaling chunk should have the lowest distance score because its meaning is closest to your question.

Seeing a ModuleNotFoundError?

If you see an error mentioning knowledge_base, make sure both knowledge_base.py and ingest.py are in the same directory. If you see an error about chromadb, confirm your virtual environment is active (you should see its name in your terminal prompt).

help me debug my ingest.py import error

Your documents are now embedded and searchable. Next up, you will connect this vector store to Ollama and build the full RAG pipeline that retrieves context and generates answers with Langfuse tracing.

Build the RAG Pipeline with Langfuse Tracing

You have documents embedded in a vector store and can retrieve relevant chunks. Now you need to close the loop by sending those chunks to an LLM and getting a grounded answer back.

In this step, you will build the full RAG pipeline. It accepts a question, retrieves context from ChromaDB, constructs a prompt with that context, and calls your local Ollama model via the Langfuse OpenAI wrapper. Every LLM call is automatically traced so you can inspect latency, token usage, and prompt/response pairs in your dashboard.

In this step, get ready to:
  • Create the RAG pipeline file with Langfuse and Ollama client initialization.
  • Engineer system prompts and build the retrieve and generate functions.
  • Run test questions through the pipeline and view traces in Langfuse.
Create the pipeline file with Langfuse and Ollama initialization

The RAG pipeline needs three clients working together: Langfuse for observability, an OpenAI-compatible client pointing at Ollama for generation, and ChromaDB for retrieval.

  • Create a new file called rag_pipeline.py in your project folder and add the following imports and initialization code:
"""RAG pipeline with Langfuse tracing."""

import os
from dotenv import load_dotenv
from langfuse.openai import OpenAI
from langfuse import get_client
import chromadb
from knowledge_base import DOCUMENTS

load_dotenv()

# Initialize Langfuse client
langfuse = get_client()

# Verify connection
if langfuse.auth_check():
    print("Langfuse connected successfully.")
else:
    print("Langfuse authentication failed. Check your keys.")
    exit(1)

# Initialize Ollama client via Langfuse OpenAI wrapper
llm_client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",
)

# Initialize ChromaDB and ingest documents
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(
    name="knowledge_base",
    metadata={"hnsw:space": "cosine"},
)
collection.add(
    ids=[doc["id"] for doc in DOCUMENTS],
    documents=[doc["text"] for doc in DOCUMENTS],
    metadatas=[doc["metadata"] for doc in DOCUMENTS],
)

What does this code do?

This sets up three key connections your pipeline needs:

  • from langfuse.openai import OpenAI imports a drop-in replacement for the standard OpenAI client. It behaves identically but automatically traces every call to Langfuse in the background.
  • The llm_client points at your local Ollama server using the OpenAI-compatible API. The api_key='ollama' value is required by the SDK but ignored by Ollama.
  • ChromaDB is initialized in-memory and populated with your knowledge base documents, just like in ingest.py. Each script maintains its own in-memory collection.
  • Save rag_pipeline.py and test the initialization by running:
python rag_pipeline.py

You should see Langfuse connected successfully. printed in your terminal.

Seeing an authentication error?

  • Confirm your .env file has the correct LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY values from your Langfuse project settings.
  • Make sure LANGFUSE_BASE_URL is set to https://cloud.langfuse.com in your .env file.
  • Check that Ollama is still running in the background.

Help me debug my Langfuse authentication error in rag_pipeline.py

Define prompt versions and build the retrieve and generate functions

A RAG system needs two things: a way to find the right context, and a carefully crafted prompt that tells the LLM how to use it. You will define three prompt versions so you can compare their effectiveness later.

  • In rag_pipeline.py, add the following prompt definitions and the retrieve() function below the ChromaDB setup:
# Prompt versions
PROMPTS = {
    "v1": "Answer the user's question based on the provided context.",
    "v2": (
        "You are a technical assistant. Answer the user's question using ONLY "
        "the provided context. If the context does not contain enough information "
        "to answer, say 'I don't have enough information to answer that.' "
        "Cite which part of the context you used. Keep your answer concise (2-3 sentences)."
    ),
    "v3": (
        "You are a technical assistant. Answer the user's question using ONLY "
        "the provided context. If the context does not contain enough information "
        "to answer, say 'I don't have enough information to answer that.'\n\n"
        "Format your answer as:\n"
        "Answer: [your answer]\n"
        "Source: [which context chunk you used]\n\n"
        "Example:\n"
        "Context: Docker is a platform for building and running containers.\n"
        "Question: What is Docker?\n"
        "Answer: Docker is a platform for building and running containers.\n"
        "Source: Context chunk 1"
    ),
}


def retrieve(question, n_results=3):
    """Retrieve relevant documents from ChromaDB."""
    results = collection.query(
        query_texts=[question],
        n_results=n_results,
    )
    return results["documents"][0], results["distances"][0]

What do these prompt versions do?

Each version gets progressively more specific about how the LLM should behave:

  • v1 is a minimal instruction. The model gets almost no guidance on format or behavior.
  • v2 adds constraints: answer only from context, cite sources, stay concise, and admit when you don't know.
  • v3 is a few-shot prompt. It includes a worked example so the model learns the exact output format you expect.

The retrieve() function queries ChromaDB for the top 3 most similar chunks to the question. It returns both the document text and the distance scores.

You won't see a change yet. The next chunk builds the generation function that uses these prompts.

  • Still in rag_pipeline.py, add the generate() function below retrieve():
def generate(question, context_docs, prompt_version="v2", model="llama3.1"):
    """Generate an answer using the LLM with retrieved context."""
    context_str = "\n\n".join(
        [f"Context chunk {i + 1}: {doc}" for i, doc in enumerate(context_docs)]
    )

    system_prompt = PROMPTS[prompt_version]

    user_message = f"Context:\n{context_str}\n\nQuestion: {question}"

    response = llm_client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        langfuse_prompt_version=prompt_version,
        langfuse_tags=["rag-pipeline", f"model:{model}"],
    )

    return response.choices[0].message.content

What does the generate function do?

This function takes the retrieved chunks and constructs a full prompt for the LLM:

  • It formats each chunk with a numbered label (Context chunk 1, Context chunk 2, etc.) so the model can reference them by number.
  • The langfuse_prompt_version parameter tags the trace with which prompt version was used. This lets you filter and compare in the dashboard later.
  • The langfuse_tags parameter adds searchable labels to the trace. You can use these to filter traces by pipeline type or model.
  • Setting temperature=0.1 keeps responses deterministic. For a Q&A system grounded in documents, you want minimal creativity.

You won't see a change yet. The next chunk ties everything together and runs the pipeline.

Run the pipeline and view traces in Langfuse

Now you will add the orchestrator function that chains retrieval and generation together, then run a set of test questions through the full pipeline.

  • Add the ask() function and the __main__ block at the bottom of rag_pipeline.py:
def ask(question, prompt_version="v2", model="llama3.1"):
    """Full RAG pipeline: retrieve then generate."""
    print(f"\nQuestion: {question}")
    print(f"Prompt version: {prompt_version} | Model: {model}")
    print("-" * 50)

    # Retrieve
    docs, distances = retrieve(question)
    print(f"Retrieved {len(docs)} chunks (distances: {[f'{d:.3f}' for d in distances]})")

    # Generate
    answer = generate(question, docs, prompt_version, model)
    print(f"Answer: {answer}")
    return answer


if __name__ == "__main__":
    questions = [
        "What is a Pod in Kubernetes?",
        "How do I scale my application automatically?",
        "What is the difference between a Deployment and a Service?",
        "How do I store configuration in Kubernetes?",
    ]

    for q in questions:
        ask(q, prompt_version="v2", model="llama3.1")
        print()

    # Flush traces to Langfuse
    langfuse.flush()
    print("\nAll traces sent to Langfuse. Check your dashboard!")

What does this code do?

The ask() function is the orchestrator. It calls retrieve() to get relevant chunks, then passes them to generate() for the LLM to produce an answer. It also prints the distance scores so you can see how relevant the retrieved chunks were.

The langfuse.flush() call at the end ensures all buffered traces are sent to the Langfuse server before the script exits. Without it, some traces might be lost if the script finishes before the background sender completes.

  • Save rag_pipeline.py and run the full pipeline:
python rag_pipeline.py

Each question takes a few seconds to process as Ollama generates the response locally. You should see 4 questions answered with retrieved context and distance scores printed for each.

First run might be slow

The first call to Ollama after a fresh start may take 10-20 seconds as the model loads into memory. Subsequent calls will be much faster.

  • Open your Langfuse dashboard at https://cloud.langfuse.com and navigate to the Traces section.
  • You should see 4 new traces, one for each question. Click any trace to inspect the prompt, response, model name, latency, and token count.

Not seeing traces in Langfuse?

  • Make sure the script printed Langfuse connected successfully. at the start. If not, your environment variables are incorrect.
  • Check that you see All traces sent to Langfuse. Check your dashboard! at the end of the output. This confirms langfuse.flush() ran successfully.
  • Traces can take a few seconds to appear in the dashboard. Refresh the page if they are not immediately visible.

Help me troubleshoot why my traces are not appearing in the Langfuse dashboard

✔️ Awesome, I've got everything!

Great. Double check you have saved rag_pipeline.py and that all 4 traces are visible in your Langfuse dashboard.

ⓧ I'd like to double check the full code

Here is the complete rag_pipeline.py file:

"""RAG pipeline with Langfuse tracing."""

import os
from dotenv import load_dotenv
from langfuse.openai import OpenAI
from langfuse import get_client
import chromadb
from knowledge_base import DOCUMENTS

load_dotenv()

# Initialize Langfuse client
langfuse = get_client()

# Verify connection
if langfuse.auth_check():
    print("Langfuse connected successfully.")
else:
    print("Langfuse authentication failed. Check your keys.")
    exit(1)

# Initialize Ollama client via Langfuse OpenAI wrapper
llm_client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",
)

# Initialize ChromaDB and ingest documents
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(
    name="knowledge_base",
    metadata={"hnsw:space": "cosine"},
)
collection.add(
    ids=[doc["id"] for doc in DOCUMENTS],
    documents=[doc["text"] for doc in DOCUMENTS],
    metadatas=[doc["metadata"] for doc in DOCUMENTS],
)

# Prompt versions
PROMPTS = {
    "v1": "Answer the user's question based on the provided context.",
    "v2": (
        "You are a technical assistant. Answer the user's question using ONLY "
        "the provided context. If the context does not contain enough information "
        "to answer, say 'I don't have enough information to answer that.' "
        "Cite which part of the context you used. Keep your answer concise (2-3 sentences)."
    ),
    "v3": (
        "You are a technical assistant. Answer the user's question using ONLY "
        "the provided context. If the context does not contain enough information "
        "to answer, say 'I don't have enough information to answer that.'\n\n"
        "Format your answer as:\n"
        "Answer: [your answer]\n"
        "Source: [which context chunk you used]\n\n"
        "Example:\n"
        "Context: Docker is a platform for building and running containers.\n"
        "Question: What is Docker?\n"
        "Answer: Docker is a platform for building and running containers.\n"
        "Source: Context chunk 1"
    ),
}


def retrieve(question, n_results=3):
    """Retrieve relevant documents from ChromaDB."""
    results = collection.query(
        query_texts=[question],
        n_results=n_results,
    )
    return results["documents"][0], results["distances"][0]


def generate(question, context_docs, prompt_version="v2", model="llama3.1"):
    """Generate an answer using the LLM with retrieved context."""
    context_str = "\n\n".join(
        [f"Context chunk {i + 1}: {doc}" for i, doc in enumerate(context_docs)]
    )

    system_prompt = PROMPTS[prompt_version]

    user_message = f"Context:\n{context_str}\n\nQuestion: {question}"

    response = llm_client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        langfuse_prompt_version=prompt_version,
        langfuse_tags=["rag-pipeline", f"model:{model}"],
    )

    return response.choices[0].message.content


def ask(question, prompt_version="v2", model="llama3.1"):
    """Full RAG pipeline: retrieve then generate."""
    print(f"\nQuestion: {question}")
    print(f"Prompt version: {prompt_version} | Model: {model}")
    print("-" * 50)

    # Retrieve
    docs, distances = retrieve(question)
    print(f"Retrieved {len(docs)} chunks (distances: {[f'{d:.3f}' for d in distances]})")

    # Generate
    answer = generate(question, docs, prompt_version, model)
    print(f"Answer: {answer}")
    return answer


if __name__ == "__main__":
    questions = [
        "What is a Pod in Kubernetes?",
        "How do I scale my application automatically?",
        "What is the difference between a Deployment and a Service?",
        "How do I store configuration in Kubernetes?",
    ]

    for q in questions:
        ask(q, prompt_version="v2", model="llama3.1")
        print()

    # Flush traces to Langfuse
    langfuse.flush()
    print("\nAll traces sent to Langfuse. Check your dashboard!")

Your RAG pipeline is fully operational and instrumented. Next up, you will experiment with different prompt versions and models to see how they affect answer quality and performance.

Experiment with Prompt Engineering and Model Selection

Your RAG pipeline works, but how do you know the prompt you chose is actually the best one?

In production, teams iterate on prompts constantly. A small wording change can dramatically improve answer quality. Similarly, choosing between a larger model and a smaller one involves real trade-offs between speed and accuracy. In this step, you will test your three prompt versions head-to-head, pull a smaller model, and use Langfuse to compare everything.

In this step, get ready to:
  • Test three prompt variations and observe how output quality changes.
  • Pull a smaller model and build a systematic comparison script.
  • Analyze results in Langfuse by filtering traces by prompt version and model.
Test the three prompt variations

Your rag_pipeline.py already contains three prompt versions in the PROMPTS dictionary. Each represents a different prompt engineering strategy.

  • v1 is a minimal instruction with no format constraints.
  • v2 adds explicit boundaries (answer only from context, cite sources, stay concise).
  • v3 includes a few-shot example showing the model the exact output format you expect.

You already ran the pipeline with v2 in the previous step. Now let's generate traces for v1 and v3 so you can compare all three in Langfuse.

  • Run the following command in your terminal to test prompt versions v1 and v3 on the same question:
python -c "from rag_pipeline import ask, langfuse; ask('What is a Pod in Kubernetes?', prompt_version='v1'); ask('What is a Pod in Kubernetes?', prompt_version='v3'); langfuse.flush()"

What does this command do?

This imports the ask() function from your existing pipeline and runs the same question through two different prompt versions. Each call creates a separate trace in Langfuse tagged with the prompt version used.

The langfuse_prompt_version parameter in generate() automatically tags each trace with the prompt version. This is how Langfuse knows which prompt produced which response.

  • Look at the terminal output. You should see two answers to the same question, one from v1 and one from v3.

Notice how v1 tends to give a longer, unstructured answer. The v3 response should follow the exact format from the few-shot example, with an explicit Answer: and Source: line.

Seeing an import error?

Make sure your virtual environment is active and that you are running the command from inside your project directory where rag_pipeline.py lives.

help me debug an import error when running python -c with my rag_pipeline module

Pull a smaller model and build the comparison script

A key decision in any LLM application is model selection. Larger models produce higher-quality answers but use more memory and respond slower. Let's pull a second model so you can measure this trade-off directly.

  • Download the Llama 3.2 3B model by running this command:
ollama pull llama3.2:3b

Why a 3B model?

Llama 3.1 8B has 8 billion parameters and requires about 4.7GB of disk space. Llama 3.2 3B has only 3 billion parameters and is about 2.0GB. Fewer parameters means faster inference but potentially lower answer quality.

By running the same questions through both models, you will see exactly where the smaller model falls short and where it is good enough.

  • Wait for the download to complete. You should see a progress bar followed by a success message.

Download stalling or failing?

Check that Ollama is still running in the background. If the download stalls, press Ctrl+C and run the pull command again. Ollama resumes from where it left off.

help me troubleshoot my ollama pull command that is stalling or erroring

Now let's create a dedicated script that systematically runs the same questions through every model and prompt combination.

  • Create a new file called compare_models.py in your project directory.
  • Add the imports and client setup by pasting this code:
"""Compare models and prompt versions."""

import os
from dotenv import load_dotenv
from langfuse.openai import OpenAI
from langfuse import get_client
import chromadb
from knowledge_base import DOCUMENTS

load_dotenv()

langfuse = get_client()

llm_client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",
)

# Setup ChromaDB
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(
    name="knowledge_base_compare",
    metadata={"hnsw:space": "cosine"},
)
collection.add(
    ids=[doc["id"] for doc in DOCUMENTS],
    documents=[doc["text"] for doc in DOCUMENTS],
    metadatas=[doc["metadata"] for doc in DOCUMENTS],
)

What does this code do?

This sets up the same building blocks as your main pipeline but with a separate ChromaDB collection named knowledge_base_compare. This keeps comparison traces cleanly separated from your earlier pipeline runs.

The langfuse.openai wrapper automatically traces every LLM call without any extra code from you.

  • Add the prompt definitions, questions list, and models list below the setup code:
PROMPTS = {
    "v2": (
        "You are a technical assistant. Answer the user's question using ONLY "
        "the provided context. If the context does not contain enough information "
        "to answer, say 'I don't have enough information to answer that.' "
        "Cite which part of the context you used. Keep your answer concise (2-3 sentences)."
    ),
    "v3": (
        "You are a technical assistant. Answer the user's question using ONLY "
        "the provided context. If the context does not contain enough information "
        "to answer, say 'I don't have enough information to answer that.'\n\n"
        "Format your answer as:\n"
        "Answer: [your answer]\n"
        "Source: [which context chunk you used]\n\n"
        "Example:\n"
        "Context: Docker is a platform for building and running containers.\n"
        "Question: What is Docker?\n"
        "Answer: Docker is a platform for building and running containers.\n"
        "Source: Context chunk 1"
    ),
}

QUESTIONS = [
    "What is a Pod in Kubernetes?",
    "How does autoscaling work?",
    "What are Namespaces used for?",
]

MODELS = ["llama3.1", "llama3.2:3b"]

What does this code do?

The PROMPTS dictionary contains just v2 and v3 for this comparison. These are the two structured prompts worth comparing head-to-head (v1 is too minimal to be useful in production).

The MODELS list defines both models you will compare: the 8B-parameter llama3.1 and the 3B-parameter llama3.2:3b. Combined with two prompts and three questions, this creates 12 total traces.

  • Add the run_comparison() function and the main block at the bottom of the file:
def run_comparison():
    """Run all questions through all model + prompt combinations."""
    for model in MODELS:
        for prompt_version in PROMPTS:
            print(f"\n{'=' * 60}")
            print(f"Model: {model} | Prompt: {prompt_version}")
            print("=" * 60)

            for question in QUESTIONS:
                results = collection.query(
                    query_texts=[question],
                    n_results=3,
                )
                context_docs = results["documents"][0]
                context_str = "\n\n".join(
                    [
                        f"Context chunk {i + 1}: {doc}"
                        for i, doc in enumerate(context_docs)
                    ]
                )

                response = llm_client.chat.completions.create(
                    model=model,
                    messages=[
                        {"role": "system", "content": PROMPTS[prompt_version]},
                        {
                            "role": "user",
                            "content": f"Context:\n{context_str}\n\nQuestion: {question}",
                        },
                    ],
                    temperature=0.1,
                    langfuse_prompt_version=prompt_version,
                    langfuse_tags=["model-comparison", f"model:{model}"],
                )

                answer = response.choices[0].message.content
                print(f"\n  Q: {question}")
                print(f"  A: {answer[:200]}")

    langfuse.flush()
    print("\n\nComparison complete. Check Langfuse to compare traces!")


if __name__ == "__main__":
    run_comparison()

What does this code do?

The run_comparison() function loops through every model, prompt, and question combination. For each one it retrieves context from ChromaDB, constructs the prompt, and calls the LLM.

The langfuse_prompt_version parameter tags each trace with the prompt version. The langfuse_tags parameter adds searchable labels like model-comparison and the specific model name so you can filter traces in the dashboard.

  • Save compare_models.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved compare_models.py before moving on.

ⓧ I'd like to double check the full code

"""Compare models and prompt versions."""

import os
from dotenv import load_dotenv
from langfuse.openai import OpenAI
from langfuse import get_client
import chromadb
from knowledge_base import DOCUMENTS

load_dotenv()

langfuse = get_client()

llm_client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",
)

# Setup ChromaDB
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(
    name="knowledge_base_compare",
    metadata={"hnsw:space": "cosine"},
)
collection.add(
    ids=[doc["id"] for doc in DOCUMENTS],
    documents=[doc["text"] for doc in DOCUMENTS],
    metadatas=[doc["metadata"] for doc in DOCUMENTS],
)

PROMPTS = {
    "v2": (
        "You are a technical assistant. Answer the user's question using ONLY "
        "the provided context. If the context does not contain enough information "
        "to answer, say 'I don't have enough information to answer that.' "
        "Cite which part of the context you used. Keep your answer concise (2-3 sentences)."
    ),
    "v3": (
        "You are a technical assistant. Answer the user's question using ONLY "
        "the provided context. If the context does not contain enough information "
        "to answer, say 'I don't have enough information to answer that.'\n\n"
        "Format your answer as:\n"
        "Answer: [your answer]\n"
        "Source: [which context chunk you used]\n\n"
        "Example:\n"
        "Context: Docker is a platform for building and running containers.\n"
        "Question: What is Docker?\n"
        "Answer: Docker is a platform for building and running containers.\n"
        "Source: Context chunk 1"
    ),
}

QUESTIONS = [
    "What is a Pod in Kubernetes?",
    "How does autoscaling work?",
    "What are Namespaces used for?",
]

MODELS = ["llama3.1", "llama3.2:3b"]


def run_comparison():
    """Run all questions through all model + prompt combinations."""
    for model in MODELS:
        for prompt_version in PROMPTS:
            print(f"\n{'=' * 60}")
            print(f"Model: {model} | Prompt: {prompt_version}")
            print("=" * 60)

            for question in QUESTIONS:
                results = collection.query(
                    query_texts=[question],
                    n_results=3,
                )
                context_docs = results["documents"][0]
                context_str = "\n\n".join(
                    [
                        f"Context chunk {i + 1}: {doc}"
                        for i, doc in enumerate(context_docs)
                    ]
                )

                response = llm_client.chat.completions.create(
                    model=model,
                    messages=[
                        {"role": "system", "content": PROMPTS[prompt_version]},
                        {
                            "role": "user",
                            "content": f"Context:\n{context_str}\n\nQuestion: {question}",
                        },
                    ],
                    temperature=0.1,
                    langfuse_prompt_version=prompt_version,
                    langfuse_tags=["model-comparison", f"model:{model}"],
                )

                answer = response.choices[0].message.content
                print(f"\n  Q: {question}")
                print(f"  A: {answer[:200]}")

    langfuse.flush()
    print("\n\nComparison complete. Check Langfuse to compare traces!")


if __name__ == "__main__":
    run_comparison()
Run the comparison and analyze results in Langfuse

Time to run all the combinations and see the data in your dashboard.

  • Run the comparison script:
python compare_models.py

How long will this take?

The script runs 12 total LLM calls (3 questions x 2 prompts x 2 models). Llama 3.1 8B takes a few seconds per response. Llama 3.2 3B should be noticeably faster per response due to fewer parameters.

Watch the terminal output. You should see answers printed as they complete, grouped by model and prompt version.

  • Wait for the script to finish. You should see the message Comparison complete. Check Langfuse to compare traces! at the end.
  • Switch back to Langfuse in your browser and click **Traces** in the left sidebar.
  • Click the **Tags** filter and select model-comparison to see only the comparison traces.
  • Look at the **Latency** column. Compare response times between llama3.1 and llama3.2:3b traces.
  • Click into individual traces to compare the prompt_version metadata and see how v2 and v3 produce different output formats.

What trade-offs should you see?

The 8B model (llama3.1) typically produces more detailed and accurate answers, but each response takes longer. The 3B model (llama3.2:3b) responds faster and uses less memory, but may miss nuances or produce less precise answers.

For prompt versions, v3 (few-shot) tends to produce more consistently formatted output, while v2 gives the model more freedom in how it structures the response. Both approaches are valid depending on your use case.

Not seeing traces in Langfuse?

Traces can take a few seconds to appear. Refresh the Langfuse page. If traces are still missing, confirm your .env file has the correct LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY values.

help me figure out why my Langfuse traces are not appearing in the dashboard

You now have real data showing how prompts and models behave differently on the same task. Next up, you will take this further in the Secret Mission by building an automated scoring system that measures prompt quality with numbers instead of gut feel.

Secret mission

A/B Test Your Prompts with Metadata Tracking

You've compared prompts by eyeballing answers. But how do you know which prompt is actually better at scale? Build a mini-evaluation pipeline that scores each prompt version automatically and reports a winner in Langfuse.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally (Ollama, ChromaDB, Python), so there are no ongoing cloud costs. The only external service is Langfuse Cloud, which is on the free Hobby plan with no credit card attached.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama installation and downloaded models (llama3.1 at ~4.7GB, llama3.2:3b) stored at C:\Users\%username%\.ollama\models.
  • Python virtual environment and project files on your Desktop.
  • Python packages (langfuse, chromadb, openai, python-dotenv) installed inside the virtual environment.
  • Langfuse Cloud account and project (free Hobby plan, no billing).

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue with the next project in the series.

  • Ollama continues running in the background and serves the local API at http://localhost:11434. It uses minimal resources when idle.
  • Your project files remain on your Desktop ready to run again.
  • Langfuse Cloud retains your traces indefinitely on the free plan (up to 50k units/month).

Pause - I'll come back to this later

Shut down running processes to free up memory while keeping all files and accounts intact.

  • Stop the Ollama background process. Open Task Manager with Ctrl+Shift+Esc, find Ollama in the process list, and click End task.
  • Your project folder, virtual environment, and Langfuse account remain untouched. You can restart Ollama from the Start menu whenever you are ready to continue.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below.

  • Delete your project folder. Open PowerShell and run:
Remove-Item -Recurse -Force "$HOME\Desktop\rag-pipeline"
  • Uninstall Ollama. Open Settings on Windows, then select Apps, then Installed apps. Search for Ollama, click the three-dot menu next to it, and select Uninstall. The uninstaller will ask whether you want to remove downloaded models. Select the checkbox to remove them.
  • Delete the Langfuse Cloud project. Log in to https://cloud.langfuse.com, open your project, navigate to project Settings, scroll to the Danger Zone in the General section, and confirm project deletion.
  • Delete your Langfuse account (optional). Click your user menu in the bottom right, select Account Settings, and choose to delete your account.

Nice Work!

Nice Work!

You just built a fully instrumented RAG pipeline running entirely on your local machine. From document ingestion to vector search to LLM generation to observability, you now understand how production LLM applications are architected.

You've learned how to:

  • Build a complete RAG pipeline from scratch, covering document ingestion into ChromaDB, vector similarity search, prompt construction with retrieved context, and answer generation via a local Ollama model.
  • Apply prompt engineering techniques including minimal instructions, detailed format requirements, and few-shot examples to control LLM output quality.
  • Instrument every LLM call with Langfuse tracing for full observability, then compare Llama 3.1 8B against Llama 3.2 3B on answer quality, latency, and token usage.
  • Secret Mission: Built a mini-evaluation pipeline that scores prompt variants using keyword overlap and attaches quality scores to Langfuse traces for data-driven prompt comparison.

Ready to quiz yourself?