Evaluate and Monitor Your RAG Pipeline

Build a golden eval dataset, LLM-as-judge scoring, and user feedback loop.

Introduction

30 Second Summary

You can't improve what you can't measure. When an AI app starts giving worse answers, you need a system that catches the problem before your users do.

In this project, you will build a complete evaluation and monitoring system for your RAG pipeline. You'll create golden test datasets, run automated scoring experiments, collect user feedback, and debug quality regressions using traces.

What You'll Build

A production-grade evaluation loop that tests, scores, and monitors your existing RAG pipeline using Langfuse.

By the end of this project, you'll have:

  • A golden evaluation dataset with 8 curated QA pairs stored in Langfuse for repeatable testing.
  • A working LLM-as-judge evaluation pipeline that scores your RAG responses using your local Ollama model.
  • A user feedback loop integrated into your RAG app with thumbs up/down scores visible on traces in Langfuse.
  • Secret Mission: Set up an annotation queue and compare experiment runs side-by-side to quantify the impact of a retrieval change.

Are there any prerequisites?

This project builds directly on your RAG pipeline from Project 1. You should have Python, Ollama, ChromaDB with loaded documents, and a Langfuse Cloud account with basic tracing already working.

Before We Start

Before We Start...

Upgrade Langfuse SDK and Configure Cost Tracking

Your RAG pipeline from Project 1 is generating responses and sending traces to Langfuse Cloud. But traces alone do not tell you whether those responses are actually good, or how much each query costs.

In this step, you will upgrade the Langfuse SDK to v4 and configure cost tracking for your local Ollama model. This lays the foundation for the evaluation experiments you will build in later steps.

In this step, get ready to:
  • Upgrade the Langfuse Python SDK to v4 and verify the connection.
  • Define a custom model in Langfuse for Ollama cost tracking.
  • Confirm cost metrics appear on traced generations.
Upgrade the Langfuse Python SDK to v4 and verify connection

The Langfuse SDK v4 (released March 2026) introduced a new API surface for tracing, evaluation, and experiments. Your Project 1 code uses v2, so you need to upgrade before building evaluation pipelines.

  • Upgrade the Langfuse package by running this command:
pip install --upgrade langfuse
  • Verify the installed version by running:
pip show langfuse

You should see Version: 4.12.0 (or higher) in the output.

What changed from v2 to v4?

The key changes you will encounter in this project are:

  • Langfuse() becomes get_client() for initializing the client.
  • The @observe() decorator replaces @langfuse.generation() for tracing.
  • from langfuse.openai import OpenAI is the wrapper import for auto-tracing OpenAI-compatible calls.

The full migration guide is at langfuse.com/docs/observability/sdk/upgrade-path/python-v3-to-v4.

Seeing a dependency conflict?

Langfuse v4 requires Python 3.10 or higher. If you see a version conflict, check your Python version with python3 --version.

If you are using a virtual environment from Project 1, activate it before running the upgrade command.

help me resolve a dependency conflict when upgrading langfuse to v4

Now create a script to verify the v4 SDK connects to Langfuse Cloud successfully.

  • Create a new file called verify_setup.py in your project folder and paste the following code:
"""Verify Langfuse SDK v4 connection and setup."""
from langfuse import get_client

langfuse = get_client()

print("Langfuse SDK v4 connection test...")
print(f"Host: {langfuse._langfuse_host if hasattr(langfuse, '_langfuse_host') else 'default (cloud.langfuse.com)'}")

# Create a test score to verify write access
try:
    langfuse.create_score(
        trace_id="test-connection-verify",
        name="setup-test",
        value=1.0,
        data_type="NUMERIC",
        comment="SDK v4 connection verified"
    )
    print("Write access: OK")
except Exception as e:
    print(f"Write test result: {e}")
    print("(This error is expected if trace_id doesn't exist, but confirms API connectivity)")

print("\nSetup verified. Environment variables detected:")
import os
for var in ["LANGFUSE_PUBLIC_KEY", "LANGFUSE_SECRET_KEY", "LANGFUSE_HOST", "LANGFUSE_BASE_URL"]:
    val = os.environ.get(var)
    if val:
        print(f"  {var}: {'*' * 8}...{val[-4:]}")

print("\nReady for Project 2!")

What does this script do?

  • get_client() initializes the Langfuse client using the environment variables you set in Project 1 (LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY).
  • The create_score() call attempts a write to the Langfuse API. The trace ID does not exist, so you will see an error message. That error confirms the SDK successfully authenticated and reached the server.
  • The environment variable loop shows which credentials are detected without revealing the full values.
  • Run the verification script:
python verify_setup.py

You should see your environment variables detected and a write test result confirming API connectivity.

Seeing an authentication error?

Make sure your LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY environment variables are still set from Project 1. Run echo $LANGFUSE_PUBLIC_KEY to check.

If the variables are empty, re-export them from your Langfuse Cloud project settings.

help me debug a Langfuse SDK authentication error

Define a custom model in Langfuse for Ollama cost tracking

Ollama runs locally for free, but defining a model with simulated prices teaches the cost tracking pattern. When Langfuse sees token usage from your Ollama calls, it multiplies by these prices to show USD cost on every trace.

  • Open Langfuse Cloud in your browser and navigate to your project.
  • Click Settings in the left sidebar (the gear icon).
  • Select Models from the settings menu.
  • Click Add custom model definition.
  • Fill in the following fields:
  • Model name: llama3.1:8b
  • Match pattern: (?i)^(llama3.1:8b|llama3.1)$
  • Input price: 0.0001 USD per token.
  • Output price: 0.0001 USD per token.
  • Unit: TOKENS.
  • Save the model definition.

How does cost tracking work?

The match pattern is a regex that Langfuse uses to connect generation observations to this model definition. When your OpenAI wrapper sends a request with model="llama3.1:8b", Langfuse matches it against (?i)^(llama3.1:8b|llama3.1)$ and applies the configured prices.

The prices are simulated (Ollama is free) but they demonstrate the same cost tracking mechanism used with production APIs like OpenAI or Anthropic.

Don't see the Models page?

Make sure you are in Project Settings (not Organization Settings). The Models page lives under your specific project's settings.

help me find the Models page in Langfuse Cloud settings

Verify cost tracking works end-to-end

With the model definition saved, any traced Ollama call will now show cost data. Let's confirm this by running a query through your existing RAG app from Project 1.

  • Run your RAG app from Project 1 and submit a single query (any question about your loaded documents).
  • Wait 10-15 seconds for the trace to appear in Langfuse Cloud.
  • Navigate to Traces in Langfuse Cloud.
  • Click on the most recent trace.
  • Expand the generation observation. You should see token counts (input tokens, output tokens) and a calculated USD cost.
  • Navigate to Dashboard in the left sidebar. Confirm that cost metrics appear in the overview.

What should I see?

The generation observation in your trace should display fields like input tokens, output tokens, and a total cost in USD. The exact amounts depend on how many tokens your query used. Even a small response will show a non-zero cost calculated from the 0.0001 USD/token prices you configured.

No cost showing on the trace?

Make sure your RAG app uses the from langfuse.openai import OpenAI wrapper (not the plain openai package). The wrapper is what reports token usage to Langfuse.

Also verify that the model name in your code matches the regex. If your code uses model="llama3.1:8b", it will match. If it uses a different string, update the match pattern or model name.

help me debug why Langfuse is not showing cost on my traced Ollama generation

✔️ Awesome, I've got everything!

Great. Double check you have saved verify_setup.py and confirmed cost appears on your traces.

ⓧ I'd like to double check the full code

Here is the complete verify_setup.py file:

"""Verify Langfuse SDK v4 connection and setup."""
from langfuse import get_client

langfuse = get_client()

print("Langfuse SDK v4 connection test...")
print(f"Host: {langfuse._langfuse_host if hasattr(langfuse, '_langfuse_host') else 'default (cloud.langfuse.com)'}")

# Create a test score to verify write access
try:
    langfuse.create_score(
        trace_id="test-connection-verify",
        name="setup-test",
        value=1.0,
        data_type="NUMERIC",
        comment="SDK v4 connection verified"
    )
    print("Write access: OK")
except Exception as e:
    print(f"Write test result: {e}")
    print("(This error is expected if trace_id doesn't exist, but confirms API connectivity)")

print("\nSetup verified. Environment variables detected:")
import os
for var in ["LANGFUSE_PUBLIC_KEY", "LANGFUSE_SECRET_KEY", "LANGFUSE_HOST", "LANGFUSE_BASE_URL"]:
    val = os.environ.get(var)
    if val:
        print(f"  {var}: {'*' * 8}...{val[-4:]}")

print("\nReady for Project 2!")

Your SDK is upgraded, your model definition is live, and cost tracking is confirmed. Next up, you will build a golden evaluation dataset to systematically test your RAG pipeline's response quality.

Build a Golden Evaluation Dataset

Your Langfuse SDK is connected and cost tracking is configured. Now you need a way to consistently measure whether your RAG pipeline is actually producing good answers.

A golden evaluation dataset is a curated set of question-and-answer pairs where you already know the correct answer. By running your pipeline against these known-good pairs, you can score its performance objectively and catch regressions over time.

In this step, get ready to:
  • Create a Python script that builds a golden evaluation dataset in Langfuse.
  • Upload 8 curated QA pairs covering multiple categories and difficulty levels.
  • Verify the dataset appears in Langfuse Cloud with all items intact.
Design and create the evaluation dataset

Each item in a golden dataset needs three things: the input question, the expected output (ideal answer), and metadata that lets you slice results later by category or difficulty.

  • Create a new file called create_eval_dataset.py in your project folder.
  • Add the setup code that initializes the Langfuse client and creates the dataset container:
"""Create a golden evaluation dataset in Langfuse for RAG pipeline evaluation."""
from langfuse import get_client

langfuse = get_client()

# Create the dataset
langfuse.create_dataset(
    name="rag-golden-eval",
    description="Golden evaluation dataset for RAG pipeline quality assessment",
    metadata={"version": "1.0", "domain": "general-knowledge"}
)

What does this code do?

The get_client() call initializes the Langfuse client using your environment variables from Step 1.

The create_dataset() call creates a named container in Langfuse Cloud that will hold your evaluation items. The metadata dictionary lets you tag the dataset with version info for tracking changes over time.

Next, define the 8 golden QA pairs that will test your RAG pipeline across different dimensions. Add this below the create_dataset() call:

# Define golden QA pairs
golden_items = [
    {
        "input": {"question": "What is retrieval-augmented generation?"},
        "expected_output": "Retrieval-augmented generation (RAG) is a technique that combines information retrieval with text generation, where relevant documents are first retrieved from a knowledge base and then used as context for a language model to generate accurate, grounded responses.",
        "metadata": {"category": "definition", "difficulty": "easy"}
    },
    {
        "input": {"question": "What are the main components of a RAG pipeline?"},
        "expected_output": "The main components of a RAG pipeline are: a document store or vector database for storing embeddings, an embedding model for converting text to vectors, a retrieval mechanism for finding relevant documents, and a language model for generating responses based on the retrieved context.",
        "metadata": {"category": "architecture", "difficulty": "medium"}
    },
    {
        "input": {"question": "How does ChromaDB store document embeddings?"},
        "expected_output": "ChromaDB stores document embeddings as dense vectors in collections, along with associated metadata and the original document text. It uses approximate nearest neighbor search algorithms to efficiently retrieve similar documents based on vector similarity.",
        "metadata": {"category": "technical", "difficulty": "medium"}
    },
    {
        "input": {"question": "What is the purpose of chunking documents in RAG?"},
        "expected_output": "Chunking documents in RAG serves to break large documents into smaller, semantically meaningful pieces that fit within the context window of the language model. This improves retrieval precision by allowing the system to find and return only the most relevant portions of a document rather than entire documents.",
        "metadata": {"category": "technique", "difficulty": "easy"}
    },
    {
        "input": {"question": "What are common evaluation metrics for RAG systems?"},
        "expected_output": "Common evaluation metrics for RAG systems include: faithfulness (whether the response is grounded in retrieved context), relevance (whether retrieved documents are relevant to the query), answer correctness (whether the response accurately answers the question), and context precision (whether the most relevant documents are ranked highest).",
        "metadata": {"category": "evaluation", "difficulty": "hard"}
    },
    {
        "input": {"question": "What is the difference between sparse and dense retrieval?"},
        "expected_output": "Sparse retrieval uses keyword-based methods like TF-IDF or BM25 that represent documents as sparse vectors with most values being zero. Dense retrieval uses neural embeddings that represent documents as dense vectors in a continuous space, capturing semantic meaning rather than exact keyword matches.",
        "metadata": {"category": "retrieval", "difficulty": "medium"}
    },
    {
        "input": {"question": "Why might a RAG system hallucinate despite having retrieval?"},
        "expected_output": "A RAG system might still hallucinate because: the retrieved context may not contain the answer, the model may ignore the context and rely on its parametric knowledge, the retrieval may return irrelevant documents, or the model may misinterpret or incorrectly synthesize information from multiple retrieved passages.",
        "metadata": {"category": "failure-modes", "difficulty": "hard"}
    },
    {
        "input": {"question": "What role does the embedding model play in RAG?"},
        "expected_output": "The embedding model converts text (both documents and queries) into dense vector representations that capture semantic meaning. These vectors enable similarity-based retrieval by allowing the system to find documents whose embeddings are closest to the query embedding in vector space.",
        "metadata": {"category": "architecture", "difficulty": "easy"}
    },
]

What does this dataset cover?

The 8 items span 7 categories: definition, architecture, technical, technique, evaluation, retrieval, and failure-modes. Difficulty levels range from easy (3 items) through medium (3 items) to hard (2 items).

This variety is deliberate. Easy definition questions test basic retrieval. Hard failure-mode questions test whether the pipeline can synthesize nuanced answers from multiple documents. The metadata lets you filter experiment results later to pinpoint exactly where your pipeline struggles.

Finally, add the upload loop that sends each item to Langfuse. Add this below the golden_items list:

# Add items to the dataset
for item in golden_items:
    langfuse.create_dataset_item(
        dataset_name="rag-golden-eval",
        input=item["input"],
        expected_output=item["expected_output"],
        metadata=item["metadata"]
    )

print(f"Created dataset 'rag-golden-eval' with {len(golden_items)} items")
print("View in Langfuse Cloud: Datasets -> rag-golden-eval")

What does this code do?

The create_dataset_item() call uploads each QA pair to the dataset you created. It accepts input (the question), expected_output (the ideal answer), and metadata (category and difficulty tags).

Langfuse stores these items server-side, so any experiment you run later can pull them automatically without re-defining the data locally.

  • Save create_eval_dataset.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before running it.

ⓧ I'd like to double check the full code

"""Create a golden evaluation dataset in Langfuse for RAG pipeline evaluation."""
from langfuse import get_client

langfuse = get_client()

# Create the dataset
langfuse.create_dataset(
    name="rag-golden-eval",
    description="Golden evaluation dataset for RAG pipeline quality assessment",
    metadata={"version": "1.0", "domain": "general-knowledge"}
)

# Define golden QA pairs
golden_items = [
    {
        "input": {"question": "What is retrieval-augmented generation?"},
        "expected_output": "Retrieval-augmented generation (RAG) is a technique that combines information retrieval with text generation, where relevant documents are first retrieved from a knowledge base and then used as context for a language model to generate accurate, grounded responses.",
        "metadata": {"category": "definition", "difficulty": "easy"}
    },
    {
        "input": {"question": "What are the main components of a RAG pipeline?"},
        "expected_output": "The main components of a RAG pipeline are: a document store or vector database for storing embeddings, an embedding model for converting text to vectors, a retrieval mechanism for finding relevant documents, and a language model for generating responses based on the retrieved context.",
        "metadata": {"category": "architecture", "difficulty": "medium"}
    },
    {
        "input": {"question": "How does ChromaDB store document embeddings?"},
        "expected_output": "ChromaDB stores document embeddings as dense vectors in collections, along with associated metadata and the original document text. It uses approximate nearest neighbor search algorithms to efficiently retrieve similar documents based on vector similarity.",
        "metadata": {"category": "technical", "difficulty": "medium"}
    },
    {
        "input": {"question": "What is the purpose of chunking documents in RAG?"},
        "expected_output": "Chunking documents in RAG serves to break large documents into smaller, semantically meaningful pieces that fit within the context window of the language model. This improves retrieval precision by allowing the system to find and return only the most relevant portions of a document rather than entire documents.",
        "metadata": {"category": "technique", "difficulty": "easy"}
    },
    {
        "input": {"question": "What are common evaluation metrics for RAG systems?"},
        "expected_output": "Common evaluation metrics for RAG systems include: faithfulness (whether the response is grounded in retrieved context), relevance (whether retrieved documents are relevant to the query), answer correctness (whether the response accurately answers the question), and context precision (whether the most relevant documents are ranked highest).",
        "metadata": {"category": "evaluation", "difficulty": "hard"}
    },
    {
        "input": {"question": "What is the difference between sparse and dense retrieval?"},
        "expected_output": "Sparse retrieval uses keyword-based methods like TF-IDF or BM25 that represent documents as sparse vectors with most values being zero. Dense retrieval uses neural embeddings that represent documents as dense vectors in a continuous space, capturing semantic meaning rather than exact keyword matches.",
        "metadata": {"category": "retrieval", "difficulty": "medium"}
    },
    {
        "input": {"question": "Why might a RAG system hallucinate despite having retrieval?"},
        "expected_output": "A RAG system might still hallucinate because: the retrieved context may not contain the answer, the model may ignore the context and rely on its parametric knowledge, the retrieval may return irrelevant documents, or the model may misinterpret or incorrectly synthesize information from multiple retrieved passages.",
        "metadata": {"category": "failure-modes", "difficulty": "hard"}
    },
    {
        "input": {"question": "What role does the embedding model play in RAG?"},
        "expected_output": "The embedding model converts text (both documents and queries) into dense vector representations that capture semantic meaning. These vectors enable similarity-based retrieval by allowing the system to find documents whose embeddings are closest to the query embedding in vector space.",
        "metadata": {"category": "architecture", "difficulty": "easy"}
    },
]

# Add items to the dataset
for item in golden_items:
    langfuse.create_dataset_item(
        dataset_name="rag-golden-eval",
        input=item["input"],
        expected_output=item["expected_output"],
        metadata=item["metadata"]
    )

print(f"Created dataset 'rag-golden-eval' with {len(golden_items)} items")
print("View in Langfuse Cloud: Datasets -> rag-golden-eval")
Run the script and verify in Langfuse
  • Execute the dataset creation script by running:
python create_eval_dataset.py
  • You should see output confirming all 8 items were created:
Created dataset 'rag-golden-eval' with 8 items
View in Langfuse Cloud: Datasets -> rag-golden-eval

Seeing an authentication error?

Make sure your LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY environment variables are still set in your current terminal session. If you opened a new terminal since Step 1, you may need to export them again.

help me debug a Langfuse authentication error when running my dataset creation script

  • Open Langfuse Cloud in your browser.
  • Navigate to Datasets in the left sidebar.
  • Click on rag-golden-eval.
  • Verify all 8 items appear in the list. Each item shows its question as the input and the expected answer as the expected output.

What should I see?

The dataset detail page shows a table with 8 rows. Each row displays the input question, the expected output text, and the metadata (category and difficulty). You can click into any item to see the full content.

Understand dataset design principles

Why these specific questions?

The dataset deliberately covers edge cases that reveal pipeline weaknesses. Easy definition questions (like "What is RAG?") test basic retrieval accuracy. Hard questions about failure modes and evaluation metrics test whether the pipeline can synthesize nuanced answers from multiple documents.

The metadata categories let you filter experiment results to find patterns. If your pipeline consistently fails on "failure-modes" questions but passes "definition" questions, you know retrieval depth is the issue, not basic matching.

Langfuse also supports a source_trace_id parameter on dataset items, which lets you pull real production traces into datasets later. This is how teams evolve from hand-written golden sets to production-informed evaluation.

Your golden evaluation dataset is ready. Next up, you will run offline evaluations against this dataset using custom evaluators, including an LLM-as-judge powered by your local Ollama model.

Run Offline Evaluations with LLM-as-Judge

Your golden dataset is ready in Langfuse with 8 curated QA pairs. Now you need a way to automatically score your RAG pipeline's responses against those expected answers.

In this step, you will build an evaluation script that runs your entire RAG pipeline against the golden dataset and scores each response with three different evaluators. One of those evaluators uses your local Ollama model as a judge to rate correctness, completeness, and relevance.

In this step, get ready to:
  • Write three item-level evaluators and one run-level evaluator.
  • Wire your RAG pipeline as an experiment task that retrieves from ChromaDB and generates with Ollama.
  • Run the baseline experiment and interpret scores in Langfuse.
Write custom evaluators

Langfuse evaluators are Python functions that receive the input, output, and expected output for each dataset item, then return an Evaluation object with a name, numeric value, and comment. You will write three item-level evaluators that score individual responses, plus one run-level evaluator that computes an aggregate metric across all items.

  • Create a new file called run_evaluation.py in your project folder.
  • Add the imports and client initialization by pasting this code at the top of the file:
"""Run offline evaluation experiments on the RAG pipeline with custom evaluators."""
import json
from langfuse import get_client, Evaluation
from langfuse.openai import OpenAI
import chromadb

langfuse = get_client()

# Initialize OpenAI-compatible client pointing to Ollama
ollama_client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

# Initialize ChromaDB
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_collection("documents")

What does this code do?

This sets up three clients your evaluation script needs. get_client() connects to Langfuse Cloud using your environment variables. The OpenAI wrapper points at your local Ollama server so that every generation is automatically traced in Langfuse. Finally, ChromaDB connects to the same document collection your RAG pipeline uses.

The Evaluation import is the return type every evaluator function must produce. It contains a name, a numeric value (the score), and an optional comment.

  • Below the setup code, add the first evaluator. This one checks keyword overlap between the RAG response and the expected answer:

# --- Evaluators ---

def contains_key_concepts(*, input, output, expected_output, **kwargs):
    """Check if the output contains key concepts from the expected answer."""
    if not expected_output or not output:
        return Evaluation(
            name="key_concepts", value=0.0, comment="Missing output or expected output"
        )

    expected_words = set(expected_output.lower().split())
    output_words = set(output.lower().split())

    stop_words = {
        "the", "a", "an", "is", "are", "was", "were", "in", "on", "at",
        "to", "for", "of", "and", "or", "that", "this", "it", "with",
        "as", "by", "from", "be", "can", "will", "has", "have", "been",
    }
    expected_keywords = expected_words - stop_words
    output_keywords = output_words - stop_words

    if not expected_keywords:
        return Evaluation(name="key_concepts", value=1.0, comment="No keywords to check")

    overlap = expected_keywords & output_keywords
    score = len(overlap) / len(expected_keywords)

    return Evaluation(
        name="key_concepts",
        value=round(score, 2),
        comment=f"Matched {len(overlap)}/{len(expected_keywords)} key concepts",
    )

What does this evaluator do?

This is a deterministic evaluator. It strips common stop words from both the expected answer and the actual output, then calculates the percentage of expected keywords that appear in the response.

A score of 1.0 means the response contains all key concepts. A score of 0.5 means half are missing. This gives you a quick, repeatable quality signal without needing an LLM call.

  • Below contains_key_concepts, add the LLM-as-judge evaluator. This one calls Ollama to score each response:

def llm_as_judge(*, input, output, expected_output, **kwargs):
    """Use Ollama as an LLM judge to evaluate response quality."""
    if not output:
        return Evaluation(name="llm_judge", value=0.0, comment="No output to evaluate")

    judge_prompt = (
        "You are an evaluation judge. Score the following response on a scale of 0 to 1.\n\n"
        f"Question: {input['question']}\n\n"
        f"Expected Answer: {expected_output}\n\n"
        f"Actual Response: {output}\n\n"
        "Evaluate based on:\n"
        "1. Correctness: Does the response contain accurate information?\n"
        "2. Completeness: Does it cover the key points from the expected answer?\n"
        "3. Relevance: Is the response relevant to the question?\n\n"
        'Respond with ONLY a JSON object: {"score": 0.0, "reasoning": "brief explanation"}'
    )

    judge_response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0.0,
    )

    try:
        result = json.loads(judge_response.choices[0].message.content)
        score = max(0.0, min(1.0, float(result["score"])))
        reasoning = result.get("reasoning", "No reasoning provided")
    except (json.JSONDecodeError, KeyError, ValueError):
        score = 0.5
        reasoning = "Could not parse judge response"

    return Evaluation(name="llm_judge", value=score, comment=reasoning)

How does LLM-as-judge work?

This evaluator sends a structured prompt to your local Llama 3.1 8B model asking it to score the response on correctness, completeness, and relevance. The judge returns a JSON object with a 0-1 score and brief reasoning.

The try/except block handles cases where the judge model doesn't return valid JSON. In those cases it defaults to 0.5 rather than crashing the entire experiment. Setting temperature=0.0 makes the judge as deterministic as possible.

Why use a local model as judge?

In production, teams often use GPT-4 class models as judges for higher reliability. Using your local Ollama model keeps this project completely free while teaching the same LLM-as-judge pattern. The evaluator structure is identical. You only swap the model name if you upgrade later.

  • Add the final two evaluators below llm_as_judge. The first checks response length. The second is a run-level evaluator that computes the average LLM judge score across all items:

def response_length_check(*, input, output, **kwargs):
    """Check that response is neither too short nor too long."""
    if not output:
        return Evaluation(name="response_length", value=0.0, comment="No output")

    word_count = len(output.split())

    if word_count < 10:
        score = 0.3
        comment = f"Too short ({word_count} words)"
    elif word_count > 300:
        score = 0.5
        comment = f"Too long ({word_count} words)"
    else:
        score = 1.0
        comment = f"Good length ({word_count} words)"

    return Evaluation(name="response_length", value=score, comment=comment)


# --- Run-level Evaluator ---

def average_scores(*, item_results, **kwargs):
    """Calculate average LLM judge score across all items."""
    judge_scores = [
        eval.value
        for result in item_results
        for eval in result.evaluations
        if eval.name == "llm_judge" and eval.value is not None
    ]

    if not judge_scores:
        return Evaluation(name="avg_llm_judge", value=None, comment="No scores to average")

    avg = sum(judge_scores) / len(judge_scores)
    return Evaluation(
        name="avg_llm_judge",
        value=round(avg, 2),
        comment=f"Average LLM judge score: {avg:.2%} across {len(judge_scores)} items",
    )

What do these evaluators do?

The response_length_check function flags responses that are suspiciously short (under 10 words) or excessively long (over 300 words). Responses in the acceptable range get a perfect 1.0 score.

The average_scores function is a run-level evaluator. Instead of scoring a single item, it receives all item results and computes the mean LLM judge score across the entire dataset. This aggregate metric attaches to the experiment run in Langfuse for easy comparison.

  • Save run_evaluation.py.
Wire up the RAG pipeline as the experiment task

Langfuse's experiment runner needs a task function that takes a dataset item, runs your pipeline, and returns the output. You will now define your RAG pipeline as that task function, plus a degraded variant you will use for regression testing in a later step.

  • In run_evaluation.py, add the RAG pipeline task function. Place this between the client initialization section and the # --- Evaluators --- comment:

# --- RAG Pipeline Task ---

def rag_pipeline(*, item, **kwargs):
    """The RAG pipeline wrapped as an experiment task."""
    question = item.input["question"]

    # Retrieve relevant documents
    results = collection.query(
        query_texts=[question],
        n_results=3
    )

    context = "\n\n".join(results["documents"][0])

    # Generate response using Ollama
    response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the following context. "
                    "If the context does not contain the answer, say so.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content

What does this task function do?

This is your full RAG pipeline packaged as an experiment task. It takes a dataset item (which has a question in its input), queries ChromaDB for the 3 most relevant documents, joins them as context, and sends everything to Ollama for generation.

The experiment runner calls this function once per dataset item. Because you are using the Langfuse OpenAI wrapper, every generation is automatically traced without extra instrumentation.

  • Directly below rag_pipeline, add the degraded pipeline variant. You will use this in Step 5 for regression testing:

# --- Degraded Pipeline (for Step 5 regression test) ---

def rag_pipeline_degraded(*, item, **kwargs):
    """Degraded RAG pipeline: retrieves only 1 document instead of 3."""
    question = item.input["question"]

    results = collection.query(
        query_texts=[question],
        n_results=1
    )

    context = "\n\n".join(results["documents"][0])

    response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the following context. "
                    "If the context does not contain the answer, say so.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content

Why define a degraded pipeline now?

This variant retrieves only 1 document instead of 3. It simulates a common real-world regression where someone changes a retrieval parameter without evaluating the impact. You will use it in Step 5 to run a comparison experiment, but defining it here keeps all pipeline code in one place.

  • Finally, add the main block at the bottom of the file. This wires everything together with Langfuse's experiment runner:

# --- Main ---

if __name__ == "__main__":
    import sys

    dataset = langfuse.get_dataset("rag-golden-eval")

    if "--degraded" in sys.argv:
        # Step 5: Run degraded experiment for regression comparison
        print("Running DEGRADED experiment (1 document retrieval)...")
        result = dataset.run_experiment(
            name="degraded-retrieval-v1",
            description="Degraded retrieval (1 doc) to demonstrate regression detection",
            task=rag_pipeline_degraded,
            evaluators=[contains_key_concepts, llm_as_judge, response_length_check],
            run_evaluators=[average_scores],
        )
    else:
        # Step 3: Run baseline experiment
        print("Running BASELINE experiment (3 document retrieval)...")
        result = dataset.run_experiment(
            name="baseline-eval-v1",
            description="Baseline evaluation of RAG pipeline with LLM-as-judge",
            task=rag_pipeline,
            evaluators=[contains_key_concepts, llm_as_judge, response_length_check],
            run_evaluators=[average_scores],
        )

    print(result.format())
    print("\nView results: Langfuse Cloud -> Datasets -> rag-golden-eval -> Experiments")

What does the main block do?

The dataset.run_experiment() call is where everything comes together. It fetches your golden dataset from Langfuse, runs the task function on each of the 8 items, then passes each output through all three evaluators.

The experiment runner handles concurrent execution, automatic tracing, and error isolation. If one item fails, the rest still complete. The run_evaluators parameter adds the aggregate average_scores metric to the full run.

The --degraded flag lets you run the degraded variant from the same script. You will use this in Step 5.

  • Save run_evaluation.py.

Seeing an import error?

If you get ImportError: cannot import name 'Evaluation', your Langfuse SDK version may be older than v4. Run pip show langfuse and confirm the version is 4.12.0 or higher.

If you get a ChromaDB collection error, confirm your ./chroma_db folder exists with the documents collection from Project 1.

help me debug an import or collection error in my evaluation script

Run the experiment and interpret results

Your evaluation script is complete. Time to run the baseline experiment against all 8 golden dataset items.

  • Make sure Ollama is running locally, then execute the evaluation script by running:
python run_evaluation.py

What should I expect?

This takes a few minutes because the LLM-as-judge evaluator makes an additional Ollama call for each of the 8 items (16 total LLM calls). You will see a formatted results table in your terminal showing per-item scores and the aggregate avg_llm_judge metric.

  • Once the script completes, open Langfuse Cloud in your browser.
  • Navigate to **Datasets** in the left sidebar.
  • Click **rag-golden-eval**.
  • Click the **Experiments** tab.
  • You should see the baseline-eval-v1 experiment run with scores on each item.
  • Click into any individual item to view its trace. You will see the full retrieval (3 documents) and generation pipeline with all three evaluator scores attached.

Finding weak spots in your pipeline

Look at which items scored lowest on the llm_judge metric. These reveal where your RAG pipeline struggles. Common patterns include low scores on hard difficulty questions or questions where the ChromaDB collection does not contain strong matches.

Experiment not showing in Langfuse?

If the experiment does not appear after 30 seconds, check that your LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY environment variables are set in the terminal session where you ran the script.

If the script ran but some items show errors in Langfuse, that is expected. The experiment runner isolates failures so one broken item does not stop the rest. Check the trace for error details.

help me troubleshoot why my experiment results aren't appearing in Langfuse

✔️ Awesome, I've got everything!

Great. Double check you have saved run_evaluation.py and confirmed the experiment appears in Langfuse.

ⓧ I'd like to double check the full code

Here is the complete run_evaluation.py file. Note that the pipeline functions appear before the evaluators. Function order does not affect execution in Python.

"""Run offline evaluation experiments on the RAG pipeline with custom evaluators."""
import json
from langfuse import get_client, Evaluation
from langfuse.openai import OpenAI
import chromadb

langfuse = get_client()

# Initialize OpenAI-compatible client pointing to Ollama
ollama_client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

# Initialize ChromaDB
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_collection("documents")


# --- RAG Pipeline Task ---

def rag_pipeline(*, item, **kwargs):
    """The RAG pipeline wrapped as an experiment task."""
    question = item.input["question"]

    # Retrieve relevant documents
    results = collection.query(
        query_texts=[question],
        n_results=3
    )

    context = "\n\n".join(results["documents"][0])

    # Generate response using Ollama
    response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the following context. "
                    "If the context does not contain the answer, say so.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content


# --- Degraded Pipeline (for Step 5 regression test) ---

def rag_pipeline_degraded(*, item, **kwargs):
    """Degraded RAG pipeline: retrieves only 1 document instead of 3."""
    question = item.input["question"]

    results = collection.query(
        query_texts=[question],
        n_results=1
    )

    context = "\n\n".join(results["documents"][0])

    response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the following context. "
                    "If the context does not contain the answer, say so.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content


# --- Evaluators ---

def contains_key_concepts(*, input, output, expected_output, **kwargs):
    """Check if the output contains key concepts from the expected answer."""
    if not expected_output or not output:
        return Evaluation(
            name="key_concepts", value=0.0, comment="Missing output or expected output"
        )

    expected_words = set(expected_output.lower().split())
    output_words = set(output.lower().split())

    stop_words = {
        "the", "a", "an", "is", "are", "was", "were", "in", "on", "at",
        "to", "for", "of", "and", "or", "that", "this", "it", "with",
        "as", "by", "from", "be", "can", "will", "has", "have", "been",
    }
    expected_keywords = expected_words - stop_words
    output_keywords = output_words - stop_words

    if not expected_keywords:
        return Evaluation(name="key_concepts", value=1.0, comment="No keywords to check")

    overlap = expected_keywords & output_keywords
    score = len(overlap) / len(expected_keywords)

    return Evaluation(
        name="key_concepts",
        value=round(score, 2),
        comment=f"Matched {len(overlap)}/{len(expected_keywords)} key concepts",
    )


def llm_as_judge(*, input, output, expected_output, **kwargs):
    """Use Ollama as an LLM judge to evaluate response quality."""
    if not output:
        return Evaluation(name="llm_judge", value=0.0, comment="No output to evaluate")

    judge_prompt = (
        "You are an evaluation judge. Score the following response on a scale of 0 to 1.\n\n"
        f"Question: {input['question']}\n\n"
        f"Expected Answer: {expected_output}\n\n"
        f"Actual Response: {output}\n\n"
        "Evaluate based on:\n"
        "1. Correctness: Does the response contain accurate information?\n"
        "2. Completeness: Does it cover the key points from the expected answer?\n"
        "3. Relevance: Is the response relevant to the question?\n\n"
        'Respond with ONLY a JSON object: {"score": 0.0, "reasoning": "brief explanation"}'
    )

    judge_response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0.0,
    )

    try:
        result = json.loads(judge_response.choices[0].message.content)
        score = max(0.0, min(1.0, float(result["score"])))
        reasoning = result.get("reasoning", "No reasoning provided")
    except (json.JSONDecodeError, KeyError, ValueError):
        score = 0.5
        reasoning = "Could not parse judge response"

    return Evaluation(name="llm_judge", value=score, comment=reasoning)


def response_length_check(*, input, output, **kwargs):
    """Check that response is neither too short nor too long."""
    if not output:
        return Evaluation(name="response_length", value=0.0, comment="No output")

    word_count = len(output.split())

    if word_count < 10:
        score = 0.3
        comment = f"Too short ({word_count} words)"
    elif word_count > 300:
        score = 0.5
        comment = f"Too long ({word_count} words)"
    else:
        score = 1.0
        comment = f"Good length ({word_count} words)"

    return Evaluation(name="response_length", value=score, comment=comment)


# --- Run-level Evaluator ---

def average_scores(*, item_results, **kwargs):
    """Calculate average LLM judge score across all items."""
    judge_scores = [
        eval.value
        for result in item_results
        for eval in result.evaluations
        if eval.name == "llm_judge" and eval.value is not None
    ]

    if not judge_scores:
        return Evaluation(name="avg_llm_judge", value=None, comment="No scores to average")

    avg = sum(judge_scores) / len(judge_scores)
    return Evaluation(
        name="avg_llm_judge",
        value=round(avg, 2),
        comment=f"Average LLM judge score: {avg:.2%} across {len(judge_scores)} items",
    )


# --- Main ---

if __name__ == "__main__":
    import sys

    dataset = langfuse.get_dataset("rag-golden-eval")

    if "--degraded" in sys.argv:
        # Step 5: Run degraded experiment for regression comparison
        print("Running DEGRADED experiment (1 document retrieval)...")
        result = dataset.run_experiment(
            name="degraded-retrieval-v1",
            description="Degraded retrieval (1 doc) to demonstrate regression detection",
            task=rag_pipeline_degraded,
            evaluators=[contains_key_concepts, llm_as_judge, response_length_check],
            run_evaluators=[average_scores],
        )
    else:
        # Step 3: Run baseline experiment
        print("Running BASELINE experiment (3 document retrieval)...")
        result = dataset.run_experiment(
            name="baseline-eval-v1",
            description="Baseline evaluation of RAG pipeline with LLM-as-judge",
            task=rag_pipeline,
            evaluators=[contains_key_concepts, llm_as_judge, response_length_check],
            run_evaluators=[average_scores],
        )

    print(result.format())
    print("\nView results: Langfuse Cloud -> Datasets -> rag-golden-eval -> Experiments")

You now have a repeatable evaluation pipeline that scores your RAG responses across multiple dimensions. Next up, you will add user feedback to your RAG app so real interactions contribute scores alongside these automated evaluations.

Add User Feedback and Review the Monitoring Dashboard

Your baseline experiment gives you a structured view of pipeline quality across 8 questions. But in production, users encounter your RAG app with questions you never anticipated.

Automated scores tell you how the pipeline performs on paper. User feedback tells you how it performs in practice. In this step, you'll close the loop by letting users rate responses directly, then review the full monitoring picture in Langfuse's Dashboard.

In this step, get ready to:
  • Create a RAG app with user feedback collection.
  • Test the feedback loop with live queries.
  • Review the Langfuse monitoring dashboard for cost and score trends.
Add user feedback to the RAG app

Your evaluation script runs offline against a fixed dataset. Now you need a way to capture real-time quality signals from users as they interact with the pipeline.

  • Create a new file called rag_with_feedback.py in your project folder.
  • Add the imports and client initialization by pasting this code:
"""RAG pipeline with user feedback (thumbs up/down) and cost tracking."""
from langfuse import get_client, observe
from langfuse.openai import OpenAI
import chromadb

langfuse = get_client()

# Initialize OpenAI-compatible client pointing to Ollama
ollama_client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

# Initialize ChromaDB
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_collection("documents")

What does this code do?

This sets up the same foundation as your evaluation script: a Langfuse client, an OpenAI-compatible wrapper pointing to Ollama, and a ChromaDB connection.

The key difference is importing observe alongside get_client. The @observe() decorator automatically creates traced observations for each function call.

  • Below the initialization code, add the two pipeline helper functions:

@observe()
def retrieve_documents(question: str) -> list:
    """Retrieve relevant documents from ChromaDB."""
    results = collection.query(
        query_texts=[question],
        n_results=3
    )
    return results["documents"][0]


@observe()
def generate_response(question: str, context: str) -> str:
    """Generate a response using Ollama with retrieved context."""
    response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the following context. "
                    "If the context does not contain the answer, say so.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )
    return response.choices[0].message.content

What does this code do?

Each function is decorated with @observe(), which means Langfuse automatically creates a child observation (span) every time the function runs. This gives you a trace tree showing retrieval time separately from generation time.

The OpenAI wrapper from langfuse.openai also auto-captures token usage on the generation call, which feeds into cost tracking from your custom model definition.

  • Now add the core function that ties retrieval, generation, and feedback collection together:

@observe(name="rag-with-feedback")
def handle_query(question: str) -> str:
    """Full RAG pipeline with feedback collection in a single trace."""
    # Retrieve and generate
    documents = retrieve_documents(question)
    context = "\n\n".join(documents)
    response = generate_response(question, context)

    # Display response
    print(f"\nAnswer: {response}")
    print(f"(Sources: {len(documents)} documents retrieved)")

    # Collect feedback while still inside the trace context
    feedback = input("\nWas this helpful? (y/n/skip): ").strip().lower()

    if feedback in ("y", "n"):
        comment = None
        if feedback == "n":
            comment = input("What was wrong? (optional, press Enter to skip): ").strip()
            if not comment:
                comment = None

        langfuse.score_current_trace(
            name="user-feedback",
            value=1 if feedback == "y" else 0,
            data_type="BOOLEAN",
            comment=comment,
        )
        print("Feedback recorded!")

    return response

What does this code do?

This is the most important function in this step. The @observe(name="rag-with-feedback") decorator creates a parent trace that wraps everything: retrieval, generation, and feedback.

The langfuse.score_current_trace() call attaches a BOOLEAN score to the active trace. Value 1 means thumbs up, value 0 means thumbs down. Because the feedback prompt happens inside the @observe() context, the score is automatically linked to the correct trace.

On negative feedback, the user can optionally explain what went wrong. This comment appears alongside the score in Langfuse, making it easy to triage quality issues later.

  • Finally, add the interactive loop that runs the app:

def main():
    """Interactive RAG with feedback collection."""
    print("=" * 50)
    print("RAG Pipeline with User Feedback")
    print("Type 'quit' to exit")
    print("=" * 50)

    while True:
        question = input("\nQuestion: ").strip()
        if question.lower() == "quit":
            break
        if not question:
            continue

        handle_query(question)

    print("\nSession complete. View traces and feedback in Langfuse Cloud.")
    print("Filter traces by 'user-feedback < 1' to find negative feedback.")


if __name__ == "__main__":
    main()

What does this code do?

The main() function creates a simple interactive loop. Each question triggers a full traced RAG query with a feedback prompt. Type quit to exit the session.

  • Save rag_with_feedback.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

"""RAG pipeline with user feedback (thumbs up/down) and cost tracking."""
from langfuse import get_client, observe
from langfuse.openai import OpenAI
import chromadb

langfuse = get_client()

# Initialize OpenAI-compatible client pointing to Ollama
ollama_client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

# Initialize ChromaDB
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_collection("documents")


@observe()
def retrieve_documents(question: str) -> list:
    """Retrieve relevant documents from ChromaDB."""
    results = collection.query(
        query_texts=[question],
        n_results=3
    )
    return results["documents"][0]


@observe()
def generate_response(question: str, context: str) -> str:
    """Generate a response using Ollama with retrieved context."""
    response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the following context. "
                    "If the context does not contain the answer, say so.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )
    return response.choices[0].message.content


@observe(name="rag-with-feedback")
def handle_query(question: str) -> str:
    """Full RAG pipeline with feedback collection in a single trace."""
    # Retrieve and generate
    documents = retrieve_documents(question)
    context = "\n\n".join(documents)
    response = generate_response(question, context)

    # Display response
    print(f"\nAnswer: {response}")
    print(f"(Sources: {len(documents)} documents retrieved)")

    # Collect feedback while still inside the trace context
    feedback = input("\nWas this helpful? (y/n/skip): ").strip().lower()

    if feedback in ("y", "n"):
        comment = None
        if feedback == "n":
            comment = input("What was wrong? (optional, press Enter to skip): ").strip()
            if not comment:
                comment = None

        langfuse.score_current_trace(
            name="user-feedback",
            value=1 if feedback == "y" else 0,
            data_type="BOOLEAN",
            comment=comment,
        )
        print("Feedback recorded!")

    return response


def main():
    """Interactive RAG with feedback collection."""
    print("=" * 50)
    print("RAG Pipeline with User Feedback")
    print("Type 'quit' to exit")
    print("=" * 50)

    while True:
        question = input("\nQuestion: ").strip()
        if question.lower() == "quit":
            break
        if not question:
            continue

        handle_query(question)

    print("\nSession complete. View traces and feedback in Langfuse Cloud.")
    print("Filter traces by 'user-feedback < 1' to find negative feedback.")


if __name__ == "__main__":
    main()
Test the feedback loop

Time to run the app and generate some traced interactions with real feedback attached.

  • Start the interactive RAG app by running:
python rag_with_feedback.py
  • Submit 3-4 questions and give a mix of positive and negative feedback. Try questions like:
  • "What is RAG?" (then press y for thumbs up).
  • "How does chunking work?" (then press y for thumbs up).
  • "What is the capital of France?" (then press n for thumbs down, and enter a comment like "not in documents").
  • "Explain dense retrieval" (then press y for thumbs up).
  • Type quit to exit the app.
  • Open Langfuse Cloud in your browser.
  • Click Traces in the left sidebar.
  • You should see new traces named rag-with-feedback. Each trace shows the full pipeline tree: the parent observation, a retrieve_documents child, and a generate_response child.
  • Click into any trace. Under Scores, verify that thumbs up shows value 1 and thumbs down shows value 0.

Filter for negative feedback

In the Traces list, use the score filter to show traces where user-feedback < 1. This surfaces all negative-feedback interactions in one view, making it easy to triage quality issues.

Don't see any traces?

Make sure Ollama is running locally. The OpenAI wrapper needs Ollama to respond at http://localhost:11434/v1.

Check that your LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY environment variables are still set in your terminal session. If you opened a new terminal, re-export them.

Help me debug why my traces aren't appearing in Langfuse Cloud.

Review the Langfuse monitoring dashboard

With traces and scores flowing in, the Langfuse Dashboard gives you a single-pane view of your pipeline's health over time.

  • In Langfuse Cloud, click Dashboard in the left sidebar.
  • Observe the following metrics:
  • Total traces: the count of all traced interactions (experiments + interactive sessions).
  • Cost over time: each trace shows a USD cost calculated from your custom model definition in Step 1.
  • Latency distribution: shows how long each generation takes.
  • Score trends: the user-feedback score distribution shows overall satisfaction across your sessions.
  • Click into an individual trace to see the cost breakdown per generation observation. The token count and calculated cost come from the llama3.1:8b model definition you created earlier.

What about the Hobby tier limits?

The Langfuse Hobby tier provides 50k units per month and 30 days of data access. Each trace plus its observations and scores counts toward the unit total. A typical interactive session of 4 queries produces roughly 4 traces + 12 observations + 4 scores = 20 units. You have plenty of headroom for this project.

You now have a complete feedback loop: automated evaluations for offline quality checks and user feedback for live production signals. Next up, you'll use both to detect and debug a quality regression.

Debug a Quality Regression

Your evaluation pipeline is running, your dashboard is tracking costs, and user feedback is flowing in. But what happens when something quietly breaks?

In production, regressions rarely announce themselves. Someone changes a retrieval parameter, tweaks a prompt, or updates a dependency. Scores drift downward. In this step, you will simulate a real-world regression, detect it with your evaluation tooling, and trace through Langfuse to pinpoint the root cause.

In this step, get ready to:
  • Run a degraded experiment that simulates a retrieval parameter change.
  • Compare two experiment runs side-by-side in Langfuse to quantify the regression.
  • Trace through a low-scoring item to identify the root cause.
Understand the deliberate regression

A common production failure is someone changing a retrieval parameter without evaluating the downstream impact. Your run_evaluation.py already contains a degraded pipeline that simulates exactly this.

  • In run_evaluation.py, find the rag_pipeline_degraded function you wrote in Step 3:
def rag_pipeline_degraded(*, item, **kwargs):
    """Degraded RAG pipeline: retrieves only 1 document instead of 3."""
    question = item.input["question"]

    results = collection.query(
        query_texts=[question],
        n_results=1
    )

    context = "\n\n".join(results["documents"][0])

    response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the following context. "
                    "If the context does not contain the answer, say so.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content

What makes this a regression?

The only difference between this function and rag_pipeline() is n_results=1 instead of n_results=3. The pipeline retrieves a single document instead of three.

This is a realistic scenario. A developer might reduce retrieval count to speed up latency or reduce token usage, not realizing it degrades answer quality across the board.

The script also already has the --degraded flag wired up in the __main__ block. When you pass this flag, the experiment runner uses rag_pipeline_degraded instead of rag_pipeline.

Run the degraded experiment

Now you will run the degraded pipeline against the same golden dataset to produce a second experiment that you can compare against your baseline.

  • Run the degraded experiment by executing this command in your terminal:
python run_evaluation.py --degraded

What is happening during this run?

The experiment runner is processing all 8 items from your rag-golden-eval dataset. For each item, it retrieves only 1 document from ChromaDB, generates a response with Ollama, and then runs all three evaluators (key concepts, LLM-as-judge, response length).

This creates a new experiment run named degraded-retrieval-v1 in Langfuse with scores attached to every item.

  • Wait for the script to finish. You should see formatted results printed to your terminal showing per-item scores and an aggregate avg_llm_judge score.
  • Notice that the aggregate LLM judge score is lower than what you saw in your baseline run.

Script taking a long time?

Each of the 8 items requires two Ollama calls (one for generation, one for the LLM judge). With Llama 3.1 8B running locally, this can take 5-10 minutes depending on your hardware. Make sure Ollama is still running in the background.

If you see connection errors, confirm Ollama is accessible by running ollama list in a separate terminal window.

Help me debug why my degraded evaluation run is failing.

  • Return to Langfuse Cloud in your browser.
  • Navigate to **Datasets** in the left sidebar.
  • Click on rag-golden-eval.
  • Click the **Experiments** tab.
  • You should now see both baseline-eval-v1 and degraded-retrieval-v1 listed as experiment runs.

What should I see in the comparison?

The experiments view shows both runs side-by-side with their aggregate scores. The degraded run should have noticeably lower scores across all three evaluators, especially key_concepts and llm_judge. This is your first signal that something regressed.

Trace through the regression to find root cause

Scores tell you something is wrong. Traces tell you exactly where it happened. This is the production debugging workflow you will use whenever quality drops.

  • In the degraded-retrieval-v1 experiment, find an item with a low llm_judge score.
  • Click on that item to open its trace.
  • Examine the trace. In the retrieval span, observe that only 1 document was retrieved.
  • Note the limited context that was passed to the generation step.
  • Now navigate back to the experiments list.
  • Open the same question in the baseline-eval-v1 experiment.
  • Click into its trace. Observe that 3 documents were retrieved, providing richer context for the generation.
  • Compare the two answers. The baseline response should be more complete and accurate because it had access to more relevant context.

What is the production debugging workflow?

You just followed the same pattern teams use in production to diagnose quality issues:

  • Scores alert you to a problem (the degraded run has lower aggregate scores).
  • Experiment comparison quantifies the impact (you can see exactly how much scores dropped).
  • Traces show you where the regression happened (the retrieval span reveals only 1 document instead of 3).

Why this approach over manual testing?

Without automated evaluation, a parameter change like n_results=1 might go unnoticed for weeks. The pipeline still runs without errors. Answers still look plausible on a quick spot check.

Golden datasets catch these silent regressions by testing the same questions every time. The comparison view gives you quantitative evidence to revert the change or justify keeping it.

You have now identified the root cause. The n_results=1 parameter change in rag_pipeline_degraded reduced retrieval context, which caused lower quality responses across all 8 evaluation items.

✔️ Awesome, I've got everything!

Great. You have successfully debugged a quality regression using the full Langfuse workflow.

ⓧ I'd like to double check the full code

Here is the complete run_evaluation.py file. No changes were made in this step. The degraded pipeline and --degraded flag were already written in Step 3.

"""Run offline evaluation experiments on the RAG pipeline with custom evaluators."""
import json
from langfuse import get_client, Evaluation
from langfuse.openai import OpenAI
import chromadb

langfuse = get_client()

# Initialize OpenAI-compatible client pointing to Ollama
ollama_client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

# Initialize ChromaDB
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_collection("documents")


# --- RAG Pipeline Task ---

def rag_pipeline(*, item, **kwargs):
    """The RAG pipeline wrapped as an experiment task."""
    question = item.input["question"]

    # Retrieve relevant documents
    results = collection.query(
        query_texts=[question],
        n_results=3
    )

    context = "\n\n".join(results["documents"][0])

    # Generate response using Ollama
    response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the following context. "
                    "If the context does not contain the answer, say so.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content


# --- Degraded Pipeline (for Step 5 regression test) ---

def rag_pipeline_degraded(*, item, **kwargs):
    """Degraded RAG pipeline: retrieves only 1 document instead of 3."""
    question = item.input["question"]

    results = collection.query(
        query_texts=[question],
        n_results=1
    )

    context = "\n\n".join(results["documents"][0])

    response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the following context. "
                    "If the context does not contain the answer, say so.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )

    return response.choices[0].message.content


# --- Evaluators ---

def contains_key_concepts(*, input, output, expected_output, **kwargs):
    """Check if the output contains key concepts from the expected answer."""
    if not expected_output or not output:
        return Evaluation(
            name="key_concepts", value=0.0, comment="Missing output or expected output"
        )

    expected_words = set(expected_output.lower().split())
    output_words = set(output.lower().split())

    stop_words = {
        "the", "a", "an", "is", "are", "was", "were", "in", "on", "at",
        "to", "for", "of", "and", "or", "that", "this", "it", "with",
        "as", "by", "from", "be", "can", "will", "has", "have", "been",
    }
    expected_keywords = expected_words - stop_words
    output_keywords = output_words - stop_words

    if not expected_keywords:
        return Evaluation(name="key_concepts", value=1.0, comment="No keywords to check")

    overlap = expected_keywords & output_keywords
    score = len(overlap) / len(expected_keywords)

    return Evaluation(
        name="key_concepts",
        value=round(score, 2),
        comment=f"Matched {len(overlap)}/{len(expected_keywords)} key concepts",
    )


def llm_as_judge(*, input, output, expected_output, **kwargs):
    """Use Ollama as an LLM judge to evaluate response quality."""
    if not output:
        return Evaluation(name="llm_judge", value=0.0, comment="No output to evaluate")

    judge_prompt = (
        "You are an evaluation judge. Score the following response on a scale of 0 to 1.\n\n"
        f"Question: {input['question']}\n\n"
        f"Expected Answer: {expected_output}\n\n"
        f"Actual Response: {output}\n\n"
        "Evaluate based on:\n"
        "1. Correctness: Does the response contain accurate information?\n"
        "2. Completeness: Does it cover the key points from the expected answer?\n"
        "3. Relevance: Is the response relevant to the question?\n\n"
        'Respond with ONLY a JSON object: {"score": 0.0, "reasoning": "brief explanation"}'
    )

    judge_response = ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0.0,
    )

    try:
        result = json.loads(judge_response.choices[0].message.content)
        score = max(0.0, min(1.0, float(result["score"])))
        reasoning = result.get("reasoning", "No reasoning provided")
    except (json.JSONDecodeError, KeyError, ValueError):
        score = 0.5
        reasoning = "Could not parse judge response"

    return Evaluation(name="llm_judge", value=score, comment=reasoning)


def response_length_check(*, input, output, **kwargs):
    """Check that response is neither too short nor too long."""
    if not output:
        return Evaluation(name="response_length", value=0.0, comment="No output")

    word_count = len(output.split())

    if word_count < 10:
        score = 0.3
        comment = f"Too short ({word_count} words)"
    elif word_count > 300:
        score = 0.5
        comment = f"Too long ({word_count} words)"
    else:
        score = 1.0
        comment = f"Good length ({word_count} words)"

    return Evaluation(name="response_length", value=score, comment=comment)


# --- Run-level Evaluator ---

def average_scores(*, item_results, **kwargs):
    """Calculate average LLM judge score across all items."""
    judge_scores = [
        eval.value
        for result in item_results
        for eval in result.evaluations
        if eval.name == "llm_judge" and eval.value is not None
    ]

    if not judge_scores:
        return Evaluation(name="avg_llm_judge", value=None, comment="No scores to average")

    avg = sum(judge_scores) / len(judge_scores)
    return Evaluation(
        name="avg_llm_judge",
        value=round(avg, 2),
        comment=f"Average LLM judge score: {avg:.2%} across {len(judge_scores)} items",
    )


# --- Main ---

if __name__ == "__main__":
    import sys

    dataset = langfuse.get_dataset("rag-golden-eval")

    if "--degraded" in sys.argv:
        # Step 5: Run degraded experiment for regression comparison
        print("Running DEGRADED experiment (1 document retrieval)...")
        result = dataset.run_experiment(
            name="degraded-retrieval-v1",
            description="Degraded retrieval (1 doc) to demonstrate regression detection",
            task=rag_pipeline_degraded,
            evaluators=[contains_key_concepts, llm_as_judge, response_length_check],
            run_evaluators=[average_scores],
        )
    else:
        # Step 3: Run baseline experiment
        print("Running BASELINE experiment (3 document retrieval)...")
        result = dataset.run_experiment(
            name="baseline-eval-v1",
            description="Baseline evaluation of RAG pipeline with LLM-as-judge",
            task=rag_pipeline,
            evaluators=[contains_key_concepts, llm_as_judge, response_length_check],
            run_evaluators=[average_scores],
        )

    print(result.format())
    print("\nView results: Langfuse Cloud -> Datasets -> rag-golden-eval -> Experiments")

Secret mission

Annotation Queue and Experiment Comparison

You have automated scores from your evaluators. But in production, human judgment is the ground truth. Set up an annotation queue to bring structured human review into your pipeline, then use the experiment comparison view to quantify exactly how much your retrieval regression cost you.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally with a free Langfuse Cloud Hobby tier, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Local Python files: verify_setup.py, create_eval_dataset.py, run_evaluation.py, rag_with_feedback.py
  • Langfuse Cloud dataset: rag-golden-eval (8 items, 2 experiment runs).
  • Langfuse Cloud custom model definition: llama3.1:8b.
  • Langfuse Cloud annotation queue and score config (answer-quality).
  • Langfuse Cloud traces with user-feedback scores.

Keep everything running

No action needed. Choose this if you plan to continue building on this evaluation pipeline or want to reference your experiment results later.

  • Your Langfuse Cloud Hobby tier is free (50k units/month). All datasets, experiments, traces, and scores persist indefinitely on the default retention setting.
  • Your local Python files remain in your project folder ready to rerun.
  • Ollama continues running locally but uses no resources when idle.

Pause - I'll come back to this later

Free up local memory by stopping Ollama. Your Langfuse Cloud data persists on the free tier with no action needed.

  • Stop Ollama by quitting the Ollama application from your system tray or menu bar.
  • Your local files and ChromaDB data remain on disk for when you return.
  • All Langfuse Cloud resources (dataset, experiments, traces, model definition, annotation queue) stay intact at no cost.

Delete - I don't want to use this again

Remove all project resources created in this project. This does not affect resources from Project 1.

  • Delete your local Python files by running this command from your project folder:
  • macOS/Linux:
rm verify_setup.py create_eval_dataset.py run_evaluation.py rag_with_feedback.py
  • Windows:
del verify_setup.py, create_eval_dataset.py, run_evaluation.py, rag_with_feedback.py
  • Delete the custom model definition in Langfuse Cloud. Navigate to Project Settings then Models. Find the llama3.1:8b entry and delete it.
  • Delete traces created during this project. Navigate to Traces in Langfuse Cloud. Select the traces from this project, then choose Delete from the Actions dropdown.
  • Delete dataset items and experiment runs. Navigate to Datasets then click into rag-golden-eval. For each item, click the dots menu next to the item and select Delete.
  • Archive the score config. Navigate to Project Settings then Scores / Evaluation. Click the archive icon next to the answer-quality score config.

Nice Work!

Nice Work!

You just built a complete evaluation and monitoring system for your RAG pipeline. Your pipeline now has automated scoring, human feedback capture, cost tracking, and the ability to detect and debug quality regressions from traces.

You've learned how to:

  • Build a golden evaluation dataset and run automated experiments with an LLM-as-judge powered by your local Ollama model.
  • Capture user feedback (thumbs up/down) in your RAG app and attach scores to traces for filtering and analysis in Langfuse.
  • Debug a quality regression by comparing experiment runs side-by-side and tracing through individual failures to find the root cause.
  • Secret Mission: Set up an annotation queue for structured human review and quantified the impact of a retrieval change using experiment comparison data.

Ready to quiz yourself?