Evaluate a RAG Pipeline with DeepEval

Build a RAG evaluation pipeline using the RAG Triad metrics in DeepEval.

Introduction

30 Second Summary

You ask an AI chatbot a question, and it confidently gives you the wrong answer. The frustrating part is not the mistake itself. It is figuring out whether the system retrieved the wrong information or made something up from good information.

In this project, you will build an evaluation pipeline that measures a RAG system's quality using three independent metrics. You will diagnose exactly where failures happen and learn to fix them systematically.

What You'll Build

A Python-based RAG evaluation pipeline that tests faithfulness, answer relevancy, and contextual relevancy using DeepEval and OpenAI.

By the end of this project, you'll have:

  • A simple RAG system with in-memory document retrieval and OpenAI-powered generation.
  • An evaluation test suite using DeepEval's RAG Triad metrics with pytest-style assertions.
  • A golden dataset of test cases for reproducible, automated evaluation.
  • Secret Mission: Build a custom source citation metric that checks whether your RAG system properly attributes information to specific documents.

Are there any prerequisites?

You should have basic Python knowledge and a conceptual understanding of what RAG (Retrieval Augmented Generation) is. You do not need prior experience with evaluation frameworks or LLM tooling.

Set Up Your Python Environment

Before you can evaluate a RAG pipeline, you need a clean Python environment with the right packages installed.

In this step, you will install Python, create an isolated project directory with a virtual environment, and install DeepEval and OpenAI packages.

In this step, get ready to:
  • Install Python 3.10+ and verify it works.
  • Create a project directory with a virtual environment.
  • Install DeepEval and OpenAI packages and configure your API key.
Install Python and verify your version

DeepEval requires Python 3.10 or higher. Let's check whether you already have the right version installed.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.10 or higher

Python is ready. Move on to the next substep below.

ⓧ I see an older version

You have Python installed, but it is too old for DeepEval. You need version 3.10 or higher.

  • Download the latest Python installer from python.org/downloads.
  • Run the installer and follow the prompts.
  • On macOS, the system Python is often outdated. After installing from python.org, use python3.12 (the full versioned binary) instead of python3 if the default still points to the old version.
  • Close and reopen your terminal, then run the version check again.

ⓧ Command not found

Python is not installed yet. Let's fix that.

  • Go to python.org/downloads and download the latest Python installer for your operating system.
  • Run the installer.
  • On Windows, check the box that says Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal after installation.
  • Run the version check command again to confirm Python is working.
Create your project directory and virtual environment

A virtual environment keeps your project's packages isolated from your system Python. This prevents version conflicts between projects.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-evaluation
cd rag-evaluation
  • Confirm the folder exists by running:
ls

The folder will be empty for now. That's expected.

  • Create a virtual environment by running:

macOS

python3 -m venv venv

Windows

python -m venv venv
  • Activate your virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is a self-contained folder that holds its own copy of Python and any packages you install. Packages installed inside it do not affect your system Python or other projects.

The (venv) prefix in your terminal tells you the environment is active. Every pip install you run from now on goes into this environment only.

Don't see (venv) in your prompt?

Make sure you ran the activate command from inside the rag-evaluation folder (where the venv folder was created).

On Windows, if you see an error about execution policies, run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned and then try the activate command again.

Help me activate my Python virtual environment

Install DeepEval and OpenAI packages

DeepEval is the evaluation framework you will use to measure your RAG pipeline's quality. The OpenAI package powers both the RAG system's generation and DeepEval's LLM-as-judge metrics.

  • Install both packages by running:
pip install -U deepeval openai

What does this command do?

  • pip install -U installs packages and upgrades them to the latest version.
  • deepeval is the LLM evaluation framework with built-in RAG metrics like faithfulness and answer relevancy.
  • openai is the official Python client for OpenAI's API, used for both generating answers and judging them.

The installation may take a minute. You should see a success message once it finishes.

Seeing installation errors?

Make sure your virtual environment is active (you should see (venv) in your prompt). If you closed your terminal, navigate back to your rag-evaluation folder and activate the environment again.

If you see permission errors, do not use sudo. Instead, confirm you are inside an activated virtual environment.

Help me fix my pip install error for deepeval

Now set your OpenAI API key as an environment variable. DeepEval needs this key to call OpenAI's models when scoring your evaluation metrics.

  • Set your API key by running:

macOS

export OPENAI_API_KEY=your-api-key-here

Windows

$env:OPENAI_API_KEY="your-api-key-here"
  • Replace your-api-key-here with your actual OpenAI API key.

Why do I need an OpenAI API key?

DeepEval's RAG metrics are LLM-as-a-Judge metrics. They call OpenAI's models to evaluate your RAG system's outputs. The same key also powers the RAG system's answer generation in the next step.

This environment variable only lasts for your current terminal session. If you close your terminal, you will need to set it again.

Finally, open VS Code to your project directory.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Select the rag-evaluation folder on your Desktop, then click Open.

You should see the rag-evaluation folder name in the VS Code sidebar with a venv subfolder inside it.

Your environment is ready. Next up, you will build a minimal RAG system with an in-memory document store and OpenAI-powered answer generation.

Build a Simple RAG System

Your Python environment is ready with all the packages installed. Now it's time to build the RAG system that your evaluation pipeline will test.

A RAG system has two core components: a retriever that finds relevant documents, and a generator that produces answers from those documents. You'll build both in a single Python file, keeping things simple with an in-memory knowledge base and keyword-based search.

In this step, get ready to:
  • Create an in-memory knowledge base of customer support documents.
  • Implement a keyword-based retriever function.
  • Connect the retriever to OpenAI's API to generate answers from retrieved context.
Create a knowledge base of documents stored in memory

Every RAG system needs a collection of source documents to retrieve from. You'll create a small knowledge base of four customer support documents covering return policies, shipping, product care, and sizing.

  • In VS Code, create a new file called rag_system.py in your rag-evaluation folder.
  • Add the imports, client setup, and knowledge base by pasting this code:
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

# Knowledge base documents
DOCUMENTS = [
    {
        "id": "doc1",
        "title": "Return Policy",
        "content": "All customers are eligible for a 30 day full refund at no extra cost. "
        "Items must be in original condition with tags attached. "
        "Refunds are processed within 5-7 business days after we receive the returned item."
    },
    {
        "id": "doc2",
        "title": "Shipping Information",
        "content": "Standard shipping takes 5-7 business days. "
        "Express shipping takes 1-2 business days and costs $15. "
        "Free shipping is available on orders over $50. "
        "International shipping is available to select countries."
    },
    {
        "id": "doc3",
        "title": "Product Care",
        "content": "Machine wash cold with like colors. Tumble dry low. "
        "Do not bleach. Iron on low heat if needed. "
        "Store in a cool, dry place away from direct sunlight."
    },
    {
        "id": "doc4",
        "title": "Size Guide",
        "content": "Our sizes run true to standard US sizing. "
        "For the best fit, measure your chest, waist, and hips. "
        "If you are between sizes, we recommend sizing up for a more comfortable fit. "
        "See our detailed measurement chart on each product page."
    },
]

What does this code do?

The first two lines import the os module and the OpenAI client. The client reads your API key from the environment variable you set in Step 1.

The DOCUMENTS list acts as your knowledge base. Each document has an ID, a title, and content text. In a production system, this would be a vector database. For evaluation purposes, four documents are enough to produce predictable, interpretable results.

  • Save rag_system.py.
  • Verify the file loads correctly by running this in your terminal:
python -c "from rag_system import DOCUMENTS; print(len(DOCUMENTS))"

You should see 4 printed, confirming all four documents loaded successfully.

Seeing a ModuleNotFoundError?

Make sure your terminal is in the rag-evaluation directory where rag_system.py lives. Run ls to confirm you can see the file listed.

Also check that your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.

help me debug a ModuleNotFoundError when importing rag_system

Implement a keyword-based retriever that finds relevant documents

The retriever's job is to find the most relevant documents for a given question. Your retriever scores each document by counting how many words it shares with the query, then returns the top matches.

  • In rag_system.py, add the retrieve() function below the DOCUMENTS list:
def retrieve(query: str, top_k: int = 2) -> list[str]:
    """Simple keyword-based retriever that scores documents by keyword overlap."""
    query_words = set(query.lower().split())
    scored_docs = []

    for doc in DOCUMENTS:
        doc_words = set(doc["content"].lower().split())
        title_words = set(doc["title"].lower().split())
        overlap = len(query_words & (doc_words | title_words))
        scored_docs.append((overlap, doc["content"]))

    scored_docs.sort(key=lambda x: x[0], reverse=True)
    return [content for _, content in scored_docs[:top_k]]

What does this code do?

The retrieve() function splits the query into individual words and compares them against each document's content and title. The more words that overlap, the higher the document scores.

The top_k=2 parameter means it returns the 2 highest-scoring documents. This gives the generator enough context without overwhelming it with irrelevant information.

  • Save rag_system.py.
  • Test the retriever by running:
python -c "from rag_system import retrieve; print(retrieve('return policy'))"

You should see a list containing two document strings. The first one mentions "30 day full refund" because the Return Policy document has the highest keyword overlap with your query.

Seeing an empty list or wrong documents?

Check that DOCUMENTS is defined as a module-level variable above the retrieve() function. Python reads files top to bottom, so the function needs the list to already exist.

help me debug why my retrieve function returns unexpected results

Why keyword matching instead of embeddings?

Production RAG systems use embedding models and vector databases for semantic search. For this project, keyword matching keeps the setup minimal. No vector database, no embedding model, no extra dependencies.

The evaluation techniques you'll learn work identically regardless of how retrieval is implemented. You can swap in a vector database later and your test suite stays the same.

Connect the retriever to OpenAI's API to generate answers

The generator takes the retrieved documents and uses them as context to answer the user's question. You'll use OpenAI's gpt-4o-mini model with a system prompt that instructs it to answer only from the provided context.

  • In rag_system.py, add the generate_answer() function below retrieve():
def generate_answer(query: str, context: list[str]) -> str:
    """Generate an answer using OpenAI with the retrieved context."""
    context_text = "\n\n".join(context)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful customer support assistant. "
                    "Answer the user's question based ONLY on the provided context. "
                    "If the context does not contain enough information to answer, say so."
                ),
            },
            {
                "role": "user",
                "content": f"Context:\n{context_text}\n\nQuestion: {query}",
            },
        ],
        temperature=0,
    )
    return response.choices[0].message.content

What does this code do?

The system prompt tells the model to act as a customer support assistant and only answer from the provided context. This is critical for faithfulness. Without this instruction, the model might draw on its training data and hallucinate information not in your documents.

Setting temperature=0 makes responses deterministic. This is important for evaluation because you want consistent outputs when running the same test multiple times.

You won't see a change yet. The next function ties retrieval and generation together into a single pipeline.

  • Add the query_rag() function below generate_answer():
def query_rag(query: str) -> dict:
    """Run the full RAG pipeline: retrieve then generate."""
    retrieved_contexts = retrieve(query)
    answer = generate_answer(query, retrieved_contexts)
    return {
        "query": query,
        "answer": answer,
        "retrieved_contexts": retrieved_contexts,
    }

What does this code do?

The query_rag() function orchestrates the full pipeline. It calls retrieve() to get relevant documents, passes them to generate_answer(), and returns everything in a dictionary.

Returning the retrieved_contexts alongside the answer is essential for evaluation. DeepEval needs to see exactly which documents were retrieved to measure faithfulness and contextual relevancy.

  • Save rag_system.py.
  • Test the full RAG pipeline by running:
python -c "from rag_system import query_rag; result = query_rag('What is your return policy?'); print(result['answer'])"

You should see a response about a 30-day full refund, original condition requirements, and 5-7 business day processing time. This answer is generated entirely from your Return Policy document.

Seeing an AuthenticationError?

This means your OPENAI_API_KEY environment variable is not set or contains an invalid key. Check it by running:

echo $OPENAI_API_KEY

If it prints nothing, you need to export the variable again in your current terminal session. If it prints a value, double-check that the key is valid in your OpenAI dashboard.

help me fix an OpenAI AuthenticationError in my RAG system

✔️ Awesome, I've got everything!

Great. Double check you've saved rag_system.py.

ⓧ I'd like to double check the full code

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

# Knowledge base documents
DOCUMENTS = [
    {
        "id": "doc1",
        "title": "Return Policy",
        "content": "All customers are eligible for a 30 day full refund at no extra cost. "
        "Items must be in original condition with tags attached. "
        "Refunds are processed within 5-7 business days after we receive the returned item."
    },
    {
        "id": "doc2",
        "title": "Shipping Information",
        "content": "Standard shipping takes 5-7 business days. "
        "Express shipping takes 1-2 business days and costs $15. "
        "Free shipping is available on orders over $50. "
        "International shipping is available to select countries."
    },
    {
        "id": "doc3",
        "title": "Product Care",
        "content": "Machine wash cold with like colors. Tumble dry low. "
        "Do not bleach. Iron on low heat if needed. "
        "Store in a cool, dry place away from direct sunlight."
    },
    {
        "id": "doc4",
        "title": "Size Guide",
        "content": "Our sizes run true to standard US sizing. "
        "For the best fit, measure your chest, waist, and hips. "
        "If you are between sizes, we recommend sizing up for a more comfortable fit. "
        "See our detailed measurement chart on each product page."
    },
]


def retrieve(query: str, top_k: int = 2) -> list[str]:
    """Simple keyword-based retriever that scores documents by keyword overlap."""
    query_words = set(query.lower().split())
    scored_docs = []

    for doc in DOCUMENTS:
        doc_words = set(doc["content"].lower().split())
        title_words = set(doc["title"].lower().split())
        overlap = len(query_words & (doc_words | title_words))
        scored_docs.append((overlap, doc["content"]))

    scored_docs.sort(key=lambda x: x[0], reverse=True)
    return [content for _, content in scored_docs[:top_k]]


def generate_answer(query: str, context: list[str]) -> str:
    """Generate an answer using OpenAI with the retrieved context."""
    context_text = "\n\n".join(context)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful customer support assistant. "
                    "Answer the user's question based ONLY on the provided context. "
                    "If the context does not contain enough information to answer, say so."
                ),
            },
            {
                "role": "user",
                "content": f"Context:\n{context_text}\n\nQuestion: {query}",
            },
        ],
        temperature=0,
    )
    return response.choices[0].message.content


def query_rag(query: str) -> dict:
    """Run the full RAG pipeline: retrieve then generate."""
    retrieved_contexts = retrieve(query)
    answer = generate_answer(query, retrieved_contexts)
    return {
        "query": query,
        "answer": answer,
        "retrieved_contexts": retrieved_contexts,
    }

Your RAG system is complete. It retrieves relevant documents and generates grounded answers. Next up, you'll build an evaluation test suite that measures exactly how well this pipeline performs using the RAG Triad metrics.

Create Your Golden Dataset and Run the RAG Triad Evaluation

Your RAG system can retrieve documents and generate answers. But how do you know if those answers are actually good?

In this step, you'll create a golden dataset of test questions and evaluate your RAG system using three metrics called the RAG Triad. These metrics independently measure whether the retriever pulled relevant context, and whether the generator produced faithful, on-topic answers.

In this step, get ready to:
  • Define test cases with inputs, actual outputs, and retrieval context.
  • Configure DeepEval's FaithfulnessMetric, AnswerRelevancyMetric, and ContextualRelevancyMetric.
  • Run the evaluation and interpret per-metric scores.
Define your golden dataset

A golden dataset is a curated list of test questions that you run through your RAG pipeline repeatedly. Each test case captures the query, the RAG system's actual output, and the retrieved context chunks. This gives the evaluation metrics everything they need to score your system.

  • In VS Code, create a new file called run_evaluation.py inside your rag-evaluation folder.
  • Add the imports and the build_golden_dataset function by pasting the following code:
"""Run RAG evaluation outside of pytest for quick iteration."""

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRelevancyMetric,
)
from rag_system import query_rag


def build_golden_dataset() -> list[LLMTestCase]:
    """Build a golden dataset by running queries through the RAG pipeline."""
    queries = [
        "What is your return policy?",
        "How long does express shipping take?",
        "How should I wash my clothes?",
        "What if I'm between sizes?",
        "Can I get free shipping?",
    ]

    test_cases = []
    for query in queries:
        result = query_rag(query)
        test_cases.append(
            LLMTestCase(
                input=result["query"],
                actual_output=result["answer"],
                retrieval_context=result["retrieved_contexts"],
            )
        )
    return test_cases

What does this code do?

The build_golden_dataset function defines 5 test queries that cover different topics in your knowledge base (returns, shipping, product care, sizing, and free shipping).

For each query, it calls query_rag() to get the actual answer and retrieved context. Then it wraps those into an LLMTestCase with the three fields that the RAG Triad metrics need: input, actual_output, and retrieval_context.

  • Below the build_golden_dataset function, add the main() function that runs the full evaluation:
def main():
    print("Building golden dataset by querying RAG system...")
    test_cases = build_golden_dataset()
    print(f"Created {len(test_cases)} test cases\n")

    metrics = [
        FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True),
        AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True),
        ContextualRelevancyMetric(
            threshold=0.7, model="gpt-4o-mini", include_reason=True
        ),
    ]

    print("Running RAG Triad evaluation...")
    results = evaluate(test_cases=test_cases, metrics=metrics)

    print("\n--- Evaluation Summary ---")
    print(f"Overall pass rate: {results.count_pass()}/{results.count_pass() + results.count_fail()}")
    print("\nPer-metric breakdown:")
    for metric_result in results.metrics_results:
        print(f"  {metric_result.metric_name}: {metric_result.score:.2f}")


if __name__ == "__main__":
    main()

What does this code do?

The main() function creates all three RAG Triad metrics with a threshold of 0.7. A test case passes a metric only if the score is 0.7 or higher.

It then calls evaluate() which runs every metric on every test case concurrently, making LLM judge calls to score each one. The summary at the end prints how many test cases passed overall and per-metric averages.

  • Save run_evaluation.py.

Seeing a syntax error?

Make sure the if __name__ == "__main__": block is at the bottom of the file, outside of any function. Also check that your f-strings use matching curly braces.

help me fix a syntax error in my run_evaluation.py file

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

"""Run RAG evaluation outside of pytest for quick iteration."""

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRelevancyMetric,
)
from rag_system import query_rag


def build_golden_dataset() -> list[LLMTestCase]:
    """Build a golden dataset by running queries through the RAG pipeline."""
    queries = [
        "What is your return policy?",
        "How long does express shipping take?",
        "How should I wash my clothes?",
        "What if I'm between sizes?",
        "Can I get free shipping?",
    ]

    test_cases = []
    for query in queries:
        result = query_rag(query)
        test_cases.append(
            LLMTestCase(
                input=result["query"],
                actual_output=result["answer"],
                retrieval_context=result["retrieved_contexts"],
            )
        )
    return test_cases


def main():
    print("Building golden dataset by querying RAG system...")
    test_cases = build_golden_dataset()
    print(f"Created {len(test_cases)} test cases\n")

    metrics = [
        FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True),
        AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True),
        ContextualRelevancyMetric(
            threshold=0.7, model="gpt-4o-mini", include_reason=True
        ),
    ]

    print("Running RAG Triad evaluation...")
    results = evaluate(test_cases=test_cases, metrics=metrics)

    print("\n--- Evaluation Summary ---")
    print(f"Overall pass rate: {results.count_pass()}/{results.count_pass() + results.count_fail()}")
    print("\nPer-metric breakdown:")
    for metric_result in results.metrics_results:
        print(f"  {metric_result.metric_name}: {metric_result.score:.2f}")


if __name__ == "__main__":
    main()
Configure the RAG Triad metrics and write test functions

The run_evaluation.py script is great for quick iteration. Now you'll create a pytest-integrated test file that gives you pass/fail assertions. This is the approach you'd use in CI/CD pipelines to catch quality regressions automatically.

  • Create a new file called test_rag_evaluation.py inside your rag-evaluation folder.
  • Add the imports and metric configuration by pasting the following code:
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRelevancyMetric,
)
from rag_system import query_rag


# Configure metrics
faithfulness_metric = FaithfulnessMetric(
    threshold=0.7,
    model="gpt-4o-mini",
    include_reason=True,
)

answer_relevancy_metric = AnswerRelevancyMetric(
    threshold=0.7,
    model="gpt-4o-mini",
    include_reason=True,
)

contextual_relevancy_metric = ContextualRelevancyMetric(
    threshold=0.7,
    model="gpt-4o-mini",
    include_reason=True,
)

What does this code do?

Each metric is configured with three parameters. The threshold=0.7 means a test case must score at least 0.7 (out of 1.0) to pass. The model="gpt-4o-mini" tells DeepEval which LLM to use as the judge. The include_reason=True makes DeepEval explain why it gave a particular score.

The LLM-as-judge pattern means a second LLM evaluates the output of your first LLM. It reads the retrieved context and the generated answer, then decides whether the answer is faithful to that context.

  • Below the metric configuration, add the helper function and the individual test functions:

def create_test_case(query: str) -> LLMTestCase:
    """Run the RAG pipeline and create a test case from the results."""
    result = query_rag(query)
    return LLMTestCase(
        input=result["query"],
        actual_output=result["answer"],
        retrieval_context=result["retrieved_contexts"],
    )


def test_return_policy_faithfulness():
    """Test that answers about returns are grounded in retrieved context."""
    test_case = create_test_case("What is your return policy?")
    assert_test(test_case, [faithfulness_metric])


def test_return_policy_answer_relevancy():
    """Test that answers about returns are relevant to the question asked."""
    test_case = create_test_case("What is your return policy?")
    assert_test(test_case, [answer_relevancy_metric])


def test_shipping_contextual_relevancy():
    """Test that retrieved context for shipping questions is relevant."""
    test_case = create_test_case("How long does express shipping take?")
    assert_test(test_case, [contextual_relevancy_metric])


def test_rag_triad_combined():
    """Test all three RAG Triad metrics together on a single query."""
    test_case = create_test_case("Can I get a refund if the shoes don't fit?")
    assert_test(
        test_case,
        [faithfulness_metric, answer_relevancy_metric, contextual_relevancy_metric],
    )

What does this code do?

The create_test_case helper runs your RAG pipeline for a given query and wraps the result into an LLMTestCase. Each test function then passes that test case to assert_test with one or more metrics.

The first three tests each isolate one metric. The test_rag_triad_combined function runs all three metrics together on a single query. If any metric scores below 0.7, the test fails.

  • Finally, add the hallucination detection test below the combined test:

def test_hallucination_detection():
    """Test that faithfulness catches hallucinated content.

    This test uses a manually constructed test case with a hallucinated output
    to demonstrate how faithfulness detects claims not supported by context.
    """
    test_case = LLMTestCase(
        input="What is your return policy?",
        actual_output=(
            "We offer a 90-day money-back guarantee with free return shipping "
            "and a $10 store credit bonus on every return."
        ),
        retrieval_context=[
            "All customers are eligible for a 30 day full refund at no extra cost. "
            "Items must be in original condition with tags attached. "
            "Refunds are processed within 5-7 business days after we receive the returned item."
        ],
    )
    # We expect this to FAIL because the output contradicts the context
    metric = FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True)
    metric.measure(test_case)
    print(f"\nFaithfulness score for hallucinated output: {metric.score}")
    print(f"Reason: {metric.reason}")
    assert metric.score < 0.5, "Hallucinated output should score low on faithfulness"

Why build a hallucination test?

This test proves that the faithfulness metric actually catches problems. The actual_output claims a 90-day guarantee and free return shipping, but the context only mentions 30 days and says nothing about free shipping.

Instead of using assert_test, this test calls metric.measure(test_case) directly so you can inspect the raw score and reason. It then asserts the score is below 0.5 to confirm that hallucinations get caught.

  • Save test_rag_evaluation.py.

Seeing an import error?

Make sure your virtual environment is activated. Also verify that rag_system.py is in the same folder as test_rag_evaluation.py. Python needs both files in the same directory to resolve the import.

help me fix an import error in my DeepEval test file

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRelevancyMetric,
)
from rag_system import query_rag


# Configure metrics
faithfulness_metric = FaithfulnessMetric(
    threshold=0.7,
    model="gpt-4o-mini",
    include_reason=True,
)

answer_relevancy_metric = AnswerRelevancyMetric(
    threshold=0.7,
    model="gpt-4o-mini",
    include_reason=True,
)

contextual_relevancy_metric = ContextualRelevancyMetric(
    threshold=0.7,
    model="gpt-4o-mini",
    include_reason=True,
)


def create_test_case(query: str) -> LLMTestCase:
    """Run the RAG pipeline and create a test case from the results."""
    result = query_rag(query)
    return LLMTestCase(
        input=result["query"],
        actual_output=result["answer"],
        retrieval_context=result["retrieved_contexts"],
    )


def test_return_policy_faithfulness():
    """Test that answers about returns are grounded in retrieved context."""
    test_case = create_test_case("What is your return policy?")
    assert_test(test_case, [faithfulness_metric])


def test_return_policy_answer_relevancy():
    """Test that answers about returns are relevant to the question asked."""
    test_case = create_test_case("What is your return policy?")
    assert_test(test_case, [answer_relevancy_metric])


def test_shipping_contextual_relevancy():
    """Test that retrieved context for shipping questions is relevant."""
    test_case = create_test_case("How long does express shipping take?")
    assert_test(test_case, [contextual_relevancy_metric])


def test_rag_triad_combined():
    """Test all three RAG Triad metrics together on a single query."""
    test_case = create_test_case("Can I get a refund if the shoes don't fit?")
    assert_test(
        test_case,
        [faithfulness_metric, answer_relevancy_metric, contextual_relevancy_metric],
    )


def test_hallucination_detection():
    """Test that faithfulness catches hallucinated content.

    This test uses a manually constructed test case with a hallucinated output
    to demonstrate how faithfulness detects claims not supported by context.
    """
    test_case = LLMTestCase(
        input="What is your return policy?",
        actual_output=(
            "We offer a 90-day money-back guarantee with free return shipping "
            "and a $10 store credit bonus on every return."
        ),
        retrieval_context=[
            "All customers are eligible for a 30 day full refund at no extra cost. "
            "Items must be in original condition with tags attached. "
            "Refunds are processed within 5-7 business days after we receive the returned item."
        ],
    )
    # We expect this to FAIL because the output contradicts the context
    metric = FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True)
    metric.measure(test_case)
    print(f"\nFaithfulness score for hallucinated output: {metric.score}")
    print(f"Reason: {metric.reason}")
    assert metric.score < 0.5, "Hallucinated output should score low on faithfulness"
Run the evaluation and interpret the scores

You now have two ways to run your evaluation. The deepeval test run command gives you pytest-style pass/fail output. The standalone script gives you a summary view. Let's try both.

  • Run the pytest-integrated evaluation by running this command:
deepeval test run test_rag_evaluation.py

What should I expect?

This command takes 30-60 seconds because each metric makes multiple LLM calls to the judge model. You'll see each test function reported as passed or failed with its score.

DeepEval may prompt you about logging into Confident AI. You can press Enter to skip this. It is optional and not required for local evaluation.

  • After the test run finishes, look at the output. Each test shows a score between 0 and 1 along with a pass or fail status.
  • Now run the standalone evaluation script by running this command:
python run_evaluation.py
  • Wait for the evaluation to complete. You should see a summary showing how many test cases passed and a per-metric breakdown.

How do I interpret these scores?

All metric scores range from 0 to 1. A score of 1.0 is perfect. Here's what each metric tells you:

  • Faithfulness: Extracts claims from the answer and checks each one against the retrieved context. Score = number of truthful claims divided by total claims. Low score means the generator is hallucinating.
  • Answer Relevancy: Checks whether the answer actually addresses the user's question. Low score means the response is off-topic.
  • Contextual Relevancy: Checks whether the retrieved chunks are relevant to answering the question. Low score means your retriever is pulling in the wrong documents.

Two evaluation workflows

Use deepeval test run when you want pass/fail assertions in a CI/CD pipeline. Use python run_evaluation.py when you want a quick summary for local iteration. Both produce the same scores.

Tests failing or timing out?

If you see rate limit errors, wait a minute and try again. OpenAI throttles rapid API calls. If all tests fail with authentication errors, verify your OPENAI_API_KEY environment variable is set correctly in your terminal session.

help me troubleshoot failing DeepEval tests with API errors

Your evaluation pipeline is working. Next up, you'll intentionally break things to see how the metrics respond and learn to diagnose retrieval versus generation failures.

Diagnose and Improve Your RAG Pipeline

In the previous step, you built a full evaluation suite and ran it against your RAG pipeline. You saw scores and reasons for each metric.

Scores alone are not enough. The real power of the RAG Triad is using each metric as a diagnostic signal. In this step, you will learn to read those signals, pinpoint whether a failure comes from retrieval or generation, and observe how evaluation catches hallucination in action.

In this step, get ready to:
  • Analyze metric scores to distinguish retrieval failures from generation failures.
  • Run the hallucination detection test and observe a low faithfulness score.
  • Compare normal RAG scores against hallucinated output to see how evaluation catches regressions.
Analyze metric scores to diagnose retrieval vs generation failures

Each metric in the RAG Triad isolates a different failure mode. Understanding which metric dropped tells you exactly where to focus your debugging effort.

  • Run the standalone evaluation script by running this command:
python run_evaluation.py

The script prints per-metric scores for five test queries across all three RAG Triad metrics.

How to read each metric as a diagnostic signal

  • Low faithfulness = generation problem. The LLM is making claims that are not supported by the retrieved context. It is hallucinating or contradicting what it was given.
  • Low answer relevancy = prompt or retrieval problem. The answer does not address the user's question. Either irrelevant context was retrieved, or the system prompt did not guide the model properly.
  • Low contextual relevancy = retrieval problem. The retrieved text chunks are not relevant to producing a good answer. Your retriever is pulling in the wrong documents.
  • Look at your terminal output and find any scores below 0.7 (the threshold you configured).
  • Read the reason strings printed alongside each score. These explanations tell you which specific claims or context chunks caused the score to drop.

What does a reason string look like?

DeepEval's LLM judge produces a natural language explanation for every score. For faithfulness, it identifies specific claims in the output and states whether each one is supported by the retrieval context.

For contextual relevancy, it evaluates each retrieved chunk and explains whether it is relevant to answering the input question. This granularity lets you trace failures to specific documents or claims.

High scores across all three metrics

If all your scores are above 0.7, that is a good sign. Your keyword retriever works well for these simple queries because the document titles contain the query keywords. The real test comes when we deliberately break things.

Run the hallucination detection test

Your test suite includes test_hallucination_detection() which uses a manually crafted output that contradicts the retrieved context. This simulates what happens when an LLM hallucinates.

  • Run your full test suite with verbose output by running this command:
deepeval test run test_rag_evaluation.py -v

What does the -v flag do?

The -v flag enables verbose mode for all metrics in the test run. It prints the intermediate steps each metric uses to calculate its score, so you can see exactly how claims are extracted and classified.

  • Look for the test_hallucination_detection output in your terminal.
  • Find the printed faithfulness score. It should be below 0.5 because the test's actual_output contains fabricated claims.

The test constructs an output claiming a 90-day guarantee, free return shipping, and a $10 bonus. None of these exist in the retrieval context, which says 30 days and no extra cost.

How does faithfulness catch this?

The FaithfulnessMetric extracts individual claims from the actual_output and checks each one against the retrieval_context.

The score equals the number of truthful claims divided by the total claims. When most claims are fabricated, the score drops toward 0. The reason string names each unsupported claim, making it clear what went wrong.

Test taking too long or timing out?

Each test makes multiple API calls to the LLM judge. If you hit rate limits, wait 60 seconds and re-run. You can also run just the hallucination test by passing -k test_hallucination_detection after the filename.

Help me troubleshoot my deepeval test run

Compare scores to understand how evaluation catches regressions

Now that you have seen both passing and failing test results, you can see the evaluation framework in action as a regression detector.

  • Compare the faithfulness score from test_return_policy_faithfulness (which uses the real RAG pipeline) against the score from test_hallucination_detection (which uses the fabricated output).
  • Note how the real pipeline scores high (above 0.7) because the generated answer sticks to claims supported by the retrieved context.
  • Note how the fabricated output scores low (below 0.5) because it introduces claims that contradict the context.

What does this comparison teach you?

In a real project, you would run this test suite on every code change. If someone swaps the model, changes the system prompt, or modifies the retriever, a drop in faithfulness immediately signals a generation regression.

A drop in contextual relevancy would signal that retrieval got worse. This separation is what makes the RAG Triad more useful than a single end-to-end quality score.

  • Look at the contextual relevancy score for test_shipping_contextual_relevancy. This should score well because the query "How long does express shipping take?" contains keywords that match the Shipping Information document title.
  • Consider what would happen if the query had no keyword overlap with any document title. The retriever would pull irrelevant documents, and contextual relevancy would drop. That is your signal to improve the retriever (e.g. by adding embeddings).

The diagnostic decision tree

  • All three metrics high: your pipeline is working well for this query.
  • Contextual relevancy low, others high: your retriever is pulling wrong documents but the generator is doing its best with what it gets.
  • Faithfulness low, contextual relevancy high: the retriever found good context but the generator ignored or contradicted it.
  • Answer relevancy low: the final answer is off-topic regardless of whether retrieval or generation caused it.

You now have a complete mental model for interpreting DeepEval metric scores. Every time a score drops, you know exactly which component to investigate and how the reason string guides you to the root cause.

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved. Your evaluation pipeline is complete and you understand how to use it diagnostically.

ⓧ I'd like to double check the full code

Here is the full code for all three files. Your files should match these exactly since no code changes were made in this step.

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

# Knowledge base documents
DOCUMENTS = [
    {
        "id": "doc1",
        "title": "Return Policy",
        "content": "All customers are eligible for a 30 day full refund at no extra cost. "
        "Items must be in original condition with tags attached. "
        "Refunds are processed within 5-7 business days after we receive the returned item."
    },
    {
        "id": "doc2",
        "title": "Shipping Information",
        "content": "Standard shipping takes 5-7 business days. "
        "Express shipping takes 1-2 business days and costs $15. "
        "Free shipping is available on orders over $50. "
        "International shipping is available to select countries."
    },
    {
        "id": "doc3",
        "title": "Product Care",
        "content": "Machine wash cold with like colors. Tumble dry low. "
        "Do not bleach. Iron on low heat if needed. "
        "Store in a cool, dry place away from direct sunlight."
    },
    {
        "id": "doc4",
        "title": "Size Guide",
        "content": "Our sizes run true to standard US sizing. "
        "For the best fit, measure your chest, waist, and hips. "
        "If you are between sizes, we recommend sizing up for a more comfortable fit. "
        "See our detailed measurement chart on each product page."
    },
]


def retrieve(query: str, top_k: int = 2) -> list[str]:
    """Simple keyword-based retriever that scores documents by keyword overlap."""
    query_words = set(query.lower().split())
    scored_docs = []

    for doc in DOCUMENTS:
        doc_words = set(doc["content"].lower().split())
        title_words = set(doc["title"].lower().split())
        overlap = len(query_words & (doc_words | title_words))
        scored_docs.append((overlap, doc["content"]))

    scored_docs.sort(key=lambda x: x[0], reverse=True)
    return [content for _, content in scored_docs[:top_k]]


def generate_answer(query: str, context: list[str]) -> str:
    """Generate an answer using OpenAI with the retrieved context."""
    context_text = "\n\n".join(context)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful customer support assistant. "
                    "Answer the user's question based ONLY on the provided context. "
                    "If the context does not contain enough information to answer, say so."
                ),
            },
            {
                "role": "user",
                "content": f"Context:\n{context_text}\n\nQuestion: {query}",
            },
        ],
        temperature=0,
    )
    return response.choices[0].message.content


def query_rag(query: str) -> dict:
    """Run the full RAG pipeline: retrieve then generate."""
    retrieved_contexts = retrieve(query)
    answer = generate_answer(query, retrieved_contexts)
    return {
        "query": query,
        "answer": answer,
        "retrieved_contexts": retrieved_contexts,
    }
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRelevancyMetric,
)
from rag_system import query_rag


# Configure metrics
faithfulness_metric = FaithfulnessMetric(
    threshold=0.7,
    model="gpt-4o-mini",
    include_reason=True,
)

answer_relevancy_metric = AnswerRelevancyMetric(
    threshold=0.7,
    model="gpt-4o-mini",
    include_reason=True,
)

contextual_relevancy_metric = ContextualRelevancyMetric(
    threshold=0.7,
    model="gpt-4o-mini",
    include_reason=True,
)


def create_test_case(query: str) -> LLMTestCase:
    """Run the RAG pipeline and create a test case from the results."""
    result = query_rag(query)
    return LLMTestCase(
        input=result["query"],
        actual_output=result["answer"],
        retrieval_context=result["retrieved_contexts"],
    )


def test_return_policy_faithfulness():
    """Test that answers about returns are grounded in retrieved context."""
    test_case = create_test_case("What is your return policy?")
    assert_test(test_case, [faithfulness_metric])


def test_return_policy_answer_relevancy():
    """Test that answers about returns are relevant to the question asked."""
    test_case = create_test_case("What is your return policy?")
    assert_test(test_case, [answer_relevancy_metric])


def test_shipping_contextual_relevancy():
    """Test that retrieved context for shipping questions is relevant."""
    test_case = create_test_case("How long does express shipping take?")
    assert_test(test_case, [contextual_relevancy_metric])


def test_rag_triad_combined():
    """Test all three RAG Triad metrics together on a single query."""
    test_case = create_test_case("Can I get a refund if the shoes don't fit?")
    assert_test(
        test_case,
        [faithfulness_metric, answer_relevancy_metric, contextual_relevancy_metric],
    )


def test_hallucination_detection():
    """Test that faithfulness catches hallucinated content.

    This test uses a manually constructed test case with a hallucinated output
    to demonstrate how faithfulness detects claims not supported by context.
    """
    test_case = LLMTestCase(
        input="What is your return policy?",
        actual_output=(
            "We offer a 90-day money-back guarantee with free return shipping "
            "and a $10 store credit bonus on every return."
        ),
        retrieval_context=[
            "All customers are eligible for a 30 day full refund at no extra cost. "
            "Items must be in original condition with tags attached. "
            "Refunds are processed within 5-7 business days after we receive the returned item."
        ],
    )
    # We expect this to FAIL because the output contradicts the context
    metric = FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True)
    metric.measure(test_case)
    print(f"\nFaithfulness score for hallucinated output: {metric.score}")
    print(f"Reason: {metric.reason}")
    assert metric.score < 0.5, "Hallucinated output should score low on faithfulness"
"""Run RAG evaluation outside of pytest for quick iteration."""

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRelevancyMetric,
)
from rag_system import query_rag


def build_golden_dataset() -> list[LLMTestCase]:
    """Build a golden dataset by running queries through the RAG pipeline."""
    queries = [
        "What is your return policy?",
        "How long does express shipping take?",
        "How should I wash my clothes?",
        "What if I'm between sizes?",
        "Can I get free shipping?",
    ]

    test_cases = []
    for query in queries:
        result = query_rag(query)
        test_cases.append(
            LLMTestCase(
                input=result["query"],
                actual_output=result["answer"],
                retrieval_context=result["retrieved_contexts"],
            )
        )
    return test_cases


def main():
    print("Building golden dataset by querying RAG system...")
    test_cases = build_golden_dataset()
    print(f"Created {len(test_cases)} test cases\n")

    metrics = [
        FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True),
        AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True),
        ContextualRelevancyMetric(
            threshold=0.7, model="gpt-4o-mini", include_reason=True
        ),
    ]

    print("Running RAG Triad evaluation...")
    results = evaluate(test_cases=test_cases, metrics=metrics)

    print("\n--- Evaluation Summary ---")
    print(f"Overall pass rate: {results.count_pass()}/{results.count_pass() + results.count_fail()}")
    print("\nPer-metric breakdown:")
    for metric_result in results.metrics_results:
        print(f"  {metric_result.metric_name}: {metric_result.score:.2f}")


if __name__ == "__main__":
    main()

You have built a complete diagnostic toolkit for RAG evaluation. Next up, tackle the Secret Mission to extend your pipeline with a custom metric that checks source citations.

Secret mission

Build a Custom Source Citation Metric

Your RAG pipeline can measure faithfulness, relevancy, and contextual quality. But what about source attribution? In real-world applications, users need to know WHERE information came from. Build a custom GEval metric that checks whether your RAG system properly cites its source documents in responses.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine. There are no cloud resources left running or accumulating charges. The only cost was the OpenAI API calls you made during evaluation runs.

Decide whether to keep your project files for future experimentation, or delete them entirely.

Resources you used:

  • Project directory with rag_system.py, test_rag_evaluation.py, and run_evaluation.py.
  • Python virtual environment (inside your project directory).
  • OpenAI API usage (already billed per call, no ongoing charges).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your evaluation pipeline or adding new test cases.

  • Your project files stay on your Desktop.
  • Your virtual environment stays ready to activate whenever you want to run evaluations again.
  • Your OPENAI_API_KEY environment variable only lasts for the current terminal session unless you added it to your shell profile.

Pause - I'll come back to this later

Since this project is entirely local with no running processes, pausing and keeping are effectively the same. Your files sit idle on disk until you come back.

  • Close your terminal to clear the OPENAI_API_KEY from the session.
  • Close VS Code if you have the project open.
  • When you come back, reactivate your virtual environment and re-export your API key.

Delete - I don't want to use this again

Remove all project files from your machine. This deletes your RAG system, evaluation scripts, and the virtual environment.

  • Close VS Code if you have the project open.
  • Delete the project directory by running this command (macOS/Linux):
rm -rf ~/Desktop/rag-evaluation

On Windows, run this instead:

rmdir /s /q %USERPROFILE%\Desktop\rag-evaluation
  • This removes the virtual environment, all Python files, and any .deepeval cache created during test runs.
  • Your OpenAI API key remains valid for other projects. No action needed on the OpenAI side unless you want to revoke it from your OpenAI account settings.

That's a Wrap!

That's a Wrap!

Nice work! You have built a complete RAG evaluation pipeline that measures retrieval and generation quality independently using DeepEval.

You've learned how to:

  • Built a RAG evaluation pipeline using the RAG Triad metrics: faithfulness, answer relevancy, and contextual relevancy.
  • Created a golden dataset of test cases and ran pytest-style evaluation tests that catch quality regressions.
  • Diagnosed retrieval vs generation failures by interpreting per-metric scores and DeepEval's structured explanations.
  • Built a custom source citation metric using GEval to extend evaluation beyond built-in metrics.

Ready to quiz yourself?