Faithfulness Eval for Financial RAG

Build a pytest evaluation pipeline that catches hallucinations in SEC filing RAG.

Introduction

30 Second Summary

When an AI system reports financial figures, a single made-up number can mean millions in losses. Unlike a chatbot getting a recipe wrong, financial AI needs every claim to trace back to a real document.

In this project, you will build a faithfulness evaluation pipeline for a RAG system that answers questions from real SEC 10-K filings. You will create domain-specific LLM judges that catch grounding failures, then wire them into a pytest CI gate that blocks on faithfulness regressions.

What You'll Build

A production-grade faithfulness evaluation pipeline that tests whether a financial RAG system's answers are grounded in real SEC 10-K filings.

By the end of this project, you'll have:

  • A RAG pipeline over real SEC 10-K filings using Claude Sonnet 4.5, FAISS vector search, and sentence-transformers embeddings.
  • A golden evaluation dataset and a DeepEval faithfulness test suite that uses an LLM-as-judge to score grounding quality.
  • A custom financial faithfulness evaluator that detects domain-specific hallucination patterns like fabricated figures and wrong-entity attribution, wired into a pytest CI gate that blocks on faithfulness regressions.
  • Secret Mission: Benchmark your custom judge against the generic one on adversarial financial test cases, proving domain-specific evaluation catches failures generic judges miss.

Are there any prerequisites?

You will need an Anthropic API key with credits loaded, since the project uses Claude for both generation and evaluation.

You should also be comfortable with Python classes, decorators, and pytest, and familiar with RAG concepts like retrieval and grounding.

Set Up the Project Environment

Before you can evaluate whether a RAG pipeline hallucinates, you need the tools to build one and measure its faithfulness.

In this step, you will create the project directory, install the libraries that power retrieval, generation, and evaluation, and wire up your API credentials so everything can talk to each other.

In this step, get ready to:
  • Create the project directory with subdirectories for data, evaluations, and golden test cases.
  • Install all Python dependencies in a virtual environment.
  • Configure API credentials and verify connectivity with Claude.
Create the project directory and install dependencies

This project requires Python 3.10 or higher because one of the core libraries, edgartools, needs it to fetch SEC filings.

  • Check your Python version by running this command:
python3 --version

✔️ I see version 3.10 or higher

You are all set. Move on to creating the project directory below.

ⓧ I see an older version

Your system Python is below 3.10. macOS often ships with an older Python that stays even after you install a newer one.

  • Download the latest Python 3 installer from the official Python website.
  • Run the installer and follow the prompts.
  • Close and reopen your Terminal after installation.
  • Run python3 --version again to confirm it now shows 3.10 or higher.
  • If the version still shows the old number, try running the version-specific binary directly (for example python3.12 --version). Use that binary name in place of python3 for the rest of this project.

Homebrew alternative

If you use Homebrew, you can install Python with brew install python@3.12 instead. After installing, the binary is available as python3.12.

ⓧ Command not found

Python 3 is not installed on your machine.

  • Download the latest Python 3 installer from the official Python website.
  • Run the installer and follow the prompts.
  • Close and reopen your Terminal after installation.
  • Run python3 --version again to confirm it shows 3.10 or higher.
  • Open your Terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), then type Terminal and press Enter.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project directory and its subdirectories by running these commands:
mkdir fin-rag-eval
cd fin-rag-eval
mkdir data evals golden
  • Confirm the directories were created by running:
ls

You should see data, evals, and golden listed in the output.

What are these directories for?

The data/ directory stores raw SEC filing text that the RAG pipeline retrieves from. The golden/ directory holds the evaluation dataset of test cases with known-good and known-bad outputs. The evals/ directory contains the pytest test files that gate on faithfulness scores.

A virtual environment isolates this project's packages from your system Python so nothing conflicts with other projects on your machine.

  • Create and activate a virtual environment by running these commands:
python3 -m venv venv
source venv/bin/activate

Your terminal prompt now shows (venv) at the beginning, confirming the virtual environment is active.

Don't see (venv) in your prompt?

Make sure you ran source venv/bin/activate (not just python3 -m venv venv). The first command creates the environment. The second one activates it.

If you still see an error, confirm you are inside the fin-rag-eval/ directory by running pwd.

Help me fix my virtual environment activation.

Now install all the project dependencies. This downloads several packages including a sentence embedding model, so it may take a few minutes.

  • Install the project packages by running:
pip install deepeval==4.0.5 anthropic edgartools sentence-transformers faiss-cpu python-dotenv

What do these packages do?

  • DeepEval provides faithfulness metrics and a pytest integration for LLM evaluation. You are pinning version 4.0.5 because a later release has a known import bug.
  • anthropic is the official SDK for calling Claude models.
  • edgartools fetches and parses SEC EDGAR filings directly into Python objects.
  • sentence-transformers provides the local embedding model that turns filing text into vectors for similarity search.
  • faiss-cpu is Facebook's FAISS library for fast vector similarity search.
  • python-dotenv loads environment variables from a .env file so your credentials stay out of source code.

Installation failing?

Make sure you see (venv) in your prompt before running pip install. Installing into your system Python instead of the virtual environment can cause permission errors.

If you see a compilation error for faiss-cpu, make sure your Python version is 3.10 or higher. Older Python versions may not have prebuilt wheels available.

Help me fix my pip install errors.

  • Verify that DeepEval installed correctly by running:
python -c "from deepeval.metrics import FaithfulnessMetric; print('DeepEval OK')"
  • Verify that edgartools installed correctly by running:
python -c "from edgar import Company; print('EdgarTools OK')"

You should see DeepEval OK and EdgarTools OK printed without any errors.

Seeing an ImportError?

Double-check that you activated the virtual environment before running the verification commands. If you opened a new terminal tab, you need to run source venv/bin/activate again from inside the fin-rag-eval/ directory.

Help me fix my import errors after installing packages.

Configure API credentials and verify connectivity

The project needs two credentials. Your Anthropic API key lets you call Claude for generation and evaluation. Your EDGAR identity string lets the SEC know who is accessing its filing data.

What is EDGAR identity?

The SEC requires all users of its EDGAR API to provide a name and email address for identification. No registration or API key is needed. You just supply a plain-text string like YourName your.email@example.com.

  • Create the .env.example template file by running this command:
cat > .env.example << 'EOF'
ANTHROPIC_API_KEY=your-api-key-here
EDGAR_IDENTITY=YourName your.email@example.com
EOF

What does this command do?

The cat > ... << 'EOF' pattern is called a heredoc. It writes everything between the two EOF markers into the file. This creates a template that shows teammates what environment variables the project expects without exposing real credentials.

  • Copy the template to create your actual .env file by running:
cp .env.example .env

Now you need to edit .env to replace the placeholders with your real credentials.

  • Open .env in a terminal text editor by running:
open -e .env

What does open -e do?

On macOS, open -e opens a file in TextEdit. If you prefer a different editor, you can use any text editor you are comfortable with.

  • Replace your-api-key-here with your actual Anthropic API key.
  • Replace YourName your.email@example.com with your real name and email address (for example, Jane Doe jane@example.com).
  • Save and close the file.

Your project scripts use python-dotenv to read from .env at runtime. For the verification commands below, you also need the variables exported in your current shell session.

  • Export all variables from your .env file into your shell by running:
set -a && source .env && set +a

What does this command do?

The set -a flag tells your shell to automatically export every variable it reads. Then source .env reads your .env file and sets the variables. Finally, set +a turns off auto-export. This loads your credentials without typing them directly into the terminal.

  • Test your Claude API connectivity by running:
python -c '
from anthropic import Anthropic
c = Anthropic()
r = c.messages.create(
    model="claude-sonnet-4-5-20250514",
    max_tokens=50,
    messages=[{"role": "user", "content": "Say hello in one sentence."}]
)
print(r.content[0].text)
'

You should see a short greeting from Claude printed in your terminal. The exact text varies, but any response confirms your API key is working.

Seeing an AuthenticationError?

This means your API key was not loaded correctly. Run echo $ANTHROPIC_API_KEY to check whether the variable is set. If it prints nothing, re-run the set -a && source .env && set +a command.

Also confirm that your .env file has no extra spaces around the = sign. The format must be ANTHROPIC_API_KEY=sk-... with no spaces.

Help me fix my Anthropic API connection.

  • Confirm the files exist by running:
ls -a .env*

You should see both .env and .env.example listed.

✔️ Awesome, I've got everything!

Great. Double check you have saved your .env file with your real credentials.

ⓧ I'd like to double check the full code

Your .env.example should look exactly like this (with placeholder values). Your .env file has the same structure but with your real API key and identity.

ANTHROPIC_API_KEY=your-api-key-here
EDGAR_IDENTITY=YourName your.email@example.com

Your environment is ready. All packages are installed, your credentials are configured, and Claude is responding. Next up, you will build the RAG pipeline that retrieves and answers questions from real SEC filings.

Build the RAG Pipeline

Your environment is set up and your API credentials are connected. Now it's time to build the core of this project: a RAG pipeline that retrieves real financial data and generates grounded answers.

You'll fetch a real 10-K filing from the SEC EDGAR database, split it into searchable chunks, embed those chunks into a FAISS vector index, and wire up Claude as the answer generator. By the end of this step, you'll have a working pipeline that answers financial questions using only the filing as its source of truth.

In this step, get ready to:
  • Chunk a 10-K filing and build a FAISS vector index with sentence-transformers embeddings.
  • Implement retrieval and generation functions using Claude Sonnet 4.5.
  • Test the pipeline with sample financial questions and verify grounded responses.
Chunk the filing text and build a FAISS vector index

Before the pipeline can answer any questions, it needs data to search through. A 10-K filing is a massive document, often over 100,000 words. Searching the whole thing at once would be slow and imprecise.

The solution is to split the filing into overlapping chunks, convert each chunk into a numerical vector (an embedding), and store those vectors in a searchable index. When a question comes in, the pipeline finds the chunks most similar to the question.

  • Create a new file called rag_pipeline.py in your fin-rag-eval folder.
  • Add the imports, module-level setup, and the fetch_filing() function by copying this code:
import os
import json
import numpy as np
import faiss
from anthropic import Anthropic
from sentence_transformers import SentenceTransformer
from edgar import Company, set_identity
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Identify yourself to the SEC EDGAR API (required for all requests)
set_identity(os.getenv("EDGAR_IDENTITY", "FinRAGEval fin-rag-eval@example.com"))

# Initialize the Claude client and embedding model
client = Anthropic()
embedder = SentenceTransformer("all-MiniLM-L6-v2")


def fetch_filing(ticker: str = "MSFT") -> str:
    # Look up the company and get its most recent 10-K filing
    company = Company(ticker)
    filings = company.get_filings(form="10-K")
    filing = filings[0]
    text = filing.text()
    # Save the raw text locally for the RAG pipeline
    os.makedirs("data", exist_ok=True)
    filepath = f"data/{ticker}_10k.txt"
    with open(filepath, "w", encoding="utf-8") as f:
        f.write(text)
    print(f"Saved {len(text):,} characters to {filepath}")
    return text

What does this code do?

The module-level setup loads your .env file and configures two clients. set_identity() tells the SEC who is making requests, which is required for all EDGAR API access. Anthropic() reads your ANTHROPIC_API_KEY from the environment automatically.

The SentenceTransformer("all-MiniLM-L6-v2") line loads a local embedding model that converts text into 384-dimensional vectors. On first run, it downloads about 90 MB of model weights from Hugging Face.

The fetch_filing() function uses edgartools to pull Microsoft's most recent 10-K filing from SEC EDGAR, extracts the plain text with filing.text(), and saves it to data/MSFT_10k.txt.

Why sentence-transformers instead of an API embedding model?

Claude does not offer an embeddings API. Using a local model like all-MiniLM-L6-v2 keeps the project to a single paid dependency (Anthropic) and runs for free on your machine.

FAISS is a vector search library built by Meta. It handles similarity search over dense vectors. For a dataset this size, it is fast and lightweight.

You won't see any output yet. The next code block defines the functions that process the filing text.

  • Add the chunk_text() and build_index() functions below fetch_filing() by copying this code:

def chunk_text(text: str, chunk_size: int = 512, overlap: int = 64) -> list[str]:
    # Split the filing into word-level chunks with overlap
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = " ".join(words[start:end])
        # Skip chunks that are too short to be meaningful
        if len(chunk.strip()) > 50:
            chunks.append(chunk)
        start += chunk_size - overlap
    return chunks


def build_index(chunks: list[str]) -> tuple[faiss.IndexFlatIP, np.ndarray]:
    # Encode all chunks into normalized embeddings for cosine similarity
    embeddings = embedder.encode(chunks, show_progress_bar=True, normalize_embeddings=True)
    embeddings = np.array(embeddings, dtype="float32")
    # Create a FAISS inner-product index (equivalent to cosine similarity with normalized vectors)
    index = faiss.IndexFlatIP(embeddings.shape[1])
    index.add(embeddings)
    return index, embeddings

What does this code do?

  • chunk_text() splits the filing into 512-word chunks with a 64-word overlap. The overlap ensures important context at chunk boundaries is not lost.
  • build_index() encodes every chunk into a vector using the embedding model, then stores them in a FAISS inner-product index. Because the embeddings are normalized, inner product is equivalent to cosine similarity. This makes retrieval fast and accurate.

You still won't see any output. These functions are called later when you run the pipeline.

Implement the retrieval and generation pipeline

With the data indexed, the pipeline needs two more abilities: finding the right chunks for a question, and generating a grounded answer from those chunks.

  • Add the retrieve() function below build_index() by copying this code:

def retrieve(query: str, index: faiss.IndexFlatIP, chunks: list[str], top_k: int = 5) -> list[str]:
    # Embed the query and find the top-k most similar chunks
    query_embedding = embedder.encode([query], normalize_embeddings=True)
    query_embedding = np.array(query_embedding, dtype="float32")
    scores, indices = index.search(query_embedding, top_k)
    return [chunks[i] for i in indices[0] if i < len(chunks)]

What does this code do?

retrieve() takes a question, encodes it into the same vector space as the chunks, and asks FAISS for the 5 most similar chunks. These chunks become the context the generator uses to answer the question.

You won't see output yet. The next chunk adds the generation logic that calls Claude.

  • Add the generate() and rag_pipeline() functions below retrieve() by copying this code:

def generate(query: str, context_chunks: list[str]) -> str:
    # Join the retrieved chunks into a single context block
    context = "\n\n---\n\n".join(context_chunks)
    # Call Claude with a grounded system prompt that restricts answers to the context
    message = client.messages.create(
        model="claude-sonnet-4-5-20250514",
        max_tokens=1024,
        system=(
            "You are a financial analyst assistant. Answer the question ONLY using "
            "the provided context from SEC filings. If the context does not contain "
            "enough information to answer, say so explicitly. Do not use any knowledge "
            "outside the provided context. Cite specific figures and sections when possible."
        ),
        messages=[
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}",
            }
        ],
    )
    return message.content[0].text


def rag_pipeline(query: str, index: faiss.IndexFlatIP, chunks: list[str], top_k: int = 5) -> dict:
    # Chain retrieval and generation into a single pipeline call
    context_chunks = retrieve(query, index, chunks, top_k)
    answer = generate(query, context_chunks)
    return {
        "input": query,
        "actual_output": answer,
        "retrieval_context": context_chunks,
    }

What does this code do?

  • generate() takes the retrieved chunks and builds a prompt for Claude Sonnet 4.5. The system prompt is critical: it tells Claude to answer only from the provided context and to cite specific figures. This grounding instruction is what makes the output evaluable for faithfulness.
  • rag_pipeline() chains retrieval and generation together. It returns a dictionary with the input, actual_output, and retrieval_context. These three fields match exactly what DeepEval's LLMTestCase expects, which makes evaluation seamless later.

You still won't see output. Save the file and move on to the final substep where you add the test runner.

Test the pipeline with sample financial questions

The pipeline functions are all defined. Now you need a way to actually run them. The if __name__ == "__main__" block fetches the filing, builds the index, and runs three sample questions through the full pipeline.

  • Add the test runner block at the bottom of rag_pipeline.py, below all the function definitions:

if __name__ == "__main__":
    # Fetch the filing and prepare the data
    text = fetch_filing("MSFT")
    chunks = chunk_text(text)
    print(f"Created {len(chunks)} chunks")

    # Build the vector index from the chunks
    index, _ = build_index(chunks)

    # Test the pipeline with sample financial questions
    questions = [
        "What were Microsoft's total revenue and net income for the most recent fiscal year?",
        "What are the primary risk factors mentioned in the filing?",
        "How does Microsoft describe its cloud computing business segment?",
    ]

    for q in questions:
        result = rag_pipeline(q, index, chunks)
        print(f"\nQ: {q}")
        print(f"A: {result['actual_output'][:300]}...")
        print(f"Context chunks used: {len(result['retrieval_context'])}")

What does this code do?

  • fetch_filing("MSFT") downloads Microsoft's most recent 10-K filing from SEC EDGAR and saves it to data/MSFT_10k.txt.
  • chunk_text() splits the filing into overlapping chunks, and build_index() embeds them into the FAISS index.
  • The loop runs three financial questions through the full pipeline and prints the first 300 characters of each answer, plus how many context chunks were used.
  • Save rag_pipeline.py.
  • Run the pipeline by executing this command in your terminal:
python rag_pipeline.py

What should I expect to see?

This command takes 1 to 3 minutes to complete. Here is what happens during that time:

  • The embedding model downloads from Hugging Face (about 90 MB on first run).
  • The 10-K filing downloads from SEC EDGAR and saves to data/MSFT_10k.txt.
  • The chunks are embedded and indexed (you'll see a progress bar).
  • Three questions are answered by Claude using only the retrieved context.

You should see output showing the number of characters saved, the number of chunks created, and three Q&A pairs. Each answer should reference specific figures or sections from the filing.

  • Verify that the answers reference specific numbers or filing sections. For example, the revenue question should cite a dollar amount from the filing, not generic knowledge.

Not seeing any output or getting an error?

  • If you see an error about EDGAR_IDENTITY not being set, make sure your .env file contains the EDGAR_IDENTITY line and that it includes an email address.
  • If you see a connection error downloading the embedding model, check your internet connection and try running the command again. The model downloads from Hugging Face and requires network access.
  • If you see an authentication error from Anthropic, verify that your ANTHROPIC_API_KEY is correct in your .env file and that your account has credits.

Still stuck? Help me debug my RAG pipeline. I'm running python rag_pipeline.py and seeing an error.

✔️ Awesome, I've got everything!

Great. Double check you've saved rag_pipeline.py and that data/MSFT_10k.txt exists in your project folder.

ⓧ I'd like to double check the full code

import os
import json
import numpy as np
import faiss
from anthropic import Anthropic
from sentence_transformers import SentenceTransformer
from edgar import Company, set_identity
from dotenv import load_dotenv

load_dotenv()

set_identity(os.getenv("EDGAR_IDENTITY", "FinRAGEval fin-rag-eval@example.com"))

client = Anthropic()
embedder = SentenceTransformer("all-MiniLM-L6-v2")


def fetch_filing(ticker: str = "MSFT") -> str:
    company = Company(ticker)
    filings = company.get_filings(form="10-K")
    filing = filings[0]
    text = filing.text()
    os.makedirs("data", exist_ok=True)
    filepath = f"data/{ticker}_10k.txt"
    with open(filepath, "w", encoding="utf-8") as f:
        f.write(text)
    print(f"Saved {len(text):,} characters to {filepath}")
    return text


def chunk_text(text: str, chunk_size: int = 512, overlap: int = 64) -> list[str]:
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = " ".join(words[start:end])
        if len(chunk.strip()) > 50:
            chunks.append(chunk)
        start += chunk_size - overlap
    return chunks


def build_index(chunks: list[str]) -> tuple[faiss.IndexFlatIP, np.ndarray]:
    embeddings = embedder.encode(chunks, show_progress_bar=True, normalize_embeddings=True)
    embeddings = np.array(embeddings, dtype="float32")
    index = faiss.IndexFlatIP(embeddings.shape[1])
    index.add(embeddings)
    return index, embeddings


def retrieve(query: str, index: faiss.IndexFlatIP, chunks: list[str], top_k: int = 5) -> list[str]:
    query_embedding = embedder.encode([query], normalize_embeddings=True)
    query_embedding = np.array(query_embedding, dtype="float32")
    scores, indices = index.search(query_embedding, top_k)
    return [chunks[i] for i in indices[0] if i < len(chunks)]


def generate(query: str, context_chunks: list[str]) -> str:
    context = "\n\n---\n\n".join(context_chunks)
    message = client.messages.create(
        model="claude-sonnet-4-5-20250514",
        max_tokens=1024,
        system=(
            "You are a financial analyst assistant. Answer the question ONLY using "
            "the provided context from SEC filings. If the context does not contain "
            "enough information to answer, say so explicitly. Do not use any knowledge "
            "outside the provided context. Cite specific figures and sections when possible."
        ),
        messages=[
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}",
            }
        ],
    )
    return message.content[0].text


def rag_pipeline(query: str, index: faiss.IndexFlatIP, chunks: list[str], top_k: int = 5) -> dict:
    context_chunks = retrieve(query, index, chunks, top_k)
    answer = generate(query, context_chunks)
    return {
        "input": query,
        "actual_output": answer,
        "retrieval_context": context_chunks,
    }


if __name__ == "__main__":
    text = fetch_filing("MSFT")
    chunks = chunk_text(text)
    print(f"Created {len(chunks)} chunks")

    index, _ = build_index(chunks)

    questions = [
        "What were Microsoft's total revenue and net income for the most recent fiscal year?",
        "What are the primary risk factors mentioned in the filing?",
        "How does Microsoft describe its cloud computing business segment?",
    ]

    for q in questions:
        result = rag_pipeline(q, index, chunks)
        print(f"\nQ: {q}")
        print(f"A: {result['actual_output'][:300]}...")
        print(f"Context chunks used: {len(result['retrieval_context'])}")

Your RAG pipeline is live and answering financial questions from real SEC data. Next up, you'll build a golden evaluation dataset to systematically test whether those answers stay faithful to the source.

Build the Golden Evaluation Dataset and Run DeepEval Faithfulness

Your RAG pipeline retrieves context from SEC filings and generates grounded answers using Claude. But how do you know those answers are actually faithful to the retrieved context?

In this step, you will build a golden evaluation dataset that mixes real RAG answers with intentionally unfaithful ones. Then you will use DeepEval's FaithfulnessMetric to score each answer, revealing how well a generic LLM-as-judge catches hallucinations in financial output.

In this step, get ready to:
  • Construct a golden evaluation dataset with faithful and adversarial financial test cases.
  • Run DeepEval's FaithfulnessMetric and a custom financial judge to score each test case.
Construct the golden evaluation dataset

A golden dataset is a fixed set of test cases with known expected outcomes. You will generate 5 faithful test cases by running real questions through your RAG pipeline, then manually craft 3 adversarial cases with fabricated figures, wrong entity attributions, and unsourced statistics.

  • Create a new file called golden/build_golden.py in your fin-rag-eval project folder.
  • Add the imports, function definition, question list, and faithful test case generation loop by pasting the following code:
import json
import os
from rag_pipeline import fetch_filing, chunk_text, build_index, rag_pipeline


def build_golden_dataset():
    text = fetch_filing("MSFT")
    chunks = chunk_text(text)
    index, _ = build_index(chunks)

    questions = [
        "What were Microsoft's total revenue for the most recent fiscal year reported?",
        "What are the key risk factors Microsoft identifies in its 10-K filing?",
        "How does Microsoft describe its Intelligent Cloud segment?",
        "What was Microsoft's operating income?",
        "What does the filing say about Microsoft's research and development expenses?",
    ]

    test_cases = []

    for q in questions:
        result = rag_pipeline(q, index, chunks)
        test_cases.append(
            {
                "input": result["input"],
                "actual_output": result["actual_output"],
                "retrieval_context": result["retrieval_context"],
                "expected_faithful": True,
            }
        )

What does this code do?

The script imports your existing RAG pipeline functions from rag_pipeline.py. It re-fetches the Microsoft 10-K filing, chunks it, and builds a fresh FAISS index.

The loop sends 5 financial questions through your full RAG pipeline. Each question produces a real answer grounded in retrieved context. These become your faithful test cases with expected_faithful set to True.

  • You won't see a change yet. The next chunk adds the adversarial cases and saves the dataset.
  • Still in golden/build_golden.py, add the adversarial test cases and the save logic below the faithful loop by pasting this code directly after the last line you added:

    adversarial_cases = [
        {
            "input": "What were Microsoft's total revenue?",
            "actual_output": (
                "Microsoft reported total revenue of $245.1 billion for fiscal year 2024, "
                "representing a 18.2% increase year-over-year. The company's cloud division "
                "alone generated $140 billion, surpassing Amazon Web Services for the first time."
            ),
            "retrieval_context": test_cases[0]["retrieval_context"],
            "expected_faithful": False,
            "adversarial_type": "fabricated_figures",
        },
        {
            "input": "What are the risk factors?",
            "actual_output": (
                "Apple's primary risk factors include supply chain concentration in China, "
                "declining iPhone market share in Europe, and regulatory challenges with the "
                "App Store. The company also faces risks from its growing services revenue "
                "dependency."
            ),
            "retrieval_context": test_cases[1]["retrieval_context"],
            "expected_faithful": False,
            "adversarial_type": "wrong_entity_attribution",
        },
        {
            "input": "Describe the cloud computing business.",
            "actual_output": (
                "Microsoft's cloud business grew 45% year-over-year to reach $100 billion "
                "in annual recurring revenue. Azure's market share increased to 35%, making "
                "it the largest cloud provider globally by revenue."
            ),
            "retrieval_context": test_cases[2]["retrieval_context"],
            "expected_faithful": False,
            "adversarial_type": "unsourced_statistics",
        },
    ]

    test_cases.extend(adversarial_cases)

    os.makedirs("golden", exist_ok=True)
    with open("golden/financial_faithfulness.json", "w", encoding="utf-8") as f:
        json.dump(test_cases, f, indent=2, ensure_ascii=False)

    print(f"Built golden dataset with {len(test_cases)} test cases")
    print(f"  Faithful: {sum(1 for tc in test_cases if tc['expected_faithful'])}")
    print(f"  Adversarial: {sum(1 for tc in test_cases if not tc['expected_faithful'])}")


if __name__ == "__main__":
    build_golden_dataset()

What do the adversarial cases test?

  • fabricated_figures: Revenue numbers and growth rates that look plausible but are completely made up. The retrieval context contains real figures, so this tests whether the judge catches numerical fabrication.
  • wrong_entity_attribution: The answer discusses Apple's risk factors using Microsoft's retrieval context. This is the most common hallucination pattern in financial RAG, where facts from one company get attributed to another.
  • unsourced_statistics: Market share and growth claims that sound authoritative but do not appear anywhere in the retrieved context.

Each adversarial case reuses the retrieval context from a faithful case. The output is manually written to contain claims the context does not support.

  • Save golden/build_golden.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved golden/build_golden.py before moving on.

ⓧ I'd like to double check the full code

import json
import os
from rag_pipeline import fetch_filing, chunk_text, build_index, rag_pipeline


def build_golden_dataset():
    text = fetch_filing("MSFT")
    chunks = chunk_text(text)
    index, _ = build_index(chunks)

    questions = [
        "What were Microsoft's total revenue for the most recent fiscal year reported?",
        "What are the key risk factors Microsoft identifies in its 10-K filing?",
        "How does Microsoft describe its Intelligent Cloud segment?",
        "What was Microsoft's operating income?",
        "What does the filing say about Microsoft's research and development expenses?",
    ]

    test_cases = []

    for q in questions:
        result = rag_pipeline(q, index, chunks)
        test_cases.append(
            {
                "input": result["input"],
                "actual_output": result["actual_output"],
                "retrieval_context": result["retrieval_context"],
                "expected_faithful": True,
            }
        )

    adversarial_cases = [
        {
            "input": "What were Microsoft's total revenue?",
            "actual_output": (
                "Microsoft reported total revenue of $245.1 billion for fiscal year 2024, "
                "representing a 18.2% increase year-over-year. The company's cloud division "
                "alone generated $140 billion, surpassing Amazon Web Services for the first time."
            ),
            "retrieval_context": test_cases[0]["retrieval_context"],
            "expected_faithful": False,
            "adversarial_type": "fabricated_figures",
        },
        {
            "input": "What are the risk factors?",
            "actual_output": (
                "Apple's primary risk factors include supply chain concentration in China, "
                "declining iPhone market share in Europe, and regulatory challenges with the "
                "App Store. The company also faces risks from its growing services revenue "
                "dependency."
            ),
            "retrieval_context": test_cases[1]["retrieval_context"],
            "expected_faithful": False,
            "adversarial_type": "wrong_entity_attribution",
        },
        {
            "input": "Describe the cloud computing business.",
            "actual_output": (
                "Microsoft's cloud business grew 45% year-over-year to reach $100 billion "
                "in annual recurring revenue. Azure's market share increased to 35%, making "
                "it the largest cloud provider globally by revenue."
            ),
            "retrieval_context": test_cases[2]["retrieval_context"],
            "expected_faithful": False,
            "adversarial_type": "unsourced_statistics",
        },
    ]

    test_cases.extend(adversarial_cases)

    os.makedirs("golden", exist_ok=True)
    with open("golden/financial_faithfulness.json", "w", encoding="utf-8") as f:
        json.dump(test_cases, f, indent=2, ensure_ascii=False)

    print(f"Built golden dataset with {len(test_cases)} test cases")
    print(f"  Faithful: {sum(1 for tc in test_cases if tc['expected_faithful'])}")
    print(f"  Adversarial: {sum(1 for tc in test_cases if not tc['expected_faithful'])}")


if __name__ == "__main__":
    build_golden_dataset()
  • Run the golden dataset builder by running this command from your fin-rag-eval directory:
python golden/build_golden.py

This takes a few minutes. The script fetches the filing, builds embeddings, and calls Claude 5 times to generate the faithful answers.

  • You should see output that ends with something like:
Built golden dataset with 8 test cases
  Faithful: 5
  Adversarial: 3
  • Confirm the JSON file was created by running:
ls golden/
  • You should see build_golden.py and financial_faithfulness.json in the output.

Script failing or hanging?

  • If you see an authentication error, make sure your ANTHROPIC_API_KEY is set in your .env file and your virtual environment is activated.
  • If the SEC download fails, verify your EDGAR_IDENTITY environment variable is set. The SEC requires identification for API access.
  • If the embedding step is slow, this is normal. The first run downloads the all-MiniLM-L6-v2 model. Subsequent runs are faster.

Still stuck? Help me debug my golden dataset builder script.

Run DeepEval's FaithfulnessMetric using AnthropicModel

Now you will build an evaluation script that runs both a generic faithfulness judge and a custom financial faithfulness judge against your golden dataset. The FaithfulnessMetric works by extracting every claim from an answer, then checking whether each claim contradicts the retrieval context.

  • Create a new file called evals/run_evaluation.py in your fin-rag-eval project folder.
  • Add the imports and data-loading utility functions by pasting the following code:
import json
from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, GEval
from deepeval.test_case import SingleTurnParams
from deepeval.models import AnthropicModel


def load_golden_dataset(path: str = "golden/financial_faithfulness.json") -> list[dict]:
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)


def build_test_cases(dataset: list[dict]) -> list[LLMTestCase]:
    return [
        LLMTestCase(
            input=tc["input"],
            actual_output=tc["actual_output"],
            retrieval_context=tc["retrieval_context"],
        )
        for tc in dataset
    ]

What does this code do?

  • load_golden_dataset() reads the JSON file you just generated and returns it as a list of dictionaries.
  • build_test_cases() converts each dictionary into a DeepEval LLMTestCase object with input, actual_output, and retrieval_context. These three fields are what FaithfulnessMetric needs to evaluate grounding.

The AnthropicModel import lets DeepEval use Claude as the judge instead of the default OpenAI model.

  • You won't see a change yet. The next chunk adds the custom financial judge function.
  • Add the custom financial faithfulness judge function below the code you just pasted:

def create_financial_judge(model: AnthropicModel) -> GEval:
    return GEval(
        name="FinancialFaithfulness",
        criteria=(
            "Evaluate whether the actual output is faithfully grounded in the retrieval context, "
            "applying strict financial domain standards:\n"
            "1. Every numerical claim (revenue, profit, growth rates, percentages, ratios) must "
            "exactly match a figure present in the retrieval context.\n"
            "2. Every entity attribution (company name, subsidiary, product line, segment name) "
            "must correctly correspond to the entity discussed in the retrieval context. Attributing "
            "facts from one company to another is a critical failure.\n"
            "3. Claims about financial trends or comparisons must be supported by specific data "
            "points and time periods present in the context.\n"
            "4. Market share claims, rankings, or competitive comparisons must be explicitly "
            "stated in the context, not inferred.\n"
            "5. Any claim not explicitly supported by the retrieval context is unfaithful, "
            "even if it might be true in the real world."
        ),
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.RETRIEVAL_CONTEXT],
        model=model,
        threshold=0.7,
        strict_mode=False,
    )

What is GEval?

GEval is DeepEval's framework for creating custom LLM-based evaluation metrics. You provide natural language criteria describing how to judge the output, and GEval uses the specified model to score each test case from 0 to 1.

This custom judge targets financial-specific hallucination patterns that generic faithfulness metrics often miss. You will explore the design decisions behind these criteria in the next step.

  • You won't see a change yet. The next chunk adds the main comparison function.
  • Add the comparison runner and main block below the code you just pasted:

def run_comparison():
    dataset = load_golden_dataset()
    test_cases = build_test_cases(dataset)

    model = AnthropicModel("claude-sonnet-4-5")
    generic_metric = FaithfulnessMetric(threshold=0.7, model=model, include_reason=True)
    financial_metric = create_financial_judge(model)

    print("=" * 60)
    print("Running Generic FaithfulnessMetric...")
    print("=" * 60)
    generic_results = []
    for i, tc in enumerate(test_cases):
        generic_metric.measure(tc)
        generic_results.append(
            {
                "input": tc.input[:80],
                "score": generic_metric.score,
                "reason": generic_metric.reason,
                "expected_faithful": dataset[i]["expected_faithful"],
            }
        )
        print(f"  [{i+1}] Score: {generic_metric.score:.2f} | Expected faithful: {dataset[i]['expected_faithful']}")

    print("\n" + "=" * 60)
    print("Running Custom Financial Faithfulness Judge...")
    print("=" * 60)
    financial_results = []
    for i, tc in enumerate(test_cases):
        financial_metric.measure(tc)
        financial_results.append(
            {
                "input": tc.input[:80],
                "score": financial_metric.score,
                "reason": financial_metric.reason,
                "expected_faithful": dataset[i]["expected_faithful"],
            }
        )
        print(f"  [{i+1}] Score: {financial_metric.score:.2f} | Expected faithful: {dataset[i]['expected_faithful']}")

    print("\n" + "=" * 60)
    print("COMPARISON SUMMARY")
    print("=" * 60)
    print(f"{'Case':<6} {'Expected':<12} {'Generic':<10} {'Financial':<12} {'Generic Correct':<18} {'Financial Correct'}")
    print("-" * 80)
    generic_correct = 0
    financial_correct = 0
    for i in range(len(dataset)):
        expected = dataset[i]["expected_faithful"]
        g_score = generic_results[i]["score"]
        f_score = financial_results[i]["score"]
        g_pass = (g_score >= 0.7) == expected
        f_pass = (f_score >= 0.7) == expected
        generic_correct += int(g_pass)
        financial_correct += int(f_pass)
        print(f"  {i+1:<4} {str(expected):<12} {g_score:<10.2f} {f_score:<12.2f} {str(g_pass):<18} {str(f_pass)}")

    print(f"\nGeneric accuracy: {generic_correct}/{len(dataset)}")
    print(f"Financial accuracy: {financial_correct}/{len(dataset)}")


if __name__ == "__main__":
    run_comparison()

How does the comparison work?

The run_comparison() function runs every test case through two judges. First, DeepEval's built-in FaithfulnessMetric scores each case. Then the custom financial judge scores the same cases.

For each test case, the script calls metric.measure(tc) and reads metric.score (a float from 0 to 1) and metric.reason (a text explanation). A score at or above 0.7 means the judge considers the answer faithful.

The comparison summary table shows whether each judge correctly classified the test case based on its expected_faithful label.

  • Save evals/run_evaluation.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved evals/run_evaluation.py before running it.

ⓧ I'd like to double check the full code

import json
from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, GEval
from deepeval.test_case import SingleTurnParams
from deepeval.models import AnthropicModel


def load_golden_dataset(path: str = "golden/financial_faithfulness.json") -> list[dict]:
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)


def build_test_cases(dataset: list[dict]) -> list[LLMTestCase]:
    return [
        LLMTestCase(
            input=tc["input"],
            actual_output=tc["actual_output"],
            retrieval_context=tc["retrieval_context"],
        )
        for tc in dataset
    ]


def create_financial_judge(model: AnthropicModel) -> GEval:
    return GEval(
        name="FinancialFaithfulness",
        criteria=(
            "Evaluate whether the actual output is faithfully grounded in the retrieval context, "
            "applying strict financial domain standards:\n"
            "1. Every numerical claim (revenue, profit, growth rates, percentages, ratios) must "
            "exactly match a figure present in the retrieval context.\n"
            "2. Every entity attribution (company name, subsidiary, product line, segment name) "
            "must correctly correspond to the entity discussed in the retrieval context. Attributing "
            "facts from one company to another is a critical failure.\n"
            "3. Claims about financial trends or comparisons must be supported by specific data "
            "points and time periods present in the context.\n"
            "4. Market share claims, rankings, or competitive comparisons must be explicitly "
            "stated in the context, not inferred.\n"
            "5. Any claim not explicitly supported by the retrieval context is unfaithful, "
            "even if it might be true in the real world."
        ),
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.RETRIEVAL_CONTEXT],
        model=model,
        threshold=0.7,
        strict_mode=False,
    )


def run_comparison():
    dataset = load_golden_dataset()
    test_cases = build_test_cases(dataset)

    model = AnthropicModel("claude-sonnet-4-5")
    generic_metric = FaithfulnessMetric(threshold=0.7, model=model, include_reason=True)
    financial_metric = create_financial_judge(model)

    print("=" * 60)
    print("Running Generic FaithfulnessMetric...")
    print("=" * 60)
    generic_results = []
    for i, tc in enumerate(test_cases):
        generic_metric.measure(tc)
        generic_results.append(
            {
                "input": tc.input[:80],
                "score": generic_metric.score,
                "reason": generic_metric.reason,
                "expected_faithful": dataset[i]["expected_faithful"],
            }
        )
        print(f"  [{i+1}] Score: {generic_metric.score:.2f} | Expected faithful: {dataset[i]['expected_faithful']}")

    print("\n" + "=" * 60)
    print("Running Custom Financial Faithfulness Judge...")
    print("=" * 60)
    financial_results = []
    for i, tc in enumerate(test_cases):
        financial_metric.measure(tc)
        financial_results.append(
            {
                "input": tc.input[:80],
                "score": financial_metric.score,
                "reason": financial_metric.reason,
                "expected_faithful": dataset[i]["expected_faithful"],
            }
        )
        print(f"  [{i+1}] Score: {financial_metric.score:.2f} | Expected faithful: {dataset[i]['expected_faithful']}")

    print("\n" + "=" * 60)
    print("COMPARISON SUMMARY")
    print("=" * 60)
    print(f"{'Case':<6} {'Expected':<12} {'Generic':<10} {'Financial':<12} {'Generic Correct':<18} {'Financial Correct'}")
    print("-" * 80)
    generic_correct = 0
    financial_correct = 0
    for i in range(len(dataset)):
        expected = dataset[i]["expected_faithful"]
        g_score = generic_results[i]["score"]
        f_score = financial_results[i]["score"]
        g_pass = (g_score >= 0.7) == expected
        f_pass = (f_score >= 0.7) == expected
        generic_correct += int(g_pass)
        financial_correct += int(f_pass)
        print(f"  {i+1:<4} {str(expected):<12} {g_score:<10.2f} {f_score:<12.2f} {str(g_pass):<18} {str(f_pass)}")

    print(f"\nGeneric accuracy: {generic_correct}/{len(dataset)}")
    print(f"Financial accuracy: {financial_correct}/{len(dataset)}")


if __name__ == "__main__":
    run_comparison()

API costs for this run

This script makes multiple Claude API calls. Each test case requires at least one call per metric, so expect around 16 total calls across both judges. The cost is typically under $1 for a full run.

  • Run the evaluation comparison by running this command:
python evals/run_evaluation.py

This takes several minutes as each test case is evaluated by both judges sequentially.

  • You should see scores printed for each test case under both the generic and custom judges, followed by a comparison summary table.
  • Look at the adversarial cases (cases 6, 7, and 8). Faithful cases (1-5) should score above 0.7. Adversarial cases should score below 0.7.
  • If the generic judge scores any adversarial case above 0.7, that is a gap. Note which cases the generic judge missed. The custom financial judge should catch more of them.

Seeing API errors or timeouts?

  • If you see a rate limit error, wait 60 seconds and rerun the script. The Anthropic API has per-minute rate limits that sequential evaluation can trigger.
  • If you see a model not found error, double check that you have credits on your Anthropic account and that your API key is valid.
  • If DeepEval prints a login prompt, you can skip it by pressing Enter. The evaluation runs locally without a Confident AI account.

Still stuck? Help me debug my DeepEval evaluation run.

You now have a golden dataset and two judges scoring your RAG output. Next up, you will dive deeper into the custom financial judge's design and analyze exactly where it outperforms the generic metric.

Build a Custom Financial Faithfulness Judge

In the last step, you ran DeepEval's generic FaithfulnessMetric on your golden dataset and observed the scores. You may have noticed that some adversarial cases scored higher than expected. Generic judges check whether claims appear in the context, but they lack the domain knowledge to flag financial-specific failures like wrong-entity attribution or fabricated growth percentages.

Your evals/run_evaluation.py file already contains a custom financial faithfulness judge built with DeepEval's GEval metric. In this step, you'll dig into how its domain-specific criteria work and then run both judges side by side to see the difference in action.

In this step, get ready to:
  • Understand the 5-point financial faithfulness criteria in your custom judge.
  • Run both judges on all 8 golden test cases and compare detection rates.
Design a finance-specific faithfulness evaluator using DeepEval's GEval metric

DeepEval's generic FaithfulnessMetric extracts claims from the output and checks whether each one contradicts the retrieval context. This works well for general-purpose evaluation. But in financial documents, the failure modes are more subtle.

A revenue figure might be plausible but completely fabricated. A risk factor might be real but belong to a different company. A growth rate might be formatted correctly but reference the wrong fiscal year. These are the patterns your custom judge targets.

  • In your editor, open evals/run_evaluation.py and find the create_financial_judge() function.
def create_financial_judge(model: AnthropicModel) -> GEval:
    return GEval(
        name="FinancialFaithfulness",
        criteria=(
            "Evaluate whether the actual output is faithfully grounded in the retrieval context, "
            "applying strict financial domain standards:\n"
            "1. Every numerical claim (revenue, profit, growth rates, percentages, ratios) must "
            "exactly match a figure present in the retrieval context.\n"
            "2. Every entity attribution (company name, subsidiary, product line, segment name) "
            "must correctly correspond to the entity discussed in the retrieval context. Attributing "
            "facts from one company to another is a critical failure.\n"
            "3. Claims about financial trends or comparisons must be supported by specific data "
            "points and time periods present in the context.\n"
            "4. Market share claims, rankings, or competitive comparisons must be explicitly "
            "stated in the context, not inferred.\n"
            "5. Any claim not explicitly supported by the retrieval context is unfaithful, "
            "even if it might be true in the real world."
        ),
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.RETRIEVAL_CONTEXT],
        model=model,
        threshold=0.7,
        strict_mode=False,
    )

What is GEval?

GEval is DeepEval's framework for creating custom LLM-as-judge metrics using natural language criteria. Instead of writing a full custom metric class, you describe WHAT to evaluate in plain English and GEval handles the prompt engineering and scoring.

The evaluation_params field tells GEval which parts of the test case to send to the judge. Here you pass ACTUAL_OUTPUT and RETRIEVAL_CONTEXT so the judge can compare the RAG answer against its source documents.

What do the 5 criteria target?

Each criterion addresses a specific financial hallucination pattern that generic judges tend to miss:

  • Criterion 1 (numerical exactness) catches fabricated revenue figures, invented growth rates, and rounded-off percentages. Your fabricated_figures adversarial case targets this.
  • Criterion 2 (entity attribution) catches cross-company mix-ups. Your wrong_entity_attribution case puts Apple's risk factors in a Microsoft filing context.
  • Criterion 3 (temporal grounding) requires trend claims to reference specific time periods from the context, not vague directional statements.
  • Criterion 4 (market share sourcing) flags competitive comparisons that the filing never makes. Your unsourced_statistics case injects market share claims absent from the context.
  • Criterion 5 (explicit support) is the catch-all. Even if a claim happens to be true in the real world, it counts as unfaithful if the retrieval context does not explicitly support it.

Why GEval instead of a custom metric class?

DeepEval also lets you build fully custom metrics by subclassing DeepEvalBaseLLM. That gives you total control over prompting and scoring logic, but it requires significantly more boilerplate.

GEval hits the sweet spot for domain-specific evaluation. You write the criteria in natural language, and GEval structures the LLM judge prompt around them. For financial faithfulness, the domain expertise lives in the criteria themselves, not in custom prompt engineering.

Now look at how run_comparison() in the same file uses both judges. It runs the generic FaithfulnessMetric and your custom FinancialFaithfulness judge on every test case, then prints a comparison table with per-case scores.

Run the custom judge on the golden dataset and compare results

Time to put both judges head-to-head on your 8 golden test cases. The run_comparison() function runs the generic metric first, then the custom financial judge, and finishes with a comparison summary.

  • Run the comparison by executing this command from your fin-rag-eval/ directory:
python evals/run_evaluation.py

What does this command do?

This calls run_comparison(), which loads your golden dataset and feeds all 8 test cases through both judges. Each judge makes an API call to Claude Sonnet 4.5 for every test case, so expect approximately 16 API calls total.

The script takes a few minutes to complete because each .measure() call waits for a full LLM evaluation. You'll see per-case scores print as each one finishes.

Wait for the script to finish. At the end, you should see a comparison summary table that looks similar to this:

============================================================
COMPARISON SUMMARY
============================================================
Case   Expected     Generic    Financial    Generic Correct   Financial Correct
--------------------------------------------------------------------------------
  1    True         0.95       0.90         True              True
  2    True         0.88       0.85         True              True
  ...
  6    False        0.75       0.40         False             True
  7    False        0.30       0.20         True              True
  8    False        0.65       0.35         True              True

Generic accuracy: 6/8
Financial accuracy: 8/8

How to read the comparison table

  • Expected shows whether the test case is faithful (True) or adversarial (False).
  • Generic and Financial columns show each judge's faithfulness score (0.0 to 1.0).
  • Generic Correct and Financial Correct show whether the judge's score matched the expected verdict at the 0.7 threshold.
  • A False in the Correct column for an adversarial case means the judge MISSED the hallucination. It scored the unfaithful output above 0.7, treating it as faithful.

Your exact scores will differ because LLM evaluation is non-deterministic. The key thing to look for is whether your custom financial judge caught adversarial cases that the generic judge missed or scored too leniently.

Why does the custom judge perform better on financial cases?

The generic FaithfulnessMetric asks "does this claim contradict the context?" A claim like "revenue grew 45% year-over-year" might not directly contradict anything in the context. It just isn't supported by it.

Your custom criteria explicitly require numerical exactness and entity-level attribution. This shifts the burden of proof. Instead of asking "is this wrong?", the judge asks "is this provably right based on the context?" That distinction is critical in financial reporting.

Seeing rate limit errors?

The comparison runs approximately 16 API calls in sequence. If you hit rate limits, wait a minute and run the script again. The Anthropic API has per-minute request limits that vary by account tier.

If both judges score all adversarial cases below 0.7, that is still a valid result. LLM judges are non-deterministic, and sometimes the generic metric catches everything on a given run. The important thing is that the custom criteria give the financial judge a structural advantage on domain-specific failures.

Help me debug my evaluation comparison results.

API cost reminder

Each full comparison run makes approximately 16 Claude API calls. This costs roughly $0.50-1.00 depending on context chunk sizes. Avoid running the script repeatedly unless you need to.

You now have two faithfulness judges producing real scores on your golden dataset. The custom judge gives you domain-specific sensitivity that the generic one lacks. Next up, you'll wire both judges into a pytest test suite that gates on faithfulness regressions, so a failing score blocks your pipeline automatically.

Wire Into a Pytest CI Gate

In the last step, you compared your custom financial judge against DeepEval's generic FaithfulnessMetric and identified where domain-specific criteria caught failures the generic judge missed.

Running evaluations manually is useful for development, but production systems need automated guardrails. In this step, you'll wire both judges into a pytest test suite that automatically blocks deployments when faithfulness drops below your threshold. This turns your evaluation from a one-off comparison into a CI/CD gate that protects your pipeline from regressions.

In this step, get ready to:
  • Create a pytest test file with parameterized test cases and threshold-based assertions.
  • Run the test suite and verify pass/fail behavior on faithful and adversarial cases.
Create the pytest test file with parameterized test cases

DeepEval integrates with pytest through its assert_test() function. When a metric score falls below its threshold, assert_test() raises an assertion error, failing the test. The deepeval test run CLI command wraps pytest and returns a non-zero exit code on failure, which is exactly what a CI pipeline needs to block a deployment.

  • Create a new file called evals/test_faithfulness.py by running this command:
touch evals/test_faithfulness.py
  • Confirm the file exists by running:
ls evals/

You should see both run_evaluation.py and test_faithfulness.py listed.

  • Open evals/test_faithfulness.py in your editor and add the imports, dataset loader, and module-level metric setup by copying this code:
import json
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, GEval
from deepeval.test_case import SingleTurnParams
from deepeval.models import AnthropicModel


# Load the golden dataset at module level so all tests share it
def load_golden_dataset():
    with open("golden/financial_faithfulness.json", "r", encoding="utf-8") as f:
        return json.load(f)


# Module-level setup: load data and instantiate metrics once
DATASET = load_golden_dataset()
MODEL = AnthropicModel("claude-sonnet-4-5")

# Generic faithfulness metric from DeepEval
GENERIC_METRIC = FaithfulnessMetric(threshold=0.7, model=MODEL, include_reason=True)

# Custom financial faithfulness judge using GEval
FINANCIAL_METRIC = GEval(
    name="FinancialFaithfulness",
    criteria=(
        "Evaluate whether the actual output is faithfully grounded in the retrieval context, "
        "applying strict financial domain standards:\n"
        "1. Every numerical claim (revenue, profit, growth rates, percentages, ratios) must "
        "exactly match a figure present in the retrieval context.\n"
        "2. Every entity attribution (company name, subsidiary, product line, segment name) "
        "must correctly correspond to the entity discussed in the retrieval context.\n"
        "3. Claims about financial trends or comparisons must be supported by specific data "
        "points and time periods present in the context.\n"
        "4. Market share claims, rankings, or competitive comparisons must be explicitly "
        "stated in the context.\n"
        "5. Any claim not explicitly supported by the retrieval context is unfaithful."
    ),
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.RETRIEVAL_CONTEXT],
    model=MODEL,
    threshold=0.7,
    strict_mode=False,
)

What does this code do?

This block sets up everything the test functions need before any test runs.

  • load_golden_dataset() reads your 8 test cases from golden/financial_faithfulness.json. Loading at module level means the file is read once, not per test.
  • DATASET, MODEL, GENERIC_METRIC, and FINANCIAL_METRIC are module-level constants so pytest can reference them in parameterized test functions.
  • Both metrics use the same AnthropicModel("claude-sonnet-4-5") instance you used in run_evaluation.py, keeping the judge consistent across your evaluation and CI pipeline.

You won't see a change yet. The next chunk adds the test functions that use this setup.

  • Below the FINANCIAL_METRIC definition, add the test function for faithful cases:
@pytest.mark.parametrize(
    "test_case_data",
    [tc for tc in DATASET if tc["expected_faithful"]],
    ids=[f"faithful_{i}" for i in range(sum(1 for tc in DATASET if tc["expected_faithful"]))],
)
def test_faithful_cases_pass(test_case_data):
    # Build an LLMTestCase from each golden dataset entry
    test_case = LLMTestCase(
        input=test_case_data["input"],
        actual_output=test_case_data["actual_output"],
        retrieval_context=test_case_data["retrieval_context"],
    )
    # assert_test raises AssertionError if any metric falls below threshold
    assert_test(test_case, [GENERIC_METRIC, FINANCIAL_METRIC])

What does this code do?

  • @pytest.mark.parametrize loops through every test case where expected_faithful is True. Each one becomes its own named test (faithful_0, faithful_1, etc.).
  • assert_test() takes the test case and a list of metrics. If either metric scores below 0.7, the test fails with a detailed error message. These cases should all pass because they contain grounded RAG outputs.

You won't see a change yet. The next chunk adds the adversarial detection tests.

  • Below test_faithful_cases_pass, add the test function that verifies adversarial cases are caught:
@pytest.mark.parametrize(
    "test_case_data",
    [tc for tc in DATASET if not tc["expected_faithful"]],
    ids=[
        tc.get("adversarial_type", f"adversarial_{i}")
        for i, tc in enumerate(tc for tc in DATASET if not tc["expected_faithful"])
    ],
)
def test_adversarial_cases_detected(test_case_data):
    # Build an LLMTestCase from the adversarial golden dataset entry
    test_case = LLMTestCase(
        input=test_case_data["input"],
        actual_output=test_case_data["actual_output"],
        retrieval_context=test_case_data["retrieval_context"],
    )
    # Measure with the custom financial judge only
    FINANCIAL_METRIC.measure(test_case)
    # The score should be below 0.7 for unfaithful output
    assert FINANCIAL_METRIC.score < 0.7, (
        f"Custom judge failed to detect unfaithful output. "
        f"Score: {FINANCIAL_METRIC.score:.2f}, Reason: {FINANCIAL_METRIC.reason}"
    )

What does this code do?

This test function takes a different approach from the faithful tests.

  • Instead of using assert_test(), it calls FINANCIAL_METRIC.measure() directly and asserts the score is below 0.7. This inverts the logic: the test passes when the judge correctly identifies unfaithful output.
  • The ids parameter uses each case's adversarial_type field (fabricated_figures, wrong_entity_attribution, unsourced_statistics) as the test name, making failures easy to diagnose.
  • If the judge scores an adversarial case above 0.7, the assertion message prints the score and reason so you can see exactly what the judge missed.
  • Save evals/test_faithfulness.py.

Seeing a syntax error?

  • Check that the @pytest.mark.parametrize decorator sits directly above each def test_ line with no blank lines between them.
  • Make sure the ids list comprehension in test_adversarial_cases_detected has matching parentheses and brackets. The nested generator expression can be tricky.

Why isn't my test file valid Python?

✔️ Awesome, I've got everything!

Great. Make sure you've saved evals/test_faithfulness.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete evals/test_faithfulness.py file. Compare it against yours:

import json
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, GEval
from deepeval.test_case import SingleTurnParams
from deepeval.models import AnthropicModel


def load_golden_dataset():
    with open("golden/financial_faithfulness.json", "r", encoding="utf-8") as f:
        return json.load(f)


DATASET = load_golden_dataset()
MODEL = AnthropicModel("claude-sonnet-4-5")

GENERIC_METRIC = FaithfulnessMetric(threshold=0.7, model=MODEL, include_reason=True)

FINANCIAL_METRIC = GEval(
    name="FinancialFaithfulness",
    criteria=(
        "Evaluate whether the actual output is faithfully grounded in the retrieval context, "
        "applying strict financial domain standards:\n"
        "1. Every numerical claim (revenue, profit, growth rates, percentages, ratios) must "
        "exactly match a figure present in the retrieval context.\n"
        "2. Every entity attribution (company name, subsidiary, product line, segment name) "
        "must correctly correspond to the entity discussed in the retrieval context.\n"
        "3. Claims about financial trends or comparisons must be supported by specific data "
        "points and time periods present in the context.\n"
        "4. Market share claims, rankings, or competitive comparisons must be explicitly "
        "stated in the context.\n"
        "5. Any claim not explicitly supported by the retrieval context is unfaithful."
    ),
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.RETRIEVAL_CONTEXT],
    model=MODEL,
    threshold=0.7,
    strict_mode=False,
)


@pytest.mark.parametrize(
    "test_case_data",
    [tc for tc in DATASET if tc["expected_faithful"]],
    ids=[f"faithful_{i}" for i in range(sum(1 for tc in DATASET if tc["expected_faithful"]))],
)
def test_faithful_cases_pass(test_case_data):
    test_case = LLMTestCase(
        input=test_case_data["input"],
        actual_output=test_case_data["actual_output"],
        retrieval_context=test_case_data["retrieval_context"],
    )
    assert_test(test_case, [GENERIC_METRIC, FINANCIAL_METRIC])


@pytest.mark.parametrize(
    "test_case_data",
    [tc for tc in DATASET if not tc["expected_faithful"]],
    ids=[
        tc.get("adversarial_type", f"adversarial_{i}")
        for i, tc in enumerate(tc for tc in DATASET if not tc["expected_faithful"])
    ],
)
def test_adversarial_cases_detected(test_case_data):
    test_case = LLMTestCase(
        input=test_case_data["input"],
        actual_output=test_case_data["actual_output"],
        retrieval_context=test_case_data["retrieval_context"],
    )
    FINANCIAL_METRIC.measure(test_case)
    assert FINANCIAL_METRIC.score < 0.7, (
        f"Custom judge failed to detect unfaithful output. "
        f"Score: {FINANCIAL_METRIC.score:.2f}, Reason: {FINANCIAL_METRIC.reason}"
    )
Run the test suite and verify pass/fail behavior

The deepeval test run command wraps pytest with extra evaluation features. It prints per-test scores and reasons, and returns a non-zero exit code if any test fails. This is what makes it useful as a CI gate.

Cost awareness before running

Each full test run makes approximately 20-30 Claude API calls for generation and evaluation. Estimated cost per run is around $0.50-1.00. Keep this in mind if you plan to run the suite multiple times.

  • Run the test suite by executing this command from your fin-rag-eval/ directory:
deepeval test run evals/test_faithfulness.py

Why deepeval test run instead of pytest?

DeepEval recommends using deepeval test run instead of running pytest directly. The deepeval test run command adds LLM-specific features on top of pytest, including structured metric reporting and proper exit codes for CI gating.

This will take a few minutes. DeepEval runs each test case through the Claude-based judges and reports results as they complete.

What should I see?

The output shows pytest-style results with DeepEval metric details. You should see two categories of results:

  • The 5 faithful test cases (faithful_0 through faithful_4) should pass. Both the generic and financial metrics score them above 0.7 because the RAG outputs are grounded in the retrieved context.
  • The 3 adversarial test cases (fabricated_figures, wrong_entity_attribution, unsourced_statistics) should also pass. They pass because the custom financial judge correctly scores them below 0.7, and the test asserts that the score IS below threshold.

If all 8 tests pass, your CI gate is working correctly. Grounded outputs clear the bar, and ungrounded outputs get caught.

Some tests failing unexpectedly?

  • If a faithful case fails, check that your golden dataset was generated with the grounding prompt intact in rag_pipeline.py. The system prompt must instruct Claude to answer only from the provided context.
  • If an adversarial case fails (judge scores it above 0.7), this is the exact gap your custom judge is designed to catch. LLM judges can be non-deterministic. Try re-running the suite once.
  • If you see an import error, make sure you are running the command from the fin-rag-eval/ project root so that golden/financial_faithfulness.json resolves correctly.

Why are my DeepEval faithfulness tests failing?

Understand how the CI gate catches regressions

Your test suite now has two layers of protection. The faithful cases verify that grounded outputs stay grounded. The adversarial cases verify that the judge can detect ungrounded outputs.

Consider what would happen if someone removed the grounding instructions from the system prompt in rag_pipeline.py.

  • In rag_pipeline.py, find the system parameter in the generate() function. It currently reads:
system=(
    "You are a financial analyst assistant. Answer the question ONLY using "
    "the provided context from SEC filings. If the context does not contain "
    "enough information to answer, say so explicitly. Do not use any knowledge "
    "outside the provided context. Cite specific figures and sections when possible."
),

Why does this prompt matter for regressions?

This system prompt constrains Claude to answer strictly from the retrieved context. If someone changed it to a generic prompt like "You are a helpful assistant", Claude would start using its general knowledge to fill gaps.

The resulting answers might sound plausible but include revenue figures, growth rates, or competitive comparisons not present in the SEC filing context. Those are exactly the hallucination patterns your adversarial test cases simulate: fabricated figures, wrong entity attributions, and unsourced statistics.

If you regenerated the golden dataset with a weakened prompt and re-ran the tests, the previously-faithful cases would now fail because the new outputs would contain unsupported claims. That is the regression your CI gate catches.

Secret mission

Comparative Adversarial Benchmark

Your custom financial judge catches hallucinations the generic one misses. But how much better is it, really? Build a head-to-head adversarial benchmark with five targeted test cases designed to exploit blind spots in generic faithfulness evaluation. Prove the value of domain-specific judges with hard numbers.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing cloud costs. The only costs you incurred were per-call Anthropic API charges during test runs.

Resources you used:

  • fin-rag-eval/ project directory on your Desktop, containing all Python files, the SEC filing data, golden dataset, virtual environment, and .env credentials file.

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend the pipeline with more evaluation metrics.

  • Your project files stay on your Desktop inside fin-rag-eval/.
  • Your .env file still contains your Anthropic API key. Keep it safe and avoid committing it to version control.
  • The all-MiniLM-L6-v2 embedding model is cached on your system by sentence-transformers. It takes up roughly 80 MB but loads faster on future runs.

Pause - I'll come back to this later

Close any running processes to free up memory. Your project files stay on disk so you can pick up where you left off.

  • If your virtual environment is still active, deactivate it by running:
deactivate
  • Close any terminal windows running evaluation scripts.
  • Close your code editor.
  • When you come back, reactivate your virtual environment from inside fin-rag-eval/ by running source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows).

Delete - I don't want to use this again

Remove all project files and start fresh. This deletes your code, golden dataset, downloaded SEC filing, virtual environment, and .env credentials file.

  • If your virtual environment is still active, deactivate it first by running:
deactivate
  • Delete the entire project directory by running the command for your operating system.

macOS

rm -rf ~/Desktop/fin-rag-eval

Windows

rmdir /s /q %USERPROFILE%\Desktop\fin-rag-eval
  • Confirm the directory is gone by running:
ls ~/Desktop | grep fin-rag
  • You should see no output, confirming the directory has been removed.
  • If you exported ANTHROPIC_API_KEY or EDGAR_IDENTITY in your shell session, unset them by running:
unset ANTHROPIC_API_KEY
unset EDGAR_IDENTITY

Clear the embedding model cache

The all-MiniLM-L6-v2 model that sentence-transformers downloaded is cached in your home directory. To reclaim the disk space, delete the HuggingFace cache folder.

On macOS/Linux, run rm -rf ~/.cache/huggingface/hub/sentence-transformers*. On Windows, delete the .cache\huggingface\hub folder inside your user directory.

Nice Work!

Nice Work!

Nice work. You just built a production-grade faithfulness evaluation pipeline for a financial document RAG system, from real SEC filings all the way to automated CI gates.

What you learned:

  • Built a RAG pipeline over real SEC 10-K filings using Claude Sonnet 4.5 for generation and FAISS with sentence-transformers embeddings for retrieval.
  • Built a custom financial faithfulness evaluator using DeepEval's GEval metric with domain-specific criteria targeting numerical accuracy, entity attribution, and temporal grounding.
  • Wired both judges into a pytest CI gate that blocks on faithfulness regressions, using a golden evaluation dataset mixing faithful and adversarial financial test cases.
  • Secret Mission: Benchmarked your custom judge against the generic one on adversarial financial cases, proving domain-specific evaluation catches failures generic judges miss.

Ready to quiz yourself?