Multi-Agent RAG with Hallucination Detection

Introduction

30 Second Summary

AI chatbots can sound confident while saying things that are completely wrong. In production systems, a single hallucinated answer can erode user trust or cause real harm.

In this project, you will build a multi-agent RAG system that retrieves context from a knowledge base, generates grounded answers, and automatically detects and corrects hallucinations. You will also build an evaluation framework that scores every response and calibrates itself against your own human ratings.

What You'll Build

A production-grade multi-agent RAG pipeline with three specialized agents coordinated by LangGraph, plus an auto-judge evaluation framework with human-AI calibration, all running locally through Ollama.

By the end of this project, you'll have:

• A LangGraph multi-agent pipeline with a retriever, generator, and hallucination checker that self-heals by automatically regenerating unfaithful answers.
• An LLM-as-judge evaluation framework that scores every response on correctness, helpfulness, and faithfulness using rubric-based scoring.
• A human-AI calibration workflow that compares your own ratings against the LLM judge and improves its accuracy with few-shot corrections.
• A Gradio web interface with a chat tab showing agent execution in real time and an evaluation dashboard for batch scoring and calibration.
• Secret Mission: Add evaluation regression detection that tracks scores across runs and alerts when quality drops below a threshold.

Are there any prerequisites?

You should have basic Python knowledge and familiarity with APIs and JSON. You will also need Python 3.10+, Ollama, and VS Code installed on your machine (macOS).

Before We Start

Building a multi-agent RAG system requires several tools working together. You need AI models for generation and hallucination detection, an embedding model for vector search, and libraries to orchestrate the entire pipeline.

In this step, you will pull three Ollama models, create a Python project with pinned dependencies, and verify that your complete toolchain is functional.

In this step, get ready to:

• Pull three specialized Ollama models for generation, hallucination detection, and embeddings.
• Create a Python project with pinned dependencies.
• Verify that every tool in your toolchain is functional.

Pull the three Ollama models

This project uses three specialized models instead of one general-purpose model. Each model is purpose-built for a specific job in the pipeline.

`llama3.2` handles generation and evaluation. `bespoke-minicheck` is a purpose-built hallucination detection classifier. `nomic-embed-text` generates vector embeddings for document search.

• Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Terminal and press Enter to open it.

• Verify that Ollama is installed by running this command:

ollama --version

✔️ I see a version number

Ollama is ready to go. Move on to the next instruction.

ⓧ Command not found

• Download the Ollama installer from ollama.com/download.
• Open the downloaded .dmg file (macOS) or .exe installer (Windows) and follow the prompts to install.
• Close and reopen your terminal, then run ollama --version again to confirm the installation.

• Pull the main generation model by running this command:

ollama pull llama3.2

This downloads the llama3.2 model (around 2 GB). It may take a few minutes depending on your internet speed.

• Pull the hallucination detection model by running:

ollama pull bespoke-minicheck

• Pull the embedding model by running:

ollama pull nomic-embed-text

• Confirm all three models are downloaded by running:

ollama list

You should see llama3.2, bespoke-minicheck, and nomic-embed-text listed in the output.

Model pull failing or timing out?

Make sure the Ollama application is running. On macOS, look for the Ollama icon in your menu bar. If it is not there, open Ollama from your Applications folder.

If downloads keep stalling, check your internet connection and try the pull command again. Ollama resumes partial downloads automatically.

Help me troubleshoot Ollama model pulls that are failing.

• Test that llama3.2 responds correctly by running:

ollama run llama3.2 "Say hello in exactly one word."

You should see a short response from the model. The exact wording will vary, but any coherent reply confirms the model is working.

Why three separate models?

Production AI systems use specialized models for different tasks instead of one model for everything. `bespoke-minicheck` is a purpose-built factuality classifier that is faster and more consistent than a general LLM for hallucination detection.

`nomic-embed-text` is optimized for generating vector embeddings, while `llama3.2` handles the tasks that need natural language reasoning. Matching the right model to the right job is a key production pattern.

Create your project and install dependencies

• Verify your Python version by running this command:

python3 --version

✔️ I see Python 3.10 or higher

Python is ready. Move on to the next instruction.

ⓧ I see an older version or command not found

This project requires Python 3.10 or higher.

• Download the latest Python installer from python.org/downloads.
• Run the installer and follow the prompts.
• Close and reopen your terminal, then run python3 --version again.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir multi-agent-rag
cd multi-agent-rag

• Create a Python virtual environment and activate it by running these commands:

python3 -m venv venv
source venv/bin/activate

Your terminal prompt should now show (venv) at the beginning, confirming the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated Python installation for your project. It keeps your project's packages separate from your system Python, so different projects can use different package versions without conflicts.

Now you need a `requirements.txt` file to pin the exact package versions. Pinning versions ensures every learner gets the same behavior.

• Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type VS Code and press Enter to open it.
• Click File in the top menu bar.
• Click Open Folder.
• Select the multi-agent-rag folder on your Desktop, then click Open.

• Create a new file by clicking the new file icon in the Explorer sidebar, or by pressing Cmd+N (macOS) or Ctrl+N (Windows).
• Save the file immediately as requirements.txt by pressing Cmd+S (macOS) or Ctrl+S (Windows).
• Paste the following dependency list into requirements.txt:

langgraph==1.0.9
langchain-ollama==1.1.0
langchain-text-splitters==1.1.2
chromadb==1.5.8
gradio==6.13.0

What are these packages?

• `langgraph` is the graph-based orchestration framework you will use to coordinate your AI agents.
• `langchain-ollama` connects your Python code to the Ollama models running on your machine.
• `langchain-text-splitters` breaks documents into smaller chunks for embedding and retrieval.
• `chromadb` is a vector database that stores document embeddings and finds similar documents by meaning.
• `gradio` builds the web interface where you will interact with the pipeline.

• Save requirements.txt.

• Install all the dependencies by running this command in your terminal:

pip install -r requirements.txt

This will download and install all five packages and their sub-dependencies. It may take a minute or two.

Seeing an installation error?

Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt. If not, run source venv/bin/activate again from inside the multi-agent-rag folder.

If you see a permissions error, do not use sudo. Deactivate, delete the venv folder, and re-create it.

Help me fix a pip install error in my virtual environment.

✔️ Awesome, I've got everything!

Great. Double check you have saved requirements.txt.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt for comparison.

langgraph==1.0.9
langchain-ollama==1.1.0
langchain-text-splitters==1.1.2
chromadb==1.5.8
gradio==6.13.0

Verify the complete toolchain

Before you start building, a quick verification script catches any installation issues. This script tests every tool the project depends on in one run.

• Create a new file in VS Code called test_setup.py inside the multi-agent-rag folder.
• Paste the following verification script into test_setup.py:

# test_setup.py - Verify all tools are working
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_core.messages import HumanMessage
import chromadb

# Test Ollama connection with llama3.2
print("Testing Ollama LLM connection...")
llm = ChatOllama(model="llama3.2", temperature=0.0)
response = llm.invoke([HumanMessage(content="Say hello in one word.")])
print(f"LLM response: {response.content}")

# Test embeddings with nomic-embed-text
print("\nTesting embedding model...")
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vector = embeddings.embed_query("Test sentence for embedding.")
print(f"Embedding dimensions: {len(vector)}")

# Test ChromaDB store and retrieve
print("\nTesting ChromaDB...")
client = chromadb.Client()
collection = client.create_collection("test_collection")
collection.add(
    ids=["test_1"],
    embeddings=[vector],
    documents=["Test document for verification."],
)
results = collection.query(query_embeddings=[vector], n_results=1)
print(f"ChromaDB retrieved: {results['documents'][0][0]}")

print("\nAll systems operational!")

What does this script do?

• It sends a prompt to llama3.2 through ChatOllama to confirm the LLM connection works.
• It generates a vector embedding with nomic-embed-text to confirm the embedding model works.
• It stores a test document in ChromaDB and retrieves it to confirm the vector database works end to end.

• Save test_setup.py.
• Run the verification script by running this command in your terminal:

python test_setup.py

You should see output confirming each check passed, ending with All systems operational!. The LLM response will be a short word. The embedding dimensions should be a number like 768. The ChromaDB line should show the test document text.

Script failing with a connection error?

If you see Connection refused, Ollama is not running. Open the Ollama app from your Applications folder. You should see the Ollama icon appear in your menu bar.

If you see an ImportError, make sure your virtual environment is active ((venv) in your prompt) and that you ran pip install -r requirements.txt successfully.

Help me debug my test_setup.py verification script.

Once all checks pass, you can delete test_setup.py. It was only needed to confirm your setup.

Your environment is fully set up and verified. Next, you will create the knowledge base that your RAG pipeline will search through.

Build the RAG Knowledge Base

Your environment is set up and all three Ollama models are responding. Now it's time to give your AI agents something to work with.

Every RAG system starts with a knowledge base of documents that the retriever can search. In this step, you'll create sample documents, split them into searchable chunks, and store them as vector embeddings in ChromaDB.

In this step, get ready to:

• Create sample documents covering three AI/ML topics.
• Build an ingestion pipeline that chunks and embeds documents into ChromaDB.
• Test retrieval by querying the knowledge base and verifying results.

Create sample documents and set up the file

Before your agents can retrieve and answer questions, they need a body of text to search through. You'll create three detailed documents about AI/ML topics that serve as your knowledge base.

• Create a new file called knowledge_base.py in your multi-agent-rag folder.
• Add the following imports, constants, and sample documents:

import chromadb
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings

# Path where ChromaDB stores its data on disk
PERSIST_DIR = "./chroma_db"
COLLECTION_NAME = "rag_knowledge_base"

# Three documents covering core AI/ML topics for the knowledge base
SAMPLE_DOCUMENTS = [
    {
        "title": "Transformer Architecture",
        "content": (
            "The Transformer architecture was introduced in the paper "
            "'Attention Is All You Need' by Vaswani et al. in 2017. It relies "
            "entirely on self-attention mechanisms, dispensing with recurrence "
            "and convolutions entirely. The model uses multi-head attention to "
            "jointly attend to information from different representation "
            "subspaces at different positions. The architecture consists of an "
            "encoder and decoder, each composed of stacked layers of multi-head "
            "self-attention and position-wise fully connected feed-forward "
            "networks. Positional encodings are added to input embeddings to "
            "inject sequence order information since the model contains no "
            "recurrence or convolution. The original Transformer achieved "
            "state-of-the-art results on WMT 2014 English-to-German and "
            "English-to-French translation benchmarks while being more "
            "parallelizable and requiring significantly less time to train "
            "than recurrent architectures. The encoder maps an input sequence "
            "of symbol representations to a sequence of continuous "
            "representations, and the decoder generates an output sequence "
            "one element at a time in an autoregressive fashion."
        ),
    },
    {
        "title": "Retrieval-Augmented Generation",
        "content": (
            "Retrieval-Augmented Generation (RAG) is a technique that enhances "
            "large language models by retrieving relevant documents from an "
            "external knowledge base before generating a response. The process "
            "has two main stages: retrieval and generation. During retrieval, "
            "the user query is converted to an embedding vector and compared "
            "against document embeddings in a vector database to find the most "
            "semantically similar chunks. During generation, the retrieved "
            "context is prepended to the user query and passed to the LLM, "
            "which generates an answer grounded in the provided context. RAG "
            "reduces hallucinations by providing factual grounding, enables "
            "knowledge updates without model retraining, and allows "
            "attribution of answers to source documents. Common vector "
            "databases used in RAG include ChromaDB, Pinecone, Weaviate, and "
            "Qdrant. Key parameters that affect RAG quality include chunk "
            "size, chunk overlap, top-K retrieval count, and the embedding "
            "model used. A typical chunk size is 500 to 1000 tokens with 10 "
            "to 20 percent overlap between consecutive chunks."
        ),
    },
    {
        "title": "LLM Hallucinations",
        "content": (
            "LLM hallucinations occur when a model generates plausible-sounding "
            "but factually incorrect or ungrounded information. There are two "
            "main types: intrinsic hallucinations contradict the provided source "
            "context, while extrinsic hallucinations add information not present "
            "in any provided context. Hallucinations arise from several factors "
            "including pattern completion tendencies, training data memorization, "
            "authority bias, and context window limitations. Detection methods "
            "include faithfulness scoring which checks if claims are supported "
            "by context, semantic similarity comparison between answer and "
            "source, Natural Language Inference (NLI) verification, and "
            "stochastic consistency checking across multiple generations. The "
            "standard production pipeline for hallucination detection involves "
            "three steps: extracting atomic claims from the generated answer, "
            "verifying each claim against source documents using an NLI model "
            "or LLM-as-judge, and computing a faithfulness score as the ratio "
            "of supported claims to total claims. A faithfulness threshold of "
            "0.8 or higher is commonly used in production systems. Companies "
            "like Stripe and Notion run real-time faithfulness scoring on every "
            "LLM response in production."
        ),
    },
]

What does this code do?

• The three imports bring in ChromaDB for vector storage, RecursiveCharacterTextSplitter for breaking documents into chunks, and OllamaEmbeddings for generating vector embeddings locally.
• PERSIST_DIR tells ChromaDB where to save its data on disk so your knowledge base survives between script runs.
• SAMPLE_DOCUMENTS contains three articles about Transformers, RAG, and LLM Hallucinations. These give the retriever enough variety to return different results depending on the question.

• Save knowledge_base.py.
• You won't see a change yet. The next chunk adds the functions that process these documents.

Why does chunk size and overlap matter?

When a document is too long to fit in one embedding, you split it into smaller chunks. Chunk size controls how much text each piece contains.

If chunks are too large, your retriever returns noisy results that dilute the relevant information. If chunks are too small, you lose the surrounding context that makes a fact meaningful.

Overlap (the shared text between consecutive chunks) prevents important sentences from being cut in half at a boundary. A typical setup is 500 to 1000 characters per chunk with 10 to 20 percent overlap.

Embed documents and store in ChromaDB

With the raw documents in place, you need a function that splits them into chunks, converts each chunk into a vector embedding, and stores everything in ChromaDB. This is the standard RAG ingestion pipeline.

• In knowledge_base.py, add the create_knowledge_base() function below the SAMPLE_DOCUMENTS list:

def create_knowledge_base():
    # Set up the embedding model and text splitter
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,       # Max characters per chunk
        chunk_overlap=50,     # Shared characters between consecutive chunks
        separators=["\n\n", "\n", ". ", " ", ""],
    )

    # Create a fresh ChromaDB collection (delete old one if it exists)
    client = chromadb.PersistentClient(path=PERSIST_DIR)
    try:
        client.delete_collection(COLLECTION_NAME)
    except ValueError:
        pass

    collection = client.create_collection(
        name=COLLECTION_NAME,
        metadata={"hnsw:space": "cosine"},
    )

    # Split each document into chunks and store as embeddings
    doc_id = 0
    for doc in SAMPLE_DOCUMENTS:
        chunks = text_splitter.split_text(doc["content"])
        for chunk in chunks:
            embedding = embeddings.embed_query(chunk)
            collection.add(
                ids=[f"doc_{doc_id}"],
                embeddings=[embedding],
                documents=[chunk],
                metadatas=[{"title": doc["title"]}],
            )
            doc_id += 1

    print(f"Knowledge base created with {doc_id} chunks in {PERSIST_DIR}")
    return collection

What does this function do?

• RecursiveCharacterTextSplitter splits each document into chunks of at most 500 characters, with 50 characters of overlap between neighbors.
• The separators list tells the splitter to prefer splitting at paragraph breaks first, then newlines, then sentences, then words. This keeps chunks as coherent as possible.
• The function deletes any existing collection before re-creating it, so running the script twice doesn't duplicate data.
• Each chunk is embedded with nomic-embed-text and stored in ChromaDB with its source document title as metadata.

• Save knowledge_base.py.
• You won't see a change yet. The next chunk adds the retrieval function and a test runner.

Why cosine similarity?

The "hnsw:space": "cosine" metadata tells ChromaDB to measure similarity using cosine distance. Cosine distance compares the direction of two vectors, ignoring their magnitude. This means a short chunk and a long chunk about the same topic still score as similar.

Test retrieval quality

The ingestion function stores documents, but you also need a way to query them. You'll add a retrieval function and a test runner that populates the knowledge base and runs a sample query.

• In knowledge_base.py, add the retrieve_documents() function and the __main__ test block below create_knowledge_base():

def retrieve_documents(query, top_k=3):
    # Connect to the existing ChromaDB collection
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    client = chromadb.PersistentClient(path=PERSIST_DIR)
    collection = client.get_collection(COLLECTION_NAME)

    # Embed the query and find the closest chunks
    query_embedding = embeddings.embed_query(query)
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=top_k,
    )

    # Package results into a list of dicts with content, metadata, and distance
    documents = []
    for i, doc_text in enumerate(results["documents"][0]):
        documents.append(
            {
                "content": doc_text,
                "metadata": results["metadatas"][0][i],
                "distance": results["distances"][0][i]
                if results.get("distances")
                else None,
            }
        )
    return documents


if __name__ == "__main__":
    create_knowledge_base()
    results = retrieve_documents("What is RAG?")
    for r in results:
        title = r["metadata"]["title"]
        dist = r["distance"]
        print(f"\n[{title}] (distance: {dist:.4f})")
        print(r["content"][:200])

What does this code do?

• retrieve_documents() embeds the query using the same nomic-embed-text model, then asks ChromaDB for the top_k closest chunks by cosine similarity.
• Each result includes the chunk text, its source document title, and the cosine distance. Lower distance means higher similarity.
• The __main__ block first populates the knowledge base, then queries it with "What is RAG?" and prints the top 3 matching chunks.

• Save knowledge_base.py.
• Run the script by running this command:

python knowledge_base.py

You should see output showing the knowledge base was created, followed by three retrieved chunks. The chunks from the "Retrieval-Augmented Generation" document should appear with the lowest distances.

What should I see?

The first line confirms how many chunks were created (you should see around 10 to 15 depending on how the splitter divides the text). Below that, each result shows a document title, a cosine distance score, and a preview of the chunk content.

For the query "What is RAG?", the top results should come from the "Retrieval-Augmented Generation" document with distances well below 1.0. If you see results primarily from the other two documents, something went wrong with the embeddings.

Seeing an error when running the script?

• If you see ModuleNotFoundError, make sure your virtual environment is activated. Run source venv/bin/activate and try again.
• If you see a connection error mentioning Ollama, confirm Ollama is running. Check for the Ollama icon in your macOS menu bar.
• If the distance scores are all very close to each other, the embeddings may not be loading correctly. Verify nomic-embed-text is pulled by running ollama list.

Help me debug my knowledge_base.py script

✔️ Awesome, I've got everything!

Great. Double check you've saved knowledge_base.py and that your terminal showed successful retrieval results.

ⓧ I'd like to double check the full code

Here is the complete knowledge_base.py file. Compare it against yours and fix any differences.

import chromadb
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings

PERSIST_DIR = "./chroma_db"
COLLECTION_NAME = "rag_knowledge_base"

SAMPLE_DOCUMENTS = [
    {
        "title": "Transformer Architecture",
        "content": (
            "The Transformer architecture was introduced in the paper "
            "'Attention Is All You Need' by Vaswani et al. in 2017. It relies "
            "entirely on self-attention mechanisms, dispensing with recurrence "
            "and convolutions entirely. The model uses multi-head attention to "
            "jointly attend to information from different representation "
            "subspaces at different positions. The architecture consists of an "
            "encoder and decoder, each composed of stacked layers of multi-head "
            "self-attention and position-wise fully connected feed-forward "
            "networks. Positional encodings are added to input embeddings to "
            "inject sequence order information since the model contains no "
            "recurrence or convolution. The original Transformer achieved "
            "state-of-the-art results on WMT 2014 English-to-German and "
            "English-to-French translation benchmarks while being more "
            "parallelizable and requiring significantly less time to train "
            "than recurrent architectures. The encoder maps an input sequence "
            "of symbol representations to a sequence of continuous "
            "representations, and the decoder generates an output sequence "
            "one element at a time in an autoregressive fashion."
        ),
    },
    {
        "title": "Retrieval-Augmented Generation",
        "content": (
            "Retrieval-Augmented Generation (RAG) is a technique that enhances "
            "large language models by retrieving relevant documents from an "
            "external knowledge base before generating a response. The process "
            "has two main stages: retrieval and generation. During retrieval, "
            "the user query is converted to an embedding vector and compared "
            "against document embeddings in a vector database to find the most "
            "semantically similar chunks. During generation, the retrieved "
            "context is prepended to the user query and passed to the LLM, "
            "which generates an answer grounded in the provided context. RAG "
            "reduces hallucinations by providing factual grounding, enables "
            "knowledge updates without model retraining, and allows "
            "attribution of answers to source documents. Common vector "
            "databases used in RAG include ChromaDB, Pinecone, Weaviate, and "
            "Qdrant. Key parameters that affect RAG quality include chunk "
            "size, chunk overlap, top-K retrieval count, and the embedding "
            "model used. A typical chunk size is 500 to 1000 tokens with 10 "
            "to 20 percent overlap between consecutive chunks."
        ),
    },
    {
        "title": "LLM Hallucinations",
        "content": (
            "LLM hallucinations occur when a model generates plausible-sounding "
            "but factually incorrect or ungrounded information. There are two "
            "main types: intrinsic hallucinations contradict the provided source "
            "context, while extrinsic hallucinations add information not present "
            "in any provided context. Hallucinations arise from several factors "
            "including pattern completion tendencies, training data memorization, "
            "authority bias, and context window limitations. Detection methods "
            "include faithfulness scoring which checks if claims are supported "
            "by context, semantic similarity comparison between answer and "
            "source, Natural Language Inference (NLI) verification, and "
            "stochastic consistency checking across multiple generations. The "
            "standard production pipeline for hallucination detection involves "
            "three steps: extracting atomic claims from the generated answer, "
            "verifying each claim against source documents using an NLI model "
            "or LLM-as-judge, and computing a faithfulness score as the ratio "
            "of supported claims to total claims. A faithfulness threshold of "
            "0.8 or higher is commonly used in production systems. Companies "
            "like Stripe and Notion run real-time faithfulness scoring on every "
            "LLM response in production."
        ),
    },
]


def create_knowledge_base():
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=50,
        separators=["\n\n", "\n", ". ", " ", ""],
    )

    client = chromadb.PersistentClient(path=PERSIST_DIR)
    try:
        client.delete_collection(COLLECTION_NAME)
    except ValueError:
        pass

    collection = client.create_collection(
        name=COLLECTION_NAME,
        metadata={"hnsw:space": "cosine"},
    )

    doc_id = 0
    for doc in SAMPLE_DOCUMENTS:
        chunks = text_splitter.split_text(doc["content"])
        for chunk in chunks:
            embedding = embeddings.embed_query(chunk)
            collection.add(
                ids=[f"doc_{doc_id}"],
                embeddings=[embedding],
                documents=[chunk],
                metadatas=[{"title": doc["title"]}],
            )
            doc_id += 1

    print(f"Knowledge base created with {doc_id} chunks in {PERSIST_DIR}")
    return collection


def retrieve_documents(query, top_k=3):
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    client = chromadb.PersistentClient(path=PERSIST_DIR)
    collection = client.get_collection(COLLECTION_NAME)

    query_embedding = embeddings.embed_query(query)
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=top_k,
    )

    documents = []
    for i, doc_text in enumerate(results["documents"][0]):
        documents.append(
            {
                "content": doc_text,
                "metadata": results["metadatas"][0][i],
                "distance": results["distances"][0][i]
                if results.get("distances")
                else None,
            }
        )
    return documents


if __name__ == "__main__":
    create_knowledge_base()
    results = retrieve_documents("What is RAG?")
    for r in results:
        title = r["metadata"]["title"]
        dist = r["distance"]
        print(f"\n[{title}] (distance: {dist:.4f})")
        print(r["content"][:200])

Your knowledge base is populated and retrieval is working. Next up, you'll build the multi-agent RAG pipeline that uses this knowledge base to retrieve context, generate answers, and automatically detect hallucinations.

Build the Multi-Agent RAG Pipeline with LangGraph

Your knowledge base is loaded and retrieval is working. Now it's time to build the brain of the system: a multi-agent pipeline that retrieves context, generates answers, and automatically checks those answers for hallucinations.

You'll use LangGraph to orchestrate three specialized agents through an explicit state graph. When the hallucination checker detects unfaithful claims, the graph routes back to the generator for a stricter retry. This is the self-healing pattern used in production AI systems.

In this step, get ready to:

• Define the shared pipeline state and implement three agent nodes (retriever, generator, hallucination checker).
• Wire the agents into a state graph with conditional self-healing routing.
• Test the complete pipeline end-to-end with different question types.

Design the graph state and implement agent nodes

Every agent in the pipeline reads from and writes to a single shared state object. This is how LangGraph coordinates multiple agents without them needing to call each other directly.

Why a shared state instead of direct agent calls?

In a typical function call chain, Agent A calls Agent B which calls Agent C. If you want to re-route (say, skip Agent C or loop back to Agent B), you have to rewrite the chain.

With a shared state graph, each agent reads what it needs from the state and writes its results back. The graph decides what runs next based on conditions. This makes self-healing loops and dynamic routing simple to add.

• Create a new file called rag_graph.py in your multi-agent-rag folder.
• Add the imports, state definition, and retriever agent by pasting this code:

import json
from typing import TypedDict, List

from langgraph.graph import StateGraph, END
from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage

# Import the retrieval function from your knowledge base module
from knowledge_base import retrieve_documents


# Shared state that every agent reads from and writes to
class RAGState(TypedDict):
    question: str
    retrieved_docs: List[dict]
    context: str
    answer: str
    hallucination_checks: List[dict]
    faithfulness_score: float
    retry_count: int
    max_retries: int
    agent_log: List[str]


def retrieve_node(state: RAGState) -> dict:
    """Query ChromaDB and combine the top chunks into a single context string."""
    docs = retrieve_documents(state["question"], top_k=3)
    context = "\n\n".join([d["content"] for d in docs])
    return {
        "retrieved_docs": docs,
        "context": context,
        "agent_log": state["agent_log"] + [f"[Retriever] Found {len(docs)} relevant chunks"],
    }

What does this code do?

RAGState is a TypedDict that holds every piece of data the pipeline needs. Each agent reads only the fields it cares about and returns a partial update. LangGraph merges the update into the shared state automatically.

retrieve_node is the first agent. It calls the retrieve_documents function you built in the previous step and joins the returned chunks into one context string. The agent_log field tracks what each agent did so you can debug the pipeline later.

You won't see a change yet. The next chunk adds the generator agent that uses this context.

The generator is the agent that takes the retrieved context and produces an answer. It also needs to handle a special case: when it's retrying after a failed hallucination check, it uses a stricter system prompt.

• In rag_graph.py, add the generator agent below retrieve_node by pasting this code:

def generate_node(state: RAGState) -> dict:
    """Generate an answer grounded in the retrieved context."""
    llm = ChatOllama(model="llama3.2", temperature=0.1)
    is_retry = state["retry_count"] > 0
    # Use a stricter prompt if this is a retry after a hallucination check failure
    if is_retry:
        system_prompt = (
            "You are a precise assistant. Your previous answer contained "
            "claims not supported by the context. This time you MUST:\n"
            "1. ONLY use information explicitly stated in the context.\n"
            "2. If the context lacks information, say so clearly.\n"
            "3. Do NOT add anything from your training data.\n"
            "4. Quote or closely paraphrase the source material."
        )
    else:
        system_prompt = (
            "You are a helpful assistant. Answer the question using ONLY "
            "the context provided below. If the context does not contain "
            "enough information to answer, say so clearly. Do not make up "
            "or infer information beyond what the context states."
        )
    prompt = (
        f"{system_prompt}\n\n"
        f"Context:\n{state['context']}\n\n"
        f"Question: {state['question']}\n\n"
        f"Answer:"
    )
    response = llm.invoke([HumanMessage(content=prompt)])
    answer = response.content
    verb = "Regenerated" if is_retry else "Generated"
    log_msg = f"[Generator] {verb} answer ({len(answer)} chars)"
    return {"answer": answer, "agent_log": state["agent_log"] + [log_msg]}

What does this code do?

• is_retry checks whether this is the first attempt or a do-over. On a retry, the prompt is much stricter and demands exact grounding in the source material.
• ChatOllama(model="llama3.2", temperature=0.1) creates a low-temperature LLM connection. Low temperature means more deterministic outputs, which is what you want for grounded answers.
• The agent returns only the fields it changes (answer and agent_log). LangGraph merges these into the full state.

You won't see a change yet. The next chunk adds the hallucination checker that verifies the generator's output.

The hallucination checker is the most complex agent. It works in three stages: extract atomic claims from the answer, verify each claim against the source context, and compute a faithfulness score.

• In rag_graph.py, add the hallucination checker below generate_node by pasting this code:

def check_hallucination_node(state: RAGState) -> dict:
    """Extract claims from the answer and verify each one against the context."""
    llm = ChatOllama(model="llama3.2", temperature=0.0)
    # Step 1: Ask llama3.2 to break the answer into atomic factual claims
    extract_prompt = (
        "Extract all factual claims from the following answer as a JSON "
        "list of strings. Each claim should be a single, self-contained "
        "statement with no pronouns.\n\n"
        f"Answer: {state['answer']}\n\n"
        "Respond with ONLY a JSON list of strings. "
        'Example: ["claim 1", "claim 2"]'
    )
    extract_response = llm.invoke([HumanMessage(content=extract_prompt)])
    # Parse the JSON list with fallback handling for malformed responses
    try:
        raw = extract_response.content
        start = raw.find("[")
        end = raw.rfind("]") + 1
        if start != -1 and end > start:
            claims = json.loads(raw[start:end])
        else:
            claims = [state["answer"]]
    except (json.JSONDecodeError, TypeError):
        claims = [state["answer"]]
    if not isinstance(claims, list) or len(claims) == 0:
        claims = [state["answer"]]
    # Step 2: Check each claim against context using bespoke-minicheck
    minicheck = ChatOllama(model="bespoke-minicheck", temperature=0.0)
    checks = []
    supported_count = 0
    for claim in claims:
        # bespoke-minicheck expects Document: ... / Claim: ... format
        check_prompt = f"Document: {state['context']}\nClaim: {claim}"
        result = minicheck.invoke([HumanMessage(content=check_prompt)])
        verdict = result.content.strip().lower().startswith("yes")
        checks.append({"claim": claim, "supported": verdict, "raw_response": result.content.strip()})
        if verdict:
            supported_count += 1
    # Step 3: Compute faithfulness as ratio of supported claims to total claims
    score = supported_count / len(claims) if claims else 0.0
    log_msg = f"[Hallucination Checker] Faithfulness: {score:.2f} ({supported_count}/{len(claims)} claims supported)"
    return {
        "hallucination_checks": checks,
        "faithfulness_score": score,
        "retry_count": state["retry_count"] + 1,
        "agent_log": state["agent_log"] + [log_msg],
    }

What does this code do?

This function implements the three-stage faithfulness pipeline used by production evaluation frameworks like RAGAS and DeepEval.

• Stage 1 uses llama3.2 to break the answer into atomic claims. Each claim is a single, self-contained statement with no pronouns.
• Stage 2 uses bespoke-minicheck, a purpose-built factuality classifier. It takes a Document + Claim pair and returns Yes or No. This is faster and more consistent than using a general-purpose LLM for verification.
• Stage 3 computes faithfulness as the ratio of supported claims to total claims. A score of 0.8 means 80% of the claims are backed by the source context.

• Save rag_graph.py.

Seeing a syntax error on save?

• Check for mismatched parentheses in the long string concatenations. Each opening ( in the prompt strings needs a matching ).
• Make sure the f-string quotes inside extract_prompt use matching quote types. The outer strings use double quotes, and the JSON example inside uses escaped double quotes.
• Help me debug a syntax error in my check_hallucination_node function.

Wire the LangGraph with conditional self-healing

You have three agents, but they don't know about each other yet. The next piece is the routing logic that decides what happens after the hallucination check, plus the graph that connects everything.

• In rag_graph.py, add the routing function, graph builder, and query runner below check_hallucination_node by pasting this code:

def should_retry(state: RAGState) -> str:
    """Decide whether to retry generation or accept the answer."""
    if state["faithfulness_score"] >= 0.8:
        return "accept"
    if state["retry_count"] >= state["max_retries"]:
        return "accept"
    return "retry"


def build_rag_graph():
    """Build and compile the multi-agent RAG graph."""
    graph = StateGraph(RAGState)
    # Register each agent as a named node
    graph.add_node("retrieve", retrieve_node)
    graph.add_node("generate", generate_node)
    graph.add_node("check_hallucination", check_hallucination_node)
    # Set the entry point and connect nodes in sequence
    graph.set_entry_point("retrieve")
    graph.add_edge("retrieve", "generate")
    graph.add_edge("generate", "check_hallucination")
    # Conditional edge: retry generation or accept the answer
    graph.add_conditional_edges(
        "check_hallucination",
        should_retry,
        {"retry": "generate", "accept": END},
    )
    return graph.compile()


def run_query(question, max_retries=2):
    """Run a question through the full RAG pipeline."""
    graph = build_rag_graph()
    initial_state = {
        "question": question,
        "retrieved_docs": [],
        "context": "",
        "answer": "",
        "hallucination_checks": [],
        "faithfulness_score": 0.0,
        "retry_count": 0,
        "max_retries": max_retries,
        "agent_log": [],
    }
    result = graph.invoke(initial_state)
    return result

What does this code do?

• should_retry is the routing function. If the faithfulness score is 0.8 or higher, the answer is accepted. If the score is below 0.8 but retries remain, it routes back to the generator for another attempt.
• build_rag_graph assembles the state graph. add_conditional_edges is what makes self-healing possible. It maps the string returned by should_retry to the next node in the graph.
• run_query is a convenience wrapper. It builds the graph, sets up a blank initial state, and invokes the full pipeline.

• Save rag_graph.py.

You won't see a change yet. The next section adds the test script that runs the pipeline.

Test the pipeline end-to-end

The graph is fully wired. Time to send a question through it and watch all three agents work in sequence.

• In rag_graph.py, add the test block at the bottom of the file by pasting this code:

if __name__ == "__main__":
    result = run_query("What is retrieval-augmented generation and how does it work?")
    print("\n=== Agent Execution Log ===")
    for log_entry in result["agent_log"]:
        print(log_entry)
    print(f"\n=== Answer ===\n{result['answer']}")
    print(f"\n=== Faithfulness: {result['faithfulness_score']:.2f} ===")
    for check in result["hallucination_checks"]:
        status = "SUPPORTED" if check["supported"] else "NOT SUPPORTED"
        print(f"  [{status}] {check['claim']}")

What does this code do?

This test block sends a question about RAG through the entire pipeline. It prints the agent execution log (showing which agents ran), the generated answer, the overall faithfulness score, and the verdict for each individual claim.

• Save rag_graph.py.
• Make sure Ollama is running, then run the pipeline by running this command:

python rag_graph.py

This takes 30 to 60 seconds because it runs three models in sequence. You should see output that includes the agent execution log, a generated answer about RAG, a faithfulness score, and a list of individual claim verdicts.

What should I see?

The agent execution log shows three entries: the Retriever finding chunks, the Generator producing an answer, and the Hallucination Checker reporting a faithfulness score. If the score is below 0.8, you'll see additional log entries for the retry loop.

Each claim shows either SUPPORTED or NOT SUPPORTED. A faithfulness score of 0.80 or higher means the pipeline accepted the answer on the first try.

Getting a connection error or timeout?

• Make sure Ollama is running. Look for the Ollama icon in your macOS menu bar. If it's not there, open Ollama from your Applications folder.
• Verify all three models are pulled by running ollama list in a separate terminal window. You should see llama3.2, bespoke-minicheck, and nomic-embed-text listed.
• If you see a ChromaDB error about the collection not being found, re-run python knowledge_base.py to recreate the knowledge base first.
• Help me debug my RAG pipeline. It's failing when I run python rag_graph.py.

✔️ Awesome, I've got everything!

Double check you've saved rag_graph.py.

ⓧ I'd like to double check the full code

Here is the complete rag_graph.py file. Compare it against yours and fix any differences.

import json
from typing import TypedDict, List

from langgraph.graph import StateGraph, END
from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage

from knowledge_base import retrieve_documents


class RAGState(TypedDict):
    question: str
    retrieved_docs: List[dict]
    context: str
    answer: str
    hallucination_checks: List[dict]
    faithfulness_score: float
    retry_count: int
    max_retries: int
    agent_log: List[str]


def retrieve_node(state: RAGState) -> dict:
    docs = retrieve_documents(state["question"], top_k=3)
    context = "\n\n".join([d["content"] for d in docs])
    return {
        "retrieved_docs": docs,
        "context": context,
        "agent_log": state["agent_log"] + [f"[Retriever] Found {len(docs)} relevant chunks"],
    }


def generate_node(state: RAGState) -> dict:
    llm = ChatOllama(model="llama3.2", temperature=0.1)
    is_retry = state["retry_count"] > 0
    if is_retry:
        system_prompt = (
            "You are a precise assistant. Your previous answer contained "
            "claims not supported by the context. This time you MUST:\n"
            "1. ONLY use information explicitly stated in the context.\n"
            "2. If the context lacks information, say so clearly.\n"
            "3. Do NOT add anything from your training data.\n"
            "4. Quote or closely paraphrase the source material."
        )
    else:
        system_prompt = (
            "You are a helpful assistant. Answer the question using ONLY "
            "the context provided below. If the context does not contain "
            "enough information to answer, say so clearly. Do not make up "
            "or infer information beyond what the context states."
        )
    prompt = (
        f"{system_prompt}\n\n"
        f"Context:\n{state['context']}\n\n"
        f"Question: {state['question']}\n\n"
        f"Answer:"
    )
    response = llm.invoke([HumanMessage(content=prompt)])
    answer = response.content
    verb = "Regenerated" if is_retry else "Generated"
    log_msg = f"[Generator] {verb} answer ({len(answer)} chars)"
    return {"answer": answer, "agent_log": state["agent_log"] + [log_msg]}


def check_hallucination_node(state: RAGState) -> dict:
    llm = ChatOllama(model="llama3.2", temperature=0.0)
    extract_prompt = (
        "Extract all factual claims from the following answer as a JSON "
        "list of strings. Each claim should be a single, self-contained "
        "statement with no pronouns.\n\n"
        f"Answer: {state['answer']}\n\n"
        "Respond with ONLY a JSON list of strings. "
        'Example: ["claim 1", "claim 2"]'
    )
    extract_response = llm.invoke([HumanMessage(content=extract_prompt)])
    try:
        raw = extract_response.content
        start = raw.find("[")
        end = raw.rfind("]") + 1
        if start != -1 and end > start:
            claims = json.loads(raw[start:end])
        else:
            claims = [state["answer"]]
    except (json.JSONDecodeError, TypeError):
        claims = [state["answer"]]
    if not isinstance(claims, list) or len(claims) == 0:
        claims = [state["answer"]]
    minicheck = ChatOllama(model="bespoke-minicheck", temperature=0.0)
    checks = []
    supported_count = 0
    for claim in claims:
        check_prompt = f"Document: {state['context']}\nClaim: {claim}"
        result = minicheck.invoke([HumanMessage(content=check_prompt)])
        verdict = result.content.strip().lower().startswith("yes")
        checks.append({"claim": claim, "supported": verdict, "raw_response": result.content.strip()})
        if verdict:
            supported_count += 1
    score = supported_count / len(claims) if claims else 0.0
    log_msg = f"[Hallucination Checker] Faithfulness: {score:.2f} ({supported_count}/{len(claims)} claims supported)"
    return {
        "hallucination_checks": checks,
        "faithfulness_score": score,
        "retry_count": state["retry_count"] + 1,
        "agent_log": state["agent_log"] + [log_msg],
    }


def should_retry(state: RAGState) -> str:
    if state["faithfulness_score"] >= 0.8:
        return "accept"
    if state["retry_count"] >= state["max_retries"]:
        return "accept"
    return "retry"


def build_rag_graph():
    graph = StateGraph(RAGState)
    graph.add_node("retrieve", retrieve_node)
    graph.add_node("generate", generate_node)
    graph.add_node("check_hallucination", check_hallucination_node)
    graph.set_entry_point("retrieve")
    graph.add_edge("retrieve", "generate")
    graph.add_edge("generate", "check_hallucination")
    graph.add_conditional_edges(
        "check_hallucination",
        should_retry,
        {"retry": "generate", "accept": END},
    )
    return graph.compile()


def run_query(question, max_retries=2):
    graph = build_rag_graph()
    initial_state = {
        "question": question,
        "retrieved_docs": [],
        "context": "",
        "answer": "",
        "hallucination_checks": [],
        "faithfulness_score": 0.0,
        "retry_count": 0,
        "max_retries": max_retries,
        "agent_log": [],
    }
    result = graph.invoke(initial_state)
    return result


if __name__ == "__main__":
    result = run_query("What is retrieval-augmented generation and how does it work?")
    print("\n=== Agent Execution Log ===")
    for log_entry in result["agent_log"]:
        print(log_entry)
    print(f"\n=== Answer ===\n{result['answer']}")
    print(f"\n=== Faithfulness: {result['faithfulness_score']:.2f} ===")
    for check in result["hallucination_checks"]:
        status = "SUPPORTED" if check["supported"] else "NOT SUPPORTED"
        print(f"  [{status}] {check['claim']}")

Your multi-agent pipeline is working. Three agents coordinate through shared state, and the graph automatically self-heals when it detects unfaithful responses. Next up, you'll take a closer look at how the hallucination detection works and test it with adversarial examples designed to trip it up.

Deep Dive into Hallucination Detection

Your multi-agent pipeline can retrieve context, generate answers, and check for hallucinations automatically. But how does that hallucination checker actually decide which claims are real and which are fabricated?

In this step, you'll unpack the three-stage detection pipeline that production systems like RAGAS and DeepEval use. You'll compare two different verification approaches side by side and stress-test the system with adversarial inputs designed to break it.

In this step, get ready to:

• Understand the three-stage hallucination detection pipeline (claim extraction, NLI verification, faithfulness scoring).
• Compare bespoke-minicheck against LLM-as-judge for claim verification.
• Test the pipeline with adversarial examples designed to trigger hallucinations.

Understand claim extraction and NLI verification

The hallucination detection approach your pipeline uses follows a three-stage pattern. This is the same pattern used by production evaluation frameworks.

• Stage 1: Extract atomic factual claims from the generated answer. Each claim is a single, self-contained statement with no pronouns.
• Stage 2: Verify each claim against the source context using an NLI model or LLM-as-judge. The verifier answers: can this claim be directly inferred from the context?
• Stage 3: Compute a faithfulness score as the ratio of supported claims to total claims. A score of 0.8 or higher is the standard production threshold.

Why decompose into atomic claims?

A single sentence can contain multiple factual claims. Checking the whole sentence at once means a verifier might miss a single fabricated detail buried inside mostly-true information.

By splitting into atomic claims first, you catch partial hallucinations that simpler methods miss. This is why both RAGAS and DeepEval use this decomposition approach.

Your check_hallucination_node in rag_graph.py already implements all three stages. Let's isolate the claim extraction piece so you can see exactly what it produces.

• Create a scratch file called explore_hallucination.py in your multi-agent-rag folder. This file is for exploration only and won't be part of your final project.
• Add the following imports, test data, and standalone extract_claims function by pasting this code:

import json
from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage

# A sample context and answer pair to experiment with
test_context = (
    "Retrieval-Augmented Generation (RAG) is a technique that enhances "
    "large language models by retrieving relevant documents from an "
    "external knowledge base before generating a response. The process "
    "has two main stages: retrieval and generation. RAG reduces "
    "hallucinations by providing factual grounding, enables knowledge "
    "updates without model retraining, and allows attribution of answers "
    "to source documents."
)

test_answer = (
    "RAG is a two-stage technique involving retrieval and generation. "
    "It was invented by Facebook AI Research in 2020. RAG reduces "
    "hallucinations by grounding responses in retrieved documents."
)


def extract_claims(answer):
    """Extract atomic factual claims from an answer using llama3.2."""
    llm = ChatOllama(model="llama3.2", temperature=0.0)

    # Ask the LLM to decompose the answer into individual factual claims
    prompt = (
        "Extract all factual claims from the following answer as a JSON "
        "list of strings. Each claim should be a single, self-contained "
        "statement with no pronouns.\n\n"
        f"Answer: {answer}\n\n"
        "Respond with ONLY a JSON list of strings. "
        'Example: ["claim 1", "claim 2"]'
    )

    response = llm.invoke([HumanMessage(content=prompt)])

    # Parse the JSON from the response, with a fallback
    try:
        raw = response.content
        start = raw.find("[")
        end = raw.rfind("]") + 1
        if start != -1 and end > start:
            claims = json.loads(raw[start:end])
        else:
            claims = [answer]
    except (json.JSONDecodeError, TypeError):
        claims = [answer]

    return claims


# Run claim extraction and display results
print("=== Claim Extraction ===")
print(f"Answer: {test_answer}\n")
claims = extract_claims(test_answer)
for i, claim in enumerate(claims):
    print(f"  Claim {i + 1}: {claim}")

What does this code do?

• test_context holds a known passage from your knowledge base. test_answer contains a deliberately mixed response: two true claims and one fabricated claim ("invented by Facebook AI Research in 2020").
• extract_claims() sends the answer to llama3.2 with instructions to decompose it into atomic, pronoun-free statements. The JSON parsing includes a fallback in case the model's output is malformed.

• Save explore_hallucination.py.
• Run the scratch script by running this command:

python explore_hallucination.py

You should see the answer decomposed into individual claims. The fabricated claim about Facebook AI Research should appear as its own separate line.

Why are pronoun-free claims important?

If a claim says "It was invented in 2020," the verifier has no idea what "it" refers to. A self-contained claim like "RAG was invented by Facebook AI Research in 2020" gives the verifier everything it needs to check against the source context.

Seeing a JSON parsing error?

Small local models sometimes return malformed JSON. The fallback logic handles this by treating the entire answer as one claim. If you see only one claim containing the full answer, that means the JSON parsing fell through to the fallback.

Try running the script again. LLM outputs vary between runs, and the next attempt often produces valid JSON.

Help me debug my claim extraction output.

Compare bespoke-minicheck vs LLM-as-judge for hallucination detection

Your pipeline currently uses bespoke-minicheck to verify claims. This is a purpose-built classifier that gives a binary Yes/No verdict per claim. But there's another approach: using a general-purpose LLM as an NLI judge that provides richer reasoning alongside its verdict.

Each approach has real tradeoffs. Let's run both on the same claims so you can see where they agree and where they diverge.

• In explore_hallucination.py, find the last line that says print(f" Claim {i + 1}: {claim}").
• Add the side-by-side comparison functions below it by pasting this code:

def verify_with_minicheck(claim, context):
    """Verify a claim using bespoke-minicheck (binary classifier)."""
    minicheck = ChatOllama(model="bespoke-minicheck", temperature=0.0)
    # bespoke-minicheck expects Document/Claim format
    prompt = f"Document: {context}\nClaim: {claim}"
    result = minicheck.invoke([HumanMessage(content=prompt)])
    verdict = result.content.strip().lower().startswith("yes")
    return {"method": "minicheck", "supported": verdict, "raw": result.content.strip()}


def verify_with_llm_judge(claim, context):
    """Verify a claim using llama3.2 as an NLI judge (with reasoning)."""
    llm = ChatOllama(model="llama3.2", temperature=0.0)
    # The NLI prompt asks for both a verdict and reasoning
    prompt = (
        "You are a fact-checking judge. Determine whether the following "
        "claim can be directly inferred from the document.\n\n"
        f"Document: {context}\n\n"
        f"Claim: {claim}\n\n"
        "Respond with ONLY a JSON object: "
        '{"verdict": "supported" or "not_supported", '
        '"reasoning": "brief explanation"}'
    )
    result = llm.invoke([HumanMessage(content=prompt)])
    try:
        raw = result.content
        start = raw.find("{")
        end = raw.rfind("}") + 1
        if start != -1 and end > start:
            parsed = json.loads(raw[start:end])
            supported = parsed.get("verdict", "").lower() == "supported"
            reasoning = parsed.get("reasoning", "No reasoning provided")
        else:
            supported = False
            reasoning = "Could not parse response"
    except (json.JSONDecodeError, TypeError):
        supported = False
        reasoning = "Could not parse response"
    return {"method": "llm_judge", "supported": supported, "reasoning": reasoning, "raw": result.content.strip()}

What does this code do?

• verify_with_minicheck() sends each claim to bespoke-minicheck using the Document: ... Claim: ... format. The model returns a binary Yes or No with no reasoning.
• verify_with_llm_judge() sends each claim to llama3.2 with an NLI prompt that asks for both a verdict and a brief explanation. This gives richer output but is slower and can itself hallucinate.

• Now add the comparison runner at the bottom of the same file by pasting this code:

# Run both verification methods side by side on each claim
print("\n=== Side-by-Side Comparison ===")
for i, claim in enumerate(claims):
    print(f"\nClaim {i + 1}: {claim}")

    mc_result = verify_with_minicheck(claim, test_context)
    print(f"  Minicheck: {'SUPPORTED' if mc_result['supported'] else 'NOT SUPPORTED'}")
    print(f"    Raw: {mc_result['raw']}")

    judge_result = verify_with_llm_judge(claim, test_context)
    print(f"  LLM Judge: {'SUPPORTED' if judge_result['supported'] else 'NOT SUPPORTED'}")
    print(f"    Reasoning: {judge_result['reasoning']}")

    if mc_result["supported"] != judge_result["supported"]:
        print("  >>> DISAGREEMENT DETECTED <<<")

What does this code do?

For each extracted claim, this code runs both verification methods and prints their results side by side. When the two methods disagree on a claim, it flags that disagreement. These disagreements reveal the strengths and weaknesses of each approach.

• Save explore_hallucination.py and run it again:

python explore_hallucination.py

You should see each claim with verdicts from both methods. The fabricated claim about Facebook AI Research should be flagged as NOT SUPPORTED by both. The LLM judge should also provide reasoning explaining why.

Which approach is better for production?

bespoke-minicheck is faster and more consistent because it is a purpose-built classifier. It gives you a reliable binary signal. The LLM judge provides richer reasoning that helps you understand why a claim was flagged. But the LLM judge can itself hallucinate its reasoning.

Production systems often use both for defense in depth. The specialized classifier runs on every response for speed. The LLM judge runs on flagged responses for explainability.

Both methods agreeing on everything?

If you don't see any disagreements, that's actually a good sign for this test case. The fabricated claim is fairly obvious. Disagreements are more common on subtle or paraphrased claims. You'll see more divergence with the adversarial examples in the next substep.

Help me understand my side-by-side comparison results.

Test with adversarial examples

Knowing the system works on straightforward cases is useful. Knowing where it breaks is essential. Adversarial testing reveals the blind spots in your hallucination detection.

You'll craft three types of tricky inputs and run them through your full pipeline to observe how each one is handled.

• Create a new file called test_adversarial.py in your multi-agent-rag folder. This is another scratch file for exploration.
• Add the adversarial test cases by pasting this code:

from rag_graph import run_query

# Three types of adversarial questions designed to trigger hallucinations
adversarial_questions = [
    {
        # Type 1: Requires info NOT in the knowledge base
        "type": "Out-of-scope",
        "question": "What is the learning rate schedule used to train GPT-4?",
        "why": "The knowledge base has no information about GPT-4 training.",
    },
    {
        # Type 2: Context has similar but different facts
        "type": "Confusable facts",
        "question": "Did the Transformer achieve state-of-the-art on Chinese-to-English translation?",
        "why": "The context says English-to-German and English-to-French, not Chinese-to-English.",
    },
    {
        # Type 3: Likely to confuse related concepts
        "type": "Concept confusion",
        "question": "Are intrinsic hallucinations the same as extrinsic hallucinations?",
        "why": "These are distinct types that a model might conflate.",
    },
]

# Run each adversarial question through the full pipeline
for test in adversarial_questions:
    print(f"\n{'=' * 60}")
    print(f"Type: {test['type']}")
    print(f"Question: {test['question']}")
    print(f"Why this is tricky: {test['why']}")
    print("=" * 60)

    result = run_query(test["question"], max_retries=2)

    # Show the agent execution log
    print("\nAgent Log:")
    for entry in result["agent_log"]:
        print(f"  {entry}")

    # Show the answer
    print(f"\nAnswer: {result['answer'][:300]}")
    print(f"\nFaithfulness: {result['faithfulness_score']:.2f}")

    # Show which claims passed and which failed
    print("\nClaim Checks:")
    for check in result["hallucination_checks"]:
        status = "SUPPORTED" if check["supported"] else "NOT SUPPORTED"
        print(f"  [{status}] {check['claim']}")

    # Flag if self-healing triggered
    if result["retry_count"] > 1:
        print(f"\n  Self-healing triggered: {result['retry_count'] - 1} retries")
    else:
        print("\n  No self-healing needed.")

What does this code do?

• The out-of-scope question asks about GPT-4 training, which is completely absent from your knowledge base. The model should either say it doesn't know or hallucinate details from its training data.
• The confusable facts question references Chinese-to-English translation. The knowledge base says English-to-German and English-to-French. The model might swap one for the other because the concepts are so similar.
• The concept confusion question asks whether two distinct types of hallucination are the same. The model might blur the definitions or add fabricated connections.

• Save test_adversarial.py and run it by running this command:

python test_adversarial.py

This will take a few minutes since it runs three full pipeline cycles. Watch the agent logs closely to see whether the self-healing loop triggers for any of the adversarial cases.

What patterns should I look for?

• Out-of-scope questions often trigger hallucinations because the model fills in gaps with its training data. If the faithfulness score drops below 0.8, you should see the self-healing loop kick in.
• Confusable facts are the hardest to catch. The model might say "Chinese-to-English" and the checker might still mark it as supported because the surrounding context about translation benchmarks is very similar.
• Concept confusion tests whether the checker can distinguish between claims that merge two distinct definitions. bespoke-minicheck may give false positives here if the claim is semantically close to the source but factually wrong.

Pipeline running very slowly?

Each adversarial question runs the full retrieve-generate-check cycle, and some may trigger retries. On a laptop, each question can take 30 to 90 seconds depending on your hardware. Three questions with retries can take up to 5 minutes total.

If it seems stuck, check that Ollama is still running by opening a new terminal tab and running ollama list. You should see your three models listed.

My adversarial test pipeline is very slow or seems stuck.

What can't this checker detect?

No hallucination detection system is perfect. bespoke-minicheck can give false positives on paraphrased claims that are semantically similar to the source but factually different. It can also miss extrinsic hallucinations when the added information sounds plausible in context.

This is why production teams combine multiple detection methods. Your pipeline uses both a specialized classifier and the self-healing loop as a second line of defense.

You now understand exactly how the three-stage detection pipeline works, where the two verification approaches diverge, and which hallucination patterns are hardest to catch. Next up, you'll build an automated evaluation framework that scores every response on correctness, helpfulness, and faithfulness.

Build the Auto-Judge Evaluation Framework

Your RAG pipeline can now detect hallucinations and self-heal. But how do you know the overall quality of every answer it produces? In production, teams run automated evaluation suites that score every response before it reaches a user.

In this step, you will build an LLM-as-judge evaluation framework. This framework uses llama3.2 itself to score RAG outputs on three criteria: correctness, helpfulness, and faithfulness. You will also prepare a calibration function so you can compare the judge's ratings against your own human ratings in the next step.

In this step, get ready to:

• Design three evaluation rubrics with explicit 1-5 scoring guides.
• Implement an LLM-as-judge pipeline that scores responses with structured JSON output.
• Run a batch evaluation suite across five test questions and review per-criterion scores.

Design evaluation rubrics for three criteria

Research shows that clear evaluation criteria are the single most important factor for judge reliability. They matter more than chain-of-thought prompting or even the choice of model.

You will define a test dataset of five questions with reference answers, plus three scoring rubrics. Each rubric describes what a 1-5 score means for that criterion.

• Create a new file called evaluator.py inside your multi-agent-rag folder.
• Add the imports and test question dataset by pasting the following code:

import json
from typing import List

from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage

from rag_graph import run_query

# Five test questions with reference answers for evaluation
TEST_QUESTIONS = [
    {"question": "What is the Transformer architecture and who introduced it?", "reference": "The Transformer architecture was introduced in 'Attention Is All You Need' by Vaswani et al. in 2017. It relies on self-attention mechanisms and consists of an encoder and decoder."},
    {"question": "How does RAG reduce hallucinations?", "reference": "RAG reduces hallucinations by retrieving relevant documents from an external knowledge base and using them as factual grounding for the LLM to generate answers."},
    {"question": "What are the two types of LLM hallucinations?", "reference": "Intrinsic hallucinations contradict the provided source context. Extrinsic hallucinations add information not present in any provided context."},
    {"question": "What faithfulness threshold is used in production?", "reference": "A faithfulness threshold of 0.8 or higher is commonly used in production systems."},
    {"question": "What vector databases are commonly used with RAG?", "reference": "Common vector databases include ChromaDB, Pinecone, Weaviate, and Qdrant."},
]

What does this code do?

The imports bring in ChatOllama for the judge model, HumanMessage for prompting it, and run_query from your RAG pipeline so the evaluator can run questions end-to-end.

TEST_QUESTIONS is your evaluation benchmark. Each entry has a question the RAG pipeline will answer and a reference answer you know is correct. The judge compares the pipeline's output against these references.

Now you need the rubrics the judge will use to score each response.

• Add the evaluation criteria dictionary below TEST_QUESTIONS by pasting this code:

# Three scoring rubrics: what each criterion measures
EVALUATION_CRITERIA = {
    "correctness": "Is the answer factually correct based on the provided context? Does it accurately represent the information in the source documents without distorting or misrepresenting facts?",
    "helpfulness": "Is the answer helpful and complete? Does it directly address the question asked and provide sufficient detail to be useful to the person asking?",
    "faithfulness": "Is the answer grounded in the provided context? Does it avoid adding information, opinions, or claims not present in the retrieved documents?",
}

Why these three criteria?

Each criterion catches a different failure mode. Correctness catches factual errors. Helpfulness catches answers that are technically correct but miss the point or lack detail. Faithfulness catches hallucinations where the model invents information beyond the retrieved context.

Notice how each rubric is written as a question with specific, observable conditions. Vague criteria like "is it good?" would give inconsistent scores. The more explicit the rubric, the more reliable the judge becomes.

• Save evaluator.py.

Seeing a red underline on the imports?

Make sure your virtual environment is activated. Your editor needs the environment where you installed the packages. Also confirm that rag_graph.py is in the same folder as evaluator.py.

Why can't my editor find the imports in evaluator.py?

Implement the LLM-as-judge scoring pipeline

The judge needs a structured prompt that includes the rubric, a scoring guide, and instructions to return JSON. The function also needs robust parsing because llama3.2 does not always produce perfectly formatted output.

• In evaluator.py, add the judge_response and evaluate_response functions below EVALUATION_CRITERIA by pasting this code:

def judge_response(question, answer, context, criteria):
    """Use an LLM to judge a response on a specific criterion."""
    llm = ChatOllama(model="llama3.2", temperature=0.0)

    # Build the judge prompt with the rubric and scoring guide
    prompt = (
        "You are an expert evaluator. Score the following AI response on a scale of 1 to 5.\n\n"
        f"Criterion: {criteria}\n\n"
        "Scoring Guide:\n1 - Completely fails the criterion\n2 - Mostly fails with minor correct elements\n3 - Partially meets the criterion\n4 - Mostly meets the criterion with minor issues\n5 - Fully meets the criterion\n\n"
        f"Question: {question}\n\nRetrieved Context: {context}\n\nAI Response: {answer}\n\n"
        'Respond with ONLY a JSON object in this exact format:\n{"score": <1-5>, "reasoning": "<brief explanation>"}'
    )

    response = llm.invoke([HumanMessage(content=prompt)])

    # Parse the JSON from the response, handling messy output
    try:
        raw = response.content
        start = raw.find("{")
        end = raw.rfind("}") + 1
        if start != -1 and end > start:
            result = json.loads(raw[start:end])
            if "score" in result and "reasoning" in result:
                # Clamp score to valid 1-5 range
                result["score"] = max(1, min(5, int(result["score"])))
                return result
    except (json.JSONDecodeError, TypeError, ValueError):
        pass

    # Fallback if parsing fails
    return {"score": 3, "reasoning": "Could not parse judge response"}


def evaluate_response(question, answer, context):
    """Evaluate a response on all criteria."""
    scores = {}
    for name, description in EVALUATION_CRITERIA.items():
        scores[name] = judge_response(question, answer, context, description)
    return scores

What does this code do?

• judge_response sends the question, answer, context, and one rubric to llama3.2 at temperature=0.0 for reproducible scores. It parses the JSON response and clamps the score to the 1-5 range.
• The JSON parsing uses find("{") and rfind("}") to extract the JSON object even when the model wraps it in extra text. This is the same robust parsing pattern you used in the hallucination checker.
• evaluate_response is a thin wrapper that loops through all three criteria and returns a complete evaluation dictionary.

• Save evaluator.py.

Why temperature 0.0 for the judge?

Setting temperature to 0.0 makes the judge deterministic. Running the same evaluation twice should produce the same scores. Research shows that non-deterministic sampling can actually improve alignment with human preferences, but for a learning project, reproducibility makes it easier to debug and compare results.

Worried about the fallback score of 3?

The fallback returns a neutral score of 3 when parsing fails. This prevents the evaluation from crashing on malformed output. You will see the reasoning text "Could not parse judge response" in the results, which flags that the judge did not produce valid JSON for that evaluation.

Why is my LLM judge returning fallback scores instead of real evaluations?

Run batch evaluation on the test questions

Individual scoring is useful, but production teams run evaluation suites across entire test datasets. This catches quality regressions before deployment.

You will add two more functions: one that runs the full batch evaluation, and one that compares LLM judge scores against human ratings. The calibration function prepares you for the next step where you will rate outputs yourself.

• Add the run_evaluation_suite function below evaluate_response by pasting this code:

def run_evaluation_suite(questions=None):
    """Run the full evaluation suite on a set of test questions."""
    if questions is None:
        questions = TEST_QUESTIONS

    results = []
    for i, q in enumerate(questions):
        print(f"\nEvaluating {i + 1}/{len(questions)}: {q['question'][:50]}...")

        # Run the question through the full RAG pipeline
        rag_result = run_query(q["question"])

        # Score the RAG output on all three criteria
        eval_scores = evaluate_response(q["question"], rag_result["answer"], rag_result["context"])

        # Bundle the RAG result and evaluation scores together
        results.append({"question": q["question"], "reference": q.get("reference", ""), "answer": rag_result["answer"], "context": rag_result["context"], "faithfulness_score": rag_result["faithfulness_score"], "evaluation": eval_scores, "agent_log": rag_result["agent_log"]})

    return results

What does this code do?

run_evaluation_suite is the orchestrator. For each test question, it runs the full RAG pipeline (retrieve, generate, hallucination check), then passes the answer through the LLM judge on all three criteria. It collects everything into a results list.

The function accepts an optional questions parameter. When it's None, it defaults to the full TEST_QUESTIONS list. This lets you run a smaller subset for quick testing.

• Save evaluator.py.

Next, add the calibration function that will compare LLM judge scores against human ratings.

• Add the compute_calibration function below run_evaluation_suite by pasting this code:

def compute_calibration(eval_results, human_ratings):
    """Compare LLM judge scores against human ratings."""
    criteria_names = ["correctness", "helpfulness", "faithfulness"]
    agreement = {c: [] for c in criteria_names}
    disagreements = []

    for eval_r, human_r in zip(eval_results, human_ratings):
        for criterion in criteria_names:
            llm_score = eval_r["evaluation"][criterion]["score"]
            human_score = human_r.get(criterion, 3)

            # Track score difference per criterion
            diff = abs(llm_score - human_score)
            agreement[criterion].append(diff)

            # Flag major disagreements where scores differ by 2+ points
            if diff >= 2:
                disagreements.append({"question": eval_r["question"], "criterion": criterion, "llm_score": llm_score, "human_score": human_score, "llm_reasoning": eval_r["evaluation"][criterion]["reasoning"], "answer": eval_r["answer"][:200]})

    # Compute agreement metrics per criterion
    summary = {}
    for criterion, diffs in agreement.items():
        if not diffs:
            continue
        avg_diff = sum(diffs) / len(diffs)
        exact_match = sum(1 for d in diffs if d == 0) / len(diffs)
        within_one = sum(1 for d in diffs if d <= 1) / len(diffs)
        summary[criterion] = {"avg_difference": round(avg_diff, 2), "exact_match_rate": round(exact_match, 2), "within_one_rate": round(within_one, 2)}

    return {"summary": summary, "disagreements": disagreements}

What does this code do?

• compute_calibration takes two lists: the LLM judge's evaluation results and your human ratings. It compares scores for each criterion on each question.
• It computes three agreement metrics: average score difference, exact match rate (how often you and the judge gave the same score), and within-one-point rate (how often scores are close).
• Any question where the LLM and human scores differ by 2 or more points is flagged as a "major disagreement." These disagreements reveal the judge's blind spots and become the basis for improvement.

• Save evaluator.py.

Finally, add the main block that runs the full evaluation suite when you execute the file.

• Add the main block at the bottom of evaluator.py by pasting this code:

if __name__ == "__main__":
    print("Running evaluation suite...")
    results = run_evaluation_suite()

    print("\n=== Evaluation Results ===")
    for r in results:
        print(f"\nQ: {r['question']}")
        print(f"Faithfulness (minicheck): {r['faithfulness_score']:.2f}")
        for criterion, score_data in r["evaluation"].items():
            print(f"  {criterion}: {score_data['score']}/5 - {score_data['reasoning']}")

What does this code do?

The main block runs all five test questions through the RAG pipeline and then through the LLM judge. For each question, it prints the faithfulness score from bespoke-minicheck alongside the three judge scores. This gives you two independent quality signals to compare.

• Save evaluator.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved evaluator.py before running it.

ⓧ I'd like to double check the full code

import json
from typing import List

from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage

from rag_graph import run_query

TEST_QUESTIONS = [
    {"question": "What is the Transformer architecture and who introduced it?", "reference": "The Transformer architecture was introduced in 'Attention Is All You Need' by Vaswani et al. in 2017. It relies on self-attention mechanisms and consists of an encoder and decoder."},
    {"question": "How does RAG reduce hallucinations?", "reference": "RAG reduces hallucinations by retrieving relevant documents from an external knowledge base and using them as factual grounding for the LLM to generate answers."},
    {"question": "What are the two types of LLM hallucinations?", "reference": "Intrinsic hallucinations contradict the provided source context. Extrinsic hallucinations add information not present in any provided context."},
    {"question": "What faithfulness threshold is used in production?", "reference": "A faithfulness threshold of 0.8 or higher is commonly used in production systems."},
    {"question": "What vector databases are commonly used with RAG?", "reference": "Common vector databases include ChromaDB, Pinecone, Weaviate, and Qdrant."},
]

EVALUATION_CRITERIA = {
    "correctness": "Is the answer factually correct based on the provided context? Does it accurately represent the information in the source documents without distorting or misrepresenting facts?",
    "helpfulness": "Is the answer helpful and complete? Does it directly address the question asked and provide sufficient detail to be useful to the person asking?",
    "faithfulness": "Is the answer grounded in the provided context? Does it avoid adding information, opinions, or claims not present in the retrieved documents?",
}


def judge_response(question, answer, context, criteria):
    """Use an LLM to judge a response on a specific criterion."""
    llm = ChatOllama(model="llama3.2", temperature=0.0)

    prompt = (
        "You are an expert evaluator. Score the following AI response on a scale of 1 to 5.\n\n"
        f"Criterion: {criteria}\n\n"
        "Scoring Guide:\n1 - Completely fails the criterion\n2 - Mostly fails with minor correct elements\n3 - Partially meets the criterion\n4 - Mostly meets the criterion with minor issues\n5 - Fully meets the criterion\n\n"
        f"Question: {question}\n\nRetrieved Context: {context}\n\nAI Response: {answer}\n\n"
        'Respond with ONLY a JSON object in this exact format:\n{"score": <1-5>, "reasoning": "<brief explanation>"}'
    )

    response = llm.invoke([HumanMessage(content=prompt)])

    try:
        raw = response.content
        start = raw.find("{")
        end = raw.rfind("}") + 1
        if start != -1 and end > start:
            result = json.loads(raw[start:end])
            if "score" in result and "reasoning" in result:
                result["score"] = max(1, min(5, int(result["score"])))
                return result
    except (json.JSONDecodeError, TypeError, ValueError):
        pass

    return {"score": 3, "reasoning": "Could not parse judge response"}


def evaluate_response(question, answer, context):
    """Evaluate a response on all criteria."""
    scores = {}
    for name, description in EVALUATION_CRITERIA.items():
        scores[name] = judge_response(question, answer, context, description)
    return scores


def run_evaluation_suite(questions=None):
    """Run the full evaluation suite on a set of test questions."""
    if questions is None:
        questions = TEST_QUESTIONS

    results = []
    for i, q in enumerate(questions):
        print(f"\nEvaluating {i + 1}/{len(questions)}: {q['question'][:50]}...")

        rag_result = run_query(q["question"])

        eval_scores = evaluate_response(q["question"], rag_result["answer"], rag_result["context"])

        results.append({"question": q["question"], "reference": q.get("reference", ""), "answer": rag_result["answer"], "context": rag_result["context"], "faithfulness_score": rag_result["faithfulness_score"], "evaluation": eval_scores, "agent_log": rag_result["agent_log"]})

    return results


def compute_calibration(eval_results, human_ratings):
    """Compare LLM judge scores against human ratings."""
    criteria_names = ["correctness", "helpfulness", "faithfulness"]
    agreement = {c: [] for c in criteria_names}
    disagreements = []

    for eval_r, human_r in zip(eval_results, human_ratings):
        for criterion in criteria_names:
            llm_score = eval_r["evaluation"][criterion]["score"]
            human_score = human_r.get(criterion, 3)

            diff = abs(llm_score - human_score)
            agreement[criterion].append(diff)

            if diff >= 2:
                disagreements.append({"question": eval_r["question"], "criterion": criterion, "llm_score": llm_score, "human_score": human_score, "llm_reasoning": eval_r["evaluation"][criterion]["reasoning"], "answer": eval_r["answer"][:200]})

    summary = {}
    for criterion, diffs in agreement.items():
        if not diffs:
            continue
        avg_diff = sum(diffs) / len(diffs)
        exact_match = sum(1 for d in diffs if d == 0) / len(diffs)
        within_one = sum(1 for d in diffs if d <= 1) / len(diffs)
        summary[criterion] = {"avg_difference": round(avg_diff, 2), "exact_match_rate": round(exact_match, 2), "within_one_rate": round(within_one, 2)}

    return {"summary": summary, "disagreements": disagreements}


if __name__ == "__main__":
    print("Running evaluation suite...")
    results = run_evaluation_suite()

    print("\n=== Evaluation Results ===")
    for r in results:
        print(f"\nQ: {r['question']}")
        print(f"Faithfulness (minicheck): {r['faithfulness_score']:.2f}")
        for criterion, score_data in r["evaluation"].items():
            print(f"  {criterion}: {score_data['score']}/5 - {score_data['reasoning']}")

Now run the evaluation suite. This will take several minutes because it runs all five questions through the full RAG pipeline and then through three separate judge calls each.

• Make sure Ollama is running in the background.
• Run the evaluation suite by executing this command in your terminal:

python evaluator.py

This will take a few minutes

Each of the 5 questions goes through the full RAG pipeline (retrieval, generation, hallucination checking) plus 3 separate judge evaluations. That is at least 40 LLM calls running locally. Expect this to take 3-8 minutes depending on your machine.

• You should see progress output like Evaluating 1/5: What is the Transformer architecture and who... followed by the full evaluation results showing each question with its faithfulness score and three judge scores.

Seeing connection errors or timeouts?

Confirm that Ollama is running by opening a new terminal and running ollama list. You should see llama3.2 and bespoke-minicheck in the list. If not, start Ollama from your menu bar or by running ollama serve.

If the script is taking a very long time on a single question, your machine may be running low on memory. Close other applications and try again.

My evaluator.py is failing with connection errors when calling Ollama models.

What should I see?

For each question, you should see a faithfulness score from bespoke-minicheck (a number between 0.0 and 1.0) plus three judge scores from 1-5 with reasoning. The scores will vary each time, but well-grounded answers typically score 4 or 5 on all three criteria.

Your evaluation framework is now producing automated quality scores for every RAG response. Next up, you will calibrate this judge by rating outputs yourself and measuring how well the LLM judge aligns with your human judgment.

Calibrate the Judge Against Human Ratings

In the previous step, you built an automated evaluation framework that scores every RAG response on correctness, helpfulness, and faithfulness.

Automated scores are only useful if they match what a human would rate. In this step, you will rate a batch of outputs yourself, measure where the LLM judge agrees and disagrees with you, and improve the judge using few-shot corrections from those disagreements.

In this step, get ready to:

• Rate RAG pipeline outputs using the same 1-5 evaluation rubrics.
• Compute agreement metrics between your ratings and the LLM judge.
• Improve the judge with few-shot corrections from major disagreements.

Rate a batch of outputs yourself

An LLM judge is only as trustworthy as its agreement with human judgment. To measure that agreement, you need a "gold standard" dataset of your own ratings.

You already ran the evaluation suite in the previous step. Now you will review those results, rate each output yourself, and compare.

• Run the evaluation suite again to see the full output by running this command:

python3 evaluator.py

What should I see?

The script runs all 5 test questions through the RAG pipeline and the LLM judge. For each question you will see the faithfulness score from bespoke-minicheck and the judge's 1-5 scores for correctness, helpfulness, and faithfulness.

This takes several minutes because each question passes through retrieval, generation, hallucination checking, and three evaluation calls. Let it finish completely before moving on.

• Review the printed results in your terminal.
• For each of the 5 questions, note the answer and the judge's scores.

Now rate each answer yourself. Use the same 1-5 scale and the same three criteria: correctness (is it factually accurate?), helpfulness (does it address the question with enough detail?), and faithfulness (does it avoid adding info not in the context?).

Be strict and honest

The whole point of calibration is to find where the judge is wrong. If you inflate your ratings to match the judge, the calibration is meaningless. Rate what you genuinely think each answer deserves.

Write your ratings as a JSON list with one object per question. Each object needs three keys: correctness, helpfulness, and faithfulness. The order must match the 5 test questions.

✔️ I'll give it a go!

Write your 5 ratings in a JSON list. Here is the format you need:

[
  {"correctness": <1-5>, "helpfulness": <1-5>, "faithfulness": <1-5>},
  {"correctness": <1-5>, "helpfulness": <1-5>, "faithfulness": <1-5>},
  {"correctness": <1-5>, "helpfulness": <1-5>, "faithfulness": <1-5>},
  {"correctness": <1-5>, "helpfulness": <1-5>, "faithfulness": <1-5>},
  {"correctness": <1-5>, "helpfulness": <1-5>, "faithfulness": <1-5>}
]

Save your ratings somewhere handy. You will paste them into a Python session in the next substep.

ⓧ Maybe a little help...

Here is an example set of human ratings. These are intentionally generous on some questions and strict on others so that you get interesting disagreements with the LLM judge.

[
  {"correctness": 4, "helpfulness": 4, "faithfulness": 5},
  {"correctness": 5, "helpfulness": 4, "faithfulness": 4},
  {"correctness": 4, "helpfulness": 3, "faithfulness": 5},
  {"correctness": 5, "helpfulness": 3, "faithfulness": 5},
  {"correctness": 4, "helpfulness": 4, "faithfulness": 4}
]

You can use these as a starting point. Adjust any scores based on what you actually saw in the evaluation output.

Compute agreement metrics and identify disagreements

Your evaluator.py already has a compute_calibration() function that compares LLM judge scores against human scores across all three criteria. It calculates three metrics per criterion: average score difference, exact match rate, and within-one-point rate.

It also flags "major disagreements" where the LLM and human scores differ by 2 or more points. These disagreements reveal the judge's blind spots.

• Open an interactive Python session by running:

python3

• In the Python session, run the evaluation suite and compute calibration against your human ratings by typing these commands:

import json
from evaluator import run_evaluation_suite, compute_calibration

# Run the evaluation suite on all 5 test questions
results = run_evaluation_suite()

# Paste your human ratings here (one dict per question)
human_ratings = [
    {"correctness": 4, "helpfulness": 4, "faithfulness": 5},
    {"correctness": 5, "helpfulness": 4, "faithfulness": 4},
    {"correctness": 4, "helpfulness": 3, "faithfulness": 5},
    {"correctness": 5, "helpfulness": 3, "faithfulness": 5},
    {"correctness": 4, "helpfulness": 4, "faithfulness": 4},
]

# Compare your ratings against the LLM judge
calibration = compute_calibration(results, human_ratings)

# Print the agreement summary
print("\n=== Calibration Summary ===")
for criterion, stats in calibration["summary"].items():
    print(f"\n{criterion.title()}:")
    print(f"  Avg difference: {stats['avg_difference']}")
    print(f"  Exact match rate: {stats['exact_match_rate']:.0%}")
    print(f"  Within 1 point: {stats['within_one_rate']:.0%}")

# Print major disagreements
if calibration["disagreements"]:
    print("\n=== Major Disagreements (diff >= 2) ===")
    for d in calibration["disagreements"]:
        print(f"\nQ: {d['question'][:60]}...")
        print(f"  Criterion: {d['criterion']}")
        print(f"  LLM score: {d['llm_score']}, Human score: {d['human_score']}")
        print(f"  LLM reasoning: {d['llm_reasoning']}")
else:
    print("\nNo major disagreements found.")

What does this code do?

This re-runs the full evaluation suite so the results are available as Python objects. Then it calls compute_calibration() to compare the LLM judge scores against your human ratings for every question and criterion.

The summary shows three metrics per criterion. Average difference tells you how far off the judge is on average. Exact match rate shows how often it agrees exactly. Within-one-point rate shows how often it is close enough to be useful.

• Replace the human_ratings list with your own ratings if you created different ones in the previous substep.
• Wait for the evaluation suite to finish running. This takes several minutes.

You should see a calibration summary printed for each criterion, followed by any major disagreements where scores differed by 2 or more points.

How do I interpret these results?

A within-one-point rate above 80% means the judge is reasonably well calibrated. Below 60% means the judge needs significant correction.

Major disagreements are the most valuable output. Each one tells you exactly where the judge's reasoning diverges from yours. These become the training data for improving the judge.

Seeing no disagreements at all?

If you have zero major disagreements, try being more critical in your ratings. Small local models like llama3.2 tend to give generous scores. If you rate a mediocre answer as a 2 on helpfulness but the judge gave it a 4, that becomes a useful disagreement.

Having trouble interpreting your calibration results?

Improve the judge with few-shot examples

The calibration loop used in production systems follows a specific pattern: collect human corrections, turn them into few-shot examples, inject those examples into the judge prompt, and measure whether agreement improves.

Each major disagreement from the previous substep becomes a concrete "here is what the correct score should be" example for the judge.

What is few-shot correction?

Few-shot correction means adding worked examples directly into the judge's prompt. Instead of just giving the judge a rubric, you also show it concrete cases: "For this question and answer, the correct score is 2 because the answer adds information not in the context."

This is how LangSmith's Align Evals feature works at scale. Teams collect human corrections over time, build a growing set of few-shot examples, and track agreement metrics as the judge improves.

• In your Python session, inspect the disagreements list to find a concrete example by running:

# Build few-shot examples from disagreements
few_shot_examples = ""
for d in calibration["disagreements"]:
    few_shot_examples += (
        f"\nExample - Question: {d['question'][:80]}\n"
        f"Answer excerpt: {d['answer'][:100]}\n"
        f"Criterion: {d['criterion']}\n"
        f"Correct score: {d['human_score']} "
        f"(the model gave {d['llm_score']} which was too "
        f"{'high' if d['llm_score'] > d['human_score'] else 'low'})\n"
    )

print(few_shot_examples if few_shot_examples else "No disagreements to use as examples.")

What does this code do?

This loops through the major disagreements and formats each one as a few-shot example. Each example includes the question, an excerpt of the answer, which criterion was misjudged, and what the correct score should have been.

Now test whether adding these examples to the judge prompt improves its accuracy. You will re-run the judge on the same questions with the few-shot examples injected into the prompt.

• Run the improved judge and compare calibration by typing these commands in your Python session:

from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage
from evaluator import EVALUATION_CRITERIA
import json

# Create an improved judge that includes few-shot correction examples
def improved_judge(question, answer, context, criteria):
    llm = ChatOllama(model="llama3.2", temperature=0.0)
    prompt = (
        "You are an expert evaluator. Score the following AI response "
        "on a scale of 1 to 5.\n\n"
        f"Criterion: {criteria}\n\n"
        "Scoring Guide:\n"
        "1 - Completely fails the criterion\n"
        "2 - Mostly fails with minor correct elements\n"
        "3 - Partially meets the criterion\n"
        "4 - Mostly meets the criterion with minor issues\n"
        "5 - Fully meets the criterion\n\n"
        "Here are examples of correct scoring:\n"
        f"{few_shot_examples}\n\n"
        f"Question: {question}\n\n"
        f"Retrieved Context: {context}\n\n"
        f"AI Response: {answer}\n\n"
        'Respond with ONLY a JSON object: '
        '{"score": <1-5>, "reasoning": "<brief explanation>"}'
    )
    response = llm.invoke([HumanMessage(content=prompt)])
    try:
        raw = response.content
        start = raw.find("{")
        end = raw.rfind("}") + 1
        if start != -1 and end > start:
            result = json.loads(raw[start:end])
            if "score" in result and "reasoning" in result:
                result["score"] = max(1, min(5, int(result["score"])))
                return result
    except (json.JSONDecodeError, TypeError, ValueError):
        pass
    return {"score": 3, "reasoning": "Could not parse response"}

# Re-evaluate a subset with the improved judge
for r in results[:3]:
    print(f"\nQ: {r['question'][:60]}...")
    for crit_name, crit_desc in EVALUATION_CRITERIA.items():
        original = r["evaluation"][crit_name]["score"]
        improved = improved_judge(
            r["question"], r["answer"], r["context"], crit_desc
        )
        print(
            f"  {crit_name}: original={original}, "
            f"improved={improved['score']}"
        )

What does this code do?

The improved_judge function uses the same prompt structure as the original judge_response() but injects the few-shot correction examples between the scoring guide and the actual question. This gives the judge concrete calibration references before it scores.

The loop re-evaluates the first 3 questions and prints the original score alongside the improved score for each criterion. You can see whether the few-shot examples pushed the judge closer to your human ratings.

• Wait for the improved evaluation to finish. Compare the original and improved scores for each criterion.
• Exit the Python session by typing exit().

You should see at least some of the improved scores shift closer to your human ratings on the criteria where disagreements existed.

Why not save this improvement permanently?

In a production system, few-shot examples are stored in a separate calibration dataset and injected into the prompt dynamically. You would not hardcode them into the function because the examples grow over time as more human corrections are collected.

The pattern you just ran is the core calibration loop: evaluate, collect human corrections, build few-shot examples, re-evaluate, track improvement. At scale, this runs continuously as part of an evaluation CI/CD pipeline.

Improved scores look the same as the originals?

Small models like llama3.2 may not always respond to few-shot examples as dramatically as larger models. If the scores barely changed, that is still a valid observation. In production, this same technique with GPT-4 or Claude shows much stronger improvement.

Also check that the few_shot_examples variable is not empty. If there were no major disagreements, the few-shot string will be blank and the judge prompt is unchanged.

Need help understanding why your improved scores did or did not change?

You now have the complete human-AI calibration workflow running locally. Next up, you will bring everything together in a Gradio web interface with a chat tab and an evaluation dashboard.

Build the Gradio Interface

You now have a complete multi-agent RAG pipeline, a hallucination detector, an auto-judge evaluation framework, and a human calibration workflow. All of these pieces run from the terminal, which makes testing awkward and results hard to visualize.

In this step, you will build a Gradio web interface that ties everything together into two tabs. A Chat tab lets you ask questions and watch the agent execution flow in real time. An Evaluation tab lets you run batch evaluations and calibrate the judge against your own ratings.

In this step, get ready to:

• Create a chat tab that displays answers, agent logs, retrieved documents, and hallucination check results.
• Add an evaluation and calibration tab for batch scoring and human-AI comparison.
• Launch and test the complete system end-to-end through the browser.

Create the chat tab with agent execution visualization

The chat tab is the primary interface for your RAG system. When a user submits a question, it needs to run the full pipeline and display four pieces of information: the generated answer, the agent execution log, the retrieved document chunks, and the hallucination check results.

• Create a new file called app.py in your multi-agent-rag folder.
• Add the imports and the knowledge base safety check by copying this code into app.py:

import json
import os

import gradio as gr

# Import functions from our existing modules
from knowledge_base import create_knowledge_base, PERSIST_DIR
from rag_graph import run_query
from evaluator import (
    evaluate_response,
    run_evaluation_suite,
    compute_calibration,
    TEST_QUESTIONS,
)


def ensure_knowledge_base():
    # Auto-create the vector database if it doesn't exist yet
    if not os.path.exists(PERSIST_DIR):
        print("Creating knowledge base...")
        create_knowledge_base()

What does this code do?

The imports pull in everything the UI needs from your three existing modules. ensure_knowledge_base() checks whether the ChromaDB directory already exists before trying to create it. This prevents the app from re-embedding documents every time you restart the server.

• Save app.py.

You won't see a change yet. The next chunk adds the function that powers the chat tab.

• Below ensure_knowledge_base() in app.py, add the chat_with_rag() function by copying this code:

def chat_with_rag(question):
    # Guard against empty input
    if not question.strip():
        return "Please enter a question.", "", "", ""

    # Run the full multi-agent pipeline
    result = run_query(question)

    # Format the agent execution log as a newline-separated string
    agent_log = "\n".join(result["agent_log"])

    # Build a Markdown display for each retrieved chunk
    docs_display = ""
    for i, doc in enumerate(result["retrieved_docs"]):
        title = doc["metadata"].get("title", "Unknown")
        distance = doc.get("distance", 0)
        similarity = 1 - distance if distance else 0
        docs_display += f"**Chunk {i + 1}** [{title}] (similarity: {similarity:.2f})\n\n{doc['content'][:300]}...\n\n---\n\n"

    # Build a Markdown display for the hallucination verdict on each claim
    checks_display = f"**Faithfulness Score: {result['faithfulness_score']:.2f}**\n\n"
    for check in result["hallucination_checks"]:
        if check["supported"]:
            checks_display += f"\u2705 **Supported**: {check['claim']}\n\n"
        else:
            checks_display += f"\u274c **NOT Supported**: {check['claim']}\n\n"

    return result["answer"], agent_log, docs_display, checks_display

What does this function do?

This function is the bridge between the Gradio UI and your RAG pipeline. It calls run_query() from rag_graph.py, then formats the raw results into four display-ready strings.

Similarity is calculated as 1 - distance because ChromaDB's cosine distance returns lower values for more similar documents. The hallucination checks are formatted with checkmarks and crosses so the user can instantly see which claims passed verification.

• Save app.py.

You won't see a visible change yet. The next substep adds the evaluation functions, and then you will wire everything into the Gradio UI.

Add the evaluation and calibration tab

The evaluation tab needs two backend functions. One runs the batch evaluation suite and formats the results. The other accepts your human ratings and compares them against the LLM judge.

• Below chat_with_rag() in app.py, add the run_eval_batch() function by copying this code:

def run_eval_batch():
    # Only evaluate the first 3 questions to keep response times reasonable
    questions_subset = TEST_QUESTIONS[:3]
    results = run_evaluation_suite(questions_subset)

    # Build a Markdown summary of each evaluation result
    display = ""
    for r in results:
        display += f"### Q: {r['question']}\n\n"
        display += f"**Answer**: {r['answer'][:300]}...\n\n"
        display += f"**Faithfulness (minicheck)**: {r['faithfulness_score']:.2f}\n\n"
        for criterion, score_data in r["evaluation"].items():
            display += f"- **{criterion}**: {score_data['score']}/5 - {score_data['reasoning']}\n"
        display += "\n---\n\n"

    # Also return raw JSON so the calibration step can parse it
    raw_json = json.dumps(results, indent=2, default=str)
    return display, raw_json

What does this function do?

This function runs your evaluation suite on the first 3 test questions (to keep wait times short) and returns two things. The first is a human-readable Markdown summary. The second is the raw JSON string, which gets passed to the calibration function later.

• Save app.py.

You won't see a change yet. The next chunk adds the human calibration handler.

• Below run_eval_batch() in app.py, add the submit_human_ratings() function by copying this code:

def submit_human_ratings(ratings_json, eval_results_json):
    # Parse both JSON inputs with error handling
    try:
        human_ratings = json.loads(ratings_json)
        eval_results = json.loads(eval_results_json)
    except (json.JSONDecodeError, TypeError):
        return "Error: Could not parse JSON. Check the format and try again."

    # Run calibration comparison between human and LLM judge scores
    calibration = compute_calibration(eval_results, human_ratings)

    # Format the calibration summary as Markdown
    display = "## Calibration Results\n\n"
    for criterion, stats in calibration["summary"].items():
        display += f"### {criterion.title()}\n"
        display += f"- Average score difference: {stats['avg_difference']}\n"
        display += f"- Exact match rate: {stats['exact_match_rate']:.0%}\n"
        display += f"- Within 1 point rate: {stats['within_one_rate']:.0%}\n\n"

    # Show major disagreements if any exist
    if calibration["disagreements"]:
        display += "## Major Disagreements (diff >= 2)\n\n"
        for d in calibration["disagreements"]:
            display += f"- **{d['question'][:50]}...** ({d['criterion']}): LLM={d['llm_score']}, Human={d['human_score']}\n"
            display += f"  LLM reasoning: {d['llm_reasoning']}\n\n"
    else:
        display += "No major disagreements found. The LLM judge is well-calibrated with your ratings.\n"

    return display

What does this function do?

This function takes two JSON strings: your human ratings and the raw evaluation results from the previous step. It parses both, calls compute_calibration() from evaluator.py, and formats the agreement metrics into a readable report.

Any disagreement of 2 or more points between your score and the LLM judge's score gets flagged. These are the cases where the judge needs improvement through few-shot examples.

• Save app.py.

You won't see a change yet. The next substep wires all four functions into the Gradio layout and launches the server.

Build the UI layout and launch the server

Gradio's Blocks API lets you compose complex layouts with tabs, rows, and columns. You will create one function that defines the entire interface, then a main block that starts the server.

• Below submit_human_ratings() in app.py, add the build_ui() function by copying this code:

def build_ui():
    with gr.Blocks(title="Multi-Agent RAG with Hallucination Detection") as app:
        gr.Markdown("# Multi-Agent RAG Pipeline\n### Hallucination Detection and Auto-Judge Evaluation")

        # === Chat Tab: question input, answer, logs, docs, hallucination checks ===
        with gr.Tab("Chat"):
            gr.Markdown("Ask a question and watch the multi-agent pipeline retrieve context, generate an answer, and verify each claim for hallucinations.")

            with gr.Row():
                question_input = gr.Textbox(label="Your Question", placeholder="e.g., What is the Transformer architecture?", lines=2, scale=4)
                submit_btn = gr.Button("Ask", variant="primary", scale=1)

            answer_output = gr.Textbox(label="Answer", lines=4)

            # Side-by-side agent log and retrieved documents
            with gr.Row():
                with gr.Column():
                    agent_log_output = gr.Textbox(label="Agent Execution Log", lines=6)
                with gr.Column():
                    docs_output = gr.Markdown(label="Retrieved Documents")

            hallucination_output = gr.Markdown(label="Hallucination Check Results")

            # Wire the Ask button to chat_with_rag
            submit_btn.click(fn=chat_with_rag, inputs=[question_input], outputs=[answer_output, agent_log_output, docs_output, hallucination_output])

        # === Evaluation Tab: batch eval suite + human calibration ===
        with gr.Tab("Evaluation"):
            gr.Markdown("Run the auto-judge evaluation suite on test questions and calibrate the judge against your own ratings.")

            eval_btn = gr.Button("Run Evaluation Suite (3 questions)", variant="primary")
            eval_display = gr.Markdown(label="Evaluation Results")
            eval_raw = gr.Textbox(label="Raw Results JSON (for calibration)", lines=6, visible=True)

            # Wire the eval button to run_eval_batch
            eval_btn.click(fn=run_eval_batch, outputs=[eval_display, eval_raw])

            gr.Markdown("---\n## Human Calibration")
            gr.Markdown("After reviewing the results above, enter your own ratings for each question using the same 1-5 scale. Format as a JSON list with one object per question.")

            human_ratings_input = gr.Textbox(label="Your Ratings (JSON list)", placeholder='[{"correctness": 4, "helpfulness": 5, "faithfulness": 3}, ...]', lines=5)
            calibrate_btn = gr.Button("Compare with LLM Judge")
            calibration_output = gr.Markdown(label="Calibration Results")

            # Wire the calibration button to submit_human_ratings
            calibrate_btn.click(fn=submit_human_ratings, inputs=[human_ratings_input, eval_raw], outputs=[calibration_output])

    return app

What's happening in this function?

The gr.Blocks() context manager creates the app shell. Inside it, gr.Tab() creates the two tabs. Each tab's widgets are defined in order from top to bottom.

The .click() calls wire buttons to your Python functions. Gradio handles the rest: it serializes the inputs, calls your function, and pushes the outputs back to the browser. The eval_raw textbox serves double duty. It displays the raw JSON for inspection and is also passed as an input to the calibration button so the human ratings can be compared.

• Save app.py.

You won't see a change yet. The next chunk adds the main block that starts the server.

• At the very bottom of app.py, add the main block that launches the server by copying this code:

if __name__ == "__main__":
    # Create the knowledge base on first run, then build and launch the UI
    ensure_knowledge_base()
    app = build_ui()
    app.launch(share=False)

What does this do?

The main block first ensures the knowledge base exists, then builds the Gradio layout and starts a local web server. The share=False flag keeps the server local to your machine. If you set it to True, Gradio would create a temporary public URL.

• Save app.py.
• Make sure your virtual environment is activated, then launch the app by running:

python app.py

You should see output like Running on local URL: http://127.0.0.1:7860 in your terminal. Open that URL in your browser to see the Gradio interface with the Chat and Evaluation tabs.

Not seeing the Gradio interface?

• Make sure Ollama is running in the background. Open a separate terminal and run ollama list to confirm the models are available.
• Check that your virtual environment is activated. Your terminal prompt should show (venv) at the start.
• If you see an ImportError, verify that all packages are installed by running pip install -r requirements.txt.

Help me debug why my Gradio app won't start

Now test the Chat tab.

• Type What is the Transformer architecture? in the Your Question field.
• Click Ask.

This will take 30-60 seconds because the full pipeline runs: retrieval, generation, claim extraction, and hallucination checking. When it finishes, you should see the answer text, an agent execution log listing each agent that ran, the retrieved document chunks with similarity scores, and a hallucination check section with supported/unsupported claim verdicts.

First query is slowest

Ollama loads models into memory on first use. Subsequent queries will be noticeably faster because the models stay cached in RAM.

Next, test the Evaluation tab.

• Click the Evaluation tab at the top of the interface.
• Click Run Evaluation Suite (3 questions).

This takes several minutes because it runs three questions through the full RAG pipeline and judges each response on three criteria. When it finishes, you should see the evaluation results as a Markdown summary and the raw JSON below it.

• Scroll down to the Human Calibration section.
• In the Your Ratings (JSON list) field, paste your own ratings using the same format shown in the placeholder. For example:

[
  {"correctness": 4, "helpfulness": 4, "faithfulness": 5},
  {"correctness": 5, "helpfulness": 4, "faithfulness": 4},
  {"correctness": 4, "helpfulness": 3, "faithfulness": 5}
]

What scores should I give?

Read through the evaluation results above and rate each answer yourself on the 1-5 scale. There are 3 questions, so you need 3 objects in the JSON list. Be honest with your ratings. The whole point of calibration is to find where you and the LLM judge disagree.

• Click Compare with LLM Judge.

You should see the calibration results appear below, showing the average score difference, exact match rate, and within-one-point rate for each criterion. Any major disagreements are listed with the LLM judge's reasoning.

Seeing a JSON parse error?

Make sure your ratings JSON is a valid list with exactly 3 objects (matching the 3 evaluated questions). Each object must have correctness, helpfulness, and faithfulness keys with integer values from 1 to 5.

Help me fix a JSON parse error in the calibration step

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py.

ⓧ I'd like to double check the full code

Here is the complete app.py file. Compare yours against it:

import json
import os

import gradio as gr

from knowledge_base import create_knowledge_base, PERSIST_DIR
from rag_graph import run_query
from evaluator import (
    evaluate_response,
    run_evaluation_suite,
    compute_calibration,
    TEST_QUESTIONS,
)


def ensure_knowledge_base():
    if not os.path.exists(PERSIST_DIR):
        print("Creating knowledge base...")
        create_knowledge_base()


def chat_with_rag(question):
    if not question.strip():
        return "Please enter a question.", "", "", ""
    result = run_query(question)
    agent_log = "\n".join(result["agent_log"])
    docs_display = ""
    for i, doc in enumerate(result["retrieved_docs"]):
        title = doc["metadata"].get("title", "Unknown")
        distance = doc.get("distance", 0)
        similarity = 1 - distance if distance else 0
        docs_display += f"**Chunk {i + 1}** [{title}] (similarity: {similarity:.2f})\n\n{doc['content'][:300]}...\n\n---\n\n"
    checks_display = f"**Faithfulness Score: {result['faithfulness_score']:.2f}**\n\n"
    for check in result["hallucination_checks"]:
        if check["supported"]:
            checks_display += f"\u2705 **Supported**: {check['claim']}\n\n"
        else:
            checks_display += f"\u274c **NOT Supported**: {check['claim']}\n\n"
    return result["answer"], agent_log, docs_display, checks_display


def run_eval_batch():
    questions_subset = TEST_QUESTIONS[:3]
    results = run_evaluation_suite(questions_subset)
    display = ""
    for r in results:
        display += f"### Q: {r['question']}\n\n"
        display += f"**Answer**: {r['answer'][:300]}...\n\n"
        display += f"**Faithfulness (minicheck)**: {r['faithfulness_score']:.2f}\n\n"
        for criterion, score_data in r["evaluation"].items():
            display += f"- **{criterion}**: {score_data['score']}/5 - {score_data['reasoning']}\n"
        display += "\n---\n\n"
    raw_json = json.dumps(results, indent=2, default=str)
    return display, raw_json


def submit_human_ratings(ratings_json, eval_results_json):
    try:
        human_ratings = json.loads(ratings_json)
        eval_results = json.loads(eval_results_json)
    except (json.JSONDecodeError, TypeError):
        return "Error: Could not parse JSON. Check the format and try again."
    calibration = compute_calibration(eval_results, human_ratings)
    display = "## Calibration Results\n\n"
    for criterion, stats in calibration["summary"].items():
        display += f"### {criterion.title()}\n"
        display += f"- Average score difference: {stats['avg_difference']}\n"
        display += f"- Exact match rate: {stats['exact_match_rate']:.0%}\n"
        display += f"- Within 1 point rate: {stats['within_one_rate']:.0%}\n\n"
    if calibration["disagreements"]:
        display += "## Major Disagreements (diff >= 2)\n\n"
        for d in calibration["disagreements"]:
            display += f"- **{d['question'][:50]}...** ({d['criterion']}): LLM={d['llm_score']}, Human={d['human_score']}\n"
            display += f"  LLM reasoning: {d['llm_reasoning']}\n\n"
    else:
        display += "No major disagreements found. The LLM judge is well-calibrated with your ratings.\n"
    return display


def build_ui():
    with gr.Blocks(title="Multi-Agent RAG with Hallucination Detection") as app:
        gr.Markdown("# Multi-Agent RAG Pipeline\n### Hallucination Detection and Auto-Judge Evaluation")
        with gr.Tab("Chat"):
            gr.Markdown("Ask a question and watch the multi-agent pipeline retrieve context, generate an answer, and verify each claim for hallucinations.")
            with gr.Row():
                question_input = gr.Textbox(label="Your Question", placeholder="e.g., What is the Transformer architecture?", lines=2, scale=4)
                submit_btn = gr.Button("Ask", variant="primary", scale=1)
            answer_output = gr.Textbox(label="Answer", lines=4)
            with gr.Row():
                with gr.Column():
                    agent_log_output = gr.Textbox(label="Agent Execution Log", lines=6)
                with gr.Column():
                    docs_output = gr.Markdown(label="Retrieved Documents")
            hallucination_output = gr.Markdown(label="Hallucination Check Results")
            submit_btn.click(fn=chat_with_rag, inputs=[question_input], outputs=[answer_output, agent_log_output, docs_output, hallucination_output])
        with gr.Tab("Evaluation"):
            gr.Markdown("Run the auto-judge evaluation suite on test questions and calibrate the judge against your own ratings.")
            eval_btn = gr.Button("Run Evaluation Suite (3 questions)", variant="primary")
            eval_display = gr.Markdown(label="Evaluation Results")
            eval_raw = gr.Textbox(label="Raw Results JSON (for calibration)", lines=6, visible=True)
            eval_btn.click(fn=run_eval_batch, outputs=[eval_display, eval_raw])
            gr.Markdown("---\n## Human Calibration")
            gr.Markdown("After reviewing the results above, enter your own ratings for each question using the same 1-5 scale. Format as a JSON list with one object per question.")
            human_ratings_input = gr.Textbox(label="Your Ratings (JSON list)", placeholder='[{"correctness": 4, "helpfulness": 5, "faithfulness": 3}, ...]', lines=5)
            calibrate_btn = gr.Button("Compare with LLM Judge")
            calibration_output = gr.Markdown(label="Calibration Results")
            calibrate_btn.click(fn=submit_human_ratings, inputs=[human_ratings_input, eval_raw], outputs=[calibration_output])
    return app


if __name__ == "__main__":
    ensure_knowledge_base()
    app = build_ui()
    app.launch(share=False)

Your entire multi-agent RAG system now has a proper frontend. You can ask questions, watch the agents work, run batch evaluations, and calibrate the judge all from the browser. Next up, you will tackle the Secret Mission and add evaluation regression detection to track quality across runs.

Secret mission

Add Evaluation Regression Detection

Your RAG pipeline can detect hallucinations in real time. But what happens when quality silently degrades over multiple runs? In this mission, you will build a regression detection system that tracks evaluation scores across runs and alerts you when quality drops below threshold.

Before you go

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. The three Ollama models use approximately 3-4 GB of disk space combined. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

• Three Ollama models: llama3.2, bespoke-minicheck, and nomic-embed-text.
• ChromaDB database stored in the local ./chroma_db directory.
• Python virtual environment with installed packages.
• Project files: knowledge_base.py, rag_graph.py, evaluator.py, app.py, and requirements.txt.

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment further with the RAG pipeline and evaluation framework.

• Your Ollama models stay pulled and ready to use.
• Your ChromaDB knowledge base remains populated in ./chroma_db.
• Everything runs locally, so there is no ongoing cost.

Pause - I'll come back to this later

Free up memory by stopping any running processes while keeping your files and models in place.

• Stop the Gradio server if it is still running by pressing Ctrl+C in the terminal.
• Deactivate your Python virtual environment by running:

deactivate

• Your Ollama models and project files stay on disk, ready for when you return.

Delete - I don't want to use this again

Remove all project resources and free up disk space. This deletes your models, database, and code.

• Remove the three Ollama models by running:

ollama rm llama3.2
ollama rm bespoke-minicheck
ollama rm nomic-embed-text

• Deactivate your virtual environment if it is still active:

deactivate

• Delete the project directory including the ChromaDB data and virtual environment by running:

cd ~/Desktop
rm -rf rag-multi-agent

• This removes all project files, the ./chroma_db database, and the venv folder with installed packages.

Mission accomplished

That's a Wrap!

Nice work! You just built a production-grade multi-agent RAG system with automated quality assurance from scratch.

You've learned how to:

• Build a LangGraph multi-agent pipeline with three specialized agents (retriever, generator, and hallucination checker) coordinated through explicit state management and conditional self-healing routing that automatically regenerates unfaithful answers.
• Implement claim-level hallucination detection using bespoke-minicheck alongside LLM-as-judge evaluation for defense in depth.
• Design evaluation rubrics for correctness, helpfulness, and faithfulness, then calibrate the LLM judge against your own human ratings using few-shot corrections from disagreements.
• Add evaluation regression detection that tracks scores across runs and alerts when quality drops below threshold.

Ready to quiz yourself?