RAG Hallucination Detection System

Build a self-correcting RAG pipeline that detects and fixes its own hallucinations.

Introduction

30 Second Summary

Imagine asking an AI a question about your research papers and getting a confident, detailed answer that's completely fabricated. That's hallucination, and it's one of the biggest barriers to trusting AI in high-stakes work like academic research.

In this project, you will build a RAG system that detects when its own answers contain hallucinations and automatically corrects them before delivering results to you.

What You'll Build

You'll ask your system a tricky cross-paper question, watch it catch itself making up details, rewrite its own search query, retrieve better evidence, and deliver a corrected answer with a confidence score proving it's grounded in the source material.

By the end of this project, you'll have:

A question-answering pipeline over research papers that returns real-time faithfulness scores alongside every answer.
A self-correcting workflow built with LangGraph that catches hallucinations, rewrites queries, and regenerates grounded answers automatically.
A side-by-side comparison proving measurable hallucination reduction between a naive pipeline and your uncertainty-aware system.
Secret Mission: Generate publication-ready evaluation metrics using RAGAS for your thesis or final year report.

Are there any prerequisites?

You need basic programming knowledge in any language. No prior Python or machine learning experience is required. You should have VS Code installed and a machine with a GPU (this project uses an RTX 3050 for local embeddings).

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

RAG systems can confidently deliver fabricated information with no warning to the user. You're going to build a system that catches itself hallucinating and fixes its own answers before they reach the user.

Set Up Your Python Environment and API Access

Your uncertainty-aware RAG system needs several moving parts: an LLM for generating answers, an embedding model for turning text into vectors, and a vector database for storing and retrieving those vectors. Before you can wire these together, each tool needs to be installed and tested.

In this step, you'll set up a Python virtual environment with all the packages you need. You'll also grab a free API key for Google Gemini (your LLM) and confirm that your GPU can run the embedding model locally.

In this step, get ready to:

Install the RAG stack into a Python virtual environment.
Create a free Gemini API key from Google AI Studio.
Run a test script that confirms both LLM access and GPU embedding.

Create your virtual environment and install packages

A virtual environment isolates your project's packages from the rest of your system. This prevents version conflicts between projects.

Check your Python version by running this command in the VS Code terminal (open it with Ctrl+`):

python --version

✔️ I see version 3.10 or higher

Your Python version is compatible. Move on to creating your project folder below.

ⓧ I see a version lower than 3.10

LangChain and LangGraph require Python 3.10 or higher. Download and install the latest Python from python.org/downloads.

During installation, check the box labeled Add python.exe to PATH.
After installation, close and reopen your terminal, then run python --version again to confirm 3.10 or higher.

ⓧ Command not found

Python is not on your PATH. Download and install Python from python.org/downloads.

During installation, check the box labeled Add python.exe to PATH.
Close and reopen your terminal after installation.
Run python --version to confirm it prints 3.10 or higher.

Move to your Desktop by running this command:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir rag-hallucination-detector
cd rag-hallucination-detector

Create a virtual environment by running:

python -m venv venv

Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

.\venv\Scripts\Activate.ps1

Seeing a script execution error?

If PowerShell blocks the activation script, run this command first to allow local scripts:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Then try the activation command again.

Your terminal prompt now shows (venv) at the beginning, confirming the environment is active.

Install all the project packages by running this command:

pip install langchain langgraph chromadb sentence-transformers langchain-google-genai google-generativeai ragas pypdf numpy python-dotenv

What are these packages?

langchain and langgraph provide the orchestration framework for building your RAG pipeline and correction loop.
chromadb is a lightweight vector database that runs locally with zero configuration.
sentence-transformers loads the embedding model on your GPU to convert text into vectors.
langchain-google-genai and google-generativeai connect your pipeline to the Gemini API.
ragas provides standardized RAG evaluation metrics for the Secret Mission.
pypdf reads PDF research papers so you can chunk and embed them.

The install takes a few minutes because sentence-transformers downloads PyTorch with CUDA support. You'll see a stream of download and installation progress messages.

Install taking forever or failing?

Check that (venv) appears in your prompt. If not, re-run the activation command.
If you see a network timeout, try running the pip install command again. Pip resumes partial downloads.

Help me troubleshoot my pip install errors.

Set up your Gemini API key

Google Gemini is the LLM your pipeline uses to generate answers and evaluate faithfulness. The free tier provides enough quota for development and testing.

Open your browser and go to Google AI Studio.
Sign in with your Google account.
If you are a new user, Google AI Studio automatically creates a default project and API key after you accept the Terms of Service. Copy the key from the API keys page.
If you do not see a key, click Get API key and then click Create API key to generate a new one.
Copy the API key to your clipboard. You will only see it once.

Seeing a permissions or region error?

The Gemini free tier may not be available in all countries. If you see an access restricted error, you may be in an unsupported region.

Help me resolve a Google AI Studio access error.

Back in VS Code, create a file called .env in the root of your rag-hallucination-detector folder by running:

echo GOOGLE_API_KEY=your_gemini_api_key_here > .env

Open the .env file in VS Code and replace your_gemini_api_key_here with the actual API key you copied from Google AI Studio.
Save the file.

Why a .env file?

Storing your API key in a .env file keeps it out of your source code. The python-dotenv package loads it into your environment at runtime so your scripts can access it without hardcoding secrets.

Confirm the file exists by running:

ls .env

You should see .env listed in the output.

Verify your setup with a test script

Now you'll run a script that tests both your Gemini API connection and your local GPU embedding model. This catches configuration problems before you build the full pipeline.

Create test_setup.py and add the following code to test the Gemini API:

import os
from dotenv import load_dotenv
import google.generativeai as genai
from sentence_transformers import SentenceTransformer
import torch

load_dotenv()

# Test Gemini API
genai.configure(api_key=os.getenv('GOOGLE_API_KEY'))
model = genai.GenerativeModel('gemini-1.5-flash')
response = model.generate_content('Say hello in one sentence.')
print('Gemini response:', response.text)

What does this code do?

load_dotenv() reads your .env file and loads GOOGLE_API_KEY into the environment.
genai.configure() authenticates with the Gemini API using your key.
GenerativeModel('gemini-1.5-flash') selects the fast, free-tier Gemini model.
generate_content() sends a prompt and returns the model's response.

Save test_setup.py and run it:

python test_setup.py

You should see a short greeting printed to your terminal, something like: Gemini response: Hello! How can I help you today?.

Seeing an API key error?

Open your .env file and confirm it contains GOOGLE_API_KEY= followed by your key with no spaces around the = sign.
Make sure you copied the full key from Google AI Studio with no trailing whitespace.

Help me debug my Gemini API connection.

Now add the GPU embedding test below your existing code in test_setup.py. Find the last line (print('Gemini response:', response.text)) and add this code below it:


# Test local embedding model on GPU
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'Using device: {device}')
embedder = SentenceTransformer('all-MiniLM-L6-v2', device=device)
vector = embedder.encode('Test sentence for embedding.')
print(f'Embedding shape: {vector.shape}')

What does this code do?

torch.cuda.is_available() checks whether your NVIDIA GPU is accessible. If yes, it sets device to cuda.
SentenceTransformer('all-MiniLM-L6-v2') loads a lightweight embedding model that produces 384-dimensional vectors.
embedder.encode() converts a text string into a numerical vector your RAG pipeline can store and search.

Save test_setup.py.

Before you run this, what do you expect to see printed for the device and embedding shape?

Run the complete test script:

python test_setup.py

You should see three lines of output: the Gemini greeting, Using device: cuda, and Embedding shape: (384,). The cuda device confirms your RTX 3050 is being used. The 384 dimensions match the all-MiniLM-L6-v2 model's output size.

Seeing device: cpu instead of cuda?

Your NVIDIA drivers may be outdated. Update them from the NVIDIA website.
PyTorch may have installed without CUDA support. Try reinstalling with the CUDA-specific wheel from pytorch.org/get-started.
The pipeline still works on CPU, but embedding will be slower during retrieval.

Help me get CUDA working with PyTorch on my RTX 3050.

✔️ Awesome, I've got everything!

Great. Double check you've saved test_setup.py and confirm your output shows all three lines.

ⓧ I'd like to double check the full code

Here is the complete test_setup.py file:

import os
from dotenv import load_dotenv
import google.generativeai as genai
from sentence_transformers import SentenceTransformer
import torch

load_dotenv()

# Test Gemini API
genai.configure(api_key=os.getenv('GOOGLE_API_KEY'))
model = genai.GenerativeModel('gemini-1.5-flash')
response = model.generate_content('Say hello in one sentence.')
print('Gemini response:', response.text)

# Test local embedding model on GPU
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'Using device: {device}')
embedder = SentenceTransformer('all-MiniLM-L6-v2', device=device)
vector = embedder.encode('Test sentence for embedding.')
print(f'Embedding shape: {vector.shape}')

And your .env file should contain:

GOOGLE_API_KEY=your_gemini_api_key_here

Replace your_gemini_api_key_here with your actual Gemini API key.

Your environment is ready. You have a working LLM connection and a GPU-accelerated embedding model. Next up, you'll build a naive RAG pipeline that retrieves and generates answers from research papers.

Build a Naive RAG Pipeline Over Research Papers

Your environment is verified and ready. Now you need actual data to work with and a pipeline that can answer questions about it.

In this step, you'll build a naive RAG pipeline. The word "naive" is deliberate. This pipeline retrieves chunks from research papers and passes them to Gemini with no quality checks whatsoever. On simple factual questions it works fine. On complex questions that require synthesizing across multiple papers, it will confidently fabricate details. That visible failure is exactly what you need as a baseline before adding hallucination detection in Step 3.

In this step, get ready to:

Download arXiv papers, chunk them, and embed them into ChromaDB.
Build a retrieve-and-generate chain using LangChain and Gemini.
Test with questions that reveal hallucination behavior.

Download and ingest research papers into ChromaDB

RAG systems need a knowledge base to retrieve from. You'll download a few arXiv papers on attention mechanisms and transformers, then chunk and embed them into a local vector database.

Create a papers/ directory to store your PDFs by running this command:

mkdir papers

Now you need 3-5 research papers as PDFs. You'll download them from arXiv, a free repository of academic papers.

Download papers on attention mechanisms by running these commands:

curl -L -o papers/attention_is_all_you_need.pdf https://arxiv.org/pdf/1706.03762.pdf
curl -L -o papers/bert.pdf https://arxiv.org/pdf/1810.04805.pdf
curl -L -o papers/vision_transformer.pdf https://arxiv.org/pdf/2010.11929.pdf

Why these papers?

These are foundational papers on attention mechanisms and transformers. They contain detailed training configurations (optimizers, learning rates, datasets) that make good test cases. Your simple factual question will target details from one paper, while your complex question will ask for cross-paper comparisons that the naive pipeline cannot reliably answer.

You can substitute any 3-5 PDFs on a topic you care about. Just make sure at least one paper has explicit training details you can ask about.

Confirm the papers downloaded by running:

ls papers/

You should see your 3 PDF files listed.

Downloads failing?

If curl returns an error, check your internet connection. You can also download the papers manually by visiting arxiv.org, searching for the paper titles, clicking the PDF link, and saving each file into your papers/ folder.

Help me troubleshoot my paper downloads.

Now you need a script that reads those PDFs, splits them into chunks, and stores them as vector embeddings in ChromaDB.

Create a new file called ingest.py in your project folder and add this code:

import os
import glob
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings

load_dotenv()

embeddings = HuggingFaceEmbeddings(
    model_name='all-MiniLM-L6-v2',
    model_kwargs={'device': 'cuda'}
)

docs = []
for pdf_path in glob.glob('papers/*.pdf'):
    loader = PyPDFLoader(pdf_path)
    docs.extend(loader.load())

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

vectorstore = Chroma.from_documents(
    chunks,
    embeddings,
    persist_directory='chroma_db'
)
print(f'Ingested {len(chunks)} chunks into ChromaDB.')

What does this code do?

HuggingFaceEmbeddings loads the all-MiniLM-L6-v2 model onto your GPU. This is the same model you tested in Step 1, now wrapped in LangChain's interface.
The glob loop finds every PDF in papers/ and loads it page-by-page using PyPDFLoader.
RecursiveCharacterTextSplitter breaks pages into 500-character chunks with 50-character overlap, so no information is lost at chunk boundaries.
Chroma.from_documents embeds every chunk and stores the vectors in a local database at chroma_db/.

Save ingest.py.
Run the ingestion script:

python ingest.py

This takes 30-60 seconds depending on how many papers you downloaded. You should see a message like Ingested 312 chunks into ChromaDB. (your number will vary based on paper length).

Seeing an import error?

If you see an error about pypdf not being found, install it by running pip install pypdf in your terminal. The PyPDFLoader class depends on this package to parse PDF files.

Help me fix my ingest.py import errors.

✔️ Awesome, I've got everything!

Great. Double check you've saved ingest.py and confirmed the chunk count printed to your terminal.

ⓧ I'd like to double check the full code

import os
import glob
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings

load_dotenv()

embed dings = HuggingFaceEmbeddings(
    model_name='all-MiniLM-L6-v2',
    model_kwargs={'device': 'cuda'}
)

docs = []
for pdf_path in glob.glob('papers/*.pdf'):
    loader = PyPDFLoader(pdf_path)
    docs.extend(loader.load())

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

vectorstore = Chroma.from_documents(
    chunks,
    embeddings,
    persist_directory='chroma_db'
)
print(f'Ingested {len(chunks)} chunks into ChromaDB.')

Build the naive retrieve-and-generate chain

Your vector store is populated. Now you need a pipeline that takes a user's question, retrieves the most relevant chunks, and passes them to Gemini to generate an answer.

Create a new file called naive_rag.py and add the chain setup:

import os
from dotenv import load_dotenv
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain.chains import RetrievalQA

load_dotenv()

embed dings = HuggingFaceEmbeddings(
    model_name='all-MiniLM-L6-v2',
    model_kwargs={'device': 'cuda'}
)

vectorstore = Chroma(
    persist_directory='chroma_db',
    embedding_function=embeddings
)

llm = ChatGoogleGenerativeAI(
    model='gemini-1.5-flash',
    google_api_key=os.getenv('GOOGLE_API_KEY')
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={'k': 4}),
    return_source_documents=True
)

What does this code do?

The script reconnects to your existing chroma_db vector store using the same embedding model from ingestion.
ChatGoogleGenerativeAI creates a LangChain-compatible wrapper around Gemini 1.5 Flash.
RetrievalQA.from_chain_type wires everything together. When you send a question, it retrieves the top 4 most relevant chunks (k=4), stuffs them into a prompt, and asks Gemini to answer based on that context.
There are zero quality checks here. Whatever Gemini returns is accepted as the final answer.

Add the query function and test questions below the existing code in naive_rag.py:

def naive_rag_query(question: str) -> dict:
    result = qa_chain({'query': question})
    return {
        'question': question,
        'answer': result['result'],
        'source_docs': result['source_documents']
    }

if __name__ == '__main__':
    # Factual single-hop question
    q1 = 'What specific optimizer and learning rate did the attention mechanism paper use?'
    r1 = naive_rag_query(q1)
    print('Q1 (factual):', r1['answer'])

    # Complex cross-paper question likely to hallucinate
    q2 = 'Compare the exact training configurations and datasets used across all papers in the collection.'
    r2 = naive_rag_query(q2)
    print('Q2 (complex/hallucination-prone):', r2['answer'])

What does this code do?

naive_rag_query wraps the chain call and returns the answer plus source documents in a clean dictionary.
Q1 is a factual single-hop question. The answer (Adam optimizer, specific learning rate schedule) exists in a single paper's text. The pipeline should handle this correctly.
Q2 asks for a cross-paper comparison of exact training details. This forces Gemini to synthesize across multiple chunks from different papers. When the retrieved context does not contain complete information, the model tends to fill in gaps with plausible-sounding fabrications.

Save naive_rag.py.

✔️ Awesome, I've got everything!

Great. Confirm you have both the chain setup and the query function with test questions in naive_rag.py.

ⓧ I'd like to double check the full code

Your naive_rag.py should contain two sections. Scroll up and confirm:

The first section has imports, embeddings, vectorstore, llm, and qa_chain setup.
The second section has the naive_rag_query function and the if __name__ == '__main__' block with both test questions.

Run the pipeline and observe hallucination

This is the moment of truth. You have a complete RAG pipeline. Time to see how it handles different types of questions.

Before you run the script, predict: do you think both questions will get accurate, well-grounded answers? Or will one behave differently from the other?
Run the naive pipeline:

python naive_rag.py

Watch both answers print to your terminal. Q1 should return a factual answer grounded in the paper's actual content (the "Attention Is All You Need" paper uses the Adam optimizer with a specific warmup learning rate schedule). Q2 is where things get interesting.

Read Q2's answer carefully. You'll notice it presents specific details about training configurations "from each paper" with confident, authoritative language. Compare those claims against what you actually know is in the papers. Many of these details are fabricated.

Why does Q2 hallucinate?

The retriever pulls the top 4 chunks, but a cross-paper comparison question needs details scattered across many more chunks than 4. When the retrieved context has gaps, Gemini fills them with plausible-sounding information rather than admitting it does not know. This is the core problem with naive RAG: there is no mechanism to detect when the answer goes beyond what the context supports.

This is not a Gemini-specific problem. Every LLM does this. The fix is not a better model. The fix is a detection layer that catches unsupported claims before they reach the user.

Getting a rate limit or API error?

The Gemini free tier has per-minute rate limits. If you see a rate limit error, wait 60 seconds and run the script again. If you see an authentication error, double-check that your .env file contains your GOOGLE_API_KEY and that the key is valid in Google AI Studio.

Help me debug my Gemini API connection in naive_rag.py.

You now have documented proof that naive RAG hallucinates on complex questions. The pipeline confidently fabricated cross-paper details without any indication that its answer is unreliable. In the next step, you'll build a detection layer that catches these unsupported claims before they reach the user.

Add Hallucination Detection with Faithfulness Scoring

In Step 2, you saw something troubling. Your naive RAG pipeline answered a factual single-hop question correctly, but when asked to compare details across papers, it confidently fabricated information.

That is the core problem with naive RAG. There is no mechanism to verify whether the generated answer is actually grounded in the retrieved context. In this step, you will build a detection layer that decomposes each answer into individual claims and checks every claim against the source material, producing a faithfulness score between 0.0 and 1.0.

In this step, get ready to:

Build a faithfulness scorer that extracts claims from an answer and verifies each one against retrieved context chunks.
Add a confidence threshold that classifies answers as PASS or FLAGGED.
Run the same test questions from Step 2 through the detector and confirm it catches the hallucinated answer.

Build the claim extraction and verification functions

The detection approach uses an LLM-as-a-judge pattern. Instead of a separate trained model, you will use Gemini itself to evaluate whether each claim in a generated answer is supported by the context that was retrieved.

Create a new file called faithfulness_scorer.py in your project folder by adding the following code:

import os
from dotenv import load_dotenv
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain.schema import HumanMessage

load_dotenv()

FAITHFULNESS_THRESHOLD = 0.7

llm = ChatGoogleGenerativeAI(
    model='gemini-1.5-flash',
    google_api_key=os.getenv('GOOGLE_API_KEY')
)

def extract_claims(answer: str) -> list[str]:
    prompt = f"""Break the following answer into individual factual claims. Return each claim on a new line, no numbering.

Answer: {answer}

Claims:"""
    response = llm.invoke([HumanMessage(content=prompt)])
    claims = [c.strip() for c in response.content.strip().split('\n') if c.strip()]
    return claims

What does this code do?

This sets up the faithfulness scoring infrastructure and defines the first key function:

FAITHFULNESS_THRESHOLD = 0.7 defines the cutoff. Any answer scoring below this is flagged as potentially hallucinated.
extract_claims() uses Gemini to decompose a full answer into individual factual statements. Each claim can then be verified independently against the source material.

Add the claim verification function below extract_claims in faithfulness_scorer.py:

def check_claim_supported(claim: str, context_chunks: list[str]) -> bool:
    context = '\n\n'.join(context_chunks)
    prompt = f"""Given the following context, is the claim supported? Answer only YES or NO.

Context:
{context}

Claim: {claim}

Answer:"""
    response = llm.invoke([HumanMessage(content=prompt)])
    return response.content.strip().upper().startswith('YES')

What does this function do?

check_claim_supported() takes a single claim and all the retrieved context chunks. It asks Gemini a binary question: is this claim supported by the context? A strict YES/NO format prevents the LLM from hedging.

By checking each claim individually, you get granular insight into exactly which parts of an answer are grounded and which are fabricated.

Save faithfulness_scorer.py.
Test that claim extraction works by running this command:

python -c "from faithfulness_scorer import extract_claims; claims = extract_claims('The model uses Adam optimizer with lr=0.001. Training runs for 100 epochs on ImageNet.'); print(claims)"

You should see a list of individual claims printed to terminal, each a separate factual statement extracted from the input sentence.

Seeing an import error?

Make sure your virtual environment is activated. You should see (venv) in your terminal prompt.
Confirm you saved faithfulness_scorer.py in the same folder as naive_rag.py and .env.
If you see an API error, check that your GOOGLE_API_KEY is correctly set in .env.

help me debug this import error in faithfulness_scorer.py

Add scoring logic with a confidence threshold

Now you have functions that extract claims and check each one individually. The next piece brings them together into a single score and applies the 0.7 threshold to classify answers as trustworthy or suspect.

Add the scoring and evaluation functions below check_claim_supported in faithfulness_scorer.py:

def score_faithfulness(answer: str, source_docs: list) -> dict:
    context_chunks = [doc.page_content for doc in source_docs]
    claims = extract_claims(answer)
    if not claims:
        return {'score': 1.0, 'claims': [], 'supported': [], 'unsupported': []}
    supported = []
    unsupported = []
    for claim in claims:
        if check_claim_supported(claim, context_chunks):
            supported.append(claim)
        else:
            unsupported.append(claim)
    score = len(supported) / len(claims)
    return {
        'score': score,
        'claims': claims,
        'supported': supported,
        'unsupported': unsupported
    }

def evaluate_answer(answer: str, source_docs: list) -> dict:
    result = score_faithfulness(answer, source_docs)
    result['status'] = 'PASS' if result['score'] >= FAITHFULNESS_THRESHOLD else 'FLAGGED'
    return result

How does the scoring work?

score_faithfulness() orchestrates the full check. It extracts all claims, verifies each one, and computes the ratio of supported claims to total claims. If 6 out of 8 claims are supported, the score is 0.75.

evaluate_answer() adds the threshold decision. Scores at or above 0.7 get a PASS status. Scores below 0.7 are FLAGGED as potentially hallucinated. This threshold matches established baselines from corrective RAG research.

Save faithfulness_scorer.py.

Run the detector on the test questions

Time to prove the detector works. You will run the same two questions from Step 2 through the faithfulness scorer and see whether it correctly distinguishes the factual answer from the hallucinated one.

Add the test block at the bottom of faithfulness_scorer.py:

if __name__ == '__main__':
    from naive_rag import naive_rag_query

    q1 = 'What specific optimizer and learning rate did the attention mechanism paper use?'
    r1 = naive_rag_query(q1)
    eval1 = evaluate_answer(r1['answer'], r1['source_docs'])
    print(f"Q1 faithfulness score: {eval1['score']:.2f} -> {eval1['status']}")

    q2 = 'Compare the exact training configurations and datasets used across all papers in the collection.'
    r2 = naive_rag_query(q2)
    eval2 = evaluate_answer(r2['answer'], r2['source_docs'])
    print(f"Q2 faithfulness score: {eval2['score']:.2f} -> {eval2['status']}")

What does this test block do?

This reuses the exact same questions from Step 2. It imports naive_rag_query to get both the answer and the source documents, then runs each through evaluate_answer to compute and print the faithfulness score and status.

The factual question (Q1) should produce an answer well-supported by the retrieved chunks. The cross-paper comparison question (Q2) should produce claims that go beyond what the context contains.

Save faithfulness_scorer.py.

Before you run this, think about what you expect. Will Q1 pass or get flagged? What about Q2, the question that made naive RAG fabricate details?

Run the faithfulness scorer by executing:

python faithfulness_scorer.py

You should see two lines printed to terminal. Q1 (the factual question) scores at or above 0.70 with a PASS status. Q2 (the cross-paper comparison) scores below 0.70 with a FLAGGED status.

Why does this work?

The factual question asks about details from a single paper. The retrieved chunks contain that information directly, so the claims in the answer are verifiable. The cross-paper question forces the LLM to synthesize details that may not exist together in any single chunk.

When Gemini checks those fabricated claims against the actual context, it correctly identifies them as unsupported. The ratio of supported-to-total claims drops below 0.7, triggering the FLAGGED status.

Both questions showing PASS?

Your Q2 answer may have happened to stay closer to the context this time. Try running it again. LLM outputs vary between runs.
If Q2 consistently passes, try a more aggressive cross-paper question like: 'List the exact batch sizes, dropout rates, and GPU hours used by every paper in the collection.'
Check that your chroma_db contains papers with enough distinct training details to make cross-paper comparisons genuinely difficult.

help me understand why my faithfulness scorer is not flagging hallucinated answers

✔️ Awesome, I've got everything!

Great. Double check you have saved faithfulness_scorer.py and confirm both scores printed to terminal.

ⓧ I'd like to double check the full code

Your faithfulness_scorer.py should contain the following sections in order. Scroll up and confirm each one is present:

Imports (os, dotenv, ChatGoogleGenerativeAI, HumanMessage) and load_dotenv() call.
FAITHFULNESS_THRESHOLD = 0.7 constant and llm initialization with gemini-1.5-flash.
extract_claims() function that prompts Gemini to break an answer into individual claims.
check_claim_supported() function that asks Gemini YES/NO whether a claim is supported by context.
score_faithfulness() function that loops through claims and returns a score dict.
evaluate_answer() function that adds PASS/FLAGGED status based on the threshold.
if __name__ == '__main__': block that runs both test questions and prints scores.

Your detection layer is working. It correctly distinguishes answers grounded in retrieved context from answers that contain unsupported claims. In the next step, you will build a correction loop that takes those FLAGGED answers and rewrites the query to produce a grounded response.

Build the Correction Loop with LangGraph

Your faithfulness scorer correctly flags hallucinated answers. But flagging alone does not fix the problem.

In this step, you will build a self-correcting workflow using LangGraph. When the scorer flags an answer, the system automatically rewrites the query to target the unsupported claims, re-retrieves better evidence, and regenerates a grounded answer.

In this step, get ready to:

Define a LangGraph state machine with nodes for Retrieve, Generate, Check Faithfulness, Rewrite Query, and Safe Decline.
Wire conditional edges that route answers to delivery, rewriting, or graceful decline.
Run the cross-paper question and watch the system self-correct in real time.

Define the state machine structure

A LangGraph state machine needs three things: a typed state that flows between nodes, the node functions themselves, and edges that connect them. You will start by setting up the state schema and infrastructure.

Create a new file called correction_loop.py in your project folder and add the imports, infrastructure connections, and state definition:

import os
from typing import TypedDict, Annotated
from dotenv import load_dotenv
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain.schema import HumanMessage
from langgraph.graph import StateGraph, END
from faithfulness_scorer import evaluate_answer, FAITHFULNESS_THRESHOLD

load_dotenv()

# Maximum correction attempts before declining
MAX_RETRIES = 2

# Reuse the same embedding model and vector store from previous steps
embeddings = HuggingFaceEmbeddings(
    model_name='all-MiniLM-L6-v2',
    model_kwargs={'device': 'cuda'}
)
vectorstore = Chroma(
    persist_directory='chroma_db',
    embedding_function=embeddings
)
llm = ChatGoogleGenerativeAI(
    model='gemini-1.5-flash',
    google_api_key=os.getenv('GOOGLE_API_KEY')
)

What does this code do?

This sets up the shared infrastructure. The key import is StateGraph and END from langgraph.graph. These give you the building blocks for a state machine with conditional routing.

You also import evaluate_answer from your faithfulness scorer. This is the function that will be called inside the detection node to score each answer.

Below the infrastructure setup, add the state definition that will flow through every node:

class RAGState(TypedDict):
    question: str
    current_query: str
    source_docs: list
    answer: str
    faithfulness_result: dict
    retry_count: int
    final_answer: str
    status: str

What is RAGState?

Every node in the graph receives this state dictionary and returns an updated copy. The question field holds the original user question (never modified), while current_query holds the active search query (rewritten on each retry).

The retry_count field tracks how many correction loops have run. Once it hits MAX_RETRIES, the system declines gracefully instead of looping forever.

Save correction_loop.py and verify the imports resolve by running:

python correction_loop.py

You should see no output and no errors. This confirms LangGraph, your faithfulness scorer, and all dependencies are correctly linked.

Seeing an import error?

If you see an error about langgraph, confirm your virtual environment is activated and run pip install -U langgraph again.
If you see an error about faithfulness_scorer, make sure faithfulness_scorer.py is in the same directory as correction_loop.py.

Help me debug my LangGraph import error in correction_loop.py.

Add the pipeline node functions

Each node in the state machine is a plain Python function. It receives the current state, performs one job, and returns an updated state. You need five processing nodes plus a router function.

Below the RAGState class in correction_loop.py, add the retrieve and generate nodes:

def retrieve_node(state: RAGState) -> RAGState:
    print(f'[RETRIEVE] Query: {state["current_query"]}')
    docs = vectorstore.similarity_search(state['current_query'], k=4)
    return {**state, 'source_docs': docs}

def generate_node(state: RAGState) -> RAGState:
    print('[GENERATE] Generating answer...')
    context = '\n\n'.join([doc.page_content for doc in state['source_docs']])
    prompt = f'Answer the question using only the provided context.\n\nContext:\n{context}\n\nQuestion: {state["current_query"]}\n\nAnswer:'
    response = llm.invoke([HumanMessage(content=prompt)])
    return {**state, 'answer': response.content}

What do these nodes do?

retrieve_node runs a similarity search against ChromaDB using the current_query (which changes on rewrites). It stores the top 4 chunks in source_docs.
generate_node builds a prompt from the retrieved chunks and asks Gemini to answer using only that context. The print statements create the workflow trace you will see in the terminal.

Next, add the detection and correction nodes below the generate node:

def check_faithfulness_node(state: RAGState) -> RAGState:
    print('[CHECK] Scoring faithfulness...')
    result = evaluate_answer(state['answer'], state['source_docs'])
    print(f'[CHECK] Score: {result["score"]:.2f} -> {result["status"]}')
    return {**state, 'faithfulness_result': result}

def rewrite_query_node(state: RAGState) -> RAGState:
    print('[REWRITE] Rewriting query to target unsupported claims...')
    unsupported = state['faithfulness_result'].get('unsupported', [])
    unsupported_text = '\n'.join(unsupported)
    prompt = f'The original question was: {state["question"]}\n\nThe following claims in the answer were not supported by the retrieved context:\n{unsupported_text}\n\nRewrite the search query to better retrieve evidence for these claims. Return only the new query.'
    response = llm.invoke([HumanMessage(content=prompt)])
    new_query = response.content.strip()
    print(f'[REWRITE] New query: {new_query}')
    return {**state, 'current_query': new_query, 'retry_count': state['retry_count'] + 1}

How does query rewriting work?

When check_faithfulness_node flags an answer, it also stores the list of unsupported claims. The rewrite_query_node feeds those specific claims to Gemini and asks it to write a better search query targeting the missing evidence.

This is the key insight of Corrective RAG. Instead of retrying the same query, you ask a more targeted question based on exactly what went wrong.

Add the terminal nodes and router function below the rewrite node:

def safe_decline_node(state: RAGState) -> RAGState:
    print('[DECLINE] Max retries reached. Declining gracefully.')
    return {**state, 'final_answer': 'I was unable to generate a sufficiently grounded answer for this question based on the available documents.', 'status': 'DECLINED'}

def deliver_answer_node(state: RAGState) -> RAGState:
    print('[DELIVER] Answer passed faithfulness check.')
    return {**state, 'final_answer': state['answer'], 'status': 'PASS'}

def faithfulness_router(state: RAGState) -> str:
    if state['faithfulness_result']['status'] == 'PASS':
        return 'deliver'
    elif state['retry_count'] >= MAX_RETRIES:
        return 'decline'
    else:
        return 'rewrite'

What is the router function?

The faithfulness_router is not a node. It is a decision function that LangGraph calls after the check_faithfulness node to decide where to go next. It returns a string key that maps to the next node.

Three outcomes are possible: deliver the answer (score passed), rewrite the query (score failed but retries remain), or decline (score failed and retries exhausted).

Save correction_loop.py and run it again to verify no syntax errors:

python correction_loop.py

Again, no output and no errors means all node functions are syntactically correct.

Wire the graph edges and run the correction loop

With all nodes defined, you now wire them into a state machine. The conditional edge after check_faithfulness is what creates the self-correction loop.

Below the faithfulness_router function in correction_loop.py, add the graph construction:

# Build the state machine graph
graph = StateGraph(RAGState)
graph.add_node('retrieve', retrieve_node)
graph.add_node('generate', generate_node)
graph.add_node('check_faithfulness', check_faithfulness_node)
graph.add_node('rewrite_query', rewrite_query_node)
graph.add_node('safe_decline', safe_decline_node)
graph.add_node('deliver_answer', deliver_answer_node)

# Wire the edges
graph.set_entry_point('retrieve')
graph.add_edge('retrieve', 'generate')
graph.add_edge('generate', 'check_faithfulness')
graph.add_conditional_edges('check_faithfulness', faithfulness_router, {
    'deliver': 'deliver_answer',
    'decline': 'safe_decline',
    'rewrite': 'rewrite_query'
})
graph.add_edge('rewrite_query', 'retrieve')
graph.add_edge('deliver_answer', END)
graph.add_edge('safe_decline', END)

app = graph.compile()

How does the graph flow?

The entry point is retrieve. Every query starts by fetching chunks from ChromaDB.
After generate, the answer is checked for faithfulness. The add_conditional_edges call routes the flow based on the router function's return value.
The rewrite edge loops back to retrieve. This creates the correction cycle: retrieve, generate, check, rewrite, retrieve again.
Both deliver_answer and safe_decline terminate the graph by connecting to END.

Now add the query function and test block at the bottom of correction_loop.py:

def uncertainty_aware_query(question: str) -> dict:
    initial_state = RAGState(
        question=question,
        current_query=question,
        source_docs=[],
        answer='',
        faithfulness_result={},
        retry_count=0,
        final_answer='',
        status=''
    )
    result = app.invoke(initial_state)
    return result

if __name__ == '__main__':
    q = 'Compare the exact training configurations and datasets used across all papers in the collection.'
    result = uncertainty_aware_query(q)
    print('\nFinal answer:', result['final_answer'])
    print('Status:', result['status'])

What does uncertainty_aware_query do?

This function is the public interface to your correction loop. It creates the initial state (with retry_count at 0 and current_query set to the original question), then invokes the compiled graph. The graph runs through all nodes and conditional edges automatically.

The test uses the same cross-paper comparison question that caused hallucinations in the naive pipeline. This time, the system should catch the hallucination and self-correct.

Save correction_loop.py.

Before you run this, think about what you expect to happen. The cross-paper comparison question hallucinated in naive RAG. Will it pass on the first faithfulness check, or will the system need to rewrite and retry?

Run the correction loop:

python correction_loop.py

You should see a workflow trace printed to terminal showing the self-correction in action. The output follows this pattern:

[RETRIEVE] Query: Compare the exact training configurations...
[GENERATE] Generating answer...
[CHECK] Scoring faithfulness...
[CHECK] Score: 0.35 -> FLAGGED
[REWRITE] Rewriting query to target unsupported claims...
[REWRITE] New query: <rewritten query targeting unsupported claims>
[RETRIEVE] Query: <rewritten query>
[GENERATE] Generating answer...
[CHECK] Scoring faithfulness...
[CHECK] Score: 0.78 -> PASS
[DELIVER] Answer passed faithfulness check.

Final answer: <grounded answer>
Status: PASS

What if the status says DECLINED?

If the system reaches MAX_RETRIES (2 correction attempts) without producing a faithful answer, it declines gracefully. This is the safe behavior. It means the available documents genuinely do not contain enough information to answer the question without hallucinating.

Both PASS and DECLINED are valid outcomes. The important thing is that the system never delivers an unchecked hallucination.

Seeing a recursion or timeout error?

If you see a recursion limit error, confirm MAX_RETRIES = 2 is set at the top of the file. Without it, the loop would never exit.
If you hit a Gemini rate limit error, wait 60 seconds and try again. The correction loop makes multiple API calls (generate, check claims, rewrite), so it uses more quota than a single query.
If the faithfulness score is always 1.0 (everything passes), check that your check_claim_supported function in faithfulness_scorer.py correctly returns False for unsupported claims.

Help me debug my correction loop. It's not producing the expected workflow trace.

Finally, clean up faithfulness_scorer.py by removing the test block at the bottom. The correction loop now handles testing.
In faithfulness_scorer.py, find and delete everything from if __name__ == '__main__': to the end of the file.
Save faithfulness_scorer.py.

✔️ Awesome, I've got everything!

Great. Confirm you have saved both correction_loop.py and faithfulness_scorer.py.

ⓧ I'd like to double check the full code

Your correction_loop.py file is built from the following chunks (scroll up to confirm each one is present, in order):

Imports, infrastructure setup (load_dotenv, MAX_RETRIES, embeddings, vectorstore, llm).
RAGState TypedDict class with 8 fields.
retrieve_node and generate_node functions.
check_faithfulness_node and rewrite_query_node functions.
safe_decline_node, deliver_answer_node, and faithfulness_router functions.
Graph construction with StateGraph, all nodes, edges, conditional edges, and app = graph.compile().
uncertainty_aware_query function and if __name__ == '__main__' test block.

Your faithfulness_scorer.py should no longer have the if __name__ == '__main__': block at the bottom.

Your RAG system can now detect hallucinations and fix them automatically. Next up, you will put both pipelines head-to-head on a full test set to prove the improvement with numbers.

Compare Naive vs. Uncertainty-Aware Pipeline

Your uncertainty-aware pipeline catches hallucinations and self-corrects on individual questions. But a single example is anecdotal.

To prove the system actually works, you need quantitative evidence across a range of questions. In this step, you'll build a comparison script that runs both pipelines side-by-side on 10 test questions and outputs a formatted table showing where the correction loop earns its keep.

In this step, get ready to:

Create a test set of 10 questions mixing simple single-hop and complex multi-hop types.
Run both pipelines on all questions and collect faithfulness scores.
Generate a comparison table and summary statistics showing hallucination reduction.

Create a test set of 10 questions

A fair comparison needs both question types. Simple single-hop questions ("What dataset did paper 1 use?") test whether the uncertainty-aware pipeline preserves already-correct answers. Complex multi-hop questions ("Compare training configs across all papers") target the failure mode you observed in Step 2.

Before building the comparison script, you need to clean up your existing pipeline modules so they work as quiet imports (no debug prints cluttering the comparison output).

In correction_loop.py, find every print() line inside the node functions and delete it.
Also in correction_loop.py, remove Annotated from the typing import so it reads:

from typing import TypedDict

Remove the entire if __name__ == '__main__': block at the bottom of correction_loop.py.
In naive_rag.py, also remove the if __name__ == '__main__': block at the bottom.

Why remove the prints and main blocks?

When you import these files as modules in the comparison script, any top-level print statements would fire during import and drown out your comparison output. Removing __main__ blocks keeps the modules focused on exposing their functions without running test queries on import.

Now create compare_pipelines.py in your project folder with the imports and test questions by pasting the following:

import os
import csv
from naive_rag import naive_rag_query
from correction_loop import uncertainty_aware_query
from faithfulness_scorer import evaluate_answer

TEST_QUESTIONS = [
    # Simple single-hop questions
    {'question': 'What dataset was used in paper 1?', 'type': 'simple'},
    {'question': 'What is the main contribution of paper 2?', 'type': 'simple'},
    {'question': 'What evaluation metric did paper 3 use?', 'type': 'simple'},
    {'question': 'What model architecture is described in paper 1?', 'type': 'simple'},
    {'question': 'What baseline did paper 2 compare against?', 'type': 'simple'},
    # Complex multi-hop / comparison questions
    {'question': 'Compare the training datasets used across all papers.', 'type': 'complex'},
    {'question': 'Which paper achieved the highest reported accuracy and what method did it use?', 'type': 'complex'},
    {'question': 'How do the optimization strategies differ between paper 1 and paper 3?', 'type': 'complex'},
    {'question': 'Synthesize the common limitations acknowledged across all papers.', 'type': 'complex'},
    {'question': 'Compare the computational costs reported in each paper.', 'type': 'complex'},
]

What does this code do?

The imports pull in naive_rag_query and uncertainty_aware_query from your existing modules, plus the faithfulness scorer to evaluate both pipelines' answers.
The 5 simple questions target single facts that should be easily retrievable from individual papers.
The 5 complex questions require synthesizing across multiple papers, which is exactly where naive RAG tends to fabricate details.

Save compare_pipelines.py.
Confirm the file exists by running:

ls compare_pipelines.py

You should see compare_pipelines.py listed in the output.

File not found?

Make sure you saved the file in the same directory as naive_rag.py and correction_loop.py.
Check that your terminal is in the project directory by running ls naive_rag.py.

Still stuck? Help me find where my project files are located.

Run both pipelines on all questions

The evaluation loop runs each question through both the naive pipeline and the uncertainty-aware pipeline, then scores both answers for faithfulness against the retrieved context.

Add the following evaluation loop below the TEST_QUESTIONS list in compare_pipelines.py:

results = []
for item in TEST_QUESTIONS:
    q = item['question']
    print(f'\nEvaluating: {q}')

    # Run naive RAG and score its answer
    naive = naive_rag_query(q)
    naive_eval = evaluate_answer(naive['answer'], naive['source_docs'])

    # Run uncertainty-aware pipeline and score its answer
    ua = uncertainty_aware_query(q)
    ua_docs = ua['source_docs']
    ua_eval = evaluate_answer(ua['final_answer'], ua_docs)

    # Store comparison results
    results.append({
        'question': q,
        'type': item['type'],
        'naive_score': round(naive_eval['score'], 3),
        'ua_score': round(ua_eval['score'], 3),
        'improvement': round(ua_eval['score'] - naive_eval['score'], 3)
    })

What does this code do?

For each question, naive_rag_query runs the baseline retrieve-and-generate chain with no quality checks.
Then uncertainty_aware_query runs the full LangGraph state machine with hallucination detection and correction.
Both answers are scored by evaluate_answer using the same faithfulness scorer. The difference between scores shows how much the correction loop improved grounding.

Save compare_pipelines.py.

Generate the comparison table and run the evaluation

The final piece formats results into a readable table with summary statistics and saves everything to a CSV file for your records.

Add the following output code below the evaluation loop in compare_pipelines.py:

# Print formatted comparison table
print('\n{:<60} {:>6} {:>6} {:>12}'.format('Question', 'Naive', 'UA', 'Improvement'))
print('-' * 90)
for r in results:
    print('{:<60} {:>6.3f} {:>6.3f} {:>12.3f}'.format(r['question'][:58], r['naive_score'], r['ua_score'], r['improvement']))

# Print summary statistics by question type
simple = [r for r in results if r['type'] == 'simple']
complex_ = [r for r in results if r['type'] == 'complex']
print(f'\nSimple questions - Naive avg: {sum(r["naive_score"] for r in simple)/len(simple):.3f}, UA avg: {sum(r["ua_score"] for r in simple)/len(simple):.3f}')
print(f'Complex questions - Naive avg: {sum(r["naive_score"] for r in complex_)/len(complex_):.3f}, UA avg: {sum(r["ua_score"] for r in complex_)/len(complex_):.3f}')

# Save results to CSV for later analysis
os.makedirs('results', exist_ok=True)
with open('results/comparison_results.csv', 'w', newline='') as f:
    writer = csv.DictWriter(f, fieldnames=['question', 'type', 'naive_score', 'ua_score', 'improvement'])
    writer.writeheader()
    writer.writerows(results)
print('\nResults saved to results/comparison_results.csv')

What does this code do?

The table uses Python string formatting to align columns so you can scan results quickly.
Summary statistics split by question type reveal the key insight: simple questions may show similar scores, but complex questions should show dramatic improvement.
The CSV export gives you a permanent record in results/comparison_results.csv for your report.

Save compare_pipelines.py.

✔️ Awesome, I've got everything!

Great. Double-check you've saved compare_pipelines.py and that your edits to correction_loop.py and naive_rag.py are saved too.

ⓧ I'd like to double check the full code

Your compare_pipelines.py should contain three sections in order:

Imports and TEST_QUESTIONS list (5 simple, 5 complex dictionaries).
The evaluation loop that iterates over TEST_QUESTIONS, calls both pipelines, scores both answers, and appends to results.
The table printing, summary statistics, and CSV export block.

Scroll up and confirm each chunk is present in order. Make sure correction_loop.py has no print() lines in its node functions and no if __name__ block.

Before you run this, think: do you expect the simple questions to show improvement, or mostly the complex ones? Why?

Run the comparison script:

python compare_pipelines.py

What should I see?

The script takes a few minutes because it runs 10 questions through both pipelines (many API calls). You'll see "Evaluating: ..." printed for each question as it processes.

When it finishes, a formatted table appears showing each question's naive score, uncertainty-aware score, and improvement. Below the table, summary statistics reveal the pattern: simple questions show similar scores for both pipelines (naive RAG handles them fine), but complex questions show clear improvement from roughly 0.4 average to 0.75+ on the uncertainty-aware side.

The final line confirms your results are saved to results/comparison_results.csv.

Rate limits during batch evaluation

If you hit rate limit errors from the Gemini API, add a short delay between questions. Insert import time at the top of the file and time.sleep(2) inside the loop after each question finishes processing.

Seeing an ImportError?

Check that correction_loop.py no longer has Annotated in its typing import. The target import line should be from typing import TypedDict.
Make sure you removed the if __name__ == '__main__': blocks from both naive_rag.py and correction_loop.py.
Confirm your virtual environment is activated (you should see the environment name in your terminal prompt).

Help me debug this import error in my comparison script.

Confirm the CSV was created by running:

ls results/

You should see comparison_results.csv listed in the results/ directory.

You now have quantitative proof that your uncertainty-aware pipeline reduces hallucinations where it matters most. The comparison table tells a clear story: naive RAG handles simple factual questions fine, but fabricates details on complex cross-paper questions. Your correction loop catches those fabrications and fixes them.

Secret mission

Add RAGAS Evaluation for Publication-Ready Metrics

Your comparison table from Step 5 proves the system works, but a thesis reviewer wants standardized metrics. RAGAS is a widely-recognized RAG evaluation framework that produces per-question faithfulness, answer relevancy, and context precision scores. In this secret mission, you'll wire your 10 test questions into a RAGAS-compatible dataset and generate a publication-ready results table comparing both pipelines.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with a free-tier API key, so there are no ongoing costs.

Resources you used:

Project folder on your Desktop containing all Python files, the virtual environment, ChromaDB database, research papers, and results.
Google Gemini API key created in Google AI Studio.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend the project soon.

Your virtual environment and project files stay on your Desktop.
Your Gemini API key remains active on the free tier with no charges.
The ChromaDB database persists in the chroma_db/ folder, ready for future queries.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files stay intact so you can pick up where you left off.

If your virtual environment is still active, deactivate it by running:

deactivate

Close any terminal sessions running Python scripts.
Your project folder, ChromaDB database, and API key all remain in place for when you return.

Delete - I don't want to use this again

Remove all project resources and revoke your API key. This is a complete teardown.

Deactivate your virtual environment if it is still active:

deactivate

Delete the entire project folder from your Desktop by running:

cd ~/Desktop
rm -rf uncertainty-rag

Windows users

If you are on Windows, use this command instead in PowerShell:

cd ~\Desktop
Remove-Item -Recurse -Force uncertainty-rag

Delete your Gemini API key to prevent future misuse.
Open the Google Cloud API Credentials page.
Find the API key you used for this project and click the Actions icon next to it.
Select Delete API key.
In the Delete credential modal, click Delete to confirm.

Nice Work!

Nice work! You just built a self-correcting RAG system that detects its own hallucinations, rewrites its queries, and proves the improvement with hard numbers.

You've learned how to:

Build a LangGraph state machine that orchestrates a full detect-correct-verify loop, automatically catching hallucinated answers and re-retrieving grounded evidence.
Implement faithfulness scoring using an LLM-as-a-judge pattern that decomposes answers into individual claims and checks each one against retrieved context.
Prove measurable hallucination reduction through a side-by-side comparison of naive RAG vs. uncertainty-aware RAG across simple and complex questions.
Secret Mission: Generated publication-ready evaluation metrics using RAGAS to quantitatively compare both pipelines on faithfulness, answer relevancy, and context precision.

Ready to quiz yourself?

RAG Hallucination Detection System

Introduction

30 Second Summary

What You'll Build

Are there any prerequisites?

Before We Start

Before We Start...

Set Up Your Python Environment and API Access

In this step, get ready to:

Create your virtual environment and install packages

✔️ I see version 3.10 or higher

ⓧ I see a version lower than 3.10

ⓧ Command not found

macOS

Windows

Seeing a script execution error?

What are these packages?

Install taking forever or failing?

Set up your Gemini API key

Seeing a permissions or region error?

Why a .env file?

Verify your setup with a test script

What does this code do?

Seeing an API key error?

What does this code do?

Seeing device: cpu instead of cuda?

✔️ Awesome, I've got everything!

ⓧ I'd like to double check the full code

Build a Naive RAG Pipeline Over Research Papers

In this step, get ready to:

Download and ingest research papers into ChromaDB

Why these papers?

Downloads failing?

What does this code do?

Seeing an import error?

✔️ Awesome, I've got everything!

ⓧ I'd like to double check the full code

Build the naive retrieve-and-generate chain

What does this code do?

What does this code do?

✔️ Awesome, I've got everything!

ⓧ I'd like to double check the full code

Run the pipeline and observe hallucination

Why does Q2 hallucinate?

Getting a rate limit or API error?

Add Hallucination Detection with Faithfulness Scoring

In this step, get ready to:

Build the claim extraction and verification functions

What does this code do?

What does this function do?

Seeing an import error?

Add scoring logic with a confidence threshold

How does the scoring work?

Run the detector on the test questions

What does this test block do?

Why does this work?

Both questions showing PASS?

✔️ Awesome, I've got everything!

ⓧ I'd like to double check the full code

Build the Correction Loop with LangGraph

In this step, get ready to:

Define the state machine structure

What does this code do?

What is RAGState?

Seeing an import error?

Add the pipeline node functions

What do these nodes do?

How does query rewriting work?

What is the router function?

Wire the graph edges and run the correction loop

How does the graph flow?

What does uncertainty_aware_query do?

What if the status says DECLINED?

Seeing a recursion or timeout error?

✔️ Awesome, I've got everything!

ⓧ I'd like to double check the full code

Compare Naive vs. Uncertainty-Aware Pipeline

In this step, get ready to:

Create a test set of 10 questions

Why remove the prints and __main__ blocks?

Why remove the prints and main blocks?