Build a Corrective RAG System

Build a self-correcting RAG system that grades docs and falls back to web search.

Introduction

30 Second Summary

You ask your AI assistant a question, and it answers with total confidence. The problem is the answer is completely wrong. It pulled irrelevant context and hallucinated the rest.

In this project, you will build a Corrective RAG system that grades its own retrieval before generating an answer. When its internal documents fall short, it rewrites your query and searches the web for better context instead of guessing.

What You'll Build

You run a question through your system, watch it retrieve PDF chunks, grade each one for relevance, and either generate a grounded answer or pivot to web search. The full decision trace prints to your terminal so you can see exactly why each path fired.

By the end of this project, you'll have:

  • A PDF ingestion pipeline that loads documents into a local FAISS vector store using Ollama embeddings, so your system can retrieve relevant chunks from any PDF you drop in.
  • A document grading gate powered by LangGraph that evaluates every retrieved chunk before it reaches generation. You ask a question outside your PDF content and watch the system detect weak retrieval, rewrite the query, and fetch fresh context from the web.
  • A conditional state machine with the full execution trace visible in your terminal. You see exactly which nodes fired and why for every query.
  • Secret Mission: Add a hallucination grader that checks whether the generated answer is actually supported by the retrieved documents, and loops back to regenerate if not.

Are there any prerequisites?

This project assumes basic Python knowledge and familiarity with RAG concepts (retrieval + generation). You should also have Ollama installed on your machine already.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters. This step is your moment to commit to the goal and understand the problem you're solving.

Set Up the Project and Dependencies

Your Corrective RAG system relies on several moving parts. You need a Python project manager, AI and retrieval libraries, a local embedding model, and an API key for the cloud LLM.

In this step, you will install uv (a fast Python project manager), scaffold the crag-system project, pull all dependencies, and prepare the file structure your pipeline will use.

In this step, get ready to:
  • Install uv and create the crag-system project with all dependencies.
  • Pull the Ollama embedding model for local vector search.
  • Create the environment file and docs folder for your PDFs.
Install uv and create the project

You will run all commands in your macOS Terminal.

  • Press Cmd+Space to open your search bar.
  • Type Terminal and press Enter to open it.
  • Check if uv is already installed by running:
uv --version

✔️ I see a version number

uv is ready to go. Skip ahead to the next instruction below the tabs.

ⓧ Command not found

  • Install uv by running this command:
curl -LsSf https://astral.sh/uv/install.sh | sh
  • Close and reopen your Terminal so the new path takes effect.
  • Confirm the install worked by running:
uv --version

You should now see a version number printed.

Still seeing command not found?

Make sure you fully closed the old Terminal window and opened a fresh one. The installer adds uv to your shell path, but an already-open session will not pick it up.

help me troubleshoot uv installation on macOS

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project by running this command:
uv init crag-system

You should see a message like Initialized project `crag-system` confirming the project directory was created.

What did uv init create?

This command created a crag-system/ folder on your Desktop containing a pyproject.toml (project metadata), a .python-version file (pins the Python version uv detected on your system), and a README.md.

uv manages your virtual environment and lockfile automatically. You never activate a venv by hand.

  • Move into the project folder and install all dependencies by running these commands:
cd crag-system
uv add langgraph langchain-core langchain-ollama faiss-cpu pypdf ddgs langchain-openai pydantic

uv will resolve versions, create a .venv, and install packages. You should see lines like Resolved X packages and Installed X packages in the output.

What are these dependencies?

  • langgraph provides the state machine framework for building conditional AI workflows.
  • langchain-core and langchain-ollama connect your code to Ollama embeddings.
  • faiss-cpu is the local vector store for similarity search.
  • pypdf extracts text from PDF files.
  • ddgs provides DuckDuckGo web search for the fallback path.
  • langchain-openai lets you call OpenRouter (OpenAI-compatible API) for the LLM.
  • pydantic defines structured output schemas for document grading.

Seeing a resolution error?

If uv reports a conflict, make sure your .python-version file shows Python 3.10 or higher. LangGraph requires Python 3.10 or above.

help me fix a uv dependency resolution error

Pull the Ollama embedding model

Your CRAG system uses nomic-embed-text to turn PDF chunks into vectors locally. This model is only 274 MB and runs comfortably on 8 GB of RAM.

  • Download the embedding model by running:
ollama pull nomic-embed-text

You will see progress bars as Ollama downloads the model layers. The final line should read success.

  • Verify the model is available by running:
ollama list

You should see nomic-embed-text:latest in the list of installed models.

Pull hanging or failing?

Make sure the Ollama app is running. Look for the llama icon in your macOS menu bar. If it is not there, open Ollama from your Applications folder.

help me troubleshoot ollama pull

Create the environment file and docs folder

Your project needs an OpenRouter API key for LLM calls and a folder to hold the PDF documents you will ingest.

  • Create the .env file by running this command (replace your-api-key-here with your actual OpenRouter API key):
echo "OPENROUTER_API_KEY=your-api-key-here" > .env

Where do I find my OpenRouter API key?

Log in to your OpenRouter account and navigate to the Keys page. Copy the key you created during sign-up. The free tier gives you 50 requests per day, which is plenty for building and testing.

  • Create the docs folder and confirm both items exist by running:
mkdir docs
ls -a

You should see .env and docs/ listed alongside pyproject.toml and the other project files.

  • Add at least one PDF to the docs/ folder. You can copy any PDF you have on hand into the folder. For example, drag a file into the docs/ folder using Finder, or use a command like:
cp ~/Downloads/any-document.pdf docs/

Choose a PDF you can ask questions about

Pick a document with factual content you can later verify. A short whitepaper, a product manual, or lecture notes all work well. You will ask questions about this PDF in the next steps.

  • Confirm the PDF landed in the folder by running:
ls docs/

You should see your PDF file name listed.

Before you run the final check, what do you think will happen if LangGraph installed correctly?

  • Verify everything is wired together by running:
uv run python -c "from langgraph.graph import StateGraph; print('LangGraph ready')"

You should see LangGraph ready printed to the terminal. This confirms uv, Python, and all dependencies are working together.

Seeing an import error?

Make sure you are inside the crag-system/ directory when you run the command. The uv run command looks for pyproject.toml in the current folder to find the project environment.

help me debug my uv run import error

Your project environment is ready and all dependencies are in place. Next up, you will write the PDF ingestion pipeline that loads your documents into a searchable vector store.

Build the PDF Ingestion and Retrieval Pipeline

Your project has dependencies installed and an Ollama embedding model ready. Now you need actual documents inside a searchable vector store before any retrieval can happen.

In this step, you will load PDFs, split them into chunks, embed each chunk with your local Ollama model, and save the resulting vectors to a FAISS index on disk. Then you will query that index to prove retrieval works.

In this step, get ready to:
  • Write an ingestion script that loads PDFs, chunks them, and builds a FAISS vector store.
  • Write a retrieval test that loads the index and prints the top 4 matching chunks for a query.
Add the community integration package

The FAISS vector store wrapper and PDF loader live in the langchain-community package. The text splitter lives in langchain-text-splitters.

  • Add both packages by running:
uv add langchain-community langchain-text-splitters

Why a separate package?

LangChain splits integrations into dedicated packages. The langchain-community package contains the FAISS and PyPDFLoader wrappers that talk to the underlying faiss-cpu and pypdf libraries you already installed.

Write the ingestion script

This script will read every PDF in your docs/ folder, split each into overlapping chunks, embed them locally, and persist the index to disk.

  • Create a new file called ingest.py in the root of your crag-system folder and paste in the following imports and loader logic:
import os
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import FAISS

# Load all PDFs from the docs/ folder
docs = []
for filename in os.listdir("docs"):
    if filename.endswith(".pdf"):
        loader = PyPDFLoader(os.path.join("docs", filename))
        docs.extend(loader.load())

What does this code do?

The os.listdir("docs") call finds every file in your docs/ folder. For each PDF, PyPDFLoader uses the pypdf library to extract text page by page, returning a list of Document objects with content and metadata.

  • Below the loader code in ingest.py, add the chunking and embedding logic:
# Split documents into overlapping chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = text_splitter.split_documents(docs)
print(f"Split {len(docs)} pages into {len(chunks)} chunks")

# Embed chunks using local Ollama model and save to disk
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = FAISS.from_documents(chunks, embeddings)
vectorstore.save_local("faiss_index")
print("FAISS index saved to faiss_index/")

What does this code do?

RecursiveCharacterTextSplitter breaks each page into chunks of about 1000 characters, with 200 characters of overlap so context is not lost at chunk boundaries.

OllamaEmbeddings sends each chunk to your local nomic-embed-text model to produce a 768-dimension vector. FAISS.from_documents builds the index in memory, then save_local writes it to disk as two files inside faiss_index/.

  • Save ingest.py.
  • Run the ingestion script:
uv run python ingest.py

You should see output like:

Split 12 pages into 47 chunks
FAISS index saved to faiss_index/

The exact numbers depend on your PDF, but you should see both print statements confirming the split and save.

  • Confirm the index was written to disk by running:
ls faiss_index/

You should see index.faiss and index.pkl listed.

Not seeing the faiss_index folder?

  • Make sure Ollama is running. Open a separate terminal and run ollama serve if you stopped it.
  • Check that your docs/ folder contains at least one .pdf file.
  • If you see an import error, confirm you ran uv add langchain-community langchain-text-splitters earlier.

Need more help? Why is my ingest.py failing to create the FAISS index?

Write the retrieval test

Now that your vectors live on disk, you need to prove you can load them and search. A quick similarity search will confirm the pipeline works end to end.

  • Create a new file called main.py in the root of your crag-system folder and paste in the following code:
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import FAISS

# Load the FAISS index from disk
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = FAISS.load_local(
    "faiss_index",
    embeddings,
    allow_dangerous_deserialization=True
)

# Run a test similarity search
question = "What is this document about?"
results = vectorstore.similarity_search(question, k=4)

print(f"Query: {question}")
print(f"Found {len(results)} results:\n")
for i, doc in enumerate(results):
    print(f"--- Result {i+1} ---")
    print(doc.page_content[:200])
    print()

What does this code do?

FAISS.load_local reads the index from faiss_index/ back into memory. The allow_dangerous_deserialization=True flag is required because FAISS stores a pickle file. Since you created this file yourself, it is safe.

similarity_search embeds your question with the same Ollama model and finds the 4 closest chunks by vector distance. The results are printed with the first 200 characters of each chunk so you can verify the content is relevant.

  • Save main.py.

Before you run this, what do you expect to see? Will it return chunks from your PDF, or something else?

  • Run the retrieval test:
uv run python main.py

You should see your query printed, followed by 4 chunks of text pulled from your PDF. Each chunk shows the first 200 characters so you can confirm it comes from the document you loaded.

Getting zero results or an error?

  • If you see a file-not-found error, make sure you ran ingest.py first to create the faiss_index/ folder.
  • If you see a deserialization error, confirm allow_dangerous_deserialization=True is set in load_local.
  • If results are empty, check that ingest.py printed a non-zero chunk count.

Need more help? Why is my FAISS similarity search returning no results?

✔️ Awesome, I've got everything!

Great. Make sure you have saved both ingest.py and main.py.

ⓧ I'd like to double check the full code

Here is the complete ingest.py:

import os
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import FAISS

# Load all PDFs from the docs/ folder
docs = []
for filename in os.listdir("docs"):
    if filename.endswith(".pdf"):
        loader = PyPDFLoader(os.path.join("docs", filename))
        docs.extend(loader.load())

# Split documents into overlapping chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = text_splitter.split_documents(docs)
print(f"Split {len(docs)} pages into {len(chunks)} chunks")

# Embed chunks using local Ollama model and save to disk
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = FAISS.from_documents(chunks, embeddings)
vectorstore.save_local("faiss_index")
print("FAISS index saved to faiss_index/")

Here is the complete main.py:

from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import FAISS

# Load the FAISS index from disk
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = FAISS.load_local(
    "faiss_index",
    embeddings,
    allow_dangerous_deserialization=True
)

# Run a test similarity search
question = "What is this document about?"
results = vectorstore.similarity_search(question, k=4)

print(f"Query: {question}")
print(f"Found {len(results)} results:\n")
for i, doc in enumerate(results):
    print(f"--- Result {i+1} ---")
    print(doc.page_content[:200])
    print()

Your PDFs are now searchable vectors on disk. Next up, you will wire an LLM to these results and build a naive RAG chain that retrieves and generates answers.

Build the Naive RAG Chain and See It Fail

In the last step, you loaded PDFs into a FAISS vector store and confirmed retrieval works. Chunks come back when you ask a question. But raw chunks are not answers.

You need a language model to read those chunks and generate a response. In this step, you will wire up a simple retrieve-then-generate pipeline. This is naive RAG. It trusts whatever it retrieves and generates blindly from it. You will see exactly why that is a problem.

In this step, get ready to:
  • Connect to an LLM via OpenRouter's free tier and build a retrieve-then-generate chain.
  • Test the chain with a question your PDFs can answer and confirm it works.
  • Test the chain with a question your PDFs cannot answer and observe a confident hallucination.
Wire up the OpenRouter LLM and build the retrieval chain

Your retriever pulls chunks from the vector store, but right now they just print to the terminal. You need an LLM to synthesize those chunks into an actual answer.

  • Open main.py in your editor.
  • Replace the entire contents of the file with the following imports and LLM configuration:
import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

# Load environment variables from .env
load_dotenv()

# Point ChatOpenAI at OpenRouter's free tier
llm = ChatOpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.getenv("OPENROUTER_API_KEY"),
    model="meta-llama/llama-3.2-3b-instruct:free",
)

# Load the FAISS index from disk
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = FAISS.load_local(
    "faiss_index", embeddings, allow_dangerous_deserialization=True
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

What does this code do?

  • load_dotenv() reads your .env file so the OpenRouter API key is available as an environment variable.
  • ChatOpenAI is LangChain's wrapper for OpenAI-compatible chat APIs. By setting base_url to OpenRouter's endpoint, you route requests there instead of to OpenAI.
  • The :free suffix on the model ID tells OpenRouter to use the free variant (50 requests per day at no cost).
  • The retriever loads your existing FAISS index and returns the top 4 most similar chunks for any query.
  • Below the retriever line, add the prompt template and the ask function that ties everything together:
# Define a simple RAG prompt
prompt = ChatPromptTemplate.from_template(
    """Answer the question based only on the following context:

{context}

Question: {question}

Answer:"""
)

# Build the naive retrieve-then-generate chain
def ask(question):
    docs = retriever.invoke(question)
    context = "\n\n".join(doc.page_content for doc in docs)
    chain = prompt | llm | StrOutputParser()
    answer = chain.invoke({"context": context, "question": question})
    print(f"\nQuestion: {question}")
    print(f"Answer: {answer}")
    print(f"\nRetrieved {len(docs)} chunks")
    return answer

# Test with a question the PDF can answer
ask("What is the main topic of this document?")

What does the ask function do?

The ask function is your entire naive RAG pipeline in one place. It retrieves the top 4 chunks, stuffs them into the prompt template as context, sends that prompt to the LLM, and prints the answer. There is no quality check on whether those chunks are actually relevant to the question.

Test with an in-scope question

Your chain is wired up. Before testing its limits, confirm it works when retrieval is good.

  • Save main.py and run it:
uv run python main.py

You should see a relevant answer drawn from your PDF content, along with a count of 4 retrieved chunks.

Seeing an authentication error?

  • Check that your .env file contains OPENROUTER_API_KEY with a valid key.
  • Make sure python-dotenv is available. If you see an import error for dotenv, run uv add python-dotenv.
  • Confirm Ollama is running so the embeddings can load. Run ollama list to check.

Help me debug an OpenRouter authentication error in my CRAG project.

Test with an out-of-scope question and observe the hallucination

The chain works when retrieval is good. Now you will push it outside the boundary of your PDFs.

  • In main.py, find this line at the bottom:
ask("What is the main topic of this document?")
  • Add a second test call below it that your PDFs definitely cannot answer:
# Test with a question the PDF CANNOT answer
ask("What were the key causes of the 2008 financial crisis?")

Before you run this, think about what you expect to happen. The retriever will still return 4 chunks from your PDF. Will the LLM refuse to answer, or will it try anyway?

  • Save main.py and run it again:
uv run python main.py

Look at the answer for the second question. The LLM produces a confident response about the 2008 financial crisis, even though the retrieved chunks have nothing to do with that topic. It fabricates an answer from irrelevant context.

Why does this happen?

This is the core problem with naive RAG. The retriever always returns something because similarity search finds the least-dissimilar chunks, not truly relevant ones. The LLM then trusts those chunks blindly and generates an answer that sounds authoritative but is grounded in irrelevant text.

In a production system, this means users get confident wrong answers with no warning. The system never says "I don't know" because it never checks whether its retrieval was actually good.

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py.

ⓧ I'd like to double check the full code

Your main.py should contain these sections in order. Scroll up and confirm each one is present:

  • Imports (7 lines): os, load_dotenv, ChatOpenAI, OllamaEmbeddings, FAISS, ChatPromptTemplate, StrOutputParser.
  • Environment and LLM config: load_dotenv() call, then ChatOpenAI pointed at https://openrouter.ai/api/v1 with the free model.
  • FAISS loading: OllamaEmbeddings, FAISS.load_local, and as_retriever with k=4.
  • Prompt template: a ChatPromptTemplate with {context} and {question} placeholders.
  • The ask function: retrieves docs, joins content, pipes through the chain, and prints the question, answer, and chunk count.
  • Two test calls: one in-scope question and one out-of-scope question about the 2008 financial crisis.

You have now seen naive RAG fail firsthand. The system produces a confident hallucination from irrelevant chunks. In the next step, you will add document grading with LangGraph so the system can detect bad retrieval before it reaches the LLM.

Add Document Grading with LangGraph

In the last step, you saw the core problem with naive RAG: it trusts retrieval blindly. When you asked a question outside your PDF content, the system retrieved irrelevant chunks and hallucinated a confident answer from them.

Document grading adds a quality gate between retrieval and generation. The LLM evaluates whether each retrieved chunk actually answers the question before passing it to the generation stage. This is where LangGraph enters the picture.

In this step, get ready to:
  • Define the LangGraph state and a Pydantic grading model.
  • Write a document grading node that scores each chunk for relevance.
  • Build a partial graph and run it to see the grader catch irrelevant documents.
Define the graph state and grading schema

A LangGraph state machine needs a typed state object that flows between nodes. Each node reads from and writes to this shared state.

  • Create a new file called graph.py in your crag-system project folder.
  • Add the imports, state definition, and grading schema by pasting this code:
import os
from typing import TypedDict
from pydantic import BaseModel, Field
from langchain_ollama import OllamaEmbeddings
from langchain_openai import ChatOpenAI
from langchain_community.vectorstores import FAISS
from langgraph.graph import StateGraph, START, END
from dotenv import load_dotenv

load_dotenv()

# Define the state that flows through the graph
class GraphState(TypedDict):
    question: str
    documents: list
    generation: str
    web_search: str

# Define structured output for document grading
class GradeDocuments(BaseModel):
    """Binary relevance score for a retrieved document."""
    score: str = Field(description="Relevance: 'relevant' or 'irrelevant'")

What does this code do?

GraphState is a TypedDict that defines the data flowing through your graph. Every node can read and update these four fields: the user's question, the retrieved documents, the generated answer, and whether web search is needed.

GradeDocuments is a Pydantic model that constrains the LLM to output only 'relevant' or 'irrelevant'. This is called structured output. It prevents the grader from rambling and keeps your routing logic simple.

  • Save graph.py.
Write the retrieve and grade_documents nodes

Now you need the two functions that will become nodes in your graph. The retrieve node pulls chunks from your FAISS index. The grade_documents node asks the LLM to judge each chunk.

  • Add the LLM setup, grader prompt, and retrieve function below your GradeDocuments class in graph.py:
# Set up the LLM via OpenRouter
llm = ChatOpenAI(
    model="meta-llama/llama-3.2-3b-instruct:free",
    base_url="https://openrouter.ai/api/v1",
    api_key=os.getenv("OPENROUTER_API_KEY"),
)

# Create a structured grader that returns GradeDocuments
structured_llm_grader = llm.with_structured_output(GradeDocuments)

# System prompt for the document grader
grader_prompt = (
    "You are a grader assessing relevance of a retrieved document to a user question.\n"
    "If the document contains keyword(s) or semantic meaning related to the question, "
    "grade it as relevant.\n"
    "Give a binary score 'relevant' or 'irrelevant' to indicate whether the document "
    "is relevant to the question."
)

def retrieve(state):
    """Retrieve documents from the FAISS index."""
    question = state["question"]
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vectorstore = FAISS.load_local(
        "faiss_index", embeddings, allow_dangerous_deserialization=True
    )
    documents = vectorstore.similarity_search(question, k=4)
    return {"documents": documents, "question": question}

What does this code do?

The llm variable creates a ChatOpenAI instance pointed at OpenRouter's free tier. Calling .with_structured_output(GradeDocuments) wraps the LLM so it always returns a validated GradeDocuments object.

The retrieve function is a LangGraph node. It receives the current state, loads your FAISS vector store, runs a similarity search for the top 4 chunks, and returns them in the state.

  • Now add the grade_documents node below the retrieve function in graph.py:
def grade_documents(state):
    """Grade each retrieved document for relevance."""
    question = state["question"]
    documents = state["documents"]

    relevant_docs = []
    web_search = "no"

    for doc in documents:
        # Ask the LLM to grade this document
        result = structured_llm_grader.invoke(
            [
                {"role": "system", "content": grader_prompt},
                {"role": "user", "content": f"Document:\n{doc.page_content}\n\nQuestion: {question}"},
            ]
        )
        grade = result.score
        if grade == "relevant":
            print(f"  [RELEVANT] {doc.page_content[:50]}...")
            relevant_docs.append(doc)
        else:
            print(f"  [IRRELEVANT] {doc.page_content[:50]}...")
            web_search = "yes"

    return {"documents": relevant_docs, "question": question, "web_search": web_search}

What does this code do?

This node loops through every retrieved document and asks the LLM: "Is this chunk relevant to the question?" The structured output forces the LLM to respond with either 'relevant' or 'irrelevant'.

Relevant documents are kept for generation. If even one chunk is irrelevant, the node sets web_search = "yes" in the state. This flag tells the graph to route to web search instead of generating from weak context.

  • Save graph.py.
Build the partial graph and test it

With both nodes written, you can wire them into a LangGraph state machine and test the grading on the same off-topic question that hallucinated in Step 3.

  • Add the graph wiring and test block at the bottom of graph.py:
# Build the partial graph
workflow = StateGraph(GraphState)

# Add nodes
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade_documents", grade_documents)

# Connect: START -> retrieve -> grade_documents -> END
workflow.add_edge(START, "retrieve")
workflow.add_edge("retrieve", "grade_documents")
workflow.add_edge("grade_documents", END)

# Compile the graph
app = workflow.compile()

# Test with an off-topic question
if __name__ == "__main__":
    print("\nGrading documents for an off-topic question...\n")
    result = app.invoke({"question": "What is the capital of France?"})
    print(f"\nWeb search needed: {result['web_search']}")
    print(f"Relevant documents kept: {len(result['documents'])}")

What does this code do?

StateGraph(GraphState) creates a new state machine using your typed state. You add two nodes and connect them with edges so data flows from retrieve to grade_documents.

The if __name__ block runs the graph with a question your PDFs cannot answer. The grader should mark all chunks as irrelevant and set web_search to "yes".

  • Save graph.py.

Before you run this, what do you think the grader will output for each chunk when the question is completely unrelated to your PDFs?

  • Run the graph by executing this command:
uv run python graph.py

You should see each chunk printed with [IRRELEVANT] next to it, followed by Web search needed: yes and Relevant documents kept: 0. The quality gate catches bad retrieval instead of passing it blindly to generation.

Not seeing the expected output?

  • If you see a connection error or 401, confirm your OPENROUTER_API_KEY is set correctly in .env.
  • If you see FileNotFoundError for faiss_index, make sure you ran ingest.py successfully in the previous step and the faiss_index/ folder exists.
  • If some chunks come back as [RELEVANT], try a more obviously off-topic question like "What is the recipe for chocolate cake?" to confirm the grader works. Free-tier models occasionally misjudge borderline cases.

Still stuck? Help me debug my document grading output

✔️ Awesome, I've got everything!

Great. Double check you have saved graph.py with all three sections (state/schema, nodes, and graph wiring).

ⓧ I'd like to double check the full code

Your graph.py has four main sections you built above. Scroll through your file and confirm they appear in this order:

  • Imports, load_dotenv(), GraphState TypedDict, and GradeDocuments Pydantic model.
  • LLM setup (llm, structured_llm_grader, grader_prompt) and the retrieve function.
  • The grade_documents function.
  • Graph wiring (StateGraph, add_node, add_edge, compile) and the if __name__ test block.

Your quality gate is working. The grader catches the same bad retrieval that caused hallucination in Step 3. Next up, you will build the web search fallback and conditional routing so the system has somewhere better to go when internal docs fail.

Add Web Search Fallback and Conditional Routing

In Step 4, you built a quality gate that catches irrelevant retrieval. But catching bad retrieval is only half the battle. Right now, the system flags poor chunks and stops. It has nowhere better to go.

The web search fallback completes the CRAG loop. When internal documents fail, the system rewrites the query for the open web, retrieves fresh context from DuckDuckGo, and routes that context to generation. This is the full corrective flow: retrieve, grade, route, generate.

In this step, get ready to:
  • Write a query rewriting node and a web search node using DuckDuckGo.
  • Write a generate node that produces answers from surviving documents.
  • Wire the full LangGraph with conditional routing and test end-to-end.
Write the query rewrite and web search nodes

When retrieval fails, you need two things: a better query shaped for web search, and a way to actually search the web. These are two separate nodes in the graph.

  • In graph.py, add the following import at the top of the file, below your existing imports:
from ddgs import DDGS
from langchain_core.documents import Document

What are these imports?

DDGS is the DuckDuckGo search client from the ddgs package. It lets you run web searches programmatically and get back results with titles, URLs, and body text.

Document is LangChain's standard document class. You'll wrap web search results in this format so the generate node can handle them the same way it handles your PDF chunks.

  • Below your grade_documents function, add the transform_query node:
def transform_query(state):
    """Rewrite the question for better web search results."""
    print("---TRANSFORM QUERY---")
    question = state["question"]

    # Prompt the LLM to rewrite the question for web search
    msg = llm.invoke(
        f"Rewrite this question to be a better web search query. "
        f"Return only the rewritten query, nothing else: {question}"
    )
    return {"question": msg.content}

What does this code do?

  • The node takes the original question from state and asks the LLM to rephrase it as a search-engine-friendly query.
  • It returns the rewritten question back into state, replacing the original. This means downstream nodes (web search, generation) use the improved query.
  • The print statement makes the execution path visible in your terminal so you can trace the flow.
  • Directly below transform_query, add the web_search node:
def web_search(state):
    """Search DuckDuckGo and return results as documents."""
    print("---WEB SEARCH---")
    question = state["question"]

    # Search the web using DuckDuckGo
    results = DDGS().text(question, max_results=3)

    # Convert search results into Document objects
    web_docs = [
        Document(page_content=r["body"], metadata={"source": r["href"]})
        for r in results
    ]
    return {"documents": web_docs}

What does this code do?

  • DDGS().text(question, max_results=3) sends the rewritten query to DuckDuckGo and returns up to 3 results. Each result is a dictionary with title, href, and body keys.
  • The list comprehension wraps each result's body text in a LangChain Document object, storing the URL in metadata. This makes web results interchangeable with your PDF chunks.
  • Save graph.py.
  • Verify the imports work by running:
uv run python -c "from ddgs import DDGS; print(DDGS().text('test', max_results=1))"

You should see a list with one dictionary containing a title, href, and body from a web search result.

Seeing a connection error?

  • Make sure you have an internet connection. The ddgs package queries external search engines.
  • If you see a timeout, try running the command again. Search backends occasionally take a moment to respond.

Why isn't my DuckDuckGo search returning results?

Write the generate node

The generate node is the final destination in the graph. It takes whatever documents survived the pipeline (either graded PDF chunks or web search results) and produces a natural language answer.

  • In graph.py, add the generate function below your web_search node:
def generate(state):
    """Generate an answer using the retrieved documents."""
    print("---GENERATE---")
    question = state["question"]
    documents = state["documents"]

    # Format documents into a single context string
    docs_text = "\n\n".join(doc.page_content for doc in documents)

    # Build the prompt with context and question
    prompt = (
        f"Answer the following question based on the provided context.\n\n"
        f"Context:\n{docs_text}\n\n"
        f"Question: {question}\n\n"
        f"Answer:"
    )

    # Generate the answer via OpenRouter
    msg = llm.invoke(prompt)
    return {"generation": msg.content}

What does this code do?

  • It joins all document contents into a single context string. These documents could be PDF chunks (if grading passed) or web search results (if grading failed and the system fell back).
  • The prompt template instructs the LLM to answer based only on the provided context. This keeps the answer grounded in retrieved evidence.
  • The result goes into state as generation, which is the final output of the graph.
  • Save graph.py.
Wire the full graph with conditional routing

You have all five nodes: retrieve, grade_documents, transform_query, web_search, and generate. Now you need to connect them with conditional logic so the graph routes to web search only when grading flags poor retrieval.

  • In graph.py, find the section where you built and compiled your partial graph (the lines with StateGraph, add_node, add_edge, and compile). Replace that entire section with the following routing function and full graph:
def decide_to_generate(state):
    """Route based on whether web search is needed."""
    if state["web_search"] == "yes":
        print("---ROUTING TO WEB SEARCH---")
        return "transform_query"
    else:
        print("---ROUTING TO GENERATE---")
        return "generate"

What does this routing function do?

This is the decision point of the entire CRAG system. After grade_documents runs, it checks the web_search field in state. If any document was graded irrelevant, that field is "yes" and the graph routes through query rewriting and web search. Otherwise it goes straight to generation with the good PDF chunks.

  • Below the routing function, add the full graph assembly and test invocation:
# Build the full CRAG graph
workflow = StateGraph(GraphState)

# Add all nodes
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade_documents", grade_documents)
workflow.add_node("transform_query", transform_query)
workflow.add_node("web_search", web_search)
workflow.add_node("generate", generate)

# Connect the edges
workflow.add_edge(START, "retrieve")
workflow.add_edge("retrieve", "grade_documents")

# Conditional edge: grade_documents decides next step
workflow.add_conditional_edges(
    "grade_documents",
    decide_to_generate,
    {
        "transform_query": "transform_query",
        "generate": "generate",
    },
)
workflow.add_edge("transform_query", "web_search")
workflow.add_edge("web_search", "generate")
workflow.add_edge("generate", END)

# Compile the graph
app = workflow.compile()

What does this graph structure do?

  • Every query starts at retrieve (pulls chunks from FAISS) then flows to grade_documents (scores each chunk).
  • add_conditional_edges calls decide_to_generate after grading. The returned string ("transform_query" or "generate") tells LangGraph which node to visit next.
  • The web search path is linear: rewrite the query, search, then generate. Both paths converge at generate and then reach END.
  • Finally, add the test invocation at the bottom of the file:
# Test with a question outside your PDF content
result = app.invoke({"question": "What is the capital of France?"})
print("\n--- FINAL ANSWER ---")
print(result["generation"])
  • Save graph.py.

Before you run this, think about what you expect to see. The question "What is the capital of France?" is almost certainly not in your PDF documents. What path do you think the graph will take?

  • Run the full CRAG system:
uv run python graph.py

You should see the full execution trace in your terminal:

  • ---RETRIEVE--- (pulls chunks from FAISS).
  • ---GRADE DOCUMENT--- repeated for each chunk, each graded irrelevant.
  • ---ROUTING TO WEB SEARCH--- (conditional edge fires).
  • ---TRANSFORM QUERY--- (rewrites the question).
  • ---WEB SEARCH--- (hits DuckDuckGo).
  • ---GENERATE--- (produces the answer from web results).
  • --- FINAL ANSWER --- followed by a correct, web-grounded answer about Paris.

Seeing a rate limit or empty answer?

  • OpenRouter's free tier allows 50 requests per day. Each CRAG run uses 2-4 LLM calls (grading, rewriting, generating). If you have already run many tests today, you may hit the limit. Wait until tomorrow or check your usage at openrouter.ai/activity.
  • If web search returns empty results, check your internet connection. The ddgs package requires network access.
  • If you see a KeyError on web_search in state, make sure your GraphState TypedDict includes the web_search field as a str.

Why is my LangGraph conditional edge not routing correctly?

✔️ Awesome, I've got everything!

Great. Double check you have saved graph.py and your terminal shows the full web-search-fallback trace.

ⓧ I'd like to double check the full code

Your graph.py is now fairly long. Scroll through it and confirm you have these sections in order:

  • Imports (including DDGS, Document, StateGraph, START, END).
  • GraphState TypedDict and GradeDocuments Pydantic model.
  • LLM initialization (ChatOpenAI pointed at OpenRouter).
  • retrieve function.
  • grade_documents function.
  • transform_query function.
  • web_search function.
  • generate function.
  • decide_to_generate routing function.
  • Full graph build (StateGraph, nodes, edges, conditional edges, compile).
  • Test invocation with app.invoke(...).

You have a fully functional Corrective RAG system. It retrieves, grades, conditionally routes to web search, and generates grounded answers. Next up: the Secret Mission challenges you to add a hallucination grader that checks whether the generated answer is actually supported by the source documents.

Secret mission

Add a Hallucination Grader

Even with good retrieval, the LLM can still hallucinate. A hallucination grader checks whether the generated answer is actually supported by the documents, and regenerates if not. You are adding a loop to the graph. LangGraph conditional edges can point backward, creating a retry cycle. This is how production systems implement self-correction.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • crag-system/ project directory (source code, FAISS index, virtual environment, .env file).
  • Ollama nomic-embed-text model weights on disk.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your CRAG system.

  • Your project files stay on your Desktop in the crag-system/ folder.
  • The nomic-embed-text model remains available for Ollama to use in other projects.
  • Your OpenRouter free-tier account stays active (50 requests per day, no charges).

Pause - I'll come back to this later

Everything in this project is local and free. There are no running processes to pause or cloud resources to stop.

  • Your files and FAISS index persist on disk until you delete them manually.
  • Ollama does not run continuously. It only activates when you call it.

Delete - I don't want to use this again

Remove all project resources and reclaim disk space.

  • Delete the project directory by running:
cd ~/Desktop
rm -rf crag-system/

On Windows, run this instead:

cd ~\Desktop
rmdir /s /q crag-system
  • Remove the Ollama embedding model to free disk space by running:
ollama rm nomic-embed-text

You should see a confirmation that the model was deleted.

That's a Wrap!

That's a Wrap!

Nice work! You just built a Corrective RAG system that knows when its retrieval falls short and automatically routes to the web for better context.

You've learned how to:

  • Built a Corrective RAG system that grades documents for relevance before generation, catching bad retrieval instead of blindly trusting it.
  • Designed a LangGraph state machine with conditional edges that route between internal docs and web search based on grading results.
  • Implemented a web search fallback with DuckDuckGo so the system handles questions outside your document corpus by rewriting the query and fetching fresh context.
  • Secret Mission: Added a hallucination grader that creates a self-correction loop, checking whether the generated answer is grounded in source documents and regenerating if not.

Ready to quiz yourself?