Build an Agentic RAG System

Build a self-correcting RAG system with LangGraph and compare 3 LLM providers.

Introduction

30 Second Summary

You know that frustrating moment when you ask a chatbot a specific question and it confidently gives you a completely wrong answer? Basic chatbots retrieve whatever documents seem vaguely related and then generate a response, even if those documents have nothing to do with your question.

In this project, you will build an Agentic RAG system that actually checks whether retrieved documents are relevant before answering. If retrieval quality is poor, it rewrites the query and tries again. You will then compare how three different AI providers handle the same questions.

What You'll Build

A business knowledge base assistant powered by a LangGraph state machine that grades retrieval quality, rewrites queries when results are poor, and compares answers across three LLM providers.

By the end of this project, you'll have:

  • A document ingestion pipeline that chunks and embeds markdown files into ChromaDB.
  • An agentic RAG graph with retrieve, grade, rewrite, and generate nodes that self-corrects when retrieval quality is low.
  • A multi-provider comparison that runs identical queries through OpenAI, Anthropic, and Ollama to measure quality, speed, and cost differences.
  • Secret Mission: Add a hallucination grading node that checks whether generated answers are actually supported by retrieved documents.

Are there any prerequisites?

You should have basic Python familiarity. You will also need an OpenAI API key, an Anthropic API key, and an Ollama account with access to the nemotron-3-super:cloud model.

Step 1 walks you through installing Python 3.12 and verifying your setup, so you do not need anything configured beyond those keys and Ollama before starting.

Set Up Your Environment

Your Agentic RAG system depends on several Python libraries that have strict compatibility requirements. ChromaDB's Rust bindings crash on Python 3.14 due to a PyO3 ABI incompatibility. LangGraph also does not yet support Python 3.14.

In this step, you'll install Python 3.12 specifically for this project, create an isolated virtual environment, install all dependencies, connect your API keys, verify your local Ollama model, and create the sample documents your RAG system will search.

In this step, get ready to:
  • Install Python 3.12 and create a virtual environment with all project dependencies.
  • Verify Ollama is signed in and can access the nemotron-3-super cloud model.
  • Create sample business documents for the knowledge base.
Install Python 3.12 and set up the project

You likely already have Python 3.14 on your machine, but this project requires 3.12 for full library compatibility.

Why Python 3.12 instead of 3.14?

ChromaDB's Rust bindings crash with a SIGSEGV on Python 3.14 due to a PyO3 ABI incompatibility. LangGraph's CLI also has unresolved Python 3.14 issues.

Python 3.12 is the safest choice across the entire dependency stack for this project.

  • Check if Python 3.12 is already installed by running this command:

macOS

python3.12 --version

Windows

py -3.12 --version

✔️ I see Python 3.12.x

Python 3.12 is ready. Continue to the next instruction below.

ⓧ I see a different version

You have a different Python version (like 3.13 or 3.14) but need 3.12 specifically. You can install 3.12 alongside your existing version.

  • Go to the Python 3.12 downloads page.
  • Download the installer for your operating system (macOS 64-bit universal2 installer or Windows 64-bit installer).
  • Run the installer and follow the prompts. On Windows, make sure to check the box to add Python to your PATH.
  • After installation completes, close and reopen your terminal, then run the version check command again to confirm it shows 3.12.

ⓧ Command not found

Python 3.12 is not installed on your machine. You need to install it.

  • Go to the Python 3.12 downloads page.
  • Download the installer for your operating system (macOS 64-bit universal2 installer or Windows 64-bit installer).
  • Run the installer and follow the prompts. On Windows, make sure to check the box to add Python to your PATH.
  • After installation completes, close and reopen your terminal, then run the version check command again to confirm it works.
  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir agentic-rag
cd agentic-rag
  • Create a Python 3.12 virtual environment by running:

macOS

python3.12 -m venv .venv

Windows

py -3.12 -m venv .venv

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. It keeps this project's dependencies separate from your system Python and any other projects.

The .venv folder holds a private copy of the Python interpreter and all packages you install.

  • Activate the virtual environment:

macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate

You should see (.venv) appear at the beginning of your terminal prompt. This confirms you are working inside the virtual environment.

Don't see (.venv) in your prompt?

On macOS, make sure you typed source before the path. On Windows, you may need to allow scripts to run first.

Run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned in PowerShell, then try activating again.

Help me activate my Python virtual environment.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Cursor and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the agentic-rag folder, then click Open.
  • Create a new file called requirements.txt by clicking the new file icon in the Explorer sidebar, then paste the following content:
langchain==1.3.11
langgraph==1.2.8
langchain-openai==1.3.3
langchain-anthropic==1.4.8
langchain-ollama==1.1.0
langchain-chroma==1.1.0
langchain-text-splitters==1.1.2
python-dotenv==1.2.2

What does this file do?

The requirements.txt file pins exact package versions so every developer gets the same setup. It includes LangChain for the AI framework, LangGraph for the stateful graph execution, provider integrations for OpenAI/Anthropic/Ollama, ChromaDB for the vector store, and python-dotenv for loading API keys from a .env file.

  • Save requirements.txt.
  • Install all dependencies by running this command in your terminal (make sure (.venv) is still showing in your prompt):
pip install -r requirements.txt

This downloads and installs all eight packages and their dependencies. It may take a minute or two.

Seeing installation errors?

Make sure your virtual environment is active (you see (.venv) in your prompt). If you see a permissions error, do not use sudo. Deactivate and reactivate your venv instead.

If ChromaDB fails to build, confirm you are using Python 3.12 by running python --version inside the venv. It should show 3.12.x.

Help me fix my pip install errors.

  • Create a new file called .env in the root of your agentic-rag folder. Paste the following, replacing the placeholder values with your actual API keys:
OPENAI_API_KEY=your-openai-api-key-here
ANTHROPIC_API_KEY=your-anthropic-api-key-here

Where do I find my API keys?

Your OpenAI key is at platform.openai.com under API Keys. Your Anthropic key is at console.anthropic.com under API Keys.

Replace your-openai-api-key-here and your-anthropic-api-key-here with the actual key strings from those dashboards.

  • Save .env.
  • Create a new file called .env.example and paste this content (this serves as a reference for anyone who clones your repo):
OPENAI_API_KEY=your-openai-api-key-here
ANTHROPIC_API_KEY=your-anthropic-api-key-here
  • Save .env.example.
  • Create a new file called .gitignore and paste:
.venv/
.env
chroma_db/
__pycache__/
*.pyc

What does .gitignore do?

This file tells Git to ignore certain files when committing. Your .env file (with real API keys) stays private.

The .venv/ and chroma_db/ folders are large generated directories that do not belong in version control.

  • Save .gitignore.
  • Verify all packages installed correctly by running:
python -c "import langchain; import langgraph; import langchain_openai; import langchain_anthropic; import langchain_ollama; import langchain_chroma; import langchain_text_splitters; import dotenv; print('All packages imported successfully!')"

You should see All packages imported successfully! printed with no errors.

Seeing an ImportError?

Run which python (macOS) or where python (Windows) to confirm the path points inside your .venv folder. If it points elsewhere, reactivate your virtual environment and run pip install -r requirements.txt again.

Help me fix my import errors.

  • Initialize a Git repository and make your first commit by running:
git init
git add .
git commit -m "Initial project setup with dependencies"
  • Go to github.com/new in your browser to create a new repository.
  • Name it agentic-rag and leave it as Public.
  • Click Create repository (do not add a README or .gitignore from GitHub).
  • Push your local code to GitHub by running these commands (replace the URL with your own):
git remote add origin [[REPO_URL="https://github.com/YOUR_USERNAME/agentic-rag.git"]]
git branch -M main
git push -u origin main

You should see your files appear on GitHub when you refresh the repository page.

Verify Ollama is signed in with your cloud model

Your multi-provider comparison will send queries to Ollama's cloud infrastructure using the nemotron-3-super model. Let's confirm your Ollama account is signed in and the model is accessible.

  • Confirm Ollama is running by sending a request to its local API:
curl http://localhost:11434/api/tags

You should see a JSON response containing a models array. Look for nemotron-3-super in the list of model names. If you haven't signed in yet, run ollama signin first.

Connection refused or no nemotron-3-super?

If you see "connection refused", Ollama is not running. Open a new terminal window and start it with ollama serve.

If Ollama is running but nemotron-3-super is missing from the list, sign in to your Ollama account by running ollama signin, then pull the model with ollama pull nemotron-3-super. This is a cloud model so it does not download to your machine.

Help me get Ollama running with nemotron-3-super.

  • Test that the model responds to a query by running:
curl http://localhost:11434/api/generate -d '{"model": "nemotron-3-super", "prompt": "Say hello in one sentence.", "stream": false}'

You should see a JSON response with a response field containing a greeting. This confirms the model is loaded and generating text.

Create sample business documents for the knowledge base

Your RAG system needs documents to search through. You will create a fictional company's knowledge base with three markdown files covering different business topics.

  • Create a documents folder inside your agentic-rag project by running:
mkdir documents
  • Create a new file at documents/company_overview.md in Cursor's Explorer sidebar and paste:
# NovaTech Solutions - Company Overview

NovaTech Solutions is a B2B software company founded in 2019, specializing in workflow automation for mid-market enterprises. Headquartered in Austin, Texas, with offices in London and Singapore.

## Mission

Empower businesses to eliminate repetitive manual processes through intelligent automation, freeing teams to focus on strategic work.

## Key Facts

- Founded: 2019
- Employees: 320
- Customers: 1,200+ companies across 40 countries
- Annual Revenue: $45M (2025)
- Primary Industries: Healthcare, Financial Services, Manufacturing, Legal

## Leadership

- CEO: Sarah Chen (previously VP Engineering at Salesforce)
- CTO: Marcus Williams (co-founder, PhD in Distributed Systems)
- VP Sales: David Park (previously at HubSpot)

## Company Values

1. Customer outcomes over feature counts
2. Transparency in pricing and roadmap
3. Security is non-negotiable
4. Build for scale, deploy for simplicity
  • Save documents/company_overview.md.
  • Create a new file at documents/product_catalog.md and paste:
# NovaTech Product Catalog

## FlowEngine Pro

Our flagship workflow automation platform for teams of 10-50 people.

- Visual drag-and-drop workflow builder
- 200+ pre-built integrations (Slack, Jira, Salesforce, etc.)
- Conditional logic and branching
- Audit trail and compliance logging
- Pricing: $29/user/month (annual) or $39/user/month (monthly)
- Free trial: 14 days, no credit card required

## FlowEngine Enterprise

For organizations with 50-500+ employees needing advanced governance.

- Everything in Pro, plus:
- SSO/SAML authentication
- Role-based access control (RBAC)
- Custom API rate limits
- Dedicated account manager
- 99.9% SLA with financial credits
- Pricing: Custom pricing, typically $49-79/user/month depending on volume
- Minimum commitment: 50 seats, annual contract

## FlowEngine Connect

Lightweight integration layer for developers who want API-first automation.

- RESTful API with webhooks
- SDK available in Python, JavaScript, and Go
- 50,000 API calls/month included
- Pricing: $199/month flat rate (up to 5 developers)
- Additional developers: $29/developer/month

## Add-Ons

- AI Document Processing: $15/user/month - OCR, entity extraction, classification
- Advanced Analytics: $10/user/month - custom dashboards, predictive insights
- Priority Support: $500/month - 1-hour response SLA, dedicated Slack channel
  • Save documents/product_catalog.md.
  • Create a new file at documents/support_policies.md and paste:
# NovaTech Support Policies

## Support Tiers

### Standard Support (included with all plans)
- Email support: support@novatech.io
- Response time: within 24 business hours
- Hours: Monday-Friday, 9am-6pm CT
- Knowledge base access: docs.novatech.io
- Community forum: community.novatech.io

### Priority Support ($500/month add-on)
- Dedicated Slack channel with engineering team
- Response time: within 1 hour during business hours
- Extended hours: Monday-Friday, 7am-9pm CT
- Monthly architecture review call
- Direct escalation path to engineering leads

### Enterprise Support (included with Enterprise plan)
- 24/7 phone and email support
- Response time: within 15 minutes for critical issues
- Dedicated Technical Account Manager (TAM)
- Quarterly business reviews
- Custom integration development assistance
- On-site training available (additional cost)

## Return and Refund Policy

- Monthly plans: Cancel anytime, no refund for partial months
- Annual plans: 30-day money-back guarantee from start date
- Enterprise plans: Terms negotiated per contract, typically 90-day notice for cancellation
- All plans: Data export available for 30 days after cancellation

## Escalation Process

1. Level 1: Support agent (automated triage + human response)
2. Level 2: Senior support engineer (complex technical issues)
3. Level 3: Engineering team lead (product bugs, architectural issues)
4. Level 4: VP Engineering (critical outages, data incidents)

## SLA Details (Enterprise Only)

- Uptime guarantee: 99.9% monthly
- Credit structure: 10% credit for each 0.1% below SLA
- Maximum credit: 30% of monthly bill
- Exclusions: scheduled maintenance (announced 48hr in advance), force majeure
- Measurement: Based on API endpoint availability, measured every 60 seconds

## Contact Information

- General support: support@novatech.io
- Security issues: security@novatech.io
- Billing questions: billing@novatech.io
- Phone (Enterprise only): +1-512-555-0147
- Status page: status.novatech.io
  • Save documents/support_policies.md.

Why these three documents?

Each file covers a different category of business information: company background, product details, and support policies.

This variety lets you test how well the RAG system retrieves the right context for different types of questions. A question about pricing should pull from the product catalog, while a question about response times should pull from support policies.

  • Confirm all three files exist by running:
ls documents/

You should see company_overview.md, product_catalog.md, and support_policies.md listed.

  • Stage and commit the new documents by running:
git add .
git commit -m "Add business knowledge base documents"
git push

✔️ Awesome, I've got everything!

Great. Make sure all your files are saved in Cursor.

ⓧ I'd like to double check the full code

Here is requirements.txt:

langchain==1.3.11
langgraph==1.2.8
langchain-openai==1.3.3
langchain-anthropic==1.4.8
langchain-ollama==1.1.0
langchain-chroma==1.1.0
langchain-text-splitters==1.1.2
python-dotenv==1.2.2

Here is .env.example:

OPENAI_API_KEY=your-openai-api-key-here
ANTHROPIC_API_KEY=your-anthropic-api-key-here

Here is .gitignore:

.venv/
.env
chroma_db/
__pycache__/
*.pyc

Here is documents/company_overview.md:

# NovaTech Solutions - Company Overview

NovaTech Solutions is a B2B software company founded in 2019, specializing in workflow automation for mid-market enterprises. Headquartered in Austin, Texas, with offices in London and Singapore.

## Mission

Empower businesses to eliminate repetitive manual processes through intelligent automation, freeing teams to focus on strategic work.

## Key Facts

- Founded: 2019
- Employees: 320
- Customers: 1,200+ companies across 40 countries
- Annual Revenue: $45M (2025)
- Primary Industries: Healthcare, Financial Services, Manufacturing, Legal

## Leadership

- CEO: Sarah Chen (previously VP Engineering at Salesforce)
- CTO: Marcus Williams (co-founder, PhD in Distributed Systems)
- VP Sales: David Park (previously at HubSpot)

## Company Values

1. Customer outcomes over feature counts
2. Transparency in pricing and roadmap
3. Security is non-negotiable
4. Build for scale, deploy for simplicity

Here is documents/product_catalog.md:

# NovaTech Product Catalog

## FlowEngine Pro

Our flagship workflow automation platform for teams of 10-50 people.

- Visual drag-and-drop workflow builder
- 200+ pre-built integrations (Slack, Jira, Salesforce, etc.)
- Conditional logic and branching
- Audit trail and compliance logging
- Pricing: $29/user/month (annual) or $39/user/month (monthly)
- Free trial: 14 days, no credit card required

## FlowEngine Enterprise

For organizations with 50-500+ employees needing advanced governance.

- Everything in Pro, plus:
- SSO/SAML authentication
- Role-based access control (RBAC)
- Custom API rate limits
- Dedicated account manager
- 99.9% SLA with financial credits
- Pricing: Custom pricing, typically $49-79/user/month depending on volume
- Minimum commitment: 50 seats, annual contract

## FlowEngine Connect

Lightweight integration layer for developers who want API-first automation.

- RESTful API with webhooks
- SDK available in Python, JavaScript, and Go
- 50,000 API calls/month included
- Pricing: $199/month flat rate (up to 5 developers)
- Additional developers: $29/developer/month

## Add-Ons

- AI Document Processing: $15/user/month - OCR, entity extraction, classification
- Advanced Analytics: $10/user/month - custom dashboards, predictive insights
- Priority Support: $500/month - 1-hour response SLA, dedicated Slack channel

Here is documents/support_policies.md:

# NovaTech Support Policies

## Support Tiers

### Standard Support (included with all plans)
- Email support: support@novatech.io
- Response time: within 24 business hours
- Hours: Monday-Friday, 9am-6pm CT
- Knowledge base access: docs.novatech.io
- Community forum: community.novatech.io

### Priority Support ($500/month add-on)
- Dedicated Slack channel with engineering team
- Response time: within 1 hour during business hours
- Extended hours: Monday-Friday, 7am-9pm CT
- Monthly architecture review call
- Direct escalation path to engineering leads

### Enterprise Support (included with Enterprise plan)
- 24/7 phone and email support
- Response time: within 15 minutes for critical issues
- Dedicated Technical Account Manager (TAM)
- Quarterly business reviews
- Custom integration development assistance
- On-site training available (additional cost)

## Return and Refund Policy

- Monthly plans: Cancel anytime, no refund for partial months
- Annual plans: 30-day money-back guarantee from start date
- Enterprise plans: Terms negotiated per contract, typically 90-day notice for cancellation
- All plans: Data export available for 30 days after cancellation

## Escalation Process

1. Level 1: Support agent (automated triage + human response)
2. Level 2: Senior support engineer (complex technical issues)
3. Level 3: Engineering team lead (product bugs, architectural issues)
4. Level 4: VP Engineering (critical outages, data incidents)

## SLA Details (Enterprise Only)

- Uptime guarantee: 99.9% monthly
- Credit structure: 10% credit for each 0.1% below SLA
- Maximum credit: 30% of monthly bill
- Exclusions: scheduled maintenance (announced 48hr in advance), force majeure
- Measurement: Based on API endpoint availability, measured every 60 seconds

## Contact Information

- General support: support@novatech.io
- Security issues: security@novatech.io
- Billing questions: billing@novatech.io
- Phone (Enterprise only): +1-512-555-0147
- Status page: status.novatech.io

Your environment is fully configured. Next up, you will build the document ingestion pipeline that turns these markdown files into searchable vector embeddings.

Build the Document Ingestion Pipeline

Your environment is ready, and you have three markdown files representing a fictional company's knowledge base.

Now you need a way to make those documents searchable by meaning, not just by keywords. In this step, you'll build a document ingestion pipeline that reads your markdown files, splits them into small chunks, and stores them as embeddings in a vector store.

In this step, get ready to:
  • Load and chunk markdown documents using RecursiveCharacterTextSplitter.
  • Embed chunks and store them in ChromaDB.
  • Verify retrieval with a test query.
Load markdown files and split them into chunks

Before your RAG system can answer questions, it needs to break large documents into smaller pieces that fit inside an LLM's context window. A text splitter handles this by cutting documents at natural boundaries (headings, paragraphs) while keeping each chunk under a size limit.

  • In Cursor, create a new file called ingest.py in your project root (the same folder that contains requirements.txt).
  • Add the imports, environment loading, and document loading logic by pasting this code:
"""Document ingestion pipeline - loads, chunks, and embeds documents into ChromaDB."""

from pathlib import Path
from dotenv import load_dotenv
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

load_dotenv()


def ingest_documents():
    """Load markdown files, split into chunks, and store in ChromaDB."""

    # Load documents
    docs_dir = Path("documents")
    documents = []
    for md_file in sorted(docs_dir.glob("*.md")):
        content = md_file.read_text(encoding="utf-8")
        documents.append(
            Document(page_content=content, metadata={"source": md_file.name})
        )

    print(f"Loaded {len(documents)} documents")

    # Split into chunks
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=50,
        separators=["\n## ", "\n### ", "\n\n", "\n", " ", ""],
    )
    splits = text_splitter.split_documents(documents)
    print(f"Split into {len(splits)} chunks")

What does this code do?

The script reads every .md file from the documents/ folder and wraps each one in a LangChain Document object. The metadata dict stores the filename so you can trace results back to their source later.

RecursiveCharacterTextSplitter breaks each document into chunks of 500 characters with 50 characters of overlap between consecutive chunks. The custom separators list tells the splitter to prefer cutting at markdown headings and paragraph breaks before resorting to mid-sentence splits.

💡 Why 500 characters with 50 overlap?

Smaller chunks give the retriever more precision. A 500-character chunk is roughly one paragraph of business content.

The 50-character overlap ensures that sentences straddling a chunk boundary still appear in at least one chunk, so meaning isn't lost at the edges.

  • Save ingest.py.

Seeing an import error?

Make sure your virtual environment is activated. You should see (.venv) at the start of your terminal prompt. If it's missing, activate it again with source .venv/bin/activate (macOS/Linux) or .venv\Scripts\activate (Windows).

Help me debug an import error in ingest.py

Embed chunks and store in ChromaDB

With your documents chunked, the next step is to convert each chunk into a numerical vector (an embedding) and store it in a database optimized for similarity search. ChromaDB is a local vector store that persists embeddings to disk so you don't need to re-embed every time you run a query.

  • In ingest.py, add the following code directly below the print(f"Split into {len(splits)} chunks") line, still inside the ingest_documents() function:
    # Embed and store in ChromaDB
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectorstore = Chroma.from_documents(
        documents=splits,
        embedding=embeddings,
        persist_directory="./chroma_db",
    )

    print(f"Stored {len(splits)} chunks in ChromaDB at ./chroma_db/")

What does this code do?

OpenAIEmbeddings calls OpenAI's text-embedding-3-small model to convert each text chunk into a 1536-dimensional vector. These vectors capture the semantic meaning of each chunk, so "refund policy" and "money-back guarantee" end up near each other in vector space.

Chroma.from_documents embeds all chunks in one call and saves them to the ./chroma_db/ directory. Because it persists to disk, you only run ingestion once. Future scripts can load this store without re-embedding.

  • Save ingest.py.

Getting an OpenAI authentication error?

Double-check that your .env file contains OPENAI_API_KEY with a valid key. The load_dotenv() call at the top of the file reads this automatically.

Help me fix an OpenAI API key error in my ingestion script

Verify retrieval with a test query

The final section of the script runs a test query against the vector store to confirm that semantic search works. This proves the embeddings are stored correctly and that relevant chunks come back for a natural-language question.

  • In ingest.py, add the verification code below the print(f"Stored {len(splits)} chunks in ChromaDB at ./chroma_db/") line, still inside the function:
    # Verify with a test query
    results = vectorstore.similarity_search("What is the return policy?", k=2)
    print(f"\nVerification - test query returned {len(results)} results:")
    for i, doc in enumerate(results):
        print(f"  {i+1}. [{doc.metadata['source']}] {doc.page_content[:80]}...")
  • After the function definition, add the entry point at the very bottom of the file (no indentation):
if __name__ == "__main__":
    ingest_documents()

What does this code do?

similarity_search takes a plain-text question, embeds it, and returns the 2 closest chunks from the vector store. The results are printed with their source filename and a preview of the content.

The if __name__ == "__main__" block ensures the function runs only when you execute the file directly, not when you import it from another script later.

  • Save ingest.py.
  • Run the ingestion pipeline by executing this command in your terminal:
python ingest.py

You should see output showing 3 documents loaded, a number of chunks created, and 2 verification results from the test query.

Seeing zero results or a connection error?

If the script fails with a network error, check your internet connection. The text-embedding-3-small model requires an API call to OpenAI's servers.

If you see 0 results, confirm that your documents/ folder contains the three .md files and that you're running the command from the project root folder (the one containing ingest.py).

Help me debug why my ingestion script isn't returning results

  • Confirm the chroma_db/ directory was created by running:
ls chroma_db/

You should see files inside the directory. This confirms your embeddings are persisted to disk.

✔️ Awesome, I've got everything!

Great. Double check you've saved ingest.py and that your terminal shows the verification output with 2 results.

ⓧ I'd like to double check the full code

"""Document ingestion pipeline - loads, chunks, and embeds documents into ChromaDB."""

from pathlib import Path
from dotenv import load_dotenv
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

load_dotenv()


def ingest_documents():
    """Load markdown files, split into chunks, and store in ChromaDB."""

    # Load documents
    docs_dir = Path("documents")
    documents = []
    for md_file in sorted(docs_dir.glob("*.md")):
        content = md_file.read_text(encoding="utf-8")
        documents.append(
            Document(page_content=content, metadata={"source": md_file.name})
        )

    print(f"Loaded {len(documents)} documents")

    # Split into chunks
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=50,
        separators=["\n## ", "\n### ", "\n\n", "\n", " ", ""],
    )
    splits = text_splitter.split_documents(documents)
    print(f"Split into {len(splits)} chunks")

    # Embed and store in ChromaDB
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectorstore = Chroma.from_documents(
        documents=splits,
        embedding=embeddings,
        persist_directory="./chroma_db",
    )

    print(f"Stored {len(splits)} chunks in ChromaDB at ./chroma_db/")

    # Verify with a test query
    results = vectorstore.similarity_search("What is the return policy?", k=2)
    print(f"\nVerification - test query returned {len(results)} results:")
    for i, doc in enumerate(results):
        print(f"  {i+1}. [{doc.metadata['source']}] {doc.page_content[:80]}...")


if __name__ == "__main__":
    ingest_documents()
  • Commit your progress to Git by running:
git add ingest.py
git commit -m "Add document ingestion pipeline"
git push

Your knowledge base is now searchable by meaning. Next up, you'll build the agentic RAG graph that retrieves these chunks, grades their relevance, and rewrites queries when results are poor.

Build the LangGraph Agentic RAG Graph

Your knowledge base is ready with embedded documents sitting in ChromaDB. A basic RAG system would just retrieve chunks and pass them to an LLM. The problem is that retrieval often returns irrelevant results, and the LLM hallucinates confidently from bad context.

In this step, you will build an agentic RAG graph using LangGraph that solves this problem. The graph evaluates retrieved documents for relevance before generating an answer. If the documents are not relevant, it rewrites the query and tries again.

In this step, get ready to:
  • Define the graph state, grading schema, and prompt templates.
  • Implement four graph nodes: retrieve, grade, generate, and rewrite.
  • Wire the graph with conditional edges and test it end-to-end.
Define the graph state and grading schema

Every LangGraph workflow needs a state object that travels through each node. You also need a structured schema so the LLM returns machine-readable relevance grades instead of freeform text.

  • Create a new file called graph.py in your project folder by running:
touch graph.py
  • Confirm the file exists by running:
ls

You should see graph.py listed alongside ingest.py and your other project files.

  • Open graph.py in Cursor and add the imports, state definition, and grading schema:
"""Agentic RAG graph using LangGraph with retrieval grading and query rewriting."""

from typing import Literal
from typing_extensions import TypedDict
from pydantic import BaseModel, Field
from dotenv import load_dotenv
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.language_models import BaseChatModel
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
from langgraph.graph import StateGraph, START, END

load_dotenv()


# --- State ---
class GraphState(TypedDict):
    question: str
    generation: str
    documents: list[Document]
    retry_count: int


# --- Grading Schema ---
class GradeDocuments(BaseModel):
    """Grade whether a document is relevant to a question."""

    relevant: bool = Field(
        description="Whether the document is relevant to the question"
    )

What does this code do?

  • GraphState is a TypedDict that defines what data flows between nodes. Every node reads from and writes to this shared state.
  • GradeDocuments is a Pydantic model that forces the LLM to return a structured boolean instead of freeform text. This makes routing decisions reliable.
  • The retry_count field tracks how many times the graph has rewritten the query, preventing infinite loops.

You won't see a change yet. The next chunk adds the prompt templates that instruct the LLM.

  • Add the three prompt templates below your GradeDocuments class:
# --- Prompts ---
GRADE_PROMPT = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a grader assessing whether a retrieved document is relevant "
            "to a user question. If the document contains keywords or semantic meaning "
            "related to the question, grade it as relevant. Give a binary 'relevant' score.",
        ),
        ("human", "Document:\n{document}\n\nQuestion: {question}"),
    ]
)

GENERATE_PROMPT = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an assistant answering questions using ONLY the provided context. "
            "If the context does not contain enough information to answer, say so clearly. "
            "Keep answers concise and cite which part of the context supports your answer.",
        ),
        ("human", "Context:\n{context}\n\nQuestion: {question}"),
    ]
)

REWRITE_PROMPT = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a query rewriter. Given a question that failed to retrieve "
            "relevant documents, rewrite it to be more specific and better suited "
            "for semantic search against a business knowledge base. "
            "Return only the rewritten question, nothing else.",
        ),
        ("human", "Original question: {question}\n\nRewritten question:"),
    ]
)

What do these prompts do?

  • GRADE_PROMPT asks the LLM to judge whether a single document chunk is relevant to the user's question.
  • GENERATE_PROMPT instructs the LLM to answer using ONLY the provided context. This prevents hallucination from the model's training data.
  • REWRITE_PROMPT triggers when retrieval fails. It reformulates the question to improve semantic search results on the next attempt.

You won't see a change yet. The next chunk creates the retriever connection.

  • Add the get_retriever() function below your prompt templates:
def get_retriever():
    """Load the existing ChromaDB vector store and return a retriever."""
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectorstore = Chroma(
        persist_directory="./chroma_db", embedding_function=embeddings
    )
    return vectorstore.as_retriever(search_kwargs={"k": 4})

What does this function do?

This loads the existing ChromaDB database you created in the previous step and wraps it as a LangChain retriever. The k=4 setting returns the top 4 most similar chunks for any query.

Using the same text-embedding-3-small model ensures the retriever can find the embeddings you stored earlier.

  • Save graph.py.

You won't see a visible change yet. The next substep builds the node functions that use these definitions.

Seeing import errors in Cursor?

  • Make sure your virtual environment is selected as the Python interpreter in Cursor. Check the bottom-right status bar for the correct .venv path.
  • Confirm you installed all packages in the previous step. Run pip list and check that langgraph and langchain-core appear.

Why are my imports showing errors in graph.py?

Implement the four graph nodes

Nodes are the individual steps your graph executes. Each node is a Python function that reads from the graph state, does one job, and returns updates to the state.

  • Add the create_rag_graph() function with its chain setup, retrieve node, and grade_documents node below get_retriever():
def create_rag_graph(llm: BaseChatModel, retriever):
    """Create an agentic RAG graph with the given LLM and retriever."""

    # Build chains using LCEL
    grader_llm = llm.with_structured_output(GradeDocuments)
    grade_chain = GRADE_PROMPT | grader_llm
    generate_chain = GENERATE_PROMPT | llm
    rewrite_chain = REWRITE_PROMPT | llm

    # --- Nodes ---
    def retrieve(state: GraphState) -> dict:
        """Retrieve documents from the vector store."""
        question = state["question"]
        documents = retriever.invoke(question)
        return {"documents": documents}

    def grade_documents(state: GraphState) -> dict:
        """Grade each retrieved document for relevance."""
        question = state["question"]
        documents = state["documents"]
        relevant_docs = []

        for doc in documents:
            grade = grade_chain.invoke(
                {"question": question, "document": doc.page_content}
            )
            if grade.relevant:
                relevant_docs.append(doc)

        print(
            f"  [grade] {len(relevant_docs)}/{len(documents)} documents relevant"
        )
        return {"documents": relevant_docs}

What does this code do?

  • The chain setup uses LCEL (LangChain Expression Language) to pipe prompts into LLMs. The with_structured_output(GradeDocuments) call forces the LLM to return a GradeDocuments object with a boolean relevant field.
  • The retrieve node queries ChromaDB with the user's question and returns the top 4 document chunks.
  • The grade_documents node loops through each retrieved chunk, asks the LLM if it is relevant, and keeps only the relevant ones. This filtering step is what makes the system "agentic."

You won't see a change yet. The next chunk adds the remaining two nodes.

  • Add the generate and rewrite_query nodes below grade_documents, still inside create_rag_graph():
    def generate(state: GraphState) -> dict:
        """Generate an answer from relevant documents."""
        question = state["question"]
        documents = state["documents"]
        context = "\n\n".join(doc.page_content for doc in documents)

        if not context:
            return {
                "generation": "I don't have enough information in the knowledge base to answer this question."
            }

        response = generate_chain.invoke(
            {"question": question, "context": context}
        )
        return {"generation": response.content}

    def rewrite_query(state: GraphState) -> dict:
        """Rewrite the query for better retrieval."""
        question = state["question"]
        retry_count = state.get("retry_count", 0)

        response = rewrite_chain.invoke({"question": question})
        new_question = response.content.strip()
        print(f"  [rewrite] '{question}' -> '{new_question}'")
        return {"question": new_question, "retry_count": retry_count + 1}

What do these nodes do?

  • The generate node joins all relevant document chunks into a single context string and asks the LLM to answer. If no relevant documents survived grading, it returns a clear "not enough information" message instead of hallucinating.
  • The rewrite_query node asks the LLM to reformulate the question for better semantic search results. It also increments retry_count so the graph knows how many attempts have been made.
  • Save graph.py.

You won't see a runtime change yet. The next substep wires these nodes together into a working graph.

Indentation issues?

  • Both generate and rewrite_query must be indented one level inside create_rag_graph(). They are nested functions, not top-level definitions.
  • If Cursor shows a syntax error, check that these functions are at the same indentation level as retrieve and grade_documents above them.

Help me fix indentation errors in my graph.py node functions.

Wire the graph with conditional edges and compile

With all four nodes defined, you need to tell LangGraph how they connect. This is where the "agentic" behavior comes from. Conditional edges let the graph choose its own path based on runtime results.

  • Add the routing function and graph construction below rewrite_query, still inside create_rag_graph():
    # --- Routing ---
    def route_after_grading(
        state: GraphState,
    ) -> Literal["generate", "rewrite_query"]:
        """Route based on grading results."""
        if state["documents"]:
            return "generate"
        if state.get("retry_count", 0) >= 2:
            print("  [route] Max retries reached, generating with empty context")
            return "generate"
        return "rewrite_query"

    # --- Build Graph ---
    workflow = StateGraph(GraphState)

    workflow.add_node("retrieve", retrieve)
    workflow.add_node("grade_documents", grade_documents)
    workflow.add_node("generate", generate)
    workflow.add_node("rewrite_query", rewrite_query)

    workflow.add_edge(START, "retrieve")
    workflow.add_edge("retrieve", "grade_documents")
    workflow.add_conditional_edges(
        "grade_documents",
        route_after_grading,
        ["generate", "rewrite_query"],
    )
    workflow.add_edge("rewrite_query", "retrieve")
    workflow.add_edge("generate", END)

    return workflow.compile()

What does this code do?

  • route_after_grading() checks if any relevant documents survived grading. If yes, it routes to generate. If no relevant docs remain and retries are under the limit, it routes to rewrite_query.
  • add_conditional_edges() is the LangGraph method that creates a branching point. The routing function decides which node runs next at runtime.
  • The rewrite_query -> retrieve edge creates the retry loop. This is what makes LangGraph essential. Plain LangChain LCEL only supports straight-line chains with no loops.
  • workflow.compile() validates the graph structure and returns an executable object you can invoke.

💡 Why LangGraph instead of plain LangChain?

Plain LangChain LCEL is limited to DAGs (directed acyclic graphs). That means no loops. The "grade and retry" pattern requires a cycle: rewrite goes back to retrieve, which goes back to grade.

LangGraph adds cyclic graph execution on top of LangChain. This single capability is what separates a naive RAG chatbot from a self-correcting agentic system.

  • Save graph.py.
  • Test the graph by running this command in your terminal:
python -c "
from graph import create_rag_graph, get_retriever
from langchain_openai import ChatOpenAI

retriever = get_retriever()
llm = ChatOpenAI(model='gpt-4o', temperature=0)
graph = create_rag_graph(llm, retriever)
result = graph.invoke({'question': 'What is the return policy?', 'generation': '', 'documents': [], 'retry_count': 0})
print(result['generation'])
"

You should see output showing the graph traversal followed by an answer about NovaTech's return policy. Look for the [grade] line showing how many documents were marked relevant.

What should I see?

A successful run prints something like [grade] 3/4 documents relevant followed by a concise answer about cancellation policies, refund timelines, and data export windows. The exact numbers and wording vary by LLM response.

If the graph rewrites the query, you will also see a [rewrite] line showing the original and rewritten question. This means the agentic loop is working.

🙋‍♀️ Getting an error when running the test?

  • If you see No module named 'graph', make sure your terminal is in the project folder (the same directory where graph.py lives).
  • If you see an OpenAI authentication error, confirm your .env file has a valid OPENAI_API_KEY value.
  • If you see a ChromaDB error about the collection not existing, run python ingest.py again to rebuild the vector store.

Help me debug my LangGraph agentic RAG test.

✔️ Awesome, I've got everything!

Great. Double check you have saved graph.py and your test returned an answer about the return policy.

ⓧ I'd like to double check the full code

Here is the complete graph.py file. Compare yours against this:

"""Agentic RAG graph using LangGraph with retrieval grading and query rewriting."""

from typing import Literal
from typing_extensions import TypedDict
from pydantic import BaseModel, Field
from dotenv import load_dotenv
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.language_models import BaseChatModel
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
from langgraph.graph import StateGraph, START, END

load_dotenv()


# --- State ---
class GraphState(TypedDict):
    question: str
    generation: str
    documents: list[Document]
    retry_count: int


# --- Grading Schema ---
class GradeDocuments(BaseModel):
    """Grade whether a document is relevant to a question."""

    relevant: bool = Field(
        description="Whether the document is relevant to the question"
    )


# --- Prompts ---
GRADE_PROMPT = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a grader assessing whether a retrieved document is relevant "
            "to a user question. If the document contains keywords or semantic meaning "
            "related to the question, grade it as relevant. Give a binary 'relevant' score.",
        ),
        ("human", "Document:\n{document}\n\nQuestion: {question}"),
    ]
)

GENERATE_PROMPT = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an assistant answering questions using ONLY the provided context. "
            "If the context does not contain enough information to answer, say so clearly. "
            "Keep answers concise and cite which part of the context supports your answer.",
        ),
        ("human", "Context:\n{context}\n\nQuestion: {question}"),
    ]
)

REWRITE_PROMPT = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a query rewriter. Given a question that failed to retrieve "
            "relevant documents, rewrite it to be more specific and better suited "
            "for semantic search against a business knowledge base. "
            "Return only the rewritten question, nothing else.",
        ),
        ("human", "Original question: {question}\n\nRewritten question:"),
    ]
)


def get_retriever():
    """Load the existing ChromaDB vector store and return a retriever."""
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectorstore = Chroma(
        persist_directory="./chroma_db", embedding_function=embeddings
    )
    return vectorstore.as_retriever(search_kwargs={"k": 4})


def create_rag_graph(llm: BaseChatModel, retriever):
    """Create an agentic RAG graph with the given LLM and retriever."""

    # Build chains using LCEL
    grader_llm = llm.with_structured_output(GradeDocuments)
    grade_chain = GRADE_PROMPT | grader_llm
    generate_chain = GENERATE_PROMPT | llm
    rewrite_chain = REWRITE_PROMPT | llm

    # --- Nodes ---
    def retrieve(state: GraphState) -> dict:
        """Retrieve documents from the vector store."""
        question = state["question"]
        documents = retriever.invoke(question)
        return {"documents": documents}

    def grade_documents(state: GraphState) -> dict:
        """Grade each retrieved document for relevance."""
        question = state["question"]
        documents = state["documents"]
        relevant_docs = []

        for doc in documents:
            grade = grade_chain.invoke(
                {"question": question, "document": doc.page_content}
            )
            if grade.relevant:
                relevant_docs.append(doc)

        print(
            f"  [grade] {len(relevant_docs)}/{len(documents)} documents relevant"
        )
        return {"documents": relevant_docs}

    def generate(state: GraphState) -> dict:
        """Generate an answer from relevant documents."""
        question = state["question"]
        documents = state["documents"]
        context = "\n\n".join(doc.page_content for doc in documents)

        if not context:
            return {
                "generation": "I don't have enough information in the knowledge base to answer this question."
            }

        response = generate_chain.invoke(
            {"question": question, "context": context}
        )
        return {"generation": response.content}

    def rewrite_query(state: GraphState) -> dict:
        """Rewrite the query for better retrieval."""
        question = state["question"]
        retry_count = state.get("retry_count", 0)

        response = rewrite_chain.invoke({"question": question})
        new_question = response.content.strip()
        print(f"  [rewrite] '{question}' -> '{new_question}'")
        return {"question": new_question, "retry_count": retry_count + 1}

    # --- Routing ---
    def route_after_grading(
        state: GraphState,
    ) -> Literal["generate", "rewrite_query"]:
        """Route based on grading results."""
        if state["documents"]:
            return "generate"
        if state.get("retry_count", 0) >= 2:
            print("  [route] Max retries reached, generating with empty context")
            return "generate"
        return "rewrite_query"

    # --- Build Graph ---
    workflow = StateGraph(GraphState)

    workflow.add_node("retrieve", retrieve)
    workflow.add_node("grade_documents", grade_documents)
    workflow.add_node("generate", generate)
    workflow.add_node("rewrite_query", rewrite_query)

    workflow.add_edge(START, "retrieve")
    workflow.add_edge("retrieve", "grade_documents")
    workflow.add_conditional_edges(
        "grade_documents",
        route_after_grading,
        ["generate", "rewrite_query"],
    )
    workflow.add_edge("rewrite_query", "retrieve")
    workflow.add_edge("generate", END)

    return workflow.compile()
  • Commit your progress to GitHub by running:
git add graph.py
git commit -m "Add agentic RAG graph with retrieval grading and query rewriting"
git push

Your agentic RAG graph is working. It retrieves, grades, rewrites when needed, and generates grounded answers. Next up, you will put this graph to the test by running the same queries through three different LLM providers and comparing their performance.

Run the Multi-Provider Comparison

Your agentic RAG graph can now retrieve, grade, rewrite, and generate using any LLM you plug in.

In this step, you will build a comparison script that sends identical questions through three different providers: OpenAI, Anthropic, and Ollama's cloud-hosted Nemotron 3 Super model. By measuring response time, retry count, and answer quality side-by-side, you will have concrete evidence to recommend one provider over another for a business client.

In this step, get ready to:
  • Create a comparison script that initializes all three LLM providers.
  • Run identical queries through each provider and collect metrics.
  • Analyze the results for answer quality, speed, and cost implications.
Create a comparison script that initializes all three providers

The comparison script reuses the same create_rag_graph() function and retriever from graph.py. Each provider gets its own compiled graph instance, but they all share the same ChromaDB retriever so retrieval results are identical.

  • Create a new file called compare.py in your project folder.
  • Add the imports and provider initialization by pasting this code:
"""Multi-provider comparison - runs the same queries through OpenAI, Anthropic, and Ollama."""

import time
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
from langchain_ollama import ChatOllama
from graph import create_rag_graph, get_retriever

load_dotenv()


def run_comparison():
    """Run identical queries through all three providers and compare results."""

    retriever = get_retriever()

    # Initialize providers
    providers = {
        "OpenAI (gpt-4o)": ChatOpenAI(model="gpt-4o", temperature=0),
        "Anthropic (claude-sonnet-4-0)": ChatAnthropic(
            model="claude-sonnet-4-0", temperature=0
        ),
        "Ollama (nemotron-3-super:cloud)": ChatOllama(model="nemotron-3-super:cloud", temperature=0),
    }

What does this code do?

  • import time gives us the time.time() function to measure how long each provider takes.
  • from graph import create_rag_graph, get_retriever pulls in the RAG graph and retriever you built in the previous step.
  • Each provider is initialized with temperature=0 so responses are deterministic. This makes the comparison fair.
  • All three providers share the same retriever so retrieval results are identical across providers.

💡 Why the same retriever for all providers?

This comparison measures generation quality, not embedding quality. Using the same OpenAI embeddings for all providers ensures that differences in the final answers come purely from how each LLM interprets the context, not from which documents were retrieved.

Add the comparison loop and run the script

Now you need the logic that runs each question through every provider, measures timing, and prints results.

  • In compare.py, add the following code directly below the providers dictionary (still inside the run_comparison() function):
    # Test questions of varying difficulty
    questions = [
        "What is the return policy for enterprise customers?",
        "Which product is best for a team of 75 people and what does it cost?",
        "How do I get help outside of business hours if I'm on the Pro plan?",
    ]

    # Run comparison
    for question in questions:
        print(f"\n{'=' * 80}")
        print(f"QUESTION: {question}")
        print(f"{'=' * 80}")

        for provider_name, llm in providers.items():
            graph = create_rag_graph(llm, retriever)

            print(f"\n--- {provider_name} ---")
            start_time = time.time()

            try:
                result = graph.invoke(
                    {
                        "question": question,
                        "generation": "",
                        "documents": [],
                        "retry_count": 0,
                    }
                )
                elapsed = time.time() - start_time

                print(f"  Time: {elapsed:.2f}s")
                print(f"  Retries: {result.get('retry_count', 0)}")
                print(f"  Answer: {result['generation'][:200]}")
                if len(result["generation"]) > 200:
                    print(f"         ...({len(result['generation'])} chars total)")

            except Exception as e:
                elapsed = time.time() - start_time
                print(f"  ERROR after {elapsed:.2f}s: {e}")

    print(f"\n{'=' * 80}")
    print("COMPARISON COMPLETE")
    print(f"{'=' * 80}")


if __name__ == "__main__":
    run_comparison()

What does this code do?

  • Three test questions cover different difficulty levels: a direct lookup, a reasoning question requiring product comparison, and a nuanced policy question.
  • For each question, the script creates a fresh compiled graph for each provider using create_rag_graph(llm, retriever).
  • time.time() captures elapsed seconds for each provider's full graph traversal (retrieve, grade, possibly rewrite, and generate).
  • The try/except block catches provider errors (network timeouts, rate limits) so one failure does not crash the entire comparison.
  • Answers are truncated to 200 characters in the console output for readability.
  • Save compare.py.
  • Run the comparison script by running this command:
python compare.py

The script will take 1-3 minutes to complete depending on your internet speed and local hardware. You should see each question printed with a separator, followed by results from all three providers showing their time, retry count, and answer.

Seeing connection errors?

  • If Ollama (nemotron-3-super:cloud) fails, confirm Ollama is still running and you are signed in. Open a separate terminal and run ollama run nemotron-3-super:cloud "test" to check.
  • If OpenAI or Anthropic errors mention rate limits or authentication, double-check your .env file has valid keys for both OPENAI_API_KEY and ANTHROPIC_API_KEY.
  • If you see a ModuleNotFoundError, make sure your virtual environment is activated before running the script.

Help me debug my comparison script

✔️ Awesome, I've got everything!

Great. Double check you have saved compare.py and that the comparison ran successfully for all three providers.

ⓧ I'd like to double check the full code

"""Multi-provider comparison - runs the same queries through OpenAI, Anthropic, and Ollama."""

import time
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
from langchain_ollama import ChatOllama
from graph import create_rag_graph, get_retriever

load_dotenv()


def run_comparison():
    """Run identical queries through all three providers and compare results."""

    retriever = get_retriever()

    # Initialize providers
    providers = {
        "OpenAI (gpt-4o)": ChatOpenAI(model="gpt-4o", temperature=0),
        "Anthropic (claude-sonnet-4-0)": ChatAnthropic(
            model="claude-sonnet-4-0", temperature=0
        ),
        "Ollama (nemotron-3-super:cloud)": ChatOllama(model="nemotron-3-super:cloud", temperature=0),
    }

    # Test questions of varying difficulty
    questions = [
        "What is the return policy for enterprise customers?",
        "Which product is best for a team of 75 people and what does it cost?",
        "How do I get help outside of business hours if I'm on the Pro plan?",
    ]

    # Run comparison
    for question in questions:
        print(f"\n{'=' * 80}")
        print(f"QUESTION: {question}")
        print(f"{'=' * 80}")

        for provider_name, llm in providers.items():
            graph = create_rag_graph(llm, retriever)

            print(f"\n--- {provider_name} ---")
            start_time = time.time()

            try:
                result = graph.invoke(
                    {
                        "question": question,
                        "generation": "",
                        "documents": [],
                        "retry_count": 0,
                    }
                )
                elapsed = time.time() - start_time

                print(f"  Time: {elapsed:.2f}s")
                print(f"  Retries: {result.get('retry_count', 0)}")
                print(f"  Answer: {result['generation'][:200]}")
                if len(result["generation"]) > 200:
                    print(f"         ...({len(result['generation'])} chars total)")

            except Exception as e:
                elapsed = time.time() - start_time
                print(f"  ERROR after {elapsed:.2f}s: {e}")

    print(f"\n{'=' * 80}")
    print("COMPARISON COMPLETE")
    print(f"{'=' * 80}")


if __name__ == "__main__":
    run_comparison()
Analyze and interpret the results

Look at your comparison output and consider what each metric tells you about the providers.

What should I look for in the results?

Focus on three dimensions when reading the output:

  • Answer quality and grounding. Does the answer cite specific details from the knowledge base, or does it add information not present in the documents? Answers that invent details are hallucinating.
  • Response time. Cloud APIs (OpenAI, Anthropic) typically respond in 2-10 seconds. Ollama's cloud-hosted Nemotron 3 Super may vary depending on server load and model size (120B parameters with 12B active).
  • Retry count. A retry means the grader decided the retrieved documents were not relevant and the query was rewritten. More retries suggest the LLM's grading threshold is stricter.

💡 What about cost implications?

Each query to OpenAI or Anthropic costs a fraction of a cent. Ollama's cloud models are included in your Ollama plan (Free tier provides light usage). Over thousands of queries per day, costs from paid APIs add up, while Ollama's plan-based pricing keeps per-query costs predictable.

For a business client, the recommendation depends on their priorities: speed and polish favor established cloud providers, while open-weight models like Nemotron 3 Super offer transparency and flexibility for teams that want control over their inference stack.

  • Commit your work and push to GitHub by running these commands:
git add compare.py
git commit -m "Add multi-provider comparison script"
git push
  • Confirm the push succeeded by checking your GitHub repository page. You should see compare.py in the file list.

Push failed?

  • If git push fails with an authentication error, check that your GitHub credentials or SSH key are configured correctly for this repository.
  • If the error mentions the remote is ahead, run git pull --rebase first, then try pushing again.

Help me fix my git push error

You now have hard data comparing three LLM providers on the same agentic RAG task. Next up, you will take this system further by adding hallucination detection in the Secret Mission.

Secret mission

Add Hallucination Grading

Your RAG system grades retrieval quality, but what about the answer itself? An LLM can retrieve the right documents and still hallucinate claims that aren't in them. Add a hallucination grading node that checks whether the generated answer is actually supported by the source documents, and loops back to regenerate if not.

Clean Up Your Resources

Clean Up Your Resources

This project runs locally apart from the API calls you made to OpenAI, Anthropic, and Ollama's cloud service. There are no running services or ongoing costs to worry about.

Decide whether to keep your resources as they are, free up some disk space, or delete everything entirely.

Resources you used:

  • Project folder on your Desktop (includes source code, .venv/ virtual environment, and chroma_db/ vector database).
  • .env file containing your OpenAI and Anthropic API keys (inside the project folder).
  • GitHub repository with your project commits.

Keep everything running

No action needed. The project is fully local with no running services or recurring charges. Your chroma_db/ folder, source files, and virtual environment will stay on your machine until you choose to remove them.

  • Your API keys remain stored in .env locally. They are not committed to GitHub thanks to your .gitignore.
  • Ollama continues to run in the background if you started it. You can stop it anytime or leave it running for other projects.

Pause - I'll come back to this later

Free up disk space by removing the vector database while keeping your source code intact. You can regenerate the database anytime by running python ingest.py again.

  • Delete the chroma_db/ folder to reclaim disk space by running:
  • Keep your .env file so your API keys are ready when you return.
  • Keep the .venv/ folder so your dependencies stay installed.
rm -rf chroma_db/

When you come back, run python ingest.py to rebuild the vector store from your documents.

Delete - I don't want to use this again

Remove all project files from your machine and delete the GitHub repository.

  • Navigate to your Desktop and delete the project folder by running:

macOS

cd ~/Desktop
rm -rf agentic-rag-langgraph

Windows

cd ~\Desktop
rmdir /s /q agentic-rag-langgraph

This removes all source code, the .venv/ virtual environment, the chroma_db/ database, and your .env file.

  • Delete the GitHub repository:
  • Go to your repository page on GitHub.
  • Click Settings in the top menu.
  • Scroll to the bottom to find the Danger Zone section.
  • Click Delete this repository and follow the confirmation prompts.

Nice Work!

Nice Work!

Nice work! You just built a production-pattern Agentic RAG system that self-corrects its own retrieval quality and compares answers across multiple LLM providers.

You've learned how to:

  • Build a document ingestion pipeline that chunks and embeds business documents into ChromaDB for semantic retrieval.
  • Implement an Agentic RAG system using LangGraph with retrieval grading, query rewriting loops, and conditional routing.
  • Compare three LLM providers (OpenAI, Anthropic, and Ollama) on the same RAG task to evaluate answer quality, response time, and cost.
  • Secret Mission: Added hallucination grading to prevent the system from generating answers not grounded in retrieved context.

Ready to quiz yourself?