FDA Drug Safety Intelligence Agent

Build a multi-agent RAG system that answers drug safety questions using real FDA data.

Introduction

30 Second Summary

Every drug on the market has a safety profile buried in thousands of pages of FDA documentation. When patients or researchers need answers about side effects or drug interactions, they need them fast and grounded in real data.

In this project, you will build a multi-agent AI system that pulls real FDA drug labels, stores them in a vector database, and coordinates specialized agents to answer complex safety questions with cited sources. You will also build an evaluation harness that measures whether your system's answers are actually trustworthy.

What You'll Build

A multi-agent drug safety intelligence system that ingests real FDA drug labels into a local vector database, uses a LangGraph supervisor to coordinate three AI agents, and evaluates answer quality with automated pass/fail tests.

By the end of this project, you'll have:

A RAG pipeline that fetches FDA drug labels, chunks them, and stores embeddings in ChromaDB.
A LangGraph supervisor graph coordinating three specialized agents: a Retrieval Agent, an Adverse Event Agent, and a Report Agent.
An evaluation harness using DeepEval and RAGAS that measures faithfulness, answer relevancy, and context precision.
A Drug Interaction Checker agent that cross-references multiple drug labels to flag potential interactions (Secret Mission).

Are there any prerequisites?

Basic Python knowledge is all you need (functions, dictionaries, classes). No prior experience with LangChain, LangGraph, or vector databases is required.

Set Up Your Development Environment

This project uses several tools that work together to build, evaluate, and observe a multi-agent system. Before writing any code, you need a clean environment with the right Python version, all dependencies installed, and API keys configured.

In this step, you will install Python, set up a virtual environment with all project packages, and connect to two free cloud services: Groq for LLM inference and LangSmith for tracing.

In this step, get ready to:

Install Python and project dependencies in a virtual environment.
Create a Groq account and verify your API key works.
Set up LangSmith for production-grade tracing.

Install Python and create your project

This project requires Python 3.11 or higher. Let's confirm you have it installed.

Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
Type VS Code and press Enter to open it.
Open the integrated terminal by pressing Ctrl+` (macOS and Windows).
Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

I see a version number

You should see something like Python 3.11.x or higher. You are good to go.

Command not found

Download Python from the official Python website.
Download the latest 3.11+ installer for your operating system.
Run the installer. On Windows, make sure to check the box that says "Add Python to PATH" before clicking Install.
Close and reopen your VS Code terminal, then run the version check command again to confirm it works.

Move to your Desktop by running this command:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir fda-drug-safety-agent
cd fda-drug-safety-agent

Confirm the folder was created by running:

ls

The folder should be empty for now. That is expected.

A virtual environment keeps this project's dependencies isolated from other Python projects on your machine. Let's create one now.

Create and activate a virtual environment by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay local to this project and do not interfere with other Python projects on your machine.

Not seeing (venv) in your prompt?

Make sure you are inside the fda-drug-safety-agent folder before running the activation command.
On Windows, if you see an error about script execution, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned in PowerShell and try again.

Help me activate my Python virtual environment.

Now let's install the project dependencies. This project uses pinned versions to ensure everything works together reliably.

In VS Code, click File in the top menu bar.
Click Open Folder.
Select the fda-drug-safety-agent folder on your Desktop, then click Open.
Open the terminal again with Ctrl+`. If your virtual environment deactivated, reactivate it using the activation command for your platform above.
Create a new file in the Explorer sidebar by clicking the new file icon.
Name the file requirements.txt and paste the following dependencies into it:

langchain==0.3.25
langchain-core==0.3.61
langchain-groq==0.3.2
langgraph==0.4.7
langgraph-supervisor==0.0.14
chromadb==1.5.8
sentence-transformers==3.4.1
deepeval==3.8.6
ragas==0.2.16
requests==2.32.3
python-dotenv==1.1.0

What are these packages?

langchain, langchain-core, langchain-groq: The LangChain framework and its Groq integration for talking to LLMs.
langgraph, langgraph-supervisor: LangGraph for multi-agent orchestration with the supervisor pattern.
chromadb: A local vector database for storing and searching document embeddings.
sentence-transformers: Local HuggingFace embedding models that run on your machine for free.
deepeval, ragas: Evaluation frameworks for testing your RAG pipeline's accuracy.
requests, python-dotenv: Utilities for HTTP requests and loading environment variables.

Save requirements.txt.

Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

I'd like to double check the full code

langchain==0.3.25
langchain-core==0.3.61
langchain-groq==0.3.2
langgraph==0.4.7
langgraph-supervisor==0.0.14
chromadb==1.5.8
sentence-transformers==3.4.1
deepeval==3.8.6
ragas==0.2.16
requests==2.32.3
python-dotenv==1.1.0

Install all the dependencies by running this command:

pip install -r requirements.txt

This takes a few minutes. The sentence-transformers package downloads a local embedding model, so expect a larger download. You should see Successfully installed messages when it finishes.

Seeing install errors?

Make sure (venv) appears in your terminal prompt. If it does not, reactivate your virtual environment before running pip install.
If you see a permissions error, do not use sudo. Deactivate and recreate the virtual environment instead.
On Windows, if the install fails for chromadb, you may need to install the Microsoft C++ Build Tools from the Visual Studio website.

Help me fix my pip install errors.

Create a Groq account and get your API key

Groq provides free access to powerful open-source LLMs like Llama 3.3 70B. The free tier gives you 1,000 requests per day with no credit card required.

Open console.groq.com in your browser.
Click Sign Up and create an account using your Google account or email. No credit card is needed.
After signing in, click API Keys in the left sidebar (or navigate directly to console.groq.com/keys).
Click Create API Key.
Give your key a name like fda-safety-agent and click Submit.
Copy the API key immediately. You will not be able to see it again after closing the dialog.

Pro tip: save your key somewhere safe

Paste your API key into a temporary text file or password manager right away. If you lose it, you will need to create a new one.

Now let's store this key in your project so your code can use it.

In VS Code's Explorer sidebar, create a new file called .env in your fda-drug-safety-agent folder.
Paste the following line into .env, replacing the placeholder with your actual Groq API key:

GROQ_API_KEY=your-groq-api-key-here

Replace your-groq-api-key-here with the API key you just copied from Groq.
Save .env.

Why use a .env file?

A .env file keeps sensitive values like API keys out of your source code. The python-dotenv library loads these values into your Python environment automatically so your code can read them without hardcoding secrets.

Let's verify your Groq connection works before moving on.

Verify your Groq API key works by running this test command in your terminal:

python -c "from dotenv import load_dotenv; load_dotenv(); from langchain_groq import ChatGroq; llm = ChatGroq(model='llama-3.3-70b-versatile', temperature=0); print(llm.invoke('Say hello in one word').content)"

You should see a short greeting like Hello printed in your terminal. This confirms your API key is valid and Groq is reachable.

Seeing an authentication error?

Open your .env file and confirm there are no extra spaces or quotes around your API key. The format should be GROQ_API_KEY=gsk_... with no spaces.
Make sure the .env file is saved in the root of your fda-drug-safety-agent folder, not inside a subfolder.
If your key starts with a different prefix, verify you copied the full key from the Groq console.

Help me fix my Groq API authentication error.

Set up LangSmith for tracing

LangSmith records every LLM call, tool invocation, and agent decision your system makes. This is called tracing. It gives you full visibility into how your multi-agent system behaves.

Open smith.langchain.com in your browser.
Click Sign Up and create a free account using Google, GitHub, or email. No credit card is needed.
When prompted to choose a data region, select US Central. This choice cannot be changed later.
After signing in, navigate to your Settings page.
Click API Keys in the settings menu.
Click Create API Key, then copy the key and save it somewhere safe.

What is LangSmith's free tier?

The free Developer plan gives you 5,000 traces per month. Each multi-agent query uses roughly 4 to 8 traces (one per LLM call). This is more than enough for this project.

Open your .env file and add the following three lines below your GROQ_API_KEY line:

LANGSMITH_API_KEY=your-langsmith-api-key-here
LANGSMITH_TRACING=true
LANGSMITH_PROJECT=fda-drug-safety-agent

What do these variables do?

LANGSMITH_API_KEY: Authenticates your app with the LangSmith platform.
LANGSMITH_TRACING=true: Tells LangChain to send trace data to LangSmith automatically.
LANGSMITH_PROJECT: Groups all your traces under a project name in the LangSmith dashboard.

Replace your-langsmith-api-key-here with the API key you just copied from LangSmith.
Save .env.

Here is what your complete .env file should look like now.

Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

I'd like to double check the full code

GROQ_API_KEY=your-groq-api-key-here
LANGSMITH_API_KEY=your-langsmith-api-key-here
LANGSMITH_TRACING=true
LANGSMITH_PROJECT=fda-drug-safety-agent

Your environment is ready. You have Python, all project dependencies, and two configured cloud services. Next up, you will connect to Groq through LangChain and start talking to an LLM.

Understand LangChain Basics and Connect to Groq

Your development environment is ready with all the packages and API keys you need. The next step is to connect your Python code to Groq's LLM and start sending it questions.

LangChain is the framework that makes this connection simple. In this step, you'll build a basic chatbot to learn the core patterns (messages, chat models, invoke) that power every agent in your final system.

In this step, get ready to:

Build a simple chatbot that sends a question to Groq's Llama 3.3 70B and prints the response.
Add a system prompt that shapes the LLM into a drug safety expert and establish a baseline for how it responds without external data.
Verify that LangSmith is capturing traces of your test queries.

Create a simple chatbot with ChatGroq

LangChain's ChatGroq class connects your Python code to Groq's API. You call its .invoke() method with a question, and it sends that question to the LLM and returns the response.

In VS Code, create a new file called chatbot.py in the same folder as your config.py file.
Add the following code to chatbot.py:

from langchain_groq import ChatGroq
from config import GROQ_MODEL, GROQ_API_KEY

# Set up the LLM connection to Groq
llm = ChatGroq(
    model=GROQ_MODEL,
    temperature=0,
    max_retries=2,
    api_key=GROQ_API_KEY,
)

# Send a simple question to the LLM
response = llm.invoke("What is metformin used for?")
print(response.content)

What does this code do?

ChatGroq() creates a connection to Groq's API using your API key and model name from config.py. The temperature=0 setting makes the LLM give consistent, deterministic answers.
llm.invoke() sends a message to Groq's Llama 3.3 70B model and waits for the response. This is the core method you'll use throughout the entire project.
response.content contains the LLM's text reply as a plain string.

Save chatbot.py.
Run the script by entering this command in your terminal:

python chatbot.py

You should see a paragraph of text explaining what metformin is used for, pulled entirely from the LLM's training data.

Seeing an error?

If you see AuthenticationError, double-check that your GROQ_API_KEY in .env is correct. Copy it fresh from console.groq.com if needed.
If you see ModuleNotFoundError: No module named 'langchain_groq', make sure your virtual environment is activated before running the script.
If you see a rate limit error, wait a minute and try again. Groq's free tier allows 30 requests per minute.

Still stuck? Help me debug my ChatGroq connection error

Add a system prompt to establish a baseline

Right now, your chatbot answers as a general-purpose assistant. A system prompt tells the LLM what role to play and how to behave.

You'll also switch from passing a plain string to using LangChain's message types: SystemMessage and HumanMessage. These give you precise control over the conversation structure that the LLM receives.

Replace the entire contents of chatbot.py with the following code:

from langchain_groq import ChatGroq
from langchain_core.messages import SystemMessage, HumanMessage
from config import GROQ_MODEL, GROQ_API_KEY

# Set up the LLM connection to Groq
llm = ChatGroq(
    model=GROQ_MODEL,
    temperature=0,
    max_retries=2,
    api_key=GROQ_API_KEY,
)

# Define the drug safety expert system prompt
system_prompt = SystemMessage(content=(
    "You are a drug safety expert. "
    "Provide accurate, concise information about medications, "
    "their side effects, warnings, and interactions. "
    "Always note that your information is for educational purposes only "
    "and not a substitute for professional medical advice."
))

# Test with a drug safety question
question = HumanMessage(content="What are the most common side effects of metformin?")

# Send the system prompt and question together
response = llm.invoke([system_prompt, question])
print(f"Question: {question.content}\n")
print(f"Response: {response.content}")

What changed here?

SystemMessage sets the LLM's persona before it processes any user input. The LLM reads this first and shapes all of its responses accordingly.
HumanMessage wraps the user's question. Together with the system prompt, these two message types form a structured conversation that the LLM processes in order.
llm.invoke([system_prompt, question]) now takes a list of messages instead of a single string. This is the same pattern your agents will use later.

Save chatbot.py and run it again:

python chatbot.py

You should see a more focused response about metformin's side effects, with a medical disclaimer at the end.

Try changing the question to test a few more drugs. Replace the content inside HumanMessage with each of these, saving and re-running each time:

"What are the warnings for ibuprofen?"
"What drug interactions should I know about for lisinopril?"

Why does this baseline matter?

The LLM is answering from its training data alone. The responses sound confident, but you have no way to verify where the information came from or whether it's current.

This is exactly the problem that RAG (Retrieval-Augmented Generation) solves. By the end of this project, your system will pull answers from real FDA data and cite its sources.

Getting a different response each time?

Make sure temperature=0 is set in your ChatGroq() constructor. A temperature of 0 tells the model to pick the most likely response every time, which keeps outputs consistent.

Need help? Why are my LLM responses different every time I run the script?

Awesome, I've got everything!

Great. Double check you've saved chatbot.py before moving on.

I'd like to double check the full code

Here is the complete chatbot.py file. Make sure yours matches:

from langchain_groq import ChatGroq
from langchain_core.messages import SystemMessage, HumanMessage
from config import GROQ_MODEL, GROQ_API_KEY

# Set up the LLM connection to Groq
llm = ChatGroq(
    model=GROQ_MODEL,
    temperature=0,
    max_retries=2,
    api_key=GROQ_API_KEY,
)

# Define the drug safety expert system prompt
system_prompt = SystemMessage(content=(
    "You are a drug safety expert. "
    "Provide accurate, concise information about medications, "
    "their side effects, warnings, and interactions. "
    "Always note that your information is for educational purposes only "
    "and not a substitute for professional medical advice."
))

# Test with a drug safety question
question = HumanMessage(content="What are the most common side effects of metformin?")

# Send the system prompt and question together
response = llm.invoke([system_prompt, question])
print(f"Question: {question.content}\n")
print(f"Response: {response.content}")

Verify LangSmith is capturing your traces

Every time you called llm.invoke(), LangSmith was silently recording a trace of what happened. A trace captures the full lifecycle of an LLM call: the input messages, the model used, the response, how long it took, and how many tokens it consumed.

Open your browser and go to smith.langchain.com.
Sign in with the account you created in Step 1.
Click on the fda-drug-safety-agent project in your project list.

You should see a list of traces from the test questions you ran. Each trace shows the model name, latency, token count, and input/output messages.

Click on any trace to expand it.
Check that the Input section shows your system prompt and question.
Check that the Output section shows the LLM's response.

Why does tracing matter?

When your multi-agent system is running, a single user question might trigger 4-8 LLM calls across different agents. Without tracing, debugging a wrong answer would be like searching in the dark.

LangSmith shows you exactly which agent was called, what it received, and what it returned. You'll rely on this heavily in later steps.

Don't see any traces?

Confirm that your .env file has LANGSMITH_TRACING=true (not True or TRUE). It must be lowercase.
Make sure LANGSMITH_API_KEY in your .env file is correct. Copy it fresh from the LangSmith settings page if needed.
Check that LANGSMITH_PROJECT=fda-drug-safety-agent is set in your .env file. If this is missing, traces go to a default project that might not be visible.
Re-run python chatbot.py after fixing your .env file, then refresh the LangSmith dashboard.

Still stuck? Help me figure out why my LangSmith traces are not showing up

Your chatbot works and LangSmith is recording every LLM call. The LLM's answers sound plausible, but they come from training data with no citations or verifiable sources. In the next step, you'll fix that by ingesting real FDA drug labels into a vector database so your system can ground its answers in official data.

Ingest FDA Drug Labels into ChromaDB

In the previous step, you sent drug safety questions to the LLM and saw how it responds without any external data. The answers were generic and potentially inaccurate because the model was relying purely on its training data.

Now you will fix that by feeding your system real FDA drug labels. You will fetch official drug label data from the openFDA API, split it into searchable chunks, and store those chunks as embeddings in a local ChromaDB vector database. This is the foundation of your RAG pipeline.

In this step, get ready to:

Fetch drug labels from the openFDA API for five common drugs.
Chunk the drug label text into searchable pieces with metadata.
Embed the chunks using a local HuggingFace model and store them in ChromaDB.

Add ingestion config to your project

Your config.py file currently holds Groq and LangSmith settings. You need to add configuration for the drugs to fetch, the openFDA API endpoints, ChromaDB storage, the embedding model, and chunking parameters.

In config.py, find the line that says LANGSMITH_API_KEY = os.getenv("LANGSMITH_API_KEY"), then add the following code below it:

TARGET_DRUGS = [
    "metformin",
    "ibuprofen",
    "lisinopril",
    "amoxicillin",
    "atorvastatin",
]

CHROMA_COLLECTION_NAME = "fda_drug_labels"
CHROMA_PERSIST_DIR = "./chroma_db"

OPENFDA_LABEL_URL = "https://api.fda.gov/drug/label.json"
OPENFDA_EVENT_URL = "https://api.fda.gov/drug/event.json"

EMBEDDING_MODEL = "all-MiniLM-L6-v2"

CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200

What do these config values control?

TARGET_DRUGS lists five common drugs whose labels you will ingest.
CHROMA_PERSIST_DIR tells ChromaDB to save data to a local folder so it survives restarts.
EMBEDDING_MODEL names the HuggingFace model that converts text into numerical vectors. all-MiniLM-L6-v2 runs entirely on your machine at zero cost.
CHUNK_SIZE and CHUNK_OVERLAP control how drug label text is split into smaller pieces for embedding.

Save config.py.

You won't see a visible change yet. These values are used by the ingestion script you will write next.

Fetch drug labels from the openFDA API

The openFDA API provides free, public access to FDA drug label data. No API key is required. Your script will call the /drug/label.json endpoint to fetch the official label for each drug, then extract key sections like warnings, adverse reactions, and drug interactions.

Create a new file called ingest.py in your project folder (the same folder as config.py).
Add the imports and the first two functions by copying the following code into ingest.py:

import requests
import chromadb
from sentence_transformers import SentenceTransformer
from langchain_text_splitters import RecursiveCharacterTextSplitter
from config import (
    TARGET_DRUGS,
    OPENFDA_LABEL_URL,
    CHROMA_COLLECTION_NAME,
    CHROMA_PERSIST_DIR,
    EMBEDDING_MODEL,
    CHUNK_SIZE,
    CHUNK_OVERLAP,
)


def fetch_drug_label(drug_name):
    """Fetch drug label from openFDA API."""
    params = {
        "search": f'openfda.generic_name:"{drug_name}"',
        "limit": 1,
    }
    response = requests.get(OPENFDA_LABEL_URL, params=params)
    if response.status_code == 200:
        results = response.json().get("results", [])
        if results:
            return results[0]
    return None


def extract_label_sections(label, drug_name):
    """Extract key sections from a drug label."""
    sections = [
        "indications_and_usage",
        "warnings",
        "adverse_reactions",
        "drug_interactions",
        "dosage_and_administration",
        "contraindications",
    ]
    documents = []
    for section in sections:
        content = label.get(section, [])
        if content:
            text = content[0] if isinstance(content, list) else content
            documents.append({
                "text": text,
                "drug_name": drug_name,
                "section": section,
                "source": f"openFDA drug label: {drug_name}",
            })
    return documents

What do these functions do?

fetch_drug_label() calls the openFDA API, searching by generic drug name. It returns the first matching label as a dictionary, or None if no result is found.
extract_label_sections() pulls six key sections from the raw label (warnings, adverse reactions, drug interactions, etc.). Each section becomes a document with metadata tracking the drug name, section name, and source.

Save ingest.py.

You won't see a change yet. The next chunk adds the function that splits these documents into smaller pieces.

Chunk the drug label text

Drug labels can be thousands of characters long. An LLM cannot process them efficiently in one piece, and embedding a giant block of text produces a low-quality vector. Chunking splits each label section into smaller, overlapping pieces so each piece captures a focused topic.

Why 1000 characters with 200 overlap?

A 1000-character chunk is large enough to hold a complete thought (like a full warning paragraph) but small enough to embed accurately. The 200-character overlap ensures that sentences straddling a chunk boundary appear in both chunks, so the retriever does not miss relevant context.

In ingest.py, add the following function below extract_label_sections:

def chunk_documents(documents):
    """Split documents into chunks with metadata."""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=CHUNK_SIZE,
        chunk_overlap=CHUNK_OVERLAP,
        separators=["\n\n", "\n", ". ", " "],
    )
    chunks = []
    for doc in documents:
        splits = splitter.split_text(doc["text"])
        for i, split in enumerate(splits):
            chunks.append({
                "text": split,
                "metadata": {
                    "drug_name": doc["drug_name"],
                    "section": doc["section"],
                    "source": doc["source"],
                    "chunk_index": i,
                },
            })
    return chunks

What does this code do?

RecursiveCharacterTextSplitter from LangChain tries to split on double newlines first, then single newlines, then sentence boundaries, then spaces. This keeps paragraphs and sentences intact whenever possible.
Each chunk carries its parent document's metadata (drug_name, section, source). This metadata lets the retriever cite exactly where a piece of information came from.

Save ingest.py.

You won't see a change yet. The next function embeds these chunks and stores them in ChromaDB, where you can run and verify the full pipeline.

Embed chunks and store them in ChromaDB

With the fetching and chunking functions ready, you need the final two pieces: a function that embeds chunks and writes them to ChromaDB, and a main() function that orchestrates the entire pipeline.

In ingest.py, add the following two functions and entry point below chunk_documents:

def ingest_to_chromadb(chunks):
    """Embed chunks and store in ChromaDB."""
    model = SentenceTransformer(EMBEDDING_MODEL)
    client = chromadb.PersistentClient(path=CHROMA_PERSIST_DIR)

    try:
        client.delete_collection(CHROMA_COLLECTION_NAME)
    except ValueError:
        pass

    collection = client.create_collection(
        name=CHROMA_COLLECTION_NAME,
        metadata={"hnsw:space": "cosine"},
    )

    texts = [c["text"] for c in chunks]
    embeddings = model.encode(texts).tolist()
    ids = [f"chunk_{i}" for i in range(len(chunks))]
    metadatas = [c["metadata"] for c in chunks]

    collection.add(
        ids=ids,
        embeddings=embeddings,
        documents=texts,
        metadatas=metadatas,
    )
    print(f"Ingested {len(chunks)} chunks into ChromaDB.")
    return collection


def main():
    all_documents = []
    for drug in TARGET_DRUGS:
        print(f"Fetching label for {drug}...")
        label = fetch_drug_label(drug)
        if label:
            docs = extract_label_sections(label, drug)
            all_documents.extend(docs)
            print(f"  Found {len(docs)} sections for {drug}")
        else:
            print(f"  No label found for {drug}")

    chunks = chunk_documents(all_documents)
    print(f"\nTotal chunks: {len(chunks)}")

    ingest_to_chromadb(chunks)
    print("Ingestion complete!")


if __name__ == "__main__":
    main()

What does this code do?

ingest_to_chromadb() loads the all-MiniLM-L6-v2 model locally and converts every chunk into a 384-dimensional vector. It then creates a persistent ChromaDB collection using cosine similarity and stores the vectors alongside their text and metadata.
The try: client.delete_collection() block deletes any existing collection first. This makes the script safe to re-run without creating duplicates.
main() ties the entire pipeline together: it loops through each drug, fetches labels, extracts sections, chunks them, and ingests them into ChromaDB.

Save ingest.py.
Run the ingestion script by running this command in your terminal:

python ingest.py

What should I see?

The first run downloads the all-MiniLM-L6-v2 model (about 80 MB), so it may take a minute. After that, you should see output like this:

Fetching label for metformin...
  Found 6 sections for metformin
Fetching label for ibuprofen...
  Found 6 sections for ibuprofen
Fetching label for lisinopril...
  Found 6 sections for lisinopril
Fetching label for amoxicillin...
  Found 6 sections for amoxicillin
Fetching label for atorvastatin...
  Found 6 sections for atorvastatin

Total chunks: <number>
Ingested <number> chunks into ChromaDB.
Ingestion complete!

The exact chunk count depends on how long each drug's label sections are. Expect roughly 80 to 150 chunks total.

Confirm that a chroma_db folder appeared in your project directory by running:

ls

You should see chroma_db listed alongside config.py and ingest.py. This folder contains your persisted vector database.

Seeing an error when running the script?

If you see ModuleNotFoundError: No module named 'chromadb' or similar, make sure your virtual environment is activated and that you ran pip install -r requirements.txt in Step 1.
If you see No label found for every drug, check your internet connection. The openFDA API does not require a key but it does need network access.
If the script hangs on the first run, it is likely downloading the embedding model. Give it a minute or two to complete.

Still stuck? Help me debug my ingest.py script

Awesome, I've got everything!

Double check you have saved both config.py and ingest.py.

I'd like to double check the full code

Here is the full config.py at this point:

import os
from dotenv import load_dotenv

load_dotenv()

GROQ_API_KEY = os.getenv("GROQ_API_KEY")
GROQ_MODEL = "llama-3.3-70b-versatile"

LANGSMITH_API_KEY = os.getenv("LANGSMITH_API_KEY")

TARGET_DRUGS = [
    "metformin",
    "ibuprofen",
    "lisinopril",
    "amoxicillin",
    "atorvastatin",
]

CHROMA_COLLECTION_NAME = "fda_drug_labels"
CHROMA_PERSIST_DIR = "./chroma_db"

OPENFDA_LABEL_URL = "https://api.fda.gov/drug/label.json"
OPENFDA_EVENT_URL = "https://api.fda.gov/drug/event.json"

EMBEDDING_MODEL = "all-MiniLM-L6-v2"

CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200

Here is the full ingest.py:

import requests
import chromadb
from sentence_transformers import SentenceTransformer
from langchain_text_splitters import RecursiveCharacterTextSplitter
from config import (
    TARGET_DRUGS,
    OPENFDA_LABEL_URL,
    CHROMA_COLLECTION_NAME,
    CHROMA_PERSIST_DIR,
    EMBEDDING_MODEL,
    CHUNK_SIZE,
    CHUNK_OVERLAP,
)


def fetch_drug_label(drug_name):
    """Fetch drug label from openFDA API."""
    params = {
        "search": f'openfda.generic_name:"{drug_name}"',
        "limit": 1,
    }
    response = requests.get(OPENFDA_LABEL_URL, params=params)
    if response.status_code == 200:
        results = response.json().get("results", [])
        if results:
            return results[0]
    return None


def extract_label_sections(label, drug_name):
    """Extract key sections from a drug label."""
    sections = [
        "indications_and_usage",
        "warnings",
        "adverse_reactions",
        "drug_interactions",
        "dosage_and_administration",
        "contraindications",
    ]
    documents = []
    for section in sections:
        content = label.get(section, [])
        if content:
            text = content[0] if isinstance(content, list) else content
            documents.append({
                "text": text,
                "drug_name": drug_name,
                "section": section,
                "source": f"openFDA drug label: {drug_name}",
            })
    return documents


def chunk_documents(documents):
    """Split documents into chunks with metadata."""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=CHUNK_SIZE,
        chunk_overlap=CHUNK_OVERLAP,
        separators=["\n\n", "\n", ". ", " "],
    )
    chunks = []
    for doc in documents:
        splits = splitter.split_text(doc["text"])
        for i, split in enumerate(splits):
            chunks.append({
                "text": split,
                "metadata": {
                    "drug_name": doc["drug_name"],
                    "section": doc["section"],
                    "source": doc["source"],
                    "chunk_index": i,
                },
            })
    return chunks


def ingest_to_chromadb(chunks):
    """Embed chunks and store in ChromaDB."""
    model = SentenceTransformer(EMBEDDING_MODEL)
    client = chromadb.PersistentClient(path=CHROMA_PERSIST_DIR)

    try:
        client.delete_collection(CHROMA_COLLECTION_NAME)
    except ValueError:
        pass

    collection = client.create_collection(
        name=CHROMA_COLLECTION_NAME,
        metadata={"hnsw:space": "cosine"},
    )

    texts = [c["text"] for c in chunks]
    embeddings = model.encode(texts).tolist()
    ids = [f"chunk_{i}" for i in range(len(chunks))]
    metadatas = [c["metadata"] for c in chunks]

    collection.add(
        ids=ids,
        embeddings=embeddings,
        documents=texts,
        metadatas=metadatas,
    )
    print(f"Ingested {len(chunks)} chunks into ChromaDB.")
    return collection


def main():
    all_documents = []
    for drug in TARGET_DRUGS:
        print(f"Fetching label for {drug}...")
        label = fetch_drug_label(drug)
        if label:
            docs = extract_label_sections(label, drug)
            all_documents.extend(docs)
            print(f"  Found {len(docs)} sections for {drug}")
        else:
            print(f"  No label found for {drug}")

    chunks = chunk_documents(all_documents)
    print(f"\nTotal chunks: {len(chunks)}")

    ingest_to_chromadb(chunks)
    print("Ingestion complete!")


if __name__ == "__main__":
    main()

Your vector database is loaded with real FDA drug data. Next up, you will build the retrieval tools that let your agents search this data and query live adverse event reports.

Build the RAG Pipeline with Retrieval Tools

Your ChromaDB vector database is loaded with FDA drug label data. The next challenge is giving your AI agents a structured way to search through it.

In this step, you'll create two LangChain tools. One queries your vector database for drug label information. The other calls the live openFDA API for real adverse event statistics.

In this step, get ready to:

Create a vector search tool that retrieves drug label chunks from ChromaDB.
Create an adverse event query tool that fetches live data from the openFDA API.
Test both tools independently to verify they return useful results.

Create the search_drug_labels tool

LangChain's @tool decorator turns a regular Python function into something an LLM agent can call. The agent reads the function's docstring to decide when to use it, and LangChain handles the input/output plumbing automatically.

Create a new file called tools.py in your project folder (the same folder that contains ingest.py and config.py).
Add the imports and module-level setup by copying this code into tools.py:

import requests
import chromadb
from sentence_transformers import SentenceTransformer
from langchain_core.tools import tool
from config import (
    CHROMA_COLLECTION_NAME,
    CHROMA_PERSIST_DIR,
    EMBEDDING_MODEL,
    OPENFDA_EVENT_URL,
)

# Initialize the embedding model and ChromaDB client at module level
# so they're reused across all tool calls
embedding_model = SentenceTransformer(EMBEDDING_MODEL)
chroma_client = chromadb.PersistentClient(path=CHROMA_PERSIST_DIR)

What does this code do?

The @tool import from langchain_core.tools is a decorator that converts a plain function into a LangChain tool. Your agents will use this to interact with external data sources.
embedding_model and chroma_client are created at the module level so they're initialized once and reused every time a tool is called. This avoids reloading the embedding model on every query.

You won't see any output yet. The next section adds the first tool function that uses these imports.

The first tool your agents need is a way to search the drug label data you ingested into ChromaDB. This tool takes a natural language query, embeds it with the same HuggingFace model used during ingestion, and retrieves the most relevant chunks.

In tools.py, add the search_drug_labels function below the module-level setup you just wrote:

@tool
def search_drug_labels(query: str) -> str:
    """Search FDA drug labels in the vector database for relevant information about drugs, side effects, warnings, dosage, and interactions."""
    # Connect to the existing ChromaDB collection
    collection = chroma_client.get_collection(CHROMA_COLLECTION_NAME)
    # Embed the search query using the same model used during ingestion
    query_embedding = embedding_model.encode([query]).tolist()

    # Retrieve the top 5 most relevant chunks
    results = collection.query(
        query_embeddings=query_embedding,
        n_results=5,
        include=["documents", "metadatas"],
    )

    # Return a message if no results found
    if not results["documents"][0]:
        return "No relevant drug label information found."

    # Format each result with its source metadata for citation
    formatted = []
    for doc, meta in zip(results["documents"][0], results["metadatas"][0]):
        formatted.append(
            f"[Source: {meta['source']} | Section: {meta['section']}]\n{doc}"
        )
    return "\n\n---\n\n".join(formatted)

What does this code do?

The docstring is critical. LangChain agents read it to decide when this tool is relevant to a question. A vague docstring means the agent won't know when to call the tool.
query_embedding = embedding_model.encode([query]).tolist() converts the search query into a vector using the same all-MiniLM-L6-v2 model that embedded the original chunks. Using the same model for both ingestion and querying is essential for accurate similarity search.
n_results=5 retrieves the top 5 most similar chunks. Each result includes the original text and its metadata (drug name, section, source) so the agent can cite where the information came from.

Save tools.py.

You won't see a change yet. We'll test this tool after adding the second one.

Create the query_adverse_events tool

Your first tool searches static drug label data stored locally. This second tool goes live. It queries the openFDA FAERS (FDA Adverse Event Reporting System) database to fetch real-time counts of reported side effects for any drug.

In tools.py, add the query_adverse_events function below the search_drug_labels function:

@tool
def query_adverse_events(drug_name: str) -> str:
    """Query the FDA adverse event database for reported side effects of a specific drug. Returns the top 10 most commonly reported reactions."""
    # Build the API request to search by generic drug name
    params = {
        "search": f'patient.drug.openfda.generic_name:"{drug_name}"',
        "count": "patient.reaction.reactionmeddrapt.exact",
        "limit": 10,
    }
    response = requests.get(OPENFDA_EVENT_URL, params=params)

    # Handle API errors
    if response.status_code != 200:
        return f"Error querying FDA adverse events for {drug_name}: HTTP {response.status_code}"

    results = response.json().get("results", [])
    if not results:
        return f"No adverse event data found for {drug_name}."

    # Format the results with reaction names and report counts
    lines = [f"Top reported adverse events for {drug_name} (from FDA FAERS database):"]
    for r in results:
        lines.append(f"  - {r['term']}: {r['count']} reports")
    return "\n".join(lines)

What does this code do?

The count parameter tells the openFDA API to aggregate results by the exact reaction term. Instead of returning individual reports, it returns the top 10 most frequently reported adverse reactions with their counts.
The openFDA API requires no API key for basic use. It allows up to 240 requests per minute without a key, which is more than enough for this project.
Error handling is important here because this tool calls a live API. Network issues or invalid drug names could cause failures, and the agent needs a clear error message to handle those gracefully.

Why two tools instead of one?

These tools serve fundamentally different purposes. search_drug_labels searches your local vector database for official drug label text (warnings, dosage, interactions). query_adverse_events hits a live government API for real-world side effect statistics. Together, they give your agents both official documentation and real-world data.

Save tools.py.

Awesome, I've got everything!

Great. Double check you've saved tools.py before moving on to testing.

I'd like to double check the full code

import requests
import chromadb
from sentence_transformers import SentenceTransformer
from langchain_core.tools import tool
from config import (
    CHROMA_COLLECTION_NAME,
    CHROMA_PERSIST_DIR,
    EMBEDDING_MODEL,
    OPENFDA_EVENT_URL,
)


embedding_model = SentenceTransformer(EMBEDDING_MODEL)
chroma_client = chromadb.PersistentClient(path=CHROMA_PERSIST_DIR)


@tool
def search_drug_labels(query: str) -> str:
    """Search FDA drug labels in the vector database for relevant information about drugs, side effects, warnings, dosage, and interactions."""
    collection = chroma_client.get_collection(CHROMA_COLLECTION_NAME)
    query_embedding = embedding_model.encode([query]).tolist()

    results = collection.query(
        query_embeddings=query_embedding,
        n_results=5,
        include=["documents", "metadatas"],
    )

    if not results["documents"][0]:
        return "No relevant drug label information found."

    formatted = []
    for doc, meta in zip(results["documents"][0], results["metadatas"][0]):
        formatted.append(
            f"[Source: {meta['source']} | Section: {meta['section']}]\n{doc}"
        )
    return "\n\n---\n\n".join(formatted)


@tool
def query_adverse_events(drug_name: str) -> str:
    """Query the FDA adverse event database for reported side effects of a specific drug. Returns the top 10 most commonly reported reactions."""
    params = {
        "search": f'patient.drug.openfda.generic_name:"{drug_name}"',
        "count": "patient.reaction.reactionmeddrapt.exact",
        "limit": 10,
    }
    response = requests.get(OPENFDA_EVENT_URL, params=params)

    if response.status_code != 200:
        return f"Error querying FDA adverse events for {drug_name}: HTTP {response.status_code}"

    results = response.json().get("results", [])
    if not results:
        return f"No adverse event data found for {drug_name}."

    lines = [f"Top reported adverse events for {drug_name} (from FDA FAERS database):"]
    for r in results:
        lines.append(f"  - {r['term']}: {r['count']} reports")
    return "\n".join(lines)

Test both tools independently

Before wiring these tools into your agents, you should verify they actually return useful data. LangChain tools can be called directly using their .invoke() method.

Test the vector search tool by running this command in your terminal:

python -c "from tools import search_drug_labels; print(search_drug_labels.invoke('metformin side effects'))"

You should see five chunks of text, each prefixed with a source tag like [Source: openFDA drug label: metformin | Section: adverse_reactions]. The text should contain real FDA drug label content about metformin.

Seeing an error about the collection?

If you see a ValueError about the collection not existing, make sure you ran ingest.py in the previous step. The search_drug_labels tool depends on the ChromaDB collection that ingest.py creates.
If you see an ImportError, check that you're running the command from the same folder where tools.py and config.py live.
Still stuck? Help me debug why search_drug_labels is failing. I'm getting this error when I run the test command.

Test the adverse event tool by running this command:

python -c "from tools import query_adverse_events; print(query_adverse_events.invoke('ibuprofen'))"

You should see a list of the top 10 reported adverse events for ibuprofen, each with a reaction name and report count. For example, you might see entries like Drug ineffective: 12345 reports.

Seeing an HTTP error or empty results?

An HTTP 404 or 500 error usually means the openFDA API is temporarily unavailable. Wait a minute and try again.
If you see No adverse event data found, double check that you're passing a valid generic drug name (e.g. ibuprofen not a brand name like Advil).
Need help? Help me debug why query_adverse_events is returning an error or empty results when I test it.

What is the RAG pattern you just built?

RAG (Retrieval-Augmented Generation) is the pattern of retrieving relevant data first, then passing it to an LLM to generate an answer. Your search_drug_labels tool is the "retrieval" half of RAG. In the next step, you'll connect it to an agent that uses the retrieved data to generate grounded answers.

Both tools are working and returning real FDA data. Next up, you'll build the multi-agent system with LangGraph that coordinates specialized agents to use these tools together.

Build the Multi-Agent System with LangGraph

Your retrieval tools are ready. You have search_drug_labels querying your ChromaDB vector database and query_adverse_events pulling live data from the FDA API.

Now you need something to coordinate these tools. In this step, you'll build a multi-agent system using LangGraph where a supervisor agent routes questions to three specialized agents, each with its own tools and expertise.

In this step, get ready to:

Understand how LangGraph uses graphs to orchestrate multiple agents.
Create three specialized agents with different tools and prompts.
Wire the agents together using a supervisor that routes questions to the right agent.

Understand how LangGraph orchestrates agents

A single LLM can answer questions, but it has no way to decide when to search a database versus when to call an API versus when to write a report. That decision-making layer is what LangGraph provides.

What is LangGraph?

LangGraph models your AI application as a graph. Each node in the graph is a step your application can take (calling an LLM, running a tool, making a decision). Edges connect nodes and define the flow.

The core building blocks are:

StateGraph defines the overall graph structure and what data flows between nodes.
Nodes are the individual processing steps (an LLM call, a tool call, a routing decision).
Edges connect nodes and control which node runs next.
compile() turns the graph definition into a runnable application.

You could build these graphs from scratch using StateGraph, adding nodes and edges manually. But LangGraph provides two powerful shortcuts that handle the wiring for you.

What are create_react_agent and create_supervisor?

create_react_agent builds a complete agent graph. It creates an LLM node and a tool node, then wires them together so the LLM can decide when to call its tools and when to return a final answer.
create_supervisor builds a supervisor graph that sits above multiple agents. It routes incoming questions to the right agent and coordinates the overall flow.

Under the hood, both functions create a StateGraph with nodes, edges, and typed state. You get the benefits of graph-based orchestration without manually wiring every connection.

Time to put this into practice. You'll create a new file called agents.py that defines your three agents and wires them together with a supervisor.

Create the retrieval agent

Your first agent specializes in searching the FDA drug label data you ingested into ChromaDB. It uses the search_drug_labels tool and returns raw information without interpretation.

Create a new file called agents.py in your project folder (the same folder as tools.py).
Add the imports, LLM setup, and retrieval agent by copying this code:

from langchain_groq import ChatGroq
from langgraph.prebuilt import create_react_agent
from langgraph_supervisor import create_supervisor
from tools import search_drug_labels, query_adverse_events
from config import GROQ_MODEL, GROQ_API_KEY


# Initialize the LLM that all agents will share
llm = ChatGroq(
    model=GROQ_MODEL,
    temperature=0,
    max_retries=2,
    api_key=GROQ_API_KEY,
)

# Agent 1: searches FDA drug labels in ChromaDB
retrieval_agent = create_react_agent(
    model=llm,
    tools=[search_drug_labels],
    name="retrieval_agent",
    prompt=(
        "You are a drug label retrieval specialist. "
        "Use the search_drug_labels tool to find relevant FDA drug label information. "
        "Always cite the source and section of the information you find. "
        "Return the raw information without interpretation."
    ),
)

What does this code do?

The imports bring in ChatGroq (the LLM), create_react_agent (builds individual agents), and create_supervisor (coordinates agents). It also imports the two tools you built in the previous step.
All three agents share the same llm instance. temperature=0 makes the output deterministic, which is important for consistent safety information.
create_react_agent builds a full agent graph with an LLM node and a tool node. The prompt tells the agent its role. The name is how the supervisor identifies it.

You won't see any output yet. The next chunk adds the remaining two agents.

Add the adverse event and report agents

Your second agent queries the live FDA adverse event database. Your third agent has no tools at all. Its job is to synthesize the findings from the other agents into a structured report.

In agents.py, add both agents below the retrieval_agent you just created:

# Agent 2: queries live FDA adverse event data
adverse_event_agent = create_react_agent(
    model=llm,
    tools=[query_adverse_events],
    name="adverse_event_agent",
    prompt=(
        "You are an adverse event data analyst. "
        "Use the query_adverse_events tool to look up reported side effects from the FDA FAERS database. "
        "Present the data clearly with report counts. "
        "Note that adverse event reports do not prove causation."
    ),
)

# Agent 3: synthesizes findings into a structured report
report_agent = create_react_agent(
    model=llm,
    tools=[],
    name="report_agent",
    prompt=(
        "You are a drug safety report writer. "
        "Synthesize the information provided by other agents into a clear, structured safety report. "
        "Always include citations indicating where each piece of information came from. "
        "Include a disclaimer that this is informational only and not medical advice."
    ),
)

What does this code do?

adverse_event_agent gets the query_adverse_events tool. Its prompt tells it to present adverse event counts clearly and note that reports don't prove causation.
report_agent gets an empty tools list (tools=[]). This agent doesn't fetch data. It reads the messages from the other agents and writes a final report with citations and a medical disclaimer.
Each agent's prompt defines its specialization. The retrieval agent fetches raw data, the adverse event agent fetches statistics, and the report agent synthesizes everything.

You won't see output yet. The next substep wires these agents together with a supervisor.

Why does the report agent have no tools?

The report agent works entirely from the message history. By the time the supervisor routes a question to it, the retrieval agent and adverse event agent have already fetched the data and written their findings into the shared message state. The report agent reads those messages and synthesizes them into a single structured response.

Build the supervisor graph and run it

Your three agents exist, but nothing connects them. The supervisor is the orchestration layer that decides which agent handles each question.

What is the supervisor pattern?

In the supervisor pattern, one LLM acts as a manager. It reads the user's question, decides which agent should handle it, and coordinates the flow between agents. The supervisor can route to one agent for simple questions or chain multiple agents together for complex ones.

In agents.py, add the supervisor, the compiled app, and the ask function below the report_agent:

# Supervisor coordinates all three agents
supervisor = create_supervisor(
    [retrieval_agent, adverse_event_agent, report_agent],
    model=llm,
    prompt=(
        "You are a drug safety investigation supervisor coordinating three agents:\n"
        "- retrieval_agent: searches FDA drug label data (use for drug info, warnings, dosage, interactions)\n"
        "- adverse_event_agent: queries FDA adverse event reports (use for side effect statistics)\n"
        "- report_agent: writes final safety reports (use after other agents have gathered data)\n\n"
        "For most questions, first send the query to retrieval_agent, "
        "then to adverse_event_agent for complementary data, "
        "then to report_agent to synthesize a final answer.\n"
        "For simple factual questions, one agent may be sufficient."
    ),
)

# Compile the supervisor into a runnable graph
app = supervisor.compile()


def ask(question: str) -> str:
    """Ask the drug safety agent a question."""
    result = app.invoke({
        "messages": [{"role": "user", "content": question}],
    })
    return result["messages"][-1].content


if __name__ == "__main__":
    question = "What are the most common side effects of metformin?"
    print(f"Question: {question}\n")
    answer = ask(question)
    print(f"Answer:\n{answer}")

What does this code do?

create_supervisor takes the list of three agents, the LLM, and a prompt that describes each agent's capabilities. The supervisor uses this prompt to decide which agent to route each question to.
supervisor.compile() turns the graph definition into a runnable application stored in app.
The ask() function sends a question to the supervisor as a user message. The supervisor routes it through the agent graph and returns the final response.
The if __name__ block lets you run this file directly to test with a sample question about metformin.

Save agents.py.
Run your multi-agent system by running this command in your terminal:

python agents.py

This takes 15-30 seconds because the supervisor makes multiple LLM calls. You should see the question printed, followed by a structured safety report about metformin's side effects with cited sources.

What is happening behind the scenes?

When you ask about metformin side effects, the supervisor typically follows this flow:

The supervisor reads your question and routes it to retrieval_agent.
The retrieval agent calls search_drug_labels to find relevant chunks from your ChromaDB database.
The supervisor routes to adverse_event_agent to get real-world adverse event statistics.
The supervisor routes to report_agent to synthesize everything into a final report.

Each step is a separate LLM call, which is why the response takes longer than a single chat completion. A single query typically uses 4-8 LLM calls.

Not seeing any output?

Make sure you ran python ingest.py in the previous step. The retrieval agent needs data in ChromaDB to search.
Check that your .env file has a valid GROQ_API_KEY. An invalid key causes a silent failure.
If you see a rate limit error, wait 60 seconds and try again. The Groq free tier allows 30 requests per minute.

Still stuck? Ask for help.

Pro tip: rate limits

Each multi-agent query uses 4-8 LLM calls. The Groq free tier gives you 1,000 requests per day on Llama 3.3 70B. That means you can run dozens of queries throughout this project without hitting the limit.

Awesome, I've got everything!

Great. Double check you've saved agents.py before moving on.

I'd like to double check the full code

Here is the complete agents.py file. Make sure yours matches exactly.

from langchain_groq import ChatGroq
from langgraph.prebuilt import create_react_agent
from langgraph_supervisor import create_supervisor
from tools import search_drug_labels, query_adverse_events
from config import GROQ_MODEL, GROQ_API_KEY


llm = ChatGroq(
    model=GROQ_MODEL,
    temperature=0,
    max_retries=2,
    api_key=GROQ_API_KEY,
)

retrieval_agent = create_react_agent(
    model=llm,
    tools=[search_drug_labels],
    name="retrieval_agent",
    prompt=(
        "You are a drug label retrieval specialist. "
        "Use the search_drug_labels tool to find relevant FDA drug label information. "
        "Always cite the source and section of the information you find. "
        "Return the raw information without interpretation."
    ),
)

adverse_event_agent = create_react_agent(
    model=llm,
    tools=[query_adverse_events],
    name="adverse_event_agent",
    prompt=(
        "You are an adverse event data analyst. "
        "Use the query_adverse_events tool to look up reported side effects from the FDA FAERS database. "
        "Present the data clearly with report counts. "
        "Note that adverse event reports do not prove causation."
    ),
)

report_agent = create_react_agent(
    model=llm,
    tools=[],
    name="report_agent",
    prompt=(
        "You are a drug safety report writer. "
        "Synthesize the information provided by other agents into a clear, structured safety report. "
        "Always include citations indicating where each piece of information came from. "
        "Include a disclaimer that this is informational only and not medical advice."
    ),
)

supervisor = create_supervisor(
    [retrieval_agent, adverse_event_agent, report_agent],
    model=llm,
    prompt=(
        "You are a drug safety investigation supervisor coordinating three agents:\n"
        "- retrieval_agent: searches FDA drug label data (use for drug info, warnings, dosage, interactions)\n"
        "- adverse_event_agent: queries FDA adverse event reports (use for side effect statistics)\n"
        "- report_agent: writes final safety reports (use after other agents have gathered data)\n\n"
        "For most questions, first send the query to retrieval_agent, "
        "then to adverse_event_agent for complementary data, "
        "then to report_agent to synthesize a final answer.\n"
        "For simple factual questions, one agent may be sufficient."
    ),
)

app = supervisor.compile()


def ask(question: str) -> str:
    """Ask the drug safety agent a question."""
    result = app.invoke({
        "messages": [{"role": "user", "content": question}],
    })
    return result["messages"][-1].content


if __name__ == "__main__":
    question = "What are the most common side effects of metformin?"
    print(f"Question: {question}\n")
    answer = ask(question)
    print(f"Answer:\n{answer}")

Your multi-agent system is built and responding to questions. Next up, you'll test it with more complex queries and examine how the supervisor makes routing decisions in LangSmith.

Test the Complete System End-to-End

In the previous step, you wired three specialized agents into a LangGraph supervisor that coordinates the full drug safety investigation workflow.

Now it is time to put the system through its paces. You will run several real drug safety questions, watch the supervisor route each query through the right agents, and inspect the full trace of every decision in LangSmith.

In this step, get ready to:

Run the multi-agent system with several test questions and observe the responses.
Examine LangSmith traces to understand how the supervisor routes queries to agents.
Identify and fix common issues like wrong routing, irrelevant retrieval, or missing citations.

Run the multi-agent system with test questions

Your agents.py file already has a __main__ block that sends a metformin side effects question through the full system. This is your first real test of the entire pipeline, from supervisor routing to vector retrieval to report synthesis.

Run the multi-agent system with the default metformin question by running this command:

python agents.py

What should I see?

The system takes 15 to 45 seconds to respond. During this time, the supervisor is routing your question to the retrieval agent, then the adverse event agent, and finally the report agent.

You should see a structured safety report about metformin that includes side effects from the drug label, adverse event statistics from the live FDA database, and citations indicating where each piece of information came from.

Pro tip: rate limits

Each multi-agent query uses 4 to 8 LLM calls to Groq. The free tier allows 30 requests per minute and 1,000 per day.

Wait about 15 seconds between test runs to stay within limits. If you see a rate limit error, wait a minute and try again.

Seeing an error before any output appears?

If you see Collection fda_drug_labels does not exist, you need to run the ingestion script first. Run python ingest.py and then try again.
If you see AuthenticationError, double-check that your GROQ_API_KEY in .env is correct and has no extra spaces.
If you see RateLimitError, wait 60 seconds and run the command again. Groq's free tier limits you to 30 requests per minute.
help me debug an error when running agents.py

Now try a different type of question. Instead of asking about one drug's side effects, test whether the system can handle a question about a specific drug's adverse event reports.

In agents.py, scroll down to the if __name__ == "__main__": block at the bottom of your file.
Find this line:

question = "What are the most common side effects of metformin?"

Replace the question string so the line looks like this:

question = "What adverse events have been reported for lisinopril?"

Save agents.py and run it again:

python agents.py

This question emphasizes adverse event data. Watch whether the supervisor sends it to the adverse event agent for live FDA statistics, the retrieval agent for label information, or both.

For the third test, try a question that pushes the system harder. A comparison question forces the supervisor to gather data on multiple drugs.

In agents.py, find the question line again and replace it so it looks like this:

question = "Compare the safety profiles of ibuprofen and acetaminophen"

Save agents.py and run it again:

python agents.py

Why does acetaminophen have limited results?

Your ChromaDB vector database only contains drug labels for the five drugs you ingested: metformin, ibuprofen, lisinopril, amoxicillin, and atorvastatin. Acetaminophen is not one of them.

The retrieval agent will only find ibuprofen label data. The adverse event agent can still query the live FDA API for both drugs. This is a realistic test of how your system handles partial data.

Before moving on, change the question back to the original metformin query so agents.py matches the complete code:

question = "What are the most common side effects of metformin?"

Save agents.py.

Examine your LangSmith traces

Every time you ran a query, LangSmith recorded a full trace of every LLM call, tool invocation, and agent handoff. These traces let you see exactly how the supervisor decided which agent to call and in what order.

Open your browser and go to smith.langchain.com.
Sign in if prompted.
Click on the fda-drug-safety-agent project in your project list.
Click on any trace to expand it and see the full sequence of calls.

What should I look for in a trace?

Each trace shows a tree of operations. Look for these key details:

The supervisor's routing decisions. You should see it calling retrieval_agent first, then adverse_event_agent, then report_agent.
Tool calls within each agent. The retrieval agent should call search_drug_labels and the adverse event agent should call query_adverse_events.
Total token count and latency for the full query. This tells you how expensive each question is.

Don't see any traces?

Make sure LANGSMITH_TRACING=true is set in your .env file.
Verify that LANGSMITH_API_KEY in your .env file matches the key in your LangSmith account settings.
Check that LANGSMITH_PROJECT=fda-drug-safety-agent is set. Without this, traces may land in a default project instead.
help me troubleshoot why LangSmith is not showing traces for my project

Identify and fix common issues

Running multiple test questions often reveals patterns in how the system succeeds or fails. Here are the three most common issues and how to address them.

Supervisor routing to the wrong agent. If the supervisor sends a side effects question only to the retrieval agent and skips the adverse event agent, check the supervisor prompt in agents.py. The prompt should clearly describe when to use each agent.
Retrieval returning irrelevant chunks. If the retrieval agent's tool results contain unrelated drug label sections, the issue is usually with the embedding quality or chunk size. You can verify what chunks are being returned by checking the tool call output in your LangSmith trace.
Report agent not citing sources. If the final report does not include citations, check the report agent's prompt. It should explicitly instruct the agent to include source references. Your current prompt includes this instruction, but the LLM may occasionally skip it.

Why might the LLM ignore prompt instructions?

Large language models are probabilistic. Even with clear instructions, they can occasionally skip formatting rules or citation requirements. Running the same question twice may produce slightly different reports.

Setting temperature=0 in your ChatGroq configuration reduces this randomness. This is exactly why you will build an evaluation harness in the next step to measure consistency.

Re-run any question that produced a poor result to confirm the fix. If you changed a prompt, save agents.py and run python agents.py again.
Check LangSmith traces for the rerun to confirm the routing or output improved.

System seems stuck or taking too long?

Complex comparison questions can take 60 seconds or more because the supervisor calls multiple agents sequentially. Each agent makes its own LLM call and tool call.
If a query takes longer than 90 seconds, it may have hit a Groq rate limit and is retrying. Press Ctrl+C to cancel, wait 60 seconds, and try again.
help me debug why my multi-agent system is hanging or timing out

Your multi-agent system is working and you can observe every decision it makes. Next up, you will build an evaluation harness that measures whether your system's answers are actually grounded in real data.

Build the Evaluation Harness with DeepEval and RAGAS

Your multi-agent system is answering drug safety questions and you can see every decision in LangSmith. But how do you know the answers are actually correct?

An AI system that sounds confident but hallucinates facts is worse than one that says nothing. In this step, you will build an evaluation harness that measures whether your system grounds its answers in real retrieved data. You will use DeepEval for pytest-style pass/fail tests and RAGAS metrics to score faithfulness, answer relevancy, and context precision.

In this step, get ready to:

Create a golden test dataset of drug safety questions with expected answers.
Write DeepEval test cases that measure faithfulness, answer relevancy, and context precision.
Run the evaluation suite and analyze the results.

Create a golden test dataset

A golden test dataset is a curated set of questions with known-good answers. You compare your system's actual output against these reference answers to measure quality.

What is a golden test dataset?

Think of it like a unit test fixture for your AI system. Each entry has an input question, an expected output, and later you will also capture the retrieval context. If the system drifts or breaks, these tests catch it before users do.

Create a new file called test_eval.py in your project folder.
Add the imports and golden dataset by pasting this code:

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    ContextualPrecisionMetric,
)
from tools import search_drug_labels, query_adverse_events
from agents import ask


# Golden test dataset: curated questions with expected answers
GOLDEN_DATASET = [
    {
        "input": "What are the common side effects of metformin?",
        "expected_output": "Common side effects of metformin include diarrhea, nausea, vomiting, and stomach upset.",
    },
    {
        "input": "What are the warnings for ibuprofen?",
        "expected_output": "Ibuprofen carries warnings about cardiovascular risk, gastrointestinal bleeding, and renal effects.",
    },
    {
        "input": "What is lisinopril used for?",
        "expected_output": "Lisinopril is used for the treatment of hypertension and heart failure.",
    },
]

What does this code do?

The imports pull in DeepEval's test assertion function, the LLMTestCase class for structuring test data, and three evaluation metrics.
The GOLDEN_DATASET list contains three representative drug safety questions. Each entry pairs an input question with a short expected output that captures the key facts.
The imports from tools and agents bring in the search_drug_labels tool (to capture retrieval context) and the ask function (to generate actual outputs from the full agent system).

Save test_eval.py.

Why only three test cases?

Three well-chosen test cases cover three different question types: side effects, warnings, and indications. This gives you a representative signal without burning through Groq API rate limits. You can expand the dataset once you confirm the harness works.

Write the evaluation test functions

Now you will define the metrics and write two test functions. One tests answer quality (faithfulness and relevancy). The other tests retrieval quality (context precision).

What are these three metrics?

Faithfulness checks whether the answer is grounded in the retrieved context. A high score means the system is not making things up.
Answer relevancy checks whether the answer actually addresses the question that was asked.
Context precision checks whether the retrieved chunks are ranked correctly, with relevant chunks appearing before irrelevant ones.

DeepEval metrics require an LLM judge

DeepEval uses an LLM as a judge to evaluate your system's outputs. By default, it uses OpenAI's GPT models and requires an OPENAI_API_KEY environment variable. If you have an OpenAI API key, add it to your .env file.

If you do not have an OpenAI key, DeepEval also supports custom LLM judges. Check the DeepEval documentation on custom models for alternatives.

In test_eval.py, add the metric definitions and the first test function below your GOLDEN_DATASET:

# Set score thresholds for pass/fail
answer_relevancy = AnswerRelevancyMetric(threshold=0.7)
faithfulness = FaithfulnessMetric(threshold=0.7)
context_precision = ContextualPrecisionMetric(threshold=0.5)


@pytest.mark.parametrize("test_data", GOLDEN_DATASET)
def test_drug_safety_agent(test_data):
    actual_output = ask(test_data["input"])

    retrieval_context = search_drug_labels.invoke(test_data["input"])
    context_list = [retrieval_context] if retrieval_context else []

    test_case = LLMTestCase(
        input=test_data["input"],
        actual_output=actual_output,
        expected_output=test_data["expected_output"],
        retrieval_context=context_list,
    )

    assert_test(test_case, [answer_relevancy, faithfulness])

What does this code do?

The three metric objects define your pass/fail thresholds. A threshold of 0.7 means the system must score at least 70% on that metric to pass.
The @pytest.mark.parametrize decorator runs the test function once for each entry in GOLDEN_DATASET. This gives you three separate test results.
Inside the test, ask() runs the full multi-agent pipeline to produce an actual answer. Then search_drug_labels.invoke() captures the retrieval context separately.
The LLMTestCase bundles everything together: the original question, the actual output, the expected output, and the retrieval context.
Finally, assert_test checks the test case against both answer relevancy and faithfulness. If either metric falls below its threshold, the test fails.

Save test_eval.py.

Now add the second test function for retrieval quality.

In test_eval.py, add this function below test_drug_safety_agent:

@pytest.mark.parametrize("test_data", GOLDEN_DATASET)
def test_retrieval_precision(test_data):
    actual_output = ask(test_data["input"])

    retrieval_context = search_drug_labels.invoke(test_data["input"])
    context_list = [retrieval_context] if retrieval_context else []

    test_case = LLMTestCase(
        input=test_data["input"],
        actual_output=actual_output,
        expected_output=test_data["expected_output"],
        retrieval_context=context_list,
    )

    assert_test(test_case, [context_precision])

What does this code do?

This test function follows the same pattern as test_drug_safety_agent but evaluates a different metric: context precision.
Context precision measures whether the most relevant retrieved chunks appear first in the results. A low score suggests your retrieval is returning noise alongside useful data.
The threshold is set lower (0.5) because context ranking is harder to optimize than answer quality.

Save test_eval.py.

Why separate test functions?

Separating answer quality tests from retrieval quality tests makes it clear which part of your pipeline is failing. If faithfulness is low but context precision is high, the problem is in your report agent's synthesis. If context precision is low, the problem is in your retrieval or chunking.

Awesome, I've got everything!

Double check you have saved test_eval.py.

I'd like to double check the full code

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    ContextualPrecisionMetric,
)
from tools import search_drug_labels, query_adverse_events
from agents import ask


GOLDEN_DATASET = [
    {
        "input": "What are the common side effects of metformin?",
        "expected_output": "Common side effects of metformin include diarrhea, nausea, vomiting, and stomach upset.",
    },
    {
        "input": "What are the warnings for ibuprofen?",
        "expected_output": "Ibuprofen carries warnings about cardiovascular risk, gastrointestinal bleeding, and renal effects.",
    },
    {
        "input": "What is lisinopril used for?",
        "expected_output": "Lisinopril is used for the treatment of hypertension and heart failure.",
    },
]


answer_relevancy = AnswerRelevancyMetric(threshold=0.7)
faithfulness = FaithfulnessMetric(threshold=0.7)
context_precision = ContextualPrecisionMetric(threshold=0.5)


@pytest.mark.parametrize("test_data", GOLDEN_DATASET)
def test_drug_safety_agent(test_data):
    actual_output = ask(test_data["input"])

    retrieval_context = search_drug_labels.invoke(test_data["input"])
    context_list = [retrieval_context] if retrieval_context else []

    test_case = LLMTestCase(
        input=test_data["input"],
        actual_output=actual_output,
        expected_output=test_data["expected_output"],
        retrieval_context=context_list,
    )

    assert_test(test_case, [answer_relevancy, faithfulness])


@pytest.mark.parametrize("test_data", GOLDEN_DATASET)
def test_retrieval_precision(test_data):
    actual_output = ask(test_data["input"])

    retrieval_context = search_drug_labels.invoke(test_data["input"])
    context_list = [retrieval_context] if retrieval_context else []

    test_case = LLMTestCase(
        input=test_data["input"],
        actual_output=actual_output,
        expected_output=test_data["expected_output"],
        retrieval_context=context_list,
    )

    assert_test(test_case, [context_precision])

Run the evaluation suite and analyze results

With your test file written, it is time to run the evaluation and see how your system scores.

Run the evaluation suite by running this command:

pytest test_eval.py -v

What does this command do?

The pytest command discovers and runs all functions prefixed with test_ in test_eval.py.
The -v flag enables verbose output so you can see each test case's pass/fail status individually.
Each parameterized test case runs the full multi-agent pipeline, so expect this to take a few minutes. The Groq API calls and the DeepEval judge LLM calls both add latency.

Pro tip: use DeepEval's CLI

You can also run your tests with deepeval test run test_eval.py instead of pytest. The DeepEval CLI provides a richer output format with metric scores and reasons displayed inline.

After the tests finish, you will see output showing each test case with its pass or fail status.

How to read the results

A PASSED test means your system scored above the threshold for every metric in that test.
A FAILED test tells you which metric fell short. DeepEval prints the score and a reason explaining why.
Low faithfulness means the answer includes claims not found in the retrieved context. Try improving your retrieval by adjusting chunk size or adding more specific system prompts.
Low answer relevancy means the answer wandered off-topic. Check your report agent's system prompt to ensure it stays focused on the question.
Low context precision means irrelevant chunks are ranked above relevant ones. Consider adjusting your embedding model or retrieval parameters.

Tests failing or erroring out?

If you see an OPENAI_API_KEY error, DeepEval needs an LLM judge to evaluate your outputs. Add your OpenAI API key to .env or check the DeepEval docs for configuring a custom evaluation model.
If you hit Groq rate limits, wait a minute and run again. Each test case triggers multiple LLM calls through the multi-agent pipeline.
If tests fail with low scores rather than errors, that is expected on the first run. Review the metric reasons in the output to understand what to improve.

Help me debug my DeepEval test failures.

Secret mission

Add a Drug Interaction Checker

Your system can answer questions about individual drugs. But what happens when a patient takes multiple medications? Build a fourth agent that cross-references drug labels to flag potential interactions between drugs.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with free-tier cloud services, so there are no ongoing costs.

Resources you used:

Python virtual environment with all project dependencies.
ChromaDB local vector database (./chroma_db directory).
Project files (config.py, ingest.py, tools.py, agents.py, test_eval.py).
Groq account and API key (free tier, no credit card).
LangSmith account and API key (free tier, 5,000 traces/month).
.env file containing your API keys.
Downloaded HuggingFace embedding model (all-MiniLM-L6-v2) cached locally.

Keep everything running

No action needed. Choose this if you are still actively building or want to extend the project with the Secret Mission.

Your ChromaDB database, virtual environment, and all project files stay as they are.
Your Groq and LangSmith free-tier accounts remain active at no cost.
The cached embedding model stays in your local HuggingFace cache for future use.

Pause - I'll come back to this later

Deactivate your virtual environment to free up your terminal. Your files and database will stay on disk for when you return.

Deactivate your Python virtual environment by running deactivate in your terminal.
Your ChromaDB data in ./chroma_db persists on disk automatically. No shutdown steps needed.
To resume later, reactivate your virtual environment and pick up where you left off.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow these steps to clean up everything.

Deactivate your virtual environment by running deactivate in your terminal.
Delete the entire project folder, which includes all Python files, the .env file, the virtual environment, and the ./chroma_db directory.
Optionally, revoke your Groq API key at console.groq.com under the API Keys section.
Optionally, revoke your LangSmith API key at smith.langchain.com under your account settings.
Optionally, clear the cached HuggingFace embedding model from your local cache at ~/.cache/huggingface.

Mission accomplished

Nice Work!

You just built a production-grade multi-agent drug safety intelligence system powered by real FDA data. This project covers every core skill that AI Engineer hiring managers look for: RAG pipelines, multi-agent orchestration, LLM evaluation, and production observability.

You've learned how to:

Build a RAG pipeline that ingests real FDA drug labels, chunks them with LangChain, embeds them with local HuggingFace models, and stores them in ChromaDB for semantic retrieval.
Orchestrate three specialized agents with a LangGraph supervisor that routes user questions, gathers data from a vector database and the live openFDA API, and synthesizes safety reports with citations.
Build a full evaluation harness with DeepEval and RAGAS that measures faithfulness, answer relevancy, and context precision with CI-style pass/fail tests. Configure LangSmith for full observability of every agent decision.
Add a Drug Interaction Checker agent that cross-references multiple drug labels to flag potential interactions between medications.

Ready to quiz yourself?