Build a Clinical Note Intelligence Agent

Build a multi-agent pipeline that codes clinical notes into FHIR R4 data.

Introduction

30 Second Summary

Every time a doctor writes a clinical note, critical medical information gets locked inside unstructured text. Hospitals spend billions trying to extract diagnoses, medications, and procedures from these notes to code them for billing, research, and patient safety.

In this project, you will build a multi-agent Clinical Note Intelligence system that takes raw clinical notes and produces structured, coded medical data using the OpenAI Agents SDK. Three specialized AI agents work together to classify documents, extract clinical entities, and map diagnoses to standardized codes.

What You'll Build

A multi-agent pipeline that processes unstructured clinical notes (SOAP notes, discharge summaries) into FHIR R4-compliant output with ICD-10 coded diagnoses, medication extraction, and confidence-scored entities.

By the end of this project, you'll have:

  • A three-agent pipeline (Triage, Extraction, Coding) that processes clinical documents end-to-end with full tracing and observability.
  • Production-grade guardrails and human-in-the-loop flagging that validates output and flags low-confidence entities for human review.
  • A FHIR R4 Bundle output format with ICD-10 coded conditions and medication statements for healthcare interoperability.
  • Secret Mission: Build an evaluation harness that measures your pipeline's accuracy with precision/recall metrics.

Are there any prerequisites?

You should have experience writing Python and a basic understanding of LLM APIs. You also need an OpenAI API key with credits loaded (this project costs approximately $2-5 in API calls).

Before We Start

Clinical notes are the backbone of patient care, but they exist as unstructured text. Doctors write SOAP notes, discharge summaries, and progress notes in free-form prose that is difficult for systems to process.

In this step, you will set up a Python project with the OpenAI Agents SDK and define the data models that will power your multi-agent clinical intelligence pipeline.

In this step, get ready to:
  • Create the project environment and install the OpenAI Agents SDK.
  • Set up the file structure and synthetic clinical notes.
  • Define Pydantic data models for structured clinical output.
Create the project environment and install dependencies

Your pipeline needs a dedicated project folder and a virtual environment to keep dependencies isolated from other Python projects on your machine.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir clinical-note-agent
cd clinical-note-agent
  • Create a Python virtual environment by running:
python3 -m venv venv

What is a virtual environment?

A virtual environment is an isolated Python installation for your project. It keeps your project's packages separate from other Python projects on your machine, preventing version conflicts.

  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

.\venv\Scripts\Activate.ps1

You should see (venv) appear at the beginning of your terminal prompt.

Windows PowerShell blocking the script?

If you see an error about execution policies, run this command first, then retry the activation:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Help me fix my PowerShell execution policy error

  • Install the project dependencies by running:
pip install openai-agents pydantic

What are these packages?

The openai-agents package (v0.17.7) is the OpenAI Agents SDK. It provides the Agent, Runner, and function_tool primitives you will use to build your multi-agent pipeline.

The pydantic package provides data validation through Python type annotations. You will use it to define the structured output schemas for each agent.

You should see installation output ending with a success message listing installed packages.

Seeing a pip error?

Make sure (venv) is showing in your terminal prompt. If it is missing, your virtual environment is not activated. Run the activation command again before retrying the install.

Help me fix my pip install error

  • Set your OpenAI API key as an environment variable so the SDK can authenticate your requests:

macOS

export OPENAI_API_KEY=your-api-key-here

Windows

$env:OPENAI_API_KEY = "your-api-key-here"

Replace your-api-key-here with your actual OpenAI API key.

Where do I find my API key?

Go to platform.openai.com/api-keys and copy an existing key or create a new one. Make sure your account has credits loaded.

  • Verify the SDK installed correctly by running:
python -c "from agents import Agent, Runner; print('SDK ready')"

You should see SDK ready printed in your terminal.

Seeing ModuleNotFoundError?

This means the openai-agents package did not install into your active environment. Confirm your virtual environment is active (look for (venv) in your prompt) and re-run the pip install command.

Help me fix the ModuleNotFoundError

Create the file structure and sample clinical notes

Your pipeline needs separate modules for each agent, data models, guardrails, and sample data. Creating this structure now keeps the code organized as you build each component.

  • Create the directory structure by running these commands:
mkdir agents models data guardrails
  • Create the empty project files by running:
touch main.py
touch agents/__init__.py agents/triage.py agents/extraction.py agents/coding.py
touch models/__init__.py models/schemas.py models/fhir.py
touch data/__init__.py data/sample_notes.py
touch guardrails/__init__.py guardrails/validation.py

Windows without bash?

If you are using PowerShell without Git Bash, you can create files with New-Item instead. Alternatively, skip these commands and create the files through VS Code's Explorer sidebar after the next step.

  • Confirm the files exist by running:
ls

You should see agents, data, guardrails, main.py, models, and venv in the output.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Select the clinical-note-agent folder on your Desktop, then click Open.

You should see the folder structure in the Explorer sidebar with all the directories and files you just created.

  • Open data/sample_notes.py in the Explorer sidebar and paste the following synthetic clinical notes:
SOAP_NOTE = """
SOAP Note - Patient: John Smith, DOB: 1965-03-15, MRN: 12345
Date: 2026-07-01

SUBJECTIVE:
Patient is a 61-year-old male presenting with complaints of increased fatigue
and shortness of breath on exertion for the past 2 weeks. He reports mild
chest tightness during physical activity. He denies chest pain at rest,
palpitations, or syncope. He notes bilateral lower extremity edema that is
worse at the end of the day. Current medications include Metformin 1000mg BID,
Lisinopril 20mg daily, and Atorvastatin 40mg at bedtime.

OBJECTIVE:
Vitals: BP 145/92, HR 88, RR 18, Temp 98.6F, SpO2 94% on room air
General: Alert, oriented, mild respiratory distress with exertion
Cardiovascular: Regular rate and rhythm, S3 gallop noted, JVD present
Lungs: Bibasilar crackles, no wheezing
Extremities: 2+ bilateral pitting edema to mid-shin
Labs: BNP 580 pg/mL (elevated), Cr 1.4 mg/dL, HbA1c 7.8%

ASSESSMENT:
1. New-onset congestive heart failure, likely HFrEF
2. Type 2 diabetes mellitus - suboptimally controlled (HbA1c 7.8%)
3. Hypertension - not at goal despite current therapy
4. Chronic kidney disease, stage 3a - Cr 1.4

PLAN:
1. Start Furosemide 40mg daily for volume overload
2. Order echocardiogram to assess ejection fraction
3. Increase Lisinopril to 40mg daily
4. Add Empagliflozin 10mg daily (benefits for HF and diabetes)
5. Recheck BMP and BNP in 2 weeks
6. Refer to cardiology for further evaluation
"""

DISCHARGE_SUMMARY = """
DISCHARGE SUMMARY
Patient: Maria Garcia, DOB: 1978-08-22, MRN: 67890
Admission Date: 2026-06-25, Discharge Date: 2026-06-28

PRINCIPAL DIAGNOSIS:
Acute appendicitis with perforation

SECONDARY DIAGNOSES:
- Localized peritonitis
- Type 1 diabetes mellitus

PROCEDURES PERFORMED:
- Laparoscopic appendectomy (converted to open due to perforation)
- Peritoneal lavage

HOSPITAL COURSE:
Patient presented to ED with RLQ pain x 24 hours, fever 101.8F, and
elevated WBC 18,200. CT abdomen showed perforated appendicitis with
localized abscess. Taken to OR for laparoscopic appendectomy which was
converted to open approach due to extensive inflammation. Peritoneal
lavage performed. Post-operative course complicated by surgical site
infection managed with IV antibiotics.

DISCHARGE MEDICATIONS:
1. Augmentin 875/125mg BID x 10 days
2. Oxycodone 5mg q6h PRN pain
3. Insulin Glargine 22 units at bedtime (home dose)
4. Insulin Lispro per sliding scale with meals

FOLLOW-UP:
- Surgery clinic in 1 week for wound check
- PCP in 2 weeks for diabetes management
"""

PROGRESS_NOTE = """
PROGRESS NOTE - ICU Day 3
Patient: Robert Chen, DOB: 1952-11-30, MRN: 54321
Date: 2026-07-02

Overnight events: Patient remained hemodynamically stable. Ventilator
weaning trial attempted - tolerated 30 min on CPAP before developing
tachypnea (RR 32). Returned to AC mode.

Current Assessment:
- Acute respiratory failure secondary to bilateral pneumonia
- Sepsis - improving on current antibiotics
- Acute kidney injury (Cr 2.1, baseline 0.9) - possibly prerenal
- Atrial fibrillation with RVR - rate controlled with Diltiazem drip

Labs (0500):
- WBC: 14.2 (down from 22.1 on admission)
- Lactate: 1.8 (normalized, was 4.2)
- Cr: 2.1 (was 2.8 yesterday)
- Procalcitonin: 3.4 (down from 12.1)

Medications:
- Piperacillin-Tazobactam 4.5g q8h (Day 5)
- Vancomycin 1g q12h (trough pending)
- Diltiazem drip at 10mg/hr
- Norepinephrine 0.05 mcg/kg/min (weaning)
- Propofol 20 mcg/kg/min

Plan: Continue current antibiotics. Attempt SBT again tomorrow AM.
Nephrology consulted for AKI management. Target MAP > 65.
"""

SAMPLE_NOTES = {
    "soap": SOAP_NOTE,
    "discharge": DISCHARGE_SUMMARY,
    "progress": PROGRESS_NOTE,
}

What are these clinical notes?

These are synthetic (fictional) clinical documents that mirror real documentation patterns used in hospitals. Each note type has a different structure your agents will need to handle.

  • The SOAP note has Subjective, Objective, Assessment, and Plan sections.
  • The discharge summary has principal/secondary diagnoses, procedures, and discharge medications.
  • The progress note is an ICU daily update with labs, medications, and a care plan.
  • Save data/sample_notes.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved data/sample_notes.py before moving on.

ⓧ I'd like to double check the full code

SOAP_NOTE = """
SOAP Note - Patient: John Smith, DOB: 1965-03-15, MRN: 12345
Date: 2026-07-01

SUBJECTIVE:
Patient is a 61-year-old male presenting with complaints of increased fatigue
and shortness of breath on exertion for the past 2 weeks. He reports mild
chest tightness during physical activity. He denies chest pain at rest,
palpitations, or syncope. He notes bilateral lower extremity edema that is
worse at the end of the day. Current medications include Metformin 1000mg BID,
Lisinopril 20mg daily, and Atorvastatin 40mg at bedtime.

OBJECTIVE:
Vitals: BP 145/92, HR 88, RR 18, Temp 98.6F, SpO2 94% on room air
General: Alert, oriented, mild respiratory distress with exertion
Cardiovascular: Regular rate and rhythm, S3 gallop noted, JVD present
Lungs: Bibasilar crackles, no wheezing
Extremities: 2+ bilateral pitting edema to mid-shin
Labs: BNP 580 pg/mL (elevated), Cr 1.4 mg/dL, HbA1c 7.8%

ASSESSMENT:
1. New-onset congestive heart failure, likely HFrEF
2. Type 2 diabetes mellitus - suboptimally controlled (HbA1c 7.8%)
3. Hypertension - not at goal despite current therapy
4. Chronic kidney disease, stage 3a - Cr 1.4

PLAN:
1. Start Furosemide 40mg daily for volume overload
2. Order echocardiogram to assess ejection fraction
3. Increase Lisinopril to 40mg daily
4. Add Empagliflozin 10mg daily (benefits for HF and diabetes)
5. Recheck BMP and BNP in 2 weeks
6. Refer to cardiology for further evaluation
"""

DISCHARGE_SUMMARY = """
DISCHARGE SUMMARY
Patient: Maria Garcia, DOB: 1978-08-22, MRN: 67890
Admission Date: 2026-06-25, Discharge Date: 2026-06-28

PRINCIPAL DIAGNOSIS:
Acute appendicitis with perforation

SECONDARY DIAGNOSES:
- Localized peritonitis
- Type 1 diabetes mellitus

PROCEDURES PERFORMED:
- Laparoscopic appendectomy (converted to open due to perforation)
- Peritoneal lavage

HOSPITAL COURSE:
Patient presented to ED with RLQ pain x 24 hours, fever 101.8F, and
elevated WBC 18,200. CT abdomen showed perforated appendicitis with
localized abscess. Taken to OR for laparoscopic appendectomy which was
converted to open approach due to extensive inflammation. Peritoneal
lavage performed. Post-operative course complicated by surgical site
infection managed with IV antibiotics.

DISCHARGE MEDICATIONS:
1. Augmentin 875/125mg BID x 10 days
2. Oxycodone 5mg q6h PRN pain
3. Insulin Glargine 22 units at bedtime (home dose)
4. Insulin Lispro per sliding scale with meals

FOLLOW-UP:
- Surgery clinic in 1 week for wound check
- PCP in 2 weeks for diabetes management
"""

PROGRESS_NOTE = """
PROGRESS NOTE - ICU Day 3
Patient: Robert Chen, DOB: 1952-11-30, MRN: 54321
Date: 2026-07-02

Overnight events: Patient remained hemodynamically stable. Ventilator
weaning trial attempted - tolerated 30 min on CPAP before developing
tachypnea (RR 32). Returned to AC mode.

Current Assessment:
- Acute respiratory failure secondary to bilateral pneumonia
- Sepsis - improving on current antibiotics
- Acute kidney injury (Cr 2.1, baseline 0.9) - possibly prerenal
- Atrial fibrillation with RVR - rate controlled with Diltiazem drip

Labs (0500):
- WBC: 14.2 (down from 22.1 on admission)
- Lactate: 1.8 (normalized, was 4.2)
- Cr: 2.1 (was 2.8 yesterday)
- Procalcitonin: 3.4 (down from 12.1)

Medications:
- Piperacillin-Tazobactam 4.5g q8h (Day 5)
- Vancomycin 1g q12h (trough pending)
- Diltiazem drip at 10mg/hr
- Norepinephrine 0.05 mcg/kg/min (weaning)
- Propofol 20 mcg/kg/min

Plan: Continue current antibiotics. Attempt SBT again tomorrow AM.
Nephrology consulted for AKI management. Target MAP > 65.
"""

SAMPLE_NOTES = {
    "soap": SOAP_NOTE,
    "discharge": DISCHARGE_SUMMARY,
    "progress": PROGRESS_NOTE,
}
Define the Pydantic data models

Each agent in your pipeline produces structured output. Pydantic models define the exact shape of that output, ensuring every agent returns validated, predictable data.

  • Open models/schemas.py in the Explorer sidebar and paste the following imports and first model:
from pydantic import BaseModel, Field


class DocumentClassification(BaseModel):
    document_type: str = Field(description="Type: soap_note, discharge_summary, progress_note, consultation_note")
    confidence: float = Field(ge=0.0, le=1.0, description="Classification confidence")
    key_sections: list[str] = Field(description="Key sections identified in the document")
    summary: str = Field(description="One-sentence summary of the document")

What does this model do?

The DocumentClassification model is the output schema for your Triage Agent. When the agent classifies a clinical note, it must return data matching this exact shape.

The Field function adds validation constraints. For example, ge=0.0, le=1.0 ensures confidence is always a value between 0 and 1.

  • Below the DocumentClassification class, add the ClinicalEntity model that represents a single extracted item:
class ClinicalEntity(BaseModel):
    text: str = Field(description="The extracted entity text")
    category: str = Field(description="Entity category: diagnosis, medication, procedure, or lab_value")
    confidence: float = Field(ge=0.0, le=1.0, description="Extraction confidence score")
    details: str = Field(default="", description="Additional details like dosage or value")
    requires_review: bool = Field(default=False, description="Flagged for human review")
    icd10_code: str | None = Field(default=None, description="ICD-10 code if applicable")

What is a ClinicalEntity?

This is the core building block for extraction. Every diagnosis, medication, procedure, or lab value the agent extracts becomes a ClinicalEntity with a confidence score and a flag for human review.

The requires_review field defaults to False and gets set to True later by your guardrails when confidence drops below a threshold.

  • Add the two result container models below ClinicalEntity:
class ExtractionResult(BaseModel):
    diagnoses: list[ClinicalEntity] = Field(default_factory=list)
    medications: list[ClinicalEntity] = Field(default_factory=list)
    procedures: list[ClinicalEntity] = Field(default_factory=list)
    lab_values: list[ClinicalEntity] = Field(default_factory=list)
    document_type: str = Field(default="unknown")
    patient_name: str = Field(default="Unknown")


class CodingResult(BaseModel):
    coded_diagnoses: list[ClinicalEntity] = Field(default_factory=list)
    coding_notes: str = Field(default="", description="Notes about coding decisions")
    flagged_for_review: list[str] = Field(default_factory=list, description="Items needing human review")

What do ExtractionResult and CodingResult hold?

The ExtractionResult groups all extracted entities by category. This is the output schema for the Extraction Agent.

The CodingResult holds diagnoses that have been mapped to ICD-10 codes by the Medical Coding Agent. It also tracks which items were flagged for human review.

  • Save models/schemas.py.
  • Verify the models load correctly by running this in your terminal:
python -c "from models.schemas import DocumentClassification, ExtractionResult; print('Models loaded')"

You should see Models loaded printed without errors.

Seeing an ImportError?

Make sure you are running the command from inside the clinical-note-agent folder (not from Desktop). Also check that models/__init__.py exists. Without the __init__.py file, Python does not recognize models as a package.

Help me fix this ImportError

✔️ Awesome, I've got everything!

Great. Double check you have saved models/schemas.py before moving on.

ⓧ I'd like to double check the full code

from pydantic import BaseModel, Field


class DocumentClassification(BaseModel):
    document_type: str = Field(description="Type: soap_note, discharge_summary, progress_note, consultation_note")
    confidence: float = Field(ge=0.0, le=1.0, description="Classification confidence")
    key_sections: list[str] = Field(description="Key sections identified in the document")
    summary: str = Field(description="One-sentence summary of the document")


class ClinicalEntity(BaseModel):
    text: str = Field(description="The extracted entity text")
    category: str = Field(description="Entity category: diagnosis, medication, procedure, or lab_value")
    confidence: float = Field(ge=0.0, le=1.0, description="Extraction confidence score")
    details: str = Field(default="", description="Additional details like dosage or value")
    requires_review: bool = Field(default=False, description="Flagged for human review")
    icd10_code: str | None = Field(default=None, description="ICD-10 code if applicable")


class ExtractionResult(BaseModel):
    diagnoses: list[ClinicalEntity] = Field(default_factory=list)
    medications: list[ClinicalEntity] = Field(default_factory=list)
    procedures: list[ClinicalEntity] = Field(default_factory=list)
    lab_values: list[ClinicalEntity] = Field(default_factory=list)
    document_type: str = Field(default="unknown")
    patient_name: str = Field(default="Unknown")


class CodingResult(BaseModel):
    coded_diagnoses: list[ClinicalEntity] = Field(default_factory=list)
    coding_notes: str = Field(default="", description="Notes about coding decisions")
    flagged_for_review: list[str] = Field(default_factory=list, description="Items needing human review")

Your project environment is ready with the SDK installed, sample data loaded, and structured output models defined. Next up, you will build the Triage and Extraction Agents that process these clinical notes.

Build the Triage and Extraction Agents

Your data models and synthetic notes are ready. Now you'll build the first two agents in your pipeline: a Triage Agent that classifies clinical documents, and an Extraction Agent that pulls structured entities from them.

Each agent is a specialized worker with its own instructions and output format. The Triage Agent determines what kind of document it's looking at. The Extraction Agent uses function tools to systematically extract every diagnosis, medication, procedure, and lab value from the text.

In this step, get ready to:
  • Build a Triage Agent that classifies clinical document types.
  • Build an Extraction Agent with four function tools for entity extraction.
  • Test both agents end-to-end with a sample SOAP note.
Build the Triage Agent

In a multi-agent system, the first agent often acts as a router. It looks at incoming data and decides what type of work needs to happen next.

Your Triage Agent classifies each clinical note by type (SOAP note, discharge summary, progress note, or consultation note). This classification tells downstream agents what structure to expect.

  • Open agents/triage.py in VS Code and add the following code:
from agents import Agent

from models.schemas import DocumentClassification

triage_agent = Agent(
    name="Clinical Triage Agent",
    instructions="""You are a clinical document classification specialist.
Analyze the incoming clinical document and classify it by type.

Document types:
- soap_note: Has Subjective, Objective, Assessment, Plan sections
- discharge_summary: Hospital discharge with diagnoses, procedures, discharge meds
- progress_note: Daily/periodic patient status update (often ICU or inpatient)
- consultation_note: Specialist consultation response

Assign a confidence score (0.0-1.0) based on how clearly the document matches.
Identify key sections present and provide a one-sentence summary.""",
    output_type=DocumentClassification,
)

What does this code do?

This file creates a single Agent instance from the OpenAI Agents SDK.

  • name gives the agent a human-readable label for tracing.
  • instructions is the system prompt. It tells the LLM exactly what document types exist and how to score confidence.
  • output_type=DocumentClassification forces the agent to return structured output matching your Pydantic model. The SDK auto-generates a JSON schema from the model and constrains the LLM output to match it.
  • Save agents/triage.py.

Seeing an import error?

  • Make sure your virtual environment is activated. You should see the environment name in your terminal prompt.
  • Confirm openai-agents is installed by running pip install openai-agents again.
  • Check that your models/schemas.py file exists and contains the DocumentClassification class from the previous step.

Why can't Python find my Agent import?

Build the Extraction Agent with function tools

The Extraction Agent needs to pull four categories of clinical entities from a document: diagnoses, medications, procedures, and lab values. Instead of asking the LLM to do everything in one pass, you give it specialized tools it can call for each category.

What is a function tool?

A function tool is a Python function decorated with @function_tool. The SDK automatically generates a JSON schema from the function's signature and docstring, then exposes it to the LLM.

When the agent decides to use a tool, the SDK executes your Python function and feeds the result back to the LLM. This gives the model a structured way to organize its extraction work into distinct categories.

  • Open agents/extraction.py and add the four function tools:
from agents import Agent, function_tool

from models.schemas import ExtractionResult


@function_tool
def extract_diagnoses(clinical_text: str) -> str:
    """Extract all diagnoses, conditions, and clinical findings from the text.
    Return each with a confidence score based on how explicitly it is stated."""
    return f"Extract diagnoses from:\n{clinical_text}"


@function_tool
def extract_medications(clinical_text: str) -> str:
    """Extract all medications with dose, route, and frequency from the text."""
    return f"Extract medications from:\n{clinical_text}"


@function_tool
def extract_procedures(clinical_text: str) -> str:
    """Extract all procedures, surgeries, tests, and referrals from the text."""
    return f"Extract procedures from:\n{clinical_text}"


@function_tool
def extract_lab_values(clinical_text: str) -> str:
    """Extract all laboratory values and vital signs with their values and units."""
    return f"Extract lab values from:\n{clinical_text}"

What do these tools do?

Each tool receives the clinical text and returns it back to the LLM with a category-specific prompt prefix. The LLM then processes the text through the lens of that specific extraction task.

  • extract_diagnoses focuses on conditions and clinical findings.
  • extract_medications focuses on drugs, dosages, and routes.
  • extract_procedures focuses on surgeries, tests, and referrals.
  • extract_lab_values focuses on lab results and vital signs.

This tool-per-category pattern gives the agent structured extraction responsibilities. The docstrings are critical because the SDK uses them to generate the tool descriptions the LLM sees.

  • Now add the Extraction Agent below the tools in the same file:
extraction_agent = Agent(
    name="Clinical Extraction Agent",
    instructions="""You are a clinical data extraction specialist. Extract ALL clinical
entities from the provided medical document using the available tools.

Use ALL four tools for each document:
1. extract_diagnoses - conditions, diseases, clinical findings
2. extract_medications - drugs, dosages, routes, frequencies
3. extract_procedures - surgeries, tests, interventions, referrals
4. extract_lab_values - lab results, vital signs, measurements

CONFIDENCE SCORING RULES:
- 1.0: Explicitly stated with clear clinical language
- 0.7-0.9: Strongly implied or clearly present but not the primary focus
- 0.5-0.7: Mentioned but with some ambiguity
- Below 0.5: Qualified with "possible", "probable", "rule out", or vague

Set requires_review=True for ANY entity with confidence below 0.5.
Extract EVERY entity mentioned. Never skip or infer entities not in the text.""",
    tools=[extract_diagnoses, extract_medications, extract_procedures, extract_lab_values],
    output_type=ExtractionResult,
)

What does this agent configuration do?

  • tools=[extract_diagnoses, extract_medications, extract_procedures, extract_lab_values] registers all four function tools so the LLM can call them.
  • The instructions tell the agent to use ALL four tools for every document. Without this explicit instruction, the LLM might skip tools it considers unnecessary.
  • The confidence scoring rules create a clear rubric. This is what separates a production system from a naive one. Entities qualified with words like "possible" or "rule out" get low confidence and are flagged for human review.
  • output_type=ExtractionResult constrains output to your Pydantic schema with typed lists of diagnoses, medications, procedures, and lab values.
  • Save agents/extraction.py.

Syntax error in the instructions string?

  • The instructions use triple-quoted strings (""".."""). Make sure you have exactly three double quotes at the start and end.
  • The escaped quotes inside the confidence rules (\"possible\") are part of the prompt text the LLM reads. They should appear as regular quotes in your file.

Help me fix a syntax error in my extraction agent

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before moving on to the test.

ⓧ I'd like to double check the full code

Here is the complete agents/triage.py:

from agents import Agent

from models.schemas import DocumentClassification

triage_agent = Agent(
    name="Clinical Triage Agent",
    instructions="""You are a clinical document classification specialist.
Analyze the incoming clinical document and classify it by type.

Document types:
- soap_note: Has Subjective, Objective, Assessment, Plan sections
- discharge_summary: Hospital discharge with diagnoses, procedures, discharge meds
- progress_note: Daily/periodic patient status update (often ICU or inpatient)
- consultation_note: Specialist consultation response

Assign a confidence score (0.0-1.0) based on how clearly the document matches.
Identify key sections present and provide a one-sentence summary.""",
    output_type=DocumentClassification,
)

Here is the complete agents/extraction.py:

from agents import Agent, function_tool

from models.schemas import ExtractionResult


@function_tool
def extract_diagnoses(clinical_text: str) -> str:
    """Extract all diagnoses, conditions, and clinical findings from the text.
    Return each with a confidence score based on how explicitly it is stated."""
    return f"Extract diagnoses from:\n{clinical_text}"


@function_tool
def extract_medications(clinical_text: str) -> str:
    """Extract all medications with dose, route, and frequency from the text."""
    return f"Extract medications from:\n{clinical_text}"


@function_tool
def extract_procedures(clinical_text: str) -> str:
    """Extract all procedures, surgeries, tests, and referrals from the text."""
    return f"Extract procedures from:\n{clinical_text}"


@function_tool
def extract_lab_values(clinical_text: str) -> str:
    """Extract all laboratory values and vital signs with their values and units."""
    return f"Extract lab values from:\n{clinical_text}"


extraction_agent = Agent(
    name="Clinical Extraction Agent",
    instructions="""You are a clinical data extraction specialist. Extract ALL clinical
entities from the provided medical document using the available tools.

Use ALL four tools for each document:
1. extract_diagnoses - conditions, diseases, clinical findings
2. extract_medications - drugs, dosages, routes, frequencies
3. extract_procedures - surgeries, tests, interventions, referrals
4. extract_lab_values - lab results, vital signs, measurements

CONFIDENCE SCORING RULES:
- 1.0: Explicitly stated with clear clinical language
- 0.7-0.9: Strongly implied or clearly present but not the primary focus
- 0.5-0.7: Mentioned but with some ambiguity
- Below 0.5: Qualified with "possible", "probable", "rule out", or vague

Set requires_review=True for ANY entity with confidence below 0.5.
Extract EVERY entity mentioned. Never skip or infer entities not in the text.""",
    tools=[extract_diagnoses, extract_medications, extract_procedures, extract_lab_values],
    output_type=ExtractionResult,
)
Test the two-agent pipeline

Both agents are defined. Now you'll run a SOAP note through the Triage Agent, then through the Extraction Agent, and verify that each produces the correct structured output.

  • Create a file called test_pipeline.py in your project root (clinical-note-agent/) and add the following test script:
# test_pipeline.py - Quick test for triage and extraction agents
from agents import Runner
from agents.triage import triage_agent
from agents.extraction import extraction_agent
from data.sample_notes import SOAP_NOTE

# Test 1: Triage Agent
print("=" * 50)
print("Testing Triage Agent...")
print("=" * 50)
triage_result = Runner.run_sync(triage_agent, SOAP_NOTE)
classification = triage_result.final_output

print(f"Document type: {classification.document_type}")
print(f"Confidence: {classification.confidence}")
print(f"Key sections: {classification.key_sections}")
print(f"Summary: {classification.summary}")

# Test 2: Extraction Agent
print("\n" + "=" * 50)
print("Testing Extraction Agent...")
print("=" * 50)
extraction_result = Runner.run_sync(extraction_agent, SOAP_NOTE)
extraction = extraction_result.final_output

print(f"Diagnoses found: {len(extraction.diagnoses)}")
for dx in extraction.diagnoses:
    print(f"  - {dx.text} (confidence: {dx.confidence})")

print(f"\nMedications found: {len(extraction.medications)}")
for med in extraction.medications:
    print(f"  - {med.text} ({med.details})")

print(f"\nProcedures found: {len(extraction.procedures)}")
print(f"Lab values found: {len(extraction.lab_values)}")

# Summary
total = (len(extraction.diagnoses) + len(extraction.medications)
         + len(extraction.procedures) + len(extraction.lab_values))
print(f"\nTotal entities extracted: {total}")
print("Pipeline test complete!")

What does this test do?

  • Runner.run_sync() executes an agent synchronously and waits for the result. The final_output attribute contains the structured Pydantic object.
  • The test feeds the same SOAP note to both agents. The Triage Agent should classify it as soap_note. The Extraction Agent should find multiple entities with confidence scores.
  • Each run makes API calls to OpenAI. This test will use a small amount of your API credits (typically a few cents).
  • Save test_pipeline.py.
  • Run the test by executing this command in your terminal (make sure you're in the clinical-note-agent/ directory):
python test_pipeline.py

The test takes 15-30 seconds because each agent makes API calls to OpenAI. You should see output showing a document type of soap_note with high confidence, followed by lists of extracted diagnoses, medications, procedures, and lab values.

What should I see?

Your Triage Agent should output document_type: soap_note with a confidence near 1.0 (the sample note has clear SOAP sections).

Your Extraction Agent should find at least 4 diagnoses (congestive heart failure, type 2 diabetes, hypertension, chronic kidney disease), 3+ medications (Metformin, Lisinopril, Atorvastatin, etc.), and multiple lab values. The exact counts may vary slightly between runs because the LLM interprets the text.

Getting an API error or timeout?

  • Check that your OPENAI_API_KEY environment variable is set in the same terminal session where you run the script.
  • Verify your OpenAI account has available credits. The Agents SDK requires a funded account.
  • If you see a rate limit error, wait 30 seconds and try again. The extraction agent makes multiple tool calls in a single run.

Help me debug my pipeline test

Your first two agents are working. The Triage Agent classifies documents and the Extraction Agent pulls structured clinical data with confidence scores. Next up, you'll build the Medical Coding Agent that maps those extracted diagnoses to ICD-10 codes and generates FHIR R4 output for healthcare interoperability.

Build the Medical Coding Agent and FHIR Output

Your Triage Agent classifies documents and your Extraction Agent pulls out clinical entities. The next challenge is turning those raw entities into standardized medical codes that hospital systems can actually use.

In this step, you will build the Medical Coding Agent that maps diagnoses to ICD-10 codes with confidence scoring. You will also build a FHIR R4 Bundle formatter that packages everything into the interoperability format used across healthcare systems.

In this step, get ready to:
  • Build the Medical Coding Agent with ICD-10 mapping.
  • Build the FHIR R4 Bundle formatter for healthcare-standard output.
  • Wire the full three-agent pipeline together in the main entry point.
Build the Medical Coding Agent

The Medical Coding Agent takes extracted diagnoses and maps each one to a standardized ICD-10-CM code. This is exactly what certified medical coders do in hospitals, except your agent does it with confidence scoring so ambiguous cases get flagged for human review.

  • Create a new file called agents/coding.py and add the following code:
from agents import Agent, function_tool

from models.schemas import CodingResult


@function_tool
def map_to_icd10(diagnosis: str) -> str:
    """Map a clinical diagnosis to its most specific ICD-10-CM code.
    Return: icd10_code|display_name|confidence_score"""
    return f"Map to ICD-10-CM: {diagnosis}"

What does this code do?

The @function_tool decorator registers map_to_icd10 as a tool the LLM agent can call. When invoked, it passes the diagnosis text back to the model for processing. The model uses its medical knowledge to determine the correct ICD-10-CM code.

This pattern (tool returns input text for model processing) is the standard OpenAI Agents SDK approach for structured extraction tasks. The tool gives the model a clear interface to work through each diagnosis individually.

  • Below the function tool in the same file, add the agent definition:
coding_agent = Agent(
    name="Medical Coding Agent",
    instructions="""You are a certified medical coding specialist.
Map each diagnosis to the most specific ICD-10-CM code available.

Use the map_to_icd10 tool for each diagnosis.

CONFIDENCE RULES:
- High (>0.8): Clear diagnosis with direct ICD-10 mapping
- Medium (0.5-0.8): Clear diagnosis but code selection has ambiguity
- Low (<0.5): Vague, qualified ("possible"/"probable"), or lacks specificity

For each coded diagnosis, set:
- icd10_code: the code (e.g., "I50.9")
- text: the diagnosis description
- confidence: your confidence in the code assignment
- requires_review: True if confidence < 0.5

Add any difficult decisions to coding_notes.
List items needing review in flagged_for_review with reasons.""",
    tools=[map_to_icd10],
    output_type=CodingResult,
)

What does this code do?

The coding_agent uses the same Agent pattern as your other agents. Its instructions define three confidence tiers that mirror how real medical coders work. Clear diagnoses like "congestive heart failure" get high confidence. Qualified diagnoses like "possible early-stage dementia" get flagged for review.

The output_type=CodingResult ensures the agent returns structured data matching your Pydantic schema, including the coded_diagnoses list and flagged_for_review items.

  • Save agents/coding.py.

Seeing an import error?

Make sure models/schemas.py contains the CodingResult class from the previous step. Also confirm your virtual environment is activated (you should see the environment name in your terminal prompt).

Help me debug this import error in agents/coding.py

✔️ Awesome, I've got everything!

Great. Make sure you have saved agents/coding.py before moving on.

ⓧ I'd like to double check the full code

from agents import Agent, function_tool

from models.schemas import CodingResult


@function_tool
def map_to_icd10(diagnosis: str) -> str:
    """Map a clinical diagnosis to its most specific ICD-10-CM code.
    Return: icd10_code|display_name|confidence_score"""
    return f"Map to ICD-10-CM: {diagnosis}"


coding_agent = Agent(
    name="Medical Coding Agent",
    instructions="""You are a certified medical coding specialist.
Map each diagnosis to the most specific ICD-10-CM code available.

Use the map_to_icd10 tool for each diagnosis.

CONFIDENCE RULES:
- High (>0.8): Clear diagnosis with direct ICD-10 mapping
- Medium (0.5-0.8): Clear diagnosis but code selection has ambiguity
- Low (<0.5): Vague, qualified ("possible"/"probable"), or lacks specificity

For each coded diagnosis, set:
- icd10_code: the code (e.g., "I50.9")
- text: the diagnosis description
- confidence: your confidence in the code assignment
- requires_review: True if confidence < 0.5

Add any difficult decisions to coding_notes.
List items needing review in flagged_for_review with reasons.""",
    tools=[map_to_icd10],
    output_type=CodingResult,
)
Build the FHIR R4 Bundle formatter

FHIR R4 is the standard data format for exchanging healthcare information between systems. Your pipeline needs to output data in this format so it can integrate with electronic health records, hospital information systems, and other clinical tools.

What is FHIR R4?

FHIR (Fast Healthcare Interoperability Resources) R4 is the most widely deployed healthcare data exchange standard in the US. A FHIR Bundle of type collection groups multiple resources (Condition entries, MedicationStatement entries) into a single package.

Each coded diagnosis becomes a FHIR Condition resource with the ICD-10 code stored using the system URI http://hl7.org/fhir/sid/icd-10-cm. Medications become MedicationStatement resources. This is what makes your output usable in real clinical workflows.

  • Create a new file called models/fhir.py and add the following code:
import json
import uuid
from datetime import datetime, timezone

from models.schemas import ExtractionResult, CodingResult


def build_fhir_bundle(
    extraction: ExtractionResult,
    coding: CodingResult,
    patient_id: str = "patient-001",
) -> dict:
    """Build a FHIR R4 Bundle from extraction and coding results."""
    entries = []
    patient_ref = {"reference": f"Patient/{patient_id}", "display": extraction.patient_name}

    for entity in coding.coded_diagnoses:
        condition = {
            "fullUrl": f"urn:uuid:{uuid.uuid4()}",
            "resource": {
                "resourceType": "Condition",
                "id": str(uuid.uuid4()),
                "clinicalStatus": {
                    "coding": [{
                        "system": "http://terminology.hl7.org/CodeSystem/condition-clinical",
                        "code": "active",
                        "display": "Active",
                    }]
                },
                "code": {
                    "coding": [{
                        "system": "http://hl7.org/fhir/sid/icd-10-cm",
                        "code": entity.icd10_code or "UNKNOWN",
                        "display": entity.text,
                    }],
                    "text": entity.text,
                },
                "subject": patient_ref,
            },
        }
        entries.append(condition)

    for med in extraction.medications:
        med_statement = {
            "fullUrl": f"urn:uuid:{uuid.uuid4()}",
            "resource": {
                "resourceType": "MedicationStatement",
                "id": str(uuid.uuid4()),
                "status": "active",
                "medicationCodeableConcept": {"text": med.text},
                "subject": patient_ref,
                "dosage": [{"text": med.details}] if med.details else [],
            },
        }
        entries.append(med_statement)

    bundle = {
        "resourceType": "Bundle",
        "type": "collection",
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "entry": entries,
    }
    return bundle


def bundle_to_json(bundle: dict) -> str:
    """Serialize FHIR Bundle to formatted JSON."""
    return json.dumps(bundle, indent=2, default=str)

What does this code do?

The build_fhir_bundle() function takes your extraction and coding results and builds a standards-compliant FHIR R4 Bundle. It creates two types of entries:

  • FHIR Condition resources for each coded diagnosis, including the ICD-10-CM code and clinical status.
  • FHIR MedicationStatement resources for each extracted medication, with dosage information attached.

The Bundle uses type: "collection" which is the FHIR standard for grouping related resources into a single package without imposing processing rules. Each entry gets a unique urn:uuid identifier as its fullUrl.

  • Save models/fhir.py.

✔️ Awesome, I've got everything!

Great. Make sure you have saved models/fhir.py before moving on.

ⓧ I'd like to double check the full code

import json
import uuid
from datetime import datetime, timezone

from models.schemas import ExtractionResult, CodingResult


def build_fhir_bundle(
    extraction: ExtractionResult,
    coding: CodingResult,
    patient_id: str = "patient-001",
) -> dict:
    """Build a FHIR R4 Bundle from extraction and coding results."""
    entries = []
    patient_ref = {"reference": f"Patient/{patient_id}", "display": extraction.patient_name}

    for entity in coding.coded_diagnoses:
        condition = {
            "fullUrl": f"urn:uuid:{uuid.uuid4()}",
            "resource": {
                "resourceType": "Condition",
                "id": str(uuid.uuid4()),
                "clinicalStatus": {
                    "coding": [{
                        "system": "http://terminology.hl7.org/CodeSystem/condition-clinical",
                        "code": "active",
                        "display": "Active",
                    }]
                },
                "code": {
                    "coding": [{
                        "system": "http://hl7.org/fhir/sid/icd-10-cm",
                        "code": entity.icd10_code or "UNKNOWN",
                        "display": entity.text,
                    }],
                    "text": entity.text,
                },
                "subject": patient_ref,
            },
        }
        entries.append(condition)

    for med in extraction.medications:
        med_statement = {
            "fullUrl": f"urn:uuid:{uuid.uuid4()}",
            "resource": {
                "resourceType": "MedicationStatement",
                "id": str(uuid.uuid4()),
                "status": "active",
                "medicationCodeableConcept": {"text": med.text},
                "subject": patient_ref,
                "dosage": [{"text": med.details}] if med.details else [],
            },
        }
        entries.append(med_statement)

    bundle = {
        "resourceType": "Bundle",
        "type": "collection",
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "entry": entries,
    }
    return bundle


def bundle_to_json(bundle: dict) -> str:
    """Serialize FHIR Bundle to formatted JSON."""
    return json.dumps(bundle, indent=2, default=str)
Wire the full three-agent pipeline

Now you will connect all three agents into a sequential pipeline. Each agent's output feeds into the next: Triage classifies the document, Extraction pulls entities, Coding maps diagnoses to ICD-10, and the FHIR formatter packages the result.

The pipeline also needs a review queue to track entities that need human verification. You will create a helper class for this before wiring main.py.

  • Create the guardrails/ directory and a file called guardrails/__init__.py (this can be empty) by running:
mkdir guardrails
touch guardrails/__init__.py
  • Create guardrails/validation.py with the ReviewQueue class that collects low-confidence entities for human review:
class ReviewQueue:
    """Collects low-confidence entities for human review."""

    def __init__(self):
        self.items: list[dict] = []

    def add(self, entity_text: str, category: str, confidence: float, reason: str):
        self.items.append({
            "entity": entity_text,
            "category": category,
            "confidence": confidence,
            "reason": reason,
        })

    def summary(self) -> str:
        if not self.items:
            return "All entities auto-approved. No items need human review."
        lines = [f"HUMAN REVIEW REQUIRED ({len(self.items)} items):"]
        for item in self.items:
            lines.append(
                f"  [{item['category'].upper()}] \"{item['entity']}\" "
                f"(confidence: {item['confidence']:.2f}) - {item['reason']}"
            )
        return "\n".join(lines)

What does ReviewQueue do?

The ReviewQueue is a simple collector for entities that the pipeline is not fully confident about. Every entity with confidence below 0.8 gets logged here with a reason. At the end of processing, the summary() method prints a structured report showing exactly which items a human coder should verify.

You will expand this file with additional guardrail functionality in the next step. For now, it provides what main.py needs to track review items.

  • Save guardrails/validation.py.
  • Now create main.py in the project root with the process_clinical_note() function that runs all three agents in sequence:
import json

from agents import Runner

from agents.triage import triage_agent
from agents.extraction import extraction_agent
from agents.coding import coding_agent
from models.fhir import build_fhir_bundle, bundle_to_json
from guardrails.validation import ReviewQueue
from data.sample_notes import SAMPLE_NOTES


def process_clinical_note(note: str, note_label: str) -> dict:
    """Process a single clinical note through the full agent pipeline."""
    print(f"\n{'='*60}")
    print(f"Processing: {note_label}")
    print(f"{'='*60}")

    review_queue = ReviewQueue()

    # Stage 1: Triage
    print("\n[1/3] Triage Agent - Classifying document...")
    triage_result = Runner.run_sync(triage_agent, note)
    classification = triage_result.final_output
    print(f"  Type: {classification.document_type} (confidence: {classification.confidence:.2f})")
    print(f"  Sections: {', '.join(classification.key_sections)}")
    print(f"  Summary: {classification.summary}")

    # Stage 2: Extraction
    print("\n[2/3] Extraction Agent - Extracting clinical entities...")
    extraction_result = Runner.run_sync(extraction_agent, note)
    extraction = extraction_result.final_output

    print(f"  Diagnoses: {len(extraction.diagnoses)}")
    print(f"  Medications: {len(extraction.medications)}")
    print(f"  Procedures: {len(extraction.procedures)}")
    print(f"  Lab Values: {len(extraction.lab_values)}")

    # Flag low-confidence items
    all_entities = (
        extraction.diagnoses + extraction.medications
        + extraction.procedures + extraction.lab_values
    )
    for entity in all_entities:
        if entity.confidence < 0.5:
            entity.requires_review = True
            review_queue.add(entity.text, entity.category, entity.confidence, "Low confidence")
        elif entity.confidence < 0.8:
            review_queue.add(entity.text, entity.category, entity.confidence, "Medium confidence - verify")

    # Stage 3: Medical Coding
    print("\n[3/3] Coding Agent - Mapping to ICD-10 codes...")
    diagnoses_text = "\n".join([f"- {d.text} (confidence: {d.confidence})" for d in extraction.diagnoses])
    coding_input = f"Map these diagnoses to ICD-10-CM codes:\n{diagnoses_text}"
    coding_result = Runner.run_sync(coding_agent, coding_input)
    coding = coding_result.final_output

    print(f"  Coded diagnoses: {len(coding.coded_diagnoses)}")
    print(f"  Flagged for review: {len(coding.flagged_for_review)}")

    # Build FHIR Bundle
    fhir_bundle = build_fhir_bundle(extraction, coding)
    print(f"\n  FHIR Bundle: {len(fhir_bundle['entry'])} entries")

    # Print coded diagnoses
    print(f"\n{'\u2500'*40}")
    print("CODED DIAGNOSES:")
    for dx in coding.coded_diagnoses:
        flag = " [NEEDS REVIEW]" if dx.requires_review else ""
        code = dx.icd10_code or "PENDING"
        print(f"  {code} - {dx.text} (confidence: {dx.confidence:.2f}){flag}")

    # Print review summary
    print(f"\n{'\u2500'*40}")
    auto_approved = sum(1 for e in all_entities if not e.requires_review)
    flagged = sum(1 for e in all_entities if e.requires_review)
    print(f"PIPELINE RESULTS:")
    print(f"  Total entities extracted: {len(all_entities)}")
    print(f"  Auto-approved (confidence >= 0.8): {auto_approved}")
    print(f"  Flagged for review: {flagged}")
    print(f"\n{review_queue.summary()}")

    return {
        "classification": classification.model_dump(),
        "extraction": extraction.model_dump(),
        "coding": coding.model_dump(),
        "fhir_bundle": fhir_bundle,
    }

What does process_clinical_note() do?

This function orchestrates the full pipeline for a single clinical note. It runs three stages sequentially using Runner.run_sync():

  • Stage 1 sends the raw note to the Triage Agent for classification.
  • Stage 2 sends the note to the Extraction Agent, then flags low-confidence entities for review.
  • Stage 3 formats the extraction results as a list and passes them to the Coding Agent for ICD-10 mapping.

After all three stages, it builds a FHIR Bundle from the combined results and prints a summary showing coded diagnoses and flagged items.

  • Below the process_clinical_note() function in the same file, add the main() function and entry point:
def main():
    """Run the Clinical Note Intelligence pipeline."""
    print("Clinical Note Intelligence Agent")
    print("Multi-Agent Pipeline: Triage -> Extraction -> Coding -> FHIR")
    print("=" * 60)

    results = []
    for label, note in SAMPLE_NOTES.items():
        result = process_clinical_note(note, label.upper())
        results.append(result)

    # Final summary
    print(f"\n\n{'='*60}")
    print("PIPELINE SUMMARY")
    print(f"{'='*60}")
    print(f"Documents processed: {len(results)}")
    total_entities = sum(
        len(r["extraction"]["diagnoses"]) + len(r["extraction"]["medications"])
        + len(r["extraction"]["procedures"]) + len(r["extraction"]["lab_values"])
        for r in results
    )
    total_coded = sum(len(r["coding"]["coded_diagnoses"]) for r in results)
    print(f"Total entities extracted: {total_entities}")
    print(f"Total diagnoses coded: {total_coded}")
    print(f"\nView agent traces: https://platform.openai.com/traces")

    # Save FHIR output
    print("\nFHIR Bundle (first document):")
    print(bundle_to_json(results[0]["fhir_bundle"])[:1500])
    print("...")


if __name__ == "__main__":
    main()

What does main() do?

The main() function iterates over all three synthetic notes in SAMPLE_NOTES and processes each one through the pipeline. After all documents are processed, it prints an aggregate summary showing total entities extracted and total diagnoses coded.

It also prints the first 1500 characters of the FHIR Bundle JSON so you can visually inspect the structured output. The traces URL at the end lets you inspect agent decision paths in the OpenAI dashboard.

  • Save main.py.
  • Run the full pipeline by executing:
python main.py

The pipeline takes about 30-60 seconds to process all three notes. You should see output for each stage showing document classification, extracted entities with counts, coded diagnoses with ICD-10 codes, and a FHIR Bundle JSON at the end.

Pipeline not running?

If you see a ModuleNotFoundError, make sure you are running the command from inside the clinical-note-agent/ directory and that your virtual environment is activated.

If you see an authentication error, verify your OPENAI_API_KEY environment variable is set correctly in your current terminal session.

Help me debug my pipeline error

✔️ Awesome, I've got everything!

Great. Double check that main.py is saved and the pipeline ran successfully.

ⓧ I'd like to double check the full code

import json

from agents import Runner

from agents.triage import triage_agent
from agents.extraction import extraction_agent
from agents.coding import coding_agent
from models.fhir import build_fhir_bundle, bundle_to_json
from guardrails.validation import ReviewQueue
from data.sample_notes import SAMPLE_NOTES


def process_clinical_note(note: str, note_label: str) -> dict:
    """Process a single clinical note through the full agent pipeline."""
    print(f"\n{'='*60}")
    print(f"Processing: {note_label}")
    print(f"{'='*60}")

    review_queue = ReviewQueue()

    # Stage 1: Triage
    print("\n[1/3] Triage Agent - Classifying document...")
    triage_result = Runner.run_sync(triage_agent, note)
    classification = triage_result.final_output
    print(f"  Type: {classification.document_type} (confidence: {classification.confidence:.2f})")
    print(f"  Sections: {', '.join(classification.key_sections)}")
    print(f"  Summary: {classification.summary}")

    # Stage 2: Extraction
    print("\n[2/3] Extraction Agent - Extracting clinical entities...")
    extraction_result = Runner.run_sync(extraction_agent, note)
    extraction = extraction_result.final_output

    print(f"  Diagnoses: {len(extraction.diagnoses)}")
    print(f"  Medications: {len(extraction.medications)}")
    print(f"  Procedures: {len(extraction.procedures)}")
    print(f"  Lab Values: {len(extraction.lab_values)}")

    # Flag low-confidence items
    all_entities = (
        extraction.diagnoses + extraction.medications
        + extraction.procedures + extraction.lab_values
    )
    for entity in all_entities:
        if entity.confidence < 0.5:
            entity.requires_review = True
            review_queue.add(entity.text, entity.category, entity.confidence, "Low confidence")
        elif entity.confidence < 0.8:
            review_queue.add(entity.text, entity.category, entity.confidence, "Medium confidence - verify")

    # Stage 3: Medical Coding
    print("\n[3/3] Coding Agent - Mapping to ICD-10 codes...")
    diagnoses_text = "\n".join([f"- {d.text} (confidence: {d.confidence})" for d in extraction.diagnoses])
    coding_input = f"Map these diagnoses to ICD-10-CM codes:\n{diagnoses_text}"
    coding_result = Runner.run_sync(coding_agent, coding_input)
    coding = coding_result.final_output

    print(f"  Coded diagnoses: {len(coding.coded_diagnoses)}")
    print(f"  Flagged for review: {len(coding.flagged_for_review)}")

    # Build FHIR Bundle
    fhir_bundle = build_fhir_bundle(extraction, coding)
    print(f"\n  FHIR Bundle: {len(fhir_bundle['entry'])} entries")

    # Print coded diagnoses
    print(f"\n{'\u2500'*40}")
    print("CODED DIAGNOSES:")
    for dx in coding.coded_diagnoses:
        flag = " [NEEDS REVIEW]" if dx.requires_review else ""
        code = dx.icd10_code or "PENDING"
        print(f"  {code} - {dx.text} (confidence: {dx.confidence:.2f}){flag}")

    # Print review summary
    print(f"\n{'\u2500'*40}")
    auto_approved = sum(1 for e in all_entities if not e.requires_review)
    flagged = sum(1 for e in all_entities if e.requires_review)
    print(f"PIPELINE RESULTS:")
    print(f"  Total entities extracted: {len(all_entities)}")
    print(f"  Auto-approved (confidence >= 0.8): {auto_approved}")
    print(f"  Flagged for review: {flagged}")
    print(f"\n{review_queue.summary()}")

    return {
        "classification": classification.model_dump(),
        "extraction": extraction.model_dump(),
        "coding": coding.model_dump(),
        "fhir_bundle": fhir_bundle,
    }


def main():
    """Run the Clinical Note Intelligence pipeline."""
    print("Clinical Note Intelligence Agent")
    print("Multi-Agent Pipeline: Triage -> Extraction -> Coding -> FHIR")
    print("=" * 60)

    results = []
    for label, note in SAMPLE_NOTES.items():
        result = process_clinical_note(note, label.upper())
        results.append(result)

    # Final summary
    print(f"\n\n{'='*60}")
    print("PIPELINE SUMMARY")
    print(f"{'='*60}")
    print(f"Documents processed: {len(results)}")
    total_entities = sum(
        len(r["extraction"]["diagnoses"]) + len(r["extraction"]["medications"])
        + len(r["extraction"]["procedures"]) + len(r["extraction"]["lab_values"])
        for r in results
    )
    total_coded = sum(len(r["coding"]["coded_diagnoses"]) for r in results)
    print(f"Total entities extracted: {total_entities}")
    print(f"Total diagnoses coded: {total_coded}")
    print(f"\nView agent traces: https://platform.openai.com/traces")

    # Save FHIR output
    print("\nFHIR Bundle (first document):")
    print(bundle_to_json(results[0]["fhir_bundle"])[:1500])
    print("...")


if __name__ == "__main__":
    main()

Your three-agent pipeline is now producing real structured healthcare output. Next, you will add production-grade guardrails that validate extraction output and implement formal human-in-the-loop flagging with confidence thresholds.

Add Guardrails and Human-in-the-Loop Flagging

Your pipeline can now extract clinical entities and code them to ICD-10. But what happens when the extraction agent returns nothing, or flags a vague diagnosis with suspiciously high confidence?

In healthcare, unchecked AI output is dangerous. This step adds two safety layers: an output guardrail that halts execution when extraction fails, and a confidence-based review queue that routes uncertain entities to a human reviewer.

In this step, get ready to:
  • Implement an output guardrail that validates extraction results and triggers a tripwire on failure.
  • Build a review queue that categorizes entities by confidence tier for human-in-the-loop flagging.
  • Run the traced pipeline against all three synthetic notes and observe agent decision paths.
Implement the output guardrail

The OpenAI Agents SDK provides an @output_guardrail decorator that runs validation logic on an agent's final output. If the guardrail's tripwire_triggered field is set to True, the SDK raises an exception and halts execution.

  • Open guardrails/validation.py in VS Code and add the output guardrail function by pasting the following code:
from agents import output_guardrail, GuardrailFunctionOutput, RunContextWrapper, Agent

from models.schemas import ExtractionResult


@output_guardrail
async def validate_extraction_output(
    ctx: RunContextWrapper, agent: Agent, output: ExtractionResult
) -> GuardrailFunctionOutput:
    """Validate that the extraction produced meaningful results."""
    total_entities = (
        len(output.diagnoses)
        + len(output.medications)
        + len(output.procedures)
        + len(output.lab_values)
    )

    if total_entities == 0:
        return GuardrailFunctionOutput(
            output_info={"error": "No entities extracted - extraction may have failed"},
            tripwire_triggered=True,
        )

    # Flag low-confidence entities for review
    all_entities = output.diagnoses + output.medications + output.procedures + output.lab_values
    flagged_count = 0
    for entity in all_entities:
        if entity.confidence < 0.5:
            entity.requires_review = True
            flagged_count += 1

    return GuardrailFunctionOutput(
        output_info={
            "total_entities": total_entities,
            "flagged_for_review": flagged_count,
            "auto_approved": total_entities - flagged_count,
        },
        tripwire_triggered=False,
    )

What does this code do?

The @output_guardrail decorator transforms this async function into an OutputGuardrail that the SDK runs automatically after the Extraction Agent produces its final output.

  • It counts all extracted entities across four categories. If the count is zero, something went wrong and the tripwire fires to halt execution.
  • It loops through every entity and marks anything with confidence below 0.5 as requires_review=True.
  • The output_info dict provides structured metadata that appears in traces for debugging.
Add the review queue class

The guardrail catches catastrophic failures. For the gray area between "definitely correct" and "definitely wrong," you need a review queue that categorizes entities into confidence tiers.

  • Still in guardrails/validation.py, add the ReviewQueue class below the guardrail function:
class ReviewQueue:
    """Collects low-confidence entities for human review."""

    def __init__(self):
        self.items: list[dict] = []

    def add(self, entity_text: str, category: str, confidence: float, reason: str):
        self.items.append({
            "entity": entity_text,
            "category": category,
            "confidence": confidence,
            "reason": reason,
        })

    def summary(self) -> str:
        if not self.items:
            return "All entities auto-approved. No items need human review."
        lines = [f"HUMAN REVIEW REQUIRED ({len(self.items)} items):"]
        for item in self.items:
            lines.append(
                f"  [{item['category'].upper()}] \"{item['entity']}\" "
                f"(confidence: {item['confidence']:.2f}) - {item['reason']}"
            )
        return "\n".join(lines)

What does the ReviewQueue do?

The ReviewQueue collects every entity that falls below the auto-approval threshold. The summary() method prints a formatted report showing exactly which items a human clinician needs to verify before the output goes into production.

This is the human-in-the-loop pattern used in regulated domains. The AI handles the volume, but uncertain cases still get expert eyes.

  • Save guardrails/validation.py.

Now you need to wire the guardrail into the Extraction Agent so the SDK actually runs it.

  • Open agents/extraction.py and find this import block at the top of the file:
from agents import Agent, function_tool

from models.schemas import ExtractionResult
  • Add the guardrail import so the top of the file looks like this:
from agents import Agent, function_tool

from models.schemas import ExtractionResult
from guardrails.validation import validate_extraction_output
  • Scroll down to the extraction_agent definition and find this line:
    output_type=ExtractionResult,
)
  • Replace it with the following to register the guardrail:
    output_type=ExtractionResult,
    output_guardrails=[validate_extraction_output],
)

Why register on the agent instead of the runner?

The output_guardrails parameter on the Agent class tells the SDK to run these checks every time this agent produces output. Any code that calls the Extraction Agent gets guardrail protection automatically without needing to remember to add checks at every call site.

  • Save agents/extraction.py.

Seeing an import error?

Make sure guardrails/validation.py is saved before running anything. Python needs the file on disk to resolve the import from agents/extraction.py.

Also check that your guardrails/ directory has an __init__.py file. If not, create an empty one so Python treats it as a package.

Help me debug my guardrail import error

Run the full pipeline with tracing

The OpenAI Agents SDK has built-in tracing that activates automatically when your OPENAI_API_KEY environment variable is set. Every agent run, tool call, and guardrail check is captured for you to inspect.

  • Your main.py already uses the ReviewQueue to categorize entities into three confidence tiers after extraction. Entities with confidence >= 0.8 are auto-approved. Entities between 0.5 and 0.8 get a warning annotation. Entities below 0.5 are flagged with requires_review=True.
  • Run the full pipeline against all three sample notes by running this command:
python main.py

What should I see?

The pipeline processes three documents sequentially. For each one, you see the Triage classification, extracted entity counts, coded diagnoses with ICD-10 codes, and a review summary.

The progress note (Robert Chen) contains "possibly prerenal" acute kidney injury. This ambiguous phrasing should produce a low-confidence entity that appears in the review queue.

What are traces?

Traces show you every decision the agents made: which tools were called, what the guardrail returned, and how long each stage took. In a production system, this is how you debug why a particular diagnosis was missed or why a guardrail fired unexpectedly.

Pipeline throws an error about a triggered tripwire?

This means the Extraction Agent returned zero entities for a note, which triggered the tripwire. Check that your sample notes in data/sample_notes.py contain actual clinical text. If the error persists, the model may be having trouble parsing the note format. Try running with just the SOAP note first to isolate the issue.

Help me debug my tripwire error

✔️ Awesome, I've got everything!

Great. Double check you have saved both guardrails/validation.py and agents/extraction.py.

ⓧ I'd like to double check the full code

Here is the complete guardrails/validation.py:

from agents import output_guardrail, GuardrailFunctionOutput, RunContextWrapper, Agent

from models.schemas import ExtractionResult


@output_guardrail
async def validate_extraction_output(
    ctx: RunContextWrapper, agent: Agent, output: ExtractionResult
) -> GuardrailFunctionOutput:
    """Validate that the extraction produced meaningful results."""
    total_entities = (
        len(output.diagnoses)
        + len(output.medications)
        + len(output.procedures)
        + len(output.lab_values)
    )

    if total_entities == 0:
        return GuardrailFunctionOutput(
            output_info={"error": "No entities extracted - extraction may have failed"},
            tripwire_triggered=True,
        )

    # Flag low-confidence entities for review
    all_entities = output.diagnoses + output.medications + output.procedures + output.lab_values
    flagged_count = 0
    for entity in all_entities:
        if entity.confidence < 0.5:
            entity.requires_review = True
            flagged_count += 1

    return GuardrailFunctionOutput(
        output_info={
            "total_entities": total_entities,
            "flagged_for_review": flagged_count,
            "auto_approved": total_entities - flagged_count,
        },
        tripwire_triggered=False,
    )


class ReviewQueue:
    """Collects low-confidence entities for human review."""

    def __init__(self):
        self.items: list[dict] = []

    def add(self, entity_text: str, category: str, confidence: float, reason: str):
        self.items.append({
            "entity": entity_text,
            "category": category,
            "confidence": confidence,
            "reason": reason,
        })

    def summary(self) -> str:
        if not self.items:
            return "All entities auto-approved. No items need human review."
        lines = [f"HUMAN REVIEW REQUIRED ({len(self.items)} items):"]
        for item in self.items:
            lines.append(
                f"  [{item['category'].upper()}] \"{item['entity']}\" "
                f"(confidence: {item['confidence']:.2f}) - {item['reason']}"
            )
        return "\n".join(lines)

Here is the complete agents/extraction.py:

from agents import Agent, function_tool

from models.schemas import ExtractionResult
from guardrails.validation import validate_extraction_output


@function_tool
def extract_diagnoses(clinical_text: str) -> str:
    """Extract all diagnoses, conditions, and clinical findings from the text.
    Return each with a confidence score based on how explicitly it is stated."""
    return f"Extract diagnoses from:\n{clinical_text}"


@function_tool
def extract_medications(clinical_text: str) -> str:
    """Extract all medications with dose, route, and frequency from the text."""
    return f"Extract medications from:\n{clinical_text}"


@function_tool
def extract_procedures(clinical_text: str) -> str:
    """Extract all procedures, surgeries, tests, and referrals from the text."""
    return f"Extract procedures from:\n{clinical_text}"


@function_tool
def extract_lab_values(clinical_text: str) -> str:
    """Extract all laboratory values and vital signs with their values and units."""
    return f"Extract lab values from:\n{clinical_text}"


extraction_agent = Agent(
    name="Clinical Extraction Agent",
    instructions="""You are a clinical data extraction specialist. Extract ALL clinical
entities from the provided medical document using the available tools.

Use ALL four tools for each document:
1. extract_diagnoses - conditions, diseases, clinical findings
2. extract_medications - drugs, dosages, routes, frequencies
3. extract_procedures - surgeries, tests, interventions, referrals
4. extract_lab_values - lab results, vital signs, measurements

CONFIDENCE SCORING RULES:
- 1.0: Explicitly stated with clear clinical language
- 0.7-0.9: Strongly implied or clearly present but not the primary focus
- 0.5-0.7: Mentioned but with some ambiguity
- Below 0.5: Qualified with "possible", "probable", "rule out", or vague

Set requires_review=True for ANY entity with confidence below 0.5.
Extract EVERY entity mentioned. Never skip or infer entities not in the text.""",
    tools=[extract_diagnoses, extract_medications, extract_procedures, extract_lab_values],
    output_type=ExtractionResult,
    output_guardrails=[validate_extraction_output],
)

Your pipeline now has two layers of safety. The guardrail catches total failures, and the review queue surfaces uncertain items for human verification. Next up, you will extend this system with an evaluation harness in the Secret Mission.

Secret mission

Build an Evaluation Harness

Your pipeline extracts and codes clinical entities, but how accurate is it really? Build a test harness with ground-truth annotations and precision/recall metrics to prove your pipeline's performance with hard numbers.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally. The only cost is OpenAI API usage (approximately $2-5 depending on how many times you ran the pipeline). There are no persistent cloud resources to worry about.

Decide whether to keep your project files, pause development, or delete everything.

Resources you used:

  • Local project directory clinical-note-agent/ with all Python files, agents, models, and evaluation harness.
  • Python virtual environment inside the project folder.
  • OpenAI API usage (token-based charges for each pipeline and evaluation run).
  • Environment variable OPENAI_API_KEY set in your terminal session.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with the pipeline, adding new clinical notes, or tuning agent instructions.

  • Your project files stay on your Desktop in the clinical-note-agent/ folder.
  • Your virtual environment stays ready to activate whenever you return.
  • OpenAI only charges when you run the pipeline. No idle costs.

Pause - I'll come back to this later

Free up your terminal session while keeping all files intact for later.

  • Deactivate your virtual environment by running:
deactivate
  • Unset the API key from your current terminal session by running:
unset OPENAI_API_KEY
  • When you return, reactivate the virtual environment and re-export your API key.

Delete - I don't want to use this again

Remove all project files and clean up your environment completely.

  • Deactivate your virtual environment first by running:
deactivate
  • Navigate to your Desktop and delete the entire project folder by running:
cd ~/Desktop
rm -rf clinical-note-agent
  • Unset the API key from your current terminal session by running:
unset OPENAI_API_KEY

What about my OpenAI API key?

Deleting the project folder does not revoke your OpenAI API key. If you want to disable the key entirely, visit the API keys page in your OpenAI account settings and delete it there.

Nice Work!

Nice Work!

You just built a production-grade multi-agent clinical NLP pipeline that transforms unstructured medical notes into structured, coded healthcare data.

You've learned how to:

  • Built a multi-agent system with three specialized agents (Triage, Extraction, Coding) using the OpenAI Agents SDK, orchestrated sequentially with Runner.run_sync().
  • Implemented function tools with the @function_tool decorator and output guardrails with confidence-based human-in-the-loop flagging for safe AI in regulated domains.
  • Generated FHIR R4-compliant Bundle output with ICD-10 coded conditions and medication statements for healthcare interoperability.
  • Secret Mission: Built an evaluation harness with precision, recall, and F1 metrics using fuzzy matching to measure pipeline accuracy against ground-truth annotations.

Ready to quiz yourself?