Build a Respiratory Finding Extractor

Extract structured respiratory findings from clinical notes using Python NLP.

Introduction

30 Second Summary

When you visit a doctor, they scribble notes like "bilateral wheezing and bibasilar crackles noted." These notes contain critical health information, but they are locked inside unstructured text that computers cannot easily search or analyze.

In this project, you will build a Python tool that reads raw clinical notes and extracts structured respiratory findings from them. You will turn messy medical shorthand into clean, queryable data.

What You'll Build

A command-line Python tool that processes clinical notes and outputs structured JSON identifying each respiratory finding, its anatomical location, and its canonical medical term.

By the end of this project, you'll have:

  • A respiratory vocabulary that maps clinical synonyms to standardized medical terms.
  • A regex-based pattern matcher that detects findings along with their laterality and location modifiers.
  • A sentence splitter tuned for clinical text that handles medical abbreviations without triggering false breaks.
  • A full extraction pipeline that chains these components together and outputs structured JSON for downstream analytics.
  • Secret Mission: Add negation detection to handle phrases like "no wheezing" and "denies crackles."

Are there any prerequisites?

You need basic Python knowledge (variables, lists, dictionaries, functions, and string methods). No prior NLP or medical informatics experience is required.

Set Up Your Python Environment

To build a clinical text extraction pipeline, you need a Python environment with a few key libraries. Python handles the core logic, and spaCy provides sentence splitting for clinical notes.

In this step, you will install Python, create an isolated project environment, and set up the file structure for your extractor.

In this step, get ready to:
  • Install and verify Python 3.12+.
  • Create the project folder and virtual environment.
  • Install spaCy and create the project files.
Install Python and create your project folder

This project requires Python 3.12 or higher. Check which version you have by running the version command in your terminal.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.12 or higher

You should see something like Python 3.14.5 or Python 3.12.10. Any version 3.12 or above works perfectly. Move on to the next instruction below.

ⓧ I see an older version

If you see a version below 3.12 (like 3.9 or 3.10), you need to install a newer version. macOS often ships with an older Python that the system uses internally.

  • Go to python.org/downloads and download the latest Python installer for your operating system.
  • Run the installer and follow the prompts.
  • On Windows, make sure to check Add Python to PATH during installation.
  • Close and reopen your terminal, then run the version command again to confirm.

Why does macOS show an old version?

macOS ships with a system Python (often 3.9) for internal tools. Installing from python.org adds a new version alongside it. On macOS, use python3 (not python) to invoke the newer version.

ⓧ Command not found

Python is not installed on your machine yet.

  • Go to python.org/downloads and download the latest Python installer for your operating system.
  • Run the installer and follow the prompts.
  • On Windows, make sure to check Add Python to PATH during installation.
  • Close and reopen your terminal, then run the version command again to confirm you see 3.12 or higher.

Now create your project folder on the Desktop so you can find it easily.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir respiratory-extractor
cd respiratory-extractor
  • Confirm you are inside the respiratory-extractor folder by running:
pwd

You should see a path ending in respiratory-extractor.

Next, create a virtual environment to keep your project dependencies isolated from other Python projects on your machine.

  • Create the virtual environment by running this command:

macOS

python3 -m venv venv

Windows

python -m venv venv
  • Activate the virtual environment:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated Python workspace. Packages you install here stay separate from your system Python, so different projects can use different library versions without conflicts.

Don't see (venv) in your prompt?

Make sure you ran the activate command from inside the respiratory-extractor folder. On Windows, if you get a permissions error, run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned first, then try activation again.

help me activate my Python virtual environment

Install spaCy for sentence splitting

The extraction pipeline uses spaCy's sentence splitting component to break clinical notes into individual sentences before processing. Python's built-in re module handles regex matching and does not need a separate install.

  • Install spaCy by running this command:
pip install spacy==3.8.14

What does this command do?

pip install spacy==3.8.14 installs an exact version of spaCy. Pinning the version ensures your results match this project exactly.

  • Download the small English language model by running:
python -m spacy download en_core_web_sm

Do I need this model?

The core project uses spacy.blank("en") for lightweight rule-based sentence splitting. The en_core_web_sm model is useful if you want to explore spaCy's full NLP capabilities later.

You should see download and installation progress messages ending with a success confirmation.

Seeing a connection or permission error?

Make sure you see (venv) in your prompt before running pip commands. If you get a timeout error, check your internet connection and try again.

help me install spaCy in my virtual environment

Create the project files and open VS Code

Your project needs three Python files, each with a specific role in the extraction pipeline.

  • Create all three project files by running these commands:

macOS

touch respiratory_extractor.py vocabulary.py test_extractor.py

Windows

New-Item -ItemType "File" -Name "respiratory_extractor.py", "vocabulary.py", "test_extractor.py"

What are these files for?

  • respiratory_extractor.py is the main pipeline that chains together sentence splitting, regex matching, and normalization.
  • vocabulary.py holds the clinical term dictionaries that map synonyms to canonical medical terms.
  • test_extractor.py contains test cases to verify your extraction logic works correctly.
  • Confirm the files exist by running:
ls

You should see respiratory_extractor.py, vocabulary.py, test_extractor.py, and the venv folder in the output.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the respiratory-extractor folder, and click Open.

You should see all three .py files listed in the Explorer sidebar on the left.

Don't see any files in the sidebar?

Make sure you opened the respiratory-extractor folder itself, not a parent folder. The three .py files should appear at the top level of the Explorer panel.

help me open my project folder in VS Code

Your environment is ready to go. Next up, you will build the respiratory vocabulary that maps clinical synonyms to their canonical medical terms.

Build the Respiratory Vocabulary

Your Python environment is set up and ready to go. Now it's time to teach your extractor what respiratory findings actually look like in clinical text.

Clinicians use many different terms for the same finding. A doctor might write "wheezes", "wheezing", or "expiratory wheeze" to describe the same thing. Your vocabulary layer maps all these variants to a single canonical term so your extractor produces consistent, queryable output.

In this step, get ready to:
  • Define a canonical findings dictionary covering 5 respiratory finding categories.
  • Build laterality and location modifier dictionaries for anatomical context.
  • Create lookup functions that normalize raw clinical text to canonical terms.
Define the canonical findings dictionary

Clinical text is full of synonyms. "Rales" and "crepitations" both mean the same thing as "crackles". A vocabulary dictionary gives your extractor one source of truth for what counts as a valid finding and what its standardized name should be.

  • Switch back to VS Code and open vocabulary.py from the Explorer sidebar.
  • Add the module docstring, the findings synonym dictionary, and the reverse lookup builder by pasting this code:
"""Respiratory findings vocabulary for clinical NLP extraction."""

# Canonical findings: maps synonyms/variants to a single canonical term
FINDING_SYNONYMS = {
    "wheezing": ["wheeze", "wheezes", "wheezing", "expiratory wheeze", "inspiratory wheeze"],
    "crackles": ["crackle", "crackles", "rales", "crepitations", "crepitant"],
    "rhonchi": ["rhonchi", "rhonchus", "coarse breath sounds"],
    "stridor": ["stridor", "stridorous", "stridulent"],
    "diminished breath sounds": ["diminished breath sounds", "decreased breath sounds",
                                  "reduced breath sounds", "dim bs", "decreased bs"],
}

# Build a reverse lookup: variant -> canonical term
VARIANT_TO_CANONICAL = {}
for canonical, variants in FINDING_SYNONYMS.items():
    for variant in variants:
        VARIANT_TO_CANONICAL[variant.lower()] = canonical

What does this code do?

FINDING_SYNONYMS is the core vocabulary. Each key is a canonical term (the "official" name), and each value is a list of all the ways clinicians might write that finding.

VARIANT_TO_CANONICAL flips that relationship. Instead of looking up "wheezing" to find its variants, you look up any variant (like "rales") and instantly get back the canonical term ("crackles"). This reverse lookup is what makes fast normalization possible.

  • Save vocabulary.py.
  • You won't see a visible change yet. The next chunks add the modifier dictionaries and lookup functions that bring this vocabulary to life.

Why a reverse lookup instead of searching every list?

Without VARIANT_TO_CANONICAL, you'd have to loop through every canonical term's list to find a match. The reverse dictionary gives you O(1) lookups. This is the same pattern used by production clinical NLP systems like the HYPHEN normalization pipeline.

Define laterality and location modifier dictionaries

Respiratory findings rarely appear alone. Clinicians attach modifiers that describe where the finding was heard. "Bilateral wheezing" tells you the sound is on both sides. "Bibasilar crackles" tells you the sound is at both lung bases.

  • In vocabulary.py, add the laterality and location modifier dictionaries below the VARIANT_TO_CANONICAL block:
# Laterality modifiers: maps variants to normalized form
LATERALITY_MODIFIERS = {
    "bilateral": "bilateral",
    "b/l": "bilateral",
    "bilat": "bilateral",
    "left": "left",
    "lt": "left",
    "l": "left",
    "right": "right",
    "rt": "right",
    "r": "right",
}

# Location modifiers: maps variants to normalized form
LOCATION_MODIFIERS = {
    "bibasilar": "bibasilar",
    "basilar": "basilar",
    "basal": "basilar",
    "apical": "apical",
    "upper lobe": "upper lobe",
    "lower lobe": "lower lobe",
    "right lower lobe": "right lower lobe",
    "left lower lobe": "left lower lobe",
    "right upper lobe": "right upper lobe",
    "left upper lobe": "left upper lobe",
}

What does this code do?

LATERALITY_MODIFIERS normalizes side-of-body terms. Clinicians write "b/l", "bilat", or "bilateral" interchangeably. This dictionary collapses them all to one canonical form.

LOCATION_MODIFIERS does the same for anatomical locations within the lungs. "Basal" and "basilar" both normalize to "basilar". Multi-word locations like "right lower lobe" are kept intact as a single key.

  • Save vocabulary.py.
  • You won't see a change yet. The next chunk adds the lookup functions that use these dictionaries.
Create the normalization lookup functions

Your dictionaries hold the data, but you need functions to perform the actual lookups. Each function takes raw text, cleans it, and returns the normalized canonical form.

  • In vocabulary.py, add the three normalization functions below the LOCATION_MODIFIERS dictionary:
def normalize_finding(text):
    """Look up a text string in the vocabulary and return its canonical term.

    Args:
        text: Raw finding text from clinical note.

    Returns:
        Canonical term string, or None if no match found.
    """
    cleaned = text.strip().lower()
    return VARIANT_TO_CANONICAL.get(cleaned, None)


def normalize_laterality(text):
    """Normalize a laterality modifier to its canonical form.

    Args:
        text: Raw laterality text.

    Returns:
        Normalized laterality string, or None if no match.
    """
    cleaned = text.strip().lower()
    return LATERALITY_MODIFIERS.get(cleaned, None)


def normalize_location(text):
    """Normalize a location modifier to its canonical form.

    Args:
        text: Raw location text.

    Returns:
        Normalized location string, or None if no match.
    """
    cleaned = text.strip().lower()
    return LOCATION_MODIFIERS.get(cleaned, None)

What does this code do?

Each function follows the same pattern: strip whitespace, lowercase the input, then look it up in the corresponding dictionary. This handles the common case sensitivity issues in clinical text ("Wheezing" vs "wheezing" vs "WHEEZING" all resolve correctly).

Returning None when no match is found lets the calling code distinguish between "this is a known finding" and "this text doesn't match anything in our vocabulary."

  • Save vocabulary.py.
  • Test the vocabulary interactively by running this command in your terminal:
python -c "from vocabulary import normalize_finding; print(normalize_finding('wheeze'))"
  • You should see wheezing printed in the terminal. This confirms that the variant "wheeze" correctly resolves to the canonical term "wheezing".

Seeing a ModuleNotFoundError?

Make sure your terminal is in the respiratory-extractor folder (the one containing vocabulary.py). Python looks for modules in the current directory.

Also confirm your virtual environment is still active. You should see (venv) at the start of your terminal prompt.

Help me debug my vocabulary import error.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete vocabulary.py file for comparison:

"""Respiratory findings vocabulary for clinical NLP extraction."""

# Canonical findings: maps synonyms/variants to a single canonical term
FINDING_SYNONYMS = {
    "wheezing": ["wheeze", "wheezes", "wheezing", "expiratory wheeze", "inspiratory wheeze"],
    "crackles": ["crackle", "crackles", "rales", "crepitations", "crepitant"],
    "rhonchi": ["rhonchi", "rhonchus", "coarse breath sounds"],
    "stridor": ["stridor", "stridorous", "stridulent"],
    "diminished breath sounds": ["diminished breath sounds", "decreased breath sounds",
                                  "reduced breath sounds", "dim bs", "decreased bs"],
}

# Build a reverse lookup: variant -> canonical term
VARIANT_TO_CANONICAL = {}
for canonical, variants in FINDING_SYNONYMS.items():
    for variant in variants:
        VARIANT_TO_CANONICAL[variant.lower()] = canonical

# Laterality modifiers: maps variants to normalized form
LATERALITY_MODIFIERS = {
    "bilateral": "bilateral",
    "b/l": "bilateral",
    "bilat": "bilateral",
    "left": "left",
    "lt": "left",
    "l": "left",
    "right": "right",
    "rt": "right",
    "r": "right",
}

# Location modifiers: maps variants to normalized form
LOCATION_MODIFIERS = {
    "bibasilar": "bibasilar",
    "basilar": "basilar",
    "basal": "basilar",
    "apical": "apical",
    "upper lobe": "upper lobe",
    "lower lobe": "lower lobe",
    "right lower lobe": "right lower lobe",
    "left lower lobe": "left lower lobe",
    "right upper lobe": "right upper lobe",
    "left upper lobe": "left upper lobe",
}


def normalize_finding(text):
    """Look up a text string in the vocabulary and return its canonical term.

    Args:
        text: Raw finding text from clinical note.

    Returns:
        Canonical term string, or None if no match found.
    """
    cleaned = text.strip().lower()
    return VARIANT_TO_CANONICAL.get(cleaned, None)


def normalize_laterality(text):
    """Normalize a laterality modifier to its canonical form.

    Args:
        text: Raw laterality text.

    Returns:
        Normalized laterality string, or None if no match.
    """
    cleaned = text.strip().lower()
    return LATERALITY_MODIFIERS.get(cleaned, None)


def normalize_location(text):
    """Normalize a location modifier to its canonical form.

    Args:
        text: Raw location text.

    Returns:
        Normalized location string, or None if no match.
    """
    cleaned = text.strip().lower()
    return LOCATION_MODIFIERS.get(cleaned, None)

Your vocabulary layer is complete. It knows 5 finding categories, handles laterality like "bilateral" and "b/l", and normalizes locations like "bibasilar" and "basal". Next up, you'll build the sentence splitter that chops clinical notes into individual sentences for processing.

Implement Sentence Splitting for Clinical Text

Your vocabulary module can now map clinical synonyms to canonical terms. The next challenge is figuring out where one clinical observation ends and the next begins.

Clinical notes are dense. A single paragraph might contain multiple findings separated by periods, but those periods also appear in abbreviations like "pt." or "Dr." In this step, you'll build a sentence splitter that handles these tricky cases.

In this step, get ready to:
  • Set up imports and configure spaCy for clinical text processing.
  • Build an abbreviation handler to prevent false sentence breaks.
  • Create reusable functions for splitting clinical notes into sentences.
Set up the extractor file with imports

Before you can split sentences, you need to import the libraries your extractor depends on. The spacy library provides the sentence-splitting engine, and your vocabulary.py module supplies the term dictionaries you built in the previous step.

  • Open respiratory_extractor.py in VS Code and paste the following imports:
"""Respiratory finding extraction pipeline for clinical NLP."""

import re
import json
import spacy
from spacy.language import Language

from vocabulary import (
    FINDING_SYNONYMS,
    normalize_finding,
    normalize_laterality,
    normalize_location,
    LATERALITY_MODIFIERS,
    LOCATION_MODIFIERS,
)

What do these imports do?

  • re is Python's built-in regex module for pattern matching (used in later steps).
  • json formats extraction results as structured output.
  • spacy provides the sentence-splitting pipeline.
  • Language from spacy.language lets you register custom pipeline components.
  • The vocabulary imports bring in your dictionaries and normalization functions from the previous step.
  • Save respiratory_extractor.py.

You won't see a change yet. The next chunk adds the abbreviation handler that these imports support.

Handle clinical abbreviations that confuse sentence splitters

Standard sentence splitters treat every period as a sentence boundary. Clinical notes are full of abbreviations like "pt.", "Dr.", and "b/l." that contain periods but are not sentence endings. Without special handling, the text "Pt. presents with wheezing." would be incorrectly split into two fragments.

  • Add this abbreviation set and handler component below your imports in respiratory_extractor.py:
# --- Sentence Splitting ---

# Clinical abbreviations that should NOT trigger sentence boundaries
CLINICAL_ABBREVIATIONS = {"pt", "dr", "approx", "b/l", "bilat", "lt", "rt", "dept", "dx", "hx"}


@Language.component("clinical_abbreviation_handler")
def clinical_abbreviation_handler(doc):
    """Prevent sentence splits after common clinical abbreviations."""
    for i, token in enumerate(doc[:-1]):
        if token.text.lower().rstrip(".") in CLINICAL_ABBREVIATIONS:
            doc[i + 1].is_sent_start = False
    return doc

What does this code do?

CLINICAL_ABBREVIATIONS is a set of common clinical shorthand terms (without their trailing periods). The @Language.component decorator registers your function as a reusable spaCy pipeline component.

The handler loops through every token in the document. When it finds a token whose text (stripped of its period) matches an abbreviation, it tells spaCy that the next token does NOT start a new sentence. This prevents "Pt." from triggering a false sentence break.

  • Save respiratory_extractor.py.

You won't see a change yet. The next chunk creates the functions that use this handler.

Create the sentence splitter functions

Now you'll wrap the spaCy pipeline setup and sentence splitting logic into two reusable functions. One builds the pipeline, the other uses it to split any clinical text into clean sentences.

  • Add these two functions below the clinical_abbreviation_handler function in respiratory_extractor.py:
def create_sentence_splitter():
    """Create a spaCy pipeline configured for clinical sentence splitting.

    Returns:
        A spaCy Language object with sentencizer and abbreviation handling.
    """
    nlp = spacy.blank("en")
    nlp.add_pipe("sentencizer")
    nlp.add_pipe("clinical_abbreviation_handler", after="sentencizer")
    return nlp


def split_sentences(text, nlp=None):
    """Split clinical text into sentences.

    Args:
        text: Raw clinical note text.
        nlp: Optional pre-created spaCy pipeline. Creates one if not provided.

    Returns:
        List of sentence strings.
    """
    if nlp is None:
        nlp = create_sentence_splitter()
    doc = nlp(text)
    return [sent.text.strip() for sent in doc.sents if sent.text.strip()]

What does this code do?

  • create_sentence_splitter() builds a blank English pipeline with spacy.blank("en"), adds the built-in sentencizer component for punctuation-based splitting, then layers your abbreviation handler on top.
  • split_sentences() accepts raw clinical text and an optional pre-built pipeline. It processes the text through spaCy and returns a list of cleaned sentence strings, filtering out any empty results.
  • The nlp=None parameter lets callers reuse a single pipeline across multiple calls for efficiency, or creates a fresh one if none is provided.
  • Save respiratory_extractor.py.
  • Test the sentence splitter on a standard clinical sentence by running this in your terminal:
python -c "from respiratory_extractor import split_sentences; print(split_sentences('Bilateral wheezing and bibasilar crackles noted. Lungs clear to auscultation bilaterally.'))"

You should see a list with exactly two sentences printed to the terminal.

  • Now test the abbreviation handling by running:
python -c "from respiratory_extractor import split_sentences; print(split_sentences('Pt. presents with wheezing. Dr. Smith noted crackles.'))"

You should see exactly 2 sentences. Without your abbreviation handler, this would incorrectly split into 4 fragments at every period.

Seeing an import error or unexpected splits?

  • If you see ModuleNotFoundError: No module named 'vocabulary', make sure your terminal is in the respiratory-extractor folder where both files live.
  • If you see ModuleNotFoundError: No module named 'spacy', confirm your virtual environment is activated (you should see (venv) in your terminal prompt).
  • If your abbreviation test splits into more than 2 sentences, double check that CLINICAL_ABBREVIATIONS includes "pt" and "dr" (without periods).

Why isn't my sentence splitter working correctly?

✔️ Awesome, I've got everything!

Great. Double check you've saved respiratory_extractor.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete respiratory_extractor.py file so far:

"""Respiratory finding extraction pipeline for clinical NLP."""

import re
import json
import spacy
from spacy.language import Language

from vocabulary import (
    FINDING_SYNONYMS,
    normalize_finding,
    normalize_laterality,
    normalize_location,
    LATERALITY_MODIFIERS,
    LOCATION_MODIFIERS,
)

# --- Sentence Splitting ---

# Clinical abbreviations that should NOT trigger sentence boundaries
CLINICAL_ABBREVIATIONS = {"pt", "dr", "approx", "b/l", "bilat", "lt", "rt", "dept", "dx", "hx"}


@Language.component("clinical_abbreviation_handler")
def clinical_abbreviation_handler(doc):
    """Prevent sentence splits after common clinical abbreviations."""
    for i, token in enumerate(doc[:-1]):
        if token.text.lower().rstrip(".") in CLINICAL_ABBREVIATIONS:
            doc[i + 1].is_sent_start = False
    return doc


def create_sentence_splitter():
    """Create a spaCy pipeline configured for clinical sentence splitting.

    Returns:
        A spaCy Language object with sentencizer and abbreviation handling.
    """
    nlp = spacy.blank("en")
    nlp.add_pipe("sentencizer")
    nlp.add_pipe("clinical_abbreviation_handler", after="sentencizer")
    return nlp


def split_sentences(text, nlp=None):
    """Split clinical text into sentences.

    Args:
        text: Raw clinical note text.
        nlp: Optional pre-created spaCy pipeline. Creates one if not provided.

    Returns:
        List of sentence strings.
    """
    if nlp is None:
        nlp = create_sentence_splitter()
    doc = nlp(text)
    return [sent.text.strip() for sent in doc.sents if sent.text.strip()]

Your sentence splitter is ready. Next up, you'll build the regex pattern matcher that scans each sentence for respiratory findings and their modifiers.

Build the Regex Pattern Matcher

Your sentence splitter can now break clinical notes into individual sentences. The next challenge is finding the actual respiratory findings inside each sentence.

Clinical sentences like "Bilateral wheezing and bibasilar crackles noted" pack multiple findings with modifiers into a single line. In this step, you will build a regex pattern matcher that automatically detects finding terms alongside their laterality and location modifiers.

In this step, get ready to:
  • Write a regex pattern builder that dynamically creates patterns from your vocabulary dictionaries.
  • Build a multi-finding extraction function that handles coordinated phrases.
  • Test the matcher against a real clinical sentence containing multiple findings.
Write regex patterns that capture laterality and finding pairs

A clinical finding rarely appears alone. It typically comes with a modifier like "bilateral" or "bibasilar" that describes where on the body the finding was observed. Your regex pattern needs to capture both pieces together.

Instead of hard-coding every possible finding term into the regex, you will build the pattern dynamically from the vocabulary dictionaries you already created. This means adding a new term to your vocabulary automatically updates the regex.

  • In respiratory_extractor.py, find the split_sentences function at the bottom of the file.
  • Add the following code below it to create the regex pattern builder:
# --- Regex Pattern Building ---

def build_finding_pattern():
    """Build a regex pattern from the vocabulary that matches all finding variants.

    Returns:
        Compiled regex pattern for finding extraction.
    """
    # Collect all variants, sorted longest-first to avoid partial matches
    all_variants = []
    for canonical, variants in FINDING_SYNONYMS.items():
        all_variants.extend(variants)
    all_variants.sort(key=len, reverse=True)

    # Escape for regex safety
    escaped_variants = [re.escape(v) for v in all_variants]
    finding_alternation = "|".join(escaped_variants)

    # Collect laterality modifiers
    lat_terms = list(LATERALITY_MODIFIERS.keys())
    lat_terms.sort(key=len, reverse=True)
    lat_alternation = "|".join(re.escape(t) for t in lat_terms)

    # Collect location modifiers
    loc_terms = list(LOCATION_MODIFIERS.keys())
    loc_terms.sort(key=len, reverse=True)
    loc_alternation = "|".join(re.escape(t) for t in loc_terms)

    # Combined pattern: optional (laterality OR location) + finding
    pattern = (
        rf"(?P<modifier>(?:{lat_alternation}|{loc_alternation}))\s+(?P<finding>{finding_alternation})"
        rf"|(?P<finding_only>{finding_alternation})"
    )

    return re.compile(pattern, re.IGNORECASE)

What does this code do?

This function dynamically constructs a regex pattern from your vocabulary dictionaries rather than hard-coding terms.

  • Sorting variants longest-first prevents partial matches. Without this, "breath sounds" might match before "diminished breath sounds" gets a chance.
  • re.escape() handles special characters like "/" in "b/l" so they are treated as literal text.
  • The pattern uses two alternatives separated by |: one for modifier + finding, and one for a finding appearing alone.
  • re.IGNORECASE ensures the pattern matches "Wheezing", "WHEEZING", and "wheezing" equally.
  • Save respiratory_extractor.py.
  • Verify the pattern compiles by running this in your terminal:
python -c "from respiratory_extractor import build_finding_pattern; p = build_finding_pattern(); print(p.pattern[:80] + '...')"

You should see the first 80 characters of the generated regex pattern printed, confirming it compiled without errors.

Seeing an ImportError?

  • Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.
  • Confirm both vocabulary.py and respiratory_extractor.py are saved in the same respiratory-extractor folder.
  • Check that your import statement at the top of respiratory_extractor.py includes LATERALITY_MODIFIERS and LOCATION_MODIFIERS.

Why is my regex pattern not compiling?

Handle multi-finding extraction from coordinated phrases

Clinical sentences frequently contain multiple findings connected by conjunctions. "Bilateral wheezing and bibasilar crackles noted" has two distinct findings in one sentence. Your extraction function needs to find them all.

Python's finditer() method handles this naturally. It returns every non-overlapping match in the string, so coordinated findings are captured without needing to manually split on conjunctions.

  • In respiratory_extractor.py, add the following code below the build_finding_pattern() function:
# --- Multi-Finding Extraction ---

def extract_findings(sentence, pattern=None):
    """Extract respiratory findings from a single sentence.

    Handles coordinated findings separated by 'and', commas, or 'or'.

    Args:
        sentence: A single sentence string from clinical text.
        pattern: Optional pre-compiled regex pattern.

    Returns:
        List of dictionaries with keys: finding, raw_text, laterality, location.
    """
    if pattern is None:
        pattern = build_finding_pattern()

    results = []

    for match in pattern.finditer(sentence):
        raw_text = match.group(0).strip()

        # Determine which alternative matched
        modifier_text = match.group("modifier") if match.group("modifier") else None
        finding_text = match.group("finding") if match.group("finding") else match.group("finding_only")

        # Normalize the finding
        canonical = normalize_finding(finding_text)
        if canonical is None:
            continue

        # Determine if modifier is laterality or location
        laterality = None
        location = None

        if modifier_text:
            lat_result = normalize_laterality(modifier_text)
            loc_result = normalize_location(modifier_text)
            if lat_result:
                laterality = lat_result
            elif loc_result:
                location = loc_result

        results.append({
            "finding": canonical,
            "raw_text": raw_text,
            "laterality": laterality,
            "location": location,
        })

    return results

What does this code do?

  • pattern.finditer(sentence) iterates over every match in the sentence. If "wheezing" and "crackles" both appear, it finds both.
  • Named groups (modifier, finding, finding_only) identify which part of the regex matched. A finding with a modifier uses the first alternative. A finding alone uses the second.
  • Each match is normalized through your vocabulary functions. If normalize_finding() returns None, the match is skipped.
  • The modifier is classified as either laterality or location by checking both normalization functions.

Why does the pattern parameter default to None?

Compiling a regex pattern is relatively expensive. By accepting a pre-compiled pattern as a parameter, the caller can compile it once and reuse it across many sentences. This matters when processing a full clinical note with dozens of sentences.

  • Save respiratory_extractor.py.

Seeing a regex error about named groups?

  • Check that you have the pipe character | between the two rf-strings in build_finding_pattern(). The second line of the pattern must start with rf"|.
  • Ensure you did not accidentally add extra spaces around the named group syntax (?P<name>...).

Help me debug my regex named groups in the respiratory extractor.

Test the regex matcher against a clinical sentence

Time to see if the pattern matcher correctly identifies multiple findings with their modifiers from a single clinical sentence.

  • Run this test in your terminal:
python -c "
from respiratory_extractor import extract_findings, build_finding_pattern
import json

pattern = build_finding_pattern()
results = extract_findings('Bilateral wheezing and bibasilar crackles noted.', pattern)
print(json.dumps(results, indent=2))
"

You should see two findings in the output: one for "wheezing" with laterality "bilateral", and one for "crackles" with location "bibasilar".

What should I see?

Your output should look similar to this:

[
  {
    "finding": "wheezing",
    "raw_text": "Bilateral wheezing",
    "laterality": "bilateral",
    "location": null
  },
  {
    "finding": "crackles",
    "raw_text": "bibasilar crackles",
    "laterality": null,
    "location": "bibasilar"
  }
]

The matcher correctly separated "Bilateral wheezing and bibasilar crackles" into two distinct findings, each with its own modifier classified as either laterality or location.

Only seeing one finding instead of two?

  • Confirm that your vocabulary's FINDING_SYNONYMS dictionary includes both "wheezing" and "crackles" as variant entries.
  • Make sure the sort in build_finding_pattern() uses key=len, reverse=True so longer terms are matched first.
  • Check that you included re.IGNORECASE in the re.compile() call. Without it, "Bilateral" (capitalized) will not match.

Why is my regex only finding one match in the sentence?

✔️ Awesome, I've got everything!

Great. Double check you have saved respiratory_extractor.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete respiratory_extractor.py as it should look at the end of this step:

"""Respiratory finding extraction pipeline for clinical NLP."""

import re
import json
import spacy
from spacy.language import Language

from vocabulary import (
    FINDING_SYNONYMS,
    normalize_finding,
    normalize_laterality,
    normalize_location,
    LATERALITY_MODIFIERS,
    LOCATION_MODIFIERS,
)

# --- Sentence Splitting ---

CLINICAL_ABBREVIATIONS = {"pt", "dr", "approx", "b/l", "bilat", "lt", "rt", "dept", "dx", "hx"}


@Language.component("clinical_abbreviation_handler")
def clinical_abbreviation_handler(doc):
    """Prevent sentence splits after common clinical abbreviations."""
    for i, token in enumerate(doc[:-1]):
        if token.text.lower().rstrip(".") in CLINICAL_ABBREVIATIONS:
            doc[i + 1].is_sent_start = False
    return doc


def create_sentence_splitter():
    nlp = spacy.blank("en")
    nlp.add_pipe("sentencizer")
    nlp.add_pipe("clinical_abbreviation_handler", after="sentencizer")
    return nlp


def split_sentences(text, nlp=None):
    if nlp is None:
        nlp = create_sentence_splitter()
    doc = nlp(text)
    return [sent.text.strip() for sent in doc.sents if sent.text.strip()]


# --- Regex Pattern Building ---

def build_finding_pattern():
    """Build a regex pattern from the vocabulary that matches all finding variants.

    Returns:
        Compiled regex pattern for finding extraction.
    """
    # Collect all variants, sorted longest-first to avoid partial matches
    all_variants = []
    for canonical, variants in FINDING_SYNONYMS.items():
        all_variants.extend(variants)
    all_variants.sort(key=len, reverse=True)

    # Escape for regex safety
    escaped_variants = [re.escape(v) for v in all_variants]
    finding_alternation = "|".join(escaped_variants)

    # Collect laterality modifiers
    lat_terms = list(LATERALITY_MODIFIERS.keys())
    lat_terms.sort(key=len, reverse=True)
    lat_alternation = "|".join(re.escape(t) for t in lat_terms)

    # Collect location modifiers
    loc_terms = list(LOCATION_MODIFIERS.keys())
    loc_terms.sort(key=len, reverse=True)
    loc_alternation = "|".join(re.escape(t) for t in loc_terms)

    # Combined pattern: optional (laterality OR location) + finding
    pattern = (
        rf"(?P<modifier>(?:{lat_alternation}|{loc_alternation}))\s+(?P<finding>{finding_alternation})"
        rf"|(?P<finding_only>{finding_alternation})"
    )

    return re.compile(pattern, re.IGNORECASE)


# --- Multi-Finding Extraction ---

def extract_findings(sentence, pattern=None):
    """Extract respiratory findings from a single sentence.

    Handles coordinated findings separated by 'and', commas, or 'or'.

    Args:
        sentence: A single sentence string from clinical text.
        pattern: Optional pre-compiled regex pattern.

    Returns:
        List of dictionaries with keys: finding, raw_text, laterality, location.
    """
    if pattern is None:
        pattern = build_finding_pattern()

    results = []

    for match in pattern.finditer(sentence):
        raw_text = match.group(0).strip()

        # Determine which alternative matched
        modifier_text = match.group("modifier") if match.group("modifier") else None
        finding_text = match.group("finding") if match.group("finding") else match.group("finding_only")

        # Normalize the finding
        canonical = normalize_finding(finding_text)
        if canonical is None:
            continue

        # Determine if modifier is laterality or location
        laterality = None
        location = None

        if modifier_text:
            lat_result = normalize_laterality(modifier_text)
            loc_result = normalize_location(modifier_text)
            if lat_result:
                laterality = lat_result
            elif loc_result:
                location = loc_result

        results.append({
            "finding": canonical,
            "raw_text": raw_text,
            "laterality": laterality,
            "location": location,
        })

    return results

Your pattern matcher can now extract multiple respiratory findings from a single sentence and classify their modifiers. Next up, you will wire everything together into a full extraction pipeline that processes multi-sentence clinical notes and outputs structured JSON.

Assemble the Full Extraction Pipeline

You now have all the individual pieces: a sentence splitter for clinical text, a vocabulary-based normalizer, and a regex pattern matcher.

In this step, you will chain these components into a single pipeline function that accepts raw clinical text and outputs structured JSON data. This is the core orchestration pattern used in production clinical NLP systems.

In this step, get ready to:
  • Create the main pipeline function that chains all extraction stages together.
  • Define a structured output schema with finding, location, laterality, and sentence metadata.
  • Run the full pipeline on a multi-sentence clinical note and verify the JSON output.
Create the main extraction pipeline function

A pipeline function orchestrates each stage in sequence: split the text into sentences, match patterns in each sentence, normalize the matches, and assemble the results into a structured list.

  • In respiratory_extractor.py, scroll to the bottom of the file (after the extract_findings function).
  • Add the full pipeline function by pasting the following code at the end of the file:
# --- Full Pipeline ---

def extract_respiratory_findings(clinical_text):
    """Run the full respiratory finding extraction pipeline.

    Pipeline stages:
    1. Sentence splitting
    2. Regex matching per sentence
    3. Vocabulary normalization
    4. Structured output assembly

    Args:
        clinical_text: Raw clinical note text (one or more sentences).

    Returns:
        List of structured finding dictionaries.
    """
    nlp = create_sentence_splitter()
    pattern = build_finding_pattern()
    sentences = split_sentences(clinical_text, nlp)

    all_findings = []

    for idx, sentence in enumerate(sentences):
        findings = extract_findings(sentence, pattern)
        for finding in findings:
            finding["sentence_index"] = idx
            finding["source_sentence"] = sentence
            all_findings.append(finding)

    return all_findings

What does this code do?

This function is the orchestrator that chains your four pipeline stages together in sequence.

  • create_sentence_splitter() and build_finding_pattern() are called once upfront so you are not rebuilding the spaCy pipeline and regex for every sentence.
  • The loop iterates over each sentence, runs extract_findings() to get raw matches, then enriches each match with sentence_index and source_sentence metadata.
  • Each finding dictionary ends up with six keys: finding, raw_text, laterality, location, sentence_index, and source_sentence.
  • Save respiratory_extractor.py.

You will not see a change yet. The next chunk adds the entry point that actually calls this function.

Add the main entry point

The entry point defines a sample clinical note, runs the pipeline, and prints the results as formatted JSON so you can verify the output schema.

  • In respiratory_extractor.py, paste the following code at the very bottom of the file (below the extract_respiratory_findings function you just added):
# --- Main Entry Point ---

if __name__ == "__main__":
    # Example clinical text
    clinical_note = "Bilateral wheezing and bibasilar crackles noted. No rhonchi. Diminished breath sounds at left base."

    print("=" * 60)
    print("RESPIRATORY FINDING EXTRACTOR")
    print("=" * 60)
    print(f"\nInput: {clinical_note}\n")

    # Run extraction
    findings = extract_respiratory_findings(clinical_note)

    # Display results
    print("Extracted Findings:")
    print("-" * 40)
    print(json.dumps(findings, indent=2))

    print(f"\nTotal findings extracted: {len(findings)}")

What does this code do?

  • if __name__ == "__main__": ensures this block only runs when you execute the file directly (not when importing from it).
  • The clinical_note string contains three sentences with different findings, modifiers, and edge cases.
  • json.dumps(findings, indent=2) prints the list of dictionaries as nicely indented JSON so you can inspect each field.
  • Save respiratory_extractor.py.

Seeing a syntax error on save?

Make sure the if __name__ block is at the top level of the file (no indentation), not nested inside another function.

Check that the f-string quotes match. The line print(f"\nInput: {clinical_note}\n") uses double quotes for the f-string and curly braces for the variable.

Help me fix a syntax error in my respiratory_extractor.py file.

Run the full pipeline and verify the output
  • Run the extraction pipeline by executing this command in your terminal:
python respiratory_extractor.py

You should see a formatted output with the header, the input text, and a JSON array of extracted findings.

What should I see?

The output should show findings extracted from the clinical note. Look for these key results:

  • A finding with "finding": "wheezing" and "laterality": "bilateral" from the first sentence.
  • A finding with "finding": "crackles" and "location": "bibasilar" from the first sentence.
  • A finding with "finding": "rhonchi" from the second sentence.
  • A finding with "finding": "diminished breath sounds" from the third sentence.
  • Each finding includes a sentence_index (0, 1, or 2) and a source_sentence showing which sentence it came from.

Not seeing any findings in the output?

  • Make sure your virtual environment is still active. You should see (venv) in your terminal prompt.
  • Check that vocabulary.py is in the same folder as respiratory_extractor.py. The import will fail silently if the file is missing.
  • Verify the clinical_note string is on a single line with no accidental line breaks inside the quotes.

Help me debug why my respiratory extractor is not finding any matches.

✔️ Awesome, I've got everything!

Great. Double check you have saved respiratory_extractor.py and your pipeline output matches the expected findings.

ⓧ I'd like to double check the full code

Here is the complete respiratory_extractor.py file as it should look at this point:

"""Respiratory finding extraction pipeline for clinical NLP."""

import re
import json
import spacy
from spacy.language import Language

from vocabulary import (
    FINDING_SYNONYMS,
    normalize_finding,
    normalize_laterality,
    normalize_location,
    LATERALITY_MODIFIERS,
    LOCATION_MODIFIERS,
)

# --- Sentence Splitting ---

CLINICAL_ABBREVIATIONS = {"pt", "dr", "approx", "b/l", "bilat", "lt", "rt", "dept", "dx", "hx"}


@Language.component("clinical_abbreviation_handler")
def clinical_abbreviation_handler(doc):
    """Prevent sentence splits after common clinical abbreviations."""
    for i, token in enumerate(doc[:-1]):
        if token.text.lower().rstrip(".") in CLINICAL_ABBREVIATIONS:
            doc[i + 1].is_sent_start = False
    return doc


def create_sentence_splitter():
    nlp = spacy.blank("en")
    nlp.add_pipe("sentencizer")
    nlp.add_pipe("clinical_abbreviation_handler", after="sentencizer")
    return nlp


def split_sentences(text, nlp=None):
    if nlp is None:
        nlp = create_sentence_splitter()
    doc = nlp(text)
    return [sent.text.strip() for sent in doc.sents if sent.text.strip()]


# --- Regex Pattern Building ---

def build_finding_pattern():
    all_variants = []
    for canonical, variants in FINDING_SYNONYMS.items():
        all_variants.extend(variants)
    all_variants.sort(key=len, reverse=True)

    escaped_variants = [re.escape(v) for v in all_variants]
    finding_alternation = "|".join(escaped_variants)

    lat_terms = list(LATERALITY_MODIFIERS.keys())
    lat_terms.sort(key=len, reverse=True)
    lat_alternation = "|".join(re.escape(t) for t in lat_terms)

    loc_terms = list(LOCATION_MODIFIERS.keys())
    loc_terms.sort(key=len, reverse=True)
    loc_alternation = "|".join(re.escape(t) for t in loc_terms)

    pattern = (
        rf"(?P<modifier>(?:{lat_alternation}|{loc_alternation}))\s+(?P<finding>{finding_alternation})"
        rf"|(?P<finding_only>{finding_alternation})"
    )

    return re.compile(pattern, re.IGNORECASE)


# --- Multi-Finding Extraction ---

def extract_findings(sentence, pattern=None):
    if pattern is None:
        pattern = build_finding_pattern()

    results = []

    for match in pattern.finditer(sentence):
        raw_text = match.group(0).strip()

        modifier_text = match.group("modifier") if match.group("modifier") else None
        finding_text = match.group("finding") if match.group("finding") else match.group("finding_only")

        canonical = normalize_finding(finding_text)
        if canonical is None:
            continue

        laterality = None
        location = None

        if modifier_text:
            lat_result = normalize_laterality(modifier_text)
            loc_result = normalize_location(modifier_text)
            if lat_result:
                laterality = lat_result
            elif loc_result:
                location = loc_result

        results.append({
            "finding": canonical,
            "raw_text": raw_text,
            "laterality": laterality,
            "location": location,
        })

    return results


# --- Full Pipeline ---

def extract_respiratory_findings(clinical_text):
    """Run the full respiratory finding extraction pipeline.

    Pipeline stages:
    1. Sentence splitting
    2. Regex matching per sentence
    3. Vocabulary normalization
    4. Structured output assembly

    Args:
        clinical_text: Raw clinical note text (one or more sentences).

    Returns:
        List of structured finding dictionaries.
    """
    nlp = create_sentence_splitter()
    pattern = build_finding_pattern()
    sentences = split_sentences(clinical_text, nlp)

    all_findings = []

    for idx, sentence in enumerate(sentences):
        findings = extract_findings(sentence, pattern)
        for finding in findings:
            finding["sentence_index"] = idx
            finding["source_sentence"] = sentence
            all_findings.append(finding)

    return all_findings


# --- Main Entry Point ---

if __name__ == "__main__":
    # Example clinical text
    clinical_note = "Bilateral wheezing and bibasilar crackles noted. No rhonchi. Diminished breath sounds at left base."

    print("=" * 60)
    print("RESPIRATORY FINDING EXTRACTOR")
    print("=" * 60)
    print(f"\nInput: {clinical_note}\n")

    # Run extraction
    findings = extract_respiratory_findings(clinical_note)

    # Display results
    print("Extracted Findings:")
    print("-" * 40)
    print(json.dumps(findings, indent=2))

    print(f"\nTotal findings extracted: {len(findings)}")

Your pipeline now transforms raw clinical notes into structured, queryable data. Next up, you will handle edge cases and write a proper test suite to make sure the extractor is robust.

Handle Edge Cases and Write Tests

Your extraction pipeline is working, but clinical text is messy. Doctors write in uppercase, use abbreviations, and mix modifiers with multiple findings in a single phrase.

In this step, you will verify that your extractor handles these edge cases correctly and then build a test suite that proves it. A passing test suite is the difference between a demo script and a reliable tool.

In this step, get ready to:
  • Verify edge case handling for case sensitivity, partial matches, and modifier scope.
  • Confirm vocabulary coverage for abbreviations and variant forms.
  • Write a test suite with 8 test functions covering vocabulary, pipeline, and edge cases.
Verify edge case handling in your extractor

Clinical notes come in all shapes. A doctor might type "WHEEZING" in caps, write "bilateral wheezing and crackles" where the modifier scope is ambiguous, or use shorthand your regex could trip over. Your extractor already handles the most common issues thanks to design decisions baked into the previous steps.

  • Verify that case sensitivity is handled by running this command in your terminal:
python -c "from respiratory_extractor import extract_findings, build_finding_pattern; p = build_finding_pattern(); print(extract_findings('BILATERAL WHEEZING noted.', p))"

You should see a result with "finding": "wheezing" and "laterality": "bilateral" despite the input being fully uppercase.

How does case insensitivity work here?

The re.IGNORECASE flag in build_finding_pattern() makes the regex match regardless of capitalization. The normalize_finding() function also calls .lower() before looking up the vocabulary. Together, these ensure "WHEEZING", "Wheezing", and "wheezing" all resolve to the same canonical term.

Two other edge cases are worth understanding before you write tests.

What about partial matches and modifier scope?

Partial matches occur when a finding term appears inside a negation phrase like "no wheezing." Your extractor currently matches it as a positive finding. Negation detection is a separate concern that you will tackle in the Secret Mission.

Modifier scope is an inherent ambiguity in clinical text. In "bilateral wheezing and crackles," does "bilateral" apply to both findings or just "wheezing"? Your regex attaches the modifier only to the immediately following finding. This is the safer default because over-applying modifiers introduces false information.

Confirm vocabulary coverage for abbreviations and variants

Clinical abbreviations and plural/singular variations are handled by your vocabulary dictionaries. The FINDING_SYNONYMS dictionary maps both "crackle" (singular) and "crackles" (plural) to the same canonical term.

  • Verify abbreviation handling by running this command:
python -c "from vocabulary import normalize_finding; print(normalize_finding('dim bs'))"

You should see diminished breath sounds printed. The abbreviation "dim bs" resolves to the full canonical term.

  • Test a compound location modifier by running:
python -c "from vocabulary import normalize_location; print(normalize_location('right lower lobe'))"

You should see right lower lobe printed. Your LOCATION_MODIFIERS dictionary already includes compound modifiers as complete entries.

Why list compound modifiers explicitly?

Compound modifiers like "right lower lobe" cannot be reliably assembled from individual tokens ("right" + "lower" + "lobe") without a parser. Listing them as whole phrases in the dictionary avoids ambiguity and keeps the extraction logic simple.

Write your test suite

You have verified edge cases interactively. Now you will codify these checks into a repeatable test file so you can re-run them whenever you change the extractor.

  • Open test_extractor.py in VS Code. Add the imports and vocabulary test functions by pasting this code:
"""Test suite for the respiratory finding extractor."""

import json
from respiratory_extractor import extract_respiratory_findings, split_sentences, extract_findings, build_finding_pattern
from vocabulary import normalize_finding, normalize_laterality


def test_normalize_finding():
    """Test vocabulary normalization."""
    assert normalize_finding("wheeze") == "wheezing"
    assert normalize_finding("Wheezing") == "wheezing"
    assert normalize_finding("rales") == "crackles"
    assert normalize_finding("crepitations") == "crackles"
    assert normalize_finding("unknown_term") is None
    print("[PASS] test_normalize_finding")


def test_normalize_laterality():
    """Test laterality normalization."""
    assert normalize_laterality("bilateral") == "bilateral"
    assert normalize_laterality("b/l") == "bilateral"
    assert normalize_laterality("left") == "left"
    assert normalize_laterality("rt") == "right"
    print("[PASS] test_normalize_laterality")

What do these tests cover?

These two functions test the vocabulary layer in isolation. test_normalize_finding() verifies that synonyms resolve to canonical terms, mixed case is handled, and unknown terms return None.

test_normalize_laterality() confirms that abbreviations like "b/l" and "rt" map to their full forms.

  • Add the pipeline test functions below the vocabulary tests:
def test_sentence_splitting():
    """Test clinical sentence splitting."""
    text = "Bilateral wheezing and bibasilar crackles noted. Lungs clear to auscultation bilaterally."
    sentences = split_sentences(text)
    assert len(sentences) == 2
    assert "wheezing" in sentences[0]
    assert "Lungs clear" in sentences[1]
    print("[PASS] test_sentence_splitting")


def test_single_finding_extraction():
    """Test extracting a single finding."""
    pattern = build_finding_pattern()
    results = extract_findings("Wheezing noted on exam.", pattern)
    assert len(results) == 1
    assert results[0]["finding"] == "wheezing"
    assert results[0]["laterality"] is None
    print("[PASS] test_single_finding_extraction")


def test_multi_finding_extraction():
    """Test extracting multiple findings from one sentence."""
    pattern = build_finding_pattern()
    results = extract_findings("Bilateral wheezing and bibasilar crackles noted.", pattern)
    assert len(results) == 2
    assert results[0]["finding"] == "wheezing"
    assert results[0]["laterality"] == "bilateral"
    assert results[1]["finding"] == "crackles"
    assert results[1]["location"] == "bibasilar"
    print("[PASS] test_multi_finding_extraction")


def test_full_pipeline():
    """Test the full extraction pipeline on a multi-sentence note."""
    clinical_note = "Bilateral wheezing and bibasilar crackles noted. Diminished breath sounds at left base."
    results = extract_respiratory_findings(clinical_note)
    assert len(results) >= 3
    findings_set = {r["finding"] for r in results}
    assert "wheezing" in findings_set
    assert "crackles" in findings_set
    assert "diminished breath sounds" in findings_set
    print("[PASS] test_full_pipeline")

What do these tests cover?

test_sentence_splitting() confirms the spaCy sentencizer correctly separates two clinical sentences. test_single_finding_extraction() verifies the simplest case: one finding, no modifiers.

test_multi_finding_extraction() tests the coordinated phrase pattern where two findings share one sentence. test_full_pipeline() runs the entire chain end-to-end across multiple sentences and checks all expected findings appear.

  • Add the edge case tests and the main runner block at the bottom of the file:
def test_case_insensitivity():
    """Test that extraction is case-insensitive."""
    pattern = build_finding_pattern()
    results = extract_findings("BILATERAL WHEEZING noted.", pattern)
    assert len(results) == 1
    assert results[0]["finding"] == "wheezing"
    assert results[0]["laterality"] == "bilateral"
    print("[PASS] test_case_insensitivity")


def test_abbreviation_handling():
    """Test abbreviation in vocabulary."""
    result = normalize_finding("dim bs")
    assert result == "diminished breath sounds"
    print("[PASS] test_abbreviation_handling")


if __name__ == "__main__":
    print("Running Respiratory Extractor Tests")
    print("=" * 40)
    test_normalize_finding()
    test_normalize_laterality()
    test_sentence_splitting()
    test_single_finding_extraction()
    test_multi_finding_extraction()
    test_full_pipeline()
    test_case_insensitivity()
    test_abbreviation_handling()
    print("=" * 40)
    print("All tests passed!")

What do these tests cover?

test_case_insensitivity() exercises the re.IGNORECASE flag by feeding fully uppercase input. test_abbreviation_handling() proves that clinical shorthand like "dim bs" resolves to the full canonical form.

The __main__ block runs all 8 tests in sequence. If any assertion fails, Python raises an AssertionError and stops execution at that point.

  • Save test_extractor.py.
  • Run the test suite by running this command:
python test_extractor.py

You should see 8 [PASS] lines followed by "All tests passed!" at the bottom.

Seeing an AssertionError?

The error message tells you which assertion failed. Check the line number and compare the expected value with what your code returns.

If test_normalize_finding fails, make sure vocabulary.py includes all variants in the FINDING_SYNONYMS dictionary. If extraction tests fail, confirm that respiratory_extractor.py has the re.IGNORECASE flag in build_finding_pattern().

Why are my tests failing with an AssertionError?

✔️ Awesome, I've got everything!

Great. Double check you have saved test_extractor.py and all 8 tests pass.

ⓧ I'd like to double check the full code

"""Test suite for the respiratory finding extractor."""

import json
from respiratory_extractor import extract_respiratory_findings, split_sentences, extract_findings, build_finding_pattern
from vocabulary import normalize_finding, normalize_laterality


def test_normalize_finding():
    """Test vocabulary normalization."""
    assert normalize_finding("wheeze") == "wheezing"
    assert normalize_finding("Wheezing") == "wheezing"
    assert normalize_finding("rales") == "crackles"
    assert normalize_finding("crepitations") == "crackles"
    assert normalize_finding("unknown_term") is None
    print("[PASS] test_normalize_finding")


def test_normalize_laterality():
    """Test laterality normalization."""
    assert normalize_laterality("bilateral") == "bilateral"
    assert normalize_laterality("b/l") == "bilateral"
    assert normalize_laterality("left") == "left"
    assert normalize_laterality("rt") == "right"
    print("[PASS] test_normalize_laterality")


def test_sentence_splitting():
    """Test clinical sentence splitting."""
    text = "Bilateral wheezing and bibasilar crackles noted. Lungs clear to auscultation bilaterally."
    sentences = split_sentences(text)
    assert len(sentences) == 2
    assert "wheezing" in sentences[0]
    assert "Lungs clear" in sentences[1]
    print("[PASS] test_sentence_splitting")


def test_single_finding_extraction():
    """Test extracting a single finding."""
    pattern = build_finding_pattern()
    results = extract_findings("Wheezing noted on exam.", pattern)
    assert len(results) == 1
    assert results[0]["finding"] == "wheezing"
    assert results[0]["laterality"] is None
    print("[PASS] test_single_finding_extraction")


def test_multi_finding_extraction():
    """Test extracting multiple findings from one sentence."""
    pattern = build_finding_pattern()
    results = extract_findings("Bilateral wheezing and bibasilar crackles noted.", pattern)
    assert len(results) == 2
    assert results[0]["finding"] == "wheezing"
    assert results[0]["laterality"] == "bilateral"
    assert results[1]["finding"] == "crackles"
    assert results[1]["location"] == "bibasilar"
    print("[PASS] test_multi_finding_extraction")


def test_full_pipeline():
    """Test the full extraction pipeline on a multi-sentence note."""
    clinical_note = "Bilateral wheezing and bibasilar crackles noted. Diminished breath sounds at left base."
    results = extract_respiratory_findings(clinical_note)
    assert len(results) >= 3
    findings_set = {r["finding"] for r in results}
    assert "wheezing" in findings_set
    assert "crackles" in findings_set
    assert "diminished breath sounds" in findings_set
    print("[PASS] test_full_pipeline")


def test_case_insensitivity():
    """Test that extraction is case-insensitive."""
    pattern = build_finding_pattern()
    results = extract_findings("BILATERAL WHEEZING noted.", pattern)
    assert len(results) == 1
    assert results[0]["finding"] == "wheezing"
    assert results[0]["laterality"] == "bilateral"
    print("[PASS] test_case_insensitivity")


def test_abbreviation_handling():
    """Test abbreviation in vocabulary."""
    result = normalize_finding("dim bs")
    assert result == "diminished breath sounds"
    print("[PASS] test_abbreviation_handling")


if __name__ == "__main__":
    print("Running Respiratory Extractor Tests")
    print("=" * 40)
    test_normalize_finding()
    test_normalize_laterality()
    test_sentence_splitting()
    test_single_finding_extraction()
    test_multi_finding_extraction()
    test_full_pipeline()
    test_case_insensitivity()
    test_abbreviation_handling()
    print("=" * 40)
    print("All tests passed!")

Your extractor now has a verified test suite covering vocabulary normalization, sentence splitting, multi-finding extraction, and edge cases. Next up, you will take on the Secret Mission and add negation detection so phrases like "no wheezing" are correctly flagged.

Secret mission

Add Negation Detection

Your extractor currently treats every finding as affirmed. But clinical notes are full of negated statements like "No wheezing" and "Denies crackles." Can you build a negation-aware layer that distinguishes what the patient HAS from what the patient DOES NOT have?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • respiratory-extractor project folder on your Desktop (contains vocabulary.py, respiratory_extractor.py, test_extractor.py).
  • venv virtual environment (inside the project folder).
  • spaCy 3.8.14 package (installed inside the virtual environment).

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend the extractor with additional clinical finding categories.

  • Your project folder, virtual environment, and all installed packages remain on your Desktop.
  • To resume work later, open a terminal in the respiratory-extractor folder and reactivate the virtual environment.

Pause - I'll come back to this later

Deactivate your virtual environment to free up your terminal session. Your files stay intact on disk.

  • Deactivate the virtual environment by running:
deactivate
  • When you want to resume, navigate back to respiratory-extractor and reactivate with source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows).

Delete - I don't want to use this again

Remove all project files and the virtual environment. This deletes everything you built in this project.

  • First, deactivate your virtual environment if it is still active:
deactivate
  • Delete the entire project folder. On macOS/Linux, run:
rm -rf ~/Desktop/respiratory-extractor
  • On Windows, run:
rmdir /s /q %USERPROFILE%\Desktop\respiratory-extractor
  • This removes all three Python files, the venv folder (which contains spaCy and all installed packages), and the project folder itself.

Nice Work!

Nice Work!

Congratulations! You just built a clinical NLP pipeline from scratch that extracts structured respiratory findings from unstructured clinical text.

You've learned how to:

  • Build a vocabulary-based extraction system with regex pattern matching that maps clinical synonyms to canonical medical terms and handles laterality and location modifiers.
  • Configure a sentence splitter for clinical text that handles abbreviations like pt. and Dr. without triggering false sentence boundaries.
  • Assemble a multi-finding extraction pipeline that produces structured JSON output with edge case handling for case sensitivity, partial matches, and modifier scope.
  • Add negation detection using a windowed cue-word approach, which is the foundation of NegEx-style algorithms used in production clinical NLP systems.

Ready to quiz yourself?