Clinical Negation Detection with NLP

Detect negated clinical findings in medical text using medspaCy.

Introduction

30 Second Summary

When a doctor writes "no evidence of pneumonia" in a patient's chart, you instantly understand that the patient does not have pneumonia. Computers, however, often just see the word "pneumonia" and flag it as a diagnosis. This single misread can corrupt research data, trigger wrong alerts, and affect patient care.

In this project, you will build a clinical negation detection system in Python that correctly identifies which medical findings are present versus absent in doctor's notes.

What You'll Build

A Python pipeline that processes sample clinical notes and uses the ConText algorithm (via medspaCy) to determine whether mentioned conditions are actually present or negated.

By the end of this project, you'll have:

  • A naive regex-based negation detector that demonstrates three common failure modes of simple pattern matching.
  • A medspaCy pipeline with ConText that correctly handles forward, backward, and conjunction negation.
  • Custom ConTextRules demonstrating negation scope, termination terms, and pseudo-negation handling.
  • Secret Mission: Build a reusable Negation Audit Report function that processes a batch of clinical sentences and outputs a formatted summary of each entity's negation status.

Are there any prerequisites?

You only need basic Python familiarity (variables, functions, loops). No NLP or medical background is required.

See Why Regex Alone Fails

Before reaching for a powerful NLP tool, it helps to see exactly why a simpler approach breaks down. Medical text has patterns that look straightforward on the surface but hide real complexity underneath.

In this step, you will build a naive regex-based negation detector and watch it fail on real clinical sentence patterns. These failures will motivate everything you build with medspaCy in the following steps.

In this step, get ready to:
  • Write a naive regex-based negation detector.
  • Identify the three failure modes that regex cannot handle.
  • Build a comparison table that proves where regex gets it wrong.
Write a naive regex negation detector

The simplest approach to negation detection is searching for words like "no" or "not" near a medical term. If the negation word appears within a short window before the term, mark it as negated.

  • In negation_detection.py, find your existing imports at the top of the file. Copy the following function below them:
def naive_negation_check(text, term):
    """
    A simple regex approach: check if 'no', 'not', or 'denies'
    appears within 5 words before the term.
    Returns True if negated, False otherwise.
    """
    pattern = r'\b(no|not|denies|absent)\b.{0,30}\b' + re.escape(term) + r'\b'
    match = re.search(pattern, text.lower())
    return match is not None

What does this code do?

This function takes a sentence and a medical term, then checks whether a negation word appears within 30 characters before that term.

  • r'\b(no|not|denies|absent)\b' matches common negation cue words as whole words.
  • .{0,30} allows up to 30 characters of anything between the cue and the term (roughly 5 words).
  • re.escape(term) safely handles any special characters in the medical term.
  • Save the file. You will not see output yet because the function has not been called.
Identify the three failure modes

The regex approach looks reasonable, but clinical language has three patterns that break it completely.

What are the three failure modes?

  • Backward negation is where the negation cue follows the finding. In "Crackles absent.", the word "absent" comes after "crackles." A regex looking only before the term misses this entirely.
  • Pseudo-negation is where a phrase looks like negation but is not. In "No increase in wheezing", the word "no" modifies "increase", not "wheezing." The patient still has wheezing. A regex sees "no" before "wheezing" and incorrectly flags it as negated.
  • Conjunction scope is where a single negation cue negates multiple findings joined by "or" or "and." In "No wheezing or crackles", both findings are negated by the single "No." A regex that only looks within a fixed window before each term may miss the second finding if it falls outside the window.

These are not edge cases. They appear constantly in real clinical documentation. Let's prove it by running the detector on test sentences.

Build a comparison table

Now you will define test sentences, specify the correct answers, and print a table that shows where regex gets it right and where it fails.

  • In negation_detection.py, copy the following code below the naive_negation_check function:
# Test sentences covering all three failure modes
test_sentences = [
    "No wheezing.",
    "Crackles absent.",
    "No wheezing or crackles.",
    "No increase in wheezing.",
    "No evidence of CHF but there is pneumonia.",
]

# Map each sentence to the terms we want to check
terms_to_check = {
    "No wheezing.": ["wheezing"],
    "Crackles absent.": ["crackles"],
    "No wheezing or crackles.": ["wheezing", "crackles"],
    "No increase in wheezing.": ["wheezing"],
    "No evidence of CHF but there is pneumonia.": ["chf", "pneumonia"],
}

print("=" * 60)
print("NAIVE REGEX NEGATION RESULTS")
print("=" * 60)
print(f"{'Sentence':<45} {'Term':<12} {'Regex Says':<12} {'Correct'}")
print("-" * 60)

# The correct answers for each (sentence, term) pair
expected = {
    ("No wheezing.", "wheezing"): True,
    ("Crackles absent.", "crackles"): True,
    ("No wheezing or crackles.", "wheezing"): True,
    ("No wheezing or crackles.", "crackles"): True,
    ("No increase in wheezing.", "wheezing"): False,
    ("No evidence of CHF but there is pneumonia.", "chf"): True,
    ("No evidence of CHF but there is pneumonia.", "pneumonia"): False,
}

for sentence in test_sentences:
    for term in terms_to_check[sentence]:
        result = naive_negation_check(sentence, term)
        correct = expected[(sentence, term)]
        status = "CORRECT" if result == correct else "WRONG"
        print(f"{sentence:<45} {term:<12} {str(result):<12} {correct} ({status})")

What does this code do?

  • test_sentences holds five clinical sentences, each designed to test a different negation pattern.
  • terms_to_check maps each sentence to the medical terms we want to check for negation.
  • expected contains the clinically correct answers. True means the term is actually negated in the sentence. False means the term is affirmed (present).
  • The loop runs each sentence through naive_negation_check and compares the result to the correct answer, printing CORRECT or WRONG.
  • Save negation_detection.py and run the script.
  • Run your script by executing:
python negation_detection.py
  • You should see a printed table. Look for rows labelled WRONG. You should see at least two WRONG results: one for "No increase in wheezing" (pseudo-negation) and one for "No evidence of CHF but there is pneumonia" where regex incorrectly negates "pneumonia."

Not seeing any output?

  • Make sure you saved the file before running it.
  • Check that the import re line is still at the top of your file from Step 1.
  • If you see a NameError, make sure the function definition appears above the code that calls it.

Why isn't my negation script printing any output?

Why does this matter for patient safety?

Approximately half of all conditions mentioned in clinical reports are negated. If an NLP system indexes "pneumonia" from the sentence "There is no evidence of pneumonia," it creates a dangerous false positive.

That false positive could affect patient care, research cohorts, and clinical decision support systems. Getting negation wrong is not just a technical issue. It is a safety issue.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

import medspacy
from medspacy.ner import TargetRule
from medspacy.context import ConTextRule
import re

def naive_negation_check(text, term):
    """
    A simple regex approach: check if 'no', 'not', or 'denies'
    appears within 5 words before the term.
    Returns True if negated, False otherwise.
    """
    pattern = r'\b(no|not|denies|absent)\b.{0,30}\b' + re.escape(term) + r'\b'
    match = re.search(pattern, text.lower())
    return match is not None


test_sentences = [
    "No wheezing.",
    "Crackles absent.",
    "No wheezing or crackles.",
    "No increase in wheezing.",
    "No evidence of CHF but there is pneumonia.",
]

terms_to_check = {
    "No wheezing.": ["wheezing"],
    "Crackles absent.": ["crackles"],
    "No wheezing or crackles.": ["wheezing", "crackles"],
    "No increase in wheezing.": ["wheezing"],
    "No evidence of CHF but there is pneumonia.": ["chf", "pneumonia"],
}

print("=" * 60)
print("NAIVE REGEX NEGATION RESULTS")
print("=" * 60)
print(f"{'Sentence':<45} {'Term':<12} {'Regex Says':<12} {'Correct'}")
print("-" * 60)

expected = {
    ("No wheezing.", "wheezing"): True,
    ("Crackles absent.", "crackles"): True,
    ("No wheezing or crackles.", "wheezing"): True,
    ("No wheezing or crackles.", "crackles"): True,
    ("No increase in wheezing.", "wheezing"): False,
    ("No evidence of CHF but there is pneumonia.", "chf"): True,
    ("No evidence of CHF but there is pneumonia.", "pneumonia"): False,
}

for sentence in test_sentences:
    for term in terms_to_check[sentence]:
        result = naive_negation_check(sentence, term)
        correct = expected[(sentence, term)]
        status = "CORRECT" if result == correct else "WRONG"
        print(f"{sentence:<45} {term:<12} {str(result):<12} {correct} ({status})")

Your comparison table now clearly shows that regex gets several cases wrong. In the next step, you will use medspaCy's ConText algorithm to handle all three failure modes correctly.

Understand Forward, Backward, and Conjunction Negation

You now have a clear picture of where regex-based negation falls apart. The next question is: how does a purpose-built algorithm handle these same sentences correctly?

The ConText algorithm answers this by assigning a direction to every negation cue. "No" scopes forward (to the right). "Absent" scopes backward (to the left). In this step, you will load medspaCy's pipeline, teach it which clinical terms to look for, and watch it correctly negate all three patterns that tripped up regex.

In this step, get ready to:
  • Load the medspaCy pipeline and register clinical target rules.
  • Process a forward-negation sentence and confirm the entity is negated.
  • Process backward-negation and conjunction-negation sentences and confirm correct results.
Define target rules for clinical findings

Before medspaCy can detect negation, it needs to know which clinical terms to look for in the text. You define these using TargetRule objects that pair a literal phrase with a category label.

  • Add a new cell below your existing code and load the medspaCy pipeline with target rules by pasting this code:
print("\n\n" + "=" * 60)
print("MEDSPACY CONTEXT NEGATION DETECTION")
print("=" * 60)

# Load the default medspaCy pipeline (includes sentence splitter + ConText)
nlp = medspacy.load()
print(f"Pipeline components: {nlp.pipe_names}")

# Retrieve the target matcher component from the pipeline
target_matcher = nlp.get_pipe("medspacy_target_matcher")

# Define which clinical terms ConText should look for
target_rules = [
    TargetRule("wheezing", "PROBLEM"),
    TargetRule("crackles", "PROBLEM"),
    TargetRule("pneumonia", "PROBLEM"),
    TargetRule("chest pain", "PROBLEM"),
    TargetRule("CHF", "PROBLEM"),
    TargetRule("fever", "PROBLEM"),
    TargetRule("shortness of breath", "PROBLEM"),
    TargetRule("cough", "PROBLEM"),
    TargetRule("bronchitis", "PROBLEM"),
]
target_matcher.add(target_rules)

What does this code do?

medspacy.load() creates a full NLP pipeline that includes a sentence splitter and the ConText algorithm. It is ready to detect negation out of the box.

nlp.get_pipe("medspacy_target_matcher") retrieves the entity matching component from the pipeline. Each TargetRule tells it to recognise a specific clinical phrase and label it with the category PROBLEM. You are registering all the terms that will appear in the test sentences.

  • Run the cell.
  • You should see output showing the pipeline components list, which includes medspacy_target_matcher and medspacy_context.

Seeing a ModuleNotFoundError?

Make sure you ran the pip install medspacy cell from Step 1 first. The kernel must have medspaCy installed before you can import it.

help me fix a ModuleNotFoundError when loading medspaCy

Explore forward negation

Forward negation is the most common pattern in clinical notes. The trigger term (like "No") appears before the finding, and its scope extends to the right until a termination term or the end of the sentence.

  • Process the sentence "No wheezing." through the pipeline by adding this code in a new cell:
print("\n--- Forward Negation: 'No wheezing.' ---")
doc = nlp("No wheezing.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

What does this code do?

Calling nlp("No wheezing.") runs the full pipeline on the sentence: it splits sentences, identifies the entity "wheezing" using your target rules, then runs ConText to check for negation modifiers.

ent._.is_negated is the attribute ConText sets to True when a negation modifier's scope covers the entity. In this case, "No" is a FORWARD trigger whose scope extends to the right and covers "wheezing".

  • Run the cell.
  • You should see Entity: 'wheezing' | Negated: True in the output. ConText correctly identified that "No" negates "wheezing" using forward scope.

Seeing Negated: False instead of True?

Confirm that you ran the target rules cell before this one. If the pipeline does not have target rules registered, it cannot find the entity "wheezing" in the text.

help me debug why my entity is not showing as negated

Explore backward and conjunction negation

Backward negation flips the direction. The trigger term ("absent") appears after the finding, and its scope extends to the left. Conjunction negation is when a single forward trigger like "No" negates multiple findings connected by "or" because "or" is not a termination term.

  • Process a backward-negation sentence by adding this code in a new cell:
print("\n--- Backward Negation: 'Crackles absent.' ---")
doc = nlp("Crackles absent.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

How does backward negation work?

The trigger word "absent" has direction BACKWARD in the default ConText rules. Its scope extends to the left from its position, so it reaches "crackles" even though "crackles" appears before the trigger.

This is exactly the pattern that broke the naive regex in Step 2. Regex only looked for negation cues before the term, never after it.

  • Run the cell.
  • You should see Entity: 'crackles' | Negated: True. The backward-scoping trigger "absent" correctly reached "crackles" to its left.
  • Now process a conjunction-negation sentence by adding this code in a new cell:
print("\n--- Conjunction Negation: 'No wheezing or crackles.' ---")
doc = nlp("No wheezing or crackles.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

Why are both entities negated?

"No" is a FORWARD trigger. Its scope extends to the right until it hits a termination term or the end of the sentence. The word "or" is not a termination term in ConText's default rules, so the scope continues past it and covers both "wheezing" and "crackles".

This is correct clinical interpretation. A doctor writing "No wheezing or crackles" means neither symptom is present.

  • Run the cell.
  • You should see both entities listed with Negated: True. A single "No" negated two findings in one sentence.

Only seeing one entity instead of two?

Check that you added both TargetRule("wheezing", "PROBLEM") and TargetRule("crackles", "PROBLEM") to your target rules list. Without both rules registered, the pipeline cannot find both terms as entities.

help me debug missing entities in medspaCy

✔️ Awesome, I've got everything!

Great. Double check you have all three tests running and showing the correct negation results before moving on.

ⓧ I'd like to double check the full code

Here is the complete code you should have in your notebook so far:

import medspacy
from medspacy.ner import TargetRule
from medspacy.context import ConTextRule
import re

def naive_negation_check(text, term):
    pattern = r'\b(no|not|denies|absent)\b.{0,30}\b' + re.escape(term) + r'\b'
    match = re.search(pattern, text.lower())
    return match is not None

test_sentences = [
    "No wheezing.",
    "Crackles absent.",
    "No wheezing or crackles.",
    "No increase in wheezing.",
    "No evidence of CHF but there is pneumonia.",
]

terms_to_check = {
    "No wheezing.": ["wheezing"],
    "Crackles absent.": ["crackles"],
    "No wheezing or crackles.": ["wheezing", "crackles"],
    "No increase in wheezing.": ["wheezing"],
    "No evidence of CHF but there is pneumonia.": ["chf", "pneumonia"],
}

print("=" * 60)
print("NAIVE REGEX NEGATION RESULTS")
print("=" * 60)
print(f"{'Sentence':<45} {'Term':<12} {'Regex Says':<12} {'Correct'}")
print("-" * 60)

expected = {
    ("No wheezing.", "wheezing"): True,
    ("Crackles absent.", "crackles"): True,
    ("No wheezing or crackles.", "wheezing"): True,
    ("No wheezing or crackles.", "crackles"): True,
    ("No increase in wheezing.", "wheezing"): False,
    ("No evidence of CHF but there is pneumonia.", "chf"): True,
    ("No evidence of CHF but there is pneumonia.", "pneumonia"): False,
}

for sentence in test_sentences:
    for term in terms_to_check[sentence]:
        result = naive_negation_check(sentence, term)
        correct = expected[(sentence, term)]
        status = "CORRECT" if result == correct else "WRONG"
        print(f"{sentence:<45} {term:<12} {str(result):<12} {correct} ({status})")

print("\n\n" + "=" * 60)
print("MEDSPACY CONTEXT NEGATION DETECTION")
print("=" * 60)

nlp = medspacy.load()
print(f"Pipeline components: {nlp.pipe_names}")

target_matcher = nlp.get_pipe("medspacy_target_matcher")
target_rules = [
    TargetRule("wheezing", "PROBLEM"),
    TargetRule("crackles", "PROBLEM"),
    TargetRule("pneumonia", "PROBLEM"),
    TargetRule("chest pain", "PROBLEM"),
    TargetRule("CHF", "PROBLEM"),
    TargetRule("fever", "PROBLEM"),
    TargetRule("shortness of breath", "PROBLEM"),
    TargetRule("cough", "PROBLEM"),
    TargetRule("bronchitis", "PROBLEM"),
]
target_matcher.add(target_rules)

print("\n--- Forward Negation: 'No wheezing.' ---")
doc = nlp("No wheezing.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Backward Negation: 'Crackles absent.' ---")
doc = nlp("Crackles absent.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Conjunction Negation: 'No wheezing or crackles.' ---")
doc = nlp("No wheezing or crackles.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

You now have proof that ConText handles all three negation directions your regex could not. Next up, you will explore how negation scope works in longer sentences and learn how termination terms like "but" stop a negation cue from reaching too far.

Master Negation Scope and Termination

You now know that negation can flow forward, backward, or across conjunctions. The next question is: how far does a negation cue reach?

In clinical notes, a single sentence often contains both negated and affirmed findings. The ConText algorithm uses termination terms and pseudo-negation rules to precisely control where negation stops. In this step, you will see these boundaries in action.

In this step, get ready to:
  • Observe how termination terms like "but" stop negation scope from reaching affirmed findings.
  • See how pseudo-negation prevents false triggers on phrases like "no increase in".
  • Create custom ConTextRules for FORWARD, BACKWARD, and TERMINATE directions.
Process a sentence with scope termination

Consider the sentence "No evidence of CHF but there is pneumonia." A naive forward-scoped negation would negate everything after "No" until the sentence ends. That would incorrectly negate pneumonia too.

ConText solves this with termination terms. The word "but" acts as a boundary that stops the forward scope of "No evidence of" from reaching "pneumonia."

  • Add this code to the bottom of your notebook cell to test scope termination:
print("\n\n" + "=" * 60)
print("NEGATION SCOPE AND TERMINATION")
print("=" * 60)

print("\n--- Termination: 'No evidence of CHF but there is pneumonia.' ---")
doc = nlp("No evidence of CHF but there is pneumonia.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

What does this code do?

This processes a sentence where one condition (CHF) is negated and another (pneumonia) is affirmed in the same sentence.

The word "but" is a default termination term in medspaCy's rule set. When ConText's forward scope hits "but", it stops. Everything before "but" falls within the negation scope, but "pneumonia" after it does not.

  • Run the cell.
  • You should see CHF marked as True for negated and pneumonia marked as False.

Both showing as negated?

Make sure your sentence matches exactly, including the word "but" between the two conditions. If you have a typo like "and" instead of "but", there is no termination term and both entities will be negated.

help me understand why both CHF and pneumonia are showing as negated in my termination example

See how pseudo-negation prevents false triggers

Not every phrase containing "no" is actually negating something. The phrase "no increase in wheezing" does not mean wheezing is absent. It means the wheezing is still there but has not gotten worse.

ConText handles this with the PSEUDO direction. A PSEUDO rule matches a phrase and blocks it from being treated as a negation trigger.

  • Add this code below the termination example to test pseudo-negation:
print("\n--- Pseudo-negation: 'No increase in wheezing.' ---")
doc = nlp("No increase in wheezing.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

What does this code do?

This tests a sentence where "no" appears but should not negate "wheezing". The phrase "no increase in" is a default PSEUDO rule in medspaCy.

PSEUDO rules exist specifically to block false triggers. They match a longer phrase ("no increase in") before the shorter trigger ("no") can fire, preventing incorrect negation.

  • Run the cell.
  • You should see wheezing marked as False for negated. The wheezing is present in the patient.

Why does this matter for patient safety?

If a system incorrectly marked wheezing as negated here, it would record that the patient has no wheezing. In reality, the patient still has wheezing. It just has not increased. This is exactly the kind of subtle error that leads to incorrect clinical records.

Create custom ConTextRules for each direction

medspaCy comes with hundreds of default rules, but you can also create your own. This is useful when your clinical notes use terminology that the defaults do not cover.

  • Add this code to create three custom rules (FORWARD, BACKWARD, and TERMINATE) and test them on new sentences:
print("\n--- Custom Rules Demo ---")

context = nlp.get_pipe("medspacy_context")
custom_rules = [
    ConTextRule("denies", "NEGATED_EXISTENCE", direction="FORWARD"),
    ConTextRule("is ruled out", "NEGATED_EXISTENCE", direction="BACKWARD"),
    ConTextRule("however", "TERMINATE", direction="TERMINATE"),
]
context.add(custom_rules)

doc = nlp("Denies fever.")
for ent in doc.ents:
    print(f"  'Denies fever.' -> Entity: '{ent.text}' | Negated: {ent._.is_negated}")

doc = nlp("Pneumonia is ruled out.")
for ent in doc.ents:
    print(f"  'Pneumonia is ruled out.' -> Entity: '{ent.text}' | Negated: {ent._.is_negated}")

What does this code do?

  • nlp.get_pipe("medspacy_context") retrieves the ConText component from the pipeline so you can add rules to it.
  • ConTextRule("denies", "NEGATED_EXISTENCE", direction="FORWARD") creates a rule where "denies" negates everything after it (to the right).
  • ConTextRule("is ruled out", "NEGATED_EXISTENCE", direction="BACKWARD") creates a rule where "is ruled out" negates everything before it (to the left).
  • ConTextRule("however", "TERMINATE", direction="TERMINATE") adds "however" as a scope boundary, similar to how "but" works.
  • Run the cell.
  • You should see fever marked as negated (the FORWARD rule "denies" negates what follows it).
  • You should also see pneumonia marked as negated (the BACKWARD rule "is ruled out" negates what precedes it).

Entities not showing as negated?

Check that your ConTextRule literals match the sentence text exactly. "denies" matches "Denies" because ConText matching is case-insensitive, but extra spaces or typos will prevent a match.

Also confirm you called context.add(custom_rules) before processing the test sentences.

help me debug why my custom ConTextRules are not negating entities

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook cell.

ⓧ I'd like to double check the full code

Here is the complete code your notebook cell should contain at this point:

import medspacy
from medspacy.ner import TargetRule
from medspacy.context import ConTextRule
import re

def naive_negation_check(text, term):
    pattern = r'\b(no|not|denies|absent)\b.{0,30}\b' + re.escape(term) + r'\b'
    match = re.search(pattern, text.lower())
    return match is not None

test_sentences = [
    "No wheezing.",
    "Crackles absent.",
    "No wheezing or crackles.",
    "No increase in wheezing.",
    "No evidence of CHF but there is pneumonia.",
]

terms_to_check = {
    "No wheezing.": ["wheezing"],
    "Crackles absent.": ["crackles"],
    "No wheezing or crackles.": ["wheezing", "crackles"],
    "No increase in wheezing.": ["wheezing"],
    "No evidence of CHF but there is pneumonia.": ["chf", "pneumonia"],
}

print("=" * 60)
print("NAIVE REGEX NEGATION RESULTS")
print("=" * 60)
print(f"{'Sentence':<45} {'Term':<12} {'Regex Says':<12} {'Correct'}")
print("-" * 60)

expected = {
    ("No wheezing.", "wheezing"): True,
    ("Crackles absent.", "crackles"): True,
    ("No wheezing or crackles.", "wheezing"): True,
    ("No wheezing or crackles.", "crackles"): True,
    ("No increase in wheezing.", "wheezing"): False,
    ("No evidence of CHF but there is pneumonia.", "chf"): True,
    ("No evidence of CHF but there is pneumonia.", "pneumonia"): False,
}

for sentence in test_sentences:
    for term in terms_to_check[sentence]:
        result = naive_negation_check(sentence, term)
        correct = expected[(sentence, term)]
        status = "CORRECT" if result == correct else "WRONG"
        print(f"{sentence:<45} {term:<12} {str(result):<12} {correct} ({status})")

print("\n\n" + "=" * 60)
print("MEDSPACY CONTEXT NEGATION DETECTION")
print("=" * 60)

nlp = medspacy.load()
print(f"Pipeline components: {nlp.pipe_names}")

target_matcher = nlp.get_pipe("medspacy_target_matcher")
target_rules = [
    TargetRule("wheezing", "PROBLEM"),
    TargetRule("crackles", "PROBLEM"),
    TargetRule("pneumonia", "PROBLEM"),
    TargetRule("chest pain", "PROBLEM"),
    TargetRule("CHF", "PROBLEM"),
    TargetRule("fever", "PROBLEM"),
    TargetRule("shortness of breath", "PROBLEM"),
    TargetRule("cough", "PROBLEM"),
    TargetRule("bronchitis", "PROBLEM"),
]
target_matcher.add(target_rules)

print("\n--- Forward Negation: 'No wheezing.' ---")
doc = nlp("No wheezing.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Backward Negation: 'Crackles absent.' ---")
doc = nlp("Crackles absent.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Conjunction Negation: 'No wheezing or crackles.' ---")
doc = nlp("No wheezing or crackles.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n\n" + "=" * 60)
print("NEGATION SCOPE AND TERMINATION")
print("=" * 60)

print("\n--- Termination: 'No evidence of CHF but there is pneumonia.' ---")
doc = nlp("No evidence of CHF but there is pneumonia.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Pseudo-negation: 'No increase in wheezing.' ---")
doc = nlp("No increase in wheezing.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Custom Rules Demo ---")

context = nlp.get_pipe("medspacy_context")
custom_rules = [
    ConTextRule("denies", "NEGATED_EXISTENCE", direction="FORWARD"),
    ConTextRule("is ruled out", "NEGATED_EXISTENCE", direction="BACKWARD"),
    ConTextRule("however", "TERMINATE", direction="TERMINATE"),
]
context.add(custom_rules)

doc = nlp("Denies fever.")
for ent in doc.ents:
    print(f"  'Denies fever.' -> Entity: '{ent.text}' | Negated: {ent._.is_negated}")

doc = nlp("Pneumonia is ruled out.")
for ent in doc.ents:
    print(f"  'Pneumonia is ruled out.' -> Entity: '{ent.text}' | Negated: {ent._.is_negated}")

You now have precise control over how far negation reaches in a sentence. Next, you will combine everything into a complete pipeline that processes a full clinical note with multiple sentences and patterns.

Build Your Complete Negation Pipeline

You now understand how each negation direction works in isolation. In a real clinical note, a single paragraph mixes forward, backward, conjunction, termination, and pseudo-negation patterns all at once.

In this step, you will process a realistic multi-sentence clinical note through your pipeline and see how ConText handles all of these patterns together. You will also build a results table and compare it against the regex results from Step 2.

In this step, get ready to:
  • Process a full multi-sentence clinical note through the medspaCy pipeline.
  • Print a results table showing each entity's negation status and the modifier that triggered it.
  • Summarize the key takeaways comparing ConText to regex.
Process a full clinical note

Until now you have tested single sentences. A real clinical note contains multiple sentences, each with its own negation patterns. The ConText algorithm processes each sentence independently, respecting sentence boundaries as natural scope limits.

  • Add the following code to the bottom of your notebook cell to define a realistic clinical note and process it through the pipeline:
print("\n\n" + "=" * 60)
print("FULL CLINICAL NOTE PROCESSING")
print("=" * 60)

clinical_note = """Patient presents with cough. No wheezing or crackles. Chest pain is absent. Denies fever. No increase in shortness of breath. There is no evidence of pneumonia but patient has bronchitis."""

doc = nlp(clinical_note)

What does this code do?

The clinical_note variable holds a six-sentence note that includes every negation pattern you have studied: forward ("No wheezing"), backward ("is absent"), conjunction ("wheezing or crackles"), termination ("but"), and pseudo-negation ("No increase in").

Calling nlp(clinical_note) runs the entire pipeline: sentence splitting, entity matching, and ConText negation detection all at once.

Build the results table

For each entity, you want to know three things: what it is, whether it is negated, and which modifier triggered the negation. The ent._.modifiers attribute gives you the set of ConTextModifier objects that modified each entity.

  • Add the following code directly below the previous block to iterate through the entities and print a formatted table:
print(f"\n{'Entity':<22} {'Label':<10} {'Negated':<10} {'Modifier'}")
print("-" * 65)
for ent in doc.ents:
    if ent._.is_negated:
        modifiers = ent._.modifiers
        if modifiers:
            mod = list(modifiers)[0]
            mod_text = mod.rule.literal
        else:
            mod_text = "N/A"
    else:
        mod_text = "-"
    print(f"{ent.text:<22} {ent.label_:<10} {str(ent._.is_negated):<10} {mod_text}")

What does this code do?

  • For each entity, it checks ent._.is_negated to determine the negation status.
  • If negated, it accesses ent._.modifiers to get the ConTextModifier objects. The first modifier's .rule.literal tells you which trigger phrase caused the negation.
  • If the entity is not negated, it prints a dash instead of a modifier name.
  • Run the cell.

You should see a table where "cough" and "bronchitis" are affirmed (not negated), while "wheezing", "crackles", "chest pain", "fever", and "pneumonia" are all correctly marked as negated. "Shortness of breath" should NOT be negated because "No increase in" is a pseudo-negation.

How does this compare to the regex results?

Look back at your regex results table from Step 2. The naive regex got several results WRONG: it missed backward negation ("Crackles absent"), falsely negated pseudo-negation ("No increase in wheezing"), and could not handle termination ("No evidence of CHF but there is pneumonia").

ConText handles all of these correctly because it uses directional scope, termination rules, and pseudo-negation rules instead of a fixed-window pattern match.

Seeing unexpected negation results?

  • Make sure your target rules were added before processing the clinical note. If you restarted the runtime, you need to re-run the cells that define and add the target rules.
  • Confirm the clinical_note string is on a single line inside triple quotes. Extra line breaks can create empty sentences that confuse the sentence splitter.

Help me debug my negation results table

Print the key takeaways summary

To close out the pipeline, you will print a structured summary of everything you have learned. This serves as a quick reference for the core concepts.

  • Add the following code at the bottom of your cell to print the takeaways summary:
print("\n\n" + "=" * 60)
print("KEY TAKEAWAYS")
print("=" * 60)
print("""
1. WHY NEGATION MATTERS: ~50% of conditions in clinical reports are negated.
   Missing negation = false positives = wrong patient data.

2. FORWARD NEGATION: Cue BEFORE finding. 'No wheezing' -> 'No' negates right.

3. BACKWARD NEGATION: Cue AFTER finding. 'Crackles absent' -> 'absent' negates left.

4. CONJUNCTION NEGATION: One cue negates multiple findings connected by 'or'/'and'.
   'No wheezing or crackles' -> both negated by single 'No'.

5. NEGATION SCOPE: How far a negation cue reaches. Ended by:
   - Termination terms ('but', 'however', 'although')
   - End of sentence

6. PSEUDO-NEGATION: Phrases that LOOK like negation but aren't.
   'No increase in wheezing' -> wheezing is NOT negated.

7. WHY REGEX FAILS:
   - Cannot handle backward negation (cue after finding)
   - Cannot handle pseudo-negation (needs exceptions list)
   - Cannot handle scope termination (needs linguistic knowledge)
   - Cannot handle conjunction scope (needs entity awareness)

8. HOW CONTEXT SOLVES IT:
   - Directional rules (FORWARD, BACKWARD, BIDIRECTIONAL)
   - TERMINATE rules to limit scope
   - PSEUDO rules to block false triggers
   - Entity-aware scope within sentence boundaries
""")

What does this summary cover?

  • Points 1-6 recap the five negation directions (FORWARD, BACKWARD, BIDIRECTIONAL, TERMINATE, PSEUDO) and conjunction behavior.
  • Point 7 lists the four specific failure modes of regex-based negation that you observed in Step 2.
  • Point 8 explains how ConText's architecture directly addresses each of those failures.
  • Run the cell to see the full output from the clinical note processing and the summary.

You should see the results table followed by all eight takeaways printed cleanly in your output.

Output looks garbled or cut off?

  • If using Google Colab, make sure the entire code block is inside a single cell. Splitting across cells can cause variables to be undefined.
  • Check that the triple-quoted string in the print statement has matching opening and closing triple quotes.

Help me fix my output formatting

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

import medspacy
from medspacy.ner import TargetRule
from medspacy.context import ConTextRule
import re

def naive_negation_check(text, term):
    """
    A simple regex approach: check if 'no', 'not', or 'denies'
    appears within 5 words before the term.
    Returns True if negated, False otherwise.
    """
    pattern = r'\b(no|not|denies|absent)\b.{0,30}\b' + re.escape(term) + r'\b'
    match = re.search(pattern, text.lower())
    return match is not None


test_sentences = [
    "No wheezing.",
    "Crackles absent.",
    "No wheezing or crackles.",
    "No increase in wheezing.",
    "No evidence of CHF but there is pneumonia.",
]

terms_to_check = {
    "No wheezing.": ["wheezing"],
    "Crackles absent.": ["crackles"],
    "No wheezing or crackles.": ["wheezing", "crackles"],
    "No increase in wheezing.": ["wheezing"],
    "No evidence of CHF but there is pneumonia.": ["chf", "pneumonia"],
}

print("=" * 60)
print("NAIVE REGEX NEGATION RESULTS")
print("=" * 60)
print(f"{'Sentence':<45} {'Term':<12} {'Regex Says':<12} {'Correct'}")
print("-" * 60)

expected = {
    ("No wheezing.", "wheezing"): True,
    ("Crackles absent.", "crackles"): True,
    ("No wheezing or crackles.", "wheezing"): True,
    ("No wheezing or crackles.", "crackles"): True,
    ("No increase in wheezing.", "wheezing"): False,
    ("No evidence of CHF but there is pneumonia.", "chf"): True,
    ("No evidence of CHF but there is pneumonia.", "pneumonia"): False,
}

for sentence in test_sentences:
    for term in terms_to_check[sentence]:
        result = naive_negation_check(sentence, term)
        correct = expected[(sentence, term)]
        status = "CORRECT" if result == correct else "WRONG"
        print(f"{sentence:<45} {term:<12} {str(result):<12} {correct} ({status})")

print("\n\n" + "=" * 60)
print("MEDSPACY CONTEXT NEGATION DETECTION")
print("=" * 60)

nlp = medspacy.load()
print(f"Pipeline components: {nlp.pipe_names}")

target_matcher = nlp.get_pipe("medspacy_target_matcher")
target_rules = [
    TargetRule("wheezing", "PROBLEM"),
    TargetRule("crackles", "PROBLEM"),
    TargetRule("pneumonia", "PROBLEM"),
    TargetRule("chest pain", "PROBLEM"),
    TargetRule("CHF", "PROBLEM"),
    TargetRule("fever", "PROBLEM"),
    TargetRule("shortness of breath", "PROBLEM"),
    TargetRule("cough", "PROBLEM"),
    TargetRule("bronchitis", "PROBLEM"),
]
target_matcher.add(target_rules)

print("\n--- Forward Negation: 'No wheezing.' ---")
doc = nlp("No wheezing.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Backward Negation: 'Crackles absent.' ---")
doc = nlp("Crackles absent.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Conjunction Negation: 'No wheezing or crackles.' ---")
doc = nlp("No wheezing or crackles.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n\n" + "=" * 60)
print("NEGATION SCOPE AND TERMINATION")
print("=" * 60)

print("\n--- Termination: 'No evidence of CHF but there is pneumonia.' ---")
doc = nlp("No evidence of CHF but there is pneumonia.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Pseudo-negation: 'No increase in wheezing.' ---")
doc = nlp("No increase in wheezing.")
for ent in doc.ents:
    print(f"  Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n--- Custom Rules Demo ---")

context = nlp.get_pipe("medspacy_context")
custom_rules = [
    ConTextRule("denies", "NEGATED_EXISTENCE", direction="FORWARD"),
    ConTextRule("is ruled out", "NEGATED_EXISTENCE", direction="BACKWARD"),
    ConTextRule("however", "TERMINATE", direction="TERMINATE"),
]
context.add(custom_rules)

doc = nlp("Denies fever.")
for ent in doc.ents:
    print(f"  'Denies fever.' -> Entity: '{ent.text}' | Negated: {ent._.is_negated}")

doc = nlp("Pneumonia is ruled out.")
for ent in doc.ents:
    print(f"  'Pneumonia is ruled out.' -> Entity: '{ent.text}' | Negated: {ent._.is_negated}")

print("\n\n" + "=" * 60)
print("FULL CLINICAL NOTE PROCESSING")
print("=" * 60)

clinical_note = """Patient presents with cough. No wheezing or crackles. Chest pain is absent. Denies fever. No increase in shortness of breath. There is no evidence of pneumonia but patient has bronchitis."""

doc = nlp(clinical_note)

print(f"\n{'Entity':<22} {'Label':<10} {'Negated':<10} {'Modifier'}")
print("-" * 65)
for ent in doc.ents:
    if ent._.is_negated:
        modifiers = ent._.modifiers
        if modifiers:
            mod = list(modifiers)[0]
            mod_text = mod.rule.literal
        else:
            mod_text = "N/A"
    else:
        mod_text = "-"
    print(f"{ent.text:<22} {ent.label_:<10} {str(ent._.is_negated):<10} {mod_text}")

print("\n\n" + "=" * 60)
print("KEY TAKEAWAYS")
print("=" * 60)
print("""
1. WHY NEGATION MATTERS: ~50% of conditions in clinical reports are negated.
   Missing negation = false positives = wrong patient data.

2. FORWARD NEGATION: Cue BEFORE finding. 'No wheezing' -> 'No' negates right.

3. BACKWARD NEGATION: Cue AFTER finding. 'Crackles absent' -> 'absent' negates left.

4. CONJUNCTION NEGATION: One cue negates multiple findings connected by 'or'/'and'.
   'No wheezing or crackles' -> both negated by single 'No'.

5. NEGATION SCOPE: How far a negation cue reaches. Ended by:
   - Termination terms ('but', 'however', 'although')
   - End of sentence

6. PSEUDO-NEGATION: Phrases that LOOK like negation but aren't.
   'No increase in wheezing' -> wheezing is NOT negated.

7. WHY REGEX FAILS:
   - Cannot handle backward negation (cue after finding)
   - Cannot handle pseudo-negation (needs exceptions list)
   - Cannot handle scope termination (needs linguistic knowledge)
   - Cannot handle conjunction scope (needs entity awareness)

8. HOW CONTEXT SOLVES IT:
   - Directional rules (FORWARD, BACKWARD, BIDIRECTIONAL)
   - TERMINATE rules to limit scope
   - PSEUDO rules to block false triggers
   - Entity-aware scope within sentence boundaries
""")

Your complete negation pipeline is working. You have a system that correctly interprets clinical language where naive regex fails. Next up, take on the Secret Mission to build a reusable Negation Audit Report function.

Secret mission

Build a Negation Audit Report

Your pipeline can detect negation in individual sentences. But in a real clinical setting, you would process hundreds of notes at once and need a structured summary of what was found. Can you build a reusable audit function that handles tricky edge cases like double negation?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally (or in Google Colab), so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Google Colab notebook (saved in Google Drive).
  • Python files: negation_detection.py and negation_audit.py.
  • medspaCy package (installed via pip in the Colab runtime or locally).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with clinical negation detection or want to extend your pipeline further.

  • Your Google Colab notebook stays in Google Drive and can be reopened anytime.
  • The Colab runtime will disconnect automatically after about 90 minutes of inactivity. When you reconnect, you will need to re-run your cells to reinstall medspaCy and reload your pipeline.
  • If you installed medspaCy locally, the package stays on your machine until you uninstall it.

Pause - I'll come back to this later

Shut down the Colab runtime to free up memory. Your notebook file stays safe in Google Drive.

  • In your Google Colab notebook, click **Runtime** in the top menu bar.
  • Click **Disconnect and delete runtime**.
  • Your notebook code is still saved in Google Drive. When you return, just reconnect and re-run your cells.

Delete - I don't want to use this again

Remove all project resources and start fresh.

  • Delete the Google Colab notebook from Google Drive. Open Google Drive, find the notebook file, right-click it, and select **Move to trash**.
  • If you saved negation_detection.py and negation_audit.py locally, delete them by running:

macOS

rm negation_detection.py negation_audit.py

Windows

Remove-Item negation_detection.py, negation_audit.py
  • If you installed medspaCy locally and want to remove it, run:
pip uninstall medspacy -y

Nice Work!

Nice Work!

You just built a clinical negation detection system that correctly handles the subtleties of medical language. From proving why naive regex fails to implementing a full ConText pipeline, you now understand how NLP systems keep patient data safe.

What you learned:

  • Built a naive regex negation detector to expose three critical failure modes, then implemented a medspaCy pipeline using the ConText algorithm for robust clinical negation detection.
  • Mastered forward negation (cue before finding), backward negation (cue after finding), and conjunction negation (one cue negates multiple findings connected by "or").
  • Controlled negation scope using termination terms like "but" to limit a modifier's reach, and handled pseudo-negation edge cases where phrases like "no increase in" block false triggers.
  • Built a reusable Negation Audit Report function that processes batches of clinical sentences and outputs entity details, modifier information, and summary statistics.

Ready to quiz yourself?