Build a Board Security Triage App

Introduction

30 Second Summary

A circuit board photo can show dozens of tiny parts. Their shapes alone cannot tell you what each part does or whether its security controls are configured safely.

In this project, you will build a local browser app that uses zero-shot object detection to mark component candidates on embedded-board photos. The app connects those visual leads to page-linked datasheet evidence before presenting authorized manual verification questions.

What You'll Build

Your finished app shows candidate components boxed in red beside a cautious report grounded in the datasheet evidence you provide.

By the end of this project, you'll have:

• A local board inspection app that accepts an uploaded photo or webcam snapshot. You can see candidate components marked with labels and scores.
• Adjustable detection controls that let you compare prompt wording at different confidence thresholds. You can see which candidates stay on screen.
• A structured security triage report that links matched datasheet excerpts to PDF page numbers. It keeps visual observations separate from defensive review questions.
• Secret Mission: calibrate one prompt set and confidence threshold across three photos from the same board family. You will record correct detections, false positives, and missed components.

Are there any prerequisites?

You need a 64-bit Windows computer with Python 3.10+, Visual Studio Code, internet access, a working camera, and one authorized embedded-board image. A text-based datasheet PDF is optional, while a GPU is not required.

Before We Start

This first checkpoint defines the embedded board or board family you are authorized to inspect. It gives your investigation one defensive security-review question to answer before the hands-on work begins.

Set Up the Local CPU Environment

The local assistant brings the detection model into a browser. It also annotates images from your board.

Optional PDF evidence adds another set of dependencies. All of these pieces need the same reproducible Python environment.

In this step, get ready to:

• Open the embedded-board-triage project folder in Visual Studio Code.
• Isolate the project inside the tutorial-env virtual environment.
• Pin the required package versions in requirements.txt and confirm the installed set.

Prepare an isolated workspace

A virtual environment gives this project its own Python packages. This prevents another project's package versions from changing how the assistant behaves.

• Press the Windows key to open Windows search.
• Type Visual Studio Code into the search field.
• Press Enter to launch Visual Studio Code.

Why use a virtual environment?

The assistant depends on exact library versions. An isolated environment keeps those versions attached to this project.

This makes later runs easier to reproduce. It also keeps your main Python installation clean.

• Click File in the top menu bar.
• Click Open Folder....
• Select Desktop in the folder dialog.
• Click New Folder.
• Enter embedded-board-triage as the folder name.
• Press Enter to create the folder.
• Click Select Folder to open it in Visual Studio Code.

You'll see embedded-board-triage in the Explorer sidebar. This confirms that Visual Studio Code is using the correct project folder.

• Click View in the top menu bar.
• Click Terminal to open the integrated terminal.
• Create the project environment by running this command:

python -m venv tutorial-env

What does this command do?

• The venv module creates an isolated Python environment.
• The tutorial-env name becomes the environment folder inside embedded-board-triage.

When the command finishes, you'll see the tutorial-env folder in the Explorer sidebar. Your isolated environment now exists inside the project.

Virtual environment missing?

• Confirm that the integrated terminal is open inside embedded-board-triage.
• Check that Python is available in the same terminal session.
• Run the environment creation command again if the first attempt stopped early.

Still stuck? Help me create the tutorial-env virtual environment in my VS Code project folder.

Activate the environment and record dependencies

Activation directs this terminal session to the Python installation inside tutorial-env. Package installations then stay inside the project environment.

• Activate tutorial-env by running this command:

tutorial-env\Scripts\activate

What does activation change?

Activation makes this terminal use the environment's Python executable. It also directs pip to install packages inside tutorial-env.

You'll see the environment name at the start of your terminal prompt. Keep this terminal open so the environment remains active.

Environment name not showing?

• Confirm that tutorial-env appears in the Explorer sidebar.
• Confirm that the integrated terminal is still inside embedded-board-triage.
• Run the activation command again from that folder.

Need a hand? Help me activate tutorial-env in the VS Code integrated terminal.

• Select the Explorer view in the Activity Bar.
• Click the New File... button.
• Enter requirements.txt as the file name.
• Press Enter to create the file.
• Add the pinned dependencies by pasting this code into requirements.txt:

torch==2.14.0
transformers==5.17.0
gradio==6.27.0
pymupdf==1.28.2
pillow==12.3.0
huggingface-hub==1.31.0

What do these packages provide?

• PyTorch 2.14.0 runs model inference on the CPU.
• Hugging Face Transformers 5.17.0 provides the processor and zero-shot detection model APIs.
• Gradio 6.27.0 provides the local browser interface.
• PyMuPDF 1.28.2 extracts text from optional datasheet PDFs.
• Pillow 12.3.0 draws labels and bounding boxes on board images.
• Hugging Face Hub 1.31.0 supports the model download used later.

• Save requirements.txt by pressing Ctrl+S.
• Confirm that requirements.txt appears under embedded-board-triage in the Explorer sidebar.

Good progress. Your project now records every package version needed by the local assistant.

Requirements file looks different?

• Check that the filename is exactly requirements.txt.
• Remove any extra file extension added by Windows.
• Compare each package name and version with the full-file reference below.

Want help checking the file? Help me compare my requirements file with the pinned dependency list.

✔️ Awesome, I've got everything!

Your requirements.txt file is saved with all six dependency pins.

ⓧ I'd like to double check the full code

Compare your complete requirements.txt file with this reference.

torch==2.14.0
transformers==5.17.0
gradio==6.27.0
pymupdf==1.28.2
pillow==12.3.0
huggingface-hub==1.31.0

Why compare the whole file?

Each line pins one dependency to the project version. An exact match gives every later step the expected package set.

Install and verify the pinned packages

The requirements file lets pip install the complete dependency set in one pass. Exact pins keep the environment consistent with the project code.

This installation can take several minutes because PyTorch is a large package. A quiet stretch in the terminal can still be normal.

• Install the pinned dependencies by running this command:

python -m pip install -r requirements.txt

What does this command do?

The command asks the active environment's package installer to read requirements.txt. It downloads each listed package at the recorded version.

Every package goes into tutorial-env. Your system-wide Python packages remain separate.

When installation completes, the terminal returns to its prompt. The environment name should still be visible there.

Package installation stopped?

• Confirm that the terminal still shows the active environment name.
• Check that your internet connection is available for package downloads.
• Run the installation command again if a download ended early.

Still blocked? Help me diagnose why the pinned packages are not installing in tutorial-env.

Before you check, which package names do you expect to appear in the environment?

• List the installed packages by running this command:

python -m pip list

What does this check prove?

The package list comes from the active tutorial-env environment. Matching versions confirm that the project has the dependency set recorded in requirements.txt.

• Find torch with version 2.14.0.
• Find transformers with version 5.17.0.
• Find gradio with version 6.27.0.
• Find pymupdf with version 1.28.2.
• Find pillow with version 12.3.0.
• Find huggingface-hub with version 1.31.0.

That's the environment ready. All six pinned packages are installed inside the activated project environment.

Missing a package or version?

• Confirm that requirements.txt contains all six lines from the reference tab.
• Confirm that the terminal still shows the active environment name.
• Run the installation command again before repeating the package list check.

Need help reading the results? Help me compare my installed package list with the required versions.

Your CPU environment is isolated and reproducible. Next, you'll build the browser interface for board images and optional datasheets.

Build the Upload and Camera Interface

Your isolated environment now has every pinned dependency ready. The next goal is a browser surface that can accept board evidence.

A local Gradio interface creates a visible input-and-output loop before detection logic enters the app. Returning the selected image proves the browser can pass board images into Python.

In this step, get ready to:

• Create app.py with a temporary image handler.
• Build the board-image input plus the review controls.
• Launch the local app to test uploaded images plus webcam snapshots.

Create the temporary inspection handler

The first handler has one focused job. It returns the selected image so you can prove that data reaches Python from the browser.

• Use the file-creation control in the left sidebar of VS Code to add a file inside embedded-board-triage.
• Name the new file app.py.
• Add the editable prompt defaults plus the temporary handler by pasting this code:

import gradio as gr

# Give the editable prompt field a useful starting point
DEFAULT_PROMPTS = (
    "integrated circuit, microcontroller, system on chip, memory chip, "
    "pin header, test point, usb connector, ethernet connector, "
    "sd card slot, crystal oscillator, voltage regulator"
)


# Confirm that an image can travel from the browser into Python
def inspect_board(image, datasheet_pdf, prompt_text, confidence_threshold):
    if image is None:
        return None, "# Input needed\n\nUpload a board image or take a webcam snapshot."

    return image, (
        "# Interface ready\n\n"
        "Image received. Results are review leads, not vulnerability findings."
    )

What does this code do?

• The gradio import makes the browser components available to Python.
• The DEFAULT_PROMPTS value gives the editable prompt field a starting list of visible board categories.
• The inspect_board() handler asks for an image when the input is empty.
• The same handler returns a supplied image plus a short status report.

• Save app.py in VS Code.
• Confirm that app.py appears inside embedded-board-triage in the left sidebar.

Cannot find app.py?

• Check that the file appears beside requirements.txt inside embedded-board-triage.
• Move app.py into embedded-board-triage if it was created in another folder.

Still cannot locate the file? Help me check where app.py was created in VS Code..

Build the browser interface

The gr.Blocks layout groups the evidence controls into one local page. An image preview plus a Markdown report area make the handler result visible.

• In app.py place this code directly below inspect_board():

# Arrange the evidence controls and result areas
with gr.Blocks(title="Embedded Board Security Triage Assistant") as demo:
    with gr.Row():
        with gr.Column():
            # Collect the board image and optional review context
            image_input = gr.Image(sources=["upload", "webcam"], type="pil", label="Board image")
            datasheet_input = gr.File(file_count="single", file_types=[".pdf"], type="filepath", label="Optional datasheet PDF")
            prompt_input = gr.Textbox(value=DEFAULT_PROMPTS, lines=3, label="Component prompts, separated by commas")
            threshold_input = gr.Slider(minimum=0.10, maximum=0.90, value=0.35, step=0.05, label="Confidence threshold")
            inspect_button = gr.Button("Run inspection", variant="primary")

        with gr.Column():
            # Display the returned image and review status
            image_output = gr.Image(type="pil", label="Annotated board")
            report_output = gr.Markdown()

    # Send every input to the temporary inspection handler
    inspect_button.click(
        fn=inspect_board,
        inputs=[image_input, datasheet_input, prompt_input, threshold_input],
        outputs=[image_output, report_output],
    )


if __name__ == "__main__":
    # Start the local browser interface
    demo.launch()

How does the interface work?

• The gr.Image input accepts uploaded images or webcam snapshots as Pillow images.
• The gr.File input limits the optional document to one PDF path.
• The gr.Textbox lets you adjust the component categories for a different board.
• The gr.Slider stores a confidence threshold between 0.10 and 0.90.
• The inspection button sends all four inputs to inspect_board().
• The two output components display the returned image plus the status report.

• Save app.py in VS Code.

Before you run the app, do you expect the interface to appear in the terminal or in your browser?

• Launch the local interface from the activated terminal by running:

python app.py

The terminal shows the local address http://localhost:7860. That address points to the Gradio server running on your computer.

• Keep the terminal process running while you test the interface.
• Enter http://localhost:7860 in your browser address bar.
• Press Enter to load the interface.

You'll see the board-image control beside the optional PDF input. You'll also see the editable prompts plus the confidence slider.

Page not loading?

• Check that the terminal process is still running.
• Confirm that the terminal still shows tutorial-env as the activated environment.
• Check that you entered http://localhost:7860 without adding another path.

Need another pair of eyes? Help me diagnose why my local Gradio page is not loading..

✔️ Awesome, I've got everything!

Great. Confirm that app.py is saved while the local interface remains available in your browser.

ⓧ I'd like to double check the full code

Compare your complete app.py with this version line by line.

import gradio as gr

# Give the editable prompt field a useful starting point
DEFAULT_PROMPTS = (
    "integrated circuit, microcontroller, system on chip, memory chip, "
    "pin header, test point, usb connector, ethernet connector, "
    "sd card slot, crystal oscillator, voltage regulator"
)


# Confirm that an image can travel from the browser into Python
def inspect_board(image, datasheet_pdf, prompt_text, confidence_threshold):
    if image is None:
        return None, "# Input needed\n\nUpload a board image or take a webcam snapshot."

    return image, (
        "# Interface ready\n\n"
        "Image received. Results are review leads, not vulnerability findings."
    )


# Arrange the evidence controls and result areas
with gr.Blocks(title="Embedded Board Security Triage Assistant") as demo:
    with gr.Row():
        with gr.Column():
            # Collect the board image and optional review context
            image_input = gr.Image(sources=["upload", "webcam"], type="pil", label="Board image")
            datasheet_input = gr.File(file_count="single", file_types=[".pdf"], type="filepath", label="Optional datasheet PDF")
            prompt_input = gr.Textbox(value=DEFAULT_PROMPTS, lines=3, label="Component prompts, separated by commas")
            threshold_input = gr.Slider(minimum=0.10, maximum=0.90, value=0.35, step=0.05, label="Confidence threshold")
            inspect_button = gr.Button("Run inspection", variant="primary")

        with gr.Column():
            # Display the returned image and review status
            image_output = gr.Image(type="pil", label="Annotated board")
            report_output = gr.Markdown()

    # Send every input to the temporary inspection handler
    inspect_button.click(
        fn=inspect_board,
        inputs=[image_input, datasheet_input, prompt_input, threshold_input],
        outputs=[image_output, report_output],
    )


if __name__ == "__main__":
    # Start the local browser interface
    demo.launch()

Test uploads and camera input

An upload uses an image already stored on your computer. Webcam capture requires browser permission before the local page can receive a snapshot.

• Use the upload source in Board image.
• Select one photo of the embedded board you are authorized to inspect.
• Select a text-based datasheet in Optional datasheet PDF if you have one.
• Confirm that the selected PDF filename is visible in the input.

Before you select the inspection button, do you expect the returned image to match the board photo you supplied?

• Click Run inspection.

You'll see the selected photo in Annotated board. The report confirms that the image reached the temporary handler.

The first loop is live. Your browser can now pass an uploaded board image into Python plus display the returned result.

Camera permission can feel sensitive. This local page uses it to capture the board photo you choose.

• Switch Board image to the webcam source.
• Approve the browser camera request for http://localhost:7860.
• Frame your authorized board inside the camera preview.
• Capture one snapshot.

Before you run the inspection again, do you expect the webcam snapshot to travel through the same Python handler?

• Click Run inspection.

You'll see the captured snapshot in Annotated board. This confirms that both supported image paths reach the local handler.

• Press Ctrl+C in the integrated terminal to stop the local server.

Your local interface now accepts uploaded board photos plus webcam snapshots. Next, you'll connect those images to CPU-based component detection.

Detect Visible Board Components

Your Gradio interface can already collect a board image. The interface still needs a way to interpret the visible hardware.

This step adds zero-shot object detection so the app can find component candidates from text prompts. You will also see the model's key limitation. A visual label cannot prove an exact identity or security state.

In this step, get ready to:

• Load the detector only when the first inspection needs it.
• Convert the component prompt into labels for CPU inference.
• Draw accepted candidates with red boxes and confidence scores.

Load the detector on demand

Loading Grounding DINO Tiny when Python imports the file would slow every launch. Lazy loading waits until the first inspection. Later inspections reuse the same model objects.

Why zero-shot detection?

Zero-shot detection uses text labels without requiring a custom training dataset. That makes it a practical starting point for a CPU-only inspection workflow.

A trained detector can become more specific after you collect labeled examples. This step focuses on quickly generating visual leads.

• In app.py, replace the top configuration section with the code below:

import gradio as gr
import torch
from PIL import Image, ImageDraw
from transformers import AutoModelForZeroShotObjectDetection, AutoProcessor

MODEL_ID = "IDEA-Research/grounding-dino-tiny"
DEVICE = "cpu"
DEFAULT_PROMPTS = (
    "integrated circuit, microcontroller, system on chip, memory chip, "
    "pin header, test point, usb connector, ethernet connector, "
    "sd card slot, crystal oscillator, voltage regulator"
)

_PROCESSOR = None
_MODEL = None

What does this configuration do?

• The Transformers imports provide the processor and zero-shot detection model classes.
• The model identifier tells Hugging Face which detector to download.
• The device setting keeps PyTorch inference on the CPU.
• The two private variables hold the loaded processor and model for reuse.

• Add the lazy-loading helper below _MODEL = None by pasting this code:

def get_model():
    global _PROCESSOR, _MODEL
    if _PROCESSOR is None or _MODEL is None:
        _PROCESSOR = AutoProcessor.from_pretrained(MODEL_ID)
        _MODEL = AutoModelForZeroShotObjectDetection.from_pretrained(MODEL_ID)
        _MODEL.to(DEVICE)
        _MODEL.eval()
    return _PROCESSOR, _MODEL

How does lazy loading work?

• The condition checks whether either cached object still needs to be created.
• The processor converts images and prompt labels into model inputs.
• The model moves to the selected device before entering evaluation mode.
• Every call returns the same cached objects after the first load.

• Save app.py.
• Check the new imports and configuration by running the app:

python app.py

What does this check prove?

This command imports the updated file before starting the local server. A successful launch confirms that Python can load the new detection libraries.

Your existing interface should still load at http://localhost:7860. The model remains unloaded because no inspection has requested it.

Does the app stop during import?

Confirm that the integrated terminal still shows (tutorial-env) before its prompt. An inactive environment may use a Python installation without the pinned packages.

Compare the import names with the code block above. Help me diagnose an import failure in this app.

• Stop the server by pressing Ctrl+C in the integrated terminal.

Convert prompts and draw candidates

The detector expects a clean list of text labels. The editable prompt field may contain extra spaces or an empty value.

• Add the prompt-normalization helper below get_model() by pasting this code:

def normalize_prompts(prompt_text):
    prompts = [
        item.strip().lower().rstrip(".")
        for item in (prompt_text or "").split(",")
        if item.strip()
    ]
    if not prompts:
        prompts = [item.strip() for item in DEFAULT_PROMPTS.split(",")]
    return prompts

How are prompts prepared?

• The list comprehension splits the text at each comma.
• Each label becomes lowercase text without surrounding spaces or a trailing full stop.
• An empty prompt field falls back to the default component categories.

The model returns coordinates and scores. Pillow turns those values into visible red annotations on a copy of the board image.

• Add the annotation helper below normalize_prompts() by pasting this code:

def draw_detections(image, result):
    annotated = image.copy()
    draw = ImageDraw.Draw(annotated)
    detections = []
    width, height = annotated.size

    for box, score, text_label in zip(
        result["boxes"], result["scores"], result["text_labels"]
    ):
        x0, y0, x1, y1 = [int(round(value)) for value in box.tolist()]
        x0 = max(0, min(width - 1, x0))
        y0 = max(0, min(height - 1, y0))
        x1 = max(0, min(width - 1, x1))
        y1 = max(0, min(height - 1, y1))
        if x1 <= x0 or y1 <= y0:
            continue
        label = str(text_label)
        confidence = float(score.item())
        caption = f"{label}: {confidence:.2f}"
        draw.rectangle((x0, y0, x1, y1), outline="red", width=3)
        draw.text((x0 + 4, y0 + 4), caption, fill="red", font_size=16)
        detections.append(
            {
                "label": label,
                "score": confidence,
                "box": [x0, y0, x1, y1],
            }
        )
    detections.sort(key=lambda item: item["score"], reverse=True)
    return annotated, detections

What does the annotation helper do?

• The coordinate checks keep every box inside the original image boundaries.
• The caption combines the proposed label with a confidence score.
• The red rectangle marks the candidate area on a copied image.
• The returned detection records preserve each label for the report.

• Save app.py.
• Check both helpers for syntax errors by running the app again:

python app.py

What does this second check prove?

A successful launch confirms that Python can define both helpers. The browser interface should remain available with its existing controls.

You should see the same image input and confidence slider at http://localhost:7860.

• Stop the server by pressing Ctrl+C in the integrated terminal.

Run CPU inference and inspect the result

The final handler connects the browser controls to the detector. It preserves the selected confidence threshold during post-processing at the image's original size.

• Add this placeholder PDF record below normalize_prompts() so the existing optional input remains compatible with the handler:

def inspect_datasheet(pdf_path):
    info = {
        "uploaded": False,
        "page_count": 0,
        "pages_with_text": 0,
        "findings": [],
        "error": "",
    }
    if not pdf_path:
        return info

    info["uploaded"] = True
    return info

Why keep a PDF record here?

The interface already passes its optional PDF value into the inspection handler. This small record keeps that input compatible while this step focuses on visual detection.

• Add the visual report builder below draw_detections() by pasting this code:

def build_report(detections, datasheet_info):
    lines = [
        "# Security Triage Report",
        "",
        "## Visual observations",
    ]

    if detections:
        for item in detections:
            x0, y0, x1, y1 = item["box"]
            lines.append(
                f"- {item['label']} at confidence {item['score']:.2f}; "
                f"box [{x0}, {y0}, {x1}, {y1}]"
            )
    else:
        lines.append(
            "- No candidate passed the selected threshold. This does not show that the components are absent. Try clearer lighting, a closer image, different prompt wording, or a lower threshold."
        )

    return "\n".join(lines)

What does the visual report show?

Each accepted candidate appears with its proposed label and confidence score. The report also records its box coordinates.

An empty result produces a cautious message instead of claiming that the requested components are absent.

• Replace the temporary inspect_board() handler with this detection handler:

def inspect_board(image, datasheet_pdf, prompt_text, confidence_threshold):
    if image is None:
        return None, "# Input needed\n\nUpload a board image or take a webcam snapshot."

    image = image.convert("RGB")
    prompts = normalize_prompts(prompt_text)
    processor, model = get_model()
    inputs = processor(images=image, text=[prompts], return_tensors="pt")

    with torch.no_grad():
        outputs = model(**inputs)

    result = processor.post_process_grounded_object_detection(
        outputs,
        inputs.input_ids,
        threshold=float(confidence_threshold),
        text_threshold=0.25,
        target_sizes=[image.size[::-1]],
    )[0]

    annotated, detections = draw_detections(image, result)
    datasheet_info = inspect_datasheet(datasheet_pdf)
    report = build_report(detections, datasheet_info)
    return annotated, report

How does an inspection run?

• The input image becomes an RGB image before processing.
• The processor combines the image with the normalized labels.
• The no-gradient context reduces unnecessary work during inference.
• Post-processing maps accepted boxes back to the original image dimensions.

• Save app.py.

Before you launch this version, do you think a red box can prove the exact component identity or its security state?

• Launch the updated detector by running this command:

python app.py

What does this launch command do?

The command starts the updated local app. The model download begins only after you request the first inspection.

• Return to the browser tab at http://localhost:7860.
• Select a clear board photo with the Board image control.
• Review the comma-separated labels in the Component prompts, separated by commas field.
• Keep the Confidence threshold at 0.35 for the first inspection.

The first inspection may take several minutes while Hugging Face downloads the 689 MB model file into its local cache. Later inspections reuse those files.

• Start the first detection by clicking Run inspection.

You should see one or more red bounding boxes with labels and confidence scores. A difficult image may instead produce a report stating that no candidate passed the selected threshold.

That uncertainty is the intended limitation. Each label is a visual lead that still needs confirmation from package markings and vendor documentation.

No boxes or model result?

• Use a closer photo with even lighting if small package details are difficult to distinguish.
• Try a broader prompt such as integrated circuit if a highly specific category returns nothing.
• Lower the confidence threshold one step if plausible candidates remain below the current cutoff.

Still stuck? Help me diagnose why Grounding DINO returns no visible candidates for my authorized board image.

✔️ Awesome, I've got everything!

Your app now runs CPU-based component detection. Your first successful inspection also places the model files in the local Hugging Face cache.

ⓧ I'd like to double check the full code

Compare your current app.py with the complete detection version below.

import gradio as gr
import torch
from PIL import Image, ImageDraw
from transformers import AutoModelForZeroShotObjectDetection, AutoProcessor

MODEL_ID = "IDEA-Research/grounding-dino-tiny"
DEVICE = "cpu"
DEFAULT_PROMPTS = (
    "integrated circuit, microcontroller, system on chip, memory chip, "
    "pin header, test point, usb connector, ethernet connector, "
    "sd card slot, crystal oscillator, voltage regulator"
)

_PROCESSOR = None
_MODEL = None


def get_model():
    global _PROCESSOR, _MODEL
    if _PROCESSOR is None or _MODEL is None:
        _PROCESSOR = AutoProcessor.from_pretrained(MODEL_ID)
        _MODEL = AutoModelForZeroShotObjectDetection.from_pretrained(MODEL_ID)
        _MODEL.to(DEVICE)
        _MODEL.eval()
    return _PROCESSOR, _MODEL


def normalize_prompts(prompt_text):
    prompts = [
        item.strip().lower().rstrip(".")
        for item in (prompt_text or "").split(",")
        if item.strip()
    ]
    if not prompts:
        prompts = [item.strip() for item in DEFAULT_PROMPTS.split(",")]
    return prompts


def inspect_datasheet(pdf_path):
    info = {
        "uploaded": False,
        "page_count": 0,
        "pages_with_text": 0,
        "findings": [],
        "error": "",
    }
    if not pdf_path:
        return info

    info["uploaded"] = True
    return info


def draw_detections(image, result):
    annotated = image.copy()
    draw = ImageDraw.Draw(annotated)
    detections = []
    width, height = annotated.size

    for box, score, text_label in zip(
        result["boxes"], result["scores"], result["text_labels"]
    ):
        x0, y0, x1, y1 = [int(round(value)) for value in box.tolist()]
        x0 = max(0, min(width - 1, x0))
        y0 = max(0, min(height - 1, y0))
        x1 = max(0, min(width - 1, x1))
        y1 = max(0, min(height - 1, y1))
        if x1 <= x0 or y1 <= y0:
            continue

        label = str(text_label)
        confidence = float(score.item())
        caption = f"{label}: {confidence:.2f}"
        draw.rectangle((x0, y0, x1, y1), outline="red", width=3)
        draw.text((x0 + 4, y0 + 4), caption, fill="red", font_size=16)
        detections.append(
            {
                "label": label,
                "score": confidence,
                "box": [x0, y0, x1, y1],
            }
        )

    detections.sort(key=lambda item: item["score"], reverse=True)
    return annotated, detections


def build_report(detections, datasheet_info):
    lines = [
        "# Security Triage Report",
        "",
        "## Visual observations",
    ]

    if detections:
        for item in detections:
            x0, y0, x1, y1 = item["box"]
            lines.append(
                f"- {item['label']} at confidence {item['score']:.2f}; "
                f"box [{x0}, {y0}, {x1}, {y1}]"
            )
    else:
        lines.append(
            "- No candidate passed the selected threshold. This does not show that the components are absent. Try clearer lighting, a closer image, different prompt wording, or a lower threshold."
        )

    return "\n".join(lines)


def inspect_board(image, datasheet_pdf, prompt_text, confidence_threshold):
    if image is None:
        return None, "# Input needed\n\nUpload a board image or take a webcam snapshot."

    image = image.convert("RGB")
    prompts = normalize_prompts(prompt_text)
    processor, model = get_model()
    inputs = processor(images=image, text=[prompts], return_tensors="pt")

    with torch.no_grad():
        outputs = model(**inputs)

    result = processor.post_process_grounded_object_detection(
        outputs,
        inputs.input_ids,
        threshold=float(confidence_threshold),
        text_threshold=0.25,
        target_sizes=[image.size[::-1]],
    )[0]

    annotated, detections = draw_detections(image, result)
    datasheet_info = inspect_datasheet(datasheet_pdf)
    report = build_report(detections, datasheet_info)
    return annotated, report


with gr.Blocks(title="Embedded Board Security Triage Assistant") as demo:
    gr.Markdown(
        "# Embedded Board Security Triage Assistant\n"
        "Upload a board image or take a camera snapshot. Optionally add a text-based "
        "datasheet PDF. Results are review leads, not vulnerability findings."
    )

    with gr.Row():
        with gr.Column():
            image_input = gr.Image(
                sources=["upload", "webcam"],
                type="pil",
                label="Board image",
            )
            datasheet_input = gr.File(
                file_count="single",
                file_types=[".pdf"],
                type="filepath",
                label="Optional datasheet PDF",
            )
            prompt_input = gr.Textbox(
                value=DEFAULT_PROMPTS,
                lines=3,
                label="Component prompts, separated by commas",
            )
            threshold_input = gr.Slider(
                minimum=0.10,
                maximum=0.90,
                value=0.35,
                step=0.05,
                label="Confidence threshold",
            )
            inspect_button = gr.Button("Run inspection", variant="primary")

        with gr.Column():
            image_output = gr.Image(type="pil", label="Annotated board")
            report_output = gr.Markdown()

    inspect_button.click(
        fn=inspect_board,
        inputs=[image_input, datasheet_input, prompt_input, threshold_input],
        outputs=[image_output, report_output],
    )


if __name__ == "__main__":
    demo.launch()

Your browser can now turn a board photo into visible component candidates while preserving uncertainty. Next, you will add page-linked datasheet evidence to test those visual leads against documentation.

Add Datasheet Evidence

Your Grounding DINO Tiny detector now turns a board photo into visual leads. A box can suggest a component category. Appearance alone cannot establish the component's actual function.

An optional PDF datasheet can support or contradict those visual leads. PyMuPDF extracts local page text so every matched review topic can point back to its evidence source.

In this step, get ready to:

• Define datasheet terms for five defensive review topics.
• Extract page-linked excerpts from an optional text-based PDF.
• Keep image detection working when datasheet evidence is unavailable.

Define the datasheet review map

The PDF parser needs access to file paths before it can open the optional datasheet. It also needs a fixed map connecting search terms to defensive review questions.

• In app.py, replace the import section at the top with this code:

from pathlib import Path

import gradio as gr
import pymupdf
import torch
from PIL import Image, ImageDraw
from transformers import AutoModelForZeroShotObjectDetection, AutoProcessor

What do these imports add?

• Path provides the selected file's extension so the app can reject a non-PDF path safely.
• pymupdf opens the local document and extracts text from each page.
• The existing imports preserve the detector, interface, and image annotation workflow.

• Save app.py.
• Check the updated import section for red syntax underlines in VS Code.

The import section should remain free of red syntax underlines.

Seeing an import warning?

Confirm that VS Code is still using the Python interpreter from tutorial-env. The dependency is already listed as pymupdf==1.28.2 in requirements.txt.

Need help checking the active interpreter? Help me connect VS Code to the existing tutorial-env environment.

Each review topic groups several phrases that may appear in vendor documentation. A matched phrase becomes a lead linked to a manual verification question.

• In app.py, add this dictionary below DEFAULT_PROMPTS and above _PROCESSOR.

DATASHEET_TOPICS = {
    "Debug access": {
        "terms": ["jtag", "swd", "uart", "debug interface", "debug port"],
        "question": "Does the documentation describe how debug access is disabled, authenticated, locked, or restricted in production?",
    },
    "Boot trust": {
        "terms": ["secure boot", "verified boot", "boot rom", "bootloader", "root of trust"],
        "question": "Does the documented boot chain verify each executable stage before it runs?",
    },
    "Storage and lifecycle state": {
        "terms": ["spi flash", "flash memory", "emmc", "otp", "efuse", "one-time programmable"],
        "question": "What data can be stored externally, and can security configuration or lifecycle state be locked?",
    },
    "Cryptography and key protection": {
        "terms": ["trustzone", "key storage", "cryptographic", "encryption", "security engine"],
        "question": "Which hardware-backed protections are documented, and which require explicit firmware configuration?",
    },
    "Firmware updates": {
        "terms": ["firmware update", "signed firmware", "image authentication", "rollback protection"],
        "question": "Does the documentation describe update authentication, authorization, and rollback behavior?",
    },
}

What does this review map do?

• Debug access searches for documentation about internal interfaces such as JTAG or UART.
• Boot trust searches for terms associated with verified startup stages.
• Storage and lifecycle state looks for external storage plus lockable device configuration.
• Cryptography and key protection looks for hardware-backed protection claims that still require configuration review.
• Firmware updates looks for authentication plus rollback behavior.

• Save app.py.
• Check every topic entry for matching braces in VS Code.

VS Code should show one folded dictionary with five top-level topics.

Seeing a dictionary syntax warning?

Check the comma after each nested topic entry. Confirm that the final closing brace lines up with DATASHEET_TOPICS.

Still stuck? Help me find the unmatched delimiter in DATASHEET_TOPICS.

A complete PDF page can contain thousands of characters. A short excerpt keeps the report readable while preserving the words surrounding the matched term.

• In app.py, add this helper below normalize_prompts().

def excerpt_around(text, term, radius=120):
    collapsed = " ".join(text.split())
    index = collapsed.lower().find(term.lower())
    if index < 0:
        return ""
    start = max(0, index - radius)
    end = min(len(collapsed), index + len(term) + radius)
    prefix = "..." if start else ""
    suffix = "..." if end < len(collapsed) else ""
    return prefix + collapsed[start:end] + suffix

How are excerpts kept readable?

• collapsed converts repeated whitespace into single spaces.
• index finds the first case-insensitive occurrence of the matched term.
• radius keeps nearby context on both sides of the match.
• prefix and suffix show when text was trimmed from either end.

• Save app.py.
• Check that excerpt_around() begins at the left edge of the editor.

The helper should sit outside normalize_prompts() as its own function.

Is the helper nested accidentally?

Remove any leading spaces before def excerpt_around. Its body should remain indented by four spaces.

Need an indentation check? Help me place excerpt_around() at the module level.

Extract page-linked evidence

The parser returns structured status for every outcome. Image detection can continue because missing text or a damaged document becomes report data instead of an uncaught exception.

• In app.py, add this function immediately below excerpt_around().

def inspect_datasheet(pdf_path):
    info = {
        "uploaded": False,
        "page_count": 0,
        "pages_with_text": 0,
        "findings": [],
        "error": "",
    }
    if not pdf_path:
        return info

    info["uploaded"] = True
    if Path(pdf_path).suffix.lower() != ".pdf":
        info["error"] = "The optional document must be a PDF."
        return info

    try:
        topics_seen = set()
        with pymupdf.open(pdf_path) as document:
            info["page_count"] = document.page_count
            for page_number, page in enumerate(document, start=1):
                text = page.get_text("text")
                if not text.strip():
                    continue
                info["pages_with_text"] += 1
    except Exception as exc:
        info["error"] = f"Could not read the PDF: {exc}"

    return info

What does the parser track?

• uploaded distinguishes an omitted datasheet from a supplied file.
• page_count records the document's full length.
• pages_with_text counts pages where plain text extraction succeeds.
• error captures an invalid extension or parser failure without stopping the detector.

• Save app.py.
• Check the entire try block for consistent indentation in VS Code.

The except line should align with try.

Seeing an indentation warning?

Keep with pymupdf.open(pdf_path) as document inside the try block. Keep page extraction inside the for loop.

Need help aligning the blocks? Help me correct the indentation in inspect_datasheet().

Text extraction supplies the searchable page content. The next loop keeps only the first page-linked finding for each configured topic so repeated datasheet terminology does not flood the report.

• Inside inspect_datasheet(), find the line that increments pages_with_text.
• Paste this matching loop directly below that line:

                lowered = text.lower()
                for topic, details in DATASHEET_TOPICS.items():
                    if topic in topics_seen:
                        continue
                    matched_term = next(
                        (term for term in details["terms"] if term in lowered),
                        None,
                    )
                    if matched_term:
                        info["findings"].append(
                            {
                                "topic": topic,
                                "page": page_number,
                                "term": matched_term,
                                "excerpt": excerpt_around(text, matched_term),
                                "question": details["question"],
                            }
                        )
                        topics_seen.add(topic)

How are findings correlated?

• lowered makes matching independent of capitalization.
• matched_term captures the first configured phrase found on the current page.
• findings stores the topic, page number, matched phrase, excerpt, and review question together.
• topics_seen limits each topic to one finding.

• Save app.py.
• Check that lowered = text.lower() aligns with info["pages_with_text"] += 1.

The matching loop should remain inside the page loop. This placement preserves the page number attached to every excerpt.

Are findings always empty?

Check that the matching loop is indented inside for page_number, page in enumerate(document, start=1). Confirm that your text-based PDF contains at least one configured term.

Still seeing no matches? Help me trace why inspect_datasheet() returns no findings.

Connect evidence to the report

The report already summarizes visual observations. It now needs a second evidence section that handles an omitted file, unreadable text, parser errors, and successful topic matches.

• In app.py, replace the current build_report() function with this visual-report foundation:

def build_report(detections, datasheet_info):
    lines = [
        "# Security Triage Report",
        "",
        "> Authorized defensive use only. Visual detections and keyword matches are leads for manual verification, not proof of component identity, interface state, or vulnerability.",
        "",
        "## Visual observations",
    ]

    if detections:
        for item in detections:
            x0, y0, x1, y1 = item["box"]
            lines.append(
                f"- {item['label']} at confidence {item['score']:.2f}; "
                f"box [{x0}, {y0}, {x1}, {y1}]"
            )
    else:
        lines.append(
            "- No candidate passed the selected threshold. This does not show that the components are absent. Try clearer lighting, a closer image, different prompt wording, or a lower threshold."
        )

    return "\n".join(lines)

Why pass datasheet information here?

build_report() now receives visual detections plus the structured PDF status. The current visual observations remain intact while the second input becomes available for evidence reporting.

• Save app.py.
• Check that build_report() remains outside every other function.

The function should end with its own return "\n".join(lines) line.

Is the report function nested?

Align def build_report with the other top-level function definitions. Keep its body indented by four spaces.

Need help checking the function boundary? Help me place build_report() correctly in app.py.

• Inside build_report(), insert this datasheet section immediately before return "\n".join(lines).

    lines.extend(["", "## Datasheet evidence"])
    if not datasheet_info["uploaded"]:
        lines.append(
            "- No datasheet was uploaded. The report remains visual-only and requires vendor documentation before any conclusion."
        )
    elif datasheet_info["error"]:
        lines.append(f"- {datasheet_info['error']}")
    elif datasheet_info["pages_with_text"] == 0:
        lines.append(
            f"- The PDF has {datasheet_info['page_count']} page(s), but no extractable text was found. It may be scanned; OCR is outside this project's scope."
        )
    else:
        lines.append(
            f"- Extracted text from {datasheet_info['pages_with_text']} of "
            f"{datasheet_info['page_count']} page(s)."
        )
        if datasheet_info["findings"]:
            for finding in datasheet_info["findings"]:
                lines.append(
                    f"- {finding['topic']}, page {finding['page']}, matched "
                    f"`{finding['term']}`: {finding['excerpt']}"
                )
                lines.append(f"  - Review question: {finding['question']}")
        else:
            lines.append(
                "- No configured review term was found. This is not evidence that the capability is absent; inspect the table of contents, security manual, reference manual, and errata manually."
            )

How does the report handle each PDF state?

• An omitted PDF produces a visual-only evidence message.
• A parser error appears as a report item while the annotated image remains available.
• A scanned PDF reports its page count plus the absence of extractable text.
• A text-based PDF reports pages containing text plus page-linked findings.

• Save app.py.
• Check that the datasheet section appears before the final report return line.

The report function should now finish with the datasheet branch followed by return "\n".join(lines).

Seeing unreachable report code?

Move the entire datasheet section above the final return line. Code placed below that return cannot add evidence to lines.

Need help checking the order? Help me place the datasheet branch before build_report() returns.

The inspection handler is the bridge between image detection and PDF analysis. It should inspect the optional file after drawing detections, then pass both evidence sources into the report.

• At the end of inspect_board(), replace the current report-building lines with this code:

    annotated, detections = draw_detections(image, result)
    datasheet_info = inspect_datasheet(datasheet_pdf)
    report = build_report(detections, datasheet_info)
    return annotated, report

What changes in the inspection flow?

• draw_detections() continues producing the annotated image plus visual observations.
• inspect_datasheet() converts the optional path into structured evidence status.
• build_report() combines both inputs without changing CPU inference.

• Save app.py.
• Check the final four lines of inspect_board() for matching indentation.

All four lines should remain inside inspect_board().

Does the handler mention missing arguments?

Confirm that build_report() accepts detections plus datasheet_info. Confirm that the handler passes both values in the same order.

Still stuck? Help me connect inspect_datasheet() to build_report().

✔️ Awesome, I've got everything!

Great. Double check that you saved app.py. Your existing requirements.txt already contains the required PyMuPDF pin.

ⓧ I'd like to double check the full code

Here is the complete app.py for this stage. Compare it with yours line by line.

from pathlib import Path

import gradio as gr
import pymupdf
import torch
from PIL import Image, ImageDraw
from transformers import AutoModelForZeroShotObjectDetection, AutoProcessor

MODEL_ID = "IDEA-Research/grounding-dino-tiny"
DEVICE = "cpu"
DEFAULT_PROMPTS = (
    "integrated circuit, microcontroller, system on chip, memory chip, "
    "pin header, test point, usb connector, ethernet connector, "
    "sd card slot, crystal oscillator, voltage regulator"
)

_PROCESSOR = None
_MODEL = None

DATASHEET_TOPICS = {
    "Debug access": {
        "terms": ["jtag", "swd", "uart", "debug interface", "debug port"],
        "question": "Does the documentation describe how debug access is disabled, authenticated, locked, or restricted in production?",
    },
    "Boot trust": {
        "terms": ["secure boot", "verified boot", "boot rom", "bootloader", "root of trust"],
        "question": "Does the documented boot chain verify each executable stage before it runs?",
    },
    "Storage and lifecycle state": {
        "terms": ["spi flash", "flash memory", "emmc", "otp", "efuse", "one-time programmable"],
        "question": "What data can be stored externally, and can security configuration or lifecycle state be locked?",
    },
    "Cryptography and key protection": {
        "terms": ["trustzone", "key storage", "cryptographic", "encryption", "security engine"],
        "question": "Which hardware-backed protections are documented, and which require explicit firmware configuration?",
    },
    "Firmware updates": {
        "terms": ["firmware update", "signed firmware", "image authentication", "rollback protection"],
        "question": "Does the documentation describe update authentication, authorization, and rollback behavior?",
    },
}


def get_model():
    global _PROCESSOR, _MODEL
    if _PROCESSOR is None or _MODEL is None:
        _PROCESSOR = AutoProcessor.from_pretrained(MODEL_ID)
        _MODEL = AutoModelForZeroShotObjectDetection.from_pretrained(MODEL_ID)
        _MODEL.to(DEVICE)
        _MODEL.eval()
    return _PROCESSOR, _MODEL


def normalize_prompts(prompt_text):
    prompts = [
        item.strip().lower().rstrip(".")
        for item in (prompt_text or "").split(",")
        if item.strip()
    ]
    if not prompts:
        prompts = [item.strip() for item in DEFAULT_PROMPTS.split(",")]
    return prompts


def excerpt_around(text, term, radius=120):
    collapsed = " ".join(text.split())
    index = collapsed.lower().find(term.lower())
    if index < 0:
        return ""
    start = max(0, index - radius)
    end = min(len(collapsed), index + len(term) + radius)
    prefix = "..." if start else ""
    suffix = "..." if end < len(collapsed) else ""
    return prefix + collapsed[start:end] + suffix


def inspect_datasheet(pdf_path):
    info = {
        "uploaded": False,
        "page_count": 0,
        "pages_with_text": 0,
        "findings": [],
        "error": "",
    }
    if not pdf_path:
        return info

    info["uploaded"] = True
    if Path(pdf_path).suffix.lower() != ".pdf":
        info["error"] = "The optional document must be a PDF."
        return info

    try:
        topics_seen = set()
        with pymupdf.open(pdf_path) as document:
            info["page_count"] = document.page_count
            for page_number, page in enumerate(document, start=1):
                text = page.get_text("text")
                if not text.strip():
                    continue
                info["pages_with_text"] += 1
                lowered = text.lower()
                for topic, details in DATASHEET_TOPICS.items():
                    if topic in topics_seen:
                        continue
                    matched_term = next(
                        (term for term in details["terms"] if term in lowered),
                        None,
                    )
                    if matched_term:
                        info["findings"].append(
                            {
                                "topic": topic,
                                "page": page_number,
                                "term": matched_term,
                                "excerpt": excerpt_around(text, matched_term),
                                "question": details["question"],
                            }
                        )
                        topics_seen.add(topic)
    except Exception as exc:
        info["error"] = f"Could not read the PDF: {exc}"

    return info


def draw_detections(image, result):
    annotated = image.copy()
    draw = ImageDraw.Draw(annotated)
    detections = []
    width, height = annotated.size

    for box, score, text_label in zip(
        result["boxes"], result["scores"], result["text_labels"]
    ):
        x0, y0, x1, y1 = [int(round(value)) for value in box.tolist()]
        x0 = max(0, min(width - 1, x0))
        y0 = max(0, min(height - 1, y0))
        x1 = max(0, min(width - 1, x1))
        y1 = max(0, min(height - 1, y1))
        if x1 <= x0 or y1 <= y0:
            continue

        label = str(text_label)
        confidence = float(score.item())
        caption = f"{label}: {confidence:.2f}"
        draw.rectangle((x0, y0, x1, y1), outline="red", width=3)
        draw.text((x0 + 4, y0 + 4), caption, fill="red", font_size=16)
        detections.append(
            {
                "label": label,
                "score": confidence,
                "box": [x0, y0, x1, y1],
            }
        )

    detections.sort(key=lambda item: item["score"], reverse=True)
    return annotated, detections


def build_report(detections, datasheet_info):
    lines = [
        "# Security Triage Report",
        "",
        "> Authorized defensive use only. Visual detections and keyword matches are leads for manual verification, not proof of component identity, interface state, or vulnerability.",
        "",
        "## Visual observations",
    ]

    if detections:
        for item in detections:
            x0, y0, x1, y1 = item["box"]
            lines.append(
                f"- {item['label']} at confidence {item['score']:.2f}; "
                f"box [{x0}, {y0}, {x1}, {y1}]"
            )
    else:
        lines.append(
            "- No candidate passed the selected threshold. This does not show that the components are absent. Try clearer lighting, a closer image, different prompt wording, or a lower threshold."
        )

    lines.extend(["", "## Datasheet evidence"])
    if not datasheet_info["uploaded"]:
        lines.append(
            "- No datasheet was uploaded. The report remains visual-only and requires vendor documentation before any conclusion."
        )
    elif datasheet_info["error"]:
        lines.append(f"- {datasheet_info['error']}")
    elif datasheet_info["pages_with_text"] == 0:
        lines.append(
            f"- The PDF has {datasheet_info['page_count']} page(s), but no extractable text was found. It may be scanned; OCR is outside this project's scope."
        )
    else:
        lines.append(
            f"- Extracted text from {datasheet_info['pages_with_text']} of "
            f"{datasheet_info['page_count']} page(s)."
        )
        if datasheet_info["findings"]:
            for finding in datasheet_info["findings"]:
                lines.append(
                    f"- {finding['topic']}, page {finding['page']}, matched "
                    f"`{finding['term']}`: {finding['excerpt']}"
                )
                lines.append(f"  - Review question: {finding['question']}")
        else:
            lines.append(
                "- No configured review term was found. This is not evidence that the capability is absent; inspect the table of contents, security manual, reference manual, and errata manually."
            )

    return "\n".join(lines)


def inspect_board(image, datasheet_pdf, prompt_text, confidence_threshold):
    if image is None:
        return None, "# Input needed\n\nUpload a board image or take a webcam snapshot."

    image = image.convert("RGB")
    prompts = normalize_prompts(prompt_text)
    processor, model = get_model()
    inputs = processor(images=image, text=[prompts], return_tensors="pt")

    with torch.no_grad():
        outputs = model(**inputs)

    result = processor.post_process_grounded_object_detection(
        outputs,
        inputs.input_ids,
        threshold=float(confidence_threshold),
        text_threshold=0.25,
        target_sizes=[image.size[::-1]],
    )[0]

    annotated, detections = draw_detections(image, result)
    datasheet_info = inspect_datasheet(datasheet_pdf)
    report = build_report(detections, datasheet_info)
    return annotated, report


with gr.Blocks(title="Embedded Board Security Triage Assistant") as demo:
    gr.Markdown(
        "# Embedded Board Security Triage Assistant\n"
        "Upload a board image or take a camera snapshot. Optionally add a text-based "
        "datasheet PDF. Results are review leads, not vulnerability findings."
    )

    with gr.Row():
        with gr.Column():
            image_input = gr.Image(
                sources=["upload", "webcam"],
                type="pil",
                label="Board image",
            )
            datasheet_input = gr.File(
                file_count="single",
                file_types=[".pdf"],
                type="filepath",
                label="Optional datasheet PDF",
            )
            prompt_input = gr.Textbox(
                value=DEFAULT_PROMPTS,
                lines=3,
                label="Component prompts, separated by commas",
            )
            threshold_input = gr.Slider(
                minimum=0.10,
                maximum=0.90,
                value=0.35,
                step=0.05,
                label="Confidence threshold",
            )
            inspect_button = gr.Button("Run inspection", variant="primary")

        with gr.Column():
            image_output = gr.Image(type="pil", label="Annotated board")
            report_output = gr.Markdown()

    inspect_button.click(
        fn=inspect_board,
        inputs=[image_input, datasheet_input, prompt_input, threshold_input],
        outputs=[image_output, report_output],
    )


if __name__ == "__main__":
    demo.launch()

Before you run the updated app, do you expect a missing PDF to stop image detection?

• Return to the activated integrated terminal from earlier.
• Start the updated app by running this command:

python app.py

• Open http://localhost:7860 in your browser.
• Select a clear board image in the Board image input.
• Leave Optional datasheet PDF empty.
• Click Run inspection.

You should still see the annotated board. The report should state that no datasheet was uploaded.

• Select a text-based datasheet in the Optional datasheet PDF input.
• Click Run inspection again.

You should see the PDF page count plus the number of pages containing extractable text. Any matched review topic should include its page number, excerpt, and review question.

That is the evidence loop working. Your visual detector now keeps running across successful, missing, scanned, and unreadable datasheet states.

No datasheet excerpts in the report?

• Confirm that the PDF contains selectable text. A scanned image-only document can have pages without extractable text.
• Search the document for one of the terms in DATASHEET_TOPICS.
• Check that page.get_text("text") remains inside the page loop.

Need help tracing the PDF path? Help me debug missing datasheet evidence without breaking image detection.

Your app can now connect visual leads to page-linked documentation evidence without blocking detection when a PDF is unavailable. Next, you will turn those inputs into cautious manual verification guidance.

Produce the Security Triage Report

Your local Gradio app can already mark component candidates and retrieve page-linked datasheet excerpts. You now have two useful evidence streams from the board you are authorized to inspect.

Raw detections and keyword matches can still be misleading. In this step, you will create a cautious Markdown report that separates observations from evidence. It will connect manual review questions to OWASP ISVS and NISTIR 8259A themes without claiming a component identity or vulnerability.

In this step, get ready to:

• Map broad visual categories to cautious manual verification guidance.
• Combine visual observations with optional datasheet evidence in a structured report.
• Test uploaded images, webcam snapshots, prompt changes, threshold changes, and the visual-only path.

Map visual categories to review questions

A broad label such as a pin header is a visual lead. The matching guidance should ask what an authorized reviewer must verify before reaching a security conclusion.

• In app.py, locate _MODEL = None above DATASHEET_TOPICS.
• Add the visual guidance rules between those two sections by pasting this code:

GUIDANCE_RULES = [
    (
        ("integrated circuit", "microcontroller", "system on chip", "memory chip"),
        "Record the complete package marking. Confirm the exact part number in vendor documentation, then review boot-chain controls, debug controls, key storage, and lifecycle configuration.",
    ),
    (
        ("pin header", "test point"),
        "Treat this only as a possible internal interface. Compare pin count, silkscreen, schematics, and datasheet pin functions before any powered testing. Verify that production debug access is disabled or protected.",
    ),
    (
        ("sd card slot", "memory", "flash"),
        "Review whether firmware and sensitive data are protected at rest, whether updates are authenticated, and whether removable storage is covered by the device threat model.",
    ),
    (
        ("usb connector", "ethernet connector", "connector"),
        "Inventory the exposed interface and verify that logical access, boot-time behavior, services, and update paths are restricted to authorized use.",
    ),
    (
        ("crystal oscillator", "voltage regulator", "capacitor", "resistor"),
        "Document this as supporting circuitry. Do not infer a security property from appearance alone.",
    ),
]

What do these rules do?

• Each keyword group represents a broad visual category that the detector may return.
• Each guidance string asks for supporting evidence such as package markings, schematics, vendor documentation, or production configuration.
• Supporting circuitry receives an explicit reminder that appearance alone does not establish a security property.

The report needs one helper that selects the first guidance rule matching an observed label. Unknown labels receive a general verification question instead of an unsupported interpretation.

• In app.py, locate the end of inspect_datasheet().
• Add the label guidance helper directly below that function by pasting this code:

def guidance_for_label(label):
    label_lower = label.lower()
    for keywords, guidance in GUIDANCE_RULES:
        if any(keyword in label_lower for keyword in keywords):
            return guidance
    return (
        "Confirm the component category and exact part number from markings, schematics, "
        "and vendor documentation before drawing a security conclusion."
    )

How is guidance selected?

• The helper normalizes each detected label to lowercase before comparing it with the rule keywords.
• The first matching category supplies the manual verification guidance.
• The fallback keeps unfamiliar labels in the report while preserving uncertainty.

• Save app.py.
• Check that the new mapping parses successfully by running the app from the activated terminal:

python app.py

What does this check prove?

This command loads the updated Python file and starts the local server. A successful launch confirms that the rules and helper contain valid Python syntax.

You should see the local server address in the terminal. Your existing interface remains available at http://localhost:7860.

• Stop the server by pressing Ctrl+C in the integrated terminal.

Does the app stop before showing its address?

Check that GUIDANCE_RULES sits above DATASHEET_TOPICS. Confirm that every opening bracket has a matching closing bracket.

If the terminal points to the helper, compare the indentation inside guidance_for_label() with the snippet above. Help me debug the guidance mapping in app.py.

Build the structured report

The report first distinguishes visual observations from manual verification guidance. It records boxes and confidence scores as observations while reserving security conclusions for authorized follow-up work.

• In app.py, locate the temporary report-building section below draw_detections().
• Replace that section with the first part of build_report() by pasting this code:

def build_report(detections, datasheet_info):
    lines = [
        "# Security Triage Report",
        "",
        "> Authorized defensive use only. Visual detections and keyword matches are leads for manual verification, not proof of component identity, interface state, or vulnerability.",
        "",
        "## Visual observations",
    ]

    if detections:
        for item in detections:
            x0, y0, x1, y1 = item["box"]
            lines.append(
                f"- {item['label']} at confidence {item['score']:.2f}; "
                f"box [{x0}, {y0}, {x1}, {y1}]"
            )
    else:
        lines.append(
            "- No candidate passed the selected threshold. This does not show that the components are absent. Try clearer lighting, a closer image, different prompt wording, or a lower threshold."
        )

How are observations reported?

• The opening warning defines every detection and keyword match as a lead for manual verification.
• Accepted detections include their label, confidence score, and image coordinates.
• An empty result explains that the selected threshold may have filtered candidates without claiming that components are absent.

• Continue the same build_report() function by pasting this code directly below the previous segment:

    lines.extend(["", "## Manual verification guidance"])
    seen_guidance = set()
    if detections:
        for item in detections:
            guidance = guidance_for_label(item["label"])
            if guidance not in seen_guidance:
                lines.append(f"- {item['label']}: {guidance}")
                seen_guidance.add(guidance)
    else:
        lines.append(
            "- Begin with visible package markings, board silkscreen, schematics, and the manufacturer datasheet. Record uncertainty rather than assigning an interface or function from shape alone."
        )

    return "\n".join(lines)

Why remove repeated guidance?

Several detections can belong to the same broad category. The seen_guidance set keeps repeated advice from crowding the report while preserving each distinct review question.

• Save app.py.

Before you run the app, do you expect a detected pin header to appear as a vulnerability finding or as a lead for manual verification?

• Start the updated app from the activated terminal by running:

python app.py

What changes in this run?

The existing detector still produces the annotated image. The report now adds a warning, visual observations, and deduplicated manual verification guidance.

• Return to the browser tab from earlier at http://localhost:7860.
• Select a clear board photo in the Board image control.
• Click Run inspection.

You should see a red-box annotated image beside a report with visual observations and manual verification guidance. A detected label is presented as evidence to investigate, not as proof of a vulnerability.

• Stop the server by pressing Ctrl+C in the integrated terminal.

The datasheet status already contains page counts, extraction results, and matched excerpts. The next report section turns that structured status into evidence a reviewer can trace back to a page.

• In build_report(), locate return "\n".join(lines).
• Insert the first datasheet-reporting segment immediately above that return line:

    lines.extend(["", "## Datasheet evidence"])
    if not datasheet_info["uploaded"]:
        lines.append(
            "- No datasheet was uploaded. The report remains visual-only and requires vendor documentation before any conclusion."
        )
    elif datasheet_info["error"]:
        lines.append(f"- {datasheet_info['error']}")
    elif datasheet_info["pages_with_text"] == 0:
        lines.append(
            f"- The PDF has {datasheet_info['page_count']} page(s), but no extractable text was found. It may be scanned; OCR is outside this project's scope."
        )

How are missing documents handled?

• A missing datasheet produces a complete visual-only report.
• A parser problem is shown as evidence status without blocking the annotated image.
• A scanned document reports its page count and the absence of extractable text instead of inventing an excerpt.

• Continue the datasheet branch by pasting this code directly below the previous segment:

    else:
        lines.append(
            f"- Extracted text from {datasheet_info['pages_with_text']} of "
            f"{datasheet_info['page_count']} page(s)."
        )
        if datasheet_info["findings"]:
            for finding in datasheet_info["findings"]:
                lines.append(
                    f"- {finding['topic']}, page {finding['page']}, matched "
                    f"`{finding['term']}`: {finding['excerpt']}"
                )
                lines.append(f"  - Review question: {finding['question']}")
        else:
            lines.append(
                "- No configured review term was found. This is not evidence that the capability is absent; inspect the table of contents, security manual, reference manual, and errata manually."
            )

How does the report preserve traceability?

Each finding includes its review topic, source page, matched term, excerpt, and follow-up question. A document with no configured match receives a manual review path instead of an absence claim.

• Save app.py.

Before you run this version, which report detail should let you trace a matched review topic back to its source?

• Start the app from the activated terminal by running:

python app.py

What does this run verify?

This run checks that the report can combine detector results with the structured output from inspect_datasheet(). The image path remains independent from PDF extraction problems.

• Return to the browser tab at http://localhost:7860.
• Select a board photo in the Board image control.
• Select a text-based PDF in the Optional datasheet PDF control.
• Click Run inspection.

You should see the PDF page count in the report. Any configured topic match includes a page number, excerpt, and review question.

• Stop the server by pressing Ctrl+C in the integrated terminal.

Missing page-linked evidence?

Confirm that the selected document contains selectable text. A scanned image-only PDF can have pages while still producing no extractable text.

If text is available but no topic appears, search the document manually for the configured terms in DATASHEET_TOPICS. Help me troubleshoot missing datasheet findings.

Test the complete triage workflow

A useful triage report needs a consistent review frame after the raw evidence. The final section connects each result to interface control, boot integrity, data protection, secure updates, and cybersecurity state awareness.

• In build_report(), locate return "\n".join(lines).
• Insert the review framework directly above that return line by pasting this code:

    lines.extend(
        [
            "",
            "## Review framework",
            "- OWASP ISVS: verify that production debug interfaces are disabled or protected, boot stages are authenticated, sensitive data can use secure storage, and security configuration can be locked where supported.",
            "- NISTIR 8259A: review device identification, configuration control, data protection, logical access to interfaces, secure software update, and cybersecurity state awareness.",
            "",
            "## Required next action",
            "Confirm every visual lead against package markings, board schematics, the exact vendor datasheet, and the device's documented production configuration before performing any authorized hands-on test.",
        ]
    )

Why end with a required action?

The framework provides defensive review themes for organizing follow-up work. The required action sends every visual lead back to markings, schematics, exact vendor documentation, and production configuration before testing.

• Save app.py.

Before the final run, do you expect removing the PDF to remove the whole report or only its document-based evidence?

• Start the completed app from the activated terminal by running:

python app.py

What should the completed app produce?

The app now returns an annotated image and a report with visual observations, manual guidance, datasheet status, review themes, and a required next action. Each section preserves the difference between a lead and a verified security finding.

• Return to the browser tab at http://localhost:7860.
• Switch the Board image control to its camera source.
• Allow camera access in the browser permission prompt.
• Capture a clear snapshot of the authorized board.
• Clear the Optional datasheet PDF control.
• Enter microcontroller, pin header, usb connector in Component prompts, separated by commas.
• Set the Confidence threshold to 0.25.

Before you submit the snapshot, do you expect the report to remain complete without document evidence?

• Click Run inspection.

You should see an annotated webcam snapshot and a complete visual-only report. The datasheet section states that no PDF was uploaded. The report still includes visual observations, manual verification guidance, review themes, and the warning that detections do not prove identity, configuration, or vulnerabilities.

That completes the full input test. Your earlier run proved the uploaded-image and PDF path, while this run proves the webcam and no-PDF path with a different prompt set and threshold.

Camera snapshot not reaching the report?

Confirm that your browser has permission to use the intended camera. Recapture the image if the preview is empty or shows a different camera.

If the report shows no accepted candidate, keep the image path unchanged while adjusting prompt wording or the confidence threshold. Help me troubleshoot the webcam inspection path.

✔️ Awesome, I've got everything!

Your completed app.py now produces cautious visual-only or image-plus-datasheet triage reports. Keep the final server running while you capture your checkpoint.

ⓧ I'd like to double check the full code

Compare your completed app.py with this reference line by line.

from pathlib import Path

import gradio as gr
import pymupdf
import torch
from PIL import Image, ImageDraw
from transformers import AutoModelForZeroShotObjectDetection, AutoProcessor

MODEL_ID = "IDEA-Research/grounding-dino-tiny"
DEVICE = "cpu"
DEFAULT_PROMPTS = (
    "integrated circuit, microcontroller, system on chip, memory chip, "
    "pin header, test point, usb connector, ethernet connector, "
    "sd card slot, crystal oscillator, voltage regulator"
)

_PROCESSOR = None
_MODEL = None

GUIDANCE_RULES = [
    (
        ("integrated circuit", "microcontroller", "system on chip", "memory chip"),
        "Record the complete package marking. Confirm the exact part number in vendor documentation, then review boot-chain controls, debug controls, key storage, and lifecycle configuration.",
    ),
    (
        ("pin header", "test point"),
        "Treat this only as a possible internal interface. Compare pin count, silkscreen, schematics, and datasheet pin functions before any powered testing. Verify that production debug access is disabled or protected.",
    ),
    (
        ("sd card slot", "memory", "flash"),
        "Review whether firmware and sensitive data are protected at rest, whether updates are authenticated, and whether removable storage is covered by the device threat model.",
    ),
    (
        ("usb connector", "ethernet connector", "connector"),
        "Inventory the exposed interface and verify that logical access, boot-time behavior, services, and update paths are restricted to authorized use.",
    ),
    (
        ("crystal oscillator", "voltage regulator", "capacitor", "resistor"),
        "Document this as supporting circuitry. Do not infer a security property from appearance alone.",
    ),
]

DATASHEET_TOPICS = {
    "Debug access": {
        "terms": ["jtag", "swd", "uart", "debug interface", "debug port"],
        "question": "Does the documentation describe how debug access is disabled, authenticated, locked, or restricted in production?",
    },
    "Boot trust": {
        "terms": ["secure boot", "verified boot", "boot rom", "bootloader", "root of trust"],
        "question": "Does the documented boot chain verify each executable stage before it runs?",
    },
    "Storage and lifecycle state": {
        "terms": ["spi flash", "flash memory", "emmc", "otp", "efuse", "one-time programmable"],
        "question": "What data can be stored externally, and can security configuration or lifecycle state be locked?",
    },
    "Cryptography and key protection": {
        "terms": ["trustzone", "key storage", "cryptographic", "encryption", "security engine"],
        "question": "Which hardware-backed protections are documented, and which require explicit firmware configuration?",
    },
    "Firmware updates": {
        "terms": ["firmware update", "signed firmware", "image authentication", "rollback protection"],
        "question": "Does the documentation describe update authentication, authorization, and rollback behavior?",
    },
}


def get_model():
    global _PROCESSOR, _MODEL
    if _PROCESSOR is None or _MODEL is None:
        _PROCESSOR = AutoProcessor.from_pretrained(MODEL_ID)
        _MODEL = AutoModelForZeroShotObjectDetection.from_pretrained(MODEL_ID)
        _MODEL.to(DEVICE)
        _MODEL.eval()
    return _PROCESSOR, _MODEL


def normalize_prompts(prompt_text):
    prompts = [
        item.strip().lower().rstrip(".")
        for item in (prompt_text or "").split(",")
        if item.strip()
    ]
    if not prompts:
        prompts = [item.strip() for item in DEFAULT_PROMPTS.split(",")]
    return prompts


def excerpt_around(text, term, radius=120):
    collapsed = " ".join(text.split())
    index = collapsed.lower().find(term.lower())
    if index < 0:
        return ""
    start = max(0, index - radius)
    end = min(len(collapsed), index + len(term) + radius)
    prefix = "..." if start else ""
    suffix = "..." if end < len(collapsed) else ""
    return prefix + collapsed[start:end] + suffix


def inspect_datasheet(pdf_path):
    info = {
        "uploaded": False,
        "page_count": 0,
        "pages_with_text": 0,
        "findings": [],
        "error": "",
    }
    if not pdf_path:
        return info

    info["uploaded"] = True
    if Path(pdf_path).suffix.lower() != ".pdf":
        info["error"] = "The optional document must be a PDF."
        return info

    try:
        topics_seen = set()
        with pymupdf.open(pdf_path) as document:
            info["page_count"] = document.page_count
            for page_number, page in enumerate(document, start=1):
                text = page.get_text("text")
                if not text.strip():
                    continue
                info["pages_with_text"] += 1
                lowered = text.lower()
                for topic, details in DATASHEET_TOPICS.items():
                    if topic in topics_seen:
                        continue
                    matched_term = next(
                        (term for term in details["terms"] if term in lowered),
                        None,
                    )
                    if matched_term:
                        info["findings"].append(
                            {
                                "topic": topic,
                                "page": page_number,
                                "term": matched_term,
                                "excerpt": excerpt_around(text, matched_term),
                                "question": details["question"],
                            }
                        )
                        topics_seen.add(topic)
    except Exception as exc:
        info["error"] = f"Could not read the PDF: {exc}"

    return info


def guidance_for_label(label):
    label_lower = label.lower()
    for keywords, guidance in GUIDANCE_RULES:
        if any(keyword in label_lower for keyword in keywords):
            return guidance
    return (
        "Confirm the component category and exact part number from markings, schematics, "
        "and vendor documentation before drawing a security conclusion."
    )


def draw_detections(image, result):
    annotated = image.copy()
    draw = ImageDraw.Draw(annotated)
    detections = []
    width, height = annotated.size

    for box, score, text_label in zip(
        result["boxes"], result["scores"], result["text_labels"]
    ):
        x0, y0, x1, y1 = [int(round(value)) for value in box.tolist()]
        x0 = max(0, min(width - 1, x0))
        y0 = max(0, min(height - 1, y0))
        x1 = max(0, min(width - 1, x1))
        y1 = max(0, min(height - 1, y1))
        if x1 <= x0 or y1 <= y0:
            continue

        label = str(text_label)
        confidence = float(score.item())
        caption = f"{label}: {confidence:.2f}"
        draw.rectangle((x0, y0, x1, y1), outline="red", width=3)
        draw.text((x0 + 4, y0 + 4), caption, fill="red", font_size=16)
        detections.append(
            {
                "label": label,
                "score": confidence,
                "box": [x0, y0, x1, y1],
            }
        )

    detections.sort(key=lambda item: item["score"], reverse=True)
    return annotated, detections


def build_report(detections, datasheet_info):
    lines = [
        "# Security Triage Report",
        "",
        "> Authorized defensive use only. Visual detections and keyword matches are leads for manual verification, not proof of component identity, interface state, or vulnerability.",
        "",
        "## Visual observations",
    ]

    if detections:
        for item in detections:
            x0, y0, x1, y1 = item["box"]
            lines.append(
                f"- {item['label']} at confidence {item['score']:.2f}; "
                f"box [{x0}, {y0}, {x1}, {y1}]"
            )
    else:
        lines.append(
            "- No candidate passed the selected threshold. This does not show that the components are absent. Try clearer lighting, a closer image, different prompt wording, or a lower threshold."
        )

    lines.extend(["", "## Manual verification guidance"])
    seen_guidance = set()
    if detections:
        for item in detections:
            guidance = guidance_for_label(item["label"])
            if guidance not in seen_guidance:
                lines.append(f"- {item['label']}: {guidance}")
                seen_guidance.add(guidance)
    else:
        lines.append(
            "- Begin with visible package markings, board silkscreen, schematics, and the manufacturer datasheet. Record uncertainty rather than assigning an interface or function from shape alone."
        )

    lines.extend(["", "## Datasheet evidence"])
    if not datasheet_info["uploaded"]:
        lines.append(
            "- No datasheet was uploaded. The report remains visual-only and requires vendor documentation before any conclusion."
        )
    elif datasheet_info["error"]:
        lines.append(f"- {datasheet_info['error']}")
    elif datasheet_info["pages_with_text"] == 0:
        lines.append(
            f"- The PDF has {datasheet_info['page_count']} page(s), but no extractable text was found. It may be scanned; OCR is outside this project's scope."
        )
    else:
        lines.append(
            f"- Extracted text from {datasheet_info['pages_with_text']} of "
            f"{datasheet_info['page_count']} page(s)."
        )
        if datasheet_info["findings"]:
            for finding in datasheet_info["findings"]:
                lines.append(
                    f"- {finding['topic']}, page {finding['page']}, matched "
                    f"`{finding['term']}`: {finding['excerpt']}"
                )
                lines.append(f"  - Review question: {finding['question']}")
        else:
            lines.append(
                "- No configured review term was found. This is not evidence that the capability is absent; inspect the table of contents, security manual, reference manual, and errata manually."
            )

    lines.extend(
        [
            "",
            "## Review framework",
            "- OWASP ISVS: verify that production debug interfaces are disabled or protected, boot stages are authenticated, sensitive data can use secure storage, and security configuration can be locked where supported.",
            "- NISTIR 8259A: review device identification, configuration control, data protection, logical access to interfaces, secure software update, and cybersecurity state awareness.",
            "",
            "## Required next action",
            "Confirm every visual lead against package markings, board schematics, the exact vendor datasheet, and the device's documented production configuration before performing any authorized hands-on test.",
        ]
    )
    return "\n".join(lines)


def inspect_board(image, datasheet_pdf, prompt_text, confidence_threshold):
    if image is None:
        return None, "# Input needed\n\nUpload a board image or take a webcam snapshot."

    image = image.convert("RGB")
    prompts = normalize_prompts(prompt_text)
    processor, model = get_model()
    inputs = processor(images=image, text=[prompts], return_tensors="pt")

    with torch.no_grad():
        outputs = model(**inputs)

    result = processor.post_process_grounded_object_detection(
        outputs,
        inputs.input_ids,
        threshold=float(confidence_threshold),
        text_threshold=0.25,
        target_sizes=[image.size[::-1]],
    )[0]

    annotated, detections = draw_detections(image, result)
    datasheet_info = inspect_datasheet(datasheet_pdf)
    report = build_report(detections, datasheet_info)
    return annotated, report


with gr.Blocks(title="Embedded Board Security Triage Assistant") as demo:
    gr.Markdown(
        "# Embedded Board Security Triage Assistant\n"
        "Upload a board image or take a camera snapshot. Optionally add a text-based "
        "datasheet PDF. Results are review leads, not vulnerability findings."
    )

    with gr.Row():
        with gr.Column():
            image_input = gr.Image(
                sources=["upload", "webcam"],
                type="pil",
                label="Board image",
            )
            datasheet_input = gr.File(
                file_count="single",
                file_types=[".pdf"],
                type="filepath",
                label="Optional datasheet PDF",
            )
            prompt_input = gr.Textbox(
                value=DEFAULT_PROMPTS,
                lines=3,
                label="Component prompts, separated by commas",
            )
            threshold_input = gr.Slider(
                minimum=0.10,
                maximum=0.90,
                value=0.35,
                step=0.05,
                label="Confidence threshold",
            )
            inspect_button = gr.Button("Run inspection", variant="primary")

        with gr.Column():
            image_output = gr.Image(type="pil", label="Annotated board")
            report_output = gr.Markdown()

    inspect_button.click(
        fn=inspect_board,
        inputs=[image_input, datasheet_input, prompt_input, threshold_input],
        outputs=[image_output, report_output],
    )


if __name__ == "__main__":
    demo.launch()

How should you use this reference?

Check function order, indentation, literal report text, and the final call to build_report(). The complete file should match this reference exactly.

Secret mission

Calibrate the Detector for Your Board Family

Test the detector across three authorized photos from one board family. Compare prompt wording plus confidence thresholds to create a measured baseline for future custom training labels.

Clean Up Your Resources

Clean Up Your Resources

Everything in this project runs on your Windows computer. No paid cloud service is running, so keeping these resources creates no ongoing cloud charge.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

• The local Gradio process serving the app at http://localhost:7860.
• The embedded-board-triage folder containing app.py, requirements.txt, calibration-notes.md, and the tutorial-env virtual environment.
• The local Hugging Face model cache containing IDEA-Research/grounding-dino-tiny.

Keep everything running

No action needed. Choose this if you want to keep inspecting authorized boards or refining your calibration notes.

• Keep the embedded-board-triage folder so your app, virtual environment, dependency pins, and calibration record remain available.
• Leave the local server process running if you want the app to remain available at http://localhost:7860.
• Keep the downloaded model cache so future inspection runs can reuse the local model files.

Pause - I'll come back to this later

Shut down the local server to free its memory. Your project folder and downloaded model remain ready for your next session.

• Switch back to the Visual Studio Code integrated terminal from earlier.
• Press Ctrl+C to stop the local server.

The page at http://localhost:7860 stops responding once the server has closed.

• Return to the integrated terminal in embedded-board-triage when you want to resume.
• Reactivate tutorial-env by running this command:

tutorial-env\Scripts\activate

What does this command do?

This reconnects the terminal to the isolated environment containing the project's pinned packages.

• Start the local app by running this command:

python app.py

What does this command do?

This launches app.py with the dependencies from tutorial-env. Your local interface becomes available again at http://localhost:7860.

Delete - I don't want to use this again

Deleting these local resources is permanent. Your completed app and calibration record disappear with the project folder.

• Switch back to the VS Code integrated terminal from earlier.
• Press Ctrl+C if the local server is still running.

The local page stops responding after the server process closes.

• Locate the embedded-board-triage folder in Windows File Explorer using the location shown in VS Code.
• Delete the embedded-board-triage folder.

You should no longer see embedded-board-triage in that File Explorer location. This removes app.py, requirements.txt, calibration-notes.md, and tutorial-env.

The model cache is stored outside the project folder. Removing it releases the disk space used by the downloaded model files.

• List the cached repositories by running this command:

hf cache ls

What does this command do?

This lists the repositories currently stored in your local Hugging Face cache. Use the output to identify the entry for IDEA-Research/grounding-dino-tiny.

Cache removal is selective. The workflow lets you choose the model entry before permanently deleting its cached files.

• Start the documented cache removal workflow by running this command:
• Select the cached entry for IDEA-Research/grounding-dino-tiny when the workflow displays the available entries.
• Confirm the permanent removal when the workflow asks.

hf cache rm

What does this command do?

This removes the repository or revision you select from the local model cache. Selecting only this project's model preserves unrelated cached models.

• Verify the cache cleanup by running the listing command again:

hf cache ls

What does this command do?

This refreshes the cache listing after deletion. It gives you a visible check that the selected model entry is gone.

You should no longer see IDEA-Research/grounding-dino-tiny in the cache listing. Cleanup complete. Your project folder is gone. The downloaded model files are gone too.

Still See the Cached Model?

Run the removal workflow again. Make sure the selected cache entry matches IDEA-Research/grounding-dino-tiny before confirming deletion.

Help me remove the cached Grounding DINO Tiny model.

Nice Work!

Nice Work!

You did it! Your local CPU-only assistant now turns authorized board images plus optional datasheets into cautious security review leads.

What you learned:

• Built a local Gradio interface for board images from uploads or webcam snapshots. Added optional PDF datasheet input.
• Ran Grounding DINO Tiny on CPU for zero-shot object detection. Drew bounding boxes with labels plus confidence scores using Pillow.
• Extracted page-linked datasheet evidence with PyMuPDF. Produced defensive review questions that preserve uncertainty.
• Secret Mission: Calibrated prompt wording plus a confidence threshold across three photos from one board family. Documented correct detections. Captured false positives plus misses as candidates for a future custom YOLO dataset.

Ready to quiz yourself?