Build a Scalable Sign Recognizer

Build a sign recognizer for recorded videos and live webcam input.

Introduction

30 Second Summary

Two hand signs can pause in a similar pose even when their movements mean different things. Judging either clip from one frozen moment can hide the difference.

In this project, you will build a local command-line isolated-sign recognizer for five validated Indian Sign Language categories. Your recognizer will compare 32-frame hand-landmark sequences within an ISL namespace to return a known sign ID or an honest UNKNOWN result.

What You'll Build

You will run a prerecorded clip through your recognizer to see its 32 x 128 feature sequence become ISL:SIGN_03 or an honest UNKNOWN result in the terminal.

By the end of this project, you'll have:

  • A visible landmark pipeline that reports a 32 x 128 sequence from a prerecorded clip using MediaPipe plus OpenCV.
  • A reusable reference index that classifies a held-out clip among five configured ISL sign IDs using all 32 sampled frames.
  • A language-aware result that prints a qualified label for a close match or UNKNOWN when the clip exceeds the calibrated threshold.
  • Secret Mission: Add a sixth validated ISL sign through data changes alone without editing the Python files.

Are there any prerequisites?

You need a Linux computer with Python already available plus some past machine learning experience.

Prepare five validated ISL sign categories with at least two reference clips per category, one held-out query clip, and one non-catalog clip.

Each clip must show the hands clearly and have documented permission or provenance.

Before We Start

This checkpoint commits you to a hand-landmark prototype for isolated signs. You will also identify five validated ISL categories with documented video provenance before any hands-on work begins.

Set Up the Local Vision Workspace

Your validated clips need one controlled workspace before they enter the vision pipeline. Visual Studio Code keeps the project files visible while you work.

A Python virtual environment isolates the packages this recognizer needs. A local model bundle gives the video pipeline a verified asset before you write recognition code.

In this step, get ready to:
  • Prepare Visual Studio Code with the project folder open.
  • Create .venv with the pinned vision packages.
  • Prepare the local model asset and language-namespaced data directories.
Install Visual Studio Code and open the workspace

The editor gives your source files and terminal one shared workspace. First, check whether it is already available on your Linux system.

  • Press the Super key to open your Linux application launcher.
  • Type Terminal into the search field.
  • Select Terminal from the results.
  • Check for Visual Studio Code by running this command:
code --version

What does this check do?

The command asks Visual Studio Code to print its version information. A working installation prints a version number followed by its commit ID and architecture.

✔️ I see version information

Good start. Visual Studio Code is installed and its command-line launcher is available.

ⓧ Command not found

Visual Studio Code provides Linux packages for Debian-based distributions, RPM-based distributions, and systems using Snap.

  • Open the official Visual Studio Code Linux setup guide in your browser.
  • Select the installation instructions for your Linux distribution.
  • Complete the installation using the package format supported by your distribution.
  • Return to the terminal after the installation finishes.
  • Confirm the command-line launcher by running the check again:
code --version

What confirms the installation?

The command prints the installed Visual Studio Code version. It also confirms that code is available from your terminal.

Still seeing command not found?

Close the current terminal after the installation. Open a fresh terminal so it can load the updated command path.

Confirm that you followed the instructions for your distribution. Help me troubleshoot the Visual Studio Code command on Linux.

Visual Studio Code is ready. Next, give the recognizer a dedicated folder on your Desktop.

  • Press the Super key to open your Linux application launcher.
  • Type Visual Studio Code into the search field.
  • Select Visual Studio Code from the results.
  • Click Terminal in the top menu bar.
  • Click New Terminal.
  • Move to your Desktop by running this command:
cd ~/Desktop

What does this command do?

The command moves the integrated terminal into your Desktop directory. The project folder you create next stays easy to find.

  • Create the isolated-sign-recognizer folder by running these commands:
mkdir isolated-sign-recognizer
cd isolated-sign-recognizer

What do these commands do?

  • The first command creates the isolated-sign-recognizer folder on your Desktop.
  • The second command moves the terminal into that folder.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the isolated-sign-recognizer folder from your Desktop.
  • Click Open.

You should see isolated-sign-recognizer at the top of the Explorer sidebar. Your editor and project folder now point to the same workspace.

Can't see the project folder?

Confirm that you selected the folder from your Desktop. The Explorer heading should show isolated-sign-recognizer.

If the folder is missing, return to the integrated terminal and confirm that the folder creation command completed. Help me find my Visual Studio Code project folder.

Create the virtual environment and install packages

The package pin depends on the Python version already installed on your Linux system. MediaPipe 1.1.1 supports Python 3.9 or later.

OpenCV 5.0.0.93 supplies video decoding. NumPy supplies the array operations used by the landmark sequence.

  • Click Terminal in the top menu bar.
  • Click New Terminal.
  • Check the installed Python version by running this command:
python3 --version

Why check the version first?

The version decides which NumPy wheel is compatible with your interpreter. The two paths below install the same MediaPipe and OpenCV versions.

Choose the tab that matches the major and minor version printed in your terminal.

Python 3.12 or higher

Python 3.12 or higher can use NumPy 2.5.4.

The first package download can take a few minutes. Terminal activity during the installation means the packages are still being prepared.

  • Create .venv with the compatible package set by running these commands:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install mediapipe==1.1.1 opencv-contrib-python==5.0.0.93 numpy==2.5.4

What do these commands set up?

  • The first command creates an isolated Python installation inside .venv.
  • The second command makes that environment's Python tools active in the terminal.
  • The final command installs MediaPipe 1.1.1.
  • The same command installs OpenCV 5.0.0.93 with NumPy 2.5.4.

The installer returns to the prompt after resolving the three pinned packages. Your terminal prompt should also begin with (.venv).

Package installation not completing?

Check that the terminal prompt begins with (.venv). Run the activation command again if that prefix is missing.

Confirm that your Python version is 3.12 or higher before using the NumPy 2.5.4 pin. Help me diagnose this package installation.

Python 3.9 to 3.11

Python 3.9 through 3.11 uses the NumPy 2.0.2 compatibility pin.

The first package download can take a few minutes. Terminal activity during the installation means the packages are still being prepared.

  • Create .venv with the compatible package set by running these commands:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install mediapipe==1.1.1 opencv-contrib-python==5.0.0.93 numpy==2.0.2

What do these commands set up?

  • The first command creates an isolated Python installation inside .venv.
  • The second command makes that environment's Python tools active in the terminal.
  • The final command installs MediaPipe 1.1.1.
  • The same command installs OpenCV 5.0.0.93 with NumPy 2.0.2.

The installer returns to the prompt after resolving the three pinned packages. Your terminal prompt should also begin with (.venv).

Package installation not completing?

Check that the terminal prompt begins with (.venv). Run the activation command again if that prefix is missing.

Confirm that your Python version is between 3.9 and 3.11 before using the NumPy 2.0.2 pin. Help me diagnose this package installation.

Is your Python version below 3.9?

This package set requires Python 3.9 or later. Follow the official Python guidance for Unix systems before creating .venv.

After the newer interpreter is available, repeat the version check and use the matching package tab. Help me choose a supported Python installation path.

Create the data directories and download the model

The recognizer loads its model from models/hand_landmarker.task. The data/ISL/ path keeps the validated clips inside their language namespace.

  • Create the model and ISL data directories by running this command in the activated terminal:
mkdir -p models data/ISL

What does this command create?

  • The models directory stores the Hand Landmarker bundle and the reference index created later.
  • The data/ISL directory provides the language-specific home for your sign-video folders.
  • The -p option creates the parent data directory when it is missing.
  • Click the Refresh Explorer icon in the Explorer toolbar.
  • Confirm that models appears in the Explorer sidebar.
  • Expand data in the Explorer sidebar.
  • Confirm that ISL appears inside data.

Directories missing from Explorer?

Confirm that the integrated terminal is inside isolated-sign-recognizer before running the directory command.

Refresh the Explorer sidebar after the command completes. Help me verify these project directories.

The model bundle contains the parameters used to locate hand landmarks in video frames. Keeping it inside models gives the extractor a stable local path.

  • Download the official model bundle into the project by running these commands:
wget -q https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task
mv hand_landmarker.task models/hand_landmarker.task

What do these commands download?

  • The first command downloads the official Hand Landmarker model bundle into the current project folder.
  • The -q option keeps the download quiet.
  • The second command moves the bundle to models/hand_landmarker.task.
  • Click the Refresh Explorer icon in the Explorer toolbar.
  • Expand the models directory.
  • Confirm that hand_landmarker.task appears inside models.

Model file not appearing?

Check that the file name does not have an extra download extension. Help me fix the model download.

Before you run the final check, where do you expect pip to report that it is installed?

  • Verify the active package installer by running this command:
pip -V

What does this final check prove?

The command prints the active pip version and its installation path. A path containing .venv proves that package commands are using the isolated project environment.

You should see a path containing .venv in the terminal output. That is the environment boundary that protects the vision package pins.

Does the path omit .venv?

Run the activation command from the isolated-sign-recognizer folder. The terminal prompt should begin with (.venv) afterward.

Run the final check again after activation. Help me activate the correct virtual environment.

Your editor, package environment, data directories, and local model asset are ready. Next, you will turn one validated sign video into a fixed-length landmark sequence.

Extract Landmark Sequences

Your local vision workspace is ready. The Hand Landmarker model can now turn video frames into hand landmarks.

Raw sign videos can have different durations. The recognizer needs a compact sequence with the same shape for every clip.

OpenCV decodes each frame. MediaPipe tracks up to two hands while NumPy stores the resulting features.

In this step, get ready to:
  • Prepare one validated reference video for extraction.
  • Convert detected hands into stable left and right feature slots.
  • Resample the clip into a fixed 32-frame sequence.
Prepare the reference clip and extractor

A reference clip needs a stable catalog path before the extractor can read it. Its folder identifies the language namespace plus the sign ID.

  • Expand the existing data/ISL/ directory in the VS Code Explorer sidebar.
  • Select the data/ISL/ directory.
  • Click the New Folder icon in the Explorer toolbar.
  • Enter SIGN_01 as the folder name.
  • Drag your validated reference video from the Linux file manager into SIGN_01.
  • Rename the copied video to ref-01.mp4 using the Explorer rename control.

The Explorer now shows data/ISL/SIGN_01/ref-01.mp4. This path gives the extractor a known verification clip.

  • Select the top-level project folder in the Explorer sidebar.
  • Click the New File icon in the Explorer toolbar.
  • Enter feature_extractor.py as the file name.
  • Add the module dependencies plus feature dimensions by pasting this code:
import argparse
from pathlib import Path

import cv2
import mediapipe as mp
import numpy as np
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

SEQUENCE_LENGTH = 32
HAND_FEATURES = 63
FRAME_FEATURES = (HAND_FEATURES * 2) + 2

What does this setup define?

  • The imports provide command-line parsing plus video decoding plus landmark detection plus numerical arrays.
  • Each hand contributes 63 values from 21 landmarks with three coordinates each.
  • Two hand slots plus two presence flags produce 128 values per frame.
  • Every finished sequence contains 32 sampled frames.
  • Save feature_extractor.py.
  • Confirm the file appears beside the existing models/ directory in the Explorer sidebar.

Seeing unresolved imports?

Confirm the integrated terminal still shows the active .venv environment. Select the interpreter from that environment if the editor is using a different Python installation.

The installed packages provide cv2 plus mediapipe plus numpy. Help me connect VS Code to my existing virtual environment.

The extractor validates its model path before processing any video. It also creates a fresh video-mode landmarker for each extraction.

  • Append the extractor class plus model setup to feature_extractor.py by pasting this code:
class HandSequenceExtractor:
    def __init__(self, model_path: Path):
        self.model_path = Path(model_path)
        if not self.model_path.is_file():
            raise FileNotFoundError(f"Model not found: {self.model_path}")

    def _create_landmarker(self):
        options = vision.HandLandmarkerOptions(
            base_options=python.BaseOptions(
                model_asset_path=str(self.model_path)
            ),
            running_mode=vision.RunningMode.VIDEO,
            num_hands=2,
        )
        return vision.HandLandmarker.create_from_options(options)

How is the landmarker configured?

  • The constructor stores the local model path after converting it to a Path object.
  • The file check stops execution when the model bundle cannot be found.
  • Video mode accepts decoded frames with increasing timestamps.
  • The num_hands setting allows two hands in each frame.
  • Save feature_extractor.py.
  • Confirm the class body is indented beneath HandSequenceExtractor in the editor.

Model path check failing?

Confirm hand_landmarker.task remains inside the existing models/ directory. The default path is added to the command-line interface later in this step.

Check that the class methods use four spaces of indentation. Help me inspect the model setup in my extractor.

Landmark coordinates vary with hand location plus distance from the camera. Wrist-relative coordinates preserve the wrist position while scaling finger geometry consistently.

  • Append the hand-normalization method inside HandSequenceExtractor by pasting this code:
    @staticmethod
    def _normalise_hand(landmarks):
        points = np.stack(
            [[landmark.x, landmark.y, landmark.z] for landmark in landmarks]
        )
        wrist = points[0]
        scale = float(np.linalg.norm(points[9] - wrist))
        if scale < 0.000001:
            scale = 1.0

        features = [float(wrist[0]), float(wrist[1]), float(wrist[2])]
        for point in points[1:]:
            relative = (point - wrist) / scale
            features.extend(float(value) for value in relative)
        return features

What does normalization preserve?

  • The wrist remains as three absolute image coordinates.
  • The remaining 20 landmarks become wrist-relative coordinates.
  • The wrist-to-middle-finger distance supplies a scale for hand size.
  • The tiny-scale fallback prevents division by a value near zero.
  • Save feature_extractor.py.
  • Confirm _normalise_hand() aligns with _create_landmarker() inside the class.

Normalization method outside the class?

Check that the decorator begins with four spaces. Every line inside _normalise_hand() needs another four spaces.

Keep the method directly below _create_landmarker(). Help me fix this class indentation.

Build stable temporal features

Each frame needs fixed positions for left-hand data plus right-hand data. Presence flags distinguish a missing hand from landmarks whose numeric values happen to be zero.

  • Append the frame-feature method inside HandSequenceExtractor by pasting this code:
    @classmethod
    def _frame_features(cls, result):
        left = [0.0] * HAND_FEATURES
        right = [0.0] * HAND_FEATURES
        present = [0.0, 0.0]

        for landmarks, handedness in zip(
            result.hand_landmarks, result.handedness
        ):
            category_name = handedness[0].category_name or ""
            label = category_name.lower()
            features = cls._normalise_hand(landmarks)

            if label == "left":
                left = features
                present[0] = 1.0
            elif label == "right":
                right = features
                present[1] = 1.0
            elif present[0] == 0.0:
                left = features
                present[0] = 1.0
            else:
                right = features
                present[1] = 1.0

        return left + right + present

How do the hand slots stay stable?

  • Each frame starts with zero-filled left plus right slots.
  • The handedness label chooses the matching slot for detected landmarks.
  • The fallback fills the first available slot when a label is unavailable.
  • The returned list always contains two 63-value slots plus two flags.
  • Save feature_extractor.py.
  • Confirm the final return expression contains left + right + present.

Frame feature length looks inconsistent?

Confirm both empty hand slots multiply by HAND_FEATURES. Check that present contains exactly two values.

Keep every handedness branch inside the loop. Help me verify my 128-value frame layout.

Video clips can contain different numbers of decoded frames. Resampling selects 32 evenly distributed positions while keeping the first plus last frames.

  • Append the resampling method inside HandSequenceExtractor by pasting this code:
    @staticmethod
    def _resample(sequence):
        if len(sequence) == 1:
            return np.stack(
                [sequence[0] for _ in range(SEQUENCE_LENGTH)]
            )

        indices = [
            round(index * (len(sequence) - 1) / (SEQUENCE_LENGTH - 1))
            for index in range(SEQUENCE_LENGTH)
        ]
        return np.stack([sequence[index] for index in indices])

How does resampling work?

  • A one-frame clip repeats that frame until the sequence reaches the target length.
  • Longer clips map 32 evenly spaced positions across the available frame range.
  • The final stack has one row for each sampled frame.
  • Save feature_extractor.py.
  • Confirm _resample() returns a NumPy stack in both branches.

Resampling code showing an indentation problem?

Align @staticmethod with the previous method decorators. Keep the index calculation inside the method body.

Check every opening bracket for a matching closing bracket. Help me inspect the resampling method.

The extraction method connects video decoding to landmark detection. It also prepares monotonically increasing timestamps for video mode.

  • Start the extraction method inside HandSequenceExtractor by pasting this code:
    def extract(self, video_path: Path):
        video_path = Path(video_path)
        capture = cv2.VideoCapture(str(video_path))
        if not capture.isOpened():
            raise RuntimeError(f"Could not open video: {video_path}")

        fps = capture.get(cv2.CAP_PROP_FPS)
        if fps <= 0:
            fps = 30.0

        frames = []
        frame_index = 0
        previous_timestamp = -1

What does the video setup prepare?

  • OpenCV receives the reference video path.
  • The open check stops processing when the file cannot be decoded.
  • A fallback frame rate keeps timestamp calculations valid when metadata is missing.
  • The frame list plus counters track extraction progress.
  • Save feature_extractor.py.
  • Confirm the new method remains indented inside HandSequenceExtractor.

Video setup outside the class?

Align def extract with the other method definitions. Its body needs eight spaces.

Keep previous_timestamp inside the method. Help me repair the extraction method structure.

Every decoded frame is converted from OpenCV's color order to the RGB format expected by MediaPipe. The timestamp increases before each video-mode detection call.

  • Continue the extract() method directly below previous_timestamp = -1 by pasting this code:
        try:
            with self._create_landmarker() as landmarker:
                while True:
                    success, frame = capture.read()
                    if not success:
                        break

                    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                    mp_image = mp.Image(
                        image_format=mp.ImageFormat.SRGB,
                        data=rgb_frame,
                    )
                    timestamp = max(
                        previous_timestamp + 1,
                        int(frame_index * 1000 / fps),
                    )
                    result = landmarker.detect_for_video(
                        mp_image, timestamp
                    )
                    frames.append(self._frame_features(result))
                    previous_timestamp = timestamp
                    frame_index += 1
        finally:
            capture.release()

What happens to each frame?

  • The loop reads frames until the video ends.
  • Each BGR frame becomes an RGB MediaPipe image.
  • The timestamp stays greater than the previous timestamp.
  • Each detection result becomes one fixed-length frame vector.
  • The finally block releases the video even when processing fails.
  • Save feature_extractor.py.
  • Confirm the finally block aligns with the matching try block.

Loop structure showing a syntax error?

Check the indentation beneath while True plus try plus with. The finally line must align with try.

Check the commas inside the multi-line image plus timestamp calls. Help me fix the video-processing loop.

A decoded clip still needs usable hand detections. The final checks reject empty videos plus clips where every frame lacks a detected hand.

  • Finish the extract() method directly below capture.release() by pasting this code:
        if not frames:
            raise RuntimeError(f"No frames were decoded from: {video_path}")
        if not any(frame[-2] or frame[-1] for frame in frames):
            raise RuntimeError(f"No hands were detected in: {video_path}")

        return self._resample(np.stack(frames))

Why validate before resampling?

  • An empty frame list means OpenCV decoded no usable frames.
  • The final two values in each frame show whether either hand was present.
  • A valid frame list becomes one 32-frame NumPy sequence.
  • Save feature_extractor.py.
  • Confirm the final return statement remains inside extract().

Validation checks in the wrong scope?

Place both if checks at the same indentation level as try. They must remain inside extract().

Keep the return statement below both checks. Help me verify the end of my extract method.

Run the 32-frame extraction

The command-line interface connects the default model plus a video argument to the extractor. It prints the sequence dimensions plus the number of resampled frames containing hands.

  • Append the command-line interface below the HandSequenceExtractor class by pasting this code:
def main():
    parser = argparse.ArgumentParser(
        description="Extract a fixed-length hand-landmark sequence."
    )
    parser.add_argument("video", type=Path)
    parser.add_argument(
        "--model",
        type=Path,
        default=Path("models/hand_landmarker.task"),
    )
    args = parser.parse_args()

    extractor = HandSequenceExtractor(args.model)
    sequence = extractor.extract(args.video)
    detected_frames = sum(
        1 for frame in sequence if frame[-2] or frame[-1]
    )

    print(f"Feature shape: {len(sequence)} x {len(sequence[0])}")
    print(
        f"Detected hands in {detected_frames}/{len(sequence)} "
        "resampled frames"
    )


if __name__ == "__main__":
    main()

What does the command-line interface report?

  • The positional video argument identifies the clip to extract.
  • The optional model argument defaults to models/hand_landmarker.task.
  • The detected-frame count reads the two presence flags from each sampled frame.
  • The module guard runs main() when the file is launched as a script.
  • Save feature_extractor.py.
  • Confirm main() begins outside the class indentation.

Command-line interface nested inside the class?

Place def main at the far left edge of the file. The module guard also starts at the far left edge.

Keep two blank lines between the class plus main(). Help me check the final file structure.

✔️ Awesome, I've got everything!

Great. Double-check that you saved feature_extractor.py before running it.

ⓧ I'd like to double check the full code

import argparse
from pathlib import Path

import cv2
import mediapipe as mp
import numpy as np
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

SEQUENCE_LENGTH = 32
HAND_FEATURES = 63
FRAME_FEATURES = (HAND_FEATURES * 2) + 2


class HandSequenceExtractor:
    def __init__(self, model_path: Path):
        self.model_path = Path(model_path)
        if not self.model_path.is_file():
            raise FileNotFoundError(f"Model not found: {self.model_path}")

    def _create_landmarker(self):
        options = vision.HandLandmarkerOptions(
            base_options=python.BaseOptions(
                model_asset_path=str(self.model_path)
            ),
            running_mode=vision.RunningMode.VIDEO,
            num_hands=2,
        )
        return vision.HandLandmarker.create_from_options(options)

    @staticmethod
    def _normalise_hand(landmarks):
        points = np.stack(
            [[landmark.x, landmark.y, landmark.z] for landmark in landmarks]
        )
        wrist = points[0]
        scale = float(np.linalg.norm(points[9] - wrist))
        if scale < 0.000001:
            scale = 1.0

        features = [float(wrist[0]), float(wrist[1]), float(wrist[2])]
        for point in points[1:]:
            relative = (point - wrist) / scale
            features.extend(float(value) for value in relative)
        return features

    @classmethod
    def _frame_features(cls, result):
        left = [0.0] * HAND_FEATURES
        right = [0.0] * HAND_FEATURES
        present = [0.0, 0.0]

        for landmarks, handedness in zip(
            result.hand_landmarks, result.handedness
        ):
            category_name = handedness[0].category_name or ""
            label = category_name.lower()
            features = cls._normalise_hand(landmarks)

            if label == "left":
                left = features
                present[0] = 1.0
            elif label == "right":
                right = features
                present[1] = 1.0
            elif present[0] == 0.0:
                left = features
                present[0] = 1.0
            else:
                right = features
                present[1] = 1.0

        return left + right + present

    @staticmethod
    def _resample(sequence):
        if len(sequence) == 1:
            return np.stack(
                [sequence[0] for _ in range(SEQUENCE_LENGTH)]
            )

        indices = [
            round(index * (len(sequence) - 1) / (SEQUENCE_LENGTH - 1))
            for index in range(SEQUENCE_LENGTH)
        ]
        return np.stack([sequence[index] for index in indices])

    def extract(self, video_path: Path):
        video_path = Path(video_path)
        capture = cv2.VideoCapture(str(video_path))
        if not capture.isOpened():
            raise RuntimeError(f"Could not open video: {video_path}")

        fps = capture.get(cv2.CAP_PROP_FPS)
        if fps <= 0:
            fps = 30.0

        frames = []
        frame_index = 0
        previous_timestamp = -1

        try:
            with self._create_landmarker() as landmarker:
                while True:
                    success, frame = capture.read()
                    if not success:
                        break

                    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                    mp_image = mp.Image(
                        image_format=mp.ImageFormat.SRGB,
                        data=rgb_frame,
                    )
                    timestamp = max(
                        previous_timestamp + 1,
                        int(frame_index * 1000 / fps),
                    )
                    result = landmarker.detect_for_video(
                        mp_image, timestamp
                    )
                    frames.append(self._frame_features(result))
                    previous_timestamp = timestamp
                    frame_index += 1
        finally:
            capture.release()

        if not frames:
            raise RuntimeError(f"No frames were decoded from: {video_path}")
        if not any(frame[-2] or frame[-1] for frame in frames):
            raise RuntimeError(f"No hands were detected in: {video_path}")

        return self._resample(np.stack(frames))


def main():
    parser = argparse.ArgumentParser(
        description="Extract a fixed-length hand-landmark sequence."
    )
    parser.add_argument("video", type=Path)
    parser.add_argument(
        "--model",
        type=Path,
        default=Path("models/hand_landmarker.task"),
    )
    args = parser.parse_args()

    extractor = HandSequenceExtractor(args.model)
    sequence = extractor.extract(args.video)
    detected_frames = sum(
        1 for frame in sequence if frame[-2] or frame[-1]
    )

    print(f"Feature shape: {len(sequence)} x {len(sequence[0])}")
    print(
        f"Detected hands in {detected_frames}/{len(sequence)} "
        "resampled frames"
    )


if __name__ == "__main__":
    main()

How to use this reference

Compare this file with your saved version from top to bottom. The imports plus constants plus class plus command-line interface should match exactly.

Before you run the extractor, what sequence dimensions do you expect from 32 frames with 128 values per frame?

  • Run the extractor against the validated reference clip using the integrated terminal:
python feature_extractor.py data/ISL/SIGN_01/ref-01.mp4

What should you see?

The terminal reports Feature shape: 32 x 128. A second line reports detected hands in at least one of the 32 resampled frames.

This confirms that one variable-length video now becomes a fixed-size temporal feature sequence.

Extractor not reporting the expected shape?

If the video cannot open, confirm the Explorer path is exactly data/ISL/SIGN_01/ref-01.mp4. If the model cannot load, confirm models/hand_landmarker.task still exists.

A no-hands result means the model found no hands in the decoded frames. Use a validated clip with clearly visible hands. Help me diagnose my landmark extraction result.

That is the extraction pipeline working. Your reference video now produces a consistent motion sequence for the recognizer.

Your video is now compact landmark data. Next, you will build a reference index plus a deliberately limited baseline that reveals what one-frame matching misses.

Build the Baseline Index

Your Visual Studio Code workspace from earlier can already turn one video into a fixed-length NumPy sequence. The recognizer now needs several labeled examples that it can search.

A reference index stores those examples with their labels. A language namespace keeps every candidate tied to its sign language while the baseline reveals how much motion evidence it actually uses.

In this step, get ready to:
  • Organize ten validated reference clips in a five-sign catalog.
  • Build a compressed index from the reference sequences.
  • Probe a held-out clip to measure the baseline's motion coverage.
Organize the reference catalog

A JSON catalog separates sign metadata from the Python implementation. Each entry records a validated label with its provenance note.

  • In the Explorer sidebar from earlier, expand data/ISL.
  • Create the missing sign folders shown in the catalog template below.
  • Confirm the Explorer sidebar lists SIGN_01 through SIGN_05 under data/ISL.
  • Place two validated reference clips in every sign folder.
  • Name each reference pair ref-01.mp4 and ref-02.mp4.
  • Confirm each sign folder shows its two reference files in the Explorer sidebar.
  • Place the held-out SIGN_01 clip at data/ISL/SIGN_01/query.mp4.
  • Confirm query.mp4 appears beside the two SIGN_01 references.

Why keep provenance in the catalog?

A similarity result only has meaning when each reference has a validated sign label. The source field preserves the source or consent note beside that label.

Clear provenance prevents a convenient filename from becoming an unsupported linguistic claim.

  • Click the New File icon in the Explorer toolbar.
  • Enter catalog.json as the file name.
  • Use the complete template in the double-check tab to populate catalog.json.

✔️ Awesome, I've got everything!

Great. Confirm that catalog.json contains five signs with two reference paths per sign.

ⓧ I'd like to double check the full code

{
  "languages": [
    {
      "id": "ISL",
      "display_name": "Indian Sign Language",
      "signs": [
        {
          "id": "SIGN_01",
          "display_name": "Validated ISL sign 01",
          "source": "replace-with-source-or-consent-note",
          "references": [
            "data/ISL/SIGN_01/ref-01.mp4",
            "data/ISL/SIGN_01/ref-02.mp4"
          ]
        },
        {
          "id": "SIGN_02",
          "display_name": "Validated ISL sign 02",
          "source": "replace-with-source-or-consent-note",
          "references": [
            "data/ISL/SIGN_02/ref-01.mp4",
            "data/ISL/SIGN_02/ref-02.mp4"
          ]
        },
        {
          "id": "SIGN_03",
          "display_name": "Validated ISL sign 03",
          "source": "replace-with-source-or-consent-note",
          "references": [
            "data/ISL/SIGN_03/ref-01.mp4",
            "data/ISL/SIGN_03/ref-02.mp4"
          ]
        },
        {
          "id": "SIGN_04",
          "display_name": "Validated ISL sign 04",
          "source": "replace-with-source-or-consent-note",
          "references": [
            "data/ISL/SIGN_04/ref-01.mp4",
            "data/ISL/SIGN_04/ref-02.mp4"
          ]
        },
        {
          "id": "SIGN_05",
          "display_name": "Validated ISL sign 05",
          "source": "replace-with-source-or-consent-note",
          "references": [
            "data/ISL/SIGN_05/ref-01.mp4",
            "data/ISL/SIGN_05/ref-02.mp4"
          ]
        }
      ]
    }
  ]
}

How is the catalog structured?

  • The languages array places all five entries inside the ISL namespace.
  • Each sign has a stable ID that the recognizer can return.
  • Each references array points to exactly two validated clips.
  • Each source value reserves space for the matching provenance or consent note.
  • Replace each display_name template with its validated sign name.
  • Replace each replace-with-source-or-consent-note value with the matching provenance or consent note.
  • Save catalog.json.
  • Confirm catalog.json remains visible beside feature_extractor.py in the Explorer sidebar.

Do the catalog paths look wrong?

Check that each folder name uses uppercase SIGN with a two-digit number. Confirm every filename matches the corresponding reference path exactly.

Keep each provenance note inside its existing quotation marks. Help me check my catalog paths and JSON structure.

Build the reference index

The index pairs every extracted sequence with its catalog metadata. The build process checks that each sign has at least two examples before saving the compressed file.

Why nearest exemplars?

Nearest-exemplar retrieval compares a query with stored reference sequences. A new catalog entry only needs validated clips plus an index rebuild.

A fixed classifier would require retraining whenever the vocabulary changes. The reference approach keeps this prototype data-driven.

  • Click the New File icon in the Explorer toolbar.
  • Enter recognizer.py as the file name.
  • Use the complete baseline in the double-check tab to populate recognizer.py.

✔️ Awesome, I've got everything!

Great. Confirm that recognizer.py contains the build and probe command paths.

ⓧ I'd like to double check the full code

import argparse
import json
from pathlib import Path

import numpy as np

from feature_extractor import (
    FRAME_FEATURES,
    SEQUENCE_LENGTH,
    HandSequenceExtractor,
)

DEFAULT_CATALOG = Path("catalog.json")
DEFAULT_MODEL = Path("models/hand_landmarker.task")
DEFAULT_INDEX = Path("models/sign_index.npz")
DISTANCE_NORMALISER = 64.0


def load_catalog(catalog_path: Path):
    return json.loads(catalog_path.read_text(encoding="utf-8"))


def collect_reference_items(catalog, catalog_path: Path):
    items = []
    base_directory = catalog_path.parent

    for language in catalog["languages"]:
        language_id = language["id"]
        for sign in language["signs"]:
            references = sign["references"]
            if len(references) < 2:
                raise ValueError(
                    f"{language_id}:{sign['id']} needs at least two references"
                )

            for reference in references:
                items.append(
                    {
                        "language_id": language_id,
                        "sign_id": sign["id"],
                        "display_name": sign["display_name"],
                        "source": sign["source"],
                        "path": str((base_directory / reference).resolve()),
                    }
                )

    if not items:
        raise ValueError("The catalog contains no reference videos")
    return items


def first_frame_distance(first, second):
    return float(
        np.linalg.norm(first[0] - second[0])
        / float(np.sqrt(FRAME_FEATURES))
    )


def build_index(catalog_path: Path, model_path: Path, index_path: Path):
    catalog = load_catalog(catalog_path)
    items = collect_reference_items(catalog, catalog_path)
    extractor = HandSequenceExtractor(model_path)
    embeddings = []

    for item in items:
        print(
            f"Indexing {item['language_id']}:{item['sign_id']} "
            f"from {item['path']}"
        )
        embeddings.append(extractor.extract(Path(item["path"])))

    embedding_array = np.stack(embeddings)
    metadata = {
        "items": items,
        "sequence_length": SEQUENCE_LENGTH,
        "frame_features": FRAME_FEATURES,
    }

    index_path.parent.mkdir(parents=True, exist_ok=True)
    np.savez_compressed(
        index_path,
        embeddings=embedding_array,
        metadata=np.stack([json.dumps(metadata)]),
    )
    print(f"Saved {len(items)} references to {index_path}")


def load_index(index_path: Path):
    archive = np.load(index_path)
    embeddings = archive["embeddings"]
    metadata = json.loads(str(archive["metadata"][0]))
    return embeddings, metadata


def candidate_indices(metadata, language_id: str):
    indices = [
        index
        for index, item in enumerate(metadata["items"])
        if item["language_id"] == language_id
    ]
    return indices


def nearest_match(query, embeddings, metadata, language_id, distance_function):
    indices = candidate_indices(metadata, language_id)
    distances = [
        distance_function(query, embeddings[index]) for index in indices
    ]
    local_best = min(
        range(len(distances)), key=distances.__getitem__
    )
    best_index = indices[local_best]
    return metadata["items"][best_index], distances[local_best]


def run_probe(video_path, language_id, model_path, index_path):
    embeddings, metadata = load_index(index_path)
    query = HandSequenceExtractor(model_path).extract(video_path)
    item, distance = nearest_match(
        query,
        embeddings,
        metadata,
        language_id,
        first_frame_distance,
    )
    print(
        f"Baseline result: {item['language_id']}:{item['sign_id']} "
        f"({item['display_name']})"
    )
    print(f"First-frame distance: {distance:.4f}")
    print("Motion coverage: 1/32 frames")
    print("LIMITATION: 31 frames ignored.")


def add_shared_arguments(parser):
    parser.add_argument(
        "--model", type=Path, default=DEFAULT_MODEL
    )
    parser.add_argument(
        "--index", type=Path, default=DEFAULT_INDEX
    )


def main():
    parser = argparse.ArgumentParser(
        description="Build and query a language-aware sign reference index."
    )
    subparsers = parser.add_subparsers(
        dest="command", required=True
    )

    build_parser = subparsers.add_parser("build")
    build_parser.add_argument(
        "--catalog", type=Path, default=DEFAULT_CATALOG
    )
    add_shared_arguments(build_parser)

    probe_parser = subparsers.add_parser("probe")
    probe_parser.add_argument("video", type=Path)
    probe_parser.add_argument("--language", required=True)
    add_shared_arguments(probe_parser)

    args = parser.parse_args()

    if args.command == "build":
        build_index(args.catalog, args.model, args.index)
    elif args.command == "probe":
        run_probe(args.video, args.language, args.model, args.index)


if __name__ == "__main__":
    main()

How does the baseline work?

  • The catalog loader turns the JSON document into Python data.
  • The reference collector attaches each file path to its language ID with its sign metadata.
  • The build path extracts every clip into a 32 x 128 embedding.
  • The probe path filters candidates to the requested language before calculating its baseline distance.
  • Save recognizer.py.
  • Build the reference index in the integrated terminal from earlier by running this command:
python recognizer.py build

What does this command do?

The build command reads all ten reference paths from catalog.json. It sends every clip through HandSequenceExtractor.

The command stacks the resulting sequences with their metadata. It saves the compressed index at models/sign_index.npz.

Each reference is processed frame by frame, so the terminal can pause between indexing messages. The next message confirms which clip is being processed.

You'll see ten indexing messages followed by Saved 10 references to models/sign_index.npz. Your catalog is now a reusable reference index.

  • Return to the models folder in the Explorer sidebar.
  • Confirm sign_index.npz appears beside hand_landmarker.task.

Index build not completing?

Check that every reference path in catalog.json points to an existing video. Confirm each clip shows clear hands when opened.

Make sure the integrated terminal still shows the active .venv environment. Help me diagnose why my sign reference index is not building.

Probe the baseline

The held-out query gives the index a clip it did not store. The probe reports its nearest catalog example plus the amount of sampled motion used for that comparison.

Before you run the probe, how many of the 32 sampled frames do you think the baseline uses?

  • Probe the held-out SIGN_01 clip by running this command:
python recognizer.py probe data/ISL/SIGN_01/query.mp4 --language ISL

What does this command check?

The probe command extracts the held-out clip with the same 32 x 128 representation used by the index. The --language option restricts the candidate set to ISL entries.

You'll see a baseline result with a first-frame distance. The final lines read Motion coverage: 1/32 frames and LIMITATION: 31 frames ignored.

That shortfall is intentional. The probe reduces a moving sign to one pose even though the extractor prepared 32 temporal samples.

Probe not reaching the limitation output?

Confirm models/sign_index.npz exists before running the probe. Check that the held-out file is saved at data/ISL/SIGN_01/query.mp4.

Keep ISL uppercase so it matches the catalog namespace. Help me troubleshoot why my baseline probe is not printing its motion coverage.

Your index can now retrieve a labeled reference from ten stored clips. Next, you'll replace the one-frame comparison with a distance that uses the complete sign trajectory.

Match Full Sign Sequences

Your first-frame probe exposed the baseline's blind spot. Its result used only 1/32 frames from each sign sequence.

Full-sequence matching compares the complete resampled trajectories. Nearest-exemplar retrieval uses a normalized Euclidean distance to turn differences across every frame and feature into one score.

In this step, get ready to:
  • Calculate distance across complete 32-frame sequences.
  • Retrieve the nearest reference from the requested language namespace.
  • Compare the full-sequence result with the earlier first-frame probe.
Add full-sequence distance

Each embedding has 32 rows with 128 features per row. One distance calculation can therefore measure differences across the sign's complete sampled motion.

  • In recognizer.py, find the DEFAULT_INDEX constant.
  • Add the sequence-distance normalizer below DEFAULT_INDEX by pasting this line:
DISTANCE_NORMALISER = 64.0

Why use a normalizer?

The raw distance grows as more features contribute to the comparison. DISTANCE_NORMALISER scales that magnitude into a score that is easier to inspect across predictions.

  • In recognizer.py, find the first_frame_distance function.
  • Insert the full-sequence distance function above first_frame_distance by pasting this code:
def sequence_distance(first, second):
    return float(np.linalg.norm(first - second) / DISTANCE_NORMALISER)

What does this function measure?

  • The subtraction measures feature differences between matching positions in both sequence matrices.
  • The np.linalg.norm calculation combines differences from all 32 frames into one Euclidean magnitude.
  • The division produces the normalized distance used to rank reference clips.
  • Save recognizer.py.
  • Confirm the existing baseline still runs by using this command in the integrated terminal:
python recognizer.py probe data/ISL/SIGN_01/query.mp4 --language ISL

What should you see?

You should still see a first-frame result followed by Motion coverage: 1/32 frames and LIMITATION: 31 frames ignored. This confirms the new function loads without changing the intentionally limited probe.

Seeing a Python error?

Check that DISTANCE_NORMALISER sits with the other top-level constants. Confirm that sequence_distance is aligned with first_frame_distance.

A misspelled constant prevents the file from loading. Help me check my sequence-distance code.

Create the prediction path

The existing retrieval helper already filters references through the requested language namespace. The new prediction path supplies sequence_distance so the ranking uses every sampled frame.

  • In recognizer.py, find the run_probe function.
  • Add the full-sequence prediction function below run_probe by pasting this code:
def run_prediction(video_path, language_id, model_path, index_path):
    embeddings, metadata = load_index(index_path)
    query = HandSequenceExtractor(model_path).extract(video_path)
    item, distance = nearest_match(
        query,
        embeddings,
        metadata,
        language_id,
        sequence_distance,
    )
    print(f"Compared all {SEQUENCE_LENGTH} frames")
    print(f"Best-match distance: {distance:.4f}")
    print(
        f"Prediction: {item['language_id']}:{item['sign_id']} "
        f"({item['display_name']})"
    )

How does prediction work?

  • The function loads the ten saved reference embeddings from models/sign_index.npz.
  • The HandSequenceExtractor converts the query video into the same 32-frame representation.
  • The nearest_match helper limits candidates to the requested language before ranking them with sequence_distance.
  • The final lines report temporal coverage plus the nearest configured sign.
  • Save recognizer.py.
  • Check that the new function has valid syntax by rerunning the baseline command:
python recognizer.py probe data/ISL/SIGN_01/query.mp4 --language ISL

What does this check prove?

The familiar probe output confirms that Python can load the new prediction function. The probe remains motion-blind because it still passes first_frame_distance into the shared retrieval helper.

Did the baseline stop loading?

Check the indentation inside run_prediction. Every line in the function body must sit beneath its def line.

Confirm that the closing parenthesis for the prediction output aligns with its opening print call. Help me find the syntax problem.

Expose the predict command

The command-line parser decides which recognizer path runs. A predict subcommand gives the full-sequence path the same model and index options as the baseline probe.

  • In the main function, find the line that passes shared arguments to probe_parser.
  • Add the prediction parser directly below that line by pasting this code:
    predict_parser = subparsers.add_parser("predict")
    predict_parser.add_argument("video", type=Path)
    predict_parser.add_argument("--language", required=True)
    add_shared_arguments(predict_parser)

What does this parser add?

  • The new subparser recognizes predict as a command.
  • The positional argument captures the query video path.
  • The required language argument controls which reference namespace can supply candidates.
  • The shared arguments preserve the existing default model and index paths.
  • At the bottom of main, replace the existing command dispatch with this three-path dispatch:
    if args.command == "build":
        build_index(args.catalog, args.model, args.index)
    elif args.command == "probe":
        run_probe(args.video, args.language, args.model, args.index)
    else:
        run_prediction(
            args.video, args.language, args.model, args.index
        )

How does command dispatch work?

  • The first branch keeps the index-building path unchanged.
  • The second branch preserves the deliberately limited probe.
  • The final branch sends predict requests to the full-sequence matcher.

✔️ Awesome, I've got everything!

Great. Double-check that you saved recognizer.py before rebuilding the index.

ⓧ I'd like to double check the full code

import argparse
import json
from pathlib import Path

import numpy as np

from feature_extractor import (
    FRAME_FEATURES,
    SEQUENCE_LENGTH,
    HandSequenceExtractor,
)

DEFAULT_CATALOG = Path("catalog.json")
DEFAULT_MODEL = Path("models/hand_landmarker.task")
DEFAULT_INDEX = Path("models/sign_index.npz")
DISTANCE_NORMALISER = 64.0


def load_catalog(catalog_path: Path):
    return json.loads(catalog_path.read_text(encoding="utf-8"))


def collect_reference_items(catalog, catalog_path: Path):
    items = []
    base_directory = catalog_path.parent

    for language in catalog["languages"]:
        language_id = language["id"]
        for sign in language["signs"]:
            references = sign["references"]
            if len(references) < 2:
                raise ValueError(
                    f"{language_id}:{sign['id']} needs at least two references"
                )

            for reference in references:
                items.append(
                    {
                        "language_id": language_id,
                        "sign_id": sign["id"],
                        "display_name": sign["display_name"],
                        "source": sign["source"],
                        "path": str((base_directory / reference).resolve()),
                    }
                )

    if not items:
        raise ValueError("The catalog contains no reference videos")
    return items


def sequence_distance(first, second):
    return float(np.linalg.norm(first - second) / DISTANCE_NORMALISER)


def first_frame_distance(first, second):
    return float(
        np.linalg.norm(first[0] - second[0])
        / float(np.sqrt(FRAME_FEATURES))
    )


def build_index(catalog_path: Path, model_path: Path, index_path: Path):
    catalog = load_catalog(catalog_path)
    items = collect_reference_items(catalog, catalog_path)
    extractor = HandSequenceExtractor(model_path)
    embeddings = []

    for item in items:
        print(
            f"Indexing {item['language_id']}:{item['sign_id']} "
            f"from {item['path']}"
        )
        embeddings.append(extractor.extract(Path(item["path"])))

    embedding_array = np.stack(embeddings)
    metadata = {
        "items": items,
        "sequence_length": SEQUENCE_LENGTH,
        "frame_features": FRAME_FEATURES,
    }

    index_path.parent.mkdir(parents=True, exist_ok=True)
    np.savez_compressed(
        index_path,
        embeddings=embedding_array,
        metadata=np.stack([json.dumps(metadata)]),
    )
    print(f"Saved {len(items)} references to {index_path}")


def load_index(index_path: Path):
    archive = np.load(index_path)
    embeddings = archive["embeddings"]
    metadata = json.loads(str(archive["metadata"][0]))
    return embeddings, metadata


def candidate_indices(metadata, language_id: str):
    indices = [
        index
        for index, item in enumerate(metadata["items"])
        if item["language_id"] == language_id
    ]
    return indices


def nearest_match(query, embeddings, metadata, language_id, distance_function):
    indices = candidate_indices(metadata, language_id)
    distances = [
        distance_function(query, embeddings[index]) for index in indices
    ]
    local_best = min(
        range(len(distances)), key=distances.__getitem__
    )
    best_index = indices[local_best]
    return metadata["items"][best_index], distances[local_best]


def run_probe(video_path, language_id, model_path, index_path):
    embeddings, metadata = load_index(index_path)
    query = HandSequenceExtractor(model_path).extract(video_path)
    item, distance = nearest_match(
        query,
        embeddings,
        metadata,
        language_id,
        first_frame_distance,
    )
    print(
        f"Baseline result: {item['language_id']}:{item['sign_id']} "
        f"({item['display_name']})"
    )
    print(f"First-frame distance: {distance:.4f}")
    print("Motion coverage: 1/32 frames")
    print("LIMITATION: 31 frames ignored.")


def run_prediction(video_path, language_id, model_path, index_path):
    embeddings, metadata = load_index(index_path)
    query = HandSequenceExtractor(model_path).extract(video_path)
    item, distance = nearest_match(
        query,
        embeddings,
        metadata,
        language_id,
        sequence_distance,
    )
    print(f"Compared all {SEQUENCE_LENGTH} frames")
    print(f"Best-match distance: {distance:.4f}")
    print(
        f"Prediction: {item['language_id']}:{item['sign_id']} "
        f"({item['display_name']})"
    )


def add_shared_arguments(parser):
    parser.add_argument(
        "--model", type=Path, default=DEFAULT_MODEL
    )
    parser.add_argument(
        "--index", type=Path, default=DEFAULT_INDEX
    )


def main():
    parser = argparse.ArgumentParser(
        description="Build and query a language-aware sign reference index."
    )
    subparsers = parser.add_subparsers(
        dest="command", required=True
    )

    build_parser = subparsers.add_parser("build")
    build_parser.add_argument(
        "--catalog", type=Path, default=DEFAULT_CATALOG
    )
    add_shared_arguments(build_parser)

    probe_parser = subparsers.add_parser("probe")
    probe_parser.add_argument("video", type=Path)
    probe_parser.add_argument("--language", required=True)
    add_shared_arguments(probe_parser)

    predict_parser = subparsers.add_parser("predict")
    predict_parser.add_argument("video", type=Path)
    predict_parser.add_argument("--language", required=True)
    add_shared_arguments(predict_parser)

    args = parser.parse_args()

    if args.command == "build":
        build_index(args.catalog, args.model, args.index)
    elif args.command == "probe":
        run_probe(args.video, args.language, args.model, args.index)
    else:
        run_prediction(
            args.video, args.language, args.model, args.index
        )


if __name__ == "__main__":
    main()
  • Save recognizer.py.
  • Rebuild the ten-reference index by running this command:
python recognizer.py build

What should the build report?

The terminal prints one indexing line for each reference clip. It finishes with Saved 10 references to models/sign_index.npz.

That index is refreshed. All ten 32-frame references are ready for full-sequence comparison.

Did the index rebuild stop?

Check the video path named by the last indexing line. Confirm that the referenced clip still exists under its matching sign folder.

If Python reports a model problem, confirm that models/hand_landmarker.task remains in place. Help me diagnose the failed index rebuild.

Before you run the final check, do you expect the matcher to keep using one frame or expand across the complete sequence?

  • Run the held-out SIGN_01 query through the full-sequence matcher with this command:
python recognizer.py predict data/ISL/SIGN_01/query.mp4 --language ISL

What should you see?

You should see Compared all 32 frames followed by a Best-match distance: value. The prediction line reports the nearest reference from the ISL namespace.

The earlier probe reported coverage of one frame. This result proves that motion across the full resampled trajectory now affects retrieval.

Predict command not working?

Confirm that the predict_parser block sits inside main with the same indentation as probe_parser.

Check that the final dispatch calls run_prediction for the remaining command path. Help me troubleshoot the predict command.

Your recognizer now compares all 32 temporal samples before choosing its nearest ISL reference. Next, you will add an honest boundary for clips that do not resemble the catalog.

Reject Unknown Signs

Your full-sequence matcher now compares all 32 frames. However, its nearest result still becomes a label even when the clip is unlike every catalog entry.

Unknown rejection adds a calibrated boundary for each language namespace. The compressed NumPy index stores that boundary so every prediction can show the evidence behind its decision.

In this step, get ready to:
  • Calibrate an unknown threshold from each language's reference clips.
  • Return a language-qualified prediction only within the calibrated boundary.
  • Prove that a non-catalog clip returns UNKNOWN.
Calibrate thresholds by language

For each reference, the calibration finds the closest clip with the same sign ID. The largest of those distances becomes the boundary after a 25 percent margin.

  • In recognizer.py, locate the sequence_distance() function.
  • Add the first portion of calibrate_thresholds() below sequence_distance() by pasting this code:
def calibrate_thresholds(embeddings, items):
    thresholds = {}
    language_ids = sorted({item["language_id"] for item in items})

    for language_id in language_ids:
        within_sign = []
        cross_sign = []
        candidate_indices = [
            index
            for index, item in enumerate(items)
            if item["language_id"] == language_id
        ]

        for index in candidate_indices:
            same_sign = []
            other_sign = []
            for other_index in candidate_indices:
                if index == other_index:
                    continue
                distance = sequence_distance(
                    embeddings[index], embeddings[other_index]
                )

How does calibration stay language-aware?

  • The language_ids set creates one calibration pass for each namespace.
  • The candidate_indices list keeps comparisons inside the active language.
  • Each reference is compared with every other reference in that language.
  • Continue the same calibrate_thresholds() function directly below the first portion by pasting this code:
                if items[index]["sign_id"] == items[other_index]["sign_id"]:
                    same_sign.append(distance)
                else:
                    other_sign.append(distance)

            if same_sign:
                within_sign.append(min(same_sign))
            if other_sign:
                cross_sign.append(min(other_sign))

        if not within_sign or not cross_sign:
            raise ValueError(
                f"{language_id} needs at least two signs and two references per sign"
            )

        threshold = max(within_sign) * 1.25
        thresholds[language_id] = threshold

        if min(cross_sign) <= threshold:
            print(
                f"Warning: {language_id} reference groups overlap. "
                "Add more diverse clips before trusting the labels."
            )

    return thresholds

What completes the threshold?

  • The nearest same-sign distance for each reference measures normal variation within that sign.
  • The largest same-sign distance receives a 25 percent margin.
  • The overlap check warns when a different sign is already closer than the rejection boundary.
  • In build_index(), locate the line that assigns embedding_array.
  • Replace the index metadata and save section from that line onward with this code:
    embedding_array = np.stack(embeddings)
    thresholds = calibrate_thresholds(embedding_array, items)
    metadata = {
        "items": items,
        "thresholds": thresholds,
        "sequence_length": SEQUENCE_LENGTH,
        "frame_features": FRAME_FEATURES,
    }

    index_path.parent.mkdir(parents=True, exist_ok=True)
    np.savez_compressed(
        index_path,
        embeddings=embedding_array,
        metadata=np.stack([json.dumps(metadata)]),
    )
    print(f"Saved {len(items)} references to {index_path}")
    for language_id, threshold in thresholds.items():
        print(f"{language_id} unknown threshold: {threshold:.4f}")

How does the index store calibration?

  • The build calculates thresholds from the completed reference embeddings.
  • The metadata stores each threshold under its matching language ID.
  • The final loop prints the calibrated value so you can inspect it after every rebuild.
  • Save recognizer.py.

Before you rebuild, do you expect one shared threshold or a threshold labelled for the ISL namespace?

  • Rebuild the reference index with calibrated metadata by running this command:
python recognizer.py build

What should I see?

  • You'll see each of the ten reference clips reported as it is indexed.
  • You'll see Saved 10 references to models/sign_index.npz near the end.
  • You'll see a final line beginning with ISL unknown threshold:.

Index build not completing?

Check that every catalog sign still has two readable reference paths. A missing clip prevents the calibration from receiving a complete reference set.

If the build reports overlapping groups, review clips from signs that look unusually similar. Help me diagnose my threshold calibration output.

Apply the rejection rule

The existing language filter chooses the candidate references. The threshold for that same language now decides whether the nearest candidate is close enough to name.

  • In recognizer.py, locate the existing run_prediction() function.
  • Replace the complete run_prediction() function with this version:
def run_prediction(video_path, language_id, model_path, index_path):
    embeddings, metadata = load_index(index_path)
    query = HandSequenceExtractor(model_path).extract(video_path)
    item, distance = nearest_match(
        query,
        embeddings,
        metadata,
        language_id,
        sequence_distance,
    )
    threshold = float(metadata["thresholds"][language_id])

    print(f"Compared all {SEQUENCE_LENGTH} frames")
    print(f"Best-match distance: {distance:.4f}")
    print(f"Unknown threshold: {threshold:.4f}")

    if distance > threshold:
        print("Prediction: UNKNOWN")
    else:
        print(
            f"Prediction: {item['language_id']}:{item['sign_id']} "
            f"({item['display_name']})"
        )

How does the decision work?

  • The function reads the threshold stored for the requested language_id.
  • A distance above the threshold produces Prediction: UNKNOWN.
  • A distance within the threshold prints the language-qualified sign ID from the nearest reference.
  • Save recognizer.py.

Before you test the held-out known clip, do you expect its best-match distance to sit above or below the ISL threshold?

  • Test the held-out SIGN_01 query by running this command:
python recognizer.py predict data/ISL/SIGN_01/query.mp4 --language ISL

What should a known result show?

You'll see the full-sequence distance beside the calibrated threshold. When the held-out clip sits within the boundary, the final line is Prediction: ISL:SIGN_01 followed by its display name.

Known clip returning UNKNOWN?

Compare the printed distance with the threshold. A large gap can indicate different framing or hand visibility between the query and its two references.

Confirm that the query belongs to the validated SIGN_01 category. Help me investigate this false rejection.

✔️ Awesome, I've got everything!

Great. Double-check that you saved recognizer.py before testing the non-catalog clip.

ⓧ I'd like to double check the full code

import argparse
import json
from pathlib import Path

import numpy as np

from feature_extractor import (
    FRAME_FEATURES,
    SEQUENCE_LENGTH,
    HandSequenceExtractor,
)

DEFAULT_CATALOG = Path("catalog.json")
DEFAULT_MODEL = Path("models/hand_landmarker.task")
DEFAULT_INDEX = Path("models/sign_index.npz")
DISTANCE_NORMALISER = 64.0


def load_catalog(catalog_path: Path):
    return json.loads(catalog_path.read_text(encoding="utf-8"))


def collect_reference_items(catalog, catalog_path: Path):
    items = []
    base_directory = catalog_path.parent

    for language in catalog["languages"]:
        language_id = language["id"]
        for sign in language["signs"]:
            references = sign["references"]
            if len(references) < 2:
                raise ValueError(
                    f"{language_id}:{sign['id']} needs at least two references"
                )

            for reference in references:
                items.append(
                    {
                        "language_id": language_id,
                        "sign_id": sign["id"],
                        "display_name": sign["display_name"],
                        "source": sign["source"],
                        "path": str((base_directory / reference).resolve()),
                    }
                )

    if not items:
        raise ValueError("The catalog contains no reference videos")
    return items


def sequence_distance(first, second):
    return float(np.linalg.norm(first - second) / DISTANCE_NORMALISER)


def first_frame_distance(first, second):
    return float(
        np.linalg.norm(first[0] - second[0])
        / float(np.sqrt(FRAME_FEATURES))
    )


def calibrate_thresholds(embeddings, items):
    thresholds = {}
    language_ids = sorted({item["language_id"] for item in items})

    for language_id in language_ids:
        within_sign = []
        cross_sign = []
        candidate_indices = [
            index
            for index, item in enumerate(items)
            if item["language_id"] == language_id
        ]

        for index in candidate_indices:
            same_sign = []
            other_sign = []
            for other_index in candidate_indices:
                if index == other_index:
                    continue
                distance = sequence_distance(
                    embeddings[index], embeddings[other_index]
                )
                if items[index]["sign_id"] == items[other_index]["sign_id"]:
                    same_sign.append(distance)
                else:
                    other_sign.append(distance)

            if same_sign:
                within_sign.append(min(same_sign))
            if other_sign:
                cross_sign.append(min(other_sign))

        if not within_sign or not cross_sign:
            raise ValueError(
                f"{language_id} needs at least two signs and two references per sign"
            )

        threshold = max(within_sign) * 1.25
        thresholds[language_id] = threshold

        if min(cross_sign) <= threshold:
            print(
                f"Warning: {language_id} reference groups overlap. "
                "Add more diverse clips before trusting the labels."
            )

    return thresholds


def build_index(catalog_path: Path, model_path: Path, index_path: Path):
    catalog = load_catalog(catalog_path)
    items = collect_reference_items(catalog, catalog_path)
    extractor = HandSequenceExtractor(model_path)
    embeddings = []

    for item in items:
        print(
            f"Indexing {item['language_id']}:{item['sign_id']} "
            f"from {item['path']}"
        )
        embeddings.append(extractor.extract(Path(item["path"])))

    embedding_array = np.stack(embeddings)
    thresholds = calibrate_thresholds(embedding_array, items)
    metadata = {
        "items": items,
        "thresholds": thresholds,
        "sequence_length": SEQUENCE_LENGTH,
        "frame_features": FRAME_FEATURES,
    }

    index_path.parent.mkdir(parents=True, exist_ok=True)
    np.savez_compressed(
        index_path,
        embeddings=embedding_array,
        metadata=np.stack([json.dumps(metadata)]),
    )
    print(f"Saved {len(items)} references to {index_path}")
    for language_id, threshold in thresholds.items():
        print(f"{language_id} unknown threshold: {threshold:.4f}")


def load_index(index_path: Path):
    archive = np.load(index_path)
    embeddings = archive["embeddings"]
    metadata = json.loads(str(archive["metadata"][0]))
    return embeddings, metadata


def candidate_indices(metadata, language_id: str):
    indices = [
        index
        for index, item in enumerate(metadata["items"])
        if item["language_id"] == language_id
    ]
    if not indices:
        available = sorted(metadata["thresholds"])
        raise ValueError(
            f"Unknown language {language_id}. Available: {available}"
        )
    return indices


def nearest_match(query, embeddings, metadata, language_id, distance_function):
    indices = candidate_indices(metadata, language_id)
    distances = [
        distance_function(query, embeddings[index]) for index in indices
    ]
    local_best = min(
        range(len(distances)), key=distances.__getitem__
    )
    best_index = indices[local_best]
    return metadata["items"][best_index], distances[local_best]


def run_probe(video_path, language_id, model_path, index_path):
    embeddings, metadata = load_index(index_path)
    query = HandSequenceExtractor(model_path).extract(video_path)
    item, distance = nearest_match(
        query,
        embeddings,
        metadata,
        language_id,
        first_frame_distance,
    )
    print(
        f"Baseline result: {item['language_id']}:{item['sign_id']} "
        f"({item['display_name']})"
    )
    print(f"First-frame distance: {distance:.4f}")
    print("Motion coverage: 1/32 frames")
    print("LIMITATION: 31 frames ignored.")


def run_prediction(video_path, language_id, model_path, index_path):
    embeddings, metadata = load_index(index_path)
    query = HandSequenceExtractor(model_path).extract(video_path)
    item, distance = nearest_match(
        query,
        embeddings,
        metadata,
        language_id,
        sequence_distance,
    )
    threshold = float(metadata["thresholds"][language_id])

    print(f"Compared all {SEQUENCE_LENGTH} frames")
    print(f"Best-match distance: {distance:.4f}")
    print(f"Unknown threshold: {threshold:.4f}")

    if distance > threshold:
        print("Prediction: UNKNOWN")
    else:
        print(
            f"Prediction: {item['language_id']}:{item['sign_id']} "
            f"({item['display_name']})"
        )


def add_shared_arguments(parser):
    parser.add_argument(
        "--model", type=Path, default=DEFAULT_MODEL
    )
    parser.add_argument(
        "--index", type=Path, default=DEFAULT_INDEX
    )


def main():
    parser = argparse.ArgumentParser(
        description="Build and query a language-aware sign reference index."
    )
    subparsers = parser.add_subparsers(
        dest="command", required=True
    )

    build_parser = subparsers.add_parser("build")
    build_parser.add_argument(
        "--catalog", type=Path, default=DEFAULT_CATALOG
    )
    add_shared_arguments(build_parser)

    probe_parser = subparsers.add_parser("probe")
    probe_parser.add_argument("video", type=Path)
    probe_parser.add_argument("--language", required=True)
    add_shared_arguments(probe_parser)

    predict_parser = subparsers.add_parser("predict")
    predict_parser.add_argument("video", type=Path)
    predict_parser.add_argument("--language", required=True)
    add_shared_arguments(predict_parser)

    args = parser.parse_args()

    if args.command == "build":
        build_index(args.catalog, args.model, args.index)
    elif args.command == "probe":
        run_probe(args.video, args.language, args.model, args.index)
    else:
        run_prediction(
            args.video, args.language, args.model, args.index
        )


if __name__ == "__main__":
    main()
Test a non-catalog clip

A non-catalog clip tests whether the boundary can stop an unsupported label. Its distance must exceed the calibrated ISL threshold for rejection to activate.

Before you run the final check, do you expect the unfamiliar clip to receive the nearest sign ID or cross the rejection boundary?

  • Test the non-catalog clip by running this command:
python recognizer.py predict data/unknown.mp4 --language ISL

What should the rejection show?

  • You'll see Compared all 32 frames.
  • You'll see the best-match distance beside the calibrated unknown threshold.
  • When the distance exceeds that threshold, the final line is Prediction: UNKNOWN.

What does UNKNOWN prove?

The result shows that the clip falls outside the variation represented by your catalog. It does not identify the sign or validate its linguistic meaning.

Hand landmarks omit facial expression and body posture. Treat every result as similarity evidence from the available references.

Non-catalog clip receiving a sign label?

Compare the printed distance with the threshold. A near-threshold result can mean the clip resembles one reference trajectory or the reference set is too narrow.

Use a clearly different validated test clip with visible hands. Help me diagnose this missed unknown rejection.

That completes the safeguard: your recognizer now preserves the ISL namespace while admitting when a clip falls outside its catalog.

Secret mission

Expand the Catalog Without Changing Python

Add a sixth validated ISL sign through clips and catalog data alone. Rebuild the index so a held-out query can return ISL:SIGN_06.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep this local workspace available, pause your active tools, or delete the project files. Everything runs locally, so there are no cloud resources or ongoing charges.

Resources you used:

  • A local virtual environment at .venv for Python.
  • The MediaPipe Hand Landmarker model bundle at models/hand_landmarker.task.
  • The reference index at models/sign_index.npz with twelve ISL reference embeddings.
  • The project files named catalog.json, feature_extractor.py, and recognizer.py.
  • The video data under data/ with reference clips plus held-out queries.

Keep everything running

No action is needed while you continue expanding the catalog. The workspace remains available without creating ongoing charges.

  • Retain the current workspace folder so you can add more validated signs.
  • Keep catalog.json with its provenance source notes.
  • Preserve models/sign_index.npz if you want to reuse the twelve indexed references.

Pause - I'll come back to this later

Pausing closes the active development tools while preserving every local artifact. Your sixth-sign catalog entry remains ready for your return.

  • Leave .venv by running this command in the terminal from earlier:
deactivate

What does this command do?

You have safely left the active Python virtual environment. Every local file remains unchanged.

  • Close Visual Studio Code.
  • Confirm that no project terminal or editor window remains open.

Your environment is paused. The catalog, model files, index, source files, and videos remain available on disk.

Delete - I don't want to use this again

Deleting the workspace is final after you empty Trash. The preservation check protects the source records you may need later.

  • Use your Linux file manager to copy any permissioned videos you need to a location outside the current workspace.
  • Use your Linux file manager to copy any provenance notes you need to the same external location.
  • Return to the terminal from earlier.
  • Leave .venv by running this command:
deactivate

What does this command do?

You have safely left the active virtual environment. This releases the environment before you remove its files.

  • Record the current workspace folder name here: your project folder name.
  • Close Visual Studio Code.
  • Use your Linux file manager to locate your project folder name.
  • Move your project folder name to Trash.
  • Empty Trash in your Linux file manager.
  • Confirm that your project folder name is no longer listed in its original location.

The virtual environment, model bundle, reference index, project files, and workspace copies of the videos are now removed.

Nice Work!

Nice Work!

You did it! You built a local isolated-sign retrieval MVP using MediaPipe with OpenCV.

You've learned how to:

  • Convert prerecorded sign clips into 32 x 128 hand-landmark sequences with consistent left-hand slots. Preserve right-hand slots across every sampled frame.
  • Reveal the first-frame baseline limitation by observing that it ignores 31 of 32 temporal samples.
  • Build a language-aware reference index that matches full sequences inside the ISL namespace. Apply calibrated unknown rejection to distant queries.
  • Secret Mission: Add ISL:SIGN_06 through catalog data alone. Rebuild the index to classify its held-out query without editing feature_extractor.py or recognizer.py.

Ready to quiz yourself?