Build a Real-Time Hand Gesture Detector

Use MediaPipe and OpenCV to recognize 7 hand gestures via webcam.

Introduction

30 Second Summary

Your hands are constantly communicating. A wave, a thumbs up, a peace sign. Computers can now read those gestures in real time using your webcam and a bit of machine learning.

In this project, you will build a real-time hand gesture detector that uses your webcam to recognize 7 different hand gestures, drawing landmark points on your hand and displaying the gesture name with a live confidence score.

What You'll Build

Point your webcam at your hand and watch as green dots appear on your fingers, your gesture is identified, and a confidence score ticks up and down as you move.

By the end of this project, you'll have:

  • A real-time gesture recognition app that detects Thumbs Up, Victory, Open Palm, Closed Fist, Pointing Up, Thumbs Down, and I Love You through your webcam.
  • A working computer vision pipeline that captures frames, processes them through an ML model, and displays annotated results at your webcam's full frame rate.
  • Hands-on understanding of how running modes (IMAGE vs VIDEO) affect detection stability, proven by watching the difference yourself.
  • Secret Mission: Add multi-hand detection and a gesture counter that tracks how many times you've performed each gesture.

Are there any prerequisites?

You should be comfortable with basic Python (variables, loops, functions) and know how to install packages with pip. You'll also need a working webcam on your computer.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters to you.

Display Your Live Webcam Feed

Your tools are installed and the gesture model is ready. Now it's time to build the foundation of your gesture detector.

Every real-time computer vision app follows the same core pattern: capture a frame from the camera, do something with it, display the result, and repeat. Before adding any ML, you need this loop working on its own.

In this step, get ready to:
  • Write a webcam capture loop using OpenCV.
  • Display your live mirrored webcam feed in a window.
  • Add a quit condition so you can close the window with a keypress.
Write the webcam capture loop

The capture loop is the heartbeat of your app. It grabs one frame at a time from your webcam, processes it, and shows the result on screen.

  • In gesture_detector.py, add the following code:
import cv2

# Open the default webcam (index 0)
cap = cv2.VideoCapture(0)

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    # Flip horizontally for a mirror effect
    frame = cv2.flip(frame, 1)
    cv2.imshow('Hand Gesture Recognizer', frame)

    # Quit when the user presses 'q'
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# Release the webcam and close the display window
cap.release()
cv2.destroyAllWindows()

What does this code do?

  • cv2.VideoCapture(0) opens your default webcam. The 0 refers to the first camera your computer finds.
  • cap.read() grabs one frame. ret tells you whether the frame was captured successfully.
  • cv2.flip(frame, 1) mirrors the image horizontally so it feels natural, like looking in a mirror.
  • cv2.imshow() displays the frame in a window. cv2.waitKey(1) waits 1 millisecond for a keypress, keeping the loop running at full speed.
  • cap.release() and cv2.destroyAllWindows() clean up the webcam and window when you're done.
  • Save gesture_detector.py.
Run the script and see your webcam feed

Before you run this, what do you expect to see? Think about what a capture loop displaying frames in rapid succession would look like.

  • Run your script by entering this command in the terminal:
python gesture_detector.py

A window titled Hand Gesture Recognizer opens showing your live webcam feed. You should see yourself on screen, mirrored like a selfie camera.

  • Wave your hand around and confirm the video updates smoothly at your webcam's frame rate.
  • Press q on your keyboard to close the window and stop the script.

Don't see the webcam window?

  • If you see a black or frozen window, another application might be using your webcam. Close any other video apps (Zoom, Teams, FaceTime) and try again.
  • If you get an error about camera access, check that your terminal or VS Code has camera permissions in your system settings.
  • If the window opens but closes instantly, make sure you saved gesture_detector.py and that your code matches the snippet above exactly.

Help me debug my webcam not showing in OpenCV

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here's the complete gesture_detector.py so far:

import cv2

cap = cv2.VideoCapture(0)

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    frame = cv2.flip(frame, 1)
    cv2.imshow('Hand Gesture Recognizer', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

You have a working video pipeline. Frames flow from your webcam to the screen in real-time. Next up, you'll feed those frames to MediaPipe's gesture recognizer and see it identify what your hand is doing.

Add Gesture Recognition

Your webcam feed is live and streaming frames smoothly. Now it's time to make those frames useful by feeding them to a machine learning model that can recognize hand gestures.

In this step, you'll connect MediaPipe's GestureRecognizer to your capture loop. You'll start with IMAGE running mode, the simplest approach that processes each frame independently.

In this step, get ready to:
  • Configure the GestureRecognizer with IMAGE running mode and load the pre-trained model.
  • Feed converted frames to the recognizer and display gesture labels on screen.
  • Test multiple gestures and observe how IMAGE mode behaves in real-time.
Set up the MediaPipe gesture recognizer

The GestureRecognizer needs a few things before it can analyze frames: the pre-trained model file, a running mode that tells it how to process input, and the number of hands to detect.

  • In VS Code, select all the existing code in gesture_detector.py and delete it.
  • Replace it with the following imports and recognizer setup:
import cv2
import mediapipe as mp

# Set up shortcut references to MediaPipe's gesture recognition classes
BaseOptions = mp.tasks.BaseOptions
GestureRecognizer = mp.tasks.vision.GestureRecognizer
GestureRecognizerOptions = mp.tasks.vision.GestureRecognizerOptions
VisionRunningMode = mp.tasks.vision.RunningMode

# Configure the recognizer: model file, IMAGE mode, detect one hand
options = GestureRecognizerOptions(
    base_options=BaseOptions(model_asset_path='gesture_recognizer.task'),
    running_mode=VisionRunningMode.IMAGE,
    num_hands=1
)

with GestureRecognizer.create_from_options(options) as recognizer:
    cap = cv2.VideoCapture(0)

What does this code do?

The shortcut variables (BaseOptions, GestureRecognizer, etc.) save you from typing the full mp.tasks.vision... path every time.

VisionRunningMode.IMAGE tells the recognizer to treat each frame as an independent photo with no memory of previous frames. The with statement ensures the recognizer is properly closed when the script ends.

  • Save gesture_detector.py. The capture loop that feeds frames to this recognizer comes next.
Feed frames to the recognizer

OpenCV captures frames in BGR color format, but MediaPipe expects RGB. Each frame needs a color conversion before the recognizer can process it.

  • Add the following capture loop directly below the cap = cv2.VideoCapture(0) line, still inside the with block:
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break

        frame = cv2.flip(frame, 1)
        # Convert BGR (OpenCV default) to RGB (MediaPipe requirement)
        rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_frame)

        # Run gesture recognition on this single frame
        result = recognizer.recognize(mp_image)

        # Draw gesture name and confidence if a gesture is detected
        if result.gestures:
            gesture_name = result.gestures[0][0].category_name
            confidence = result.gestures[0][0].score
            text = f"{gesture_name} ({confidence:.2f})"
            cv2.putText(frame, text, (10, 50), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (0, 255, 0), 3)

        cv2.imshow('Hand Gesture Recognizer', frame)

        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

    cap.release()
    cv2.destroyAllWindows()

What does this code do?

  • cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) converts the frame from BGR to RGB so MediaPipe can read it correctly.
  • mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_frame) wraps the frame in MediaPipe's image format.
  • recognizer.recognize(mp_image) sends the frame to the model and returns gesture results.
  • result.gestures[0][0] accesses the top gesture prediction for the first detected hand. The category_name is the gesture label and score is the confidence (0 to 1).
  • Save gesture_detector.py.
  • Run the script to confirm gesture recognition is working:
python gesture_detector.py
  • Hold up a thumbs up gesture in front of your webcam. You should see green text appear near the top-left corner showing Thumb_Up followed by a confidence score like (0.87).
  • Press q to close the window.

Not seeing any gesture text?

  • Make sure your hand is clearly visible and well-lit. The model needs to see your full hand to detect gestures.
  • Check that gesture_recognizer.task is in the same folder as gesture_detector.py. If the model file is missing, the script will crash with a file-not-found error.
  • Verify that the indentation of the while loop is inside the with block (indented one level in).

Help me debug why my gesture recognizer isn't detecting anything

Test different gestures and observe the flickering

The recognizer supports 7 gestures: Thumb_Up, Victory, Open_Palm, Closed_Fist, Pointing_Up, Thumb_Down, and ILoveYou. Time to try a few and watch how the detection behaves.

  • Run the script again:
python gesture_detector.py

Before you try the next part, make a prediction: if you hold a steady thumbs up for 5 seconds without moving, do you think the label will stay on screen the entire time?

  • Hold a steady thumbs up for 5 seconds and watch the text overlay closely.
  • Try a peace sign (Victory), an open palm, and a closed fist. Pay attention to what happens between gestures.
  • Press q to close the window when you're done observing.

You'll notice the text flickers on and off. Sometimes it says None between frames even while you're holding a steady gesture. The label might disappear for a frame or two and then reappear. This instability is expected and intentional.

Why does the recognition flicker?

IMAGE mode treats every single frame as a brand new, independent photo. It has zero memory of previous frames. Each frame goes through the full detection pipeline from scratch: find the hand, locate landmarks, classify the gesture.

If one frame has slightly different lighting or your hand is at a marginally different angle, the model might briefly fail to detect it. This causes the flickering you see. The next step solves this by switching to VIDEO mode, which tracks your hand across frames instead of re-detecting from zero.

✔️ Awesome, I've got everything!

Great. Make sure you've saved gesture_detector.py and that you observed the flickering behaviour.

ⓧ I'd like to double check the full code

Your gesture_detector.py should have two sections, in order:

  • Chunk 1 (imports and recognizer setup): The import cv2 and import mediapipe as mp lines, followed by the shortcut variables, the options block with VisionRunningMode.IMAGE, the with GestureRecognizer.create_from_options(options) line, and cap = cv2.VideoCapture(0).
  • Chunk 2 (capture loop with recognition): The while cap.isOpened(): loop containing the BGR-to-RGB conversion, recognizer.recognize(mp_image), the gesture overlay with cv2.putText, the display call, quit condition, and cleanup.

Scroll up and confirm both chunks are present in order. Make sure the entire while loop is indented inside the with block.

You have gesture recognition working, but the flickering makes it feel unreliable. Next up, you'll upgrade to VIDEO mode and add hand landmark drawing to make the detection smooth and visually rich.

Upgrade to VIDEO Mode with Landmark Drawing

In the previous step, your gesture detector worked but the recognition flickered on and off. Even while holding a steady thumbs up, the text would flash between "Thumb_Up" and nothing.

That instability happens because IMAGE mode treats every frame as brand new with no memory of what came before. VIDEO mode fixes this by tracking your hand across frames. Once it finds your hand, it follows its position rather than re-detecting from scratch each time.

In this step, get ready to:
  • Switch to VIDEO mode for smooth, stable gesture detection.
  • Draw 21 hand landmark dots on the video feed.
  • Display which hand (Left or Right) is being detected.
Switch to VIDEO mode

VIDEO mode requires one extra piece of information: a timestamp for each frame. This tells the recognizer the order of frames so it can track motion between them.

  • In gesture_detector.py, find the imports at the top of the file:
import cv2
import mediapipe as mp
  • Add import time on the next line, so your imports look like this:
import cv2
import mediapipe as mp
import time
  • Now scroll down to the options block and find this line:
    running_mode=VisionRunningMode.IMAGE,
  • Replace IMAGE with VIDEO, so the line now reads:
    running_mode=VisionRunningMode.VIDEO,
  • Next, find this line inside the while loop:
        result = recognizer.recognize(mp_image)
  • Replace that line with these two lines that compute a timestamp and call the VIDEO-mode method:
        timestamp_ms = int(time.time() * 1000)
        result = recognizer.recognize_for_video(mp_image, timestamp_ms)

Why does VIDEO mode need a timestamp?

The recognizer uses timestamps to understand how time passes between frames. This lets it predict where your hand will be in the next frame based on its movement, rather than searching the entire image from scratch.

The timestamps must be monotonically increasing (always getting larger). Using int(time.time() * 1000) satisfies this because wall-clock milliseconds always go up.

Before you run the script, think about what you expect to see. Will the gesture label still flicker like before, or will it behave differently now that VIDEO mode tracks between frames?

  • Save gesture_detector.py.
  • Run the script by entering this command in your terminal:
python gesture_detector.py
  • Hold up a thumbs up gesture and watch the label on screen.

The gesture name now stays stable on screen. No more flickering between "Thumb_Up" and nothing. That single change from IMAGE to VIDEO mode gave the recognizer memory between frames.

  • Press q to close the window.

Seeing a timestamp error?

If you get an error about timestamps not being monotonically increasing, make sure you changed recognizer.recognize(mp_image) to recognizer.recognize_for_video(mp_image, timestamp_ms) and that the timestamp_ms line sits directly above the recognize call.

Also confirm the running mode is set to VisionRunningMode.VIDEO, not VisionRunningMode.IMAGE. The method and mode must match.

Help me debug my timestamp error in VIDEO mode.

Draw hand landmarks on the video feed

The recognizer returns 21 landmark points for each detected hand. These map to your fingertips, knuckles, wrist, and palm. Each landmark has normalized x and y coordinates (values between 0 and 1) that you multiply by the frame dimensions to get pixel positions.

  • In gesture_detector.py, find the line you just added:
        result = recognizer.recognize_for_video(mp_image, timestamp_ms)
  • Add the following landmark drawing code directly below that line, before the existing if result.gestures: block:
        # Draw green circles on each of the 21 hand landmarks
        if result.hand_landmarks:
            for hand_landmarks in result.hand_landmarks:
                for landmark in hand_landmarks:
                    x = int(landmark.x * frame.shape[1])
                    y = int(landmark.y * frame.shape[0])
                    cv2.circle(frame, (x, y), 5, (0, 255, 0), -1)

What does this code do?

  • result.hand_landmarks contains a list of detected hands. Each hand has 21 landmark points.
  • landmark.x and landmark.y are normalized (0 to 1). Multiplying by frame.shape[1] (width) and frame.shape[0] (height) converts them to pixel coordinates.
  • cv2.circle(frame, (x, y), 5, (0, 255, 0), -1) draws a filled green circle with radius 5 at each landmark position.
  • Now find the if result.gestures: block (the one that draws the gesture name). Add the following code directly below that entire block to display which hand is detected:
        # Display whether it's the left or right hand
        if result.handedness:
            hand_label = result.handedness[0][0].category_name
            cv2.putText(frame, f"Hand: {hand_label}", (10, 100), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 0), 2)

What does this code do?

  • result.handedness[0][0].category_name returns either "Left" or "Right" for the first detected hand.
  • The text draws at position (10, 100), just below the gesture name which sits at (10, 50).
  • The colour (255, 255, 0) is yellow in BGR, making it visually distinct from the green gesture label.

Not sure where to place each block?

The order inside your while loop should be: the timestamp_ms and recognize_for_video lines first, then the if result.hand_landmarks: block, then the if result.gestures: block, then the if result.handedness: block, then cv2.imshow.

Help me figure out the correct order of blocks in my gesture detector loop.

Run and compare to IMAGE mode

Before you run this, predict what you'll see. You now have VIDEO mode, landmark drawing, and handedness detection all wired up. What do you expect the webcam feed to look like?

  • Save gesture_detector.py and run it:
python gesture_detector.py
  • Hold your hand in front of the webcam. You should see 21 green dots appear on your hand, tracing your fingers, knuckles, and wrist.
  • Make a thumbs up gesture. The label "Thumb_Up" with a confidence score should display at the top, along with a yellow "Hand: Left" or "Hand: Right" label below it.
  • Move your hand around quickly. Notice how the green dots stay locked to your fingers. The temporal tracking in VIDEO mode keeps them stable.
  • Press q to close the window when you're done.

Why is VIDEO mode so much smoother?

IMAGE mode runs full palm detection on every single frame. If your hand moves slightly or lighting shifts, the detector might miss it for a frame and return nothing.

VIDEO mode runs full detection once, then uses a lighter tracking algorithm to follow your hand between frames. It only re-runs the full detector if it loses track. This is faster and more consistent.

No green dots appearing?

Make sure your if result.hand_landmarks: block comes AFTER the result = recognizer.recognize_for_video(...) line and BEFORE the cv2.imshow call. Drawing must happen on the frame before it gets displayed.

Also check that you're drawing on frame (the BGR frame), not rgb_frame. OpenCV displays BGR images.

Help me figure out why my hand landmarks aren't showing on screen.

✔️ Awesome, I've got everything!

Great. Double check you've saved gesture_detector.py and that all three visual elements appear when you run: green dots, gesture label, and hand label.

ⓧ I'd like to double check the full code

Scroll up and confirm you have each of these sections in order inside your gesture_detector.py:

  • Imports: cv2, mediapipe, and time.
  • Alias setup: BaseOptions, GestureRecognizer, GestureRecognizerOptions, VisionRunningMode.
  • Options: running_mode=VisionRunningMode.VIDEO and num_hands=1.
  • Inside the while loop: frame capture, flip, RGB conversion, mp.Image creation, timestamp, recognize_for_video call.
  • Drawing blocks: hand_landmarks (green circles), gestures (green text at y=50), handedness (yellow text at y=100).
  • Display and quit: cv2.imshow, waitKey with 'q' to break, then cap.release() and cv2.destroyAllWindows().

Your gesture detector now has stable recognition, visible hand tracking, and handedness detection. You've built the complete real-time computer vision pipeline: capture, process with ML, draw results, and display. Next up, the Secret Mission challenges you to add multi-hand support and a gesture counter.

Secret mission

Multi-Hand Detection and Gesture Counter

Your gesture detector handles one hand beautifully. But real applications need to track multiple hands and remember what you've done over time. Can you upgrade your detector to recognize both hands simultaneously and keep a running count of every gesture you perform?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • gesture_detector.py (your Python script).
  • gesture_recognizer.task (pre-trained model file, ~5 MB).
  • mediapipe Python package.
  • opencv-python Python package.

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with more gestures and OpenCV features.

  • Your project files stay on your Desktop ready to run anytime.
  • The mediapipe and opencv-python packages remain installed in your Python environment.

Pause - I'll come back to this later

Since this is a local project with no running processes (the script only runs when you execute it), there is nothing to pause. Your files and packages stay where they are until you return.

  • Close VS Code to free up memory.
  • Your project folder, model file, and installed packages will still be there when you come back.

Delete - I don't want to use this again

Remove all project files and uninstall packages to start fresh.

  • Delete your project folder (the folder containing gesture_detector.py and gesture_recognizer.task) by running:

macOS

cd ~/Desktop
rm -rf gesture-detector

Windows

cd ~\Desktop
rmdir /s /q gesture-detector

Use your actual folder name

Replace gesture-detector with whatever you named your project folder in Step 1.

  • Uninstall the Python packages by running:
pip uninstall mediapipe opencv-python -y

Nice Work!

Nice Work!

You just built a real-time hand gesture recognition system using Python, MediaPipe, and OpenCV. Your webcam now doubles as a gesture-reading computer vision app that detects hands, draws landmarks, and identifies what you're doing with your fingers.

You've learned how to:

  • Build a real-time computer vision pipeline that captures webcam frames, processes them with a pre-trained ML model, and displays annotated results in a live loop.
  • Use MediaPipe's GestureRecognizer to detect 7 hand gestures with confidence scores and draw 21 landmark points on a live video feed.
  • Compare IMAGE and VIDEO running modes and understand why temporal tracking between frames eliminates flickering in real-time applications.
  • Secret Mission: Added multi-hand detection and a gesture counter with state management logic that tracks gesture transitions over time.

Ready to quiz yourself?