Build a Hand Gesture Controller

Use your webcam and MediaPipe to detect hand gestures in real time.

Introduction

30 Second Summary

You use your hands to communicate every day. Waving hello, pointing at something, or giving a thumbs up are all gestures that people understand instantly. Computers are now learning to see and interpret those same movements in real time.

In this project, you will build a real-time hand gesture controller using Python and your webcam. Your app will track your hand, draw a skeleton overlay on screen, and recognize different gestures as you make them.

What You'll Build

A Python application that uses your webcam to track 21 landmarks on your hand, recognizes gestures, and displays visual feedback on a live video feed.

By the end of this project, you'll have:

  • A live webcam capture pipeline that displays mirrored video using OpenCV.
  • Real-time hand landmark detection that draws a 21-point skeleton overlay on your hand using MediaPipe.
  • A gesture recognition engine that identifies Open Hand, Fist, and Pointing gestures from landmark positions.
  • Secret Mission: Turn your gestures into a game controller that simulates keyboard inputs.

Are there any prerequisites?

No prior experience with computer vision or machine learning is needed. This guide walks you through installing every tool from scratch, including Python, MediaPipe, and OpenCV.

Before We Start

Before We Start...

Set Up Your Python Environment

Before you can track hand gestures through your webcam, you need a Python environment with the right tools installed.

In this step, you will install Python, set up an isolated workspace called a virtual environment, and install two key libraries: MediaPipe for hand detection and OpenCV for webcam video processing.

In this step, get ready to:
  • Install Python 3.11 and create a virtual environment.
  • Install MediaPipe and OpenCV packages.
  • Verify everything works by importing the libraries.
Install Python and create your project folder

Python is the programming language that powers your gesture controller. This project requires Python 3.11 or higher.

  • Open VS Code's integrated terminal by pressing Ctrl+` (macOS) or Ctrl+` (Windows).
  • Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.11 or higher

Your Python version is ready to go. Continue to the next instruction below.

ⓧ I see an older version

Your system has an older Python installed that will not work with this project's dependencies. You need to install Python 3.11 or higher.

  • Download the latest Python 3.11 installer from python.org/downloads.
  • Run the installer. On Windows, check the Add Python to PATH checkbox before clicking Install Now.
  • Close and reopen your terminal after installation completes.
  • Run the version check command again to confirm you see 3.11 or higher.

macOS users with an older default Python

After installing from python.org, macOS may still point python3 to the older version.

Try running python3.11 --version instead. If that works, use python3.11 in place of python3 for all commands in this project.

ⓧ Command not found

Python is not installed on your machine yet. Follow these steps to install it.

  • Go to python.org/downloads and download the latest Python 3.11 installer for your operating system.
  • Run the installer. On Windows, check the Add Python to PATH checkbox before clicking Install Now.
  • On macOS, open the downloaded .pkg file and follow the on-screen instructions.
  • Close and reopen your terminal after installation completes.
  • Run the version check command again to confirm Python is installed.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir hand-gesture-controller
cd hand-gesture-controller
  • Confirm the folder was created by running:
ls

The output should be empty since the folder is brand new.

  • Open this folder in VS Code by clicking File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the hand-gesture-controller folder, then click Open.

VS Code now has your project folder loaded. If a trust dialog appears, click Yes, I trust the authors.

  • Open a fresh terminal in VS Code by pressing Ctrl+` (macOS) or Ctrl+` (Windows).

This new terminal automatically starts inside your hand-gesture-controller folder.

A virtual environment is an isolated Python workspace. It keeps this project's packages separate from your system Python so nothing conflicts.

  • Create a virtual environment by running this command:

macOS

python3 -m venv venv

Windows

python -m venv venv
  • Confirm the virtual environment was created by running:
ls

You should see a venv folder in the output.

  • Activate the virtual environment by running this command:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

Don't see (venv) in your prompt?

On Windows, you may need to allow script execution first. Run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned in PowerShell, then try the activate command again.

Make sure you ran the activate command from inside the hand-gesture-controller folder where the venv folder exists.

Still stuck? Help me activate my Python virtual environment

Install MediaPipe and OpenCV

MediaPipe is Google's framework for real-time hand landmark detection. OpenCV handles the webcam video capture and display.

  • Install both packages by running this command:
pip install mediapipe opencv-python

What does this command do?

pip is Python's package manager. It downloads and installs libraries from the Python Package Index (PyPI).

mediapipe provides pre-trained machine learning models for hand tracking. opencv-python gives you tools to capture video from your webcam and display it on screen.

The installation may take a minute as it downloads the packages and their dependencies. Wait until you see your terminal prompt return.

Seeing an installation error?

Make sure your virtual environment is active. You should see (venv) at the start of your prompt. If not, run the activate command from the previous substep.

If you see a permissions error, do NOT use sudo. Instead, confirm you are inside the virtual environment.

Need more help? Help me fix my pip install error

Verify your installation

The best way to confirm the packages installed correctly is to import them in a Python shell. If no errors appear, everything is working.

  • Open a Python shell by running:
python

You should see a >>> prompt. This means you are inside the Python interpreter.

  • Test both imports by typing these lines one at a time:
import cv2
import mediapipe as mp
print(cv2.__version__)
print(mp.__version__)

What should I see?

You should see two version numbers printed with no error messages. The exact numbers may vary, but you should see something like 4.13.0.92 for OpenCV and 0.10.35 for MediaPipe.

If you see a ModuleNotFoundError, the package did not install correctly. Exit the shell and re-run the pip install command from the previous substep.

  • Exit the Python shell by typing:
exit()

Seeing ModuleNotFoundError?

This means the Python shell is not using the virtual environment where you installed the packages. Make sure (venv) appears in your terminal prompt before running python.

If you closed and reopened your terminal, you need to activate the virtual environment again before launching Python.

Still seeing errors? Help me fix my import errors for mediapipe and opencv

Your Python environment is ready with both libraries installed and verified. Next up, you will write your first script to capture live webcam video.

Capture Live Webcam Video

Your Python environment is ready with all the packages you need.

Now it's time to write code that actually does something visual. In this step, you'll create a Python script that opens your webcam and streams live video to your screen using OpenCV. This is the foundation that every computer vision project starts with.

In this step, get ready to:
  • Write a Python script that opens your webcam and displays a live video feed.
  • Add a mirror-effect frame flip so the display feels natural.
  • Test that the webcam feed displays correctly and closes cleanly.
Write your webcam capture script

Every computer vision app starts the same way: open the camera, read frames in a loop, and display them on screen. Your script will do exactly this, creating a live video window that you can close by pressing Q.

  • In VS Code, create a new file called hand_gesture_controller.py in your project folder.
  • Copy the following code into the file:
import cv2
import time

# Initialize webcam
cap = cv2.VideoCapture(0)

while cap.isOpened():
    success, frame = cap.read()
    if not success:
        break

    # Display frame
    cv2.imshow("Hand Gesture Controller", frame)

    # Quit on Q key
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

What does this code do?

  • cv2.VideoCapture(0) opens your default webcam (camera index 0).
  • cap.read() grabs one frame at a time inside a while loop, creating the illusion of live video.
  • cv2.imshow() displays each frame in a window titled "Hand Gesture Controller".
  • cv2.waitKey(1) waits 1 millisecond for a keypress. If the learner presses Q, the loop breaks and the script exits cleanly.
  • cap.release() and cv2.destroyAllWindows() free the webcam and close the display window when the loop ends.
  • Save hand_gesture_controller.py.

You won't see a change yet. The next substep adds the mirror flip, then you'll run and test the full script.

Seeing a red underline on import cv2?

Make sure your VS Code Python interpreter is set to the virtual environment you created in Step 1. Click the Python version in the bottom status bar and select the interpreter inside your project's virtual environment folder.

help me fix the cv2 import error in VS Code

Add frame flipping for a mirror effect

Right now, the webcam feed would display raw camera output. If you move your hand left, the on-screen hand moves right. That feels unnatural. A single line fix flips the frame horizontally so the display acts like a mirror.

  • In hand_gesture_controller.py, find this line inside your while loop:
    # Display frame
    cv2.imshow("Hand Gesture Controller", frame)
  • Add these two lines directly above # Display frame:
    # Flip frame for mirror effect
    frame = cv2.flip(frame, 1)

Why flip the frame?

cv2.flip(frame, 1) flips the image horizontally. The 1 means "flip around the y-axis" (left-right). This makes the video act like a mirror, so moving your hand left moves the on-screen hand left too.

  • Save hand_gesture_controller.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file, then move on to testing below.

ⓧ I'd like to double check the full code

Here's the complete hand_gesture_controller.py so far:

import cv2
import time

# Initialize webcam
cap = cv2.VideoCapture(0)

while cap.isOpened():
    success, frame = cap.read()
    if not success:
        break

    # Flip frame for mirror effect
    frame = cv2.flip(frame, 1)

    # Display frame
    cv2.imshow("Hand Gesture Controller", frame)

    # Quit on Q key
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()
Test your webcam feed

Time to see your script in action. You'll run the file and confirm that a live video window appears with a mirrored feed.

Camera permission on macOS

On first run, macOS will ask you to grant camera access to Terminal (or VS Code, depending on where you run the script). Click Allow when the system dialog appears. Without this permission, the webcam feed will show a blank frame or the script will fail silently.

  • Run your script by entering this command in your terminal:
python3 hand_gesture_controller.py
  • You should see a window titled "Hand Gesture Controller" appear with a live mirrored video feed from your webcam.
  • Wave your hand to confirm the feed is mirrored. Moving your hand left should move the on-screen hand left.
  • Press Q on your keyboard to quit. The window should close and your terminal returns to the prompt.

Don't see a video window?

If you're on macOS and the window is blank or doesn't appear, check that you granted camera permission. Go to System Settings then Privacy & Security then Camera and make sure Terminal (or VS Code) is toggled on.

If you see an error about the camera index, another app may be using your webcam. Close any video call or camera apps and try again.

help me fix my webcam feed not showing up

Your live webcam pipeline is working. Next up, you'll layer hand landmark detection on top of this feed so the app can actually see your hand and track every joint.

Detect Hand Landmarks

Your webcam feed is running and displaying live video. Now it's time to teach your app to actually understand what it sees.

In this step, you'll integrate Google's MediaPipe framework to detect 21 landmarks on your hand in real time. MediaPipe provides a pre-trained hand tracking model that works without a GPU, making it ideal for local computer vision projects.

In this step, get ready to:
  • Initialize MediaPipe Hands and process each video frame for hand detection.
  • Draw the 21-point hand skeleton overlay on your live video feed.
  • Test landmark detection by moving your hand in front of the camera.
Initialize MediaPipe Hands and process video frames

MediaPipe's Hands solution uses a machine learning model to find 21 specific points (called landmarks) on your hand. Before it can do this, you need to import the library, configure the model, and feed it each video frame in the right color format.

  • In hand_gesture_controller.py, find import cv2 at the top of your file.
  • Add import mediapipe as mp on the line directly after import cv2.
  • Below import time, add the following initialization code to set up MediaPipe Hands:
# Initialize MediaPipe Hands
mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils
hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.5
)

What does this code do?

This block configures the hand detection model before any video frames are processed.

  • mp_hands gives you access to the MediaPipe Hands solution (the pre-trained model).
  • mp_drawing provides utilities for drawing detected landmarks onto image frames.
  • max_num_hands=1 tells the model to track only one hand, which is faster and simpler for gesture recognition.
  • min_detection_confidence=0.7 means the model needs to be at least 70% sure it found a hand before reporting it.
  • Save hand_gesture_controller.py.

You won't see a change on screen yet. The next chunk connects this model to your video frames.

MediaPipe expects image data in RGB format, but OpenCV captures frames in BGR format. You need to convert each frame before passing it to the hand detector.

  • Inside the while cap.isOpened() loop, find this line:
    # Flip frame for mirror effect
    frame = cv2.flip(frame, 1)
  • Directly below the flip line, add the following frame processing code:
    h, w, _ = frame.shape

    # Convert BGR to RGB for MediaPipe
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgb_frame)

What does this code do?

This code prepares each frame for the hand detection model and runs the detection.

  • h, w, _ = frame.shape captures the frame's height and width for later use (you'll need these dimensions in upcoming steps).
  • cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) converts the frame from BGR (OpenCV's default) to RGB (what MediaPipe expects).
  • hands.process(rgb_frame) runs the hand detection model on the frame and stores any detected landmarks in results.
  • Save hand_gesture_controller.py.

You won't see a visible change yet. The detection is running, but the next substep draws the results on screen.

Draw the hand skeleton overlay

The detection results contain the x, y, and z coordinates of all 21 hand landmarks. Now you need to render these as a connected skeleton on your video feed so you can see the tracking in action.

  • In hand_gesture_controller.py, find the line results = hands.process(rgb_frame) you just added.
  • Directly below it, add the following code to draw detected landmarks onto the frame:
    # Process hand landmarks
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            # Draw hand skeleton
            mp_drawing.draw_landmarks(
                frame, hand_landmarks, mp_hands.HAND_CONNECTIONS
            )

What does this code do?

This block only runs when MediaPipe successfully detects a hand in the frame.

  • results.multi_hand_landmarks is None when no hand is detected. The if check prevents errors when your hand is out of frame.
  • The for loop iterates through detected hands (we configured max_num_hands=1, so this loops once at most).
  • mp_drawing.draw_landmarks() draws colored dots at each of the 21 landmark positions and connects them with lines based on HAND_CONNECTIONS (the natural bone structure of a hand).
  • Save hand_gesture_controller.py.
  • Run your script by entering this command in the terminal:
python3 hand_gesture_controller.py

Hold your hand in front of the webcam. You should see green dots and lines drawn on your hand, forming a connected skeleton that follows your hand movements.

Don't see the skeleton on your hand?

  • Make sure your hand is well-lit and clearly visible in the frame. Low light makes detection unreliable.
  • Check that the drawing code is placed BEFORE the cv2.imshow() line. If it comes after, the frame is displayed before landmarks are drawn.
  • On macOS, if you see a blank or black window, you may need to grant camera permission. Go to System Settings then Privacy & Security then Camera, and enable access for your terminal app.
  • Press Q to close the window and help me troubleshoot my hand skeleton not appearing.
Test landmark detection

With the skeleton drawing in place, take a moment to explore how the tracking responds to different hand positions.

  • Run the script if it's not already running.
  • Move your hand slowly toward and away from the camera. The skeleton should scale with your hand size.
  • Try opening and closing your fingers. Notice how individual landmark points track each fingertip.
  • Move your hand to the edges of the frame. The tracking may drop off when your hand is partially out of view.
  • Press Q to quit when you're done exploring.

What are the 21 landmarks?

MediaPipe Hands detects specific anatomical points: the wrist (landmark 0), thumb joints (1-4), index finger joints (5-8), middle finger joints (9-12), ring finger joints (13-16), and pinky joints (17-20).

Each landmark has an x, y, and z coordinate. In the next step, you'll compare these coordinates to determine which fingers are extended and which are curled.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete hand_gesture_controller.py so far:

import cv2
import mediapipe as mp
import time

# Initialize MediaPipe Hands
mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils
hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.5
)

# Initialize webcam
cap = cv2.VideoCapture(0)

while cap.isOpened():
    success, frame = cap.read()
    if not success:
        break

    # Flip frame for mirror effect
    frame = cv2.flip(frame, 1)
    h, w, _ = frame.shape

    # Convert BGR to RGB for MediaPipe
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgb_frame)

    # Process hand landmarks
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            # Draw hand skeleton
            mp_drawing.draw_landmarks(
                frame, hand_landmarks, mp_hands.HAND_CONNECTIONS
            )

    # Display frame
    cv2.imshow("Hand Gesture Controller", frame)

    # Quit on Q key
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Your app can now see and track your hand in real time. Next up, you'll build the gesture recognition engine that interprets those landmark positions into meaningful gestures like Open Hand, Fist, and Pointing.

Build the Gesture Recognition Engine

Your webcam now tracks 21 landmarks on your hand in real time. Those landmarks are raw coordinates in space, but they don't yet tell you what your hand is doing.

In this step, you'll build a gesture recognition engine that interprets landmark positions into named gestures. By comparing where fingertips sit relative to their joints, the engine determines which fingers are extended or curled, then maps those states to gestures like Open Hand, Fist, and Pointing.

In this step, get ready to:
  • Write a finger state detection function that checks which fingers are extended.
  • Define three named gestures based on finger states.
  • Display the detected gesture name on the live video feed.
Write the finger state detection function

MediaPipe's 21 landmarks have specific indices. Fingertips sit at landmarks 8, 12, 16, and 20. The joints just below them (PIP joints) sit at landmarks 6, 10, 14, and 18.

If a fingertip's y-coordinate is lower than its PIP joint (remember, y increases downward on screen), the finger is extended. The thumb is special because it moves laterally, so we compare x-coordinates instead.

  • In hand_gesture_controller.py, find the line that says cap = cv2.VideoCapture(0). Copy the following code below it, leaving one blank line between:
current_gesture = "None"
prev_gesture = "None"


def get_finger_states(hand_landmarks):
    """Determine which fingers are extended based on landmark positions."""
    finger_tips = [8, 12, 16, 20]  # Index, Middle, Ring, Pinky tip landmarks
    finger_pips = [6, 10, 14, 18]  # Corresponding PIP joints

    fingers = []

    # Thumb: compare tip x to IP joint x (works for right hand facing camera)
    thumb_tip = hand_landmarks.landmark[4]
    thumb_ip = hand_landmarks.landmark[3]
    if thumb_tip.x < thumb_ip.x:
        fingers.append(1)
    else:
        fingers.append(0)

    # Other fingers: compare tip y to PIP joint y (lower y = higher on screen)
    for tip, pip in zip(finger_tips, finger_pips):
        if hand_landmarks.landmark[tip].y < hand_landmarks.landmark[pip].y:
            fingers.append(1)
        else:
            fingers.append(0)

    return fingers

What does this code do?

current_gesture and prev_gesture track what gesture is detected each frame. You'll use prev_gesture later to detect when a gesture changes.

The get_finger_states() function loops through each finger and checks whether its tip is above its PIP joint. It returns a list of 5 values (one per finger), where 1 means extended and 0 means curled.

Why is the thumb handled differently?

The other four fingers curl vertically, so comparing y-coordinates works. The thumb moves sideways relative to the palm, so you compare the x-coordinate of landmark 4 (thumb tip) against landmark 3 (thumb IP joint) instead.

  • Save hand_gesture_controller.py.

Seeing a syntax error?

  • Make sure the function is placed after the cap = cv2.VideoCapture(0) line and before the while cap.isOpened(): loop.
  • Check that the indentation inside the function uses 4 spaces consistently. Mixing tabs and spaces causes Python errors.

Help me fix a syntax error in my finger state detection function.

Define the gesture recognition function

Now that you can detect which fingers are up or down, you need a function that translates those five finger states into a named gesture.

  • In hand_gesture_controller.py, copy the following function directly below get_finger_states(), leaving one blank line between:
def recognize_gesture(fingers):
    """Map finger states to gesture names."""
    if fingers == [1, 1, 1, 1, 1]:
        return "Open Hand"
    elif fingers == [0, 0, 0, 0, 0]:
        return "Fist"
    elif fingers[1] == 1 and fingers[2] == 0 and fingers[3] == 0 and fingers[4] == 0:
        return "Pointing"
    else:
        return "Unknown"

What does this code do?

  • [1, 1, 1, 1, 1] means all five fingers are extended. That's an Open Hand.
  • [0, 0, 0, 0, 0] means all fingers are curled. That's a Fist.
  • The Pointing check only requires index finger (position 1) to be extended while middle, ring, and pinky are curled. The thumb state is ignored for this gesture.
  • Any other combination returns "Unknown" as a fallback.
  • Save hand_gesture_controller.py.

IndentationError when saving?

  • Confirm there is exactly one blank line between the end of get_finger_states() and the def recognize_gesture(fingers): line.
  • Make sure the function starts at column 0 (no indentation on the def line). It should be at the same level as get_finger_states().

Help me fix an indentation error in my gesture recognition code.

Display the detected gesture on the video feed

The two functions are ready. Now you need to call them inside the main loop and show the result on screen using OpenCV's text rendering.

  • In hand_gesture_controller.py, find the # Process hand landmarks section inside the while loop. It currently looks like this:
    # Process hand landmarks
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            # Draw hand skeleton
            mp_drawing.draw_landmarks(
                frame, hand_landmarks, mp_hands.HAND_CONNECTIONS
            )

    # Display frame
    cv2.imshow("Hand Gesture Controller", frame)
  • Replace that entire section (from # Process hand landmarks down to but not including # Display frame) with the following code:
    # Process hand landmarks
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            # Draw hand skeleton
            mp_drawing.draw_landmarks(
                frame, hand_landmarks, mp_hands.HAND_CONNECTIONS
            )

            # Recognize gesture
            fingers = get_finger_states(hand_landmarks)
            current_gesture = recognize_gesture(fingers)
    else:
        current_gesture = "No Hand"

    # Display gesture name
    cv2.putText(frame, current_gesture, (45, 38),
                cv2.FONT_HERSHEY_SIMPLEX, 0.9, (255, 255, 255), 2)

    prev_gesture = current_gesture

What does this code do?

  • Inside the for loop, after drawing the skeleton, you call get_finger_states() and recognize_gesture() to identify the current gesture.
  • The else clause sets the gesture to "No Hand" when MediaPipe detects nothing in the frame.
  • cv2.putText() draws the gesture name directly onto the video frame at position (45, 38) in white text.
  • prev_gesture = current_gesture stores the current gesture so you can detect changes in the next step.
  • Save hand_gesture_controller.py.
  • Run your script by running:
python hand_gesture_controller.py
  • Hold your hand open in front of the camera. You should see "Open Hand" displayed on the video feed.
  • Make a fist. The text should change to "Fist".
  • Point with your index finger. The text should change to "Pointing".
  • Press Q to close the window when you're done testing.

Gesture not changing when you move your hand?

  • Make sure the gesture recognition code is indented inside the for loop (8 spaces in from the left). If it's at the wrong indent level, it won't have access to hand_landmarks.
  • Check that the else clause aligns with the if results.multi_hand_landmarks: line (4 spaces in). It belongs to the if block, not the for loop.
  • Try holding your hand further from the camera. If your hand fills the entire frame, landmark detection can be unreliable.

Help me debug why my gesture detection isn't working.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here's the full hand_gesture_controller.py as it should look at the end of this step:

import cv2
import mediapipe as mp
import time

# Initialize MediaPipe Hands
mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils
hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.5
)

# Initialize webcam
cap = cv2.VideoCapture(0)

current_gesture = "None"
prev_gesture = "None"


def get_finger_states(hand_landmarks):
    """Determine which fingers are extended based on landmark positions."""
    finger_tips = [8, 12, 16, 20]  # Index, Middle, Ring, Pinky tip landmarks
    finger_pips = [6, 10, 14, 18]  # Corresponding PIP joints

    fingers = []

    # Thumb: compare tip x to IP joint x (works for right hand facing camera)
    thumb_tip = hand_landmarks.landmark[4]
    thumb_ip = hand_landmarks.landmark[3]
    if thumb_tip.x < thumb_ip.x:
        fingers.append(1)
    else:
        fingers.append(0)

    # Other fingers: compare tip y to PIP joint y (lower y = higher on screen)
    for tip, pip in zip(finger_tips, finger_pips):
        if hand_landmarks.landmark[tip].y < hand_landmarks.landmark[pip].y:
            fingers.append(1)
        else:
            fingers.append(0)

    return fingers


def recognize_gesture(fingers):
    """Map finger states to gesture names."""
    if fingers == [1, 1, 1, 1, 1]:
        return "Open Hand"
    elif fingers == [0, 0, 0, 0, 0]:
        return "Fist"
    elif fingers[1] == 1 and fingers[2] == 0 and fingers[3] == 0 and fingers[4] == 0:
        return "Pointing"
    else:
        return "Unknown"


while cap.isOpened():
    success, frame = cap.read()
    if not success:
        break

    # Flip frame for mirror effect
    frame = cv2.flip(frame, 1)
    h, w, _ = frame.shape

    # Convert BGR to RGB for MediaPipe
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgb_frame)

    # Process hand landmarks
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            # Draw hand skeleton
            mp_drawing.draw_landmarks(
                frame, hand_landmarks, mp_hands.HAND_CONNECTIONS
            )

            # Recognize gesture
            fingers = get_finger_states(hand_landmarks)
            current_gesture = recognize_gesture(fingers)
    else:
        current_gesture = "No Hand"

    # Display gesture name
    cv2.putText(frame, current_gesture, (45, 38),
                cv2.FONT_HERSHEY_SIMPLEX, 0.9, (255, 255, 255), 2)

    prev_gesture = current_gesture

    # Display frame
    cv2.imshow("Hand Gesture Controller", frame)

    # Quit on Q key
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Your gesture engine is live. You can now wave, point, or make a fist and see the app recognize each one in real time. Next up, you'll add a polished heads-up display with an FPS counter and color-coded gesture indicators.

Add the Heads-Up Display

Your gesture recognition engine works, but the raw text label floating on the video feed is hard to read.

In this step, you will build a proper heads-up display (HUD) that gives you real-time performance metrics, color-coded gesture indicators, and visual feedback when a gesture changes.

In this step, get ready to:
  • Add an FPS counter to monitor real-time performance.
  • Create a styled overlay with a colored gesture indicator.
  • Add visual feedback when a gesture changes.
Add an FPS counter to monitor real-time performance

FPS (frames per second) tells you how fast your pipeline processes video. A higher FPS means smoother, more responsive tracking.

To calculate FPS, you measure the time between consecutive frames. You will also set up a color variable for the HUD indicator you will build next.

  • In hand_gesture_controller.py, find these two lines near the top of the file:
current_gesture = "None"
prev_gesture = "None"
  • Add a comment and the two new variables above and below so the section now looks like this:
# Variables for FPS calculation
prev_time = 0
current_gesture = "None"
prev_gesture = "None"
gesture_color = (200, 200, 200)

What do these variables do?

  • prev_time stores the timestamp of the last frame so you can calculate elapsed time.
  • gesture_color starts as a neutral gray and will update each frame to match the detected gesture.

Now you will replace the old display code with FPS calculation logic.

  • Scroll down to the while loop and find this section:
    # Display gesture name
    cv2.putText(frame, current_gesture, (45, 38),
                cv2.FONT_HERSHEY_SIMPLEX, 0.9, (255, 255, 255), 2)

    prev_gesture = current_gesture
  • Delete those five lines entirely.
  • In their place (between the current_gesture = "No Hand" line and the # Display frame line), add the FPS calculation by pasting this code:
    # Calculate FPS
    current_time = time.time()
    fps = int(1 / (current_time - prev_time)) if (current_time - prev_time) > 0 else 0
    prev_time = current_time

What does this code do?

  • time.time() grabs the current timestamp in seconds.
  • Dividing 1 by the elapsed time between frames gives frames per second.
  • The conditional guard prevents a division-by-zero error on the very first frame.
  • Save hand_gesture_controller.py.

The FPS is now calculated each frame but not displayed yet. You will render it on screen in the next substep.

Seeing a syntax error?

Make sure the FPS code is indented with exactly 4 spaces to match the while loop body.

Check you deleted the old # Display gesture name block completely, including the prev_gesture = current_gesture line below it.

help me debug a syntax error in my FPS calculation code

Create a styled overlay showing the current gesture with a colored indicator

A good HUD needs to be instantly readable. You will draw a black bar across the top of the frame, then place a color-coded circle and text labels on it.

  • In hand_gesture_controller.py, add the following code directly below the prev_time = current_time line you just added:
    # Set gesture color
    if current_gesture == "Open Hand":
        gesture_color = (0, 255, 0)
    elif current_gesture == "Fist":
        gesture_color = (0, 0, 255)
    elif current_gesture == "Pointing":
        gesture_color = (255, 165, 0)
    else:
        gesture_color = (200, 200, 200)

    # Draw HUD background
    cv2.rectangle(frame, (0, 0), (w, 60), (0, 0, 0), -1)

    # Draw gesture name with color indicator
    cv2.circle(frame, (25, 30), 12, gesture_color, -1)
    cv2.putText(frame, current_gesture, (45, 38),
                cv2.FONT_HERSHEY_SIMPLEX, 0.9, (255, 255, 255), 2)

    # Draw FPS
    cv2.putText(frame, f"FPS: {fps}", (w - 120, 38),
                cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)

What does this code do?

  • The if/elif block picks a color based on the gesture: green for Open Hand, red for Fist, orange for Pointing, and gray for anything else.
  • cv2.rectangle with -1 as the last argument draws a filled black bar across the top 60 pixels of the frame.
  • cv2.circle draws a small filled dot in the gesture color, acting as a quick visual signal.
  • The two cv2.putText calls render the gesture name on the left and the FPS counter on the right.
  • Save hand_gesture_controller.py and run the script:
python hand_gesture_controller.py

You should see a black bar across the top of the video feed with a colored circle on the left, the gesture name next to it, and an FPS number on the right.

HUD not visible or overlapping the video?

Ensure the HUD code appears after the FPS calculation and before the # Display frame line.

If the circle or text appears in the wrong position, double check that the coordinate values match the code above exactly.

help me fix my HUD overlay positioning

Add visual feedback when a gesture changes

Right now the HUD updates silently. A brief highlight bar when the gesture changes draws the eye and confirms the system responded.

  • In hand_gesture_controller.py, add the following code directly below the # Draw FPS putText line (the last line you added in the previous substep):
    # Highlight on gesture change
    if current_gesture != prev_gesture and current_gesture not in ["None", "No Hand", "Unknown"]:
        cv2.rectangle(frame, (0, 55), (w, 65), gesture_color, -1)
    prev_gesture = current_gesture

What does this code do?

The condition checks two things: has the gesture changed since the last frame, and is it a meaningful gesture (not a placeholder like "None" or "No Hand").

When both are true, a thin colored bar flashes just below the HUD for one frame. Then prev_gesture updates so the highlight only fires once per change.

  • Save hand_gesture_controller.py and run the script:
python hand_gesture_controller.py

Switch between gestures (open hand, fist, pointing finger). Each time you change, a brief colored bar should flash below the HUD.

Not seeing the highlight bar?

The bar only appears for a single frame, so it is very brief. Try switching gestures slowly and watch closely at the bottom edge of the black bar.

Make sure prev_gesture = current_gesture appears after the if-block, not before it. If it is above the condition, the comparison will always be equal and the bar will never show.

help me debug my gesture change highlight

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

Here is the complete hand_gesture_controller.py file for comparison:

import cv2
import mediapipe as mp
import time

# Initialize MediaPipe Hands
mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils
hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.5
)

# Initialize webcam
cap = cv2.VideoCapture(0)

# Variables for FPS calculation
prev_time = 0
current_gesture = "None"
prev_gesture = "None"
gesture_color = (200, 200, 200)


def get_finger_states(hand_landmarks):
    """Determine which fingers are extended based on landmark positions."""
    finger_tips = [8, 12, 16, 20]  # Index, Middle, Ring, Pinky tip landmarks
    finger_pips = [6, 10, 14, 18]  # Corresponding PIP joints

    fingers = []

    # Thumb: compare tip x to IP joint x (works for right hand facing camera)
    thumb_tip = hand_landmarks.landmark[4]
    thumb_ip = hand_landmarks.landmark[3]
    if thumb_tip.x < thumb_ip.x:
        fingers.append(1)
    else:
        fingers.append(0)

    # Other fingers: compare tip y to PIP joint y (lower y = higher on screen)
    for tip, pip in zip(finger_tips, finger_pips):
        if hand_landmarks.landmark[tip].y < hand_landmarks.landmark[pip].y:
            fingers.append(1)
        else:
            fingers.append(0)

    return fingers


def recognize_gesture(fingers):
    """Map finger states to gesture names."""
    if fingers == [1, 1, 1, 1, 1]:
        return "Open Hand"
    elif fingers == [0, 0, 0, 0, 0]:
        return "Fist"
    elif fingers[1] == 1 and fingers[2] == 0 and fingers[3] == 0 and fingers[4] == 0:
        return "Pointing"
    else:
        return "Unknown"


while cap.isOpened():
    success, frame = cap.read()
    if not success:
        break

    # Flip frame for mirror effect
    frame = cv2.flip(frame, 1)
    h, w, _ = frame.shape

    # Convert BGR to RGB for MediaPipe
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgb_frame)

    # Process hand landmarks
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            # Draw hand skeleton
            mp_drawing.draw_landmarks(
                frame, hand_landmarks, mp_hands.HAND_CONNECTIONS
            )

            # Recognize gesture
            fingers = get_finger_states(hand_landmarks)
            current_gesture = recognize_gesture(fingers)
    else:
        current_gesture = "No Hand"

    # Calculate FPS
    current_time = time.time()
    fps = int(1 / (current_time - prev_time)) if (current_time - prev_time) > 0 else 0
    prev_time = current_time

    # Set gesture color
    if current_gesture == "Open Hand":
        gesture_color = (0, 255, 0)
    elif current_gesture == "Fist":
        gesture_color = (0, 0, 255)
    elif current_gesture == "Pointing":
        gesture_color = (255, 165, 0)
    else:
        gesture_color = (200, 200, 200)

    # Draw HUD background
    cv2.rectangle(frame, (0, 0), (w, 60), (0, 0, 0), -1)

    # Draw gesture name with color indicator
    cv2.circle(frame, (25, 30), 12, gesture_color, -1)
    cv2.putText(frame, current_gesture, (45, 38),
                cv2.FONT_HERSHEY_SIMPLEX, 0.9, (255, 255, 255), 2)

    # Draw FPS
    cv2.putText(frame, f"FPS: {fps}", (w - 120, 38),
                cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)

    # Highlight on gesture change
    if current_gesture != prev_gesture and current_gesture not in ["None", "No Hand", "Unknown"]:
        cv2.rectangle(frame, (0, 55), (w, 65), gesture_color, -1)
    prev_gesture = current_gesture

    # Display frame
    cv2.imshow("Hand Gesture Controller", frame)

    # Quit on Q key
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Your hand gesture controller now has a polished heads-up display with real-time performance monitoring and visual feedback. Next up, take on the Secret Mission and turn your gestures into actual keyboard controls.

Secret mission

Turn Your Gestures Into a Game Controller

Your gesture detector already knows what your hand is doing. What if it could translate those gestures into keyboard presses? In this secret mission, you will wire up PyAutoGUI so your hand movements control the Chrome dinosaur game in real time.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • Your project folder on the Desktop (contains hand_gesture_controller.py and your Python virtual environment with mediapipe, opencv-python, and pyautogui installed).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your gesture controller or want to build on the project.

  • Your project folder stays on your Desktop and takes up minimal disk space.
  • The virtual environment keeps your packages isolated from the rest of your system.
  • You can run hand_gesture_controller.py anytime by activating the virtual environment and running the script.

Pause - I'll come back to this later

Close any running processes to free up system resources. Your files stay intact for when you return.

  • If the gesture controller is still running, press Q in the video window to close it.
  • If your virtual environment is active in the terminal, deactivate it by running:
deactivate
  • Close VS Code if you are done for now.

Delete - I don't want to use this again

Remove all project files from your machine. This permanently deletes your code and installed packages.

  • If the gesture controller is still running, press Q in the video window to close it.
  • If your virtual environment is active, deactivate it first by running:
deactivate
  • Navigate to your Desktop and delete the project folder by running:
cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]
  • On Windows, use this command instead:
cd ~\Desktop
rmdir /s /q [[PROJECT_FOLDER="your project folder name"]]
  • Confirm the folder is gone by running ls (macOS/Linux) or dir (Windows) on your Desktop. Your project folder should no longer appear.

Nice Work!

Nice Work!

You just built a real-time computer vision application from scratch. Your webcam now tracks your hand, detects gestures, and responds to your movements instantly.

Here's what you learned:

  • Built a live webcam capture pipeline using OpenCV to process video frames in real time.
  • Integrated Google's MediaPipe to detect 21 hand landmarks and draw a skeleton overlay on your hand.
  • Created a gesture recognition engine that interprets spatial landmark data to identify Open Hand, Fist, and Pointing gestures.
  • Turned your webcam into a game controller by mapping gestures to keyboard inputs with PyAutoGUI.

Ready to quiz yourself?