Build a Gesture-Controlled Virtual Mouse

Use your webcam and hand gestures to control your mouse with Python.

Introduction

30 Second Summary

You use your mouse thousands of times a day, but what if you could just wave your hand in the air and have the computer respond?

In this project, you will build a gesture-controlled virtual mouse that uses your laptop's webcam to track your hand and translate finger movements into real cursor actions like clicking, scrolling, and dragging.

What You'll Build

A Python application that captures your live webcam feed, detects your hand using a machine learning model, and maps your finger positions to mouse actions in real-time.

By the end of this project, you'll have:

  • A working Python script that tracks your hand via webcam and moves the mouse cursor in real-time using your index finger.
  • Gesture-based controls for left click, right click, and scrolling using pinch and finger-pattern detection.
  • A visual debug overlay showing the hand skeleton, active gesture label, and mode indicators on your webcam feed.
  • A secret mission to add drag-and-drop and volume control gestures to your virtual mouse.

Are there any prerequisites?

You should have basic Python experience with variables, functions, and loops. You'll also need Python (version 3.9 to 3.12) and a code editor like VS Code already installed on your machine, along with a working webcam.

Install Libraries and Verify Your Webcam

Before you can track hand gestures, you need three libraries working together. OpenCV captures video from your webcam. MediaPipe detects 21 landmarks on your hand using a pre-trained ML model. PyAutoGUI translates those landmarks into real mouse actions on your computer. In this step, you will install all three and confirm your webcam and hand detection are working.

In This Step, Get Ready To:
  • Install OpenCV, MediaPipe, PyAutoGUI, and NumPy via pip.
  • Open your webcam with OpenCV and display a live video feed.
  • Run MediaPipe hand detection and see a 21-point hand skeleton drawn on your camera feed.
Install the Required Python Packages

Your project depends on four packages. opencv-python handles webcam capture and image display. mediapipe provides the hand landmark detection model. pyautogui controls your mouse cursor programmatically. numpy supports the numerical operations behind the scenes.

macOS

  • Open your terminal and run the following command.
pip3 install opencv-python mediapipe pyautogui numpy

PyAutoGUI Needs Accessibility Permissions on macOS

PyAutoGUI needs Accessibility permissions to control your mouse. Without this, mouse movement commands will silently fail.

  • Open System Settings.
  • Navigate to Privacy & Security, then Accessibility.
  • Add your Terminal app (or VS Code) to the allowed list.

Windows

  • Open your terminal and run the following command.
pip install opencv-python mediapipe pyautogui numpy

Linux

  • PyAutoGUI depends on a few system-level packages on Linux. Install them first.
sudo apt-get install scrot python3-tk python3-dev
  • Then install the Python packages.
pip install opencv-python mediapipe pyautogui numpy

It worked

You should see Successfully installed messages for each package. You are ready to move on.

I see an error

Having Trouble Installing?

Make sure you are using Python 3.9 through 3.12. MediaPipe does not support all Python versions. Run python --version (or python3 --version on macOS) to check.

If you are still stuck, help me troubleshoot my pip install errors.

Verify Your Webcam Works With OpenCV

Before adding hand detection, you need to confirm that OpenCV can access your webcam and display a live video feed. This quick test script opens your camera, shows the frames in a window, and closes when you press q.

  • Create a new file called test_cam.py in your project folder.
  • Paste the following code into the file.
import cv2

# Open the default webcam (index 0)
cap = cv2.VideoCapture(0)

while cap.isOpened():
    success, frame = cap.read()
    if not success:
        print("Failed to read from webcam.")
        break

    # Display the live camera feed
    cv2.imshow("Webcam Test", frame)

    # Press 'q' to quit the window
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

What Does This Code Do?

cv2.VideoCapture(0) opens your default webcam. The 0 refers to the first camera device on your system. The while loop continuously reads frames and displays them using cv2.imshow(). When you press q, the loop exits and the webcam is released.

  • Run the script from your terminal.
python test_cam.py

A window titled "Webcam Test" should appear showing your live camera feed. Wave at the camera to confirm the video is updating in real time.

  • Press q to close the window when you are done.

Webcam Window Not Appearing?

If the window does not appear or you see "Failed to read from webcam", your camera may be in use by another application. Close any apps using the camera (Zoom, FaceTime, etc.) and try again. On macOS, you may also need to grant camera permissions to your Terminal or IDE.

If your system has multiple cameras, try changing VideoCapture(0) to VideoCapture(1). Still stuck? Help me debug my webcam issue.

Verify MediaPipe Hand Detection

Now that your webcam works, it is time to add MediaPipe hand detection on top of the video feed. MediaPipe's hand tracking model detects 21 landmarks on each hand, from the wrist to every fingertip. You will see these landmarks drawn as colored dots connected by lines, forming a hand skeleton overlay.

  • Replace the contents of test_cam.py with the following code.
import cv2
import mediapipe as mp

# Initialize MediaPipe Hands
mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils
hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.5
)

# Open the default webcam
cap = cv2.VideoCapture(0)

while cap.isOpened():
    success, frame = cap.read()
    if not success:
        print("Failed to read from webcam.")
        break

    # MediaPipe expects RGB, but OpenCV captures in BGR
    frame.flags.writeable = False
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgb_frame)
    frame.flags.writeable = True

    # Draw hand landmarks if a hand is detected
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            mp_drawing.draw_landmarks(
                frame,
                hand_landmarks,
                mp_hands.HAND_CONNECTIONS
            )

    cv2.imshow("Hand Detection Test", frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

hands.close()
cap.release()
cv2.destroyAllWindows()

What Does This Code Do?

This script builds on the webcam test by adding MediaPipe's hand detection pipeline. Here is what each new piece does:

  • mp.solutions.hands.Hands() initializes the hand detection model. max_num_hands=1 limits detection to one hand for better performance. min_detection_confidence=0.7 means the model must be at least 70% confident a hand is present before reporting it.
  • cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) converts the frame from BGR (OpenCV's default) to RGB, which is what MediaPipe expects.
  • frame.flags.writeable = False before processing improves performance by passing the image by reference instead of copying it.
  • mp_drawing.draw_landmarks() draws the 21 detected landmarks and the connections between them directly onto the frame.
  • Run the updated script.
python test_cam.py
  • Hold your hand up in front of the webcam.

You should see 21 colored dots appear on your hand, connected by lines forming a skeleton. Try moving your fingers and watch the landmarks follow in real time.

What Are These 21 Landmarks?

MediaPipe tracks 21 specific points on your hand. The key ones you will use throughout this project are:

  • Landmark 0: Wrist.
  • Landmark 4: Thumb tip.
  • Landmark 8: Index finger tip.
  • Landmark 12: Middle finger tip.
  • Landmark 16: Ring finger tip.
  • Landmark 20: Pinky tip.

Each landmark has an x and y value normalized between 0.0 and 1.0, representing its position relative to the frame. You will map these coordinates to your screen in the next step.

  • Press q to close the window.

Your webcam, OpenCV, and MediaPipe are all working together. Next up, you will use those hand landmarks to control your mouse cursor in real time.

Track Your Index Finger and Move the Cursor

Your webcam is working and MediaPipe can see your hand. Now it is time to make your hand actually control something. In this step, you will extract the position of your index finger tip, map it to your screen coordinates, and move the mouse cursor in real-time.

In This Step, Get Ready To:
  • Extract the index finger tip position from MediaPipe hand landmarks.
  • Map normalized finger coordinates to screen pixels and move the mouse cursor with PyAutoGUI.
  • Add a visual debug overlay showing the tracked finger position on your webcam feed.
Extract the Index Finger Tip Position

MediaPipe detects 21 landmarks on each hand. Each landmark has an x and y value normalized between 0.0 and 1.0. The one you care about right now is landmark #8, the index finger tip. You will build your main project file and pull out that landmark's coordinates.

What Are Hand Landmarks?

Each detected hand gets 21 numbered points. The five fingertip landmarks are the ones you will use most:

  • Landmark 4: Thumb tip.
  • Landmark 8: Index finger tip.
  • Landmark 12: Middle finger tip.
  • Landmark 16: Ring finger tip.
  • Landmark 20: Pinky tip.
  • Create a new file called gesture_mouse.py.
  • Add the following code to set up the webcam, initialize MediaPipe Hands, and extract the index finger tip coordinates:
import cv2
import mediapipe as mp
import pyautogui

# Initialize MediaPipe Hands
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.7
)
mp_draw = mp.solutions.drawing_utils

# Open webcam
cap = cv2.VideoCapture(0)

while cap.isOpened():
    success, frame = cap.read()
    if not success:
        break

    # Flip the frame horizontally for a mirror view
    frame = cv2.flip(frame, 1)

    # Convert BGR to RGB for MediaPipe
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    rgb_frame.flags.writeable = False
    results = hands.process(rgb_frame)
    rgb_frame.flags.writeable = True

    if results.multi_hand_landmarks:
        hand_landmarks = results.multi_hand_landmarks[0]

        # Draw the hand skeleton
        mp_draw.draw_landmarks(frame, hand_landmarks, mp_hands.HAND_CONNECTIONS)

        # Extract index finger tip (landmark 8)
        index_tip = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]
        print(f"Index finger tip - x: {index_tip.x:.3f}, y: {index_tip.y:.3f}")

    cv2.imshow("Gesture Mouse", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

What Does This Code Do?

This script opens your webcam and flips each frame horizontally so it behaves like a mirror. It converts each frame from BGR to RGB (which MediaPipe requires), then runs hand detection. If a hand is found, it draws the skeleton overlay and prints the index finger tip's normalized coordinates to the console.

Setting rgb_frame.flags.writeable = False before processing improves performance by passing the frame by reference instead of copying it.

  • Run the script with python gesture_mouse.py (or python3 on macOS).
  • Hold your hand up to the webcam and watch the console output.

You should see a stream of coordinates like Index finger tip - x: 0.452, y: 0.318 updating as you move your finger around. The values stay between 0.0 and 1.0 regardless of your webcam resolution.

Map Finger Coordinates to Screen Coordinates

Normalized coordinates between 0 and 1 are not useful on their own. Your screen has a specific pixel resolution (like 1920x1080), so you need to convert those decimal values into actual pixel positions. You also need smoothing to prevent the cursor from jittering as the landmark position fluctuates between frames.

Why Does the Cursor Need Smoothing?

MediaPipe's landmark detection has small fluctuations from frame to frame, even when your hand is perfectly still. Without smoothing, these micro-movements translate into a shaky, unusable cursor. An exponential moving average blends each new position with the previous one, creating fluid motion.

  • Update gesture_mouse.py to add screen coordinate mapping and cursor movement. Replace the content of your file with the following:
import cv2
import mediapipe as mp
import pyautogui

# Disable PyAutoGUI's fail-safe (moving mouse to corner won't crash)
pyautogui.FAILSAFE = False

# Initialize MediaPipe Hands
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.7
)
mp_draw = mp.solutions.drawing_utils

# Get screen dimensions
screen_w, screen_h = pyautogui.size()

# Smoothing variables
smoothing = 5
prev_x, prev_y = 0, 0

# Open webcam
cap = cv2.VideoCapture(0)

while cap.isOpened():
    success, frame = cap.read()
    if not success:
        break

    frame = cv2.flip(frame, 1)
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    rgb_frame.flags.writeable = False
    results = hands.process(rgb_frame)
    rgb_frame.flags.writeable = True

    if results.multi_hand_landmarks:
        hand_landmarks = results.multi_hand_landmarks[0]
        mp_draw.draw_landmarks(frame, hand_landmarks, mp_hands.HAND_CONNECTIONS)

        # Extract index finger tip
        index_tip = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]

        # Map normalized coordinates to screen pixels
        target_x = index_tip.x * screen_w
        target_y = index_tip.y * screen_h

        # Apply exponential moving average for smoothing
        curr_x = prev_x + (target_x - prev_x) / smoothing
        curr_y = prev_y + (target_y - prev_y) / smoothing

        # Move the mouse cursor
        pyautogui.moveTo(int(curr_x), int(curr_y))

        # Store current position for next frame
        prev_x, prev_y = curr_x, curr_y

    cv2.imshow("Gesture Mouse", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

What Does the Smoothing Formula Do?

The line curr_x = prev_x + (target_x - prev_x) / smoothing is an exponential moving average. Instead of jumping directly to the new position, the cursor moves a fraction of the distance each frame. A smoothing value of 5 means the cursor moves 1/5 of the remaining distance per frame. Higher values make the cursor smoother but introduce more lag.

  • Run the updated script.
  • Hold up your index finger and move it around.

Your mouse cursor should now follow your index finger across the screen. The movement should feel smooth and responsive. If the cursor feels too sluggish, try lowering the smoothing value to 3. If it feels too jittery, try raising it to 7.

Cursor Not Moving on macOS?

PyAutoGUI needs Accessibility permissions on macOS to control the mouse.

  • Open System Settings.
  • Select Privacy & Security, then Accessibility.
  • Add your Terminal app or VS Code to the allowed list.

Without this permission, mouse movement commands silently fail with no error message. Still stuck? Help me troubleshoot my cursor not moving.

Add a Visual Debug Overlay

The hand skeleton is helpful, but you also want to see exactly where the system thinks your index finger tip is. Drawing a highlighted circle on the fingertip and displaying the mapped screen coordinates gives you a clear visual reference for debugging and calibration.

  • Add the following lines inside the if results.multi_hand_landmarks: block, right after the prev_x, prev_y = curr_x, curr_y line:
        # Get the finger tip position in frame pixel coordinates
        frame_h, frame_w, _ = frame.shape
        cx, cy = int(index_tip.x * frame_w), int(index_tip.y * frame_h)

        # Draw a circle at the index finger tip
        cv2.circle(frame, (cx, cy), 12, (0, 255, 0), cv2.FILLED)

        # Display the mapped screen coordinates on the frame
        coord_text = f"Screen: ({int(curr_x)}, {int(curr_y)})"
        cv2.putText(frame, coord_text, (10, 30),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 255, 0), 2)

What Does This Overlay Show?

The green circle marks exactly where MediaPipe detects your index finger tip on the webcam frame. The text in the top-left corner shows the corresponding screen pixel coordinates that PyAutoGUI is sending the cursor to.

If the circle tracks your finger accurately but the cursor feels off, the issue is in the coordinate mapping. If the circle itself jumps around, the issue is in MediaPipe's detection confidence.

  • Run the script again and verify that a green dot follows your index finger tip.
  • Check that the screen coordinates displayed in the top-left corner update as you move your hand.

You should see a bright green dot locked onto your fingertip, with coordinates like Screen: (960, 540) updating in real-time as you move.

Your index finger is now a working mouse pointer. Next up, you will build a finger state detector that can tell which fingers are up or down. This is the foundation for recognizing gestures like pinches and scrolls.

Build a Finger State Detector

Right now your virtual mouse can move the cursor, but it has no idea what your hand is actually doing. Is your fist closed? Are two fingers raised? To recognize gestures like clicks and scrolls, you first need a way to detect which fingers are up and which are curled down. In this step, you'll build a reusable helper module that answers that question for every frame.

In This Step, Get Ready To:
  • Create a gesture_utils.py helper module with a finger state detection function.
  • Test the detector by displaying live finger states on your webcam feed.
  • Add a distance calculator function for detecting pinch gestures.
Create the Gesture Utils Helper Module

Gesture recognition starts with one fundamental question: which fingers are up? MediaPipe gives you 21 landmarks per hand, but it doesn't tell you whether a finger is extended or curled. You need to figure that out yourself by comparing the positions of each fingertip to its corresponding joint.

  • Create a new file called gesture_utils.py in the same directory as your main script.
  • Write a function called fingers_up(hand_landmarks) that takes a MediaPipe hand landmarks object and returns a list of 5 booleans.

The detection logic is different for the thumb versus the other four fingers. The thumb moves laterally (side to side), while the other fingers move vertically. Here's how to check each one:

  • Thumb: Compare landmark 4 (thumb tip) x-position to landmark 3 (IP joint). If the tip is further from the palm center than the joint, the thumb is extended.
  • Index, middle, ring, pinky: Compare each fingertip's y-position to its PIP joint. The tip landmarks are 8, 12, 16, and 20. The PIP joint landmarks are 6, 10, 14, and 18. If the tip y-value is less than the joint y-value, the finger is up.

Why is a lower y-value "up"?

In image coordinates, y increases downward. The top of the frame is y = 0, and the bottom is y = 1.0. So when a fingertip has a smaller y-value than its joint, the tip is physically higher in the frame. That means the finger is extended.

Add the following code to gesture_utils.py:

import mediapipe as mp

mp_hands = mp.solutions.hands

# Landmark indices for fingertips and their corresponding PIP joints
TIP_IDS = [4, 8, 12, 16, 20]
PIP_IDS = [3, 6, 10, 14, 18]

def fingers_up(hand_landmarks):
    """Return a list of 5 booleans: True if the finger is extended."""
    landmarks = hand_landmarks.landmark
    fingers = []

    # Thumb: compare tip x-position to IP joint x-position
    # For a right hand, the thumb tip is to the left (lower x) when curled
    if landmarks[TIP_IDS[0]].x < landmarks[PIP_IDS[0]].x:
        fingers.append(True)
    else:
        fingers.append(False)

    # Index, middle, ring, pinky: compare tip y to PIP joint y
    for i in range(1, 5):
        if landmarks[TIP_IDS[i]].y < landmarks[PIP_IDS[i]].y:
            fingers.append(True)
        else:
            fingers.append(False)

    return fingers

What does this code do?

The fingers_up function loops through all five fingers and checks whether each one is extended. It returns a list like [True, True, False, False, False], which means the thumb and index finger are up while the other three are curled. This list is the foundation for every gesture you'll define later.

Pro tip: thumb detection is simplified

This thumb logic assumes a right hand facing the camera. For a more robust solution, you could use the handedness classification from MediaPipe and flip the comparison for left hands. For this project, the simplified version works well for most users.

Test the Finger State Detector

Before building gestures on top of this function, you need to verify it actually works. You'll import fingers_up into your main script and display the finger state array live on the webcam feed.

  • Open gesture_mouse.py.
  • Add an import at the top of the file:
from gesture_utils import fingers_up
  • Inside your main loop, after detecting hand landmarks, call fingers_up(hand_landmarks) and store the result.
  • Use cv2.putText() to display the finger state list on the webcam frame.
# Inside your main loop, after getting hand_landmarks:
finger_state = fingers_up(hand_landmarks)

# Convert booleans to 1s and 0s for a cleaner display
state_display = [int(f) for f in finger_state]
cv2.putText(frame, f'Fingers: {state_display}', (10, 50),
            cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
  • Run your script and hold your hand up to the camera.
  • Try different finger combinations and verify the array updates correctly.

What should I see?

You should see a green text overlay on your webcam feed showing something like Fingers: [1, 1, 0, 0, 0] when your thumb and index finger are raised. Raising all five fingers should show [1, 1, 1, 1, 1]. Making a fist should show [0, 0, 0, 0, 0].

Finger states not matching your hand?

The thumb detection can be tricky depending on your hand orientation. Make sure your palm is facing the camera. If the thumb state seems inverted, you may need to flip the comparison operator in the thumb check. Help me debug my finger state detection.

Add a Distance Calculator

Knowing which fingers are up is only half the picture. To detect a pinch gesture, you need to measure how close two fingertips are to each other. A pinch happens when the distance between the thumb tip and another fingertip drops below a threshold.

  • Open gesture_utils.py.
  • Add a math import at the top of the file.
  • Write a get_distance(landmark_a, landmark_b) function that calculates the Euclidean distance between two landmarks using their normalized x and y coordinates.
import math

def get_distance(landmark_a, landmark_b):
    """Calculate the Euclidean distance between two landmarks."""
    return math.sqrt(
        (landmark_a.x - landmark_b.x) ** 2 +
        (landmark_a.y - landmark_b.y) ** 2
    )

What does this function do?

This function takes two MediaPipe landmark objects and returns the straight-line distance between them. Since the coordinates are normalized (0.0 to 1.0), the returned distance is also a small decimal. A pinch gesture typically produces a distance below 0.05.

Now test the distance function in your main script to see it in action.

  • Update your import in gesture_mouse.py to include the new function:
from gesture_utils import fingers_up, get_distance
  • Inside your main loop, calculate the distance between the thumb tip (landmark 4) and index finger tip (landmark 8).
  • Display the distance value on the webcam frame using cv2.putText().
# Inside your main loop, after getting hand_landmarks:
landmarks = hand_landmarks.landmark
distance = get_distance(landmarks[4], landmarks[8])

cv2.putText(frame, f'Pinch dist: {distance:.3f}', (10, 90),
            cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 0), 2)
  • Run the script and slowly pinch your thumb and index finger together.
  • Watch the distance value drop as your fingers get closer.

You should see the distance value hover around 0.2-0.3 when your fingers are apart and drop below 0.05 when they touch. This threshold is what you'll use to trigger click gestures in the next step.

Your gesture_utils.py module now has two powerful building blocks: a finger state detector and a distance calculator. Together, these let you define any gesture as a combination of finger positions and fingertip distances. Next up, you'll use them to add click and scroll controls to your virtual mouse.

Add Click and Scroll Gestures

Your virtual mouse can track your finger and move the cursor. Now it is time to make it actually do things. In this step, you will wire up gesture detection so that pinching your fingers triggers clicks and holding two fingers up activates scroll mode.

In This Step, Get Ready To:
  • Implement left-click using a thumb-to-index-finger pinch gesture.
  • Implement right-click using a thumb-to-middle-finger pinch and add a cursor freeze mode.
  • Implement scroll mode using a two-finger-up gesture with vertical hand tracking.
Implement Left-Click With a Pinch Gesture

A pinch gesture is the most natural way to simulate a mouse click. You will measure the distance between your thumb tip and index finger tip. When those two landmarks get close enough together, the system fires a click.

The get_distance function you built in the previous step does the heavy lifting here. You just need to compare its output against a threshold value.

  • Open gesture_mouse.py.
  • At the top of your script, add a CLICK_THRESHOLD variable set to 0.05 and a CLICK_COOLDOWN variable set to 0.5 seconds.
  • Import time at the top of the file and create a variable last_click_time = 0 before your main loop.
  • Inside your main loop, after extracting the hand landmarks, use get_distance() to calculate the distance between landmark 4 (thumb tip) and landmark 8 (index finger tip).
  • Add a condition: if the distance is below CLICK_THRESHOLD and enough time has passed since the last click, call pyautogui.click() and update last_click_time.
import time

CLICK_THRESHOLD = 0.05
CLICK_COOLDOWN = 0.5
last_click_time = 0

# Inside your main loop, after getting hand_landmarks:
thumb_tip = hand_landmarks.landmark[4]
index_tip = hand_landmarks.landmark[8]

distance = get_distance(thumb_tip, index_tip)

current_time = time.time()
if distance < CLICK_THRESHOLD and (current_time - last_click_time) > CLICK_COOLDOWN:
    pyautogui.click()
    last_click_time = current_time
    cv2.putText(frame, "LEFT CLICK", (10, 60), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)

Why Do We Need a Cooldown?

Your webcam captures around 30 frames per second. Without a cooldown, a single pinch gesture that lasts half a second could trigger 15 clicks. The cooldown timer uses time.time() to enforce a minimum gap between clicks, so each pinch registers as exactly one click.

  • Display the text LEFT CLICK on the webcam overlay using cv2.putText() whenever a click fires.
  • Run your script and test the pinch gesture by slowly bringing your thumb and index finger together.

You should see "LEFT CLICK" flash on your webcam feed each time you pinch. If clicks are not registering, try lowering the threshold to 0.06 or 0.07. If you are getting accidental clicks, raise it to 0.04.

Clicks Not Registering on macOS?

PyAutoGUI needs Accessibility permissions to control the mouse on macOS. Go to System Settings > Privacy & Security > Accessibility and make sure your Terminal or VS Code is in the allowed list. Without this, mouse commands silently fail.

Still having trouble? Help me debug my pinch click gesture.

Implement Right-Click and Cursor Freeze

Right-click follows the same pinch pattern, but uses a different finger. You will also add a cursor freeze mode so you can hold your hand still without the cursor drifting while you set up a gesture.

  • Add a right-click detection block below your left-click code.
  • Calculate the distance between landmark 4 (thumb tip) and landmark 12 (middle finger tip) using get_distance().
  • If the distance is below CLICK_THRESHOLD and the cooldown has elapsed, call pyautogui.rightClick().
# Right-click detection (add after left-click block)
middle_tip = hand_landmarks.landmark[12]

right_distance = get_distance(thumb_tip, middle_tip)

if right_distance < CLICK_THRESHOLD and (current_time - last_click_time) > CLICK_COOLDOWN:
    pyautogui.rightClick()
    last_click_time = current_time
    cv2.putText(frame, "RIGHT CLICK", (10, 90), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)

Now add the cursor freeze feature. This gives you a way to pause cursor movement without closing the program.

  • Use your fingers_up() function to get the current finger state.
  • Check if the finger state matches [0, 1, 1, 1, 1] (thumb folded, all other fingers up).
  • When this pattern is detected, skip the pyautogui.moveTo() call so the cursor stays in place.
  • Display "FROZEN" on the overlay when the cursor is frozen.
# Cursor freeze detection (add before your moveTo call)
finger_state = fingers_up(hand_landmarks)

if finger_state == [0, 1, 1, 1, 1]:
    # Cursor frozen - skip mouse movement
    cv2.putText(frame, "FROZEN", (10, 120), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 0), 2)
else:
    # Normal cursor movement
    pyautogui.moveTo(current_x, current_y)

What Does the Cursor Freeze Solve?

Without a freeze mode, the cursor keeps moving as long as your hand is visible. This makes it hard to set up a precise click because your hand naturally shifts as you bring your fingers together. Freezing the cursor first lets you position it, then pinch to click without drift.

  • Run your script and test the right-click by pinching your thumb and middle finger together.
  • Test the freeze mode by holding all fingers up with your thumb folded, then verify the cursor stops moving.
Implement Scrolling

Scroll mode uses a different approach from clicking. Instead of detecting a single pinch event, you track continuous vertical hand movement while a specific finger pattern is held. Think of it like a touchpad scroll gesture, but in the air.

  • Add a SCROLL_SENSITIVITY variable at the top of your script, set to 10.
  • Create a variable prev_scroll_y = None before your main loop to track the previous frame's y-position.
  • Detect scroll mode by checking if the finger state matches [0, 1, 1, 0, 0] (index and middle fingers up, all others down).
  • When in scroll mode, compare the current y-position of landmark 8 to prev_scroll_y.
  • Calculate the difference and pass it to pyautogui.scroll() multiplied by SCROLL_SENSITIVITY.
SCROLL_SENSITIVITY = 10
prev_scroll_y = None

# Inside your main loop, after getting finger_state:
if finger_state == [0, 1, 1, 0, 0]:
    # Scroll mode active
    index_y = hand_landmarks.landmark[8].y

    if prev_scroll_y is not None:
        # Calculate vertical movement between frames
        delta_y = prev_scroll_y - index_y
        scroll_amount = int(delta_y * SCROLL_SENSITIVITY)
        if scroll_amount != 0:
            pyautogui.scroll(scroll_amount)

    prev_scroll_y = index_y
    cv2.putText(frame, "SCROLL MODE", (10, 150), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 0, 0), 2)
else:
    # Reset scroll tracking when not in scroll mode
    prev_scroll_y = None

How Does the Scroll Direction Work?

In image coordinates, y increases downward. When you move your hand up, the y value decreases, making delta_y positive. A positive value passed to pyautogui.scroll() scrolls up. Moving your hand down makes delta_y negative, which scrolls down. This matches the natural expectation of pushing content up or pulling it down.

  • Run your script and hold up your index and middle fingers with all others down.
  • Move your hand slowly up and down to test scrolling.
  • Open a long webpage in your browser and verify that the page scrolls in the correct direction.

If scrolling feels too fast, lower SCROLL_SENSITIVITY to 5. If it feels sluggish, try 15 or 20.

Scroll Mode Activating Accidentally?

If scroll mode keeps triggering when you only want to move the cursor, your fingers_up() function might be misreading your middle finger as extended. Try curling your middle finger more firmly when in move mode. You can also adjust the finger detection logic in gesture_utils.py to use a larger y-difference threshold.

Help me fix gesture conflicts between move and scroll modes.

Your virtual mouse now has real interaction capabilities. You can move, click, right-click, scroll, and freeze the cursor. Next up, you will polish the overlay with color-coded mode indicators and tune the sensitivity settings to make everything feel smooth.

Polish and Test Your Virtual Mouse

Your gesture mouse can move, click, and scroll. Right now, though, it is hard to tell what the system thinks you are doing at any given moment. A clear visual status display makes the difference between a frustrating prototype and a tool you can actually use.

In This Step, Get Ready To:
  • Add a color-coded status bar that displays the current gesture mode on your webcam overlay.
  • Extract configuration variables for easy sensitivity tuning.
  • Run a real-world test using your gesture mouse to browse the web.
Add Gesture State Display and Mode Indicators

Your webcam overlay currently flashes text like "LEFT CLICK" or "SCROLL MODE" when a gesture triggers. A persistent status bar is more useful. It always shows the current mode so you can see at a glance what the system is detecting.

  • Open gesture_mouse.py in your code editor.
  • Add a variable called current_mode before your main loop. Initialize it to "MOVE".
  • Define a dictionary that maps each mode to a BGR color tuple.
# Mode display configuration
current_mode = "MOVE"
mode_colors = {
    "MOVE": (0, 255, 0),        # Green
    "LEFT CLICK": (0, 0, 255),  # Red
    "RIGHT CLICK": (0, 0, 200), # Dark red
    "SCROLL": (255, 150, 0),    # Blue
    "FROZEN": (200, 200, 200),  # Gray
}
  • Inside your main loop, update current_mode at each gesture detection point. Set it to "LEFT CLICK" when a left pinch is detected, "RIGHT CLICK" for a right pinch, "SCROLL" when in scroll mode, and "FROZEN" when the cursor freeze pattern is active.
  • At the bottom of the loop (before cv2.imshow()), draw the status bar using a filled rectangle and text overlay.
# Draw status bar at the top of the frame
color = mode_colors.get(current_mode, (255, 255, 255))
cv2.rectangle(frame, (0, 0), (frame.shape[1], 40), color, -1)
cv2.putText(frame, f"Mode: {current_mode}", (10, 28),
            cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 0, 0), 2)

# Reset mode to MOVE at the start of each loop iteration
# so it only shows a gesture mode while that gesture is active
current_mode = "MOVE"

What does this code do?

The cv2.rectangle() call draws a filled rectangle across the top of the frame. The -1 thickness value tells OpenCV to fill the shape instead of drawing an outline. The text is drawn on top in black so it is readable against any background color.

Resetting current_mode to "MOVE" at the start of each iteration means the bar only shows a gesture label while that gesture is actively detected. The moment you stop pinching, it snaps back to "MOVE".

  • Run your script and try each gesture.

You should see a colored bar at the top of the webcam window that changes color and label as you switch between gestures. Green for move, red for click, blue for scroll, and gray for frozen.

Tune Sensitivity and Thresholds

Hard-coded values scattered throughout your script make tuning painful. Pulling all configuration into named variables at the top of the file lets you experiment quickly without hunting through gesture logic.

  • Add a configuration section at the very top of gesture_mouse.py, right after your imports.
# --- Configuration ---
SMOOTHING = 5              # Higher = smoother but laggier cursor
CLICK_THRESHOLD = 0.05     # Distance threshold for pinch detection
CLICK_COOLDOWN = 0.5       # Seconds between allowed clicks
SCROLL_SENSITIVITY = 10    # Multiplier for scroll speed

What do these values control?

  • SMOOTHING controls the exponential moving average. A value of 5 means the cursor position is a weighted blend of the previous position and the new target. Higher values (7-10) feel smoother but introduce noticeable lag.
  • CLICK_THRESHOLD sets how close your thumb and finger tips need to be (in normalized coordinates) to register as a pinch. If clicks trigger too easily, lower this value. If you have to squeeze hard, raise it.
  • CLICK_COOLDOWN prevents a single pinch from firing dozens of clicks across multiple frames. Half a second is a good starting point.
  • SCROLL_SENSITIVITY multiplies the raw hand movement delta before passing it to pyautogui.scroll(). Increase this if scrolling feels too slow.
  • Replace every hard-coded value in your gesture logic with the corresponding configuration variable.

For example, your smoothing calculation should now reference SMOOTHING instead of a raw number:

# Use the config variable instead of a magic number
current_x = previous_x + (target_x - previous_x) / SMOOTHING
current_y = previous_y + (target_y - previous_y) / SMOOTHING
  • Do the same for your pinch distance checks (use CLICK_THRESHOLD), your cooldown timer (use CLICK_COOLDOWN), and your scroll amount calculation (use SCROLL_SENSITIVITY).
  • Run the script and experiment with different values.

Pro tip: find your sweet spot

Start by adjusting one variable at a time. Try SMOOTHING values of 3, 5, and 8 to feel the difference between responsive and smooth. Then adjust CLICK_THRESHOLD based on your webcam distance. If you sit far from the camera, you may need a higher threshold like 0.07.

Run a Real-World Test

Testing in isolation is one thing. Using your gesture mouse for actual tasks reveals problems you would never catch otherwise. Things like accidental clicks when repositioning your hand, scroll jitter, or the cursor drifting when you look away from the camera.

  • Run gesture_mouse.py.
  • Position the webcam overlay window to one side of your screen so you can see both the overlay and your desktop.
  • Use your index finger to move the cursor to your browser icon and left-click to open it.
  • Navigate to any webpage by clicking the address bar and typing a URL with your keyboard.
  • Use the scroll gesture (index and middle fingers up) to scroll through the page.
  • Right-click on a link to open the context menu.
  • Use the cursor freeze gesture (all fingers up, thumb down) to pause cursor movement while you reposition your hand.

Cursor jumping or clicks not registering?

If the cursor jumps erratically, increase your SMOOTHING value. If clicks do not register, your CLICK_THRESHOLD may be too low for your webcam distance. Try raising it to 0.06 or 0.07. On macOS, double-check that your Terminal or VS Code has Accessibility permissions in System Settings > Privacy & Security > Accessibility. Help me debug my gesture mouse.

If everything works, congratulations. You have a fully functional gesture-controlled mouse. If you noticed issues during testing, go back and adjust your configuration variables until the experience feels natural.

Your gesture mouse is polished, tuned, and tested against real tasks. Next up, you will push it even further with drag-and-drop and volume control in the Secret Mission.

Secret mission

Add Drag-and-Drop and Volume Control

Your virtual mouse can move, click, and scroll. But what about dragging files across your desktop with a fist gesture? Or controlling your system volume with a pinch-and-spread motion? Take your gesture mouse to the next level with two advanced gestures and a priority system that keeps everything from firing at once.

Before you go

Clean Up Your Resources

This project runs entirely locally on your machine, so there are no ongoing cloud costs. Decide whether to keep your project files and installed packages, or remove them entirely.

Resources you used:

  • Python packages: opencv-python, mediapipe, pyautogui, numpy.
  • Project files: gesture_mouse.py, gesture_utils.py, test_cam.py.
  • macOS Accessibility permissions granted to your Terminal or VS Code (if applicable).
  • Linux system packages: scrot, python3-tk, python3-dev (if applicable).

Keep everything running

No action needed. Choose this if you want to keep experimenting with your gesture mouse or plan to extend it with new gestures.

  • Your project files and installed packages will remain on your machine.
  • There are no ongoing costs since everything runs locally.

Pause - I'll come back to this later

Keep your project files but remove the installed packages to free up disk space. You can reinstall them later when you return.

  • Uninstall the Python packages by running pip uninstall opencv-python mediapipe pyautogui numpy in your terminal.
  • Keep your project folder with gesture_mouse.py, gesture_utils.py, and test_cam.py so you can pick up where you left off.
  • To resume, run pip install opencv-python mediapipe pyautogui numpy again from your project folder.

Delete - I don't want to use this again

Remove all project resources and installed packages to start completely fresh.

  • Uninstall the Python packages by running pip uninstall opencv-python mediapipe pyautogui numpy in your terminal.
  • Delete your project folder containing gesture_mouse.py, gesture_utils.py, and test_cam.py.
  • On macOS, optionally revoke Accessibility permissions for your Terminal or VS Code in System Settings > Privacy & Security > Accessibility.
  • On Linux, optionally remove the system packages by running sudo apt-get remove scrot python3-tk python3-dev.

Mission accomplished

That's a Wrap!

Nice work! You just built a gesture-controlled virtual mouse from scratch using your laptop's webcam, a hand tracking ML model, and a few dozen lines of Python.

Here's what you learned:

  • Used MediaPipe's hand landmark detection to track 21 points on your hand in real time, then mapped finger coordinates to screen positions with smoothing for fluid cursor movement.
  • Built a finger state detector and gesture recognition system that translates pinch distances and finger patterns into left-click, right-click, and scroll actions via PyAutoGUI.
  • Created a visual debug overlay with color-coded mode indicators, making it easy to see exactly what your system detects at every moment.
  • Added drag-and-drop and volume control gestures with a priority system to prevent gesture conflicts.

You now know how to bridge ML model output to real OS-level actions. This is the same pattern used in accessibility tools, VR controllers, and touchless kiosks.

Ready to quiz yourself?