Hand-Controlled Particle Canvas

Introduction

30 Second Summary

Think about how you interact with your computer right now. What if your bare hand in the air could control what happens on screen, no mouse or keyboard needed?

In this project, you will build a browser-based creative tool that tracks your hand through your webcam and turns finger movements into glowing particle trails. Different hand gestures trigger different visual effects, making your hand the entire interface.

What You'll Build

Open the page in your browser, wave your hand in front of your webcam, and watch glowing particles trail behind your fingertip like a digital light brush that you painted with your own movement.

By the end of this project, you'll have:

• A real-time hand tracking system where your index fingertip becomes a glowing cursor that leaves fading particle trails as you move.
• Three gesture-controlled effects triggered entirely by your hand shape. Pinch to draw permanent strokes. Open your palm to fire a particle burst. Close your fist to clear the canvas.
• A position smoothing system that transforms jittery raw tracking data into fluid, responsive cursor movement you can actually paint with.
• Secret Mission: A rainbow velocity mode that shifts particle colors based on how fast you move your hand.

Are there any prerequisites?

You should be comfortable with JavaScript, HTML, and CSS fundamentals. You also need a working webcam. No packages to install. This project runs entirely from a single HTML file served locally.

Before We Start

Before you dive in, let's commit to what you're building and why it matters. A hand-controlled particle canvas sounds simple, but turning jittery camera coordinates into smooth, gesture-responsive art requires solving several distinct problems.

Set Up Your Project Files and Local Server

Your hand-tracking canvas relies on ES module imports from a CDN. Browsers block module scripts loaded from file:// URLs for security reasons. They also deny camera access to pages not served over HTTP.

To get around both restrictions, you need a local web server. In this step you will create the project, write the base HTML file, and launch it through Live Server so the browser can load the MediaPipe library and later access your webcam.

In this step, get ready to:

• Create a project folder and a single index.html file with your canvas, video, and script elements.
• Add CSS that makes the canvas fill the viewport with a dark background.
• Start a local server and confirm the page loads without errors.

Create your project folder

• Open a terminal in VS Code by pressing Ctrl+` (macOS and Windows).
• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir hand-particle-canvas
cd hand-particle-canvas

• Open this folder in VS Code by running:

code .

What does this command do?

The code . command opens the current folder in VS Code. The dot (.) means "this directory." You should see hand-particle-canvas appear in the VS Code title bar or Explorer sidebar.

Write the HTML structure

Your entire project lives in a single file. It contains a hidden video element for the webcam feed, a full-screen canvas for drawing particles, and a module script that imports the MediaPipe hand tracking library.

• In VS Code, create a new file called index.html inside the hand-particle-canvas folder.
• Paste the following code into index.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Hand Particle Canvas</title>
    <style>
    </style>
</head>
<body>
    <video id="webcam" autoplay playsinline></video>
    <canvas id="canvas"></canvas>
    <div id="status">Loading MediaPipe...</div>

    <script type="module">
        import { FilesetResolver, HandLandmarker } from
            "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.35/vision_bundle.mjs";
    </script>
</body>
</html>

What does this code do?

• The <video id="webcam"> element will receive the raw webcam feed. You will hide it visually but still read frames from it.
• The <canvas id="canvas"> element is your drawing surface for particles.
• The <div id="status"> shows a loading message so you know the page is alive before tracking starts.
• The <script type="module"> tag imports FilesetResolver and HandLandmarker from MediaPipe's CDN bundle. These are the two classes you need to run hand detection.

• Save index.html.

Start Live Server and verify the page loads

You need to serve this file over HTTP so the browser allows the module import. Live Server handles this in one click.

• Make sure index.html is the active file in the editor.
• In the VS Code bottom status bar, click the Go Live button on the right side.

Chrome opens automatically with your page. You should see the text "Loading MediaPipe..." in the top-left area of the page.

Don't see the Go Live button?

The button only appears when an HTML file is the active editor tab. If you still do not see it, press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows) to open the Command Palette, then type Live Server: Open With Live Server and press Enter.

Help me start Live Server in VS Code

Add CSS to style the page

The page loads, but the canvas has no size and the video element is visible at its default dimensions. You need CSS to make the canvas fill the screen, hide the video, and give everything a dark background.

• In index.html, find the empty <style> tags in the <head> section.
• Paste the following CSS between the opening and closing <style> tags:

        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }

        body {
            background: #0a0a0f;
            overflow: hidden;
            font-family: monospace;
        }

        canvas {
            display: block;
            width: 100vw;
            height: 100vh;
        }

What does this CSS do?

• The universal selector (*) resets all margins and padding so the canvas touches every edge of the viewport.
• The body gets a near-black background (#0a0a0f) that makes glowing particles pop.
• The canvas fills the entire viewport using 100vw and 100vh.

• Now paste the following CSS directly below the canvas block you just added (still inside the <style> tags):

        video {
            position: absolute;
            top: 0;
            left: 0;
            width: 1px;
            height: 1px;
            opacity: 0;
            pointer-events: none;
        }

        #status {
            position: fixed;
            top: 16px;
            left: 16px;
            color: #888;
            font-size: 14px;
            z-index: 10;
        }

What does this CSS do?

• The video element is shrunk to 1x1 pixels with zero opacity. The browser still reads frames from it, but you never see the raw webcam feed on screen.
• The #status indicator is pinned to the top-left corner with a high z-index so it floats above the canvas.

• Save index.html.

Before you check the browser, what do you expect to see now that the background is dark and the canvas fills the screen?

Your browser auto-refreshes thanks to Live Server. You should see a fully dark page with small grey text "Loading MediaPipe..." in the top-left corner.

• Open the browser console with Cmd+Option+J (macOS) or Ctrl+Shift+J (Windows) and confirm there are no red error messages about CORS or module loading.

Seeing a CORS or module error in the console?

This means the page was opened as a file:// URL instead of through Live Server. Check that the address bar shows http://127.0.0.1:5500 (or a similar localhost address), not a file path.

If Live Server stopped, click Go Live again in the VS Code status bar to restart it.

Help me fix this CORS or module loading error

✔️ Awesome, I've got everything!

Great. Make sure you have saved index.html and your browser shows the dark page with "Loading MediaPipe..." text.

ⓧ I'd like to double check the full code

Here is the complete index.html at this point. Compare yours against it:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Hand Particle Canvas</title>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }

        body {
            background: #0a0a0f;
            overflow: hidden;
            font-family: monospace;
        }

        canvas {
            display: block;
            width: 100vw;
            height: 100vh;
        }

        video {
            position: absolute;
            top: 0;
            left: 0;
            width: 1px;
            height: 1px;
            opacity: 0;
            pointer-events: none;
        }

        #status {
            position: fixed;
            top: 16px;
            left: 16px;
            color: #888;
            font-size: 14px;
            z-index: 10;
        }
    </style>
</head>
<body>
    <video id="webcam" autoplay playsinline></video>
    <canvas id="canvas"></canvas>
    <div id="status">Loading MediaPipe...</div>

    <script type="module">
        import { FilesetResolver, HandLandmarker } from
            "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.35/vision_bundle.mjs";
    </script>
</body>
</html>

Your project scaffolding is done. The MediaPipe library is loading in the background, and your canvas is ready to draw on. Next up, you will wire the webcam and start detecting hand landmarks in real time.

Get Webcam Feed and Hand Tracking Running

Your canvas is ready, but right now it just sits there doing nothing. Before you can control anything with your hand, you need to prove that MediaPipe can actually detect it.

In this step, you will get raw landmark data flowing from your webcam through MediaPipe's hand tracking model and onto the screen. By the end, you will see 21 dots following your hand joints in real time.

In this step, get ready to:

• Set up canvas and video element references for the rendering loop.
• Initialize the MediaPipe Hand Landmarker model and request webcam access.
• Draw all 21 hand landmarks as dots on the canvas in real time.

Set up element references and canvas sizing

The canvas element in your HTML has CSS dimensions (100vw by 100vh), but its internal pixel resolution defaults to 300x150. Drawing needs pixel-accurate coordinates, so you need to sync the canvas resolution to the actual window size.

• In VS Code, open index.html and find the <script type="module"> block.
• Directly below the existing import line, add the following code to grab DOM elements, declare the hand tracker variable, and set the canvas to full window resolution:

const canvas = document.getElementById("canvas");
const ctx = canvas.getContext("2d");
const video = document.getElementById("webcam");
const statusEl = document.getElementById("status");

let handLandmarker;

function resizeCanvas() {
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;
}
window.addEventListener("resize", resizeCanvas);
resizeCanvas();

What does this code do?

• canvas and ctx give you the drawing surface and its 2D rendering context for all particle and landmark drawing.
• video references the hidden <video> element that will receive the webcam stream.
• resizeCanvas() sets the canvas pixel dimensions to match the window, and re-runs whenever the window resizes so your drawing coordinates stay accurate.

• Save index.html and switch to Chrome.
• Open DevTools with Cmd+Option+J (macOS) or Ctrl+Shift+J (Windows) and type canvas.width in the Console.
• You should see a number matching your browser window width (e.g. 1440 or 1920) instead of the default 300.

Seeing 300 instead of your window width?

• Make sure you placed the new code below the import line, not above it.
• Check that resizeCanvas() is called at the bottom of the snippet (it runs the function immediately, not just on resize events).
• Confirm Live Server reloaded the page after you saved. If not, manually refresh Chrome.

Still stuck? Why is my canvas width still showing 300?

Initialize MediaPipe and start the webcam

The HandLandmarker model needs two things before it can track your hand: a WASM runtime (the engine that runs the ML model in the browser) and a live video stream to analyze.

• Below the resizeCanvas() call, add this init() function that loads the model and starts the webcam:

async function init() {
    const vision = await FilesetResolver.forVisionTasks(
        "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.35/wasm"
    );

    handLandmarker = await HandLandmarker.createFromOptions(vision, {
        baseOptions: {
            modelAssetPath:
                "https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task",
            delegate: "GPU"
        },
        runningMode: "VIDEO",
        numHands: 1
    });

    statusEl.textContent = "Starting webcam...";

    const stream = await navigator.mediaDevices.getUserMedia({
        video: { facingMode: "user", width: 640, height: 480 }
    });
    video.srcObject = stream;
    await video.play();

    statusEl.textContent = "Tracking";
    requestAnimationFrame(detect);
}

What does this code do?

• FilesetResolver.forVisionTasks() loads the WASM binary that powers the ML inference engine in the browser.
• HandLandmarker.createFromOptions() downloads and initializes the hand tracking model. runningMode: "VIDEO" tells it to reuse bounding boxes between frames for faster tracking.
• delegate: "GPU" hints that inference should use your GPU if available, falling back to CPU automatically.
• getUserMedia() requests webcam access from the browser. The facingMode: "user" setting selects the front-facing camera.
• Once the video is playing, requestAnimationFrame(detect) kicks off the frame-by-frame detection loop you will write next.

Why VIDEO mode instead of IMAGE mode?

In IMAGE mode, the model runs full palm detection on every single frame. In VIDEO mode, it reuses the bounding box from the previous frame and only re-triggers palm detection when tracking is lost. This makes VIDEO mode significantly faster for continuous webcam streams.

Create the detection loop

The init() function is defined but nothing calls it yet, and it references a detect function that does not exist. You need both the detection loop and the call to init() to bring everything to life.

• Below the init() function, add the detection loop and the call that starts everything:

function detect() {
    if (video.readyState >= 2) {
        const results = handLandmarker.detectForVideo(video, performance.now());
        if (results.landmarks && results.landmarks.length > 0) {
            const landmarks = results.landmarks[0];
            ctx.clearRect(0, 0, canvas.width, canvas.height);
            for (const lm of landmarks) {
                // Mirror x so moving your hand right moves the dot right
                const x = (1 - lm.x) * canvas.width;
                const y = lm.y * canvas.height;
                ctx.beginPath();
                ctx.arc(x, y, 4, 0, Math.PI * 2);
                ctx.fillStyle = "rgba(100, 200, 255, 0.8)";
                ctx.fill();
            }
        }
    }
    requestAnimationFrame(detect);
}

init().catch(err => {
    statusEl.textContent = "Error: " + err.message;
    console.error(err);
});

What does this code do?

• detect() runs every animation frame (roughly 60 times per second). It checks the video is ready, then calls detectForVideo() to get the current hand landmark positions.
• Each landmark has x and y values normalized between 0 and 1. Multiplying by canvas dimensions converts them to pixel coordinates.
• The (1 - lm.x) mirrors the x-axis so your hand movement feels natural (move right, dot goes right) since webcams capture a mirrored image.
• init().catch() at the bottom actually runs the initialization. If anything fails, the error message appears in the status indicator.

What is requestAnimationFrame?

requestAnimationFrame tells the browser to call your function right before the next screen repaint. By calling it recursively at the end of detect(), you create a continuous loop that processes one webcam frame per screen refresh.

Before you save, what do you expect to see on screen once MediaPipe finishes loading?

• Save index.html.
• Switch to Chrome. The page reloads and you will see a camera permission prompt near the address bar asking to use your camera.
• Click Allow to grant webcam access.
• Hold your open hand in front of the webcam.

You should see 21 small blue dots on the dark canvas that follow your hand joints in real time. The status indicator in the top left updates to "Tracking".

Not seeing any dots?

• Check the status indicator. If it says "Error:", open DevTools Console to see the full message.
• If you see "Starting webcam..." but it never changes, you may have blocked the camera permission. Click the camera icon in Chrome's address bar and select Allow, then refresh the page.
• Make sure your webcam is not in use by another application (Zoom, FaceTime, etc.).
• Confirm you are accessing the page via http://localhost (Live Server) and not as a file:// URL. Module imports and camera access are blocked on file://.

Still stuck? Help me debug why my hand landmarks are not appearing on the canvas.

✔️ Awesome, I've got everything!

Great. Double check that you have saved index.html and that all 21 dots follow your hand smoothly in Chrome.

ⓧ I'd like to double check the full code

Your <script type="module"> block should contain the following sections in this order. Scroll up and confirm each one is present:

• The import statement for FilesetResolver and HandLandmarker from the MediaPipe CDN.
• Four const declarations (canvas, ctx, video, statusEl) and let handLandmarker.
• The resizeCanvas() function, its resize event listener, and the immediate resizeCanvas() call.
• The async function init() that loads WASM, creates the hand landmarker, requests the webcam, and calls requestAnimationFrame(detect).
• The function detect() loop that calls detectForVideo() and draws 21 circles using ctx.arc().
• The init().catch() call at the very end.

You now have real-time hand tracking running entirely in the browser. Next up, you will turn those raw dots into a glowing particle cursor that trails behind your fingertip.

Turn Your Fingertip Into a Particle Cursor

Your 21 dots prove that MediaPipe can see your hand. But dots are not the creative tool you are building.

In this step, you will transform one landmark (the index fingertip) into a glowing cursor that leaves a fading trail of particles as you move. Each particle spawns at your fingertip, drifts outward, and fades to nothing.

In this step, get ready to:

• Build a particle system that spawns, moves, fades, and removes particles.
• Replace the raw 21-dot rendering with a glowing particle renderer using additive blending.
• Observe a jitter problem in the raw tracking data that the next step will fix.

Build the particle system

A particle system manages a pool of tiny objects. Each particle has a position, velocity, opacity, and size. On every frame, particles drift, fade, and shrink until they disappear.

• In index.html, find this line inside the <script type="module"> block:

let handLandmarker;

• Add the particles array, the spawn function, and the update function directly below that line by copying this code:

let particles = [];

// Create a new particle at the given position
function spawnParticle(x, y) {
    particles.push({
        x,
        y,
        vx: (Math.random() - 0.5) * 0.5,
        vy: (Math.random() - 0.5) * 0.5,
        opacity: 1,
        size: 6 + Math.random() * 4,
        permanent: false,
        color: "100, 200, 255"
    });
}

// Move particles, fade opacity, shrink, and remove dead ones
function updateParticles() {
    for (let i = particles.length - 1; i >= 0; i--) {
        const p = particles[i];
        p.x += p.vx;
        p.y += p.vy;
        p.vx *= 0.98;
        p.vy *= 0.98;
        p.opacity -= 0.015;
        p.size *= 0.995;
        if (p.opacity <= 0) {
            particles.splice(i, 1);
        }
    }
}

What does this code do?

Each particle starts with full opacity and a random drift velocity. On every frame, updateParticles() moves each particle, applies friction (0.98 multiplier), fades opacity by 0.015, and shrinks size by 0.5%.

The loop iterates backwards so that removing dead particles with splice() does not skip any elements.

• Save index.html.
• Refresh your browser. You still see the 21 dots following your hand as before. The new functions exist but are not called yet.
• Open DevTools with F12 and check the Console tab. Confirm there are no red error messages.

Seeing a syntax error in the console?

Check that you pasted the code after let handLandmarker; and before the resizeCanvas() function. A stray bracket or missing closing brace will cause a parse error.

help me fix a syntax error in my particle system code

Render particles as glowing orbs and wire up detection

The particle system needs two more pieces: a renderer that draws each particle as a glowing radial gradient, and an updated detection loop that feeds fingertip positions into the system.

• Add the render() function below updateParticles() by copying this code:

// Draw particles as glowing orbs with additive blending
function render() {
    ctx.fillStyle = "rgba(10, 10, 15, 0.08)";
    ctx.fillRect(0, 0, canvas.width, canvas.height);

    ctx.globalCompositeOperation = "lighter";

    for (const p of particles) {
        const gradient = ctx.createRadialGradient(p.x, p.y, 0, p.x, p.y, p.size);
        gradient.addColorStop(0, `rgba(${p.color}, ${p.opacity})`);
        gradient.addColorStop(1, `rgba(${p.color}, 0)`);
        ctx.fillStyle = gradient;
        ctx.beginPath();
        ctx.arc(p.x, p.y, p.size, 0, Math.PI * 2);
        ctx.fill();
    }

    ctx.globalCompositeOperation = "source-over";
}

What does this renderer do?

Instead of clearing the canvas each frame, it paints a nearly transparent dark rectangle over everything. This creates the fading trail effect. Old particles show through the dark overlay until they fade completely.

Setting globalCompositeOperation to "lighter" enables additive blending. Where particles overlap, their colors add together to create bright white intersections.

• Now replace the entire existing detect() function with this new version that feeds your index fingertip into the particle system:

• Find the existing detect() function that starts with function detect() { and draws 21 dots. Select the entire function from its opening line to its closing } and replace it with:

function detect() {
    if (video.readyState >= 2) {
        const results = handLandmarker.detectForVideo(video, performance.now());
        if (results.landmarks && results.landmarks.length > 0) {
            const landmarks = results.landmarks[0];
            // Index fingertip is landmark 8, mirror x for natural movement
            const rawX = (1 - landmarks[8].x) * canvas.width;
            const rawY = landmarks[8].y * canvas.height;
            spawnParticle(rawX, rawY);
        }
    }
    updateParticles();
    render();
    requestAnimationFrame(detect);
}

What changed in detect()?

Instead of drawing all 21 landmarks, the loop now picks only landmark index 8 (your index fingertip). It mirrors the x coordinate with (1 - landmarks[8].x) so that moving your hand right moves the cursor right on screen.

The three calls at the end (updateParticles(), render(), requestAnimationFrame) run every frame regardless of whether a hand is detected. This keeps existing particles animating even when your hand leaves the frame.

• Save index.html.

Canvas stays black or still showing dots?

Make sure you deleted the entire old detect() function. If both versions exist, the old one may still be called by init(). Search your file for ctx.clearRect. If you find it, the old function is still there.

Also confirm that render() is placed between updateParticles() and detect() in the file. Function order does not matter in JavaScript, but it helps you confirm all three exist.

help me debug why my particle canvas is not rendering

Observe the jitter problem

Before you look at the browser, predict what the particle trail will look like when you hold your hand perfectly still in front of the webcam. Will the cursor sit motionless, or will it move?

• Refresh your browser and hold your hand in front of the webcam.
• Move your index finger slowly across the view. You should see glowing blue particles trailing behind your fingertip, fading as they drift outward.
• Now hold your hand completely still.

The cursor vibrates and scatters particles even though your hand is not moving. The trail looks shaky rather than smooth. This is not a bug in your code. Raw landmark positions from the ML model fluctuate by several pixels every frame, even on a perfectly still hand.

Why does the cursor jitter?

The hand landmark model re-estimates positions from scratch each frame. Tiny differences in lighting, camera noise, and model uncertainty cause the predicted x/y to jump around. For a creative drawing tool, this noise makes precise control impossible.

The next step introduces position smoothing to fix this. You will apply exponential interpolation between the previous position and the new reading, which filters out the frame-to-frame noise while keeping the cursor responsive to intentional movement.

✔️ Awesome, I've got everything!

Great. Double check you have saved index.html and confirmed the glowing (but jittery) particle trail in your browser.

ⓧ I'd like to double check the full code

Scroll up and confirm each piece is present in your file, in this order:

• let particles = []; directly after let handLandmarker;.
• The spawnParticle(x, y) function that pushes a particle object with x, y, vx, vy, opacity, size, permanent, and color fields.
• The updateParticles() function that iterates backwards, moves and fades each particle, and splices dead ones.
• The render() function with the dark overlay fill, globalCompositeOperation = "lighter", and radial gradient drawing loop.
• The updated detect() function that extracts landmarks[8], calls spawnParticle(), updateParticles(), and render().

You have a glowing particle trail that proves the creative potential of hand tracking. But the jitter makes it feel broken. Next up, you will apply position smoothing that transforms the noisy raw data into a fluid, responsive cursor.

Smooth the Tracking with Position Interpolation

You have a working particle trail, but it looks shaky. Even when you hold your hand perfectly still, the cursor vibrates because MediaPipe's raw output fluctuates by several pixels between frames.

The fix is exponential smoothing. Instead of jumping to every new reading, you blend the previous position toward the new one by a fraction each frame. The result is a cursor that feels physically attached to your fingertip.

In this step, get ready to:

• Implement exponential position smoothing that filters out frame-to-frame noise.
• Tune the smoothing factor to find the sweet spot between responsive and stable.

Implement exponential position smoothing

Exponential smoothing works by computing a weighted average between where the cursor was last frame and where the new reading says it should be. A factor called alpha controls how much of the new reading gets through.

• In index.html, find the line that says let particles = []; and add the following smoothing state variables directly below it:

// Smoothed cursor coordinates
let smoothX = 0;
let smoothY = 0;
// Previous frame position (used later for velocity)
let prevX = 0;
let prevY = 0;
// Skip smoothing on the very first detection
let hasInitialPosition = false;
// Lower = smoother but laggy, higher = snappy but jittery
const SMOOTHING_ALPHA = 0.4;

What do these variables do?

• smoothX and smoothY hold the filtered cursor position that particles spawn from.
• hasInitialPosition prevents the smoothing formula from blending toward (0, 0) on the very first frame.
• SMOOTHING_ALPHA at 0.4 means each frame moves 40% of the remaining distance toward the raw reading.

• Now add the processResults function directly above your existing spawnParticle function:

function processResults(results) {
    if (results.landmarks && results.landmarks.length > 0) {
        const landmarks = results.landmarks[0];
        // Get raw fingertip position (mirrored for natural movement)
        const rawX = (1 - landmarks[8].x) * canvas.width;
        const rawY = landmarks[8].y * canvas.height;

        if (!hasInitialPosition) {
            // First frame: jump directly to detected position
            smoothX = rawX;
            smoothY = rawY;
            hasInitialPosition = true;
        } else {
            // Blend toward new position each frame
            smoothX += SMOOTHING_ALPHA * (rawX - smoothX);
            smoothY += SMOOTHING_ALPHA * (rawY - smoothY);
        }

        spawnParticle(smoothX, smoothY);

        prevX = smoothX;
        prevY = smoothY;
    }
}

What does this function do?

On the first frame a hand is detected, it snaps the smoothed position directly to the raw reading. On every subsequent frame, it applies the exponential moving average formula: smoothed = previous + alpha * (raw - previous).

This means if the raw position jumps by 10 pixels due to noise, only 4 pixels (40%) of that jump make it through to the cursor. The remaining 6 pixels get spread across the next few frames, creating fluid motion.

• Now update the detect() function to call processResults instead of handling landmarks inline.
• Find this block inside your detect() function:

if (video.readyState >= 2) {
    const results = handLandmarker.detectForVideo(video, performance.now());
    if (results.landmarks && results.landmarks.length > 0) {
        const landmarks = results.landmarks[0];
        const rawX = (1 - landmarks[8].x) * canvas.width;
        const rawY = landmarks[8].y * canvas.height;
        spawnParticle(rawX, rawY);
    }
}

• Replace that entire if block with this streamlined version:

if (video.readyState >= 2) {
    const results = handLandmarker.detectForVideo(video, performance.now());
    processResults(results);
}

• Save index.html.
• Switch to Chrome. Live Server auto-refreshes the page.
• Hold your hand still in front of the webcam and watch the cursor.

The cursor should now sit nearly motionless when your hand is still. Move your hand and the particle trail follows in a smooth, fluid arc.

Cursor still jittering?

• Check that spawnParticle(smoothX, smoothY) uses the smoothed values, not rawX and rawY.
• Make sure you removed the old inline landmark code from detect(). If both the old code and processResults run, you get double particles.
• Confirm SMOOTHING_ALPHA is declared with const at the top of your script, not inside a function.

Help me debug my position smoothing code

Tune the smoothing factor

The alpha value controls the tradeoff between responsiveness and stability. Try changing it to feel the difference.

• Change SMOOTHING_ALPHA to 0.2, save, and move your hand quickly.

The cursor feels floaty and lags behind your hand. It filters out all jitter but sacrifices responsiveness.

• Now change SMOOTHING_ALPHA to 0.7, save, and try the same movement.

The cursor snaps to your finger quickly but lets more jitter through. Small hand tremors become visible again.

• Set SMOOTHING_ALPHA back to 0.4 and save. This is the sweet spot for this project.

Why 0.4?

At 0.4, the cursor responds within 2-3 frames (under 50ms at 60fps) while still filtering out the 3-5 pixel noise that MediaPipe produces on a stationary hand. For a creative painting tool, this feels natural without being sluggish.

Before you confirm everything is working, predict: what do you expect to see when you hold your hand completely still?

• Hold your hand still in front of the camera for a few seconds.

The cursor sits nearly motionless. Move your hand and the particles follow in a smooth trail with no visible jitter. Compare this to the shaky mess from before.

✔️ Awesome, I've got everything!

Great. Double check you have saved index.html with the final SMOOTHING_ALPHA value of 0.4.

ⓧ I'd like to double check the full code

Scroll back through the code you have added in this step and confirm each piece is present in order:

• The smoothing state variables (smoothX, smoothY, prevX, prevY, hasInitialPosition, SMOOTHING_ALPHA) declared after let particles = [];.
• The processResults(results) function with the smoothing logic and spawnParticle(smoothX, smoothY) call.
• The updated detect() function that calls processResults(results) instead of handling landmarks inline.

Your cursor is now silky smooth. Next up, you will turn different hand poses into distinct visual effects, making your hand the entire interface.

Add Gesture Controls

Your smooth particle trail proves the tracking pipeline works. But a single trail is only one interaction mode.

Recognizing different hand poses from the landmark geometry gives you multiple controls without any buttons or keyboard. Your hand becomes the entire interface. In this step, you will detect pinch, open hand, and fist gestures by computing distances between landmark positions each frame.

In this step, get ready to:

• Write gesture detection functions that recognize pinch, open hand, and fist from landmark distances.
• Wire gestures to visual effects: permanent strokes, particle bursts, and canvas clearing.
• Add visual feedback with distinct colors and a HUD overlay showing the current gesture.

Add gesture HUD elements and state variables

Before detecting gestures, you need HUD elements to display which gesture is active. You also need state variables to track draw mode and pinch toggling.

• In index.html, find the #status CSS rule inside the <style> block. Add the following two new rules below it:

#gesture {
    position: fixed;
    top: 40px;
    left: 16px;
    color: #aaa;
    font-size: 13px;
    z-index: 10;
}

#mode {
    position: fixed;
    top: 64px;
    left: 16px;
    color: #4fc3f7;
    font-size: 13px;
    z-index: 10;
}

What does this code do?

These two rules position HUD text overlays in the top-left corner. #gesture shows the currently detected gesture. #mode shows whether draw mode is active, in a blue accent color.

• Now find the <div id="status">Loading MediaPipe...</div> line in the <body>. Add these two divs directly below it:

<div id="gesture"></div>
<div id="mode"></div>

• Next, add gesture state variables inside the <script type="module"> block. Find the line const statusEl = document.getElementById("status"); and add these references and variables below it:

const gestureEl = document.getElementById("gesture");
const modeEl = document.getElementById("mode");

• Find the line const SMOOTHING_ALPHA = 0.4; and add these new state variables below it:

// Pinch distance threshold in normalized coordinates
const PINCH_THRESHOLD = 0.06;
let lastPinchState = false;

What do these variables do?

PINCH_THRESHOLD defines how close the thumb and index finger must be (in normalized 0-1 coordinates) to count as a pinch. lastPinchState tracks the previous frame's pinch state so you can detect the leading edge (the moment fingers first touch) rather than triggering every frame.

• Also find the line let particles = []; and add this variable directly below it:

let drawMode = false;

• Save index.html and refresh your browser.
• Open DevTools (right-click the page, then click Inspect) and look at the Elements tab. You should see the <div id="gesture"> and <div id="mode"> elements in the DOM, currently empty.

Don't see the new divs?

Make sure you placed the <div> elements after the <div id="status"> line and before the <script> tag. Check for any typos in the id attributes.

Help me debug my gesture HUD elements

Write the gesture detection function

Gesture detection works by computing distances between specific landmarks each frame. A pinch means the thumb tip and index tip are close together. An open hand means all fingertips are far from the palm center. A fist means all fingertips are close to the palm.

• In index.html, find the closing } of the processResults function. Start the new detectGestures function below it with this pinch detection logic:

function detectGestures(landmarks) {
    const thumbTip = landmarks[4];
    const indexTip = landmarks[8];
    const middleTip = landmarks[12];
    const ringTip = landmarks[16];
    const pinkyTip = landmarks[20];
    const wrist = landmarks[0];

    // Pinch detection: thumb tip close to index tip
    const pinchDist = Math.hypot(thumbTip.x - indexTip.x, thumbTip.y - indexTip.y);
    const isPinching = pinchDist < PINCH_THRESHOLD;

    // Toggle draw mode on leading edge of pinch
    if (isPinching && !lastPinchState) {
        drawMode = !drawMode;
        modeEl.textContent = drawMode ? "Draw Mode: ON" : "";
    }
    lastPinchState = isPinching;

How does pinch detection work?

Math.hypot computes the Euclidean distance between the thumb tip (landmark 4) and index tip (landmark 8). When that distance drops below 0.06 in normalized coordinates, it counts as a pinch.

The lastPinchState check ensures draw mode toggles only once per pinch. Without it, draw mode would flip on and off every frame while your fingers are touching.

• Continue the detectGestures function by adding the open hand and fist detection directly below the lastPinchState = isPinching; line:

    // Compute palm center from wrist and two base knuckles
    const palmCenterX = (landmarks[0].x + landmarks[5].x + landmarks[17].x) / 3;
    const palmCenterY = (landmarks[0].y + landmarks[5].y + landmarks[17].y) / 3;

    // Average distance of all fingertips from palm center
    const fingertips = [thumbTip, indexTip, middleTip, ringTip, pinkyTip];
    const avgDist = fingertips.reduce((sum, tip) => {
        return sum + Math.hypot(tip.x - palmCenterX, tip.y - palmCenterY);
    }, 0) / fingertips.length;

    if (avgDist > 0.15) {
        gestureEl.textContent = "Gesture: Open Hand";
        spawnBurst(
            (1 - palmCenterX) * canvas.width,
            palmCenterY * canvas.height
        );
    } else if (avgDist < 0.08) {
        gestureEl.textContent = "Gesture: Fist";
        particles = [];
    } else if (isPinching) {
        gestureEl.textContent = "Gesture: Pinch";
    } else {
        gestureEl.textContent = "";
    }
}

How do open hand and fist detection work?

The palm center is approximated by averaging the positions of the wrist (landmark 0), index base (landmark 5), and pinky base (landmark 17). Then the average distance from all five fingertips to that center tells you how spread the hand is.

When avgDist > 0.15, fingers are spread wide (open hand). When avgDist < 0.08, fingers are curled in (fist). These thresholds work in normalized coordinate space so they adapt to any hand distance from the camera.

• Now add the spawnBurst function below spawnParticle. This creates the particle explosion triggered by the open hand gesture:

function spawnBurst(x, y) {
    // Spawn 60 particles radiating outward from palm center
    for (let i = 0; i < 60; i++) {
        const angle = (Math.PI * 2 * i) / 60;
        const speed = 2 + Math.random() * 4;
        particles.push({
            x,
            y,
            vx: Math.cos(angle) * speed,
            vy: Math.sin(angle) * speed,
            opacity: 1,
            size: 4 + Math.random() * 4,
            permanent: false,
            color: "255, 140, 50"
        });
    }
}

What does spawnBurst do?

It spawns 60 particles at the palm center, each with a velocity pointing outward at an evenly distributed angle. The warm orange color (255, 140, 50) distinguishes burst particles from the default cool blue trail.

• Finally, wire the gesture detection into your frame loop. Inside processResults, find this line:

spawnParticle(smoothX, smoothY);

• Add this line directly below it:

detectGestures(landmarks);

• Save index.html and switch to your browser.
• Try pinching your thumb and index finger together. You should see "Gesture: Pinch" appear in the HUD and "Draw Mode: ON" toggle on.
• Open your palm wide. An orange burst of particles should explode outward from your palm center.
• Close your fist. All particles should clear from the canvas instantly.

Gestures not triggering?

Make sure your detectGestures(landmarks); call is inside the if (results.landmarks && results.landmarks.length > 0) block, not outside it. If you placed it outside, the landmarks variable would be undefined.

Check the browser console for errors. A ReferenceError about spawnBurst means the function was defined after where it's called. Move it above updateParticles.

Help me debug my gesture detection

Update particles for draw mode

Gestures are detected and the burst works. But pinch-toggled draw mode does not actually produce permanent strokes yet. The spawnParticle function still sets permanent: false on every particle regardless of draw mode. You need to connect the draw mode state to particle behavior.

• In the spawnParticle function, find these two lines:

permanent: false,
color: "100, 200, 255"

• Replace them with these lines that respond to the drawMode state:

permanent: drawMode,
color: drawMode ? "255, 180, 100" : "100, 200, 255"

What changed?

When drawMode is true, particles are marked permanent (they never fade) and take a warm amber color (255, 180, 100). When draw mode is off, they use the default cool blue (100, 200, 255) and fade normally. This gives clear visual feedback when drawing.

• Now update updateParticles to respect the permanent flag. Find these lines inside the for loop:

p.vx *= 0.98;
p.vy *= 0.98;
p.opacity -= 0.015;
p.size *= 0.995;

• Replace those lines with this version that only fades non-permanent particles:

p.vx *= 0.98;
p.vy *= 0.98;

// Only fade particles that aren't permanent
if (!p.permanent) {
    p.opacity -= 0.015;
    p.size *= 0.995;
}

Before you test, what do you think will happen when you pinch to enable draw mode and then move your hand? Will the particles you draw stay on screen permanently?

• Save index.html and switch to your browser.
• Pinch your thumb and index finger together. The HUD should show "Draw Mode: ON" in blue.
• Move your hand while in draw mode. You should see warm amber strokes that stay permanently on the canvas.
• Pinch again to toggle draw mode off. Move your hand and confirm particles return to the cool blue fading trail.
• Open your palm wide. A burst of orange particles should explode outward.
• Close your fist. Everything on the canvas should clear.

Draw mode particles still fading?

Check that your if (!p.permanent) wraps BOTH the p.opacity -= 0.015 and p.size *= 0.995 lines. If only one is inside the condition, particles will still shrink to nothing.

Also confirm that spawnParticle uses permanent: drawMode (not permanent: false). The variable must reference the live drawMode boolean.

Help me fix my draw mode particle behavior

✔️ Awesome, I've got everything!

Great. Double check you have saved index.html and that all three gestures produce distinct results.

ⓧ I'd like to double check the full code

Your index.html should now contain all of these sections in order. Scroll through your file and confirm each is present:

• CSS rules for #status, #gesture, and #mode (three fixed-position overlays).
• HTML body with <video>, <canvas>, and three HUD divs (status, gesture, mode).
• State variables including drawMode, PINCH_THRESHOLD, and lastPinchState.
• processResults calling both spawnParticle and detectGestures.
• detectGestures function with pinch, open hand, and fist logic.
• spawnParticle using permanent: drawMode and conditional color.
• spawnBurst function spawning 60 orange particles outward.
• updateParticles with if (!p.permanent) guard around the fade logic.

Your hand is now a full multi-mode controller. Pinch to draw, spread to burst, fist to clear. In the Secret Mission, you will push this further by adding velocity-based rainbow coloring that responds to how fast you move.

Secret mission

Rainbow Velocity Mode

Your particle colors are static right now. What if the color responded to how fast you move your hand, creating natural rainbow gradients from slow deliberate strokes to fast sweeping gestures? In this mission, you will compute hand velocity from frame-to-frame position deltas, map it to HSL hue values, and blend the resulting colors into the existing particle system.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with no cloud services or paid APIs, so there are no ongoing costs.

Resources you used:

• Your project folder containing index.html on your Desktop.
• Webcam permission granted to localhost in Chrome.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with the particle canvas or want to revisit the gestures later.

• Your project folder stays on your Desktop, ready to open anytime.
• The webcam permission for localhost remains in Chrome so the app works immediately next time you start Live Server.

Pause - I'll come back to this later

Stop the running server to free up the port and terminal, but keep your files for later.

• In VS Code, click the Go Live button in the bottom status bar to stop Live Server. If you used python -m http.server, press Ctrl+C in the terminal to stop it.
• Close the browser tab running the particle canvas.
• Your project folder and files remain on your Desktop for next time.

Delete - I don't want to use this again

Remove all project files and revoke browser permissions for a clean slate.

• Stop Live Server by clicking the Go Live button in the VS Code status bar. If you used python -m http.server, press Ctrl+C in the terminal.
• Delete the project folder by running the command for your operating system:

macOS

rm -rf ~/Desktop/[[PROJECT_FOLDER="your project folder name"]]

• Replace the folder name with whatever you named your project folder on your Desktop.

Windows

Remove-Item -Recurse -Force ~\Desktop\[[PROJECT_FOLDER="your project folder name"]]

• Replace the folder name with whatever you named your project folder on your Desktop.

• Revoke webcam permission in Chrome: click the three-dot menu in the top right corner, then select Settings.
• In the left sidebar, click Privacy and security, then click Site settings.
• Under Permissions, click Camera.
• Find the localhost entry under Allowed to use your camera and click the trash icon to remove it.

That's a Wrap!

That's a Wrap!

Nice work! You just built a zero-dependency creative tool that turns your webcam into a light-painting controller using real-time hand tracking.

You've learned how to:

• Built a real-time hand tracking system using MediaPipe Hand Landmarker that detects 21 landmarks from webcam input and runs entirely in the browser with no backend.
• Created a particle system with additive blending and exponential position smoothing that transforms jittery raw tracking data into fluid, glowing trails.
• Implemented gesture recognition by computing distances between hand landmarks, wiring pinch, open hand, and fist poses to distinct visual effects like permanent drawing, particle bursts, and canvas clearing.
• Secret Mission: Added velocity-based rainbow coloring that shifts particle hue based on hand speed, making slow movements produce cool blues and fast movements produce warm reds.

Ready to quiz yourself?