Ambience Moodboard with Gesture Control

Build a browser ambient soundboard with ML gesture control and offline PWA.

Introduction

30 Second Summary

Ambient soundscapes help millions of people focus, relax, and sleep. Apps like Noisli and myNoise layer rain, fire, and ocean sounds to create the perfect atmosphere. But what if you could build that experience entirely in the browser, complete with hand gesture control and offline support?

In this project, you will build a browser-based ambient moodboard that mixes multiple sound layers, crossfades between mood presets, visualizes audio frequencies on a canvas, and responds to hand gestures for hands-free control.

What You'll Build

A React app combining the Web Audio API for multi-layer sound mixing, Canvas 2D for real-time frequency visualization, and MediaPipe Gesture Recognition running in a Web Worker for hands-free playback control.

By the end of this project, you'll have:

  • A custom audio engine using AudioContext, GainNodes, and AnalyserNode for multi-layer sound mixing with smooth crossfading presets.
  • A Canvas 2D visualizer rendering real-time frequency data using requestAnimationFrame.
  • MediaPipe gesture recognition offloaded to a Web Worker, keeping the main thread at 60fps while performing ML inference.
  • Secret Mission: Add a Service Worker that caches the app shell and audio assets for full offline-first PWA functionality.

Are there any prerequisites?

This project assumes you have around 3 years of React experience and are comfortable with hooks and component composition. You should have Node.js v22 installed, VS Code ready, and a free Vercel account for deployment.

Before We Start

Before We Start...

Scaffold the Project

Before you can build the moodboard, you need a working React development environment with the right libraries installed.

In this step, you will scaffold a Vite React project and install MediaPipe for gesture recognition. You will also create the folder structure and data files the app needs.

In this step, get ready to:
  • Scaffold a Vite React project and install MediaPipe.
  • Create the directory structure and audio data files.
  • Clean boilerplate and verify the dev server runs.
Scaffold the project and install dependencies
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Scaffold a new Vite React project by running this command:
npm create vite@latest ambience-moodboard -- --template react

What does this command do?

This tells npm to run the create-vite scaffolding tool. The --template react flag pre-selects React so you skip the interactive prompts entirely.

  • Move into the new project folder and install the base dependencies by running these commands:
cd ambience-moodboard
npm install
  • Install the MediaPipe vision library by running:
npm install @mediapipe/tasks-vision@0.10.35

Why a separate install for MediaPipe?

The @mediapipe/tasks-vision package gives you access to Google's pre-trained gesture recognition model. It is not part of the Vite template, so you install it as an additional dependency.

  • Confirm the packages installed by running:
ls node_modules/@mediapipe/tasks-vision

You should see a list of files inside the tasks-vision package folder.

Seeing permission errors or install failures?

Make sure you are inside the ambience-moodboard folder before running npm install. If you see EACCES errors on macOS, avoid using sudo and instead fix your npm permissions.

Help me fix npm permission issues

Add audio files and set up the project structure

The moodboard layers five ambient sounds together. You need real MP3 files for the audio to play.

  • Create the directory structure by running this command:
mkdir -p src/hooks src/workers src/components src/data public/audio

What do these folders hold?

  • src/hooks/ holds custom React hooks for the audio engine and gesture worker.
  • src/workers/ holds the Web Worker that runs ML inference off the main thread.
  • src/components/ holds the UI components (sound cards, visualizer, presets bar).
  • src/data/ holds static configuration (sound definitions and preset mappings).
  • public/audio/ holds MP3 files served as-is by Vite without processing.
  • Source five royalty-free ambient MP3 files (rain, fire, wind, birds, ocean) from a site like Pixabay or Freesound.
  • Save them into public/audio/ with these exact filenames: rain.mp3, fire.mp3, wind.mp3, birds.mp3, ocean.mp3.
  • Confirm the directories exist by running:
ls src/hooks src/workers src/components src/data public/audio

You should see each directory listed without errors.

  • Create the sound definitions file by adding this code to src/data/sounds.js:
export const sounds = [
  { id: 'rain', label: 'Rain', file: '/audio/rain.mp3' },
  { id: 'fire', label: 'Fire', file: '/audio/fire.mp3' },
  { id: 'wind', label: 'Wind', file: '/audio/wind.mp3' },
  { id: 'birds', label: 'Birds', file: '/audio/birds.mp3' },
  { id: 'ocean', label: 'Ocean', file: '/audio/ocean.mp3' },
];

What does this file do?

Each object maps a sound ID to its display label and file path. The paths point to public/audio/ because Vite serves files in public/ at the root URL.

  • Save src/data/sounds.js.
  • Create the presets file by adding this code to src/data/presets.js:
export const presets = [
  { id: 'focus', label: 'Focus', volumes: { rain: 0.6, fire: 0.3, wind: 0.2, birds: 0.1, ocean: 0 } },
  { id: 'sleep', label: 'Sleep', volumes: { rain: 0.3, fire: 0, wind: 0, birds: 0, ocean: 0.7 } },
  { id: 'storm', label: 'Storm', volumes: { rain: 0.9, fire: 0, wind: 0.8, birds: 0, ocean: 0.4 } },
  { id: 'forest', label: 'Forest', volumes: { rain: 0, fire: 0.5, wind: 0.3, birds: 0.7, ocean: 0 } },
];

What does this file do?

Each preset maps sound IDs to target volume levels (0 to 1). Later, the audio engine will crossfade between these volume targets smoothly using the Web Audio API.

  • Save src/data/presets.js.
  • Confirm your data files exist by running:
ls src/data

You should see presets.js and sounds.js in the output.

Missing files or wrong directory?

Double check you ran the mkdir -p command from inside the ambience-moodboard folder. If src/data/ does not exist, rerun the mkdir command.

Help me debug my project directory structure

Clean the scaffold boilerplate and start the dev server

The Vite scaffold comes with demo content you do not need. You will replace it with a minimal starting point.

  • Open src/App.jsx in VS Code.
  • Select all the content and replace it with this minimal component:
function App() {
  return <div></div>;
}
export default App;

Why start with an empty component?

The default scaffold includes logos, counters, and CSS you do not need. Starting empty gives you a clean canvas to build the moodboard UI in the next steps.

  • Save src/App.jsx.
  • Open src/App.css and delete all its content so the file is completely empty.
  • Save src/App.css.
  • Open src/main.jsx and replace all its content with this:
import { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import App from './App.jsx';
import './App.css';

createRoot(document.getElementById('root')).render(
  <StrictMode>
    <App />
  </StrictMode>
);

What changed in main.jsx?

The default scaffold imports index.css, but this project uses a single App.css for all styles. This replacement points the import to the correct file.

  • Save src/main.jsx.
  • Delete the files you no longer need by running:
rm src/index.css src/assets/react.svg
  • Start the dev server by running:
npm run dev
  • Open your browser and go to http://localhost:5173.

You should see a blank white page. This is your empty App component rendering successfully.

Page not loading or seeing an error?

If the terminal shows an error about missing files, make sure you saved both src/App.jsx and src/main.jsx after editing them. If main.jsx still imports index.css, the server will crash because you deleted that file.

Help me debug my Vite dev server

✔️ Awesome, I've got everything!

Great. Double check you have saved all files.

ⓧ I'd like to double check the full code

Here is the full code for each file at this point:

function App() {
  return <div></div>;
}
export default App;

Your src/App.css file should be completely empty (no content at all).

import { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import App from './App.jsx';
import './App.css';

createRoot(document.getElementById('root')).render(
  <StrictMode>
    <App />
  </StrictMode>
);
export const sounds = [
  { id: 'rain', label: 'Rain', file: '/audio/rain.mp3' },
  { id: 'fire', label: 'Fire', file: '/audio/fire.mp3' },
  { id: 'wind', label: 'Wind', file: '/audio/wind.mp3' },
  { id: 'birds', label: 'Birds', file: '/audio/birds.mp3' },
  { id: 'ocean', label: 'Ocean', file: '/audio/ocean.mp3' },
];
export const presets = [
  { id: 'focus', label: 'Focus', volumes: { rain: 0.6, fire: 0.3, wind: 0.2, birds: 0.1, ocean: 0 } },
  { id: 'sleep', label: 'Sleep', volumes: { rain: 0.3, fire: 0, wind: 0, birds: 0, ocean: 0.7 } },
  { id: 'storm', label: 'Storm', volumes: { rain: 0.9, fire: 0, wind: 0.8, birds: 0, ocean: 0.4 } },
  { id: 'forest', label: 'Forest', volumes: { rain: 0, fire: 0.5, wind: 0.3, birds: 0.7, ocean: 0 } },
];

Your development environment is ready and the project structure is in place. Next up, you will build the Web Audio engine that layers and crossfades the ambient sounds.

Build the Web Audio Engine

Your project scaffold is ready and your audio files are in place. Now you need an engine that can play multiple sounds at once, control their volumes independently, and smoothly crossfade between mood presets.

HTML5 Audio elements can play sound, but they cannot be mixed with sample-accurate timing, crossfaded with scheduling, or analyzed for visualization. The Web Audio API solves all three problems by letting you build a routing graph of audio nodes.

In this step, get ready to:
  • Create a custom React hook that builds a Web Audio API graph with per-layer volume control.
  • Implement smooth crossfading presets using scheduled gain ramps.
  • Wire the hook into a basic UI to verify audio plays and presets work.
Create the audio engine hook

The hook creates an AudioContext on first user interaction (browsers block audio until a gesture). It wires each sound through its own GainNode for independent volume control, funnels them into a master gain, then routes through an AnalyserNode (for visualization later) before reaching the speakers.

  • Create the file src/hooks/useAudioEngine.js and add the imports, state declarations, and the initAudio function by pasting this code:
import { useRef, useState, useCallback, useEffect } from 'react';
import { sounds } from '../data/sounds';

export function useAudioEngine() {
  const ctxRef = useRef(null);
  const layersRef = useRef({});
  const masterGainRef = useRef(null);
  const analyserRef = useRef(null);
  const [isPlaying, setIsPlaying] = useState(false);
  const [isMuted, setIsMuted] = useState(false);
  const [volumes, setVolumes] = useState(
    Object.fromEntries(sounds.map((s) => [s.id, 0.5]))
  );
  const [masterVolume, setMasterVolumeState] = useState(0.8);
  const [activePreset, setActivePreset] = useState(null);

  // Build the audio graph on first user interaction
  const initAudio = useCallback(() => {
    if (ctxRef.current) return;
    const ctx = new AudioContext();
    ctxRef.current = ctx;

    // Create analyser for future visualization
    const analyser = ctx.createAnalyser();
    analyser.fftSize = 256;
    analyserRef.current = analyser;

    // Master volume control
    const masterGain = ctx.createGain();
    masterGain.gain.value = 0.8;
    masterGainRef.current = masterGain;

    // Route: masterGain -> analyser -> speakers
    masterGain.connect(analyser);
    analyser.connect(ctx.destination);

    // Create a looping audio element + gain node per sound layer
    sounds.forEach((sound) => {
      const audio = new Audio(sound.file);
      audio.loop = true;
      audio.crossOrigin = 'anonymous';
      const source = ctx.createMediaElementSource(audio);
      const gain = ctx.createGain();
      gain.gain.value = volumes[sound.id];
      source.connect(gain);
      gain.connect(masterGain);
      layersRef.current[sound.id] = { audio, gain };
    });
  }, []);

What does this code do?

The audio graph routes signals like a mixing desk. Each sound layer has its own gain (volume knob), all feeding into a single master gain.

  • ctxRef holds the AudioContext, the entry point for all Web Audio operations.
  • layersRef maps each sound ID to its Audio element and GainNode.
  • crossOrigin = 'anonymous' is required on Audio elements routed through createMediaElementSource to avoid CORS issues.
  • The AnalyserNode (fftSize 256) sits between the master gain and ctx.destination so it can read frequency data without affecting the output.

You won't see a change yet. The next chunk adds playback controls that use this graph.

  • Add the playback control functions directly below the initAudio function (still inside useAudioEngine):
  // Toggle all layers between playing and paused
  const togglePlay = useCallback(() => {
    initAudio();
    if (isPlaying) {
      Object.values(layersRef.current).forEach(({ audio }) => audio.pause());
      setIsPlaying(false);
    } else {
      if (ctxRef.current.state === 'suspended') ctxRef.current.resume();
      Object.values(layersRef.current).forEach(({ audio }) => audio.play());
      setIsPlaying(true);
    }
  }, [isPlaying, initAudio]);

  // Set volume for a single sound layer
  const setLayerVolume = useCallback((id, value) => {
    const layer = layersRef.current[id];
    if (layer) layer.gain.gain.value = value;
    setVolumes((prev) => ({ ...prev, [id]: value }));
  }, []);

  // Set the master output volume
  const setMasterVolume = useCallback((value) => {
    if (masterGainRef.current) masterGainRef.current.gain.value = value;
    setMasterVolumeState(value);
  }, []);

What's happening here?

  • togglePlay() calls initAudio() first so the AudioContext is created on a user gesture (browser autoplay policy requires this).
  • If the context is suspended (common after tab backgrounding), it resumes before playing.
  • setLayerVolume writes directly to the GainNode's gain.value for instant volume changes on individual layers.

You won't see a change yet. The next substep completes the hook with crossfading logic.

Implement crossfading presets

Presets define target volumes for each layer. Jumping instantly between them sounds jarring. The Web Audio API's scheduling methods let you ramp smoothly from the current value to the target over a duration you control.

  • Add the applyPreset and toggleMute functions, the cleanup effect, and the return statement below your existing code to complete the hook:
  // Crossfade all layers to a preset's target volumes over 1.5 seconds
  const applyPreset = useCallback((preset) => {
    initAudio();
    const now = ctxRef.current.currentTime;
    Object.entries(preset.volumes).forEach(([id, target]) => {
      const layer = layersRef.current[id];
      if (layer) {
        layer.gain.gain.cancelScheduledValues(now);
        layer.gain.gain.setValueAtTime(layer.gain.gain.value, now);
        layer.gain.gain.linearRampToValueAtTime(target, now + 1.5);
      }
    });
    setVolumes(preset.volumes);
    setActivePreset(preset.id);
  }, [initAudio]);

  // Smooth mute/unmute over 0.3 seconds
  const toggleMute = useCallback(() => {
    if (!masterGainRef.current) return;
    const now = ctxRef.current.currentTime;
    if (isMuted) {
      masterGainRef.current.gain.setValueAtTime(0, now);
      masterGainRef.current.gain.linearRampToValueAtTime(masterVolume, now + 0.3);
    } else {
      masterGainRef.current.gain.setValueAtTime(masterVolume, now);
      masterGainRef.current.gain.linearRampToValueAtTime(0, now + 0.3);
    }
    setIsMuted(!isMuted);
  }, [isMuted, masterVolume]);

  // Close the AudioContext when the component unmounts
  useEffect(() => () => { if (ctxRef.current) ctxRef.current.close(); }, []);

  return {
    isPlaying, isMuted, volumes, masterVolume, activePreset,
    analyser: analyserRef.current,
    togglePlay, setLayerVolume, setMasterVolume, applyPreset, toggleMute,
  };
}

How does the crossfade work?

Three Web Audio scheduling methods work together to create a smooth ramp:

  • cancelScheduledValues(now) clears any in-progress ramp so the new preset takes over cleanly.
  • setValueAtTime(currentValue, now) anchors the starting point at whatever the gain is right now.
  • linearRampToValueAtTime(target, now + 1.5) tells the browser to interpolate linearly over 1.5 seconds to the target volume.

This runs at the audio sample rate, so transitions are perfectly smooth with no audible stepping.

Why cancel before setting?

If the learner clicks two presets quickly, the first ramp would still be running. Without cancelScheduledValues, the second ramp would queue behind the first instead of replacing it. Cancelling ensures the newest preset always wins immediately.

  • Save src/hooks/useAudioEngine.js.
  • Check that VS Code shows no red underlines or syntax errors in the file.

Seeing red underlines?

  • Make sure the closing } on the last line closes the useAudioEngine function. Every opening brace needs a matching close.
  • Check that you pasted each chunk in order without accidentally duplicating imports or the function declaration.

Seeing something else? Help me fix syntax errors in my useAudioEngine hook.

Test the engine with a basic UI

The hook is complete but nothing uses it yet. You need a minimal UI to confirm audio plays and presets crossfade before building the full moodboard.

  • Open src/App.jsx and replace its contents with this code:
import { useAudioEngine } from './hooks/useAudioEngine';
import { presets } from './data/presets';
import './App.css';

function App() {
  // Destructure only what we need for this test
  const { isPlaying, togglePlay, applyPreset } = useAudioEngine();
  return (
    <div>
      <button onClick={togglePlay}>{isPlaying ? 'Pause' : 'Play'}</button>
      {presets.map(p => <button key={p.id} onClick={() => applyPreset(p)}>{p.label}</button>)}
    </div>
  );
}
export default App;

What does this code do?

  • The Play button triggers togglePlay(), which creates the AudioContext on the first click (satisfying browser autoplay policy) and starts all sound layers.
  • Each preset button calls applyPreset(p) with the preset object, triggering 1.5-second crossfades across all layers.
  • The button label toggles between Play and Pause based on the isPlaying state.
  • Save src/App.jsx.
  • Switch to your browser at localhost:5173. You should see a Play button and four preset buttons (Focus, Sleep, Storm, Forest).
  • Click Play. You should hear all five ambient sounds playing simultaneously.
  • Click Focus, wait a moment, then click Sleep. You should hear the sound mix shift gradually over about 1.5 seconds as layers fade in and out.

No sound when you click Play?

  • Confirm your audio files exist at public/audio/rain.mp3, public/audio/fire.mp3, etc. Open the browser DevTools Network tab and check for 404 errors on the audio files.
  • Check your browser's address bar for a blocked-audio icon. Some browsers mute tabs by default until you interact with the page.
  • Open the browser console. If you see NotAllowedError, it means the AudioContext was created outside a user gesture. Make sure initAudio() is called inside togglePlay() (which fires on a click).

Still stuck? Help me debug why my Web Audio engine isn't producing sound.

✔️ Awesome, I've got everything!

Great. Double check you've saved both files before moving on.

ⓧ I'd like to double check the full code

Here is the complete src/hooks/useAudioEngine.js:

import { useRef, useState, useCallback, useEffect } from 'react';
import { sounds } from '../data/sounds';

export function useAudioEngine() {
  const ctxRef = useRef(null);
  const layersRef = useRef({});
  const masterGainRef = useRef(null);
  const analyserRef = useRef(null);
  const [isPlaying, setIsPlaying] = useState(false);
  const [isMuted, setIsMuted] = useState(false);
  const [volumes, setVolumes] = useState(
    Object.fromEntries(sounds.map((s) => [s.id, 0.5]))
  );
  const [masterVolume, setMasterVolumeState] = useState(0.8);
  const [activePreset, setActivePreset] = useState(null);

  const initAudio = useCallback(() => {
    if (ctxRef.current) return;
    const ctx = new AudioContext();
    ctxRef.current = ctx;

    const analyser = ctx.createAnalyser();
    analyser.fftSize = 256;
    analyserRef.current = analyser;

    const masterGain = ctx.createGain();
    masterGain.gain.value = 0.8;
    masterGainRef.current = masterGain;

    masterGain.connect(analyser);
    analyser.connect(ctx.destination);

    sounds.forEach((sound) => {
      const audio = new Audio(sound.file);
      audio.loop = true;
      audio.crossOrigin = 'anonymous';
      const source = ctx.createMediaElementSource(audio);
      const gain = ctx.createGain();
      gain.gain.value = volumes[sound.id];
      source.connect(gain);
      gain.connect(masterGain);
      layersRef.current[sound.id] = { audio, gain };
    });
  }, []);

  const togglePlay = useCallback(() => {
    initAudio();
    if (isPlaying) {
      Object.values(layersRef.current).forEach(({ audio }) => audio.pause());
      setIsPlaying(false);
    } else {
      if (ctxRef.current.state === 'suspended') ctxRef.current.resume();
      Object.values(layersRef.current).forEach(({ audio }) => audio.play());
      setIsPlaying(true);
    }
  }, [isPlaying, initAudio]);

  const setLayerVolume = useCallback((id, value) => {
    const layer = layersRef.current[id];
    if (layer) layer.gain.gain.value = value;
    setVolumes((prev) => ({ ...prev, [id]: value }));
  }, []);

  const setMasterVolume = useCallback((value) => {
    if (masterGainRef.current) masterGainRef.current.gain.value = value;
    setMasterVolumeState(value);
  }, []);

  const applyPreset = useCallback((preset) => {
    initAudio();
    const now = ctxRef.current.currentTime;
    Object.entries(preset.volumes).forEach(([id, target]) => {
      const layer = layersRef.current[id];
      if (layer) {
        layer.gain.gain.cancelScheduledValues(now);
        layer.gain.gain.setValueAtTime(layer.gain.gain.value, now);
        layer.gain.gain.linearRampToValueAtTime(target, now + 1.5);
      }
    });
    setVolumes(preset.volumes);
    setActivePreset(preset.id);
  }, [initAudio]);

  const toggleMute = useCallback(() => {
    if (!masterGainRef.current) return;
    const now = ctxRef.current.currentTime;
    if (isMuted) {
      masterGainRef.current.gain.setValueAtTime(0, now);
      masterGainRef.current.gain.linearRampToValueAtTime(masterVolume, now + 0.3);
    } else {
      masterGainRef.current.gain.setValueAtTime(masterVolume, now);
      masterGainRef.current.gain.linearRampToValueAtTime(0, now + 0.3);
    }
    setIsMuted(!isMuted);
  }, [isMuted, masterVolume]);

  useEffect(() => () => { if (ctxRef.current) ctxRef.current.close(); }, []);

  return {
    isPlaying, isMuted, volumes, masterVolume, activePreset,
    analyser: analyserRef.current,
    togglePlay, setLayerVolume, setMasterVolume, applyPreset, toggleMute,
  };
}

And here is the complete src/App.jsx:

import { useAudioEngine } from './hooks/useAudioEngine';
import { presets } from './data/presets';
import './App.css';

function App() {
  const { isPlaying, togglePlay, applyPreset } = useAudioEngine();
  return (
    <div>
      <button onClick={togglePlay}>{isPlaying ? 'Pause' : 'Play'}</button>
      {presets.map(p => <button key={p.id} onClick={() => applyPreset(p)}>{p.label}</button>)}
    </div>
  );
}
export default App;

Your audio engine is working. You have independent layer volumes, smooth crossfading presets, and a clean hook API. Next up, you'll build the full moodboard UI with glassmorphism cards, volume sliders, and a Canvas visualizer that reacts to the audio in real time.

Build the Moodboard UI and Canvas Visualizer

Your audio engine is humming along under the hood. Sounds mix, presets crossfade, and the AnalyserNode is ready to pipe frequency data wherever you need it.

Now you need a UI that actually feels like a moodboard. In this step, you will build dark-themed sound cards with individual volume sliders, a preset bar with visual feedback, and a Canvas 2D visualizer that renders real-time frequency data from your audio mix using requestAnimationFrame.

In this step, get ready to:
  • Create reusable SoundLayer and MoodPresets components.
  • Build a Canvas audio visualizer driven by the AnalyserNode.
  • Wire everything together with a polished dark-themed layout.
Build the moodboard layout with sound cards

Each sound layer in the moodboard needs its own card showing the sound name, a volume slider, and a percentage readout. You will create two components: one for individual sound cards, and one for the preset buttons bar.

  • Create a new file at src/components/SoundLayer.jsx and add this component:
export function SoundLayer({ sound, volume, onVolumeChange }) {
  return (
    <div className={`sound-layer ${volume > 0 ? 'active' : ''}`}>
      <span className="sound-label">{sound.label}</span>
      <input
        type="range"
        min="0"
        max="1"
        step="0.01"
        value={volume}
        onChange={(e) => onVolumeChange(sound.id, parseFloat(e.target.value))}
        className="volume-slider"
      />
      <span className="volume-value">{Math.round(volume * 100)}%</span>
      <div className="volume-bar" style={{ width: `${volume * 100}%` }} />
    </div>
  );
}

What does this component do?

  • The outer div gets an active class whenever volume is above zero, allowing CSS to highlight cards that are producing sound.
  • The range input slides from 0 to 1 in 0.01 increments, calling the parent's onVolumeChange with the sound ID and new value.
  • The volume-bar div is a visual indicator at the bottom of the card whose width matches the current volume percentage.
  • Save SoundLayer.jsx.
  • Create a new file at src/components/MoodPresets.jsx and add this component:
import { presets } from '../data/presets';

export function MoodPresets({ activePreset, onSelectPreset }) {
  return (
    <div className="presets-bar">
      {presets.map((preset) => (
        <button
          key={preset.id}
          className={`preset-btn ${activePreset === preset.id ? 'active' : ''}`}
          onClick={() => onSelectPreset(preset)}
        >
          {preset.label}
        </button>
      ))}
    </div>
  );
}

What does this component do?

MoodPresets imports the presets array and renders a button for each one. When a preset matches the activePreset prop, it gets an active class for visual highlighting. Clicking a button calls onSelectPreset with the full preset object, which triggers the crossfade logic in your audio engine.

  • Save MoodPresets.jsx.
Build the Canvas audio visualizer

The visualizer reads frequency data from the AnalyserNode every animation frame and draws gradient-colored bars on a Canvas element. It runs as a fixed background behind the entire UI.

  • Create a new file at src/components/Visualizer.jsx and add this component:
import { useRef, useEffect } from 'react';

// Map each preset to a theme color for the visualizer bars
const PRESET_COLORS = {
  focus: '#6366f1',
  sleep: '#0ea5e9',
  storm: '#64748b',
  forest: '#22c55e',
};

export function Visualizer({ analyser, activePreset }) {
  const canvasRef = useRef(null);
  const animRef = useRef(null);

  useEffect(() => {
    if (!analyser) return;
    const canvas = canvasRef.current;
    const ctx = canvas.getContext('2d');
    const bufferLength = analyser.frequencyBinCount;
    const dataArray = new Uint8Array(bufferLength);

    const draw = () => {
      // Scale canvas for crisp rendering on high-DPI displays
      canvas.width = canvas.offsetWidth * devicePixelRatio;
      canvas.height = canvas.offsetHeight * devicePixelRatio;
      ctx.scale(devicePixelRatio, devicePixelRatio);
      const w = canvas.offsetWidth;
      const h = canvas.offsetHeight;

      // Read current frequency data from the AnalyserNode
      analyser.getByteFrequencyData(dataArray);
      ctx.clearRect(0, 0, w, h);

      // Pick bar color based on active preset
      const color = PRESET_COLORS[activePreset] || PRESET_COLORS.focus;
      const barWidth = w / bufferLength;

      // Draw a bar for each frequency bin
      for (let i = 0; i < bufferLength; i++) {
        const barHeight = (dataArray[i] / 255) * h * 0.8;
        ctx.fillStyle = color;
        ctx.globalAlpha = 0.6;
        ctx.shadowColor = color;
        ctx.shadowBlur = 6;
        ctx.fillRect(i * barWidth, h - barHeight, barWidth - 1, barHeight);
      }
      ctx.globalAlpha = 1;
      animRef.current = requestAnimationFrame(draw);
    };

    draw();
    return () => { if (animRef.current) cancelAnimationFrame(animRef.current); };
  }, [analyser, activePreset]);

  return <canvas ref={canvasRef} className="visualizer" />;
}

How does the visualizer rendering loop work?

  • The useEffect kicks off a requestAnimationFrame loop that runs at ~60fps, synced to the browser's repaint cycle.
  • Each frame, getByteFrequencyData() fills an array with 128 frequency bins (half of fftSize 256), each valued 0-255.
  • The canvas dimensions are multiplied by devicePixelRatio so bars render sharply on Retina displays.
  • The shadowBlur property on the canvas context gives each bar a soft glow that matches the preset color.

Why requestAnimationFrame instead of setInterval?

requestAnimationFrame fires exactly once per display refresh and automatically pauses when the tab is hidden. This saves CPU and guarantees smooth rendering without tearing.

  • Save Visualizer.jsx.
Wire everything together with the full layout

With all three components ready, you need a stylesheet for the dark theme and glassmorphism effect, then a rewritten App.jsx that assembles the full layout.

  • Open src/App.css and replace its contents with the full stylesheet:
:root {
  --bg: #0f0f1a;
  --card: rgba(255,255,255,0.04);
  --border: rgba(255,255,255,0.08);
  --text: #eee;
  --text-dim: #888;
  --accent: #6366f1;
  --radius: 12px;
}
* { margin:0; padding:0; box-sizing:border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, sans-serif; background: var(--bg); color: var(--text); min-height: 100vh; }
.app { min-height:100vh; padding:2rem; position:relative; transition: background 0.4s ease; }
.app.preset-focus { background: linear-gradient(135deg, #0f0f1a, #1e1b4b); }
.app.preset-sleep { background: linear-gradient(135deg, #0f0f1a, #0c4a6e); }
.app.preset-storm { background: linear-gradient(135deg, #0f0f1a, #1e293b); }
.app.preset-forest { background: linear-gradient(135deg, #0f0f1a, #14532d); }

.visualizer { position:fixed; bottom:0; left:0; width:100%; height:180px; pointer-events:none; z-index:0; }
.header { position:relative; z-index:1; display:flex; justify-content:space-between; align-items:center; margin-bottom:1.5rem; }
.header h1 { font-size:1.5rem; font-weight:700; }
.controls { display:flex; align-items:center; gap:0.75rem; }
.play-btn, .mute-btn, .gesture-toggle { padding:0.5rem 1rem; border:1px solid var(--border); border-radius:var(--radius); background:var(--card); color:var(--text); cursor:pointer; backdrop-filter:blur(8px); transition:all 0.2s; }
.play-btn:hover, .mute-btn:hover, .gesture-toggle:hover { border-color:var(--accent); }
.play-btn.playing { background:var(--accent); border-color:var(--accent); }
.mute-btn.muted { background:rgba(239,68,68,0.2); border-color:rgba(239,68,68,0.5); }
.master-slider { width:100px; accent-color:var(--accent); }

.presets-bar { position:relative; z-index:1; display:flex; gap:0.5rem; margin-bottom:1.5rem; }
.preset-btn { padding:0.4rem 1rem; border:1px solid var(--border); border-radius:20px; background:var(--card); color:var(--text-dim); cursor:pointer; transition:all 0.2s; }
.preset-btn.active { border-color:var(--accent); color:var(--text); box-shadow:0 0 12px rgba(99,102,241,0.3); }

.layers-grid { position:relative; z-index:1; display:grid; grid-template-columns:repeat(auto-fill, minmax(180px, 1fr)); gap:1rem; margin-bottom:2rem; }
.sound-layer { position:relative; padding:1.25rem; background:var(--card); border:1px solid var(--border); border-radius:var(--radius); backdrop-filter:blur(8px); overflow:hidden; transition:all 0.2s; }
.sound-layer.active { border-color:var(--accent); }
.sound-label { font-weight:500; display:block; margin-bottom:0.5rem; }
.volume-slider { width:100%; accent-color:var(--accent); }
.volume-value { font-size:0.75rem; color:var(--text-dim); }
.volume-bar { position:absolute; bottom:0; left:0; height:3px; background:var(--accent); transition:width 0.3s; }

.gesture-section { position:relative; z-index:1; }
.gesture-toggle.active { background:rgba(16,185,129,0.15); border-color:rgba(16,185,129,0.5); }
.gesture-overlay { display:none; margin-top:0.75rem; width:280px; border-radius:var(--radius); overflow:hidden; border:1px solid var(--border); }
.gesture-overlay.visible { display:block; }
.gesture-video { width:100%; display:block; transform:scaleX(-1); }
.gesture-label { padding:0.5rem; text-align:center; font-size:0.85rem; color:var(--accent); min-height:2rem; }
.gesture-legend { display:flex; flex-wrap:wrap; gap:0.5rem; margin-top:0.75rem; }
.legend-item { font-size:0.7rem; color:var(--text-dim); padding:0.25rem 0.5rem; background:var(--card); border:1px solid var(--border); border-radius:6px; }

What makes this a glassmorphism design?

  • The --card variable uses rgba(255,255,255,0.04) for near-transparent backgrounds that reveal the gradient beneath.
  • backdrop-filter:blur(8px) on buttons and cards frosts the background behind them.
  • Each .app.preset-* class applies a different gradient. The transition: background 0.4s ease on .app animates the color shift when presets change.
  • Save App.css.
  • Now open src/App.jsx and replace the entire file with the full layout that imports and connects all your new components:
import { useCallback, useRef } from 'react';
import { useAudioEngine } from './hooks/useAudioEngine';
import { sounds } from './data/sounds';
import { presets } from './data/presets';
import { SoundLayer } from './components/SoundLayer';
import { MoodPresets } from './components/MoodPresets';
import { Visualizer } from './components/Visualizer';
import './App.css';

function App() {
  const {
    isPlaying, isMuted, volumes, masterVolume, activePreset, analyser,
    togglePlay, setLayerVolume, setMasterVolume, applyPreset, toggleMute,
  } = useAudioEngine();

  return (
    <div className={`app preset-${activePreset || 'focus'}`}>
      {/* Visualizer renders as a fixed canvas behind everything */}
      <Visualizer analyser={analyser} activePreset={activePreset} />
      <header className="header">
        <h1>Ambience</h1>
        <div className="controls">
          <button className={`play-btn ${isPlaying ? 'playing' : ''}`} onClick={togglePlay}>
            {isPlaying ? 'Pause' : 'Play'}
          </button>
          <button className={`mute-btn ${isMuted ? 'muted' : ''}`} onClick={toggleMute}>
            {isMuted ? 'Unmute' : 'Mute'}
          </button>
          <input type="range" min="0" max="1" step="0.01" value={masterVolume}
            onChange={(e) => setMasterVolume(parseFloat(e.target.value))} className="master-slider" />
        </div>
      </header>
      {/* Preset buttons trigger crossfade transitions */}
      <MoodPresets activePreset={activePreset} onSelectPreset={applyPreset} />
      {/* Sound cards with individual volume control */}
      <main className="layers-grid">
        {sounds.map((s) => (
          <SoundLayer key={s.id} sound={s} volume={volumes[s.id]} onVolumeChange={setLayerVolume} />
        ))}
      </main>
    </div>
  );
}

export default App;

What changed in App.jsx?

  • The component now destructures all values from useAudioEngine() including volumes, masterVolume, analyser, and the volume setters.
  • The root div uses a dynamic className like preset-focus or preset-sleep to trigger the CSS gradient backgrounds.
  • Visualizer sits first in the DOM so it renders behind everything (fixed positioning + z-index 0).
  • Each SoundLayer receives its current volume from the engine state and passes changes back via setLayerVolume.
  • Save App.jsx and check your browser at localhost:5173.
  • You should see a dark-themed moodboard with five sound cards arranged in a grid, preset buttons at the top, and Play/Mute/volume controls in the header.
  • Click Play to start audio, then click a preset button. The background gradient should shift color over 0.4 seconds and the audio volumes should crossfade over 1.5 seconds.
  • While audio plays, look at the bottom of the page. You should see colored bars animating in response to the frequency data.

Don't see the visualizer bars?

  • The visualizer only animates when audio is playing. Click Play first and make sure at least one layer has volume above 0.
  • The analyser ref is null until the first user click creates the AudioContext. The effect guards against this with if (!analyser) return, so it starts drawing only after the context exists.
  • Check that your CSS has .visualizer set to position:fixed with height:180px. A height of 0 means nothing renders.

Help me debug my Canvas visualizer

✔️ Awesome, I've got everything!

Great. Double check you have saved all five files: SoundLayer.jsx, MoodPresets.jsx, Visualizer.jsx, App.css, and App.jsx.

ⓧ I'd like to double check the full code

Here is each file in full. Compare yours character by character.

export function SoundLayer({ sound, volume, onVolumeChange }) {
  return (
    <div className={`sound-layer ${volume > 0 ? 'active' : ''}`}>
      <span className="sound-label">{sound.label}</span>
      <input
        type="range"
        min="0"
        max="1"
        step="0.01"
        value={volume}
        onChange={(e) => onVolumeChange(sound.id, parseFloat(e.target.value))}
        className="volume-slider"
      />
      <span className="volume-value">{Math.round(volume * 100)}%</span>
      <div className="volume-bar" style={{ width: `${volume * 100}%` }} />
    </div>
  );
}
import { presets } from '../data/presets';

export function MoodPresets({ activePreset, onSelectPreset }) {
  return (
    <div className="presets-bar">
      {presets.map((preset) => (
        <button
          key={preset.id}
          className={`preset-btn ${activePreset === preset.id ? 'active' : ''}`}
          onClick={() => onSelectPreset(preset)}
        >
          {preset.label}
        </button>
      ))}
    </div>
  );
}
import { useRef, useEffect } from 'react';

const PRESET_COLORS = {
  focus: '#6366f1',
  sleep: '#0ea5e9',
  storm: '#64748b',
  forest: '#22c55e',
};

export function Visualizer({ analyser, activePreset }) {
  const canvasRef = useRef(null);
  const animRef = useRef(null);

  useEffect(() => {
    if (!analyser) return;
    const canvas = canvasRef.current;
    const ctx = canvas.getContext('2d');
    const bufferLength = analyser.frequencyBinCount;
    const dataArray = new Uint8Array(bufferLength);

    const draw = () => {
      canvas.width = canvas.offsetWidth * devicePixelRatio;
      canvas.height = canvas.offsetHeight * devicePixelRatio;
      ctx.scale(devicePixelRatio, devicePixelRatio);
      const w = canvas.offsetWidth;
      const h = canvas.offsetHeight;

      analyser.getByteFrequencyData(dataArray);
      ctx.clearRect(0, 0, w, h);

      const color = PRESET_COLORS[activePreset] || PRESET_COLORS.focus;
      const barWidth = w / bufferLength;

      for (let i = 0; i < bufferLength; i++) {
        const barHeight = (dataArray[i] / 255) * h * 0.8;
        ctx.fillStyle = color;
        ctx.globalAlpha = 0.6;
        ctx.shadowColor = color;
        ctx.shadowBlur = 6;
        ctx.fillRect(i * barWidth, h - barHeight, barWidth - 1, barHeight);
      }
      ctx.globalAlpha = 1;
      animRef.current = requestAnimationFrame(draw);
    };

    draw();
    return () => { if (animRef.current) cancelAnimationFrame(animRef.current); };
  }, [analyser, activePreset]);

  return <canvas ref={canvasRef} className="visualizer" />;
}
:root {
  --bg: #0f0f1a;
  --card: rgba(255,255,255,0.04);
  --border: rgba(255,255,255,0.08);
  --text: #eee;
  --text-dim: #888;
  --accent: #6366f1;
  --radius: 12px;
}
* { margin:0; padding:0; box-sizing:border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, sans-serif; background: var(--bg); color: var(--text); min-height: 100vh; }
.app { min-height:100vh; padding:2rem; position:relative; transition: background 0.4s ease; }
.app.preset-focus { background: linear-gradient(135deg, #0f0f1a, #1e1b4b); }
.app.preset-sleep { background: linear-gradient(135deg, #0f0f1a, #0c4a6e); }
.app.preset-storm { background: linear-gradient(135deg, #0f0f1a, #1e293b); }
.app.preset-forest { background: linear-gradient(135deg, #0f0f1a, #14532d); }

.visualizer { position:fixed; bottom:0; left:0; width:100%; height:180px; pointer-events:none; z-index:0; }
.header { position:relative; z-index:1; display:flex; justify-content:space-between; align-items:center; margin-bottom:1.5rem; }
.header h1 { font-size:1.5rem; font-weight:700; }
.controls { display:flex; align-items:center; gap:0.75rem; }
.play-btn, .mute-btn, .gesture-toggle { padding:0.5rem 1rem; border:1px solid var(--border); border-radius:var(--radius); background:var(--card); color:var(--text); cursor:pointer; backdrop-filter:blur(8px); transition:all 0.2s; }
.play-btn:hover, .mute-btn:hover, .gesture-toggle:hover { border-color:var(--accent); }
.play-btn.playing { background:var(--accent); border-color:var(--accent); }
.mute-btn.muted { background:rgba(239,68,68,0.2); border-color:rgba(239,68,68,0.5); }
.master-slider { width:100px; accent-color:var(--accent); }

.presets-bar { position:relative; z-index:1; display:flex; gap:0.5rem; margin-bottom:1.5rem; }
.preset-btn { padding:0.4rem 1rem; border:1px solid var(--border); border-radius:20px; background:var(--card); color:var(--text-dim); cursor:pointer; transition:all 0.2s; }
.preset-btn.active { border-color:var(--accent); color:var(--text); box-shadow:0 0 12px rgba(99,102,241,0.3); }

.layers-grid { position:relative; z-index:1; display:grid; grid-template-columns:repeat(auto-fill, minmax(180px, 1fr)); gap:1rem; margin-bottom:2rem; }
.sound-layer { position:relative; padding:1.25rem; background:var(--card); border:1px solid var(--border); border-radius:var(--radius); backdrop-filter:blur(8px); overflow:hidden; transition:all 0.2s; }
.sound-layer.active { border-color:var(--accent); }
.sound-label { font-weight:500; display:block; margin-bottom:0.5rem; }
.volume-slider { width:100%; accent-color:var(--accent); }
.volume-value { font-size:0.75rem; color:var(--text-dim); }
.volume-bar { position:absolute; bottom:0; left:0; height:3px; background:var(--accent); transition:width 0.3s; }

.gesture-section { position:relative; z-index:1; }
.gesture-toggle.active { background:rgba(16,185,129,0.15); border-color:rgba(16,185,129,0.5); }
.gesture-overlay { display:none; margin-top:0.75rem; width:280px; border-radius:var(--radius); overflow:hidden; border:1px solid var(--border); }
.gesture-overlay.visible { display:block; }
.gesture-video { width:100%; display:block; transform:scaleX(-1); }
.gesture-label { padding:0.5rem; text-align:center; font-size:0.85rem; color:var(--accent); min-height:2rem; }
.gesture-legend { display:flex; flex-wrap:wrap; gap:0.5rem; margin-top:0.75rem; }
.legend-item { font-size:0.7rem; color:var(--text-dim); padding:0.25rem 0.5rem; background:var(--card); border:1px solid var(--border); border-radius:6px; }
import { useCallback, useRef } from 'react';
import { useAudioEngine } from './hooks/useAudioEngine';
import { sounds } from './data/sounds';
import { presets } from './data/presets';
import { SoundLayer } from './components/SoundLayer';
import { MoodPresets } from './components/MoodPresets';
import { Visualizer } from './components/Visualizer';
import './App.css';

function App() {
  const {
    isPlaying, isMuted, volumes, masterVolume, activePreset, analyser,
    togglePlay, setLayerVolume, setMasterVolume, applyPreset, toggleMute,
  } = useAudioEngine();

  return (
    <div className={`app preset-${activePreset || 'focus'}`}>
      <Visualizer analyser={analyser} activePreset={activePreset} />
      <header className="header">
        <h1>Ambience</h1>
        <div className="controls">
          <button className={`play-btn ${isPlaying ? 'playing' : ''}`} onClick={togglePlay}>
            {isPlaying ? 'Pause' : 'Play'}
          </button>
          <button className={`mute-btn ${isMuted ? 'muted' : ''}`} onClick={toggleMute}>
            {isMuted ? 'Unmute' : 'Mute'}
          </button>
          <input type="range" min="0" max="1" step="0.01" value={masterVolume}
            onChange={(e) => setMasterVolume(parseFloat(e.target.value))} className="master-slider" />
        </div>
      </header>
      <MoodPresets activePreset={activePreset} onSelectPreset={applyPreset} />
      <main className="layers-grid">
        {sounds.map((s) => (
          <SoundLayer key={s.id} sound={s} volume={volumes[s.id]} onVolumeChange={setLayerVolume} />
        ))}
      </main>
    </div>
  );
}

export default App;

Your moodboard looks and sounds great. Next up, you will add hands-free control by offloading MediaPipe gesture recognition to a Web Worker so the main thread stays at 60fps.

Add Gesture Recognition via Web Worker

Your moodboard looks and sounds great, but every interaction still requires a click or slider drag.

In this step, you will add hands-free gesture control using MediaPipe for hand recognition. The key challenge is that ML inference is expensive. Running it on the main thread would freeze your UI and drop frames from the Canvas visualizer. You will offload all ML work to a Web Worker, keeping your main thread at 60fps.

In this step, get ready to:
  • Create a Web Worker that loads MediaPipe and runs gesture inference off the main thread.
  • Build a hook that captures webcam frames and transfers them to the worker.
  • Connect gesture actions to the audio engine for hands-free playback control.
Create the gesture recognition Web Worker

MediaPipe's recognizeForVideo() method runs synchronously and blocks the calling thread. By placing it inside a Web Worker, the heavy ML computation happens on a completely separate thread. Your UI thread never stalls.

Why a Web Worker for ML inference?

The browser's main thread handles rendering, event handling, and your audio scheduling all at once. MediaPipe's gesture recognition processes every video frame through a neural network. On the main thread, that would block requestAnimationFrame and freeze the visualizer.

A Web Worker runs in its own thread with its own event loop. You send it video frames via postMessage, it runs inference, and posts results back. The main thread never waits.

  • Create the gesture worker file by adding the following code to src/workers/gesture.worker.js:
import { FilesetResolver, GestureRecognizer } from '@mediapipe/tasks-vision';

let recognizer = null;

async function init() {
  const vision = await FilesetResolver.forVisionTasks(
    'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.35/wasm'
  );
  recognizer = await GestureRecognizer.createFromOptions(vision, {
    baseOptions: {
      modelAssetPath:
        'https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/1/gesture_recognizer.task',
    },
    runningMode: 'VIDEO',
    numHands: 1,
  });
  self.postMessage({ type: 'ready' });
}

self.onmessage = async (e) => {
  const { type, frame, timestamp } = e.data;

  if (type === 'init') {
    await init();
    return;
  }

  if (type === 'detect' && recognizer) {
    const result = recognizer.recognizeForVideo(frame, timestamp);
    frame.close();

    if (result.gestures && result.gestures.length > 0) {
      const gesture = result.gestures[0][0];
      self.postMessage({
        type: 'result',
        gesture: gesture.categoryName,
        score: gesture.score,
      });
    } else {
      self.postMessage({ type: 'result', gesture: 'None', score: 0 });
    }
  }
};

What does this code do?

  • The init() function loads MediaPipe's WASM runtime from the CDN, then creates a GestureRecognizer configured for video mode with one hand.
  • When the worker receives an 'init' message, it loads the model and posts 'ready' back to the main thread.
  • On each 'detect' message, it runs inference on the transferred ImageBitmap frame, closes the frame to free memory, and posts the recognized gesture name and confidence score back.
  • Save src/workers/gesture.worker.js.

You will not see a visible change yet. The next chunk creates the hook that communicates with this worker.

Seeing an import error in the worker file?

Make sure the file is inside src/workers/ (not public/). Vite bundles worker imports differently from static files.

Confirm the @mediapipe/tasks-vision package is installed. If not, run npm install @mediapipe/tasks-vision@0.10.35 from your project root.

help me debug a Web Worker import error for MediaPipe

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import { FilesetResolver, GestureRecognizer } from '@mediapipe/tasks-vision';

let recognizer = null;

async function init() {
  const vision = await FilesetResolver.forVisionTasks(
    'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.35/wasm'
  );
  recognizer = await GestureRecognizer.createFromOptions(vision, {
    baseOptions: {
      modelAssetPath:
        'https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/1/gesture_recognizer.task',
    },
    runningMode: 'VIDEO',
    numHands: 1,
  });
  self.postMessage({ type: 'ready' });
}

self.onmessage = async (e) => {
  const { type, frame, timestamp } = e.data;

  if (type === 'init') {
    await init();
    return;
  }

  if (type === 'detect' && recognizer) {
    const result = recognizer.recognizeForVideo(frame, timestamp);
    frame.close();

    if (result.gestures && result.gestures.length > 0) {
      const gesture = result.gestures[0][0];
      self.postMessage({
        type: 'result',
        gesture: gesture.categoryName,
        score: gesture.score,
      });
    } else {
      self.postMessage({ type: 'result', gesture: 'None', score: 0 });
    }
  }
};
Create the useGestureWorker hook

The main thread needs to capture webcam frames, send them to the worker, and handle results. A custom hook encapsulates all of this communication cleanly.

  • Create the gesture hook by adding the following code to src/hooks/useGestureWorker.js:
import { useRef, useState, useCallback, useEffect } from 'react';

export function useGestureWorker(onGesture) {
  const workerRef = useRef(null);
  const videoRef = useRef(null);
  const animRef = useRef(null);
  const lastGestureRef = useRef(0);
  const [isActive, setIsActive] = useState(false);
  const [isLoading, setIsLoading] = useState(false);
  const [currentGesture, setCurrentGesture] = useState('None');
  const onGestureRef = useRef(onGesture);
  onGestureRef.current = onGesture;

  const start = useCallback(async () => {
    setIsLoading(true);

    const worker = new Worker(
      new URL('../workers/gesture.worker.js', import.meta.url),
      { type: 'module' }
    );
    workerRef.current = worker;

    worker.onmessage = (e) => {
      if (e.data.type === 'ready') {
        setIsLoading(false);
        setIsActive(true);
        detect();
      }
      if (e.data.type === 'result') {
        setCurrentGesture(e.data.gesture);
        const now = Date.now();
        if (
          e.data.gesture !== 'None' &&
          e.data.score >= 0.7 &&
          now - lastGestureRef.current > 700
        ) {
          lastGestureRef.current = now;
          onGestureRef.current(e.data.gesture);
        }
      }
    };

    const stream = await navigator.mediaDevices.getUserMedia({
      video: { width: 320, height: 240 },
    });
    videoRef.current.srcObject = stream;
    await videoRef.current.play();

    worker.postMessage({ type: 'init' });
  }, []);

  const detect = useCallback(() => {
    const video = videoRef.current;
    if (!video || video.readyState < 2) {
      animRef.current = requestAnimationFrame(detect);
      return;
    }
    createImageBitmap(video).then((frame) => {
      workerRef.current.postMessage(
        { type: 'detect', frame, timestamp: performance.now() },
        [frame]
      );
    });
    animRef.current = requestAnimationFrame(detect);
  }, []);

  const stop = useCallback(() => {
    if (animRef.current) cancelAnimationFrame(animRef.current);
    if (videoRef.current?.srcObject) {
      videoRef.current.srcObject.getTracks().forEach((t) => t.stop());
    }
    if (workerRef.current) {
      workerRef.current.terminate();
      workerRef.current = null;
    }
    setIsActive(false);
    setCurrentGesture('None');
  }, []);

  useEffect(() => () => stop(), [stop]);

  return { videoRef, isActive, isLoading, currentGesture, start, stop };
}

What does this code do?

  • The start() function spawns the worker using Vite's new URL(..., import.meta.url) pattern, opens the webcam at 320x240, and sends the 'init' message to load MediaPipe.
  • The detect() loop runs via requestAnimationFrame. Each frame is captured as an ImageBitmap and transferred (not copied) to the worker using the transferable objects array in postMessage. This avoids expensive serialization.
  • The 700ms debounce prevents a held gesture from firing dozens of times per second. Only gestures with a confidence score of 0.7 or higher trigger an action.
  • The stop() function cancels the animation loop, stops all webcam tracks, and terminates the worker to free resources.
  • Save src/hooks/useGestureWorker.js.

You will not see a change yet. The next chunk creates the UI component that uses this hook.

Seeing a syntax error about import.meta.url?

Make sure you are using the exact new URL('../workers/gesture.worker.js', import.meta.url) pattern. Vite uses this to detect worker files at build time. A dynamic string or template literal will break bundling.

help me fix an import.meta.url error in my Vite Web Worker setup

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import { useRef, useState, useCallback, useEffect } from 'react';

export function useGestureWorker(onGesture) {
  const workerRef = useRef(null);
  const videoRef = useRef(null);
  const animRef = useRef(null);
  const lastGestureRef = useRef(0);
  const [isActive, setIsActive] = useState(false);
  const [isLoading, setIsLoading] = useState(false);
  const [currentGesture, setCurrentGesture] = useState('None');
  const onGestureRef = useRef(onGesture);
  onGestureRef.current = onGesture;

  const start = useCallback(async () => {
    setIsLoading(true);

    const worker = new Worker(
      new URL('../workers/gesture.worker.js', import.meta.url),
      { type: 'module' }
    );
    workerRef.current = worker;

    worker.onmessage = (e) => {
      if (e.data.type === 'ready') {
        setIsLoading(false);
        setIsActive(true);
        detect();
      }
      if (e.data.type === 'result') {
        setCurrentGesture(e.data.gesture);
        const now = Date.now();
        if (
          e.data.gesture !== 'None' &&
          e.data.score >= 0.7 &&
          now - lastGestureRef.current > 700
        ) {
          lastGestureRef.current = now;
          onGestureRef.current(e.data.gesture);
        }
      }
    };

    const stream = await navigator.mediaDevices.getUserMedia({
      video: { width: 320, height: 240 },
    });
    videoRef.current.srcObject = stream;
    await videoRef.current.play();

    worker.postMessage({ type: 'init' });
  }, []);

  const detect = useCallback(() => {
    const video = videoRef.current;
    if (!video || video.readyState < 2) {
      animRef.current = requestAnimationFrame(detect);
      return;
    }
    createImageBitmap(video).then((frame) => {
      workerRef.current.postMessage(
        { type: 'detect', frame, timestamp: performance.now() },
        [frame]
      );
    });
    animRef.current = requestAnimationFrame(detect);
  }, []);

  const stop = useCallback(() => {
    if (animRef.current) cancelAnimationFrame(animRef.current);
    if (videoRef.current?.srcObject) {
      videoRef.current.srcObject.getTracks().forEach((t) => t.stop());
    }
    if (workerRef.current) {
      workerRef.current.terminate();
      workerRef.current = null;
    }
    setIsActive(false);
    setCurrentGesture('None');
  }, []);

  useEffect(() => () => stop(), [stop]);

  return { videoRef, isActive, isLoading, currentGesture, start, stop };
}
Build the GestureOverlay component and connect to audio

Now you need a UI component that shows the webcam feed, displays detected gestures, and provides a toggle button. Then you will wire it into App.jsx so gestures trigger real audio actions.

  • Create the overlay component by adding the following code to src/components/GestureOverlay.jsx:
import { useGestureWorker } from '../hooks/useGestureWorker';

const GESTURE_ACTIONS = {
  Open_Palm: 'Play/Pause',
  Closed_Fist: 'Mute',
  Thumb_Up: 'Vol +',
  Thumb_Down: 'Vol -',
  Victory: 'Next Preset',
};

export function GestureOverlay({ onGesture }) {
  const { videoRef, isActive, isLoading, currentGesture, start, stop } =
    useGestureWorker(onGesture);

  return (
    <div className="gesture-section">
      <button
        className={`gesture-toggle ${isActive ? 'active' : ''}`}
        onClick={isActive ? stop : start}
        disabled={isLoading}
      >
        {isLoading ? 'Loading ML...' : isActive ? 'Disable Gestures' : 'Enable Gestures'}
      </button>

      <div className={`gesture-overlay ${isActive ? 'visible' : ''}`}>
        <video ref={videoRef} className="gesture-video" muted playsInline />
        <div className="gesture-label">
          {currentGesture !== 'None' && (
            <>
              <strong>{currentGesture.replace('_', ' ')}</strong>
              <span>{GESTURE_ACTIONS[currentGesture]}</span>
            </>
          )}
        </div>
      </div>

      {isActive && (
        <div className="gesture-legend">
          {Object.entries(GESTURE_ACTIONS).map(([g, a]) => (
            <span key={g} className="legend-item">{g.replace('_', ' ')}: {a}</span>
          ))}
        </div>
      )}
    </div>
  );
}

What does this code do?

  • The GESTURE_ACTIONS map defines the five supported gestures and their corresponding audio actions.
  • The button toggles between start and stop, showing a loading state while MediaPipe initializes.
  • The webcam video element uses transform: scaleX(-1) in CSS so it mirrors the learner like a selfie camera.
  • A gesture legend appears when active, showing all available hand signals and their actions.
  • Save src/components/GestureOverlay.jsx.

You will not see the overlay in your app yet. You need to import and render it in App.jsx next.

Seeing a missing module error for useGestureWorker?

Check that the import path is '../hooks/useGestureWorker' with the correct relative directory. The file must exist at src/hooks/useGestureWorker.js from the previous chunk.

help me fix a missing module error for my useGestureWorker hook import

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import { useGestureWorker } from '../hooks/useGestureWorker';

const GESTURE_ACTIONS = {
  Open_Palm: 'Play/Pause',
  Closed_Fist: 'Mute',
  Thumb_Up: 'Vol +',
  Thumb_Down: 'Vol -',
  Victory: 'Next Preset',
};

export function GestureOverlay({ onGesture }) {
  const { videoRef, isActive, isLoading, currentGesture, start, stop } =
    useGestureWorker(onGesture);

  return (
    <div className="gesture-section">
      <button
        className={`gesture-toggle ${isActive ? 'active' : ''}`}
        onClick={isActive ? stop : start}
        disabled={isLoading}
      >
        {isLoading ? 'Loading ML...' : isActive ? 'Disable Gestures' : 'Enable Gestures'}
      </button>

      <div className={`gesture-overlay ${isActive ? 'visible' : ''}`}>
        <video ref={videoRef} className="gesture-video" muted playsInline />
        <div className="gesture-label">
          {currentGesture !== 'None' && (
            <>
              <strong>{currentGesture.replace('_', ' ')}</strong>
              <span>{GESTURE_ACTIONS[currentGesture]}</span>
            </>
          )}
        </div>
      </div>

      {isActive && (
        <div className="gesture-legend">
          {Object.entries(GESTURE_ACTIONS).map(([g, a]) => (
            <span key={g} className="legend-item">{g.replace('_', ' ')}: {a}</span>
          ))}
        </div>
      )}
    </div>
  );
}

Now update App.jsx to import the overlay, define a gesture handler, and render the component.

  • In src/App.jsx, find the Visualizer import line:
import { Visualizer } from './components/Visualizer';
  • Add this import directly below it:
import { GestureOverlay } from './components/GestureOverlay';
  • Inside the App function, find the closing brace of the useAudioEngine() destructuring. Add the preset index ref and gesture handler below it:
  const presetIndexRef = useRef(0);

  const handleGesture = useCallback((gesture) => {
    switch (gesture) {
      case 'Open_Palm': togglePlay(); break;
      case 'Closed_Fist': toggleMute(); break;
      case 'Thumb_Up': setMasterVolume(Math.min(1, masterVolume + 0.1)); break;
      case 'Thumb_Down': setMasterVolume(Math.max(0, masterVolume - 0.1)); break;
      case 'Victory':
        presetIndexRef.current = (presetIndexRef.current + 1) % presets.length;
        applyPreset(presets[presetIndexRef.current]);
        break;
    }
  }, [togglePlay, toggleMute, setMasterVolume, masterVolume, applyPreset]);
  • In the JSX return, find the closing </main> tag. Add the GestureOverlay component directly below it (before the closing </div>):
      <GestureOverlay onGesture={handleGesture} />

What does handleGesture do?

  • It maps each recognized gesture to an audio engine action: Open Palm toggles play, Closed Fist mutes, Thumb Up and Thumb Down adjust master volume, and Victory cycles through presets.
  • The presetIndexRef tracks which preset is next in the cycle so Victory always advances forward.
  • Save src/App.jsx and return to your browser.
  • Click Play to start audio, then click Enable Gestures.

Your browser will ask for camera permission. After allowing it, you should see a small mirrored webcam feed appear with a gesture legend below it. Hold up an open palm and the audio should toggle.

What should I see?

After clicking Enable Gestures, the button text changes to Loading ML... for a few seconds while MediaPipe loads in the worker. Once ready, the webcam feed appears and the button switches to Disable Gestures.

When you hold a gesture, the label below the video shows the gesture name and its mapped action. The visualizer and UI remain smooth because inference runs in the worker thread.

Gestures not being detected?

Make sure your hand is well-lit and fully visible in the webcam frame. MediaPipe needs a clear view of your fingers.

Check the browser console for errors. If you see a CORS or network error loading the WASM files, confirm you have an active internet connection. The MediaPipe model loads from Google's CDN on first use.

If the button stays on Loading ML... indefinitely, open DevTools and check for worker errors in the Console. Filter by the worker script name.

help me debug gesture recognition not working in my Web Worker MediaPipe setup

✔️ Awesome, I've got everything!

Great. Double check you have saved App.jsx before moving on.

ⓧ I'd like to double check the full code

import { useCallback, useRef } from 'react';
import { useAudioEngine } from './hooks/useAudioEngine';
import { sounds } from './data/sounds';
import { presets } from './data/presets';
import { SoundLayer } from './components/SoundLayer';
import { MoodPresets } from './components/MoodPresets';
import { Visualizer } from './components/Visualizer';
import { GestureOverlay } from './components/GestureOverlay';
import './App.css';

function App() {
  const {
    isPlaying, isMuted, volumes, masterVolume, activePreset, analyser,
    togglePlay, setLayerVolume, setMasterVolume, applyPreset, toggleMute,
  } = useAudioEngine();

  const presetIndexRef = useRef(0);

  const handleGesture = useCallback((gesture) => {
    switch (gesture) {
      case 'Open_Palm': togglePlay(); break;
      case 'Closed_Fist': toggleMute(); break;
      case 'Thumb_Up': setMasterVolume(Math.min(1, masterVolume + 0.1)); break;
      case 'Thumb_Down': setMasterVolume(Math.max(0, masterVolume - 0.1)); break;
      case 'Victory':
        presetIndexRef.current = (presetIndexRef.current + 1) % presets.length;
        applyPreset(presets[presetIndexRef.current]);
        break;
    }
  }, [togglePlay, toggleMute, setMasterVolume, masterVolume, applyPreset]);

  return (
    <div className={`app preset-${activePreset || 'focus'}`}>
      <Visualizer analyser={analyser} activePreset={activePreset} />
      <header className="header">
        <h1>Ambience</h1>
        <div className="controls">
          <button className={`play-btn ${isPlaying ? 'playing' : ''}`} onClick={togglePlay}>
            {isPlaying ? 'Pause' : 'Play'}
          </button>
          <button className={`mute-btn ${isMuted ? 'muted' : ''}`} onClick={toggleMute}>
            {isMuted ? 'Unmute' : 'Mute'}
          </button>
          <input type="range" min="0" max="1" step="0.01" value={masterVolume}
            onChange={(e) => setMasterVolume(parseFloat(e.target.value))} className="master-slider" />
        </div>
      </header>
      <MoodPresets activePreset={activePreset} onSelectPreset={applyPreset} />
      <main className="layers-grid">
        {sounds.map((s) => (
          <SoundLayer key={s.id} sound={s} volume={volumes[s.id]} onVolumeChange={setLayerVolume} />
        ))}
      </main>
      <GestureOverlay onGesture={handleGesture} />
    </div>
  );
}

export default App;

Deploy to Vercel

Your ambient moodboard is fully functional locally. Audio layers mix, presets crossfade, the visualizer animates, and gesture recognition runs hands-free.

Now it is time to ship it. In this step, you will push your code to GitHub and deploy it to Vercel as a static site so anyone can visit your moodboard from a live URL.

In this step, get ready to:
  • Initialize a Git repository and push to GitHub.
  • Import the project into Vercel and deploy.
  • Verify the live production URL.
Initialize Git and push to GitHub

Vercel deploys from a Git repository. You need to version your project with Git and push it to a GitHub repo so Vercel can pull the source and build it.

  • Stop the dev server in your terminal by pressing Ctrl+C.
  • Initialize a Git repository and create your first commit by running these commands:
git init
git add .
git commit -m "Initial commit: ambient moodboard with gesture control"

What do these commands do?

  • git init creates a new Git repository in your project folder.
  • git add . stages all project files for the first commit.
  • git commit saves a snapshot of your project at this point in time.
  • Open GitHub in your browser and create a new repository.
  • Name it ambience-moodboard.
  • Leave it set to Public and do not initialize with a README.
  • Click Create repository.
  • Copy the remote URL that GitHub gives you (it looks like https://github.com/your-username/ambience-moodboard.git).
  • Back in your terminal, connect to GitHub and push by running these commands:
git remote add origin [[REPO_URL="https://github.com/your-username/ambience-moodboard.git"]]
git branch -M main
git push -u origin main
  • Refresh your GitHub repository page. You should see all your project files listed.

Push failed or authentication error?

  • Make sure you copied the correct repository URL from GitHub.
  • If GitHub asks for authentication, follow the prompts to sign in or set up a personal access token.
  • help me fix a Git push authentication error
Import the project into Vercel and deploy

Vercel connects directly to your GitHub repository. Every time you push new code, Vercel rebuilds and redeploys automatically.

  • Open vercel.com in your browser and sign in to your Vercel account.
  • Click Add New in the top right, then select Project.
  • If you have not connected GitHub yet, Vercel will prompt you to authorize access. Follow the prompts to connect your GitHub account.
  • Find ambience-moodboard in the repository list and click Import.

What does Vercel auto-detect?

Vercel recognizes that your project uses Vite and automatically sets the build command to npm run build and the output directory to dist. You do not need to change any build settings.

  • On the configuration page, leave all settings at their defaults. Vercel has already detected the framework and configured the build.
  • Click Deploy.
  • Wait for the build to complete. Vercel will show a build log as it installs dependencies and runs npm run build.

Build failed?

  • Check the build log for specific error messages. The most common issue is a missing dependency that was not committed to package.json.
  • Make sure node_modules is listed in your .gitignore file. Vercel installs dependencies fresh from package.json during the build.
  • help me debug my Vercel build failure for my Vite React project
Verify the live deployment

Once the build succeeds, Vercel assigns your project a production URL ending in .vercel.app.

  • Click the production URL that Vercel shows on the deployment success screen.
  • Click Play and confirm you hear audio playing.
  • Click a preset button (Focus, Sleep, Storm, or Forest) and verify the audio crossfades and the background gradient shifts.
  • Confirm the canvas visualizer at the bottom animates in response to the audio.
  • Click Enable Gestures and allow camera access when prompted. Verify that your webcam feed appears and gestures are recognized.

Why does gesture recognition need HTTPS?

The getUserMedia API (webcam access) only works on secure origins. Vercel serves your site over HTTPS by default, so gesture recognition works out of the box on the deployed URL.

Your ambient moodboard is live. Next up, the Secret Mission challenges you to add offline support with a Service Worker.

Secret mission

Service Worker for Offline PWA

Your moodboard works beautifully online, but what happens when you lose connection mid-focus session? Add a Service Worker that caches the entire app shell and all audio files after the first visit. The result: a fully offline-capable PWA that keeps playing ambient sounds even without internet.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally and deploys as a free static site on Vercel. There are no ongoing costs or paid resources to worry about.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The ambience-moodboard project folder on your Desktop.
  • A Vercel deployment (static site on the free Hobby plan).
  • MediaPipe model files cached by the Service Worker (loaded from Google's CDN on first use).

Keep everything running

No action needed. Choose this if you are still actively building or want the app to remain accessible at your Vercel URL.

  • Your Vercel deployment is free on the Hobby plan. It costs nothing to leave it running.
  • The local project folder stays on your Desktop. You can reopen it any time.
  • The Service Worker cache persists in your browser until you manually clear site data.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and deployment stay intact.

  • Stop the Vite dev server by pressing Ctrl+C in the terminal where it is running.
  • Close the VS Code window for the project.
  • Your Vercel deployment remains live (free, no cost). You can return and run npm run dev whenever you want to resume local development.

Delete - I don't want to use this again

Remove all project resources and start fresh.

  • Delete the local project folder by running this command in your terminal (macOS/Linux):
rm -rf ~/Desktop/ambience-moodboard
  • On Windows, run this instead:
Remove-Item -Recurse -Force ~\Desktop\ambience-moodboard
  • Delete the Vercel project by opening your Vercel dashboard at vercel.com.
  • Select your ambience-moodboard project.
  • Click Settings in the sidebar.
  • Scroll to the bottom of the General page and click Delete in the Delete Project section.
  • In the confirmation dialog, enter your project name and type delete my project in the second field, then click Continue.
  • Clear the Service Worker cache in your browser by opening DevTools, navigating to Application > Storage, and clicking Clear site data.

That's a Wrap!

That's a Wrap!

Nice work. You just built a fully deployed ambient moodboard that layers audio, visualizes frequency data in real time, and responds to hand gestures without dropping a single frame.

You've learned how to:

  • Build a multi-layer audio engine using the Web Audio API with AudioContext, GainNodes, and AnalyserNode, including smooth crossfading presets powered by linearRampToValueAtTime.
  • Create a Canvas 2D audio visualizer driven by requestAnimationFrame and real-time frequency data from the audio mix.
  • Offload MediaPipe gesture recognition to a Web Worker using postMessage and ImageBitmap transfer, keeping the main thread at 60fps while performing ML inference.
  • Secret Mission: Add a Service Worker with a cache-first strategy for full offline PWA mode.

Ready to quiz yourself?