Build a Voice AI Assistant

Build a Jarvis-inspired voice assistant with speech recognition and an LLM.

Introduction

30 Second Summary

Voice assistants like Siri and Alexa feel like magic. You talk, they listen, and they respond out loud.

In this project, you will build your own voice AI assistant inspired by Jarvis from Iron Man. You will wire together speech recognition, an LLM brain, and text-to-speech to create a fully conversational assistant that runs in your browser.

What You'll Build

A browser-based voice assistant with a dark sci-fi themed UI that listens to your speech, thinks using Llama 3.3 70B on Groq (free tier, no credit card required), and responds with a synthesized voice.

By the end of this project, you'll have:

  • A Node.js Express server that securely proxies requests to the Groq API with a custom Jarvis personality.
  • A dark sci-fi themed UI with an animated orb that reacts to the assistant's state (idle, listening, thinking, speaking).
  • A complete voice interaction loop using the Web Speech API for speech input, AI processing, and spoken responses.
  • Secret Mission: Add always-on wake word detection so the assistant activates when you say "Hey Jarvis" without pressing any button.

Are there any prerequisites?

You will need Node.js installed on your machine, a Google account to sign up for a free Groq API key, and the Chrome browser (the voice features rely on Chrome's speech recognition engine).

Set Up the Project

Your voice assistant needs a few things before it can work: a server to handle requests, dependencies to connect to an AI model, and a free API key to power the brain.

In this step, you'll create the project folder, install the packages it needs, and grab a free API key from Groq to access Llama 3.3 70B.

In this step, get ready to:
  • Create and initialize the jarvis project folder.
  • Get a free API key from Groq for AI responses.
  • Create the project's file structure.
Create and initialize the project folder

Every Node.js project starts with a folder and a package.json file that tracks your project's dependencies.

  • In VS Code, open the integrated terminal by pressing Ctrl+` (macOS) or Ctrl+` (Windows).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir jarvis
cd jarvis
  • Initialize the project with npm by running:
npm init -y

What does this command do?

npm init -y creates a package.json file with default settings.

This file tracks your project's name, version, and most importantly, the packages it depends on.

  • Install the three packages your project needs by running:
npm install express groq-sdk dotenv

What are these packages?

  • express is a web server framework that handles HTTP requests.
  • groq-sdk is the official SDK for communicating with Groq's AI models.
  • dotenv loads environment variables from a .env file so your API keys stay out of your code.

You should see output listing the installed packages and a new node_modules folder has been created.

Seeing an error during install?

  • Make sure you're inside the jarvis folder. Run pwd to check your current directory.
  • If you see a permissions error, try running the install command with sudo on macOS/Linux.

Help me fix my npm install error

  • Now open the jarvis folder in VS Code. Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the jarvis folder, then click Open.

You should see package.json and a node_modules folder in the VS Code Explorer sidebar.

Get a free Groq API key

Groq provides free access to Llama 3.3 70B, one of the most capable open-source language models available. You need an API key to authenticate your requests.

  • Open Chrome and go to console.groq.com.
  • Click Sign up and sign in with your Google account.

Do I need a credit card?

No. Groq's free tier gives you 30 requests per minute with no credit card required. That's more than enough for a personal voice assistant.

  • Once logged in, click API Keys in the left sidebar.
  • Click Create API Key.
  • Give your key a name like jarvis and click Submit.
  • Copy the API key that appears. You will only see this key once, so keep it handy.

Copy your key before closing the dialog

Groq only shows your API key once at the moment of creation. If you close the dialog without copying it, you will need to create a new one.

  • Return to VS Code. In the Explorer sidebar, right-click in the empty area and select New File.
  • Name the file .env and press Enter.
  • Paste the following into your .env file, replacing your-api-key-here with the key you just copied:
GROQ_API_KEY=your-api-key-here

Why use a .env file?

A .env file keeps sensitive values like API keys separate from your code. The dotenv package reads this file and makes the key available to your server without hardcoding it.

  • Save the file.

Having trouble creating the .env file?

  • On some systems, files starting with a dot are hidden by default. If you don't see the file in the Explorer, click the ... menu in the Explorer panel and make sure hidden files are visible.
  • Make sure the file is named exactly .env with no extra extension like .env.txt.

Help me create my .env file

Create the project file structure

Your project needs four files: a server file for the backend, and three files inside a public folder for the frontend that your browser will load.

  • Open the VS Code terminal by pressing Ctrl+` (both macOS and Windows).
  • Create the public folder and all empty files by running these commands:
mkdir public
touch server.js public/index.html public/style.css public/app.js

What is the public folder for?

Express serves files from the public folder directly to the browser. Your HTML, CSS, and frontend JavaScript all live here so the browser can access them.

  • Confirm the files exist by running:
ls
ls public

You should see server.js, public, node_modules, package.json, and .env in the first listing. The second listing should show index.html, style.css, and app.js.

✔️ I see all the files listed

Your project structure is ready. Everything is in place to start building.

ⓧ I'm missing some files

If touch didn't work (common on Windows), create the files manually in VS Code instead.

  • In the Explorer sidebar, right-click and select New File.
  • Create server.js in the project root.
  • Right-click again and select New Folder. Name it public.
  • Right-click the public folder and create index.html, style.css, and app.js inside it.

Your project is fully set up with dependencies installed, your API key stored, and all the files ready to fill in. Next up, you'll build the Express backend that connects to Groq and gives your assistant its brain.

Build the Express Backend

Your voice assistant needs a brain. The backend server receives your spoken words as text, sends them to an LLM for processing, and returns the AI's response to the browser.

In this step, you'll build an Express server that connects to Groq's API. A system prompt gives the AI a Jarvis-like personality so it responds in character.

In this step, get ready to:
  • Set up an Express server that serves your frontend files.
  • Connect the server to Groq's API with a custom Jarvis personality.
  • Test the server to confirm it starts correctly.
Set up the Express server foundation

Before the AI can respond to anything, you need a server that can receive requests from the browser and serve your frontend files.

  • Open server.js in the VS Code Explorer sidebar.
  • Add the server foundation by copying this code into the file:
const express = require("express");
const Groq = require("groq-sdk");
const dotenv = require("dotenv");

// Load environment variables from .env file
dotenv.config();

// Create the Express app and set the port
const app = express();
const port = 3000;

// Parse incoming JSON requests and serve frontend files
app.use(express.json());
app.use(express.static("public"));

What does this code do?

  • dotenv.config() loads your GROQ_API_KEY from the .env file into the environment so your code can access it securely.
  • express.json() is middleware that parses incoming JSON request bodies. Without it, the server can't read messages sent from the browser.
  • express.static("public") tells Express to serve any file inside the public folder (your HTML, CSS, and JavaScript) directly to the browser.

You won't see a change yet. The next chunk adds the AI endpoint that brings this server to life.

Integrate the Groq SDK and create the chat endpoint

The Groq SDK connects your server to Llama 3.3 70B, a powerful open-source LLM that runs at ~280 tokens per second on Groq's infrastructure. A system prompt tells the model to behave like Jarvis.

  • Add the following code directly below the last line in server.js:
// Initialize the Groq client with your API key
const groq = new Groq({ apiKey: process.env.GROQ_API_KEY });

// Define the Jarvis personality
const SYSTEM_PROMPT = `You are Jarvis, a highly intelligent AI assistant inspired by the AI from Iron Man. You are concise, slightly witty, and always helpful. You address the user as "sir" unless told otherwise. Keep responses under 3 sentences unless the user asks for detail. You speak naturally as if having a conversation, not like a search engine. Never use markdown formatting in your responses since they will be spoken aloud.`;

// Handle chat requests from the frontend
app.post("/api/chat", async (req, res) => {
  try {
    const { messages } = req.body;

    // Send messages to Groq with the Jarvis system prompt prepended
    const completion = await groq.chat.completions.create({
      messages: [{ role: "system", content: SYSTEM_PROMPT }, ...messages],
      model: "llama-3.3-70b-versatile",
      temperature: 0.7,
      max_tokens: 256,
    });

    // Extract and return the AI's reply
    const reply = completion.choices[0].message.content;
    res.json({ reply });
  } catch (error) {
    console.error("Groq API error:", error.message);
    res.status(500).json({ error: "Failed to get response from AI" });
  }
});

// Start the server
app.listen(port, () => {
  console.log(`Jarvis server running at http://localhost:${port}`);
});

What does this code do?

  • SYSTEM_PROMPT defines the AI's personality. It tells the model to be concise, witty, address the user as "sir", and avoid markdown (since responses will be spoken aloud).
  • The /api/chat endpoint receives a messages array from the browser, prepends the system prompt, and forwards everything to Groq.
  • temperature: 0.7 makes responses creative but coherent. max_tokens: 256 keeps replies short so they sound natural when spoken.
  • The try/catch block handles API errors gracefully, logging the issue and sending a clean error response to the frontend.

Why prepend a system prompt?

Without a system prompt, the LLM responds as a generic assistant. The system message acts as persistent instructions that shape every response. It's the difference between talking to a blank chatbot and talking to Jarvis.

  • Save server.js.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before testing in the next substep.

ⓧ I'd like to double check the full code

const express = require("express");
const Groq = require("groq-sdk");
const dotenv = require("dotenv");

// Load environment variables from .env file
dotenv.config();

// Create the Express app and set the port
const app = express();
const port = 3000;

// Parse incoming JSON requests and serve frontend files
app.use(express.json());
app.use(express.static("public"));

// Initialize the Groq client with your API key
const groq = new Groq({ apiKey: process.env.GROQ_API_KEY });

// Define the Jarvis personality
const SYSTEM_PROMPT = `You are Jarvis, a highly intelligent AI assistant inspired by the AI from Iron Man. You are concise, slightly witty, and always helpful. You address the user as "sir" unless told otherwise. Keep responses under 3 sentences unless the user asks for detail. You speak naturally as if having a conversation, not like a search engine. Never use markdown formatting in your responses since they will be spoken aloud.`;

// Handle chat requests from the frontend
app.post("/api/chat", async (req, res) => {
  try {
    const { messages } = req.body;

    // Send messages to Groq with the Jarvis system prompt prepended
    const completion = await groq.chat.completions.create({
      messages: [{ role: "system", content: SYSTEM_PROMPT }, ...messages],
      model: "llama-3.3-70b-versatile",
      temperature: 0.7,
      max_tokens: 256,
    });

    // Extract and return the AI's reply
    const reply = completion.choices[0].message.content;
    res.json({ reply });
  } catch (error) {
    console.error("Groq API error:", error.message);
    res.status(500).json({ error: "Failed to get response from AI" });
  }
});

// Start the server
app.listen(port, () => {
  console.log(`Jarvis server running at http://localhost:${port}`);
});
Test the server
  • Start the server by running this command in your terminal:
node server.js

You should see this message in your terminal:

Jarvis server running at http://localhost:3000

What should I see?

The message confirms Express is running and listening for requests on port 3000. Your browser frontend will send messages to this server in the next steps.

Don't see the message?

  • If you see a syntax error, double check that all parentheses and backticks in server.js match the code above. Template literals use backticks (`), not regular quotes.
  • If you see a "Cannot find module" error, make sure you ran npm install in the jarvis folder in Step 1. The node_modules folder should exist.
  • If the terminal says the port is already in use, another process is using port 3000. Close other terminal windows or restart your computer.

Still stuck? Help me debug why my Express server won't start.

Your backend is alive and ready to receive messages. Next up, you'll build the dark sci-fi UI that your voice assistant lives inside.

Create the Jarvis UI

Your backend is up and running, ready to receive messages and return AI-generated responses. Now you need a way for users to actually interact with it.

In this step, you'll build the visual interface for your assistant. Think of it like the HUD display Tony Stark sees when talking to Jarvis. A dark, sci-fi themed page with a glowing animated orb that reacts to what the assistant is doing.

In this step, get ready to:
  • Build the HTML page structure with an orb, status text, transcript area, and talk button.
  • Style the page with a dark sci-fi theme and animated orb states.
Build the HTML structure

The HTML page is the skeleton of your assistant's interface. It defines all the elements that will be styled and wired up with JavaScript later.

  • In VS Code, open public/index.html from the Explorer sidebar.
  • Paste the following code into the file:
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>J.A.R.V.I.S.</title>
  <link rel="stylesheet" href="style.css" />
</head>
<body>
  <div class="container">
    <div class="orb-wrapper">
      <div class="orb" id="orb"></div>
    </div>
    <p class="status" id="status">Press spacebar or click to speak</p>
    <div class="transcript" id="transcript"></div>
    <button class="talk-btn" id="talkBtn">Hold to Speak</button>
  </div>
  <script src="app.js"></script>
</body>
</html>

What does this code do?

This page defines five key pieces of the interface:

  • The orb div is the central visual element. CSS will turn this into a glowing animated circle.
  • The status paragraph tells the user what state the assistant is in (idle, listening, thinking, speaking).
  • The transcript div holds the conversation history so you can see what was said.
  • The talkBtn button triggers voice recognition when held down.
  • The <script src="app.js"> tag loads the frontend logic you'll write in the next step.
  • Save index.html.
  • Open Chrome and navigate to http://localhost:3000. You should see a white page with unstyled text saying "Press spacebar or click to speak" and a "Hold to Speak" button.

Seeing a blank page with no text?

  • Make sure your server is still running in the terminal. If it stopped, run node server.js again from the jarvis folder.
  • Confirm your file is saved as public/index.html (inside the public folder, not the project root).

Why is my page blank even though the server is running?

Style the dark sci-fi layout

Now you'll transform that plain white page into a dark, futuristic interface. The CSS is split into logical sections so you can see how each piece contributes to the final look.

  • Open public/style.css in VS Code.
  • Add the base layout styles by pasting the following code:
* {
  margin: 0;
  padding: 0;
  box-sizing: border-box;
}

body {
  background: #0a0a0f;
  color: #e0e0e0;
  font-family: "Segoe UI", sans-serif;
  height: 100vh;
  overflow: hidden;
}

.container {
  display: flex;
  flex-direction: column;
  align-items: center;
  justify-content: center;
  height: 100vh;
  padding: 2rem;
}

What does this code do?

  • The * selector resets all default margins and padding so the design starts from a clean slate.
  • The body background #0a0a0f is an almost-black color that gives the page a deep space feel.
  • The container uses flexbox to center everything vertically and horizontally on the full viewport.
  • Save style.css and refresh your browser.
  • You should see the page turn dark with light gray text centered in the middle.

Page still white?

  • Check that your index.html has <link rel="stylesheet" href="style.css" /> in the head. The filename must match exactly.
  • Do a hard refresh with Cmd+Shift+R (macOS) or Ctrl+Shift+R (Windows) to clear cached styles.

Help me figure out why my CSS isn't loading.

  • Add the orb styles below the existing code in style.css:
.orb-wrapper {
  position: relative;
  margin-bottom: 2rem;
}

.orb {
  width: 120px;
  height: 120px;
  border-radius: 50%;
  background: radial-gradient(circle, #4fc3f7, #0288d1);
  box-shadow: 0 0 40px rgba(79, 195, 247, 0.4), 0 0 80px rgba(79, 195, 247, 0.2);
  animation: pulse 3s ease-in-out infinite;
  transition: all 0.3s ease;
}

.orb.listening {
  background: radial-gradient(circle, #66bb6a, #2e7d32);
  box-shadow: 0 0 60px rgba(102, 187, 106, 0.5), 0 0 120px rgba(102, 187, 106, 0.3);
  animation: expand 1s ease-in-out infinite;
}

.orb.thinking {
  background: radial-gradient(circle, #ffa726, #e65100);
  box-shadow: 0 0 60px rgba(255, 167, 38, 0.5), 0 0 120px rgba(255, 167, 38, 0.3);
  animation: spin 1s linear infinite;
}

.orb.speaking {
  background: radial-gradient(circle, #ab47bc, #6a1b9a);
  box-shadow: 0 0 60px rgba(171, 71, 188, 0.5), 0 0 120px rgba(171, 71, 188, 0.3);
  animation: ripple 0.5s ease-in-out infinite;
}

What does this code do?

The base .orb class creates a 120px glowing blue circle using a radial gradient and a soft box-shadow. Each state class (.listening, .thinking, .speaking) changes the color and animation so you can visually tell what the assistant is doing.

  • Blue (idle): gently pulsing, waiting for input.
  • Green (listening): expanding, capturing your voice.
  • Orange (thinking): spinning, waiting for the AI response.
  • Purple (speaking): rippling, reading the response aloud.
  • Save style.css and refresh your browser.
  • You should see a glowing blue circle near the center of the page. It won't animate yet because the keyframes aren't defined.
  • Add the keyframe animations below the orb styles:
@keyframes pulse {
  0%, 100% { transform: scale(1); opacity: 0.9; }
  50% { transform: scale(1.05); opacity: 1; }
}

@keyframes expand {
  0%, 100% { transform: scale(1); }
  50% { transform: scale(1.15); }
}

@keyframes spin {
  from { transform: rotate(0deg) scale(1.05); }
  to { transform: rotate(360deg) scale(1.05); }
}

@keyframes ripple {
  0%, 100% { transform: scale(1); }
  50% { transform: scale(1.1); }
}

What does this code do?

These four @keyframes rules define the animations referenced by the orb states. The pulse animation gently scales the orb up and down for the idle state. The spin animation rotates it 360 degrees continuously for the thinking state.

  • Save style.css and refresh your browser.
  • The blue orb should now be gently pulsing (slowly growing and shrinking).
  • Add the status text, transcript area, and button styles below the keyframes:
.status {
  font-size: 0.9rem;
  color: #90a4ae;
  margin-bottom: 1.5rem;
  text-transform: uppercase;
  letter-spacing: 2px;
}

.transcript {
  width: 100%;
  max-width: 600px;
  max-height: 300px;
  overflow-y: auto;
  margin-bottom: 1.5rem;
  padding: 1rem;
  border: 1px solid #1e1e2e;
  border-radius: 8px;
  background: rgba(10, 10, 15, 0.8);
}

.transcript .message {
  margin-bottom: 0.75rem;
  padding: 0.5rem 0.75rem;
  border-radius: 6px;
  font-size: 0.9rem;
  line-height: 1.4;
}

.transcript .user {
  background: rgba(79, 195, 247, 0.1);
  border-left: 3px solid #4fc3f7;
}

.transcript .assistant {
  background: rgba(171, 71, 188, 0.1);
  border-left: 3px solid #ab47bc;
}

.talk-btn {
  padding: 0.75rem 2rem;
  border: 1px solid #4fc3f7;
  border-radius: 25px;
  background: transparent;
  color: #4fc3f7;
  font-size: 0.85rem;
  text-transform: uppercase;
  letter-spacing: 2px;
  cursor: pointer;
  transition: all 0.3s ease;
}

.talk-btn:hover {
  background: rgba(79, 195, 247, 0.1);
}

.talk-btn:active, .talk-btn.active {
  background: rgba(79, 195, 247, 0.2);
  box-shadow: 0 0 20px rgba(79, 195, 247, 0.3);
}

What does this code do?

  • The .status class styles the state indicator text with uppercase letters and wide letter-spacing for a HUD feel.
  • The .transcript area has a semi-transparent dark background with a subtle border. User messages get a blue left border, assistant messages get a purple left border.
  • The .talk-btn is a pill-shaped transparent button with a cyan border that glows when pressed.
  • Save style.css and refresh your browser.
  • You should see the full Jarvis UI: a pulsing blue orb in the center, uppercase status text below it, an empty transcript area with a dark border, and a cyan "HOLD TO SPEAK" button at the bottom.

Button or transcript not visible?

  • Make sure you didn't accidentally paste the new CSS inside a previous selector's curly braces. Each selector block should be at the top level of the file.
  • Check that the class names in your CSS match the HTML exactly. For example, the button uses class="talk-btn" in the HTML and .talk-btn in the CSS.

Help me figure out why some elements aren't showing up.

✔️ Awesome, I've got everything!

Great. Double check you've saved both index.html and style.css.

ⓧ I'd like to double check the full code

Here's the complete public/index.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>J.A.R.V.I.S.</title>
  <link rel="stylesheet" href="style.css" />
</head>
<body>
  <div class="container">
    <div class="orb-wrapper">
      <div class="orb" id="orb"></div>
    </div>
    <p class="status" id="status">Press spacebar or click to speak</p>
    <div class="transcript" id="transcript"></div>
    <button class="talk-btn" id="talkBtn">Hold to Speak</button>
  </div>
  <script src="app.js"></script>
</body>
</html>

Here's the complete public/style.css:

* {
  margin: 0;
  padding: 0;
  box-sizing: border-box;
}

body {
  background: #0a0a0f;
  color: #e0e0e0;
  font-family: "Segoe UI", sans-serif;
  height: 100vh;
  overflow: hidden;
}

.container {
  display: flex;
  flex-direction: column;
  align-items: center;
  justify-content: center;
  height: 100vh;
  padding: 2rem;
}

.orb-wrapper {
  position: relative;
  margin-bottom: 2rem;
}

.orb {
  width: 120px;
  height: 120px;
  border-radius: 50%;
  background: radial-gradient(circle, #4fc3f7, #0288d1);
  box-shadow: 0 0 40px rgba(79, 195, 247, 0.4), 0 0 80px rgba(79, 195, 247, 0.2);
  animation: pulse 3s ease-in-out infinite;
  transition: all 0.3s ease;
}

.orb.listening {
  background: radial-gradient(circle, #66bb6a, #2e7d32);
  box-shadow: 0 0 60px rgba(102, 187, 106, 0.5), 0 0 120px rgba(102, 187, 106, 0.3);
  animation: expand 1s ease-in-out infinite;
}

.orb.thinking {
  background: radial-gradient(circle, #ffa726, #e65100);
  box-shadow: 0 0 60px rgba(255, 167, 38, 0.5), 0 0 120px rgba(255, 167, 38, 0.3);
  animation: spin 1s linear infinite;
}

.orb.speaking {
  background: radial-gradient(circle, #ab47bc, #6a1b9a);
  box-shadow: 0 0 60px rgba(171, 71, 188, 0.5), 0 0 120px rgba(171, 71, 188, 0.3);
  animation: ripple 0.5s ease-in-out infinite;
}

@keyframes pulse {
  0%, 100% { transform: scale(1); opacity: 0.9; }
  50% { transform: scale(1.05); opacity: 1; }
}

@keyframes expand {
  0%, 100% { transform: scale(1); }
  50% { transform: scale(1.15); }
}

@keyframes spin {
  from { transform: rotate(0deg) scale(1.05); }
  to { transform: rotate(360deg) scale(1.05); }
}

@keyframes ripple {
  0%, 100% { transform: scale(1); }
  50% { transform: scale(1.1); }
}

.status {
  font-size: 0.9rem;
  color: #90a4ae;
  margin-bottom: 1.5rem;
  text-transform: uppercase;
  letter-spacing: 2px;
}

.transcript {
  width: 100%;
  max-width: 600px;
  max-height: 300px;
  overflow-y: auto;
  margin-bottom: 1.5rem;
  padding: 1rem;
  border: 1px solid #1e1e2e;
  border-radius: 8px;
  background: rgba(10, 10, 15, 0.8);
}

.transcript .message {
  margin-bottom: 0.75rem;
  padding: 0.5rem 0.75rem;
  border-radius: 6px;
  font-size: 0.9rem;
  line-height: 1.4;
}

.transcript .user {
  background: rgba(79, 195, 247, 0.1);
  border-left: 3px solid #4fc3f7;
}

.transcript .assistant {
  background: rgba(171, 71, 188, 0.1);
  border-left: 3px solid #ab47bc;
}

.talk-btn {
  padding: 0.75rem 2rem;
  border: 1px solid #4fc3f7;
  border-radius: 25px;
  background: transparent;
  color: #4fc3f7;
  font-size: 0.85rem;
  text-transform: uppercase;
  letter-spacing: 2px;
  cursor: pointer;
  transition: all 0.3s ease;
}

.talk-btn:hover {
  background: rgba(79, 195, 247, 0.1);
}

.talk-btn:active, .talk-btn.active {
  background: rgba(79, 195, 247, 0.2);
  box-shadow: 0 0 20px rgba(79, 195, 247, 0.3);
}

Your Jarvis UI is looking sharp. The orb pulses patiently, waiting for your voice. In the next step, you'll wire up the Web Speech API to let it actually listen to you and respond.

Wire Up Voice and AI

Your server is ready to receive messages and your UI is styled and waiting. Now you need the JavaScript that actually makes the assistant listen, think, and speak.

In this step, you will write the entire frontend logic in app.js. This single file wires together the Web Speech API for voice input, your Express backend for AI responses, and SpeechSynthesis for voice output.

In this step, get ready to:
  • Set up speech recognition to capture your voice.
  • Set up speech synthesis so the assistant can talk back.
  • Connect the frontend to your Express backend for AI-powered responses.
Set up speech recognition

Chrome exposes the SpeechRecognition API that turns your microphone audio into text. You need to grab a reference to it, set up DOM elements, and initialize your state variables.

  • Open public/app.js in VS Code (it should still be empty).
  • Add the following code to set up the SpeechRecognition constructor and your state variables:
const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition;

const orb = document.getElementById("orb");
const status = document.getElementById("status");
const transcript = document.getElementById("transcript");
const talkBtn = document.getElementById("talkBtn");

const synth = window.speechSynthesis;
let conversationHistory = [];
let recognition = null;
let isListening = false;

What does this code do?

The first line grabs the SpeechRecognition constructor, using the webkit prefix as a fallback since Chrome still requires it.

The four DOM references connect to the HTML elements you built in the previous step. The state variables track the conversation history (for multi-turn context), the current recognition instance, and whether the mic is active.

Next, you need a function that creates a fresh recognition session each time the user presses the button. This handles capturing speech, detecting when the user finishes talking, and reporting errors.

  • Add the initRecognition() function below your state variables:
function initRecognition() {
  recognition = new SpeechRecognition();
  recognition.lang = "en-US";
  recognition.interimResults = true;
  recognition.continuous = false;

  recognition.onresult = (event) => {
    const result = event.results[event.results.length - 1];
    const text = result[0].transcript;

    if (result.isFinal) {
      status.textContent = "Processing...";
      setOrbState("thinking");
      handleUserMessage(text);
    } else {
      status.textContent = text;
    }
  };

  recognition.onend = () => {
    if (isListening) {
      isListening = false;
      talkBtn.classList.remove("active");
    }
  };

  recognition.onerror = (event) => {
    console.error("Speech recognition error:", event.error);
    status.textContent = "Error: " + event.error + ". Try again.";
    setOrbState("idle");
    isListening = false;
    talkBtn.classList.remove("active");
  };
}

What does this code do?

  • A new SpeechRecognition instance is created each time with interimResults = true so the user sees their words appear in real time before they finish speaking.
  • The onresult handler checks if the result is final. If so, it sends the text to the AI. If not, it updates the status display with the interim transcript.
  • The onend handler resets the button state when recognition stops.
  • The onerror handler shows the error and resets everything so the user can try again.

Now you need the functions that control the orb animation and start or stop the microphone.

  • Add the state control functions below initRecognition():
function setOrbState(state) {
  orb.className = "orb";
  if (state !== "idle") {
    orb.classList.add(state);
  }
}

function startListening() {
  if (isListening) return;
  isListening = true;
  talkBtn.classList.add("active");
  setOrbState("listening");
  status.textContent = "Listening...";
  initRecognition();
  recognition.start();
}

function stopListening() {
  if (!isListening) return;
  recognition.stop();
}

What does this code do?

  • setOrbState() strips all state classes from the orb, then adds the new one. This triggers the CSS animations you wrote in the previous step (listening = green expand, thinking = orange spin, speaking = purple ripple).
  • startListening() guards against double-starts, activates the button style, turns the orb green, and kicks off a fresh recognition session.
  • stopListening() tells the browser to stop capturing audio. The onend handler from earlier will clean up the button state.
  • Save app.js.

You will not see a visible change yet. The recognition logic needs the backend connection and event listeners (coming next) before it can run end-to-end.

Seeing a red underline or syntax error?

  • Make sure every opening brace { has a matching closing brace }. Count them in initRecognition() carefully.
  • Check that you used const for the DOM references and let for the mutable state variables.

Still stuck? Help me debug a syntax error in my app.js speech recognition code.

Set up speech synthesis

Speech recognition turns your voice into text. Speech synthesis does the reverse. The browser's SpeechSynthesis API lets you pick a voice, set the pitch and speed, and have the browser read text out loud.

  • Add the speak() function below stopListening():
function speak(text) {
  setOrbState("speaking");
  status.textContent = "Speaking...";

  const utterance = new SpeechSynthesisUtterance(text);
  utterance.rate = 1.0;
  utterance.pitch = 0.9;

  const voices = synth.getVoices();
  const preferred = voices.find(
    (v) => v.lang.startsWith("en") && (v.name.includes("Google") || v.name.includes("Microsoft"))
  );
  if (preferred) {
    utterance.voice = preferred;
  }

  utterance.onend = () => {
    setOrbState("idle");
    status.textContent = "Press spacebar or click to speak";
  };

  synth.speak(utterance);
}

What does this code do?

  • The orb turns purple (speaking state) while the voice plays.
  • A SpeechSynthesisUtterance is created with a slightly lower pitch (0.9) for a calm, Jarvis-like tone.
  • The voice selector looks for a high-quality English voice. Chrome exposes both local system voices and Google network voices. The code prefers Google or Microsoft voices because they sound more natural.
  • When the utterance finishes, onend resets the orb back to its idle blue pulse.
  • Save app.js.

You still will not see a change in the browser. The speak function only runs when it receives a response from the backend, which you will connect next.

VS Code showing an error about synth not being defined?

Make sure your first chunk of code (the one with DOM references) includes the line const synth = window.speechSynthesis; near the top of the file.

Need a hand? Help me fix a reference error in my speak function.

Connect to the backend

When the user finishes speaking, the transcript needs to be sent to your Express server's /api/chat endpoint. The server passes it to Groq, and the response comes back for the assistant to speak.

  • Add the handleUserMessage() function below speak():
async function handleUserMessage(text) {
  addToTranscript("user", text);
  conversationHistory.push({ role: "user", content: text });

  try {
    const response = await fetch("/api/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ messages: conversationHistory }),
    });

    const data = await response.json();

    if (data.error) {
      status.textContent = "Error getting response. Try again.";
      setOrbState("idle");
      return;
    }

    const reply = data.reply;
    conversationHistory.push({ role: "assistant", content: reply });
    addToTranscript("assistant", reply);
    speak(reply);
  } catch (error) {
    console.error("Fetch error:", error);
    status.textContent = "Connection error. Is the server running?";
    setOrbState("idle");
  }
}

What does this code do?

  • The user's message is added to both the visible transcript and the conversationHistory array. Sending the full history to the backend gives the LLM multi-turn context.
  • A POST request sends the conversation to /api/chat. Your Express server prepends the Jarvis system prompt, calls Groq, and returns the reply.
  • On success, the assistant's reply is stored in history, displayed in the transcript, and spoken aloud. On failure, the status updates with a helpful error message.

The transcript display needs a helper function that creates styled message elements and auto-scrolls to the latest one.

  • Add the addToTranscript() function and the event listeners below handleUserMessage():
function addToTranscript(role, text) {
  const div = document.createElement("div");
  div.className = "message " + role;
  div.textContent = (role === "user" ? "You: " : "Jarvis: ") + text;
  transcript.appendChild(div);
  transcript.scrollTop = transcript.scrollHeight;
}

talkBtn.addEventListener("mousedown", startListening);
talkBtn.addEventListener("mouseup", stopListening);

document.addEventListener("keydown", (e) => {
  if (e.code === "Space" && !e.repeat) {
    e.preventDefault();
    startListening();
  }
});

document.addEventListener("keyup", (e) => {
  if (e.code === "Space") {
    e.preventDefault();
    stopListening();
  }
});

if (synth.onvoiceschanged !== undefined) {
  synth.onvoiceschanged = () => {};
}

setOrbState("idle");

What does this code do?

  • addToTranscript() creates a new div with a class of either user or assistant, matching the CSS you wrote earlier. It also auto-scrolls the transcript container.
  • The button listeners start recognition on mousedown and stop on mouseup, creating a push-to-talk experience.
  • The keyboard listeners do the same with the spacebar. The !e.repeat check prevents the keydown event from firing repeatedly when the key is held.
  • The onvoiceschanged handler ensures Chrome loads its voice list before the speak function needs it. The final setOrbState("idle") starts the orb in its default blue pulse.
  • Save app.js.
  • Refresh http://localhost:3000 in Chrome. The page should load without any errors and the orb should be pulsing blue.
  • Open DevTools by pressing Cmd+Option+J (macOS) or Ctrl+Shift+J (Windows) and confirm the Console shows no red errors.

Seeing errors in the console?

  • If you see an error about SpeechRecognition not being defined, make sure you are using Chrome. Other browsers have limited support for the Web Speech API.
  • If you see a reference error for a function like handleUserMessage or setOrbState, check that all functions are present in your file. A missing closing brace from an earlier function can break everything below it.
  • Make sure your server is still running in the terminal. If it crashed, restart it with node server.js.

Still stuck? Help me debug console errors in my Jarvis voice assistant app.js file.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file and that Chrome shows no console errors.

ⓧ I'd like to double check the full code

Here is the complete public/app.js file. Compare it against yours:

const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition;

const orb = document.getElementById("orb");
const status = document.getElementById("status");
const transcript = document.getElementById("transcript");
const talkBtn = document.getElementById("talkBtn");

const synth = window.speechSynthesis;
let conversationHistory = [];
let recognition = null;
let isListening = false;

function initRecognition() {
  recognition = new SpeechRecognition();
  recognition.lang = "en-US";
  recognition.interimResults = true;
  recognition.continuous = false;

  recognition.onresult = (event) => {
    const result = event.results[event.results.length - 1];
    const text = result[0].transcript;

    if (result.isFinal) {
      status.textContent = "Processing...";
      setOrbState("thinking");
      handleUserMessage(text);
    } else {
      status.textContent = text;
    }
  };

  recognition.onend = () => {
    if (isListening) {
      isListening = false;
      talkBtn.classList.remove("active");
    }
  };

  recognition.onerror = (event) => {
    console.error("Speech recognition error:", event.error);
    status.textContent = "Error: " + event.error + ". Try again.";
    setOrbState("idle");
    isListening = false;
    talkBtn.classList.remove("active");
  };
}

function setOrbState(state) {
  orb.className = "orb";
  if (state !== "idle") {
    orb.classList.add(state);
  }
}

function startListening() {
  if (isListening) return;
  isListening = true;
  talkBtn.classList.add("active");
  setOrbState("listening");
  status.textContent = "Listening...";
  initRecognition();
  recognition.start();
}

function stopListening() {
  if (!isListening) return;
  recognition.stop();
}

async function handleUserMessage(text) {
  addToTranscript("user", text);
  conversationHistory.push({ role: "user", content: text });

  try {
    const response = await fetch("/api/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ messages: conversationHistory }),
    });

    const data = await response.json();

    if (data.error) {
      status.textContent = "Error getting response. Try again.";
      setOrbState("idle");
      return;
    }

    const reply = data.reply;
    conversationHistory.push({ role: "assistant", content: reply });
    addToTranscript("assistant", reply);
    speak(reply);
  } catch (error) {
    console.error("Fetch error:", error);
    status.textContent = "Connection error. Is the server running?";
    setOrbState("idle");
  }
}

function addToTranscript(role, text) {
  const div = document.createElement("div");
  div.className = "message " + role;
  div.textContent = (role === "user" ? "You: " : "Jarvis: ") + text;
  transcript.appendChild(div);
  transcript.scrollTop = transcript.scrollHeight;
}

function speak(text) {
  setOrbState("speaking");
  status.textContent = "Speaking...";

  const utterance = new SpeechSynthesisUtterance(text);
  utterance.rate = 1.0;
  utterance.pitch = 0.9;

  const voices = synth.getVoices();
  const preferred = voices.find(
    (v) => v.lang.startsWith("en") && (v.name.includes("Google") || v.name.includes("Microsoft"))
  );
  if (preferred) {
    utterance.voice = preferred;
  }

  utterance.onend = () => {
    setOrbState("idle");
    status.textContent = "Press spacebar or click to speak";
  };

  synth.speak(utterance);
}

talkBtn.addEventListener("mousedown", startListening);
talkBtn.addEventListener("mouseup", stopListening);

document.addEventListener("keydown", (e) => {
  if (e.code === "Space" && !e.repeat) {
    e.preventDefault();
    startListening();
  }
});

document.addEventListener("keyup", (e) => {
  if (e.code === "Space") {
    e.preventDefault();
    stopListening();
  }
});

if (synth.onvoiceschanged !== undefined) {
  synth.onvoiceschanged = () => {};
}

setOrbState("idle");

Your frontend is fully wired. Next up, you will test the complete voice loop end-to-end and have your first real conversation with Jarvis.

Secret mission

Add Wake Word Detection

Your voice assistant works great, but you still need to press a button to talk. What if it could listen for its name and activate automatically? In this secret mission, you'll add always-on wake word detection so the assistant responds when you say "Hey Jarvis" hands-free.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with a free Groq API tier, so there are no ongoing costs.

Resources you used:

  • The jarvis project folder on your Desktop (including node_modules, source files, and .env).
  • The Node.js server running at http://localhost:3000.
  • Your Groq API key stored in the GroqCloud Console.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your voice assistant.

  • Your server will keep running until you close the terminal or restart your computer.
  • Your Groq API key remains active on the free tier with no charges.
  • Your project files stay on your Desktop ready for future edits.

Pause - I'll come back to this later

Stop the running server to free up your terminal, but keep your files intact so you can pick up where you left off.

  • Switch to the terminal where your server is running.
  • Press Ctrl+C to stop the server.
  • Your project files and Groq API key remain untouched for next time.
  • To resume later, open a terminal in your jarvis folder and run node server.js again.

Delete - I don't want to use this again

Remove all project resources and revoke your API key so nothing is left behind.

  • Switch to the terminal where your server is running and press Ctrl+C to stop it.
  • Delete the project folder by running this command:

macOS

cd ~/Desktop
rm -rf jarvis

Windows

cd ~\Desktop
rmdir /s /q jarvis
  • Revoke your Groq API key:
  • Go to console.groq.com/keys in your browser.
  • Find the API key you created for this project and click the delete or revoke option next to it.

Nice Work!

Nice Work!

You just built a fully functional voice AI assistant from scratch. Your Jarvis responds to your voice, thinks using a powerful LLM, and speaks back to you with a slick sci-fi interface.

You've learned how to:

  • Used the Web Speech API to capture voice input with SpeechRecognition and produce spoken responses with SpeechSynthesis.
  • Integrated the Groq API with Llama 3.3 70B to power fast, free AI responses through a custom system prompt.
  • Built a full-stack voice interaction loop connecting an Express backend, browser voice APIs, and a dark-themed animated UI with real-time state feedback.
  • Secret Mission: Added always-on wake word detection so the assistant activates hands-free when you say "Hey Jarvis".

Ready to quiz yourself?