AI Video Analyzer with Gemini

Build a web app that uses Gemini AI to analyze videos with a timeline UI.

Introduction

30 Second Summary

You have probably watched a long video and wished you could instantly know who said what, when the mood shifted, and what was happening on screen at each moment. Manually scrubbing through footage to find key moments is tedious.

In this project, you will build a local web app that accepts any video file via drag-and-drop, sends it to Google Gemini for multimodal AI analysis, and renders a color-coded timeline showing who spoke, what they said, how they sounded emotionally, and what was visually happening at each timestamp.

What You'll Build

Drop a video onto the page and watch as Gemini processes both the visuals and the audio in a single shot, then delivers a scannable timeline breakdown you can export and share.

By the end of this project, you'll have:

  • A working drag-and-drop upload that sends any video file to Gemini and returns a structured analysis in seconds.
  • An interactive timeline UI with color-coded emotion segments, speaker labels, timestamped transcription, and visual scene descriptions you can scan at a glance.
  • A one-click JSON export of the full analysis so you can save, share, or feed the data into other tools.
  • Secret Mission: Add a synced video player so clicking any timeline segment jumps the video to that exact timestamp.

Are there any prerequisites?

You need Node.js installed on your machine, a Google account (for a free Gemini API key), and VS Code with GitHub Copilot enabled. Step 1 walks you through getting everything set up.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters to you.

Set Up the Project

Before you can send a video to AI, you need a way to connect to the Gemini API and a server that can handle file uploads.

In this step, you will get a free Gemini API key, scaffold a Node.js project with the right dependencies, and create a minimal Express server that you can build on in later steps.

In this step, get ready to:
  • Get a free Gemini API key from Google AI Studio.
  • Initialize a Node.js project and install dependencies.
  • Create a minimal Express server that serves static files.
Get your Gemini API key

The Gemini API is free to use for development. You just need an API key to authenticate your requests.

  • Open your browser and go to https://aistudio.google.com/apikey.
  • Sign in with your Google account if prompted.
  • Click Create API key.
  • Copy the generated key and paste it into a temporary text file or note. You will use it in a moment.

Why do I need an API key?

The API key proves to Google that you are an authorized developer. Every request your server sends to Gemini includes this key so Google knows who is calling.

Seeing a permission error?

If you see a message about not having permission to create a key, try signing in with a personal Google account rather than a work or school account. Organization-managed accounts sometimes lack the required permissions.

help me create a Gemini API key in Google AI Studio

Scaffold the project

Now you will create the project folder and install the packages your app needs.

  • Switch to VS Code and open a new terminal by pressing Ctrl+` (macOS and Windows).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir video-analyzer
cd video-analyzer
  • Initialize a new Node.js project by running:
npm init -y

You should see output showing a new package.json file was created with default values.

  • Open package.json in VS Code by clicking it in the Explorer sidebar (or run code package.json in the terminal).
  • Find the line that says "main": "index.js" and replace it with:
"main": "server.js",
"type": "module",

What does type module do?

Setting "type": "module" tells Node.js to use modern import/export syntax instead of the older require() style. The Gemini SDK documentation uses import syntax, so this keeps your code aligned with the official examples.

  • Save package.json.
  • Install the four packages your project needs by running:
npm install @google/genai express multer dotenv

You should see output confirming the packages were added. Your terminal will show something like "added 65 packages" (the exact number may vary).

What are these packages?

  • @google/genai is Google's official SDK for calling the Gemini API.
  • express is a web server framework that handles HTTP requests.
  • multer handles file uploads sent from a browser.
  • dotenv loads environment variables from a .env file so your API key stays out of your code.
  • Create a .env file in your video-analyzer folder by running:
touch .env
  • Open .env in VS Code and paste the following, replacing your-api-key-here with the API key you copied from Google AI Studio:
GEMINI_API_KEY=your-api-key-here
  • Save .env.

Never commit your .env file

The .env file contains your secret API key. If you ever push this project to GitHub, add .env to a .gitignore file so it is not uploaded.

Create a minimal Express server

With your dependencies installed, you can now create the server file that will power your app.

  • Open GitHub Copilot Chat in VS Code by pressing Ctrl+Shift+I (macOS: Cmd+Shift+I).
  • Paste the following prompt into the Copilot chat:
Create a file called server.js that:
- Imports express and dotenv
- Loads .env with dotenv/config
- Creates an Express app on port 3000
- Serves static files from a "public" folder
- Logs "Server running on http://localhost:3000" when it starts
  • Review the code Copilot generates and apply it to create server.js in your project folder.

What does this server do?

  • import "dotenv/config" loads your .env file so process.env.GEMINI_API_KEY becomes available.
  • express.static("public") serves any HTML, CSS, or JavaScript files you put in the public folder directly to the browser.
  • app.listen(PORT, ...) starts the server and prints a confirmation message so you know it is running.
  • Create a public folder and an empty index.html inside it by running:
mkdir public
touch public/index.html
  • Confirm the files exist by running:
ls public

You should see index.html listed in the output.

Before you run the server, what do you think will happen when you visit http://localhost:3000 in your browser? The index.html is empty, so think about what Express will serve.

  • Start your server by running:
node server.js

You should see this message printed in your terminal:

Server running on http://localhost:3000

That confirms Express is running and ready to serve files from the public folder. The blank page is expected because index.html is empty.

Don't see the message?

If you see an error about import not being recognized, make sure your package.json contains "type": "module". Without this line, Node.js defaults to the older module system and cannot parse import statements.

If you see "Cannot find module", run npm install again to ensure all packages downloaded correctly.

help me fix my Express server startup error

✔️ Awesome, I've got everything!

Great. Double check you have saved both package.json and server.js, then move on.

ⓧ I'd like to double check the full code

Your server.js should look like this:

import "dotenv/config";
import express from "express";

const app = express();
const PORT = 3000;

app.use(express.static("public"));

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

Your .env should look like this (with your own key):

GEMINI_API_KEY=your-api-key-here

Your package.json should contain "type": "module" and list all four dependencies. The exact version numbers may differ slightly from what is shown below:

{
  "name": "video-analyzer",
  "version": "1.0.0",
  "description": "AI-powered video analyzer using Gemini",
  "main": "server.js",
  "type": "module",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "@google/genai": "^2.15.0",
    "dotenv": "^16.4.7",
    "express": "^4.21.2",
    "multer": "^1.4.5-lts.1"
  }
}

Your server is live and ready to serve files. Next up, you will build the drag-and-drop interface that lets users choose a video to analyze.

Build the Drag-and-Drop Upload UI

Your server is running and serving static files from the public folder. Right now that folder holds an empty HTML file, so visiting http://localhost:3000 shows a blank page.

You need a way to get a video file from the user's computer to your server before the AI can analyze it. In this step, you will build a drag-and-drop interface and wire it up to an upload endpoint that saves the file to disk.

In this step, get ready to:
  • Create a drag-and-drop page with dark theme styling.
  • Wire up the frontend to send video files to the server.
  • Add a server route that saves uploaded files to disk.
Create the frontend drop zone

The page needs a large target area the user can drag a video file onto, plus visual feedback when a file is ready. You will give GitHub Copilot a prompt describing this UI and let it generate the HTML and CSS for you.

  • Open the Copilot Chat panel by pressing Ctrl+Cmd+I (macOS) or Ctrl+Alt+I (Windows).
  • Paste the following prompt into the chat input and send it:
Create public/index.html with:
- A centered page title "AI Video Analyzer"
- A large drag-and-drop zone (dashed border, 300px tall) that says "Drop your video here"
- When a file is dropped, show the file name and a "Analyze" button
- Include a link to style.css and app.js
- Use a clean dark theme

Create public/style.css with dark theme styling for the above.

What should Copilot generate?

Copilot should create two files. The index.html file contains the page structure: a title, a drop zone div with an icon and hint text, a hidden file-info section with an Analyze button, a loading spinner area, and a results container.

The style.css file applies a dark background, light text, dashed border on the drop zone, button styling, and a spinner animation. The exact styles will vary but the page should look clean on a dark background.

  • Review the generated code in the Copilot Chat panel.
  • Click Apply in Editor on each code block, or copy the code into the corresponding files.
  • Save both public/index.html and public/style.css.
  • Switch to your browser and refresh http://localhost:3000.

You should see a dark-themed page with the title "AI Video Analyzer" and a large dashed rectangle inviting you to drop a video.

Seeing a blank page still?

  • Confirm you saved index.html inside the public folder, not in the project root.
  • Check that your server is still running in the terminal. If you stopped it, run node server.js again.
  • Make sure the file is named exactly index.html (not Index.html or index.htm).

Why isn't my page showing up after saving?

Handle the upload in the frontend and backend

The drop zone looks correct but does not do anything yet. You need JavaScript that listens for drag-and-drop events, plus a server route that accepts the uploaded file and saves it to disk using multer.

  • Return to the Copilot Chat panel and paste this prompt:
In public/app.js:
- Handle dragover/drop events on the drop zone
- When "Analyze" is clicked, send the video file to POST /api/upload using FormData
- Show a loading spinner while uploading

In server.js:
- Add multer middleware configured to save uploads to an "uploads" folder
- Add a POST /api/upload route that accepts a single file field called "video"
- Return { success: true, filename: req.file.filename } as JSON

What does this prompt produce?

Copilot creates public/app.js with event listeners for dragover, dragleave, and drop. When a video file lands on the zone, it stores the file and shows the filename with an Analyze button. Clicking Analyze sends the file to your server using a FormData POST request.

It also updates server.js to import multer, configure it to save files into an uploads/ directory, and add the POST /api/upload route.

  • Review the generated code blocks in the Copilot Chat panel.
  • Apply the public/app.js code to a new file at that path.
  • Apply the server.js changes (Copilot should update the existing file to add the multer import and upload route).
  • Save all modified files.

Since you changed server.js, you need to restart the server for the new route to take effect.

  • Switch to your terminal and press Ctrl+C to stop the running server.
  • Restart it by running:
node server.js

You should see Server running on http://localhost:3000 printed again.

Now test the full upload flow. Before you try it, predict what you think will happen when you drag a video onto the page and click Analyze. Will you see a success message or an error?

  • Refresh http://localhost:3000 in your browser.
  • Find any short video file on your computer (MP4, MOV, or WEBM).
  • Drag the video onto the drop zone.
  • Confirm the file name appears and click Analyze Video.

You should see a "Video uploaded successfully" message showing the filename. The loading spinner appears briefly while the file transfers, then the success message replaces it.

  • Confirm the file was saved to disk by running this in a new terminal:
ls uploads

You should see a file listed with a random hash name. This is the uploaded video stored by multer.

Not seeing the success message?

  • Open the browser's Developer Tools console (F12) and check for errors. A 404 on /api/upload means the server route is missing. Make sure server.js includes the multer import and the app.post route.
  • If you see a CORS error or a network error, confirm the server restarted successfully after your edits.
  • If dragging a file does nothing, check that app.js references the correct element IDs. The drop zone needs id="drop-zone" in your HTML.

Help me debug why my video upload is failing.

✔️ Awesome, I've got everything!

Great. Double check you have saved all four files: public/index.html, public/style.css, public/app.js, and server.js.

ⓧ I'd like to double check the full code

Since Copilot's output can vary, check that your files contain these key elements. Here is what server.js should look like:

import "dotenv/config";
import express from "express";
import multer from "multer";

const app = express();
const PORT = 3000;

const upload = multer({ dest: "uploads/" });

app.use(express.static("public"));

app.post("/api/upload", upload.single("video"), async (req, res) => {
  if (!req.file) {
    return res.status(400).json({ error: "No video file provided" });
  }
  res.json({ success: true, filename: req.file.filename });
});

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

For public/index.html, confirm these elements are present:

  • A div with id="drop-zone" and class drop-zone.
  • A hidden div with id="file-info" containing a span with id="file-name" and a button with id="analyze-btn".
  • A hidden div with id="loading" containing a spinner.
  • A hidden div with id="results".
  • A <link> to style.css and a <script> for app.js.

For public/app.js, confirm these behaviors:

  • It grabs elements by ID: drop-zone, file-info, file-name, analyze-btn, loading, results.
  • The drop event handler checks file.type.startsWith("video/") before accepting the file.
  • The Analyze button's click handler sends a POST to /api/upload with FormData containing a field named video.
  • The loading spinner shows during the request and hides when it finishes.

You now have a working upload pipeline. A user can drag a video onto the page and your server saves it to disk. Next up, you will send that video to Gemini and get back an AI-generated analysis of everything happening in the clip.

Send the Video to Gemini

Your drag-and-drop UI is working and videos are landing on your server. But right now they just sit there. The server confirms the upload and does nothing with the file.

In this step, you'll connect your server to Google's Gemini AI. Gemini can watch and listen to video files in a single API call. You'll send it the uploaded video and get back a full text analysis describing what's happening.

In this step, get ready to:
  • Upload the saved video file to Gemini's File API.
  • Get raw text analysis describing the video's content.
  • Display the AI response in the browser.
Upload to Gemini File API and get raw analysis

Gemini's File API lets you upload a video once, wait for it to finish processing, and then ask the model to analyze it. Your server needs to send the saved file to Gemini, poll until it's ready, and then request an analysis.

  • Open GitHub Copilot Chat in VS Code (click the chat icon in the sidebar or press Cmd+Shift+I on macOS / Ctrl+Shift+I on Windows).
  • Paste the following prompt into Copilot Chat:
In server.js, update the POST /api/upload route:
- After multer saves the file, upload it to Gemini using ai.files.upload() with the saved file path and mimeType "video/mp4"
- Poll the file state every 3 seconds until it becomes "ACTIVE"
- Once active, call ai.models.generateContent() with model "gemini-3.6-flash"
- Use this prompt: "Analyze this video. Describe what is visually happening, transcribe all speech with speaker labels, note the emotion/tone of each speaker, and identify when speakers change. Provide timestamps in MM:SS format for each observation."
- The contents should include the file reference and the text prompt
- Return the response text to the client
- Import GoogleGenAI from @google/genai at the top
- Initialize with: const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY })

On the frontend (app.js):
- Display the returned text in a results div below the drop zone
- Remove the loading spinner when done

What does this prompt tell Copilot to do?

You're asking Copilot to wire up the full Gemini integration in one shot. The server uploads the video to Gemini's File API, waits for processing to complete, then asks the model to describe what's happening visually and transcribe all speech with timestamps.

On the frontend, the response text replaces the "uploaded successfully" message so you can see Gemini's raw analysis in the browser.

  • Review the code Copilot generates. Apply the changes to both server.js and public/app.js.
  • Verify that the top of server.js now has these imports and initialization:
import "dotenv/config";
import express from "express";
import multer from "multer";
import { GoogleGenAI } from "@google/genai";
import fs from "fs";

const app = express();
const PORT = 3000;

// Initialize the Gemini AI client with your API key
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const upload = multer({ dest: "uploads/" });

What's new here?

You imported GoogleGenAI from the @google/genai package and fs for file cleanup. The ai client is initialized once at the top so every request can use it.

  • Verify that the route handler now uploads the file to Gemini, polls for completion, and generates content:
app.post("/api/upload", upload.single("video"), async (req, res) => {
  try {
    if (!req.file) {
      return res.status(400).json({ error: "No video file provided" });
    }
    const filePath = req.file.path;
    const mimeType = req.file.mimetype || "video/mp4";
    // Upload the saved video to Gemini's File API
    const uploadedFile = await ai.files.upload({ file: filePath, config: { mimeType } });
    // Poll until Gemini finishes processing the video
    let file = uploadedFile;
    while (!file.state || file.state.toString() !== "ACTIVE") {
      await new Promise((resolve) => setTimeout(resolve, 3000));
      file = await ai.files.get({ name: file.name });
    }
    // Ask Gemini to analyze the video
    const prompt = "Analyze this video. Describe what is visually happening, transcribe all speech with speaker labels, note the emotion/tone of each speaker, and identify when speakers change. Provide timestamps in MM:SS format for each observation.";
    const result = await ai.models.generateContent({
      model: "gemini-3.6-flash",
      contents: [{ role: "user", parts: [{ fileData: { fileUri: file.uri, mimeType: file.mimeType } }, { text: prompt }] }],
    });
    // Clean up the local file since Gemini has its own copy
    fs.unlink(filePath, () => {});
    res.json({ text: result.text });
  } catch (error) {
    console.error("Error analyzing video:", error);
    res.status(500).json({ error: "Failed to analyze video: " + error.message });
  }
});

What does this route do?

  • ai.files.upload() sends the saved video to Gemini's cloud storage so the model can process it.
  • The while loop polls every 3 seconds until the file state becomes ACTIVE. Video processing takes time because Gemini samples frames and extracts audio.
  • ai.models.generateContent() sends the processed file plus your text prompt to gemini-3.6-flash. The model watches the entire video and returns a text response.
  • fs.unlink() deletes the local copy since Gemini already has it. This prevents your uploads folder from filling up.
  • Now check public/app.js. Find the success handler inside the analyzeBtn click listener. It currently looks like this:
    const data = await response.json();
    results.innerHTML = `<p>Video uploaded successfully: ${data.filename}</p>`;
    results.classList.remove("hidden");
  • Replace the results.innerHTML line so it displays the AI's text response instead:
    const data = await response.json();
    results.innerHTML = `<pre>${data.text}</pre>`;
    results.classList.remove("hidden");
  • Also add let analysisData = null; on a new line after let selectedFile = null; at the top of app.js. You'll use this variable in a later step.
  • Save both server.js and public/app.js.

Copilot generated something different?

Copilot may produce slightly different code depending on your VS Code context. The key things to verify are:

  • The import uses { GoogleGenAI } from "@google/genai" (not the older @google/generative-ai package).
  • The model is gemini-3.6-flash.
  • The route returns { text: result.text } as JSON.

If Copilot used a different pattern, replace the generated code with the snippets shown above. Help me fix my Gemini integration code

✔️ Awesome, I've got everything!

Great. Make sure you've saved both files before moving on.

ⓧ I'd like to double check the full code

Your server.js should have these sections in order:

  • Imports: dotenv/config, express, multer, GoogleGenAI from @google/genai, fs.
  • Initialization: app, PORT, ai (GoogleGenAI client), upload (multer).
  • Static file serving: app.use(express.static("public")).
  • POST /api/upload route: try/catch wrapping multer save, File API upload, polling loop, generateContent call, file cleanup, JSON response.
  • app.listen on port 3000.

Your public/app.js should show let analysisData = null; after let selectedFile = null; and the success handler should display data.text inside a <pre> tag.

Test with a sample video

You need a short video with someone speaking. A screen recording, a quick phone clip, or any MP4 under 2 minutes works well. Shorter videos process faster and give more accurate timestamps.

  • Stop your running server with Ctrl+C in the terminal.
  • Restart it by running:
node server.js
  • Open http://localhost:3000 in your browser.

Before you drop a video, think about what you expect to see. Will the response come back as structured data you could build a timeline from, or as free-form paragraphs?

  • Drag a short video file onto the drop zone.
  • Click Analyze Video. Wait 30-60 seconds for Gemini to process and respond.

You should see a wall of plain text appear below the drop zone. Gemini describes what's happening in the video, names speakers, and adds timestamps. But look at the format. It's paragraphs and bullet points mixed together with no consistent structure.

Why is this a problem?

You can't build a timeline UI from free-form text. There's no reliable way to parse speaker names, timestamps, or emotions out of paragraphs. The data is all there, but it's trapped in an unpredictable format.

In the next step, you'll tell Gemini exactly what shape of JSON to return. That transforms this wall of text into clean, machine-readable data.

Seeing an error instead of text?

  • Check that your .env file contains a valid GEMINI_API_KEY value (not the placeholder text).
  • Make sure you restarted the server after saving server.js. Changes don't take effect until you restart.
  • If you see a timeout, try a shorter video (under 1 minute). Longer videos take more time to process.

Help me debug my Gemini video analysis error

You've proven that Gemini can analyze video in a single API call. Next up, you'll tell it exactly what shape of data to return so you can build a real timeline from the response.

Get Structured Timeline Data

In the last step, you uploaded a video and Gemini returned a wall of free-form text. That response has timestamps, speakers, and emotions buried in prose, but there is no reliable way to extract them programmatically.

To build a visual timeline, you need predictable data. Gemini's structured output feature lets you define a JSON schema that forces the model to return data in exactly the shape you specify.

In this step, get ready to:
  • Define a JSON response schema with fields for timestamps, speakers, emotions, and visual descriptions.
  • Update the frontend to display the structured JSON response.
  • Verify the output contains all required fields for every video segment.
Add the response schema

A response schema is a JSON object that describes the exact structure of data you want back from the AI. Instead of getting a paragraph of text, you get a parsed object with named fields you can render directly.

  • Paste the following prompt into GitHub Copilot Chat (open it with Cmd+Shift+I on macOS or Ctrl+Shift+I on Windows):
In server.js, update the POST /api/upload route:
- Add import for path at the top
- Replace the prompt with this exact text: "Analyze this video completely. For each distinct moment, provide the timestamp, identify the speaker, transcribe their speech, classify their emotion, and describe what is visually happening. Mark scene changes and silent moments too. Be thorough and cover the entire video duration."
- Add a responseSchema variable that defines this structure:
  - summary (string): brief overall summary
  - segments (array of objects), each with:
    - timestamp (string): MM:SS format
    - speaker (string): e.g. "Speaker 1" or "Narrator"
    - content (string): what was said or described
    - emotion (string, enum: happy/sad/angry/neutral)
    - visual_description (string): what is visually happening
    - type (string, enum: speech/scene_change/silence)
  - All fields in each segment are required
- Update the generateContent call to include a config with responseMimeType "application/json" and the responseSchema
- Parse the JSON response with JSON.parse(result.text)
- Return the parsed analysis object to the client instead of { text: result.text }

What does a response schema do?

When you set responseMimeType to "application/json" and provide a responseSchema, Gemini is forced to return valid JSON matching your structure. Every segment will have a timestamp, speaker label, emotion classification, and visual description. No more parsing free-form text.

  • Apply Copilot's suggestion to server.js.
  • Verify the responseSchema variable in your file looks like this:
// Define the exact shape of data Gemini must return
const responseSchema = {
  type: "object",
  properties: {
    summary: { type: "string", description: "A concise summary of the entire video content." },
    segments: {
      type: "array",
      description: "List of analyzed segments with timestamps.",
      items: {
        type: "object",
        properties: {
          timestamp: { type: "string", description: "Timestamp in MM:SS format" },
          speaker: { type: "string", description: "Speaker label e.g. Speaker 1, Narrator" },
          content: { type: "string", description: "What was said or a description of the moment" },
          emotion: { type: "string", enum: ["happy", "sad", "angry", "neutral"] },
          visual_description: { type: "string", description: "What is visually happening" },
          type: { type: "string", enum: ["speech", "scene_change", "silence"] },
        },
        required: ["timestamp", "speaker", "content", "emotion", "visual_description", "type"],
      },
    },
  },
  required: ["summary", "segments"],
};

What do these fields mean?

  • timestamp forces MM:SS format so you can sort and display segments in order.
  • emotion is an enum restricted to four values. This means you can map each to a specific color in the UI.
  • type distinguishes speech from scene changes and silence, letting you style them differently.
  • required on all fields guarantees no segment is missing data. Every entry in the array will be complete.
  • Also verify the generateContent call now includes the schema config and parses the response:
// Send the video and schema to Gemini
const result = await ai.models.generateContent({
  model: "gemini-3.6-flash",
  contents: [{ role: "user", parts: [{ fileData: { fileUri: file.uri, mimeType: file.mimeType } }, { text: prompt }] }],
  config: { responseMimeType: "application/json", responseSchema },
});

// Parse the guaranteed-valid JSON response
const analysis = JSON.parse(result.text);

// Clean up the local file and return structured data
fs.unlink(filePath, () => {});
res.json(analysis);
  • Save server.js.

Copilot generated different code?

Copilot sometimes generates slight variations. The key things to verify are:

  • The config object is passed inside the generateContent call (not as a separate argument).
  • responseMimeType is set to "application/json" (not "text/plain").
  • The response is parsed with JSON.parse(result.text) before sending to the client.

Help me fix my structured output config in server.js.

Verify the structured output

The server now returns structured JSON instead of raw text. You need to update the frontend to handle this new response shape before testing.

  • In public/app.js, find the try block inside the analyzeBtn click handler.
  • Find these two lines:
const data = await response.json();
results.innerHTML = `<pre>${data.text}</pre>`;
  • Replace them with these lines, which store the full JSON object in analysisData and display it formatted:
analysisData = await response.json();
results.innerHTML = `<pre>${JSON.stringify(analysisData, null, 2)}</pre>`;
  • Save public/app.js.
  • Stop your running server with Ctrl+C in the terminal, then restart it by running:
node server.js

Before you test, think about this: you uploaded the same video last step and got a wall of text. What do you expect to see now that you have a response schema in place?

  • Open http://localhost:3000 in your browser and drop the same video file you used in the previous step.
  • Click Analyze Video and wait for the response.

Instead of a wall of prose, you should see clean JSON with a summary string and a segments array. Each segment contains timestamp, speaker, content, emotion, visual_description, and type fields.

Why structured output instead of parsing the text yourself?

You could try to regex timestamps and speaker labels out of free-form text, but the format changes between videos. Structured output guarantees every response matches your schema. The emotion field is always one of four values. The timestamp is always MM:SS. No parsing code needed.

Getting an error instead of JSON?

  • If you see a 500 error, check the terminal for details. A common cause is the responseSchema variable not being defined before the generateContent call.
  • If the browser still shows raw text (not JSON), make sure you saved both server.js and public/app.js, then restarted the server.
  • If the response has missing fields, verify all entries in the required array match the property names exactly.

Help me debug my Gemini structured output response.

✔️ Awesome, I've got everything!

Great. Double check you have saved both server.js and public/app.js.

ⓧ I'd like to double check the full code

Here is the full server.js after this step. Scroll through and confirm your file matches:

import "dotenv/config";
import express from "express";
import multer from "multer";
import path from "path";
import { GoogleGenAI } from "@google/genai";
import fs from "fs";

const app = express();
const PORT = 3000;

const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const upload = multer({ dest: "uploads/" });

app.use(express.static("public"));
app.post("/api/upload", upload.single("video"), async (req, res) => {
  try {
    if (!req.file) {
      return res.status(400).json({ error: "No video file provided" });
    }
    const filePath = req.file.path;
    const mimeType = req.file.mimetype || "video/mp4";
    const uploadedFile = await ai.files.upload({ file: filePath, config: { mimeType } });
    let file = uploadedFile;
    while (!file.state || file.state.toString() !== "ACTIVE") {
      await new Promise((resolve) => setTimeout(resolve, 3000));
      file = await ai.files.get({ name: file.name });
    }
    const prompt = `Analyze this video completely. For each distinct moment, provide the timestamp, identify the speaker, transcribe their speech, classify their emotion, and describe what is visually happening. Mark scene changes and silent moments too. Be thorough and cover the entire video duration.`;
    const responseSchema = {
      type: "object",
      properties: {
        summary: { type: "string", description: "A concise summary of the entire video content." },
        segments: {
          type: "array",
          description: "List of analyzed segments with timestamps.",
          items: {
            type: "object",
            properties: {
              timestamp: { type: "string", description: "Timestamp in MM:SS format" },
              speaker: { type: "string", description: "Speaker label e.g. Speaker 1, Narrator" },
              content: { type: "string", description: "What was said or a description of the moment" },
              emotion: { type: "string", enum: ["happy", "sad", "angry", "neutral"] },
              visual_description: { type: "string", description: "What is visually happening" },
              type: { type: "string", enum: ["speech", "scene_change", "silence"] },
            },
            required: ["timestamp", "speaker", "content", "emotion", "visual_description", "type"],
          },
        },
      },
      required: ["summary", "segments"],
    };
    const result = await ai.models.generateContent({
      model: "gemini-3.6-flash",
      contents: [{ role: "user", parts: [{ fileData: { fileUri: file.uri, mimeType: file.mimeType } }, { text: prompt }] }],
      config: { responseMimeType: "application/json", responseSchema },
    });
    const analysis = JSON.parse(result.text);
    fs.unlink(filePath, () => {});
    res.json(analysis);
  } catch (error) {
    console.error("Error analyzing video:", error);
    res.status(500).json({ error: "Failed to analyze video: " + error.message });
  }
});

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

And here is the full public/app.js:

const dropZone = document.getElementById("drop-zone");
const fileInfo = document.getElementById("file-info");
const fileName = document.getElementById("file-name");
const analyzeBtn = document.getElementById("analyze-btn");
const loading = document.getElementById("loading");
const results = document.getElementById("results");

let selectedFile = null;
let analysisData = null;

dropZone.addEventListener("dragover", (e) => { e.preventDefault(); dropZone.classList.add("dragover"); });
dropZone.addEventListener("dragleave", () => { dropZone.classList.remove("dragover"); });
dropZone.addEventListener("drop", (e) => {
  e.preventDefault();
  dropZone.classList.remove("dragover");
  const file = e.dataTransfer.files[0];
  if (file && file.type.startsWith("video/")) {
    selectedFile = file;
    fileName.textContent = file.name;
    fileInfo.classList.remove("hidden");
    results.classList.add("hidden");
    results.innerHTML = "";
  }
});
analyzeBtn.addEventListener("click", async () => {
  if (!selectedFile) return;
  fileInfo.classList.add("hidden");
  loading.classList.remove("hidden");
  results.classList.add("hidden");
  const formData = new FormData();
  formData.append("video", selectedFile);
  try {
    const response = await fetch("/api/upload", { method: "POST", body: formData });
    if (!response.ok) { const err = await response.json(); throw new Error(err.error || "Upload failed"); }
    analysisData = await response.json();
    results.innerHTML = `<pre>${JSON.stringify(analysisData, null, 2)}</pre>`;
    results.classList.remove("hidden");
  } catch (error) {
    results.innerHTML = `<p style="color: #ef5350;">Error: ${error.message}</p>`;
    results.classList.remove("hidden");
  } finally {
    loading.classList.add("hidden");
  }
});

You now have machine-readable video analysis data with guaranteed structure. Next up, you will transform this JSON into a beautiful interactive timeline UI with color-coded emotions and clickable segments.

Build the Interactive Timeline UI

You have clean, structured JSON coming back from Gemini with timestamps, speaker labels, emotions, and visual descriptions. Right now, all of that data dumps as raw JSON in the browser.

In this step, you'll transform that JSON into a scannable, color-coded timeline. Each segment becomes a visual card showing who spoke, what they said, how they sounded, and what was happening on screen.

In this step, get ready to:
  • Build a renderTimeline function that creates timeline cards from structured data.
  • Style the timeline with color-coded emotion borders and animated cards.
  • Add an Export JSON button for downloading the full analysis.
Render the timeline

The current code displays Gemini's response as a raw JSON dump. You need a function that reads the structured data and builds proper HTML elements for each segment.

  • Open the Chat view in VS Code by pressing Cmd+Shift+I (macOS) or Ctrl+Alt+I (Windows).
  • Paste the following prompt into the chat input and press Enter:
In public/app.js, when the API returns structured JSON:
- Parse the response and render a timeline UI in a results container
- Show the summary at the top in a highlighted box
- For each segment, create a timeline card showing:
  - Timestamp on the left (styled as a badge)
  - Speaker name (bold)
  - Content/transcription text
  - Visual description in italic below
  - Emotion as a colored pill badge (green=happy, blue=sad, red=angry, gray=neutral)
  - Type indicator (speech/scene_change/silence) as a small label
- Cards should stack vertically with a connecting line on the left (like a timeline)
- Add an "Export JSON" button that downloads the raw analysis as a .json file
  • Review the generated code. Copilot will update app.js with the changes shown as inline diffs in the editor.
  • Select Keep to accept all changes.

What should Copilot generate?

Copilot should produce two key changes in app.js:

  • The line that previously showed raw JSON (results.innerHTML = `<pre>...`) gets replaced with a call to renderTimeline(analysisData).
  • A new renderTimeline(data) function at the bottom of the file that builds the summary box, timeline cards, and export button.

Confirm that the click handler in your code now calls renderTimeline instead of setting innerHTML to raw JSON. Find the analyzeBtn click handler's try block and verify it looks like this:

analysisData = await response.json();
renderTimeline(analysisData);

What does renderTimeline do?

  • It clears the results container and makes it visible.
  • If a summary field exists, it creates a highlighted box with the overall video summary.
  • For each segment in the segments array, it builds a card with the timestamp, speaker, emotion badge, transcription text, and visual description.
  • Each card gets CSS classes like emotion-happy and type-speech so the stylesheet can color-code them.
  • The Export button creates a downloadable video-analysis.json file from the raw data.

Copilot generated something different?

If Copilot's output does not include a renderTimeline function or still displays raw JSON, try being more specific in a follow-up prompt. Ask it to replace the raw JSON display with a dedicated render function.

If the generated code has syntax errors (red underlines in the editor), select Undo on the inline diff and re-prompt with the same text.

Why isn't Copilot updating my app.js correctly?

Style the timeline

The timeline cards exist in the HTML now, but without CSS they look like unstyled divs. You need styles that give each card an emotion-colored left border, a connecting vertical line, and smooth fade-in animations.

  • In the same Chat view, paste this prompt and press Enter:
In public/style.css, add styles for:
- Timeline container with a vertical line on the left
- Timeline cards with subtle background, rounded corners, left border colored by emotion
- Timestamp badges (monospace font, dark background)
- Emotion pill badges with appropriate colors
- Speaker names bold and slightly larger
- Visual descriptions in a lighter/italic style
- Scene change cards styled differently (dashed border, muted colors)
- Smooth fade-in animation for cards as they appear
- Export button styled at the bottom of the results
- Make sure everything looks good on the dark theme
  • Review the generated CSS in the inline diff. You should see new classes like .timeline, .timeline-card, .emotion-badge, and .export-btn.
  • Select Keep to accept the CSS changes.
  • Save both app.js and style.css.

What do the key CSS classes do?

  • .timeline::before draws a thin vertical line connecting all the cards.
  • .timeline-card.emotion-happy sets a green left border. Sad is blue, angry is red, neutral is gray.
  • .timeline-card.type-scene_change uses a dashed border to visually separate scene transitions from speech.
  • The fadeIn animation makes cards slide up gently as they appear.

Before you test, what do you think the timeline will look like when you analyze the same video from earlier?

  • Restart the server by pressing Ctrl+C in the terminal and running:
node server.js
  • Open http://localhost:3000 in your browser.
  • Drop the same test video you used earlier onto the drop zone and click Analyze Video.

After 30-60 seconds, you should see a color-coded timeline rendered below the drop zone. Each card shows a timestamp badge, speaker name in bold, the transcription text, a colored emotion pill, and an italic visual description.

Still seeing raw JSON or a blank page?

  • Check that app.js calls renderTimeline(analysisData) inside the try block, not the old results.innerHTML line.
  • Open the browser DevTools console (F12) and look for JavaScript errors. A missing closing brace in renderTimeline is the most common issue.
  • Confirm that style.css includes a .timeline-card class. If it is missing, Copilot may not have generated the CSS. Re-run the styling prompt.
  • Help me debug my timeline UI not rendering.

✔️ Awesome, I've got everything!

Great. Double check that both app.js and style.css are saved.

ⓧ I'd like to double check the full code

Your app.js should contain these parts in order:

  • Six const declarations caching DOM elements (dropZone, fileInfo, fileName, analyzeBtn, loading, results).
  • Two let variables: selectedFile and analysisData.
  • Drag-and-drop event listeners (dragover, dragleave, drop).
  • The analyzeBtn click handler that calls renderTimeline(analysisData) on success.
  • A renderTimeline(data) function that builds the summary box, timeline cards (with classes emotion-{emotion} and type-{type}), and the export button.

Your style.css should include these class selectors:

  • .summary-box with a purple left border.
  • .timeline and .timeline::before for the vertical connecting line.
  • .timeline-card with emotion color variants (.emotion-happy, .emotion-sad, .emotion-angry, .emotion-neutral).
  • .emotion-badge color variants and .type-label.
  • .export-btn styles.
  • A @keyframes fadeIn animation.

Your video analysis now renders as a professional, scannable timeline. Next up, you'll tackle the Secret Mission and sync a video player to these timeline segments.

Secret mission

Sync a Video Player to the Timeline

Your timeline shows what happened in the video, but you still have to open the original file to actually watch those moments. What if clicking a segment jumped straight to that point in the video, and the currently-playing segment lit up as you watched?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally and uses the Gemini API free tier, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The video-analyzer project folder on your Desktop (contains all source code, node_modules, and the .env file with your API key).
  • The uploads/ folder inside the project (created automatically by multer when you uploaded videos).
  • Your Gemini API key in Google AI Studio.

Keep everything running

No action needed. Choose this if you want to keep experimenting with video analysis or plan to extend the project further.

  • Your project folder stays on your Desktop, ready to use anytime.
  • Your Gemini API key remains active in Google AI Studio for future projects.
  • To restart the server later, open a terminal in the video-analyzer folder and run node server.js.

Pause - I'll come back to this later

Stop the running server to free up your terminal. Your files stay on disk for when you return.

  • In the terminal where the server is running, press Ctrl+C to stop it.
  • Your project folder and API key remain intact. No costs accumulate while the server is stopped.

Delete - I don't want to use this again

Remove all project files and revoke your API key for a clean slate.

  • In the terminal where the server is running, press Ctrl+C to stop it.
  • Delete the project folder by running this command from your Desktop:

macOS

cd ~/Desktop
rm -rf video-analyzer

Windows

cd ~\Desktop
Remove-Item -Recurse -Force video-analyzer

Nice Work!

Nice Work!

You just built a full-stack AI video analysis tool from scratch. Drop any video onto the page, and Gemini watches it, listens to it, and delivers a structured timeline breakdown you can scan at a glance.

You've learned how to:

  • Integrate multimodal AI into a web app by uploading video files to the Gemini API and processing both visual frames and audio in a single API call.
  • Use structured JSON output schemas to get machine-readable responses from an AI model instead of raw text, with typed fields for timestamps, speakers, emotions, and visual descriptions.
  • Build an interactive timeline UI with color-coded emotion segments, speaker labels, and an export-to-JSON feature on top of an Express backend with drag-and-drop file uploads.
  • Secret Mission: Synced a video player to the timeline so clicking any segment jumps the video to that timestamp, with auto-highlighting as the video plays.

Ready to quiz yourself?