Build a Visual AI Agent Chrome Extension

Build a Chrome extension that screenshots your browser and analyzes it with local AI.

Introduction

30 Second Summary

Think about how screen time apps know exactly what you were doing on your phone all day. Now imagine building that same capability yourself, but for your browser, powered by AI that actually understands what is on screen.

In this project, you will build a Chrome extension that periodically screenshots your active tab, analyzes each image with a local vision AI model, and stores structured activity logs in a cloud database.

What You'll Build

Click your extension icon, hit Start Monitoring, then browse normally. Every 30 seconds the agent captures your screen, sends it to a local AI that describes what you are doing, and logs everything to a real database you can query.

By the end of this project, you'll have:

  • A working extension popup with a live toggle that starts and stops browser monitoring with a single click.
  • A local vision AI pipeline that analyzes screenshots and produces natural language descriptions like "User is reading a GitHub repository README about React hooks" without sending data to any external AI service.
  • A Supabase database full of structured activity logs with timestamps, URLs, tracked click events, and AI descriptions that you can query from the dashboard.
  • Secret Mission: Build a history dashboard directly in the extension popup showing your last 10 activities with timestamps.

Are there any prerequisites?

You should have Node.js installed, a GitHub account, and basic familiarity with JavaScript. Step 1 walks you through installing everything else you need.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

You're building a Visual AI Agent Chrome extension that monitors browser activity, analyzes screenshots with a local vision model, and stores structured logs in a cloud database. This is the exact type of agentic monitoring system that Neoflo uses for finance operations.

Set Up Ollama, Supabase, and the Extension Scaffold

A visual AI agent needs three things to work: a vision model that can describe what it sees, a database to store its observations, and a browser extension that ties it all together.

In this step, you will install Ollama to run a local vision AI model, set up a Supabase database for activity logs, and scaffold the Chrome extension project structure.

In this step, get ready to:
  • Install Ollama and test the moondream2 vision model locally.
  • Create a Supabase project with an activities table for storing logs.
  • Scaffold the Chrome extension folder structure and load it in the browser.
Install Ollama and pull moondream2

Ollama lets you run AI models locally on your machine. The moondream2 model is a lightweight vision model that can describe images in natural language.

  • Open your browser and go to https://ollama.com/download.
  • Download the Windows installer.
  • Run the installer and follow the prompts to complete the installation.

Once installed, you need to pull the moondream2 vision model. This downloads the model files (about 1.7GB) to your machine.

  • Press the Windows key to open your search bar.
  • Type Terminal and press Enter to open it.
  • Pull the moondream2 model by running this command:
ollama pull moondream2

✔️ I see download progress

The model is downloading. This may take a few minutes depending on your internet speed. Once it completes, you will see a success message.

ⓧ 'ollama' is not recognized

Ollama did not install correctly or is not in your system PATH.

  • Close your terminal and reopen it (new terminals pick up PATH changes).
  • If it still fails, re-run the Ollama installer from the downloaded file.
  • After reinstalling, open a fresh terminal and try ollama pull moondream2 again.

Now test that the model can actually analyze an image. You will need any image file on your machine for this test.

  • Find any image file on your Desktop (a screenshot, photo, or downloaded image). If you do not have one, right-click your Desktop, select New > Bitmap Image, and save it as test.png.
  • Before you run the next command, what do you think the AI will say about your image?
  • Test moondream2 with your image by running this command:
ollama run moondream2 "What is in this image?" --image [[IMAGEPATH="C:\Users\YourName\Desktop\test.png"]]

You should see a text description of your image printed in the terminal. The model takes a few seconds to load on first run.

What does this command do?

  • ollama run moondream2 starts the moondream2 vision model.
  • The quoted text is the prompt telling the model what to do with the image.
  • --image followed by a file path tells Ollama which image to analyze.

Seeing an empty response or timeout?

  • Make sure you are using a full file path (e.g. C:\Users\YourName\Desktop\test.png) not just a filename.
  • Try a different image file. Some very small or corrupted images produce empty results.
  • If the model takes too long, your GPU may not be detected. Close other GPU-heavy applications and retry.

Why is moondream2 not describing my image?

Create a Supabase project and activities table

Your extension needs somewhere to store the activity logs it generates. Supabase gives you a free cloud PostgreSQL database with a REST API you can call directly from JavaScript.

  • Open https://supabase.com in your browser.
  • Click Start your project and sign up with your GitHub account.
  • Once signed in, click New project.
  • Enter visual-ai-agent as the project name.
  • Set a database password (save it somewhere safe, you will not need it again for this project but cannot recover it).
  • Choose the region closest to you and click Create new project.
  • Wait for the project to finish setting up (about 1-2 minutes).

Now create the table where your extension will store activity logs.

  • In the left sidebar, click SQL Editor.
  • Paste the following SQL and click Run:
CREATE TABLE activities (
  id uuid DEFAULT gen_random_uuid() PRIMARY KEY,
  created_at timestamptz DEFAULT now(),
  url text,
  page_title text,
  event_type text,
  event_details text,
  ai_description text,
  screenshot_base64 text
);

What does this table store?

  • id is a unique identifier for each activity entry, generated automatically.
  • created_at records when the activity was captured.
  • url and page_title record which page was active.
  • ai_description holds the vision model's natural language description of the screenshot.
  • event_details stores user interaction data like clicks and navigation as JSON text.

When you click Run, Supabase may show a warning dialog about Row Level Security. This is expected.

  • If a warning dialog appears, click Run without RLS.
  • You should see a Success message confirming the table was created.

Why disable RLS for this project?

Row Level Security controls who can read and write rows in your table. For this demo, you are the only user and the extension writes directly with your API key. Disabling RLS simplifies the setup so you can focus on building the agent itself.

In a production app, you would enable RLS and create policies to restrict access per user.

Next, the extension needs to grant access for the API roles so the REST API can reach the table.

  • In the same SQL Editor, paste and run this:
GRANT SELECT, INSERT ON activities TO anon;
GRANT SELECT, INSERT ON activities TO authenticated;
GRANT SELECT, INSERT ON activities TO service_role;
  • Verify the table exists by clicking Table Editor in the left sidebar.
  • You should see activities listed with 0 rows.

Now copy your Supabase credentials. The extension will use these to connect.

  • In the left sidebar, click the gear icon to open Project Settings.
  • Click API under the Configuration section.
  • Copy the Project URL (starts with https://) and save it in a text file.
  • Copy the anon/public key (under Project API keys) and save it in the same text file.

Keep your credentials handy

You will paste these two values into your extension code in Step 5. Save them somewhere easy to find, like a Notepad file on your Desktop.

Scaffold the Chrome extension files

A Chrome extension built with Manifest V3 needs a specific file structure. The manifest tells Chrome what your extension can do, and the other files provide the logic.

  • Open your terminal (press the Windows key, type Terminal, press Enter).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir visual-ai-agent
cd visual-ai-agent
  • Confirm you are in the right folder by running:
pwd

You should see a path ending in Desktop\visual-ai-agent.

Now open this folder in your code editor.

  • Press the Windows key, type VS Code (or your editor name), and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the visual-ai-agent folder, and click Select Folder.

Create the manifest file that tells Chrome about your extension.

  • In your editor, create a new file called manifest.json and paste this content:
{
  "manifest_version": 3,
  "name": "Visual AI Agent",
  "version": "1.0.0",
  "description": "AI-powered browser activity monitor",
  "permissions": [
    "tabs",
    "activeTab",
    "alarms",
    "scripting"
  ],
  "host_permissions": [
    "<all_urls>"
  ],
  "background": {
    "service_worker": "background.js"
  },
  "content_scripts": [
    {
      "matches": ["<all_urls>"],
      "js": ["content.js"],
      "run_at": "document_idle"
    }
  ],
  "action": {
    "default_popup": "popup.html",
    "default_title": "Visual AI Agent"
  }
}

What does the manifest configure?

  • permissions grants access to Chrome APIs: tabs for screenshots, alarms for periodic capture, scripting for injecting content scripts.
  • host_permissions with <all_urls> lets the extension capture any tab the user visits.
  • background.service_worker points to the Service Worker that runs the capture and AI logic.
  • content_scripts injects content.js into every page to track user events.
  • Save manifest.json.
  • Create four more files in the same folder. For each one, create a new file in your editor with the exact name and content shown:

background.js:

// placeholder

content.js:

// placeholder

popup.html:

<!-- placeholder -->

popup.js:

// placeholder
  • Save all files.
  • Verify your folder structure by switching to your terminal and running:
ls

You should see five files listed: background.js, content.js, manifest.json, popup.html, and popup.js.

Finally, load your extension into Chrome to verify everything is wired up.

  • Open Google Chrome.
  • Type chrome://extensions in the address bar and press Enter.
  • Toggle on Developer mode in the top-right corner.
  • Click Load unpacked.
  • Navigate to your Desktop, select the visual-ai-agent folder, and click Select Folder.

Your extension card should appear on the page showing "Visual AI Agent" with no error messages.

Seeing an error on the extension card?

  • Check that manifest.json has no trailing commas or syntax errors. Even one misplaced comma breaks the entire extension.
  • Make sure all five files (manifest.json, background.js, content.js, popup.html, popup.js) exist in the folder root, not inside a subfolder.
  • Click Errors on the extension card to see the specific issue.

Why is my Chrome extension showing errors after loading?

✔️ Awesome, I've got everything!

Great. Double check you have saved all five files in the visual-ai-agent folder.

ⓧ I'd like to double check the full code

Your manifest.json should contain the full configuration above. The other four files (background.js, content.js, popup.js, popup.html) should each contain only a single placeholder comment. Scroll up to the manifest code block and confirm your file matches character-for-character.

Your development environment is ready. Ollama is running a vision model locally, Supabase is waiting to store your data, and Chrome recognizes your extension. Next up, you will give the extension eyes by capturing screenshots from the active tab.

Capture Screenshots from the Active Tab

Your extension scaffold is loaded in Chrome, but right now every file is a placeholder. The core of any visual AI agent is the ability to see what the user sees.

In this step, you will build a popup with a toggle button and wire the background service worker to capture a screenshot of the active tab every 30 seconds. By the end, you will have raw image data flowing through the extension on a timer.

In this step, get ready to:
  • Build a popup UI with a Start/Stop Monitoring toggle button.
  • Implement periodic screenshot capture in the background service worker.
  • Verify screenshots are captured and logged to the service worker console.
Build the popup UI with a start/stop toggle

The popup is the small panel that appears when you click the extension icon in Chrome's toolbar. You need a button that lets you start and stop monitoring.

  • In VS Code, open popup.html inside your visual-ai-agent folder.
  • Replace the placeholder comment with the following HTML:
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <style>
    /* Set a fixed popup width and clean font */
    body { width: 320px; padding: 16px; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; margin: 0; }
    h1 { font-size: 16px; margin: 0 0 12px 0; }
    .status { font-size: 12px; color: #666; margin-bottom: 12px; }
    /* Green text when monitoring is active */
    .status.active { color: #16a34a; }
    button { width: 100%; padding: 10px; border: none; border-radius: 6px; font-size: 14px; cursor: pointer; font-weight: 500; }
    /* Blue for start, red for stop */
    .btn-start { background: #2563eb; color: white; }
    .btn-stop { background: #dc2626; color: white; }
  </style>
</head>
<body>
  <h1>Visual AI Agent</h1>
  <p class="status" id="status">Status: Inactive</p>
  <button id="toggle-btn" class="btn-start">Start Monitoring</button>
  <script src="popup.js"></script>
</body>
</html>

What does this code do?

  • The inline CSS sets a fixed width of 320px so the popup looks tidy in Chrome's toolbar dropdown.
  • Two CSS classes, .btn-start (blue) and .btn-stop (red), let you swap the button appearance based on state.
  • The id="toggle-btn" and id="status" elements are what your JavaScript will control.
  • Save popup.html.
  • Switch back to chrome://extensions and click the reload icon on your extension card.
  • Click the extension icon in Chrome's toolbar to open the popup.

You should see a small panel with a heading, a grey "Status: Inactive" label, and a blue "Start Monitoring" button.

Don't see the popup?

  • Make sure you saved popup.html and clicked the reload icon on the extension card.
  • If the extension shows errors on chrome://extensions, click Errors to check for typos in popup.html.
  • If you do not see the extension icon, click the puzzle-piece icon in the toolbar and pin Visual AI Agent.

Help me debug my Chrome extension popup not appearing.

The button looks right, but it does not do anything yet. Next, you will wire it up with JavaScript so it can communicate with the service worker.

  • Open popup.js and replace the placeholder with the following code:
// Cache references to the toggle button and status display
const toggleBtn = document.getElementById("toggle-btn");
const statusEl = document.getElementById("status");

// On popup open, check current monitoring state
chrome.runtime.sendMessage({ type: "get-status" }, (response) => {
  if (response && response.isMonitoring) { setActiveState(); }
});

// Toggle monitoring when button is clicked
toggleBtn.addEventListener("click", () => {
  chrome.runtime.sendMessage({ type: "toggle-monitoring" }, (response) => {
    if (response && response.isMonitoring) { setActiveState(); } else { setInactiveState(); }
  });
});

// Switch UI to active state
function setActiveState() {
  statusEl.textContent = "Status: Monitoring";
  statusEl.classList.add("active");
  toggleBtn.textContent = "Stop Monitoring";
  toggleBtn.className = "btn-stop";
}
// Switch UI to inactive state
function setInactiveState() {
  statusEl.textContent = "Status: Inactive";
  statusEl.classList.remove("active");
  toggleBtn.textContent = "Start Monitoring";
  toggleBtn.className = "btn-start";
}

What does this code do?

  • chrome.runtime.sendMessage sends a message from the popup to the background service worker.
  • On popup open, it sends get-status so the button shows the correct state if monitoring was already started.
  • Clicking the button sends toggle-monitoring and updates the UI based on the response.
  • setActiveState and setInactiveState swap the button text, colour, and status label.
  • Save popup.js.
  • Reload the extension on chrome://extensions.
  • Click the extension icon to open the popup, then click Start Monitoring.

The button text changes to "Stop Monitoring" and turns red, and the status reads "Status: Monitoring". Clicking again toggles it back.

Why does the toggle reset if I close and reopen the popup?

The popup sends the message, but the background service worker still has placeholder code. It does not know how to handle the message yet, so no state is persisted. You will fix that next.

Button does not toggle?

  • Right-click the popup panel and select Inspect to open DevTools for the popup. Check the Console for errors.
  • Confirm you saved both popup.html and popup.js before reloading the extension.
  • Make sure the id values in popup.html match the getElementById strings in popup.js exactly.

Help me fix my Chrome extension popup button not responding to clicks.

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before moving on.

ⓧ I'd like to double check the full code

Here is the full popup.html file:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <style>
    body { width: 320px; padding: 16px; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; margin: 0; }
    h1 { font-size: 16px; margin: 0 0 12px 0; }
    .status { font-size: 12px; color: #666; margin-bottom: 12px; }
    .status.active { color: #16a34a; }
    button { width: 100%; padding: 10px; border: none; border-radius: 6px; font-size: 14px; cursor: pointer; font-weight: 500; }
    .btn-start { background: #2563eb; color: white; }
    .btn-stop { background: #dc2626; color: white; }
  </style>
</head>
<body>
  <h1>Visual AI Agent</h1>
  <p class="status" id="status">Status: Inactive</p>
  <button id="toggle-btn" class="btn-start">Start Monitoring</button>
  <script src="popup.js"></script>
</body>
</html>

And the full popup.js file:

const toggleBtn = document.getElementById("toggle-btn");
const statusEl = document.getElementById("status");

chrome.runtime.sendMessage({ type: "get-status" }, (response) => {
  if (response && response.isMonitoring) { setActiveState(); }
});

toggleBtn.addEventListener("click", () => {
  chrome.runtime.sendMessage({ type: "toggle-monitoring" }, (response) => {
    if (response && response.isMonitoring) { setActiveState(); } else { setInactiveState(); }
  });
});

function setActiveState() {
  statusEl.textContent = "Status: Monitoring";
  statusEl.classList.add("active");
  toggleBtn.textContent = "Stop Monitoring";
  toggleBtn.className = "btn-stop";
}

function setInactiveState() {
  statusEl.textContent = "Status: Inactive";
  statusEl.classList.remove("active");
  toggleBtn.textContent = "Start Monitoring";
  toggleBtn.className = "btn-start";
}
Implement periodic screenshot capture in the service worker

The popup can now send messages, but the service worker needs to receive them and act. You will make it start a repeating Chrome alarm and capture a screenshot on each tick.

  • Open background.js and replace the placeholder with the following message-handling code:
// Configuration and state
const SUPABASE_URL = "https://your-project-ref.supabase.co";
const SUPABASE_KEY = "your-anon-key-here";
const OLLAMA_URL = "http://localhost:11434/api/generate";
const CAPTURE_INTERVAL_MINUTES = 0.5;
let isMonitoring = false;
let eventBuffer = [];

// Listen for toggle and status messages from the popup
chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
  if (message.type === "toggle-monitoring") {
    isMonitoring = !isMonitoring;
    if (isMonitoring) {
      chrome.alarms.create("capture-cycle", {
        delayInMinutes: CAPTURE_INTERVAL_MINUTES,
        periodInMinutes: CAPTURE_INTERVAL_MINUTES
      });
      console.log("Monitoring started");
    } else {
      chrome.alarms.clear("capture-cycle");
      eventBuffer = [];
      console.log("Monitoring stopped");
    }
    sendResponse({ isMonitoring });
    return true;
  }
  if (message.type === "get-status") {
    sendResponse({ isMonitoring });
    return true;
  }
});

What does this code do?

  • The four constants at the top store your Supabase URL, API key, Ollama endpoint, and the capture interval. You will use Supabase and Ollama in later steps.
  • chrome.runtime.onMessage.addListener receives messages from the popup.
  • When toggle-monitoring arrives, it flips isMonitoring and creates or clears a repeating alarm named capture-cycle.
  • chrome.alarms.create sets a timer that fires every 0.5 minutes (30 seconds). This is the minimum interval for Manifest V3 alarms.
  • Replace the placeholder Supabase values with your own Project URL and anon key from Step 1.
  • Save background.js.
  • Reload the extension on chrome://extensions.
  • Open the service worker console by clicking the blue link next to Inspect views on your extension card.
  • Open the popup and click Start Monitoring.

In the service worker DevTools console, you should see the message Monitoring started appear.

Don't see "Monitoring started" in the console?

  • Make sure you are looking at the service worker console (opened via the Inspect views link), not the regular page console.
  • Check that the toggle-monitoring string matches exactly in both popup.js and background.js.
  • Confirm you reloaded the extension after saving background.js.

Help me debug why my service worker is not receiving messages from the popup.

The service worker now knows when to start and stop. The final piece is capturing the actual screenshot when the alarm fires.

  • In background.js, add the following alarm handler below the message listener you just wrote:
// Capture a screenshot on each alarm tick
chrome.alarms.onAlarm.addListener(async (alarm) => {
  if (alarm.name !== "capture-cycle") return;
  try {
    // Find the active tab in the current window
    const [tab] = await chrome.tabs.query({ active: true, currentWindow: true });
    if (!tab || !tab.url || tab.url.startsWith("chrome://")) return;
    // Capture the visible area as a PNG data URL
    const screenshotDataUrl = await chrome.tabs.captureVisibleTab(null, { format: "png" });
    console.log("Screenshot captured:", screenshotDataUrl.slice(0, 80) + "...");
  } catch (error) {
    console.error("Capture cycle error:", error);
  }
});

What does this code do?

  • chrome.alarms.onAlarm.addListener fires every time an alarm triggers. The guard clause checks the alarm name is capture-cycle before proceeding.
  • chrome.tabs.query finds the currently active tab. The code skips chrome:// pages because Chrome blocks screenshots of internal pages.
  • chrome.tabs.captureVisibleTab takes a screenshot and returns a Base64-encoded data URL. The log slices to 80 characters so it does not flood the console.
  • Save background.js.
  • Reload the extension on chrome://extensions.
  • Open the service worker console again via the Inspect views link.
  • Navigate to any website (for example, https://github.com).
  • Open the popup and click Start Monitoring.

Before you check the console, what do you think will appear after 30 seconds?

Wait 30 seconds. In the service worker console you should see a log line starting with Screenshot captured: data:image/png;base64,... confirming the extension grabbed an image of your active tab.

No screenshot log after 30 seconds?

  • Make sure you are on a regular webpage, not chrome://extensions or chrome://newtab. Chrome blocks screenshots of internal pages.
  • Confirm the alarm handler code is placed below (outside) the chrome.runtime.onMessage.addListener block, not inside it.
  • Check that your manifest.json includes "alarms" in the permissions array.
  • If you see a permissions error, ensure host_permissions contains "<all_urls>".

Help me debug why my Chrome extension alarm is not firing or not capturing screenshots.

Keep the service worker DevTools open

Opening DevTools via the Inspect views link keeps the service worker alive. If you close it, Chrome may terminate the worker after 30 seconds of inactivity. For testing, keep it open.

✔️ Awesome, I've got everything!

Great. Make sure background.js is saved with both the message listener and the alarm handler.

ⓧ I'd like to double check the full code

Your background.js should contain two main sections in this order. Scroll up and confirm each block is present:

  • The configuration constants (SUPABASE_URL, SUPABASE_KEY, OLLAMA_URL, CAPTURE_INTERVAL_MINUTES) and state variables (isMonitoring, eventBuffer).
  • The chrome.runtime.onMessage.addListener block handling toggle-monitoring and get-status messages.
  • The chrome.alarms.onAlarm.addListener block that queries the active tab and calls chrome.tabs.captureVisibleTab.

Your extension now captures raw screenshot data on a timer. Next up, you will track what the user is actually doing on the page with click and navigation events.

Track User Events with Content Scripts

Your extension can now take screenshots of whatever is on screen. But a screenshot only shows what is visible. It does not tell you what the user did to get there.

Tracking clicks, form inputs, and page navigations gives the full picture of user activity. This is exactly the type of workflow data that companies like Neoflo capture for finance operations. In this step, you will write a content script that listens for user interactions on every page and sends them to the service worker for buffering.

In this step, get ready to:
  • Track click, input, and navigation events from any webpage.
  • Buffer events in the service worker between screenshot captures.
Build the content script event listeners

A content script runs inside every webpage the user visits. It has full access to the page's DOM, so it can listen for clicks, form inputs, and navigation events. Each captured event gets sent to the background service worker via chrome.runtime.sendMessage.

  • In VS Code, open content.js in your visual-ai-agent folder.
  • Delete the placeholder comment and add the click event listener by pasting this code:
document.addEventListener("click", (event) => {
  const target = event.target;
  chrome.runtime.sendMessage({
    type: "user-event",
    data: {
      event_type: "click",
      tag: target.tagName,
      text: target.innerText?.slice(0, 100) || "",
      id: target.id || "",
      class_name: target.className?.toString().slice(0, 100) || "",
      url: window.location.href,
      timestamp: new Date().toISOString()
    }
  });
}, true);

What does this code do?

This attaches a click listener to the entire document using capture mode (true as the third argument). Capture mode ensures you catch clicks even if the page's own JavaScript stops the event from bubbling.

For each click, it grabs the target element's tag name, visible text (capped at 100 characters), ID, class, the current URL, and a timestamp. All of this is bundled into a user-event message sent to the service worker.

  • Save content.js.
  • Go to chrome://extensions and click the reload icon on your Visual AI Agent card.
  • Open any webpage (like https://example.com), then open DevTools with F12 and switch to the Console tab.
  • Click anywhere on the page. You should see a warning mentioning the message port or connection. This proves the content script is running and sending messages.

No warning in the console when you click?

  • Make sure you reloaded the extension after saving. Content scripts only inject into pages opened AFTER the reload.
  • Try closing the tab and opening a fresh one. Pages that were open before the reload do not get the updated content script.
  • Check chrome://extensions for any red error badges on your extension card. A syntax error in content.js prevents injection entirely.

Help me debug why my content script isn't injecting into web pages.

The warning you see is expected for now. The service worker receives the message but does not have a handler for user-event yet. You will fix that in the next substep.

  • Now add the input and navigation listeners below the click listener in content.js:
document.addEventListener("input", (event) => {
  const target = event.target;
  chrome.runtime.sendMessage({
    type: "user-event",
    data: {
      event_type: "input",
      tag: target.tagName,
      input_type: target.type || "text",
      id: target.id || "",
      url: window.location.href,
      timestamp: new Date().toISOString()
    }
  });
}, true);

window.addEventListener("beforeunload", () => {
  chrome.runtime.sendMessage({
    type: "user-event",
    data: {
      event_type: "navigation",
      from_url: window.location.href,
      page_title: document.title,
      timestamp: new Date().toISOString()
    }
  });
});

What do these listeners track?

  • The input listener fires when the user types in a text field, selects a dropdown, or interacts with any form control. It captures the element type and ID without recording the actual value (avoiding sensitive data).
  • The beforeunload listener fires right before the user leaves the page. It records the URL they are navigating away from and the page title, creating a breadcrumb trail of visited pages.
  • Save content.js.
  • Return to chrome://extensions and reload the extension again.
  • Confirm there are no red error badges on the extension card. The content script is now ready to track all three event types.

✔️ Awesome, I've got everything!

Great. Make sure you have saved content.js and reloaded the extension before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm your content.js contains these sections in order:

  • The click event listener (captures tag, text, id, class_name, url, timestamp).
  • The input event listener (captures tag, input_type, id, url, timestamp).
  • The beforeunload event listener on window (captures from_url, page_title, timestamp).
Buffer events in the service worker

The content script is now broadcasting events, but the service worker does not know what to do with them yet. You need to catch these messages, store them in a buffer, and drain the buffer each time a screenshot fires.

  • Open background.js in VS Code.
  • Find the closing block of the get-status handler. It looks like this:
  if (message.type === "get-status") {
    sendResponse({ isMonitoring });
    return true;
  }
  • Add the following user-event handler directly below that block, still inside the onMessage listener:
  if (message.type === "user-event") {
    eventBuffer.push(message.data);
    return false;
  }

What does this handler do?

Every time the content script sends a user-event message, this pushes the event data into the eventBuffer array that already exists at the top of your file. Returning false tells Chrome that no async response is coming, which prevents the message port warning you saw earlier.

Now update the alarm handler to drain the buffer and bundle events with each screenshot capture.

  • In the chrome.alarms.onAlarm handler, find this line:
    console.log("Screenshot captured:", screenshotDataUrl.slice(0, 80) + "...");
  • Replace that single line with these three lines:
    const base64Image = screenshotDataUrl.replace(/^data:image\/png;base64,/, "");
    const events = [...eventBuffer];
    eventBuffer = [];
    console.log("Screenshot captured, events buffered:", events.length);

What changed in the alarm handler?

  • base64Image strips the data:image/png;base64, prefix from the screenshot. Ollama expects raw base64 with no data URI prefix.
  • events copies the current buffer into a local variable, then immediately empties eventBuffer. This prevents the same events from being counted in the next capture cycle.
  • The updated console log now shows how many user events were buffered since the last capture.
  • Save background.js.
  • Go to chrome://extensions and reload the extension.

Before you test, what do you think you will see in the service worker console after clicking around on a page for 30 seconds?

  • On the chrome://extensions page, click the service worker link on your extension card to open its DevTools console.
  • Open the extension popup and click Start Monitoring.
  • Browse to any website and click a few links and buttons.
  • Wait 30 seconds for the alarm to fire.

In the service worker console, you should see a log like Screenshot captured, events buffered: 3 (the number depends on how many clicks you made). This confirms the full event pipeline is working.

Events buffered shows 0?

  • Make sure you opened the website in a new tab AFTER reloading the extension. Existing tabs do not receive the updated content script.
  • Check that you are browsing a regular webpage, not a chrome:// page. Chrome does not inject content scripts into internal pages.
  • Open the webpage's own DevTools console (not the service worker's) and look for errors mentioning content.js. A syntax error there prevents event tracking.

Help me figure out why my content script events are not reaching the service worker.

✔️ Awesome, I've got everything!

Great. Make sure both content.js and background.js are saved before moving on.

ⓧ I'd like to double check the full code

Your content.js should contain these three event listeners in order: the click listener, the input listener, and the beforeunload listener. Scroll up and confirm each matches.

For background.js, confirm your onMessage listener has three if blocks: toggle-monitoring, get-status, and user-event. The alarm handler should now include base64Image, events, and the eventBuffer = [] reset.

Your agent can now see the screen AND understand what the user is doing on it. Next up, you will connect a local AI model to turn those raw screenshots into natural language descriptions.

Analyze Screenshots with Local AI

Your extension captures screenshots and tracks user events. That is raw data. Without interpretation, a base64 image and a list of click coordinates tell you nothing about what the user is actually doing.

In this step, you will connect your extension to Ollama's local vision model. The model transforms each screenshot into a plain-English description like "User is reading a GitHub README about React hooks." This is the intelligence layer that makes your agent genuinely useful.

In this step, get ready to:
  • Build a function that sends screenshots to Ollama's vision API and returns an AI description.
  • Wire the AI analysis into your existing capture loop so every screenshot gets described automatically.
  • Verify the full pipeline end-to-end by seeing an AI-generated description in the console.
Send screenshots to Ollama for analysis

Ollama exposes a REST API at http://localhost:11434/api/generate. You send it a JSON payload with the model name, a text prompt, and an images array containing raw Base64 strings. It returns a JSON object with a response field containing the generated text.

  • In background.js, scroll to the very bottom of the file (after the closing }); of the alarm listener).
  • Add the analyzeScreenshot function by pasting this code:
async function analyzeScreenshot(base64Image) {
  try {
    // Send the screenshot to Ollama's vision model
    const response = await fetch(OLLAMA_URL, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        model: "moondream2",
        prompt: "Describe what the user is doing on this webpage in one sentence. Focus on the main activity visible on screen.",
        images: [base64Image],
        stream: false
      })
    });

    if (!response.ok) {
      throw new Error(`Ollama returned ${response.status}`);
    }

    // Extract the AI's description from the response
    const data = await response.json();
    return data.response || "Unable to analyze screenshot";
  } catch (error) {
    console.error("AI analysis error:", error);
    return "AI analysis unavailable - is Ollama running?";
  }
}

What does this code do?

The function takes a raw base64 image string and POSTs it to your locally running Ollama server. The images array expects raw base64 with no data:image/png;base64, prefix. Your alarm handler already strips that prefix, so the value passed here is ready to go.

The stream: false flag tells Ollama to return the complete response in one JSON object instead of streaming tokens. If anything fails (Ollama is not running, model is not loaded), the catch block returns a human-readable fallback string instead of crashing the monitoring loop.

  • Save background.js.
Wire AI analysis into the capture loop

The function exists but nothing calls it yet. You need to plug it into your alarm handler so every screenshot capture triggers an AI analysis.

  • In background.js, find this line inside the chrome.alarms.onAlarm listener:
    console.log("Screenshot captured, events buffered:", events.length);
  • Replace that entire line with these two lines:
    const aiDescription = await analyzeScreenshot(base64Image);
    console.log("Activity:", { url: tab.url, title: tab.title, events, aiDescription });

What changed?

Instead of just logging that a screenshot was captured, the alarm handler now sends the image to Ollama and waits for a description. It then logs a complete activity record containing the page URL, page title, buffered user events, and the AI's one-sentence description of what is on screen.

  • Save background.js.
  • Return to chrome://extensions and click the reload icon on your Visual AI Agent extension to pick up the changes.

Reload after every edit

Every time you edit background.js, you must reload the extension from chrome://extensions for the service worker to pick up your changes.

Before you test this, what do you think the console will show after 30 seconds of monitoring? Will Ollama return a useful description, or will it fail?

  • Open the service worker console by clicking the Service worker link on your extension's card in chrome://extensions.
  • Navigate to any website (try a news site or GitHub).
  • Click the extension icon and press Start Monitoring.
  • Wait 30 seconds for the alarm to fire.

In the service worker console, you should see a log line starting with Activity: followed by an object containing your page's URL, title, any events you triggered, and an aiDescription field with a sentence like "The user is viewing a list of trending repositories on GitHub."

Why does the first response take a while?

The first time Ollama processes an image after startup, it loads the model into GPU memory. This can take 5-10 seconds. Subsequent requests are much faster because the model stays loaded.

Seeing "AI analysis unavailable" in the log?

This means the fetch to Ollama failed. Check that Ollama is still running in the background. Open a terminal and run ollama list to confirm moondream2 is available. If Ollama closed, restart it and try again.

If you see a CORS error in the console, make sure your host_permissions in manifest.json includes <all_urls>.

Help me debug why Ollama isn't responding to my extension

✔️ Awesome, I've got everything!

Great. Double check you have saved background.js and your extension is reloaded.

ⓧ I'd like to double check the full code

Your background.js is now over 30 lines, so here is a checklist. Scroll through your file and confirm these sections exist in order:

  • Constants at the top: SUPABASE_URL, SUPABASE_KEY, OLLAMA_URL, CAPTURE_INTERVAL_MINUTES.
  • State variables: isMonitoring and eventBuffer.
  • The chrome.runtime.onMessage listener handling toggle-monitoring, get-status, and user-event.
  • The chrome.alarms.onAlarm listener that captures a screenshot, clears eventBuffer, calls analyzeScreenshot, and logs the complete activity object.
  • The analyzeScreenshot function at the bottom, POSTing to OLLAMA_URL with the model, prompt, images, and stream: false.

Your agent now sees and understands what is on screen. Next up, you will store these activity records in Supabase so nothing is lost when the browser closes.

Store Activity Logs in Supabase

Your agent can now capture screenshots and generate AI descriptions of what you're doing. But all that data lives in memory. The moment you close your browser, everything disappears.

In this step, you'll wire the capture pipeline into Supabase so every activity record persists in a real PostgreSQL database. This is exactly the type of structured, queryable data that Neoflo delivers to enterprise clients.

In this step, get ready to:
  • Create a storage function that POSTs activity records to Supabase via its REST API.
  • Connect the storage function to your capture pipeline so every cycle writes to the database.
  • Verify rows appearing in the Supabase Table Editor with timestamps, URLs, and AI descriptions.
Create the Supabase storage function

The Supabase REST API (powered by PostgREST) accepts standard HTTP requests. You'll create a function that sends a POST request with your activity data, authenticated using your project's anon key.

  • In background.js, scroll to the bottom of the file (below the analyzeScreenshot function).
  • Add the following storage function:
async function storeActivity(record) {
  try {
    const response = await fetch(`${SUPABASE_URL}/rest/v1/activities`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "apikey": SUPABASE_KEY,
        "Authorization": `Bearer ${SUPABASE_KEY}`,
        "Prefer": "return=minimal"
      },
      body: JSON.stringify(record)
    });
    if (!response.ok) throw new Error(`Supabase returned ${response.status}`);
  } catch (error) {
    console.error("Storage error:", error);
  }
}

What does this code do?

  • fetch() sends a POST request to your Supabase project's /rest/v1/activities endpoint, which maps directly to your activities table.
  • The apikey and Authorization headers both carry your anon key. PostgREST requires both to authenticate the request.
  • The Prefer: return=minimal header tells Supabase not to return the inserted row, avoiding the need for a SELECT policy.
  • The try/catch ensures that a failed database write logs the error but does not crash the monitoring loop.
  • Save background.js.
  • Go to chrome://extensions and click the reload icon on your Visual AI Agent card.
  • Confirm no errors appear on the extension card. The function is loaded and ready to be called.

Seeing an error on the extension card?

  • Check for missing backticks in the template literal around ${SUPABASE_URL}. The fetch URL must use backticks (` `) not regular quotes.
  • Make sure the function is placed outside the chrome.alarms.onAlarm.addListener block, not inside it.

Help me debug my storeActivity function in background.js

Connect storage to the capture pipeline

The storage function exists, but nothing calls it yet. You need to replace the temporary console log in the alarm handler with a real call to storeActivity.

  • In background.js, scroll to the chrome.alarms.onAlarm.addListener block.
  • Find this line near the bottom of the try block:
    console.log("Activity:", { url: tab.url, title: tab.title, events, aiDescription });
  • Delete that line and replace it with the following code:
    await storeActivity({
      url: tab.url,
      page_title: tab.title || "Unknown",
      event_type: "capture",
      event_details: JSON.stringify(events.slice(-10)),
      ai_description: aiDescription
    });

    console.log("Activity stored:", aiDescription);

What does this code do?

  • storeActivity receives an object whose keys match your Supabase table columns: url, page_title, event_type, event_details, and ai_description.
  • events.slice(-10) keeps only the last 10 events to avoid storing enormous payloads when the user clicks rapidly.
  • JSON.stringify converts the events array into a text string, since the event_details column is a text type.
  • The await keyword ensures the write completes before logging, but the outer try/catch still protects the loop if it fails.
  • Save background.js.
  • Go to chrome://extensions and click the reload icon on your Visual AI Agent card.

Now test the full pipeline end-to-end.

  • Make sure Ollama is running in a terminal (if you closed it, open a terminal and run ollama serve).
  • Click the Visual AI Agent extension icon and click Start Monitoring.
  • Browse to 2-3 different websites (e.g. GitHub, Wikipedia, a news site). Click a few links on each page.
  • Wait at least 60 seconds so at least two capture cycles fire.

Before you check Supabase, think about what you expect to see. How many rows should be in the table? What should the ai_description column contain?

  • Open your Supabase dashboard and navigate to the Table Editor in the left sidebar.
  • Click the activities table.

You should see new rows with timestamps in created_at, the URLs you visited, AI-generated descriptions in ai_description, and JSON arrays of your click events in event_details.

No rows appearing in the table?

  • Open the service worker console: on chrome://extensions, click Inspect views: service worker under your extension. Look for "Storage error" messages.
  • Confirm your SUPABASE_URL and SUPABASE_KEY constants at the top of background.js match the values from your Supabase project's Settings > API page.
  • Make sure RLS is disabled on the activities table. If enabled, the anon key cannot insert rows.
  • If you see "AI analysis unavailable" in the ai_description column, Ollama is not running. Start it with ollama serve in a terminal.

Help me debug why my Chrome extension isn't writing to Supabase

✔️ Awesome, I've got everything!

Great. Double check you've saved background.js and that rows are appearing in your Supabase Table Editor.

ⓧ I'd like to double check the full code

Your background.js should now contain these sections in order. Scroll through your file and confirm each is present:

  • Four constants at the top: SUPABASE_URL, SUPABASE_KEY, OLLAMA_URL, CAPTURE_INTERVAL_MINUTES.
  • Two state variables: isMonitoring and eventBuffer.
  • The chrome.runtime.onMessage.addListener block handling toggle-monitoring, get-status, and user-event.
  • The chrome.alarms.onAlarm.addListener block that captures, analyzes, and now calls storeActivity with the activity record.
  • The analyzeScreenshot function that calls Ollama.
  • The new storeActivity function at the bottom that POSTs to Supabase.

Your Visual AI Agent now has a complete data pipeline. It captures what you see, understands it through AI, and persists everything to a real database. Next up, you'll push this project to GitHub with a clean commit history.

Secret mission

Build a History Dashboard in the Popup

Your agent stores activity data, but you can't see any of it without opening the Supabase dashboard. Add a history view directly in the extension popup that shows your last 10 activities with timestamps, page titles, and AI descriptions.

Clean Up Your Resources

Clean Up Your Resources

This project runs on a free Supabase tier and a local AI model, so there are no ongoing costs while resources are within free limits. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Supabase project (free tier, 500MB database). No cost while within free limits.
  • Ollama with moondream2 model files stored locally (~1.7GB disk space).
  • Chrome extension loaded in Developer Mode (the visual-ai-agent folder).

Keep everything running

No action needed. Choose this if you are still actively building or want to keep experimenting with the extension.

  • The extension only monitors when you click Start Monitoring. It does nothing in the background otherwise.
  • Supabase free tier has no ongoing charges. Your project stays active as long as you use it at least once a week.
  • Ollama runs only when you explicitly call it. The model files sit on disk but consume no resources when idle.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and database stay intact for when you return.

  • In Chrome, go to chrome://extensions and toggle off the Visual AI Agent extension.
  • Supabase free projects pause automatically after 1 week of inactivity. No action needed on your part.
  • Close any Ollama processes. The model files remain on disk but do not use memory when idle.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is permanent.

  • Remove the Chrome extension. Go to chrome://extensions, find Visual AI Agent, and click Remove.
  • Delete the project folder. Open your terminal and run:
cd ~\Desktop
Remove-Item -Recurse -Force visual-ai-agent
  • Remove the moondream2 model to free ~1.7GB of disk space by running:
ollama rm moondream2
  • Delete your Supabase project. In the Supabase dashboard, open your project, go to Settings > General, scroll to the Delete project section, and click Delete Project. Enter your project name to confirm.

Nice Work!

Nice Work!

You just built a Visual AI Agent Chrome extension that monitors your browser, analyzes screenshots with local AI, and stores structured activity logs in a cloud database.

You've learned how to:

  • Build a Chrome Extension using Manifest V3 with a service worker, content scripts, and popup UI that captures periodic screenshots using chrome.tabs.captureVisibleTab and chrome.alarms.
  • Integrate a local vision AI model (moondream2 via Ollama) to generate natural language descriptions of screen activity and track user interactions through injected content scripts.
  • Store structured activity logs in a cloud PostgreSQL database (Supabase) via its REST API, creating a queryable record of browser activity with timestamps, URLs, events, and AI descriptions.
  • Secret Mission: Built a queryable history dashboard in the extension popup showing the last 10 activities.

Ready to quiz yourself?