Background Job Queue with WebSockets

Build a job queue system with real-time progress tracking via WebSockets.

Introduction

30 Second Summary

Think about when you order something online. You get a confirmation immediately, but the actual processing (packing, shipping, tracking) happens in the background without you staring at a loading screen.

In this project, you will build a background job queue system that processes tasks asynchronously and streams real-time progress updates to a live dashboard.

What You'll Build

A full-stack job processing system with an Express REST API, a custom job queue backed by SQLite, WebSocket-powered real-time updates, and a React monitoring dashboard.

By the end of this project, you'll have:

  • A custom job queue engine with concurrency control, retry logic, and exponential backoff.
  • A REST API and WebSocket server that accept job submissions and push live progress updates to connected clients.
  • A React dashboard with Tailwind CSS showing real-time job status, progress bars, and queue metrics.
  • Secret Mission: Add job prioritization so high-priority jobs jump the queue, and implement a dead-letter queue for jobs that exhaust all retries.

Are there any prerequisites?

This project assumes basic familiarity with React and Tailwind CSS for the dashboard portion. You should also be comfortable with JavaScript fundamentals like functions, arrays, and async/await.

Set Up the Project

Before you can build a job queue, you need a project structure that keeps the backend and frontend separate.

In this step, you will create a root folder with two subdirectories. server holds the Node.js backend, and client holds the React dashboard.

In this step, get ready to:
  • Create the project folder structure with separate server and client directories.
  • Initialize the server and install its backend dependencies.
  • Scaffold the React client with Vite and install Tailwind CSS.
Create the project folder structure

You need a root folder that contains both the server and client as siblings. This makes it easy to run each part independently.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir job-queue
cd job-queue
  • Create the server subdirectory by running:
mkdir server
  • Confirm the folder exists by running:
ls

You should see server listed in the output.

Don't see the server folder?

Make sure you ran mkdir server from inside the job-queue folder. Run cd with no arguments to confirm your current location ends with job-queue.

help me figure out why my project folder structure isn't right

Initialize the server and install dependencies

The server needs its own package.json configured for ES modules. You will install five packages: Express for the REST API, ws for WebSockets, better-sqlite3 for the database, cors for cross-origin requests, and uuid for generating unique job IDs.

  • Move into the server directory by running:
cd server
  • Initialize a new Node.js project by running:
npm init -y

You should see a confirmation message showing the new package.json contents printed in your terminal.

  • Open the job-queue folder in VS Code so you can edit files.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Select the job-queue folder on your Desktop, then click Select Folder.
  • Open server/package.json in the Explorer sidebar.
  • Replace the entire contents of the file with the following:
{
  "name": "job-queue-server",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "dev": "node --watch server.js"
  }
}

What does this configuration do?

"type": "module" enables ES module syntax (import/export) instead of the older CommonJS require style.

The dev script uses node --watch to automatically restart the server whenever you save changes to your code.

  • Save server/package.json by pressing Ctrl+S.
  • Install the server dependencies by running this in your terminal (make sure you are still inside the server folder):
npm install express cors uuid ws better-sqlite3

You should see output confirming packages were added. A node_modules folder and package-lock.json file will appear inside server.

Seeing a build error for better-sqlite3?

better-sqlite3 requires native compilation and ships prebuilt binaries for Node.js LTS versions. Make sure you are running Node.js 20.x or 22.x by checking node --version.

If you see errors about missing build tools on Windows, you may need to install Windows Build Tools. help me fix the better-sqlite3 installation error on Windows

  • Open server/package.json again in VS Code to confirm a dependencies section now lists all five packages.

✔️ Awesome, I've got everything!

Great. Make sure you have saved server/package.json.

ⓧ I'd like to double check the full code

Your server/package.json should look like this:

{
  "name": "job-queue-server",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "dev": "node --watch server.js"
  },
  "dependencies": {
    "better-sqlite3": "^12.10.0",
    "cors": "^2.8.5",
    "express": "^4.21.2",
    "uuid": "^11.1.0",
    "ws": "^8.18.0"
  }
}
Scaffold the React client with Vite and install Tailwind CSS

Vite is a fast build tool that scaffolds a React project with hot reloading out of the box. Tailwind CSS gives you utility classes for styling without writing custom CSS files.

  • Move back up to the job-queue root folder by running:
cd ..
  • Scaffold a new React project in a client folder by running:
npm create vite@latest client -- --template react

What does this command do?

npm create vite@latest runs the Vite project scaffolder. client is the folder name it creates. -- --template react tells Vite to use the plain React JavaScript template, skipping interactive prompts.

Still seeing interactive prompts?

PowerShell sometimes consumes the -- separator before it reaches Vite. If you see a framework selection menu, try using three dashes instead:

npm create vite@latest client --- --template react

Alternatively, open Command Prompt (cmd) instead of PowerShell and run the original command there. help me scaffold Vite on Windows without interactive prompts

You should see a message that says "Done. Now run:" followed by instructions to change into the client folder.

  • Move into the client folder and install its base dependencies by running:
cd client
npm install
  • Install Tailwind CSS and its Vite plugin by running:
npm install tailwindcss @tailwindcss/vite

Why two packages?

tailwindcss is the CSS framework itself. @tailwindcss/vite is the official Vite plugin that integrates Tailwind into Vite's build pipeline so your utility classes are compiled automatically.

  • Open client/vite.config.js in VS Code.
  • Replace its entire contents with the following to register both plugins:
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
import tailwindcss from '@tailwindcss/vite'

// Register the React and Tailwind CSS plugins
export default defineConfig({
  plugins: [
    react(),
    tailwindcss(),
  ],
})

What does this file do?

This Vite configuration registers two plugins. react() enables JSX support and fast refresh. tailwindcss() scans your components for Tailwind classes and generates the matching CSS at build time.

  • Save client/vite.config.js.
  • Open client/src/index.css in VS Code.
  • Delete everything in the file and replace it with this single line:
@import "tailwindcss";

What does this import do?

In Tailwind CSS v4, @import "tailwindcss" replaces the older @tailwind directives. This single import pulls in all of Tailwind's utility classes, reset styles, and component layers.

  • Save client/src/index.css.
  • Open client/index.html in VS Code.
  • Replace the entire file contents with the following:
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Job Queue Dashboard</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.jsx"></script>
  </body>
</html>

What changed from the default?

You removed the default Vite favicon link and updated the <title> to Job Queue Dashboard. This gives your dashboard a proper browser tab name.

  • Save client/index.html.
  • Confirm the client is working by running the dev server:
npm run dev

You should see output indicating the dev server is running, typically at http://localhost:5173.

  • Stop the dev server by pressing Ctrl+C in your terminal.

Dev server won't start?

Make sure you ran npm install inside the client folder before npm run dev. Also confirm your terminal is currently inside client (not the root job-queue folder).

help me troubleshoot the Vite dev server not starting

✔️ Awesome, I've got everything!

Great. Make sure all files are saved and you have stopped the dev server.

ⓧ I'd like to double check the full code

Your client/src/index.css should be:

@import "tailwindcss";

Your client/index.html should be:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Job Queue Dashboard</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.jsx"></script>
  </body>
</html>

Your client/src/main.jsx should be:

import { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import './index.css';
import App from './App.jsx';

createRoot(document.getElementById('root')).render(
  <StrictMode>
    <App />
  </StrictMode>
);

Your project skeleton is ready. Next up, you will build the job queue engine that stores, retrieves, and manages jobs in SQLite.

Build the Job Queue Engine

Your project structure is ready with both the server and client directories in place. Now it's time to build the brain of the system.

Every job queue needs three things: a place to store jobs, functions to manage their lifecycle, and workers that process them. In this step, you'll build all three as separate modules that work together.

In this step, get ready to:
  • Create a SQLite database with a jobs table to persist queue state.
  • Build queue functions that manage the full job lifecycle.
  • Implement a worker with concurrency control and exponential backoff retry.
Create the queue data structure with job states in SQLite

A job queue needs persistent storage so that jobs survive a server restart. You'll use SQLite through the better-sqlite3 package to store every job's state, progress, and retry history in a single local file.

  • In VS Code, create a new file called database.js inside the server folder.
  • Add the following code to initialize the database and create the jobs table:
import Database from 'better-sqlite3';
import path from 'path';
import { fileURLToPath } from 'url';

const __dirname = path.dirname(fileURLToPath(import.meta.url));
const db = new Database(path.join(__dirname, 'jobs.db'));

// Enable WAL mode for better concurrent read performance
db.pragma('journal_mode = WAL');

// Create the jobs table if it doesn't exist
db.exec(`
  CREATE TABLE IF NOT EXISTS jobs (
    id TEXT PRIMARY KEY,
    type TEXT NOT NULL,
    payload TEXT,
    status TEXT NOT NULL DEFAULT 'waiting',
    priority INTEGER NOT NULL DEFAULT 3,
    progress INTEGER NOT NULL DEFAULT 0,
    result TEXT,
    error TEXT,
    attempts INTEGER NOT NULL DEFAULT 0,
    max_retries INTEGER NOT NULL DEFAULT 3,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    started_at TEXT,
    completed_at TEXT
  )
`);

export default db;

What does this code do?

This module sets up a SQLite database file called jobs.db in your server folder. It uses fileURLToPath to resolve the file path correctly in ESM modules (since __dirname isn't available by default in ES modules).

WAL (Write-Ahead Logging) mode lets multiple readers access the database simultaneously without blocking. The jobs table tracks each job through its full lifecycle with columns for status, progress, error messages, and timestamps.

  • Save database.js.
  • You won't see a visible change yet. The next file uses this module to manage jobs.

Now you need functions to add jobs to the queue, pull the next one out, and update their status as they progress. These are the core operations every queue system needs.

  • Create a new file called queue.js inside the server folder.
  • Add the import and basic CRUD functions by pasting this code:
import db from './database.js';

// Add a new job to the queue
export function enqueueJob(job) {
  const stmt = db.prepare(`
    INSERT INTO jobs (id, type, payload, status, priority, max_retries)
    VALUES (?, ?, ?, 'waiting', ?, ?)
  `);
  stmt.run(job.id, job.type, JSON.stringify(job.payload), job.priority || 3, job.maxRetries || 3);
  return getJob(job.id);
}

// Pull the next waiting job (highest priority first, then oldest)
export function getNextJob() {
  const stmt = db.prepare(`
    SELECT * FROM jobs
    WHERE status = 'waiting'
    ORDER BY priority DESC, created_at ASC
    LIMIT 1
  `);
  return stmt.get();
}

// Retrieve a single job by ID
export function getJob(id) {
  const stmt = db.prepare('SELECT * FROM jobs WHERE id = ?');
  return stmt.get(id);
}

// Retrieve all jobs, newest first
export function getAllJobs() {
  const stmt = db.prepare('SELECT * FROM jobs ORDER BY created_at DESC');
  return stmt.all();
}

What does this code do?

These four functions handle the basic read/write operations for your queue:

  • enqueueJob inserts a new job with status waiting and returns the full job record.
  • getNextJob picks the highest-priority waiting job (priority DESC), breaking ties by age (created_at ASC).
  • getJob and getAllJobs are lookups used by the REST API you'll build in the next step.

Jobs need to transition between states as they move through the queue. The next chunk adds functions that update a job's status, cancel pending jobs, and report queue-wide statistics.

  • In queue.js, add these state management functions below getAllJobs:
// Update a job's status and optional fields (progress, result, error, attempts)
export function updateJobStatus(id, status, extras = {}) {
  const sets = ['status = ?'];
  const values = [status];

  if (extras.progress !== undefined) {
    sets.push('progress = ?');
    values.push(extras.progress);
  }
  if (extras.result !== undefined) {
    sets.push('result = ?');
    values.push(JSON.stringify(extras.result));
  }
  if (extras.error !== undefined) {
    sets.push('error = ?');
    values.push(extras.error);
  }
  if (extras.attempts !== undefined) {
    sets.push('attempts = ?');
    values.push(extras.attempts);
  }
  if (status === 'active') {
    sets.push("started_at = datetime('now')");
  }
  if (status === 'completed' || status === 'failed') {
    sets.push("completed_at = datetime('now')");
  }

  values.push(id);
  const stmt = db.prepare(`UPDATE jobs SET ${sets.join(', ')} WHERE id = ?`);
  stmt.run(...values);
  return getJob(id);
}

// Cancel a job only if it's still waiting
export function cancelJob(id) {
  const job = getJob(id);
  if (!job || job.status !== 'waiting') return null;
  const stmt = db.prepare("UPDATE jobs SET status = 'cancelled' WHERE id = ? AND status = 'waiting'");
  stmt.run(id);
  return getJob(id);
}

// Get aggregate counts by status
export function getStats() {
  const stats = db.prepare(`
    SELECT
      COUNT(*) as total,
      SUM(CASE WHEN status = 'waiting' THEN 1 ELSE 0 END) as waiting,
      SUM(CASE WHEN status = 'active' THEN 1 ELSE 0 END) as active,
      SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END) as completed,
      SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) as failed
    FROM jobs
  `).get();
  return stats;
}

What does this code do?

updateJobStatus dynamically builds a SQL UPDATE statement based on which fields you pass in. This lets the worker update progress, record errors, and track timestamps all through one function.

cancelJob only works on jobs that are still waiting. Once a job starts processing, it's too late to cancel. getStats aggregates job counts by status for the dashboard metrics panel.

  • Save queue.js.
  • Verify both new files exist by opening a terminal in VS Code (Ctrl+`) and running:
ls server/
  • You should see database.js, queue.js, and package.json in the output.

Don't see both files?

Make sure both files are saved inside the server folder, not in the project root. Check the VS Code Explorer sidebar to confirm they appear under server/.

help me find where my files were saved

✔️ Awesome, I've got everything!

Great. Double check you've saved both files before moving on.

ⓧ I'd like to double check the full code

Here is the complete server/database.js:

import Database from 'better-sqlite3';
import path from 'path';
import { fileURLToPath } from 'url';

const __dirname = path.dirname(fileURLToPath(import.meta.url));
const db = new Database(path.join(__dirname, 'jobs.db'));

db.pragma('journal_mode = WAL');

db.exec(`
  CREATE TABLE IF NOT EXISTS jobs (
    id TEXT PRIMARY KEY,
    type TEXT NOT NULL,
    payload TEXT,
    status TEXT NOT NULL DEFAULT 'waiting',
    priority INTEGER NOT NULL DEFAULT 3,
    progress INTEGER NOT NULL DEFAULT 0,
    result TEXT,
    error TEXT,
    attempts INTEGER NOT NULL DEFAULT 0,
    max_retries INTEGER NOT NULL DEFAULT 3,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    started_at TEXT,
    completed_at TEXT
  )
`);

export default db;

Here is the complete server/queue.js:

import db from './database.js';

export function enqueueJob(job) {
  const stmt = db.prepare(`
    INSERT INTO jobs (id, type, payload, status, priority, max_retries)
    VALUES (?, ?, ?, 'waiting', ?, ?)
  `);
  stmt.run(job.id, job.type, JSON.stringify(job.payload), job.priority || 3, job.maxRetries || 3);
  return getJob(job.id);
}

export function getNextJob() {
  const stmt = db.prepare(`
    SELECT * FROM jobs
    WHERE status = 'waiting'
    ORDER BY priority DESC, created_at ASC
    LIMIT 1
  `);
  return stmt.get();
}

export function getJob(id) {
  const stmt = db.prepare('SELECT * FROM jobs WHERE id = ?');
  return stmt.get(id);
}

export function getAllJobs() {
  const stmt = db.prepare('SELECT * FROM jobs ORDER BY created_at DESC');
  return stmt.all();
}

export function updateJobStatus(id, status, extras = {}) {
  const sets = ['status = ?'];
  const values = [status];

  if (extras.progress !== undefined) {
    sets.push('progress = ?');
    values.push(extras.progress);
  }
  if (extras.result !== undefined) {
    sets.push('result = ?');
    values.push(JSON.stringify(extras.result));
  }
  if (extras.error !== undefined) {
    sets.push('error = ?');
    values.push(extras.error);
  }
  if (extras.attempts !== undefined) {
    sets.push('attempts = ?');
    values.push(extras.attempts);
  }
  if (status === 'active') {
    sets.push("started_at = datetime('now')");
  }
  if (status === 'completed' || status === 'failed') {
    sets.push("completed_at = datetime('now')");
  }

  values.push(id);
  const stmt = db.prepare(`UPDATE jobs SET ${sets.join(', ')} WHERE id = ?`);
  stmt.run(...values);
  return getJob(id);
}

export function cancelJob(id) {
  const job = getJob(id);
  if (!job || job.status !== 'waiting') return null;
  const stmt = db.prepare("UPDATE jobs SET status = 'cancelled' WHERE id = ? AND status = 'waiting'");
  stmt.run(id);
  return getJob(id);
}

export function getStats() {
  const stats = db.prepare(`
    SELECT
      COUNT(*) as total,
      SUM(CASE WHEN status = 'waiting' THEN 1 ELSE 0 END) as waiting,
      SUM(CASE WHEN status = 'active' THEN 1 ELSE 0 END) as active,
      SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END) as completed,
      SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) as failed
    FROM jobs
  `).get();
  return stats;
}
Build the worker system with configurable concurrency

The worker is the engine that pulls jobs off the queue and processes them. It needs to limit how many jobs run simultaneously (concurrency control) and report progress as each job runs.

  • Create a new file called worker.js inside the server folder.
  • Add the imports, concurrency configuration, and the simulated work function by pasting this code:
import { getNextJob, updateJobStatus } from './queue.js';

// Maximum number of jobs that can run at the same time
const CONCURRENCY = 3;
let activeJobs = 0;
let onJobUpdate = null;

// Allow external code (like the WebSocket server) to listen for job changes
export function setJobUpdateCallback(callback) {
  onJobUpdate = callback;
}

function notifyUpdate(job) {
  if (onJobUpdate) onJobUpdate(job);
}

// Simulate real work with progress updates over 3-8 seconds
async function simulateWork(job) {
  const duration = 3000 + Math.random() * 5000;
  const steps = 10;
  const stepDuration = duration / steps;

  for (let i = 1; i <= steps; i++) {
    await new Promise(resolve => setTimeout(resolve, stepDuration));

    // Chaos mode: randomly fail halfway through
    if (job.type === 'chaos' && Math.random() < 0.5 && i === Math.floor(steps / 2)) {
      throw new Error('Random chaos failure!');
    }

    const progress = Math.round((i / steps) * 100);
    const updatedJob = updateJobStatus(job.id, 'active', { progress });
    notifyUpdate(updatedJob);
  }

  return { message: `Job ${job.type} completed successfully`, processedAt: new Date().toISOString() };
}

What does this code do?

CONCURRENCY = 3 means the system will only process 3 jobs at once. Any additional jobs stay in the queue until a slot opens up. This is how production systems prevent a flood of jobs from overwhelming the server.

simulateWork mimics real tasks (image resizing, report generation) by running 10 progress steps over 3-8 seconds. Each step updates the job's progress in the database and notifies listeners. The chaos job type intentionally fails 50% of the time to test retry logic.

  • You won't see a change yet. The next chunk adds the retry logic and the main polling loop that ties everything together.
Implement retry logic with exponential backoff

When a job fails, you don't want to retry it immediately. That just hammers the same broken thing over and over. Instead, you wait longer between each retry attempt. This pattern is called exponential backoff: the delay doubles each time (1s, 2s, 4s, 8s).

  • In worker.js, add the backoff calculation, job processing, and queue polling below the simulateWork function:
// Calculate retry delay: 1s, 2s, 4s, 8s... (baseDelay * 2^attempts)
function getBackoffDelay(attempts) {
  const baseDelay = 1000;
  return baseDelay * Math.pow(2, attempts);
}

// Process a single job: run it, handle success or failure
async function processJob(job) {
  activeJobs++;

  const startedJob = updateJobStatus(job.id, 'active', { attempts: job.attempts + 1, progress: 0 });
  notifyUpdate(startedJob);

  try {
    const result = await simulateWork(job);
    const completedJob = updateJobStatus(job.id, 'completed', { progress: 100, result });
    notifyUpdate(completedJob);
  } catch (error) {
    const attempts = job.attempts + 1;

    if (attempts < job.max_retries) {
      // Still have retries left: put back in queue with a delay
      const delay = getBackoffDelay(attempts);
      const failedJob = updateJobStatus(job.id, 'waiting', {
        error: error.message,
        attempts,
        progress: 0
      });
      notifyUpdate({ ...failedJob, retryIn: delay });

      setTimeout(() => pollQueue(), delay);
    } else {
      // Out of retries: mark as permanently failed
      const failedJob = updateJobStatus(job.id, 'failed', {
        error: error.message,
        attempts
      });
      notifyUpdate(failedJob);
    }
  } finally {
    activeJobs--;
    pollQueue();
  }
}

// Pull jobs from the queue until concurrency limit is reached
export function pollQueue() {
  while (activeJobs < CONCURRENCY) {
    const job = getNextJob();
    if (!job) break;
    processJob(job);
  }
}

What does this code do?

getBackoffDelay calculates the wait time before retrying. With a base delay of 1000ms, the first retry waits 2 seconds, the second waits 4 seconds, the third waits 8 seconds. This gives transient failures time to resolve.

processJob handles the full lifecycle of a single job. On success, it marks the job completed. On failure, it checks if retries remain. If yes, the job goes back to waiting with a scheduled retry. If no retries remain, it's marked failed permanently.

pollQueue is the main loop. It keeps pulling jobs as long as there are open concurrency slots and waiting jobs. The finally block in processJob ensures pollQueue runs again after every job finishes, keeping the queue moving.

Why exponential backoff instead of fixed delays?

Fixed-interval retries (e.g. always wait 2 seconds) can overwhelm a failing service with repeated requests. Exponential backoff gives the failing resource progressively more time to recover. This is the same pattern AWS, Stripe, and most production systems use for handling transient failures.

  • Save worker.js.
  • Verify the database module loads correctly by running this from your project root:
node -e "import('./server/database.js').then(() => console.log('Database initialized!'))"
  • You should see Database initialized! printed in your terminal. A new jobs.db file also appears inside your server folder.

Seeing an error about better-sqlite3?

The better-sqlite3 package requires native compilation. Make sure you ran npm install from inside the server folder in Step 1. If you see build errors, confirm you're running Node.js 20 or 22 (the versions better-sqlite3 ships prebuilt binaries for).

help me fix better-sqlite3 installation errors

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved before continuing.

ⓧ I'd like to double check the full code

Here is the complete server/worker.js:

import { getNextJob, updateJobStatus } from './queue.js';

const CONCURRENCY = 3;
let activeJobs = 0;
let onJobUpdate = null;

export function setJobUpdateCallback(callback) {
  onJobUpdate = callback;
}

function notifyUpdate(job) {
  if (onJobUpdate) onJobUpdate(job);
}

async function simulateWork(job) {
  const duration = 3000 + Math.random() * 5000;
  const steps = 10;
  const stepDuration = duration / steps;

  for (let i = 1; i <= steps; i++) {
    await new Promise(resolve => setTimeout(resolve, stepDuration));

    if (job.type === 'chaos' && Math.random() < 0.5 && i === Math.floor(steps / 2)) {
      throw new Error('Random chaos failure!');
    }

    const progress = Math.round((i / steps) * 100);
    const updatedJob = updateJobStatus(job.id, 'active', { progress });
    notifyUpdate(updatedJob);
  }

  return { message: `Job ${job.type} completed successfully`, processedAt: new Date().toISOString() };
}

function getBackoffDelay(attempts) {
  const baseDelay = 1000;
  return baseDelay * Math.pow(2, attempts);
}

async function processJob(job) {
  activeJobs++;

  const startedJob = updateJobStatus(job.id, 'active', { attempts: job.attempts + 1, progress: 0 });
  notifyUpdate(startedJob);

  try {
    const result = await simulateWork(job);
    const completedJob = updateJobStatus(job.id, 'completed', { progress: 100, result });
    notifyUpdate(completedJob);
  } catch (error) {
    const attempts = job.attempts + 1;

    if (attempts < job.max_retries) {
      const delay = getBackoffDelay(attempts);
      const failedJob = updateJobStatus(job.id, 'waiting', {
        error: error.message,
        attempts,
        progress: 0
      });
      notifyUpdate({ ...failedJob, retryIn: delay });

      setTimeout(() => pollQueue(), delay);
    } else {
      const failedJob = updateJobStatus(job.id, 'failed', {
        error: error.message,
        attempts
      });
      notifyUpdate(failedJob);
    }
  } finally {
    activeJobs--;
    pollQueue();
  }
}

export function pollQueue() {
  while (activeJobs < CONCURRENCY) {
    const job = getNextJob();
    if (!job) break;
    processJob(job);
  }
}

Your job queue engine is complete. You have a database layer, queue functions for the full job lifecycle, and a worker that processes jobs with concurrency control and retry logic. Next up, you'll expose this engine through a REST API so clients can submit and monitor jobs.

Build the REST API and WebSocket Server

Your queue engine and worker system are ready. Now you need a way for clients to actually submit jobs and receive live updates as those jobs are processed.

In this step, you will build an Express REST API for job management and attach a WebSocket server for pushing real-time progress updates to connected clients.

In this step, get ready to:
  • Create an Express server with WebSocket support on the same HTTP port.
  • Add connection management with heartbeat detection for dead clients.
  • Build REST endpoints for submitting, listing, cancelling, and monitoring jobs.
Set up Express with WebSocket integration

Express handles HTTP requests, but WebSockets are a different protocol. To serve both on a single port, you create a shared HTTP server and attach both Express and the WebSocket server to it.

  • In VS Code, create a new file called server.js inside the server folder.
  • Add the imports and server initialization by copying this code into server.js:
import express from 'express';
import cors from 'cors';
import { createServer } from 'http';
import { WebSocketServer } from 'ws';
import { v4 as uuidv4 } from 'uuid';
import { enqueueJob, getJob, getAllJobs, cancelJob, getStats } from './queue.js';
import { pollQueue, setJobUpdateCallback } from './worker.js';

// Create Express app and wrap it in an HTTP server
const app = express();
const server = createServer(app);
// Attach WebSocket server to the same HTTP server
const wss = new WebSocketServer({ server });

// Enable CORS and JSON body parsing
app.use(cors());
app.use(express.json());

Why createServer instead of app.listen?

Calling app.listen() creates a hidden HTTP server that only Express can use. By calling createServer(app) explicitly, you get a reference to the underlying HTTP server. You pass that reference to WebSocketServer so both protocols share the same port.

This is the standard pattern documented by the ws library for Express integration.

Add WebSocket connection management and broadcasting

When a browser connects via WebSocket, you need to track that connection so you can push updates to it later. You also need a way to detect and remove dead connections that silently dropped.

  • Add the connection management, heartbeat, and broadcast logic below the middleware lines in server.js:
// Track all connected WebSocket clients
const clients = new Set();

wss.on('connection', (ws) => {
  clients.add(ws);
  console.log(`Client connected. Total: ${clients.size}`);

  // Mark connection as alive for heartbeat checks
  ws.isAlive = true;
  ws.on('pong', () => { ws.isAlive = true; });

  ws.on('close', () => {
    clients.delete(ws);
    console.log(`Client disconnected. Total: ${clients.size}`);
  });
});

// Ping all clients every 30 seconds to detect dead connections
const heartbeat = setInterval(() => {
  wss.clients.forEach((ws) => {
    if (!ws.isAlive) {
      clients.delete(ws);
      return ws.terminate();
    }
    ws.isAlive = false;
    ws.ping();
  });
}, 30000);

wss.on('close', () => clearInterval(heartbeat));

// Send data to all connected clients
function broadcast(data) {
  const message = JSON.stringify(data);
  clients.forEach((ws) => {
    if (ws.readyState === 1) {
      ws.send(message);
    }
  });
}

// Wire up worker updates to broadcast to all clients
setJobUpdateCallback((job) => {
  broadcast({ type: 'job:update', job });
});

What does the heartbeat do?

Every 30 seconds, the server sends a ping frame to each client. If the client is still alive, its browser automatically responds with a pong frame, which sets ws.isAlive = true.

On the next heartbeat cycle, any client that did not respond gets terminated. This prevents memory leaks from zombie connections that the browser closed without a proper disconnect (e.g. the user closed their laptop lid).

Why ws.readyState === 1?

The number 1 corresponds to the WebSocket OPEN state. Checking this before sending prevents errors when a connection is in the process of closing.

Build the REST API endpoints and start the server

The REST API gives clients a way to submit jobs, check their status, cancel them, and view queue statistics. Each endpoint returns an immediate response. The actual processing happens in the background.

  • Add the API routes and server startup below the setJobUpdateCallback block in server.js:
// POST /api/jobs - Submit a new job to the queue
app.post('/api/jobs', (req, res) => {
  const { type, payload, priority, maxRetries } = req.body;

  if (!type) {
    return res.status(400).json({ error: 'Job type is required' });
  }

  const job = enqueueJob({
    id: uuidv4(),
    type,
    payload: payload || {},
    priority: priority || 3,
    maxRetries: maxRetries || 3
  });

  broadcast({ type: 'job:created', job });
  pollQueue();

  res.status(201).json(job);
});

// GET /api/jobs - List all jobs
app.get('/api/jobs', (req, res) => {
  const jobs = getAllJobs();
  res.json(jobs);
});

// GET /api/jobs/:id - Get a single job by ID
app.get('/api/jobs/:id', (req, res) => {
  const job = getJob(req.params.id);
  if (!job) return res.status(404).json({ error: 'Job not found' });
  res.json(job);
});

// DELETE /api/jobs/:id - Cancel a pending job
app.delete('/api/jobs/:id', (req, res) => {
  const job = cancelJob(req.params.id);
  if (!job) return res.status(400).json({ error: 'Job cannot be cancelled (not in waiting status)' });
  broadcast({ type: 'job:update', job });
  res.json(job);
});

// GET /api/stats - Queue metrics
app.get('/api/stats', (req, res) => {
  const stats = getStats();
  res.json(stats);
});

// Start server on port 3001
const PORT = 3001;
server.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
  console.log(`WebSocket server running on ws://localhost:${PORT}`);
});

What does this code do?

The POST /api/jobs endpoint validates the request, creates a job in the database, broadcasts a job:created event to all WebSocket clients, and immediately kicks the worker queue. The caller gets back a 201 response with the job data without waiting for processing to finish.

The DELETE endpoint only cancels jobs that are still in waiting status. Jobs that are already being processed cannot be cancelled. The GET /api/stats endpoint returns a count of jobs by status so the dashboard can show queue health.

  • Save server.js.
  • Open a terminal in VS Code and navigate to the server folder by running:
cd server
  • Start the server by running:
npm run dev
  • You should see two log lines in your terminal:
Server running on http://localhost:3001
WebSocket server running on ws://localhost:3001
  • Test the API by opening a second terminal and running:
curl -X POST http://localhost:3001/api/jobs -H "Content-Type: application/json" -d "{\"type\":\"image-resize\"}"
  • You should see a JSON response with the job details including an id, status of waiting or active, and a type of image-resize.

Server won't start?

  • If you see a "Cannot find module" error, make sure you ran npm install inside the server folder in an earlier step.
  • If port 3001 is already in use, close whatever process is using it or change the PORT constant in server.js.
  • If curl is not available, open http://localhost:3001/api/stats in your browser instead. You should see a JSON object with job counts.

Help me debug my server startup error.

✔️ Awesome, I've got everything!

Great. Double check you have saved server.js and that your server is running without errors.

ⓧ I'd like to double check the full code

Here is the complete server/server.js file:

import express from 'express';
import cors from 'cors';
import { createServer } from 'http';
import { WebSocketServer } from 'ws';
import { v4 as uuidv4 } from 'uuid';
import { enqueueJob, getJob, getAllJobs, cancelJob, getStats } from './queue.js';
import { pollQueue, setJobUpdateCallback } from './worker.js';

const app = express();
const server = createServer(app);
const wss = new WebSocketServer({ server });

app.use(cors());
app.use(express.json());

const clients = new Set();

wss.on('connection', (ws) => {
  clients.add(ws);
  console.log(`Client connected. Total: ${clients.size}`);

  ws.isAlive = true;
  ws.on('pong', () => { ws.isAlive = true; });

  ws.on('close', () => {
    clients.delete(ws);
    console.log(`Client disconnected. Total: ${clients.size}`);
  });
});

const heartbeat = setInterval(() => {
  wss.clients.forEach((ws) => {
    if (!ws.isAlive) {
      clients.delete(ws);
      return ws.terminate();
    }
    ws.isAlive = false;
    ws.ping();
  });
}, 30000);

wss.on('close', () => clearInterval(heartbeat));

function broadcast(data) {
  const message = JSON.stringify(data);
  clients.forEach((ws) => {
    if (ws.readyState === 1) {
      ws.send(message);
    }
  });
}

setJobUpdateCallback((job) => {
  broadcast({ type: 'job:update', job });
});

app.post('/api/jobs', (req, res) => {
  const { type, payload, priority, maxRetries } = req.body;

  if (!type) {
    return res.status(400).json({ error: 'Job type is required' });
  }

  const job = enqueueJob({
    id: uuidv4(),
    type,
    payload: payload || {},
    priority: priority || 3,
    maxRetries: maxRetries || 3
  });

  broadcast({ type: 'job:created', job });
  pollQueue();

  res.status(201).json(job);
});

app.get('/api/jobs', (req, res) => {
  const jobs = getAllJobs();
  res.json(jobs);
});

app.get('/api/jobs/:id', (req, res) => {
  const job = getJob(req.params.id);
  if (!job) return res.status(404).json({ error: 'Job not found' });
  res.json(job);
});

app.delete('/api/jobs/:id', (req, res) => {
  const job = cancelJob(req.params.id);
  if (!job) return res.status(400).json({ error: 'Job cannot be cancelled (not in waiting status)' });
  broadcast({ type: 'job:update', job });
  res.json(job);
});

app.get('/api/stats', (req, res) => {
  const stats = getStats();
  res.json(stats);
});

const PORT = 3001;
server.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
  console.log(`WebSocket server running on ws://localhost:${PORT}`);
});

Your server is live. It accepts jobs over HTTP, processes them in the background, and pushes live updates over WebSockets. Next up, you will build the React dashboard that connects to both and shows everything in real time.

Build the React Dashboard

Your backend is complete. The server handles job submissions, processes them with retry logic, and broadcasts updates over WebSockets.

Now you need a way to see all of this in action. In this step, you will build a React dashboard that submits jobs through the REST API, listens for real-time updates via WebSockets, and displays live progress bars and queue metrics.

In this step, get ready to:
  • Create the main layout with a job submission form connected to the REST API.
  • Build a live job list with progress bars, status badges, and real-time WebSocket updates.
  • Add a queue metrics panel showing jobs by status.
Create the main layout with a job submission form

The dashboard needs to display job controls and status information. You will replace the default Vite boilerplate in App.jsx with a custom component that talks to your server's REST API.

  • In VS Code, open client/src/App.jsx from the Explorer sidebar.
  • Select all the existing content and delete it.
  • Add the imports and configuration constants by pasting the following code:
import { useState, useEffect, useRef } from 'react';

const API_URL = 'http://localhost:3001/api';
const WS_URL = 'ws://localhost:3001';

const JOB_TYPES = [
  { value: 'image-resize', label: 'Image Resize' },
  { value: 'report-generation', label: 'Report Generation' },
  { value: 'data-export', label: 'Data Export' },
  { value: 'email-batch', label: 'Email Batch' },
  { value: 'chaos', label: 'Chaos Mode (50% fail rate)' },
];

const STATUS_COLORS = {
  waiting: 'bg-yellow-100 text-yellow-800',
  active: 'bg-blue-100 text-blue-800',
  completed: 'bg-green-100 text-green-800',
  failed: 'bg-red-100 text-red-800',
  cancelled: 'bg-gray-100 text-gray-800',
};

What does this code do?

API_URL and WS_URL point at your running server on port 3001. The React app will use these to make REST calls and open a WebSocket connection.

JOB_TYPES defines the options in the job type dropdown. STATUS_COLORS maps each job status to Tailwind CSS classes for colour-coded badges.

  • Below the constants, add the App component with state declarations, the initial data fetch, and the two submit functions:
function App() {
  const [jobs, setJobs] = useState([]);
  const [stats, setStats] = useState({ total: 0, waiting: 0, active: 0, completed: 0, failed: 0 });
  const [jobType, setJobType] = useState('image-resize');
  const [connected, setConnected] = useState(false);
  const wsRef = useRef(null);

  useEffect(() => {
    fetch(`${API_URL}/jobs`).then(r => r.json()).then(setJobs);
    fetch(`${API_URL}/stats`).then(r => r.json()).then(setStats);
  }, []);

  async function submitJob() {
    await fetch(`${API_URL}/jobs`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ type: jobType, priority: 3 }),
    });
  }

  async function submitBatch() {
    for (let i = 0; i < 10; i++) {
      await fetch(`${API_URL}/jobs`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ type: jobType, priority: Math.ceil(Math.random() * 5) }),
      });
    }
  }

What does this code do?

The component tracks four pieces of state: the list of jobs, aggregate stats, the currently selected job type, and whether the WebSocket is connected.

The useEffect with an empty dependency array runs once on mount, fetching the current jobs and stats from the REST API. submitJob POSTs a single job. submitBatch sends 10 jobs in a loop with random priorities between 1 and 5.

  • Still inside the App function, add the return statement with the full layout JSX:
  return (
    <div className="min-h-screen bg-gray-900 text-white p-8">
      <div className="max-w-6xl mx-auto">
        <header className="mb-8">
          <h1 className="text-3xl font-bold">Job Queue Dashboard</h1>
          <div className="flex items-center gap-2 mt-2">
            <span className={`w-3 h-3 rounded-full ${connected ? 'bg-green-400' : 'bg-red-400'}`}></span>
            <span className="text-sm text-gray-400">{connected ? 'Connected' : 'Disconnected'}</span>
          </div>
        </header>

        <div className="grid grid-cols-5 gap-4 mb-8">
          <StatCard label="Total" value={stats.total} />
          <StatCard label="Waiting" value={stats.waiting} color="text-yellow-400" />
          <StatCard label="Active" value={stats.active} color="text-blue-400" />
          <StatCard label="Completed" value={stats.completed} color="text-green-400" />
          <StatCard label="Failed" value={stats.failed} color="text-red-400" />
        </div>

        <div className="bg-gray-800 rounded-lg p-6 mb-8">
          <h2 className="text-lg font-semibold mb-4">Submit Jobs</h2>
          <div className="flex gap-4">
            <select
              value={jobType}
              onChange={e => setJobType(e.target.value)}
              className="bg-gray-700 rounded px-4 py-2 text-white"
            >
              {JOB_TYPES.map(t => (
                <option key={t.value} value={t.value}>{t.label}</option>
              ))}
            </select>
            <button onClick={submitJob} className="bg-blue-600 hover:bg-blue-700 px-6 py-2 rounded font-medium">
              Submit Job
            </button>
            <button onClick={submitBatch} className="bg-purple-600 hover:bg-purple-700 px-6 py-2 rounded font-medium">
              Submit 10 Jobs (Stress Test)
            </button>
          </div>
        </div>

        <div className="bg-gray-800 rounded-lg p-6">
          <h2 className="text-lg font-semibold mb-4">Job Queue ({jobs.length})</h2>
          <div className="space-y-3">
            {jobs.map(job => (
              <JobCard key={job.id} job={job} />
            ))}
            {jobs.length === 0 && (
              <p className="text-gray-500 text-center py-8">No jobs yet. Submit one above!</p>
            )}
          </div>
        </div>
      </div>
    </div>
  );
}

What does this JSX render?

  • A header with the title and a green/red connection indicator dot.
  • A 5-column stats grid using StatCard components (you will define this component shortly).
  • A submission panel with a dropdown for job type and two buttons (single job, batch of 10).
  • A job list section that maps over the jobs array and renders a JobCard for each one.
Build the live job list component with real-time WebSocket updates

The layout references a JobCard component and relies on a WebSocket connection to push updates live. Without the WebSocket, the dashboard would only show data from the initial fetch.

  • Inside the App function, add this useEffect directly after the initial fetch useEffect (before the submitJob function):
  useEffect(() => {
    function connect() {
      const ws = new WebSocket(WS_URL);
      wsRef.current = ws;

      ws.onopen = () => setConnected(true);
      ws.onclose = () => {
        setConnected(false);
        setTimeout(connect, 3000);
      };

      ws.onmessage = (event) => {
        const data = JSON.parse(event.data);

        if (data.type === 'job:created') {
          setJobs(prev => [data.job, ...prev]);
          setStats(prev => ({ ...prev, total: prev.total + 1, waiting: prev.waiting + 1 }));
        }

        if (data.type === 'job:update') {
          setJobs(prev => prev.map(j => j.id === data.job.id ? data.job : j));
          fetch(`${API_URL}/stats`).then(r => r.json()).then(setStats);
        }
      };
    }

    connect();
    return () => { if (wsRef.current) wsRef.current.close(); };
  }, []);

How does the WebSocket auto-reconnect work?

The connect function opens a native browser WebSocket connection. If the server drops the connection, onclose fires and schedules a reconnect attempt after 3 seconds.

When a job:created message arrives, the new job is prepended to the list. When a job:update message arrives, the matching job is replaced in state and the stats are refreshed from the API.

  • Below the App function's closing brace, add the JobCard component that renders each job with a status badge and progress bar:
function JobCard({ job }) {
  return (
    <div className="bg-gray-700 rounded-lg p-4">
      <div className="flex justify-between items-center mb-2">
        <div className="flex items-center gap-3">
          <span className="font-mono text-sm text-gray-400">{job.id.slice(0, 8)}</span>
          <span className="font-medium">{job.type}</span>
        </div>
        <span className={`px-2 py-1 rounded text-xs font-medium ${STATUS_COLORS[job.status]}`}>
          {job.status}
        </span>
      </div>
      {job.status === 'active' && (
        <div className="mt-2">
          <div className="flex justify-between text-sm text-gray-400 mb-1">
            <span>Progress</span>
            <span>{job.progress}%</span>
          </div>
          <div className="w-full bg-gray-600 rounded-full h-2">
            <div
              className="bg-blue-500 h-2 rounded-full transition-all duration-300"
              style={{ width: `${job.progress}%` }}
            ></div>
          </div>
        </div>
      )}
      {job.error && (
        <div className="mt-2 text-sm text-red-400">
          Error: {job.error} (Attempt {job.attempts}/{job.max_retries})
        </div>
      )}
    </div>
  );
}

What does JobCard display?

  • The first 8 characters of the job ID (truncated UUID) and the job type.
  • A colour-coded status badge using the STATUS_COLORS map.
  • A progress bar that only appears when the job status is active. The bar width is driven by inline style bound to job.progress.
  • An error message with the attempt count, shown only when job.error exists.
Add the queue metrics panel

The stats grid in the layout references a StatCard component. This small component renders a single metric value with a label and optional colour.

  • Above the JobCard function, add the StatCard component:
function StatCard({ label, value, color = 'text-white' }) {
  return (
    <div className="bg-gray-800 rounded-lg p-4 text-center">
      <div className={`text-2xl font-bold ${color}`}>{value}</div>
      <div className="text-sm text-gray-400">{label}</div>
    </div>
  );
}

What does StatCard show?

Each StatCard displays a number and a label. The color prop defaults to white but accepts Tailwind text colour classes. The stats grid uses five of these cards: Total, Waiting (yellow), Active (blue), Completed (green), and Failed (red).

  • At the very bottom of the file, add the default export:
export default App;
  • Save client/src/App.jsx.
  • Open a new terminal, navigate to the client folder, and start the dev server by running:
cd client
npm run dev
  • Open a second terminal, navigate to the server folder, and start the backend by running:
cd server
npm run dev
  • Open your browser to the URL shown in the client terminal (typically http://localhost:5173).
  • You should see a dark dashboard with a green "Connected" indicator, five stat cards showing zeros, a job type dropdown, and two submit buttons.
  • Click Submit Job. A job card should appear in the list, show a blue progress bar filling up, and then turn green when complete.

Dashboard shows "Disconnected" or no jobs appear?

  • Confirm both terminals are running. The server must be started from inside the server folder and the client from inside the client folder.
  • Check that the server terminal shows Server running on http://localhost:3001. If it shows an error about port 3001 already in use, close any other processes using that port and restart.
  • Open your browser console (F12) and look for CORS or WebSocket errors. If you see a CORS error, verify the server has app.use(cors()) in server.js.

Help me debug my WebSocket connection between the React client and Express server

✔️ Awesome, I've got everything!

Great. Double check you have saved client/src/App.jsx and both servers are running.

ⓧ I'd like to double check the full code

Here is the complete client/src/App.jsx file:

import { useState, useEffect, useRef } from 'react';

const API_URL = 'http://localhost:3001/api';
const WS_URL = 'ws://localhost:3001';

const JOB_TYPES = [
  { value: 'image-resize', label: 'Image Resize' },
  { value: 'report-generation', label: 'Report Generation' },
  { value: 'data-export', label: 'Data Export' },
  { value: 'email-batch', label: 'Email Batch' },
  { value: 'chaos', label: 'Chaos Mode (50% fail rate)' },
];

const STATUS_COLORS = {
  waiting: 'bg-yellow-100 text-yellow-800',
  active: 'bg-blue-100 text-blue-800',
  completed: 'bg-green-100 text-green-800',
  failed: 'bg-red-100 text-red-800',
  cancelled: 'bg-gray-100 text-gray-800',
};

function App() {
  const [jobs, setJobs] = useState([]);
  const [stats, setStats] = useState({ total: 0, waiting: 0, active: 0, completed: 0, failed: 0 });
  const [jobType, setJobType] = useState('image-resize');
  const [connected, setConnected] = useState(false);
  const wsRef = useRef(null);

  useEffect(() => {
    fetch(`${API_URL}/jobs`).then(r => r.json()).then(setJobs);
    fetch(`${API_URL}/stats`).then(r => r.json()).then(setStats);
  }, []);

  useEffect(() => {
    function connect() {
      const ws = new WebSocket(WS_URL);
      wsRef.current = ws;

      ws.onopen = () => setConnected(true);
      ws.onclose = () => {
        setConnected(false);
        setTimeout(connect, 3000);
      };

      ws.onmessage = (event) => {
        const data = JSON.parse(event.data);

        if (data.type === 'job:created') {
          setJobs(prev => [data.job, ...prev]);
          setStats(prev => ({ ...prev, total: prev.total + 1, waiting: prev.waiting + 1 }));
        }

        if (data.type === 'job:update') {
          setJobs(prev => prev.map(j => j.id === data.job.id ? data.job : j));
          fetch(`${API_URL}/stats`).then(r => r.json()).then(setStats);
        }
      };
    }

    connect();
    return () => { if (wsRef.current) wsRef.current.close(); };
  }, []);

  async function submitJob() {
    await fetch(`${API_URL}/jobs`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ type: jobType, priority: 3 }),
    });
  }

  async function submitBatch() {
    for (let i = 0; i < 10; i++) {
      await fetch(`${API_URL}/jobs`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ type: jobType, priority: Math.ceil(Math.random() * 5) }),
      });
    }
  }

  return (
    <div className="min-h-screen bg-gray-900 text-white p-8">
      <div className="max-w-6xl mx-auto">
        <header className="mb-8">
          <h1 className="text-3xl font-bold">Job Queue Dashboard</h1>
          <div className="flex items-center gap-2 mt-2">
            <span className={`w-3 h-3 rounded-full ${connected ? 'bg-green-400' : 'bg-red-400'}`}></span>
            <span className="text-sm text-gray-400">{connected ? 'Connected' : 'Disconnected'}</span>
          </div>
        </header>

        <div className="grid grid-cols-5 gap-4 mb-8">
          <StatCard label="Total" value={stats.total} />
          <StatCard label="Waiting" value={stats.waiting} color="text-yellow-400" />
          <StatCard label="Active" value={stats.active} color="text-blue-400" />
          <StatCard label="Completed" value={stats.completed} color="text-green-400" />
          <StatCard label="Failed" value={stats.failed} color="text-red-400" />
        </div>

        <div className="bg-gray-800 rounded-lg p-6 mb-8">
          <h2 className="text-lg font-semibold mb-4">Submit Jobs</h2>
          <div className="flex gap-4">
            <select
              value={jobType}
              onChange={e => setJobType(e.target.value)}
              className="bg-gray-700 rounded px-4 py-2 text-white"
            >
              {JOB_TYPES.map(t => (
                <option key={t.value} value={t.value}>{t.label}</option>
              ))}
            </select>
            <button onClick={submitJob} className="bg-blue-600 hover:bg-blue-700 px-6 py-2 rounded font-medium">
              Submit Job
            </button>
            <button onClick={submitBatch} className="bg-purple-600 hover:bg-purple-700 px-6 py-2 rounded font-medium">
              Submit 10 Jobs (Stress Test)
            </button>
          </div>
        </div>

        <div className="bg-gray-800 rounded-lg p-6">
          <h2 className="text-lg font-semibold mb-4">Job Queue ({jobs.length})</h2>
          <div className="space-y-3">
            {jobs.map(job => (
              <JobCard key={job.id} job={job} />
            ))}
            {jobs.length === 0 && (
              <p className="text-gray-500 text-center py-8">No jobs yet. Submit one above!</p>
            )}
          </div>
        </div>
      </div>
    </div>
  );
}

function StatCard({ label, value, color = 'text-white' }) {
  return (
    <div className="bg-gray-800 rounded-lg p-4 text-center">
      <div className={`text-2xl font-bold ${color}`}>{value}</div>
      <div className="text-sm text-gray-400">{label}</div>
    </div>
  );
}

function JobCard({ job }) {
  return (
    <div className="bg-gray-700 rounded-lg p-4">
      <div className="flex justify-between items-center mb-2">
        <div className="flex items-center gap-3">
          <span className="font-mono text-sm text-gray-400">{job.id.slice(0, 8)}</span>
          <span className="font-medium">{job.type}</span>
        </div>
        <span className={`px-2 py-1 rounded text-xs font-medium ${STATUS_COLORS[job.status]}`}>
          {job.status}
        </span>
      </div>
      {job.status === 'active' && (
        <div className="mt-2">
          <div className="flex justify-between text-sm text-gray-400 mb-1">
            <span>Progress</span>
            <span>{job.progress}%</span>
          </div>
          <div className="w-full bg-gray-600 rounded-full h-2">
            <div
              className="bg-blue-500 h-2 rounded-full transition-all duration-300"
              style={{ width: `${job.progress}%` }}
            ></div>
          </div>
        </div>
      )}
      {job.error && (
        <div className="mt-2 text-sm text-red-400">
          Error: {job.error} (Attempt {job.attempts}/{job.max_retries})
        </div>
      )}
    </div>
  );
}

export default App;

Your dashboard is live. Next up, you will stress test the system by triggering failures and watching the retry logic and exponential backoff in action.

Test Failure Scenarios

Your job queue system is fully built and running. But building a system is only half the story. Understanding how it behaves under pressure and when things go wrong is what separates a demo from a production-ready architecture.

In this step, you will intentionally break things. You will trigger random failures to watch your retry logic and exponential backoff in action. Then you will flood the queue to confirm that concurrency limits hold firm.

In this step, get ready to:
  • Test chaos mode to observe retry logic with exponential backoff.
  • Stress test by submitting 20+ jobs simultaneously to observe concurrency limits.
Trigger chaos mode and observe retries

Your worker code already has a chaos mode built in. When a job has the type chaos, there is a 50% chance it fails at the halfway point of processing. This simulates real-world transient failures like network timeouts or third-party API errors.

  • Return to your Job Queue Dashboard in the browser.
  • In the Submit Jobs section, click the dropdown and select Chaos Mode (50% fail rate).
  • Click Submit Job five times to create five chaos jobs.
  • Watch the Job Queue list below.

You should see jobs start processing with a blue progress bar. Around the 50% mark, some will fail and show a red error message. After a short delay, those failed jobs reappear as "waiting" and get retried automatically.

What is exponential backoff?

When a job fails, your worker does not retry immediately. It waits an increasing amount of time between each attempt. The formula is 1000 * 2^attempts milliseconds.

  • After the 1st failure: 2 second wait.
  • After the 2nd failure: 4 second wait.
  • After the 3rd failure: the job is marked as permanently failed (max retries exhausted).

This pattern prevents a failing service from getting hammered with immediate retries. AWS, Stripe, and GitHub all use this exact strategy in their SDKs.

  • Watch a chaos job go through its retry cycle. You will see the error message update with the attempt count (e.g. "Attempt 2/3").
  • Wait for at least one job to exhaust all 3 retries and land on a permanent failed status (red badge).
  • Check the stats panel at the top. The Failed counter should have increased.

What should I see?

Some chaos jobs will complete successfully on their first try (the 50% that pass). Others will fail, briefly return to "waiting" status, then retry. You might notice the retry takes a bit longer each time. That is the exponential backoff in action.

A job that fails all 3 attempts will show a red "failed" badge and display the error message Random chaos failure! with "Attempt 3/3".

Not seeing any failures?

Make sure you selected Chaos Mode (50% fail rate) from the dropdown before submitting. Jobs of other types will always succeed.

If you submitted chaos jobs but all 5 completed successfully, that is just luck (the 50% coin flip went your way every time). Submit 5 more and you should see failures.

help me debug why my chaos mode jobs aren't failing

Stress test with 20+ simultaneous jobs

Your worker has a concurrency limit of 3 set by the CONCURRENCY constant in worker.js. This means no matter how many jobs flood the queue, only 3 will process at the same time. The rest wait their turn.

  • In the dashboard, keep Chaos Mode (50% fail rate) selected in the dropdown.
  • Click Submit 10 Jobs (Stress Test) three times quickly. This queues 30 jobs.
  • Immediately look at the stats panel at the top of the dashboard.

You should see the Active counter stay at 3 (never higher), while the Waiting counter shows all the jobs lined up in the queue.

Why limit concurrency?

Without a concurrency limit, submitting 30 jobs would process all 30 at once. In a real system, that could overwhelm a database, exhaust API rate limits, or eat all available memory.

By capping at 3 concurrent workers, the queue acts as a backpressure mechanism. It protects downstream services from being overwhelmed, even when upstream demand spikes.

  • Watch the Job Queue list. You will see 3 jobs with active progress bars at any given time.
  • As each job completes (or fails and enters its backoff wait), a new waiting job immediately picks up.
  • Wait until all 30 jobs have either completed or permanently failed.

What should I see?

The dashboard becomes a live war room. Three progress bars move simultaneously. Some jobs finish green, others flash red and bounce back to waiting for a retry. The stats panel updates in real time as jobs flow through the system.

Because you are using chaos mode, roughly half the jobs will fail on their first attempt. Many will succeed on retry. A few unlucky jobs will exhaust all 3 retries and permanently fail.

Seeing more than 3 active jobs?

The stats panel fetches counts from the database. If you see the Active counter briefly spike above 3, it may be a display timing issue where the stats fetch overlaps with a job state transition. Watch the progress bars instead. You should never see more than 3 progress bars animating at the same time.

If you consistently see 4+ progress bars active simultaneously, check that CONCURRENCY is set to 3 in server/worker.js.

help me debug why my concurrency limit isn't working

You have just proven that your system handles failure gracefully and stays controlled under load. These two properties are what production systems need above all else. Next up, the Secret Mission challenges you to add priority-based scheduling and a dead-letter queue for permanently failed jobs.

Secret mission

Add a Dead-Letter Queue

What happens to jobs that fail permanently after exhausting all retries? In production systems, they don't just vanish. They go to a dead-letter queue where engineers can inspect failure details and retry them manually. Build one for your job queue system.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing cloud costs or subscriptions to worry about. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project files on your Desktop (server/ and client/ directories with dependencies and SQLite database).
  • Running Node.js server process on port 3001.
  • Running Vite dev server for the React client.

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with the job queue system.

  • Your server continues running on port 3001 and the client dev server stays active.
  • The SQLite database (jobs.db) persists your job history between restarts.
  • You can restart the servers anytime by running npm run dev in each directory.

Pause - I'll come back to this later

Shut down running processes to free up memory and ports. Your project files stay on disk so you can pick up where you left off.

  • Switch to the terminal running the Node.js server and press Ctrl+C to stop it.
  • Switch to the terminal running the Vite dev server and press Ctrl+C to stop it.
  • Your project files and database remain on your Desktop. To resume, open the project folder in VS Code and run npm run dev in both the server/ and client/ directories.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your code, dependencies, and the SQLite database permanently.

  • Switch to the terminal running the Node.js server and press Ctrl+C to stop it.
  • Switch to the terminal running the Vite dev server and press Ctrl+C to stop it.
  • Delete your project folder from the Desktop. This is the folder that contains your server/ and client/ directories.

On macOS or Linux, delete it by running:

cd ~/Desktop
rm -rf job-queue

On Windows (PowerShell), delete it by running:

cd ~\Desktop
Remove-Item -Recurse -Force job-queue

Folder name may differ

Replace job-queue in the commands above with whatever you named your project folder in Step 1.

Nice Work!

Nice Work!

You just built a production-grade job processing system from scratch. This is the exact architecture that powers task queues at companies processing millions of background jobs every day.

You've learned how to:

  • Built a custom job queue engine with SQLite persistence, configurable concurrency control, and exponential backoff retry logic.
  • Implemented real-time WebSocket communication that pushes live job state changes from server to client without polling.
  • Created a React and Tailwind CSS monitoring dashboard with live progress bars, status badges, and queue metrics updated in real time.
  • Secret Mission: Added priority-based job scheduling and a dead-letter queue for jobs that exhaust all retries, with a retry button to re-enqueue them.

Ready to quiz yourself?