Build a Webhook Delivery System

Introduction

30 Second Summary

Think about when you order something online and get a notification that it shipped. Behind the scenes, one system sent an event to another. But what happens when that delivery fails? The receiving system might be down, overloaded, or temporarily unreachable.

In this project, you will build a webhook delivery system that queues events, delivers them to registered endpoints, and retries failed deliveries using exponential backoff.

What You'll Build

A Node.js service that accepts events via a REST API, queues them with BullMQ, and delivers them to registered webhook endpoints with automatic retries and delivery tracking.

By the end of this project, you'll have:

• A REST API that registers webhook endpoints and accepts incoming events for delivery.
• A BullMQ worker that processes delivery jobs with exponential backoff retries.
• A delivery tracking system that logs every attempt with timestamps, status updates, and idempotency checks.
• Secret Mission: Add a dead letter queue that captures permanently failed deliveries and lets you replay them on demand.

Are there any prerequisites?

This project assumes basic JavaScript and Node.js knowledge. You should also be familiar with REST APIs (sending and receiving HTTP requests).

No prior experience with message queues or Docker is required. You will learn both from scratch.

Before We Start

Every modern web service needs to communicate events to other systems. A payment goes through, a pull request gets merged, a user signs up. These are all events that external systems want to know about.

In this step, you'll set up the tools you need to build a webhook delivery system. You'll install Docker Desktop (which runs Redis for your message queue), verify your Node.js version, and create your project folder.

In this step, get ready to:

• Install Docker Desktop to run Redis.
• Verify Node.js is version 18 or higher.
• Create and initialize your project directory.

Install Docker Desktop

Your webhook system uses Redis as its message queue backbone. Instead of installing Redis directly on your machine, you'll use Docker Desktop to run Redis inside a container. This keeps your system clean and makes setup identical on any machine.

• Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Docker Desktop and press Enter.

✔️ Docker Desktop opens

Docker Desktop is already installed. Make sure it is running (you should see the whale icon in your system tray or menu bar), then skip ahead to the next substep.

ⓧ It's not installed yet

Download and install Docker Desktop from the official documentation for your operating system.

macOS

• Visit the Docker Desktop for Mac install page:
• https://docs.docker.com/desktop/setup/install/mac-install/
• Click the download button for your chip type (Apple chip or Intel chip).
• Double-click the downloaded Docker.dmg file to open the installer.
• Drag the Docker icon to your Applications folder.
• Open Docker.app from your Applications folder.
• Accept the Docker Subscription Service Agreement when prompted.
• Wait for Docker Desktop to finish starting up. You will see the whale icon appear in your menu bar.

Windows

• Visit the Docker Desktop for Windows install page:
• https://docs.docker.com/desktop/setup/install/windows-install/
• Download and double-click Docker Desktop Installer.exe.
• Choose the per-user installation option (no admin privileges required).
• Ensure WSL 2 is selected as the backend.
• Click Install and wait for the installation to complete.
• Restart your computer if prompted.
• After restarting, Docker Desktop should launch automatically. You will see the whale icon in your system tray.

Linux

• Visit the Docker Desktop for Linux install page:
• https://docs.docker.com/desktop/setup/install/linux/
• Follow the instructions for your specific Linux distribution (Debian, RPM, or Arch).
• Launch Docker Desktop after installation completes.

• Verify Docker is working by running this command in your terminal:

docker --version

You should see a version number printed, confirming Docker is ready.

Docker command not found after installing?

Make sure Docker Desktop is fully started (the whale icon should be still, not animating). On Windows, you may need to close and reopen your terminal after installation.

help me troubleshoot Docker Desktop not being recognized in my terminal

What is Docker Desktop?

Docker Desktop lets you run lightweight, isolated containers on your machine. In this project, you'll use it to run a Redis server inside a container. This means you don't need to install Redis natively on your operating system.

Verify Node.js is version 18 or higher

Your webhook delivery system uses Express.js 5.x, which requires Node.js 18 or higher. The native fetch API (used for making HTTP requests to webhook endpoints) is also only available in Node.js 18+.

• Check your Node.js version by running this command in your terminal:

node --version

✔️ I see version 18 or higher

Your Node.js version is ready. Move on to the next substep.

• Confirm npm is available by running:

npm --version

You should see a version number. Any version of npm that ships with Node.js 18+ will work.

ⓧ I see an older version

You have Node.js installed but it is too old. Download the latest LTS version from the official website to upgrade.

• Visit https://nodejs.org and download the LTS version.
• Run the installer and accept the default settings.
• Close and reopen your terminal after installation.
• Run node --version again to confirm you see v18 or higher.

Why does the old version persist on macOS?

macOS may have an older Node.js version on the system path. After installing the new LTS, close all terminal windows and open a fresh one. If the old version still shows, use the full path to the new binary or install a version manager like nvm to manage versions.

ⓧ Command not found

Node.js is not installed. Download it from the official website.

• Visit https://nodejs.org and download the LTS version.
• Run the installer and accept the default settings.
• Close and reopen your terminal after installation.
• Run node --version to confirm it prints v18 or higher.
• Run npm --version to confirm npm is also available.

Create your project directory

With Docker and Node.js ready, it is time to create the folder where your webhook delivery system will live.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder, move into it, and initialize it by running these commands:

mkdir webhook-delivery-system
cd webhook-delivery-system
npm init -y

What does npm init -y do?

The npm init -y command creates a package.json file with default values. This file tracks your project's name, version, and (later) its dependencies.

The -y flag accepts all defaults so you don't have to answer prompts.

• Confirm the file was created by running:

ls

You should see package.json listed in the output.

Don't see package.json?

Make sure you ran all three commands in sequence. If you see an error on mkdir, the folder may already exist. Try running cd webhook-delivery-system and then npm init -y separately.

help me create and initialize my project folder for the webhook delivery system

Your development environment is locked and loaded. Next up, you'll spin up Redis with Docker Compose and install your project's dependencies.

Install Docker Desktop and Verify Node.js

Your development tools are in place. The next piece of the puzzle is infrastructure. Your webhook delivery system needs Redis as its message broker because BullMQ stores all job queues and state there.

In this step, you'll spin up a Redis container using Docker Compose, then install the three npm packages your project depends on.

In this step, get ready to:

• Set up Redis using Docker Compose.
• Install Express, BullMQ, and ioredis.
• Configure your package.json scripts for running the project.

Create the Docker Compose file for Redis

Docker Compose lets you define and run multi-container applications with a single YAML file. You only need one service right now: Redis.

• Create a new file called docker-compose.yml in your webhook-delivery-system folder by running:

touch docker-compose.yml

• Confirm the file exists by running:

ls

You should see docker-compose.yml and package.json in the output.

Don't see docker-compose.yml?

Make sure you ran touch docker-compose.yml from inside the webhook-delivery-system folder. Run pwd to check your current directory.

Help me figure out why my file isn't appearing

• Open docker-compose.yml in VS Code and paste the following:

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data

volumes:
  redis-data:

What does this file do?

• image: redis:7-alpine pulls a lightweight Redis 7 image based on Alpine Linux.
• ports: "6379:6379" maps the container's Redis port to your machine so BullMQ can connect via localhost:6379.
• volumes: redis-data:/data persists Redis data between container restarts so queued jobs survive a reboot.

• Save docker-compose.yml.

Start the Redis container

With your Compose file saved, you can spin up Redis in a single command. Docker will pull the image if it's not already cached locally.

• Start the Redis container in detached mode by running:

docker compose up -d

What does -d mean?

The -d flag runs the container in the background (detached mode). Without it, the Redis logs would take over your terminal.

• Verify Redis is running by checking the container status:

docker compose ps

You should see a table with your Redis container listed. The STATUS column shows it is running, and the PORTS column shows 0.0.0.0:6379->6379/tcp.

Container not running?

Make sure Docker Desktop is running in the background. If you see an error about the Docker daemon, switch back to Docker Desktop and wait for it to finish starting.

If the container exited immediately, run docker compose logs to see what went wrong.

Help me troubleshoot my Redis container

Install dependencies and configure npm scripts

Your project uses three packages: Express for the HTTP API, BullMQ for the job queue, and ioredis for the Redis connection that BullMQ requires.

• Install all three dependencies with pinned versions by running:

npm install express@5.2.1 bullmq@5.77.6 ioredis@5.11.0

Why pin exact versions?

Pinning versions ensures everyone following this project gets identical behaviour. Without version pins, npm would install the latest compatible release, which could introduce breaking changes.

• Confirm the install succeeded by running:

ls node_modules

You should see a long list of folders including express, bullmq, and ioredis.

Install failed with a network or permissions error?

Check your internet connection and retry. If you see EACCES permission errors, avoid using sudo. Instead, fix npm permissions by following the npm docs.

Help me debug my npm install error

Now update your package.json with the project description and convenience scripts you'll use to start each process.

• Open package.json in VS Code.
• Replace its entire contents with the following:

{
  "name": "webhook-delivery-system",
  "version": "1.0.0",
  "description": "A webhook delivery system with queue-based processing and retries",
  "main": "src/server.js",
  "scripts": {
    "start": "node src/server.js",
    "worker": "node src/worker.js",
    "receiver": "node src/test-receiver.js"
  },
  "dependencies": {
    "express": "5.2.1",
    "bullmq": "5.77.6",
    "ioredis": "5.11.0"
  }
}

What do these scripts do?

• npm start runs the Express API server that receives events and queues deliveries.
• npm run worker starts the BullMQ worker that processes delivery jobs.
• npm run receiver starts a test endpoint you'll use later to simulate webhook delivery.

• Save package.json.

✔️ Awesome, I've got everything!

Great. Double check you've saved both docker-compose.yml and package.json.

ⓧ I'd like to double check the full code

Here is the complete docker-compose.yml:

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data

volumes:
  redis-data:

And here is the complete package.json:

{
  "name": "webhook-delivery-system",
  "version": "1.0.0",
  "description": "A webhook delivery system with queue-based processing and retries",
  "main": "src/server.js",
  "scripts": {
    "start": "node src/server.js",
    "worker": "node src/worker.js",
    "receiver": "node src/test-receiver.js"
  },
  "dependencies": {
    "express": "5.2.1",
    "bullmq": "5.77.6",
    "ioredis": "5.11.0"
  }
}

Your infrastructure is ready. Redis is running, your dependencies are installed, and your npm scripts are configured. Next up, you'll build the Express API that accepts webhook registrations and queues events for delivery.

Set Up Redis with Docker Compose

Your project folder is initialized, Redis is running in Docker Compose, and your dependencies are installed. Now your application needs a way to talk to Redis and manage delivery jobs.

BullMQ is a job queue library that handles adding jobs, retrying them on failure, and tracking their state. In this step, you'll create a queue module that connects your app to Redis and configures how retries work.

In this step, get ready to:

• Create the src directory for your application code.
• Configure a BullMQ queue connected to Redis with automatic exponential backoff retries.

Configure the BullMQ queue connection

BullMQ needs two things to work: a connection to Redis, and configuration for how it should handle retries when a job fails. You'll create a single module that any part of your app can import to add jobs to the queue.

• Create the src folder in your project by running this command:

mkdir src

• Confirm the folder was created by running:

ls

You should see a src folder in the output alongside node_modules, package.json, and docker-compose.yml.

• In the Explorer sidebar in VS Code, right-click the src folder.
• Click New File.
• Type queue.js and press Enter.
• Add the following code to src/queue.js:

const { Queue } = require('bullmq');
const IORedis = require('ioredis');

// Connect to the Redis instance running in Docker
const connection = new IORedis({
  host: 'localhost',
  port: 6379,
  maxRetriesPerRequest: null,
});

// Create the delivery queue with retry settings
const webhookQueue = new Queue('webhook-delivery', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 1000,
    },
    removeOnComplete: 100,
    removeOnFail: 200,
  },
});

module.exports = { webhookQueue, connection };

What does this code do?

This module sets up two things: a Redis connection and a job queue that uses it.

• maxRetriesPerRequest: null is required by BullMQ workers. Without it, Redis blocking commands time out and the worker crashes.
• attempts: 5 means each job will be tried up to 5 times before being marked as permanently failed.
• backoff: { type: 'exponential', delay: 1000 } tells BullMQ to wait increasingly longer between retries. The formula is 2^(attempts-1) × 1000ms, so retries happen at 1s, 2s, 4s, 8s.
• removeOnComplete: 100 and removeOnFail: 200 keep Redis tidy by only storing the most recent completed and failed jobs.

What is exponential backoff?

Exponential backoff is a retry strategy where wait times grow exponentially between attempts. Instead of hammering a failing endpoint every second, your system waits 1s, then 2s, then 4s, then 8s.

This gives the failing service time to recover without overwhelming it with requests. It's the industry standard for webhook delivery systems.

• Save src/queue.js.
• Verify the module loads correctly by running this command in your terminal:

node -e "const { webhookQueue } = require('./src/queue'); console.log('Queue ready:', webhookQueue.name); process.exit(0)"

You should see Queue ready: webhook-delivery printed in your terminal.

Seeing an error?

• If you see a "Cannot find module" error, make sure you're running the command from the webhook-delivery-system folder (not inside src).
• If you see a connection error, confirm Redis is still running with docker compose ps. If it's not listed, run docker compose up -d to start it again.
• If you see a syntax error, double-check that your queue.js code matches the code block above exactly.

Still stuck? Help me debug my queue.js connection error

✔️ Awesome, I've got everything!

Great. Double check you've saved src/queue.js before moving on.

ⓧ I'd like to double check the full code

Here's the complete src/queue.js file for comparison:

const { Queue } = require('bullmq');
const IORedis = require('ioredis');

const connection = new IORedis({
  host: 'localhost',
  port: 6379,
  maxRetriesPerRequest: null,
});

const webhookQueue = new Queue('webhook-delivery', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 1000,
    },
    removeOnComplete: 100,
    removeOnFail: 200,
  },
});

module.exports = { webhookQueue, connection };

Your queue is configured and ready to process jobs. Next up, you'll build the Express API that accepts webhook events and feeds them into this queue for delivery.

Build the Express API

Your Redis queue is ready and waiting for jobs. Now it's time to build the REST API that accepts incoming webhook events and pushes delivery jobs onto that queue.

In this step, you'll create two files: a delivery tracking module that records the status of every attempt, and an Express server with endpoints for registering webhooks and sending events.

In this step, get ready to:

• Create a delivery tracking module with status logging.
• Build an Express API with routes for registering webhooks and sending events.
• Start the server and test the full API flow.

Create the delivery tracking module

Before building the server, you need a way to track what happens to each delivery attempt. The server will log every delivery it queues, and later the worker will update those records as attempts succeed or fail.

• Create a new file called src/deliveryLog.js and paste the following code:

const deliveries = new Map();

function createDelivery(deliveryId, eventId, endpointId, endpointUrl, payload) {
  deliveries.set(deliveryId, {
    id: deliveryId,
    eventId,
    endpointId,
    endpointUrl,
    payload,
    status: 'pending',
    attempts: 0,
    createdAt: new Date().toISOString(),
    updatedAt: new Date().toISOString(),
    lastError: null,
  });
}

function updateDelivery(deliveryId, updates) {
  const delivery = deliveries.get(deliveryId);
  if (delivery) {
    Object.assign(delivery, updates, { updatedAt: new Date().toISOString() });
  }
}

function getDelivery(deliveryId) {
  return deliveries.get(deliveryId);
}

function getAllDeliveries() {
  return Array.from(deliveries.values());
}

function isAlreadyDelivered(eventId, endpointId) {
  for (const delivery of deliveries.values()) {
    if (
      delivery.eventId === eventId &&
      delivery.endpointId === endpointId &&
      delivery.status === 'delivered'
    ) {
      return true;
    }
  }
  return false;
}

module.exports = {
  createDelivery,
  updateDelivery,
  getDelivery,
  getAllDeliveries,
  isAlreadyDelivered,
};

What does this code do?

This module uses a Map to store delivery records in memory. Each delivery tracks its status (pending, delivered, or failed), how many attempts have been made, and the last error message.

The isAlreadyDelivered function is an idempotency check. It prevents the same event from being delivered twice to the same endpoint if a job gets processed more than once.

• Save src/deliveryLog.js.
• Confirm the file exists by running:

ls src/

You should see deliveryLog.js and queue.js listed in the output.

Don't see deliveryLog.js?

Make sure you created the file inside the src/ folder, not in the project root. The file path should be src/deliveryLog.js with a capital L in Log.

Help me find my deliveryLog.js file

✔️ Awesome, I've got everything!

Great. Make sure you've saved src/deliveryLog.js before moving on.

ⓧ I'd like to double check the full code

const deliveries = new Map();

function createDelivery(deliveryId, eventId, endpointId, endpointUrl, payload) {
  deliveries.set(deliveryId, {
    id: deliveryId,
    eventId,
    endpointId,
    endpointUrl,
    payload,
    status: 'pending',
    attempts: 0,
    createdAt: new Date().toISOString(),
    updatedAt: new Date().toISOString(),
    lastError: null,
  });
}

function updateDelivery(deliveryId, updates) {
  const delivery = deliveries.get(deliveryId);
  if (delivery) {
    Object.assign(delivery, updates, { updatedAt: new Date().toISOString() });
  }
}

function getDelivery(deliveryId) {
  return deliveries.get(deliveryId);
}

function getAllDeliveries() {
  return Array.from(deliveries.values());
}

function isAlreadyDelivered(eventId, endpointId) {
  for (const delivery of deliveries.values()) {
    if (
      delivery.eventId === eventId &&
      delivery.endpointId === endpointId &&
      delivery.status === 'delivered'
    ) {
      return true;
    }
  }
  return false;
}

module.exports = {
  createDelivery,
  updateDelivery,
  getDelivery,
  getAllDeliveries,
  isAlreadyDelivered,
};

Create the Express server

The server is the front door of your webhook system. It exposes routes that let external services register their URLs and send events that need delivering.

• Create a new file called src/server.js and paste the following code to set up the server and endpoint registration routes:

const express = require('express');
const crypto = require('crypto');
const { webhookQueue } = require('./queue');
const { createDelivery, getAllDeliveries } = require('./deliveryLog');

const app = express();
app.use(express.json());

const endpoints = new Map();

app.post('/endpoints', (req, res) => {
  const { url } = req.body;
  if (!url) {
    return res.status(400).json({ error: 'url is required' });
  }
  const id = crypto.randomUUID();
  endpoints.set(id, { id, url, createdAt: new Date().toISOString() });
  res.status(201).json({ id, url });
});

app.get('/endpoints', (req, res) => {
  res.json(Array.from(endpoints.values()));
});

What does this code do?

The imports pull in webhookQueue from your queue module and createDelivery and getAllDeliveries from the delivery log you just created.

The endpoints Map stores registered webhook URLs in memory. POST /endpoints registers a new URL with a unique ID, and GET /endpoints lists all registered endpoints.

You won't see a change yet. The next chunk adds event processing and starts the server.

• Add the event processing route and server startup below the existing code in src/server.js:

app.post('/events', async (req, res) => {
  const { type, data } = req.body;
  if (!type) {
    return res.status(400).json({ error: 'type is required' });
  }

  const eventId = crypto.randomUUID();
  const event = { id: eventId, type, data, createdAt: new Date().toISOString() };

  const jobs = [];
  for (const [endpointId, endpoint] of endpoints) {
    const deliveryId = crypto.randomUUID();

    createDelivery(deliveryId, eventId, endpointId, endpoint.url, event);

    jobs.push(
      webhookQueue.add('deliver', {
        deliveryId,
        eventId,
        endpointId,
        endpointUrl: endpoint.url,
        payload: event,
      })
    );
  }

  await Promise.all(jobs);

  res.status(202).json({
    eventId,
    deliveriesQueued: jobs.length,
  });
});

app.get('/deliveries', (req, res) => {
  res.json(getAllDeliveries());
});

const PORT = 3000;
app.listen(PORT, () => {
  console.log(`Webhook API server running on http://localhost:${PORT}`);
});

What does this code do?

POST /events is the core route. When an event arrives, it loops through every registered endpoint and does two things for each: creates a delivery record in the log, and adds a job to the BullMQ queue.

The response returns a 202 Accepted status. This tells the caller that the event has been accepted for processing but delivery hasn't happened yet. That's the whole point of async queues.

GET /deliveries returns the full delivery log so you can inspect the status of every attempt.

• Save src/server.js.

Seeing a syntax error when you save?

Make sure the second chunk of code goes directly below the GET /endpoints route from the first chunk. There should be no duplicate const app = express() lines or repeated imports.

Also check that the POST /events route handler has async before (req, res). Without it, await Promise.all(jobs) will throw an error.

Help me debug my server.js syntax error

✔️ Awesome, I've got everything!

Great. Make sure you've saved src/server.js before starting the server.

ⓧ I'd like to double check the full code

const express = require('express');
const crypto = require('crypto');
const { webhookQueue } = require('./queue');
const { createDelivery, getAllDeliveries } = require('./deliveryLog');

const app = express();
app.use(express.json());

const endpoints = new Map();

app.post('/endpoints', (req, res) => {
  const { url } = req.body;
  if (!url) {
    return res.status(400).json({ error: 'url is required' });
  }
  const id = crypto.randomUUID();
  endpoints.set(id, { id, url, createdAt: new Date().toISOString() });
  res.status(201).json({ id, url });
});

app.get('/endpoints', (req, res) => {
  res.json(Array.from(endpoints.values()));
});

app.post('/events', async (req, res) => {
  const { type, data } = req.body;
  if (!type) {
    return res.status(400).json({ error: 'type is required' });
  }

  const eventId = crypto.randomUUID();
  const event = { id: eventId, type, data, createdAt: new Date().toISOString() };

  const jobs = [];
  for (const [endpointId, endpoint] of endpoints) {
    const deliveryId = crypto.randomUUID();

    createDelivery(deliveryId, eventId, endpointId, endpoint.url, event);

    jobs.push(
      webhookQueue.add('deliver', {
        deliveryId,
        eventId,
        endpointId,
        endpointUrl: endpoint.url,
        payload: event,
      })
    );
  }

  await Promise.all(jobs);

  res.status(202).json({
    eventId,
    deliveriesQueued: jobs.length,
  });
});

app.get('/deliveries', (req, res) => {
  res.json(getAllDeliveries());
});

const PORT = 3000;
app.listen(PORT, () => {
  console.log(`Webhook API server running on http://localhost:${PORT}`);
});

Start and test the server

With both files saved, you can now start the API and test the full registration-to-event flow.

• Start the server by running:

node src/server.js

You should see this output in your terminal:

Webhook API server running on http://localhost:3000

Seeing 'Cannot find module' errors?

If you see an error about ./queue or ./deliveryLog, make sure both src/queue.js and src/deliveryLog.js exist. Check that you're running the command from the webhook-delivery-system folder.

If you see an error about connecting to Redis, make sure your Docker container is still running with docker compose ps.

Help me fix my Cannot find module error

• Open a second terminal window.
• Register a test endpoint by running:

curl -X POST http://localhost:3000/endpoints -H "Content-Type: application/json" -d '{"url": "http://localhost:4000/webhook"}'

You should see a JSON response with an id and the url you registered.

• Send a test event by running:

curl -X POST http://localhost:3000/events -H "Content-Type: application/json" -d '{"type": "payment.completed", "data": {"amount": 100}}'

You should see a response with an eventId and deliveriesQueued: 1.

• Check the delivery log by running:

curl http://localhost:3000/deliveries

You should see a JSON array with one delivery entry showing status: "pending". The delivery is pending because you haven't built the worker yet.

Why is the status still pending?

The server only queues delivery jobs. It doesn't process them. A separate worker (which you'll build next) picks jobs off the queue and actually delivers the HTTP requests.

This separation is what makes the system reliable. The API stays fast because it never waits for deliveries to complete.

Your API is live and queueing delivery jobs. Next up, you'll build the worker that picks those jobs off the queue and actually delivers them with automatic retries.

Build the Delivery Worker with Retries

Your server is accepting events and queuing them in Redis, but nothing is actually processing those jobs yet. The queue is filling up with delivery tasks that have no worker to execute them.

In this step, you'll build the worker process that pulls jobs from the queue and delivers them via HTTP. When a delivery fails, BullMQ's exponential backoff will automatically retry with increasing delays between attempts.

In this step, get ready to:

• Create a BullMQ worker that processes delivery jobs with HTTP POST requests.
• Add idempotency checks to prevent duplicate deliveries.
• Start the worker and verify it processes queued events with retries.

Create the worker job processor

A worker is a separate Node.js process that connects to Redis and pulls jobs from a named queue. For each job, your worker will make an HTTP POST request to the target endpoint URL with the event payload.

If the endpoint returns a non-2xx status code, the worker throws an error. This tells BullMQ to retry the job using the exponential backoff schedule you configured in src/queue.js.

• Create a new file called src/worker.js in VS Code.
• Add the imports, Redis connection, and worker processor by pasting this code:

const { Worker } = require('bullmq');
const IORedis = require('ioredis');
const { updateDelivery, isAlreadyDelivered } = require('./deliveryLog');

// Connect to Redis with the setting BullMQ workers require
const connection = new IORedis({
  host: 'localhost',
  port: 6379,
  maxRetriesPerRequest: null,
});

// Create a worker that processes jobs from the 'webhook-delivery' queue
const worker = new Worker(
  'webhook-delivery',
  async (job) => {
    const { deliveryId, eventId, endpointId, endpointUrl, payload } = job.data;

    // Idempotency check: skip if already delivered
    if (isAlreadyDelivered(eventId, endpointId)) {
      console.log(`Skipping duplicate delivery: ${deliveryId}`);
      updateDelivery(deliveryId, { status: 'delivered', attempts: job.attemptsMade + 1 });
      return { skipped: true, reason: 'already delivered' };
    }

    console.log(
      `Delivering event ${eventId} to ${endpointUrl} (attempt ${job.attemptsMade + 1})`
    );

    // Make the HTTP POST request to the webhook endpoint
    const response = await fetch(endpointUrl, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(payload),
    });

    // If the response is not 2xx, throw to trigger a retry
    if (!response.ok) {
      const errorMsg = `HTTP ${response.status}: ${response.statusText}`;
      updateDelivery(deliveryId, {
        status: 'pending',
        attempts: job.attemptsMade + 1,
        lastError: errorMsg,
      });
      throw new Error(errorMsg);
    }

    // Success: mark delivery as completed
    updateDelivery(deliveryId, {
      status: 'delivered',
      attempts: job.attemptsMade + 1,
    });

    console.log(`Successfully delivered ${deliveryId}`);
    return { delivered: true };
  },
  { connection }
);

What does this code do?

The worker pulls jobs from the webhook-delivery queue in Redis. For each job, it:

• Checks isAlreadyDelivered() to prevent sending duplicate events to the same endpoint. This is the idempotency guard.
• Uses Node.js native fetch to POST the event payload to the registered URL.
• Throws an error on non-2xx responses. BullMQ catches this and schedules a retry using exponential backoff (1s, 2s, 4s, 8s, 16s across 5 attempts).
• Updates the delivery log after each attempt so you can track what happened.

Why throw an error instead of handling it silently?

BullMQ uses thrown errors as the signal to retry a job. If your processor returns normally, BullMQ considers the job complete and moves on. Throwing is how you tell BullMQ "this attempt failed, try again later."

• Don't save the file yet. You'll add event handlers in the next section before running the worker.

Add event handlers for delivery outcomes

BullMQ workers emit events as jobs complete or fail. By listening to these events, you can log progress and mark deliveries as permanently failed when all retry attempts are exhausted.

• Add the following event handlers at the bottom of src/worker.js, below the worker definition:

// Log when a job completes successfully
worker.on('completed', (job) => {
  console.log(`Job ${job.id} completed`);
});

// Handle job failures and mark permanently failed deliveries
worker.on('failed', (job, err) => {
  if (job) {
    console.log(
      `Job ${job.id} failed (attempt ${job.attemptsMade}/${job.opts.attempts}): ${err.message}`
    );
    // If all attempts are exhausted, mark delivery as permanently failed
    if (job.attemptsMade >= job.opts.attempts) {
      updateDelivery(job.data.deliveryId, {
        status: 'failed',
        attempts: job.attemptsMade,
        lastError: err.message,
      });
    }
  }
});

// Catch unexpected worker errors
worker.on('error', (err) => {
  console.error('Worker error:', err);
});

console.log('Webhook delivery worker started');

What do these event handlers do?

• completed fires when a job's processor returns without throwing. It logs the success.
• failed fires on each failed attempt. When job.attemptsMade >= job.opts.attempts, all retries are exhausted and the delivery is marked as permanently failed.
• error catches unexpected worker-level errors (like losing the Redis connection).

• Save src/worker.js.

Seeing a syntax error on save?

• Check that the worker definition ends with { connection } followed by a closing ); before the event handlers.
• Make sure all template literals use backticks (`) and not regular quotes.

Help me debug a syntax error in my BullMQ worker file.

✔️ Awesome, I've got everything!

Great. Make sure you've saved src/worker.js before moving on.

ⓧ I'd like to double check the full code

const { Worker } = require('bullmq');
const IORedis = require('ioredis');
const { updateDelivery, isAlreadyDelivered } = require('./deliveryLog');

const connection = new IORedis({
  host: 'localhost',
  port: 6379,
  maxRetriesPerRequest: null,
});

const worker = new Worker(
  'webhook-delivery',
  async (job) => {
    const { deliveryId, eventId, endpointId, endpointUrl, payload } = job.data;

    if (isAlreadyDelivered(eventId, endpointId)) {
      console.log(`Skipping duplicate delivery: ${deliveryId}`);
      updateDelivery(deliveryId, { status: 'delivered', attempts: job.attemptsMade + 1 });
      return { skipped: true, reason: 'already delivered' };
    }

    console.log(
      `Delivering event ${eventId} to ${endpointUrl} (attempt ${job.attemptsMade + 1})`
    );

    const response = await fetch(endpointUrl, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(payload),
    });

    if (!response.ok) {
      const errorMsg = `HTTP ${response.status}: ${response.statusText}`;
      updateDelivery(deliveryId, {
        status: 'pending',
        attempts: job.attemptsMade + 1,
        lastError: errorMsg,
      });
      throw new Error(errorMsg);
    }

    updateDelivery(deliveryId, {
      status: 'delivered',
      attempts: job.attemptsMade + 1,
    });

    console.log(`Successfully delivered ${deliveryId}`);
    return { delivered: true };
  },
  { connection }
);

worker.on('completed', (job) => {
  console.log(`Job ${job.id} completed`);
});

worker.on('failed', (job, err) => {
  if (job) {
    console.log(
      `Job ${job.id} failed (attempt ${job.attemptsMade}/${job.opts.attempts}): ${err.message}`
    );
    if (job.attemptsMade >= job.opts.attempts) {
      updateDelivery(job.data.deliveryId, {
        status: 'failed',
        attempts: job.attemptsMade,
        lastError: err.message,
      });
    }
  }
});

worker.on('error', (err) => {
  console.error('Worker error:', err);
});

console.log('Webhook delivery worker started');

Start the worker and test the full flow

The worker runs as a separate process from your API server. You need both running at the same time for the system to work end-to-end.

• Open a new terminal in VS Code by pressing Cmd+Shift+` (macOS) or Ctrl+Shift+` (Windows).
• Start the worker by running this command:

node src/worker.js

You should see Webhook delivery worker started printed in the terminal.

• Open a third terminal to send test requests. Register a test endpoint by running:

curl -X POST http://localhost:3000/endpoints \
  -H "Content-Type: application/json" \
  -d '{"url": "http://localhost:4000/webhook"}'

Why port 4000?

Nothing is listening on port 4000 yet. This is intentional. The delivery will fail, and you'll see BullMQ's retry behavior in action. You'll build a test receiver on that port in the next step.

• Send a test event by running:

curl -X POST http://localhost:3000/events \
  -H "Content-Type: application/json" \
  -d '{"type": "order.created", "data": {"orderId": "123"}}'

• Switch back to the terminal running the worker.

You should see the worker attempting delivery, failing, then retrying with increasing delays between attempts. The output will show messages like Delivering event ... to http://localhost:4000/webhook (attempt 1) followed by failure logs.

• Check the delivery status by running this in your third terminal:

curl http://localhost:3000/deliveries | node -e "process.stdin.on('data',d=>console.log(JSON.stringify(JSON.parse(d),null,2)))"

You should see the delivery entry with status: 'pending' and an increasing attempts count as retries progress.

Worker not processing jobs?

• Make sure your API server is still running in the first terminal. The worker needs the server to have queued jobs.
• Check that Redis is running with docker compose ps. Both the server and worker connect to Redis.
• Verify the queue name webhook-delivery matches in both src/queue.js and src/worker.js.

Help me debug why my BullMQ worker isn't processing jobs from the queue.

Your delivery system now has a working retry engine. Next up, you'll build a test receiver that randomly fails requests so you can see successful deliveries alongside retries in action.

Test the Full System End-to-End

Your webhook API accepts events, queues delivery jobs, and the worker processes them with exponential backoff retries. Everything is wired up. The missing piece is a realistic target to deliver events to.

In this step, you will build a test receiver that simulates a flaky endpoint, run the full system across multiple terminals, and observe how retries and idempotency work in practice.

In this step, get ready to:

• Create a test receiver that randomly fails 50% of requests.
• Run the full system and observe retry behavior with exponential backoff.
• Understand how idempotency prevents duplicate deliveries.

Create a test receiver that simulates failures

Real webhook endpoints are unreliable. They time out, return errors, and go offline. To prove your retry logic works, you need an endpoint that fails on purpose.

• Create a new file called src/test-receiver.js inside your webhook-delivery-system project by pasting the following code:

const express = require('express');

const app = express();
app.use(express.json());

// Simulate a flaky endpoint that fails 50% of the time
app.post('/webhook', (req, res) => {
  const shouldFail = Math.random() < 0.5;

  if (shouldFail) {
    console.log(`[RECEIVER] Simulating failure for event: ${req.body.id}`);
    return res.status(500).json({ error: 'Internal server error' });
  }

  console.log(`[RECEIVER] Successfully received event: ${req.body.id}`);
  res.status(200).json({ received: true });
});

const PORT = 4000;
app.listen(PORT, () => {
  console.log(`Test receiver running on http://localhost:${PORT}/webhook`);
});

What does this code do?

This is a minimal Express server that listens for POST requests on /webhook.

• Math.random() < 0.5 creates a coin flip. Half the time the endpoint returns a 500 error, half the time it returns 200 success.
• Console logs label output with [RECEIVER] so you can distinguish this process from your API server and worker.
• Port 4000 avoids conflicts with your API server on port 3000.

• Save src/test-receiver.js.

File not saving?

Make sure you created the file inside the src/ folder, not the project root. The path should be src/test-receiver.js.

Help me troubleshoot

Run the full system and test the delivery flow

You will now run three processes simultaneously: your API server, the delivery worker, and the test receiver. Each needs its own terminal window.

• Stop the server and worker processes that are currently running by pressing Ctrl+C in each of their terminal windows.
• In your first terminal, start the API server by running:

node src/server.js

• Open a second terminal window and start the worker by running:

node src/worker.js

• Open a third terminal window and start the test receiver by running:

node src/test-receiver.js

You should see three processes running with these messages in their respective terminals:

• Webhook API server running on http://localhost:3000 in terminal 1.
• Webhook delivery worker started in terminal 2.
• Test receiver running on http://localhost:4000/webhook in terminal 3.

• Open a fourth terminal window. Register the test receiver as a webhook endpoint by running:

curl -X POST http://localhost:3000/endpoints -H "Content-Type: application/json" -d '{"url": "http://localhost:4000/webhook"}'

You should see a JSON response with an id and the url you registered.

• Send a test event by running:

curl -X POST http://localhost:3000/events -H "Content-Type: application/json" -d '{"type": "payment.completed", "data": {"amount": 49.99}}'

You should see a response like {"eventId":"...","deliveriesQueued":1}.

• Watch the worker terminal (terminal 2) and the receiver terminal (terminal 3).

What should I see?

In the worker terminal, you will see delivery attempts logged with the attempt number. When the receiver returns a 500 error, the worker logs a failure and BullMQ automatically retries after an exponential delay.

In the receiver terminal, you will see [RECEIVER] Simulating failure messages for failed requests and [RECEIVER] Successfully received event for successful ones.

The delays between retries grow: 1 second, 2 seconds, 4 seconds, 8 seconds. This is exponential backoff in action.

• Send a few more events to see more retry patterns by running:

curl -X POST http://localhost:3000/events -H "Content-Type: application/json" -d '{"type": "order.shipped", "data": {"orderId": "abc-123"}}'

• After waiting about 15-30 seconds for retries to complete, check the delivery log by running:

curl http://localhost:3000/deliveries

You should see JSON entries with status values of delivered or pending, along with attempts counts and lastError messages.

Not seeing any output in the worker terminal?

• Make sure Redis is still running. Run docker compose ps to check. If it stopped, run docker compose up -d again.
• Confirm all three Node.js processes started without errors in their terminals.
• Check that you registered the endpoint before sending events. Run curl http://localhost:3000/endpoints to verify.

Help me debug my worker

Understand how idempotency prevents duplicate deliveries

In distributed systems, a job can sometimes be processed more than once. Your worker already has a safeguard built in.

• In src/worker.js, find this code near the top of the job processor function:

if (isAlreadyDelivered(eventId, endpointId)) {
  console.log(`Skipping duplicate delivery: ${deliveryId}`);
  updateDelivery(deliveryId, { status: 'delivered', attempts: job.attemptsMade + 1 });
  return { skipped: true, reason: 'already delivered' };
}

What is idempotency?

Idempotency means running the same operation multiple times produces the same result as running it once. In webhook delivery, this prevents sending duplicate events to an endpoint.

The isAlreadyDelivered function checks whether a specific event has already been successfully delivered to a specific endpoint. It uses the combination of eventId and endpointId as the uniqueness key.

If a job gets processed a second time (due to a queue glitch or restart), the worker skips the duplicate instead of sending the event again. This is how production systems guarantee at-most-once delivery.

• Watch for Skipping duplicate delivery messages in your worker terminal. These appear when BullMQ re-processes a job that already succeeded for a given event and endpoint combination.

Testing idempotency manually

In a production system with a shared database, you could force a duplicate by re-enqueuing the same job. In this local setup with in-memory storage, the idempotency check protects within the worker's process lifetime. The pattern is what matters here. In a real deployment, you would back this with a database lookup.

✔️ Awesome, I've got everything!

Great. Double check you have saved all your files.

ⓧ I'd like to double check the full code

Here is the complete src/test-receiver.js file:

const express = require('express');

const app = express();
app.use(express.json());

app.post('/webhook', (req, res) => {
  const shouldFail = Math.random() < 0.5;

  if (shouldFail) {
    console.log(`[RECEIVER] Simulating failure for event: ${req.body.id}`);
    return res.status(500).json({ error: 'Internal server error' });
  }

  console.log(`[RECEIVER] Successfully received event: ${req.body.id}`);
  res.status(200).json({ received: true });
});

const PORT = 4000;
app.listen(PORT, () => {
  console.log(`Test receiver running on http://localhost:${PORT}/webhook`);
});

All other files (src/server.js, src/worker.js, src/queue.js, src/deliveryLog.js) remain unchanged from the previous step.

You now have a fully working webhook delivery system with queue-based processing, exponential backoff retries, and idempotency protection. Next up, you will take on the Secret Mission and add a dead letter queue for permanently failed deliveries.

Secret mission

Add a Dead Letter Queue with Replay

Right now, when a delivery permanently fails after all 5 retry attempts, it just disappears. In production, that's unacceptable. Build a dead letter queue that captures permanently failed deliveries and an API endpoint to replay them on demand.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing financial costs. The Redis container uses minimal system resources while Docker Desktop is running.

Resources you used:

• Redis container running via Docker Compose on port 6379.
• Docker volume redis-data storing persisted Redis data.
• webhook-delivery-system project folder on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend the project further.

• Your Redis container continues running in the background via Docker Desktop.
• Your project folder and all source files remain on your Desktop.
• Docker Desktop stays installed and running on your machine.
• Node.js remains available for future projects.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your project files and Docker volume stay on disk so you can resume later.

• Stop any running Node.js processes (the API server, worker, or test receiver) by pressing Ctrl+C in each terminal window.
• Stop the Redis container by running this command from your webhook-delivery-system folder:

docker compose down

Your Redis data is preserved in the Docker volume. Next time you return, run docker compose up -d to start Redis again.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your code, the Redis container, and its stored data.

• Stop any running Node.js processes by pressing Ctrl+C in each terminal window.
• Stop and remove the Redis container along with its volume. Navigate to your webhook-delivery-system folder and run:

docker compose down -v

• Delete the project folder. On macOS or Linux, run:

cd ~/Desktop
rm -rf webhook-delivery-system

• On Windows, run:

cd ~\Desktop
rmdir /s /q webhook-delivery-system

• Confirm the folder is gone by running ls (macOS/Linux) or dir (Windows) on your Desktop. You should no longer see webhook-delivery-system in the output.

Nice Work!

Nice Work!

You just built a production-style webhook delivery system from scratch. This is the exact architecture that powers event delivery at companies like Stripe, GitHub, and Shopify.

You learned how to:

• Build a REST API that accepts events and queues them for asynchronous delivery using BullMQ and Redis.
• Process delivery jobs with exponential backoff retries and track each attempt with per-delivery status logging.
• Implement idempotency checks to prevent duplicate deliveries and run Docker Compose for local Redis infrastructure.
• Add a dead letter queue that captures permanently failed deliveries and a replay endpoint to re-attempt them on demand.

These patterns (queue-based async processing, exponential backoff, idempotency, dead letter queues) show up in nearly every production backend system. You will recognize them everywhere now.

Ready to quiz yourself?