Build a Real-Time Webhook Relay

Build a webhook relay with retry logic, dead letter queues, and a live dashboard.

Introduction

30 Second Summary

When you order something online, a dozen different systems need to talk to each other: the payment processor notifies the warehouse, the warehouse notifies the shipping service, and so on. If any of those systems are temporarily down, messages get lost forever. A webhook relay service solves this by sitting in the middle, catching every message, and retrying until it gets through.

In this project, you will build a real-time webhook relay service from scratch using Node.js and SQLite. Your relay receives HTTP payloads, queues them, retries failed deliveries with increasing wait times, and shows everything happening live on a dashboard.

What You'll Build

Picture opening a dashboard in your browser, clicking a button, and watching a webhook get queued, delivered, confirmed, retried, or moved to a dead letter pile. All in real-time, without ever refreshing the page.

By the end of this project, you'll have:

  • A working relay that receives test webhooks and delivers them to a destination. You can send a message, watch it flow through the job queue, and see confirmation appear on a live dashboard.
  • Automatic retry with exponential backoff. You can simulate a broken destination and watch your relay try again with increasing delays (2s, 4s, 8s, 16s) until it either succeeds or gives up.
  • A dead letter queue for permanently failed messages. You can inspect why they failed and manually retry them with a single click.
  • A real-time dashboard powered by Server-Sent Events that updates live without refreshing. No external libraries needed.
  • Secret Mission: Add rate limiting so a single sender can't flood your system with too many requests.

Are there any prerequisites?

You should have basic JavaScript knowledge (variables, functions, and objects). You also need Node.js installed on your machine. The project will walk you through everything else.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up the Project

Before you can relay webhooks between apps, you need a foundation to build on. Express handles incoming HTTP requests. SQLite stores the queue of messages waiting to be delivered.

In this step, you'll create the project folder, install both packages, and wire up the initial server and database schema.

In this step, get ready to:
  • Create the webhook-relay project and install dependencies.
  • Define the database schema with webhooks and dead_letters tables.
  • Start the server and confirm it connects to the database.
Create the project folder and install dependencies
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Open a terminal inside VS Code by pressing Cmd+` (macOS) or Ctrl+` (Windows).

What is the integrated terminal?

VS Code has a built-in terminal panel at the bottom of the window. It works exactly like a standalone terminal, but you can see your code and your commands side by side.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir webhook-relay
cd webhook-relay
  • Initialize a new Node.js project by running:
npm init -y

You'll see output confirming a package.json file was created in your folder.

  • Install Express (the web framework) and better-sqlite3 (the database) by running:
npm install express@5.2.1 better-sqlite3@13.0.1
  • Confirm the packages installed by running:
ls node_modules

You should see a list of folders including express and better-sqlite3 in the output.

Seeing an error during install?

If you see a permissions error on macOS, try running the install command with sudo in front. If you see a network error, check your internet connection and try again.

If better-sqlite3 fails with a build error, make sure you have Node.js 18 or higher by running node --version.

  • Open the webhook-relay folder in VS Code by clicking File in the top menu bar.
  • Click Open Folder.
  • Navigate to Desktop, select the webhook-relay folder, and click Open.

VS Code now shows the project files in the left Explorer sidebar.

  • Create a new file by clicking the new file icon in the Explorer sidebar.
  • Name the file server.js and paste the following code into it:
const express = require('express');

const app = express();
const PORT = 3000;

// Parse JSON request bodies
app.use(express.json());

app.listen(PORT, () => {
  console.log(`Webhook Relay running on http://localhost:${PORT}`);
});

What does this code do?

  • require('express') loads the Express web framework you just installed.
  • express.json() tells Express to automatically parse incoming JSON request bodies.
  • app.listen(PORT) starts the server on port 3000 and prints a confirmation message.
  • Save server.js by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • If your terminal closed when you opened the folder, reopen it with Cmd+` (macOS) or Ctrl+` (Windows).
  • Start your server by running:
node server.js

You should see Webhook Relay running on http://localhost:3000 printed in your terminal.

  • Stop the server by pressing Ctrl+C in the terminal.

Don't see the message?

Make sure you saved server.js before running the command. Check that you're in the webhook-relay folder by running pwd in your terminal.

If you see "Cannot find module 'express'", run npm install express@5.2.1 again.

Set up the database schema

Your relay needs to remember every webhook it receives, even if delivery fails. A database stores each webhook's payload, destination, delivery status, and retry count.

  • Create a new file called database.js in your webhook-relay folder by clicking the new file icon in the Explorer sidebar.
  • Paste the following code to initialize the database and create the webhooks table:
const Database = require('better-sqlite3');
const path = require('path');

// Create (or open) the SQLite database file
const db = new Database(path.join(__dirname, 'relay.db'));

// Enable WAL mode for better performance with concurrent reads/writes
db.pragma('journal_mode = WAL');

// Create the webhooks table - this is our job queue
db.exec(`
  CREATE TABLE IF NOT EXISTS webhooks (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    url TEXT NOT NULL,
    body TEXT NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending',
    attempts INTEGER NOT NULL DEFAULT 0,
    max_attempts INTEGER NOT NULL DEFAULT 5,
    next_attempt TEXT NOT NULL DEFAULT (datetime('now')),
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    last_error TEXT
  )
`);

What does this code do?

  • new Database(...) creates a file called relay.db in your project folder. If it already exists, it opens it.
  • WAL mode (Write-Ahead Logging) lets reads and writes happen at the same time without blocking each other.
  • The webhooks table is your job queue. Each row is a webhook waiting to be delivered, with columns tracking its status, retry count, and when to try again.
  • Add the dead letters table and the module export below the webhooks table code:
// Create the dead letters table - where permanently failed webhooks go
db.exec(`
  CREATE TABLE IF NOT EXISTS dead_letters (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    original_id INTEGER NOT NULL,
    url TEXT NOT NULL,
    body TEXT NOT NULL,
    attempts INTEGER NOT NULL,
    last_error TEXT,
    failed_at TEXT NOT NULL DEFAULT (datetime('now'))
  )
`);

module.exports = db;

What is a dead letter queue?

When a webhook fails delivery too many times, it gets moved to the dead_letters table. This is a common pattern in production systems.

Instead of losing the message forever, you store it separately so a human can inspect it, fix the problem, and retry it manually.

  • Save database.js.
  • Verify the database loads without errors by running:
node -e "require('./database')"

If no error appears and the terminal returns to the prompt, your database schema is correct. You should also see a new relay.db file appear in your Explorer sidebar.

Seeing a module not found error?

If you see "Cannot find module 'better-sqlite3'", make sure you ran npm install in the webhook-relay folder. Check that node_modules exists by running ls.

If you see a SQL syntax error, double check that your table definitions match exactly.

✔️ Awesome, I've got everything!

Make sure database.js is saved and the relay.db file appeared in your folder.

ⓧ I'd like to double check the full code

Your database.js should contain these sections in order:

  • The require statements and database initialization with WAL mode (first chunk above).
  • The webhooks table creation (end of first chunk).
  • The dead_letters table creation and module.exports = db (second chunk above).

Scroll up and confirm each chunk is present, in order.

Verify everything works together

Your server and database exist as separate files. Now you'll connect them and add a health check endpoint so you can confirm everything is wired up.

  • Open server.js and replace its entire contents with the following:
const express = require('express');
const path = require('path');
const db = require('./database');

const app = express();
const PORT = 3000;

// Parse JSON request bodies
app.use(express.json());

// --- Health Check ---
app.get('/health', (req, res) => {
  res.json({ status: 'running', uptime: process.uptime() });
});

app.listen(PORT, () => {
  console.log(`Webhook Relay running on http://localhost:${PORT}`);
});

What changed?

  • require('./database') loads your database module. This triggers the table creation code and makes the database ready for use.
  • The /health route returns a JSON object with the server's status and uptime. This gives you a quick way to confirm the server is running and the database loaded without errors.
  • Save server.js.

Before you run the server, what do you expect to see printed in your terminal?

  • Start the server by running:
node server.js

You should see Webhook Relay running on http://localhost:3000 printed in the terminal.

  • Open your browser and visit http://localhost:3000/health.

You should see a JSON response like {"status":"running","uptime":...} in your browser. That confirms Express and the database module are both loaded and working.

Seeing an error when starting the server?

If you see "Cannot find module './database'", make sure database.js exists in the same folder as server.js. Check that the spelling matches exactly.

If you see a SQL-related error, delete relay.db from your folder and try running node server.js again to recreate it fresh.

✔️ Awesome, I've got everything!

Great. Keep your server running for now.

ⓧ I'd like to double check the full code

Here is the complete server.js at this point:

const express = require('express');
const path = require('path');
const db = require('./database');

const app = express();
const PORT = 3000;

// Parse JSON request bodies
app.use(express.json());

// --- Health Check ---
app.get('/health', (req, res) => {
  res.json({ status: 'running', uptime: process.uptime() });
});

app.listen(PORT, () => {
  console.log(`Webhook Relay running on http://localhost:${PORT}`);
});

Your foundation is in place. You have a running server connected to a database with two tables ready to hold webhook data. Next up, you'll build the relay endpoint that actually receives and forwards webhooks.

Build the Basic Relay

Your server is running and your database is ready to store webhooks. Now it is time to build the core relay logic: an endpoint that accepts incoming webhooks, stores them, and immediately attempts to deliver them to their destination.

This is the naive version of the relay. It works perfectly when the destination is up. When the destination is down, the webhook is lost forever. You will feel that limitation by the end of this step.

In this step, get ready to:
  • Build a POST /relay endpoint that accepts and delivers webhooks.
  • Create test destinations that simulate working and broken servers.
  • Test both paths and feel the problem with no retry logic.
Build the ingestion endpoint

The relay endpoint is the front door of your service. It accepts a JSON payload with two fields: a destination URL and a body to deliver. It stores the webhook in your database, then immediately tries to forward it.

  • In server.js, add the following relay endpoint between app.use(express.json()) and app.get('/health', ...):
// --- Webhook Relay Endpoint ---
app.post('/relay', async (req, res) => {
  const { url, body } = req.body;

  // Reject requests missing required fields
  if (!url || !body) {
    return res.status(400).json({ error: 'Missing required fields: url and body' });
  }

  // Store the webhook in the database as pending
  const result = db.prepare(`INSERT INTO webhooks (url, body, status, next_attempt) VALUES (?, ?, 'pending', datetime('now'))`).run(url, JSON.stringify(body));
  const id = result.lastInsertRowid;

  // Immediately attempt delivery to the destination
  try {
    const response = await fetch(url, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
    if (!response.ok) throw new Error(`Destination responded with ${response.status}`);
    db.prepare(`UPDATE webhooks SET status = 'completed' WHERE id = ?`).run(id);
    res.status(202).json({ message: 'Webhook delivered', id: Number(id) });
  } catch (err) {
    // On failure, mark as failed and store the error
    db.prepare(`UPDATE webhooks SET status = 'failed', last_error = ? WHERE id = ?`).run(err.message, id);
    res.status(202).json({ message: 'Webhook failed', id: Number(id) });
  }
});

What does this code do?

  • db.prepare(...).run() inserts a row into the webhooks table with status pending before attempting delivery.
  • fetch() sends the webhook body as a POST request to the destination URL. This is Node's built-in HTTP client.
  • If the destination responds with a non-2xx status, the code throws an error and marks the webhook as failed.
  • The endpoint always responds with 202 Accepted because the webhook was received regardless of delivery outcome.
Create test destinations

To test the relay, you need controllable targets. One always succeeds and one always fails.

  • In server.js, add these two routes directly above your /relay endpoint:
// --- Test Destinations ---
// A destination that always works
app.post('/test-destination', (req, res) => {
  res.json({ received: true });
});

// A destination that always fails
app.post('/test-destination-broken', (req, res) => {
  res.status(500).json({ error: 'Service unavailable' });
});

What does this code do?

/test-destination simulates a healthy server by always returning 200 OK. /test-destination-broken simulates a crashed server by always returning 500 Internal Server Error. These let you test both success and failure paths without relying on any external service.

  • Save server.js.

✔️ Awesome, I've got everything!

Great. Make sure you have saved server.js before continuing.

ⓧ I'd like to double check the full code

Scroll up and confirm your server.js contains the following sections in this order:

  • The require statements and app.use(express.json()) at the top.
  • The /test-destination and /test-destination-broken routes.
  • The POST /relay endpoint with the async handler.
  • The /health route.
  • The app.listen() call at the bottom.
Test both paths

Time to see the relay in action. You will send a webhook to the working destination, then send one to the broken destination.

  • If your server is still running, stop it by pressing Ctrl+C in the terminal.
  • Restart the server by running:
node server.js
  • You should see Webhook Relay running on http://localhost:3000 in the terminal.
  • Open a second terminal window for sending test requests.
  • Send a webhook to the working destination by running:

macOS

curl -X POST http://localhost:3000/relay \
  -H "Content-Type: application/json" \
  -d '{"url": "http://localhost:3000/test-destination", "body": {"message": "hello"}}'

Windows

Invoke-RestMethod -Method POST -Uri "http://localhost:3000/relay" -ContentType "application/json" -Body '{"url": "http://localhost:3000/test-destination", "body": {"message": "hello"}}'

You should see a response containing "message": "Webhook delivered" and an id number. The relay accepted the webhook, stored it, delivered it to the working destination, and marked it complete.

What just happened?

Your relay received the request, inserted a row into the webhooks table with status pending, then immediately called fetch() to POST the body to /test-destination. Since that endpoint returned 200, the status was updated to completed.

Now, before you send a webhook to the broken destination, what do you think will happen when the destination returns a 500 error?

  • Send a webhook to the broken destination by running:

macOS

curl -X POST http://localhost:3000/relay \
  -H "Content-Type: application/json" \
  -d '{"url": "http://localhost:3000/test-destination-broken", "body": {"message": "hello"}}'

Windows

Invoke-RestMethod -Method POST -Uri "http://localhost:3000/relay" -ContentType "application/json" -Body '{"url": "http://localhost:3000/test-destination-broken", "body": {"message": "hello"}}'

You should see "message": "Webhook failed" in the response. The webhook was stored, delivery was attempted once, it failed, and the relay gave up immediately.

This is the problem

One failure and the message is lost forever. The relay tried exactly once, the destination was down, and that was it. No second chance. No retry.

In a real system, destinations go down temporarily all the time. A network blip, a deploy, a brief overload. A production-grade relay would wait and try again. That is exactly what you will build in the next step.

Not seeing the expected response?

  • Make sure you restarted the server after saving your changes. Press Ctrl+C then run node server.js again.
  • Check that the URL in your test command uses http://localhost:3000 (not https).
  • Confirm app.use(express.json()) appears before your routes. Without it, req.body will be undefined.

Help me debug my webhook relay endpoint.

Your relay works, but it has no resilience. Next, you will add a queue and retry engine that uses exponential backoff to give failed webhooks multiple chances to deliver.

Add the Queue and Retry Engine

In Step 2, you felt the limitation firsthand: your relay tries to deliver a webhook exactly once. If the destination is down, the message is gone forever.

A production-grade system does not give up on the first failure. It retries with exponential backoff: wait 2 seconds, then 4, then 8, then 16. Each attempt doubles the wait, giving the destination time to recover. In this step, you will build that retry intelligence from scratch.

In this step, get ready to:
  • Write a delay calculator that doubles the wait time after each failed attempt.
  • Build a processing loop that retries pending webhooks and moves permanently failed ones to a dead letter queue.
  • Wire the queue processor into your server and watch retries happen in real time.
Write the retry calculator

Your retry engine needs two pieces of logic: a function that calculates how long to wait before the next attempt, and a function that actually tries to deliver the payload.

  • Create a new file called queue.js in your webhook-relay folder.
  • Add the database import and the delay calculator by pasting this code:
const db = require('./database');

// Calculate retry delay using exponential backoff with jitter
function calculateDelay(attempt) {
  const baseDelay = 2000;
  const maxDelay = 30000;
  const exponentialDelay = Math.min(baseDelay * Math.pow(2, attempt), maxDelay);
  const jitter = Math.floor(Math.random() * 1000);
  return exponentialDelay + jitter;
}

What does this code do?

The calculateDelay function takes the current attempt number and returns a delay in milliseconds.

  • Attempt 1 waits ~2 seconds. Attempt 2 waits ~4 seconds. Attempt 3 waits ~8 seconds. The delay doubles each time, capped at 30 seconds.
  • The random jitter (0 to 1000ms) prevents a "thundering herd" where hundreds of retries all fire at the same instant.
  • Below calculateDelay, add the delivery function that attempts to send a webhook to its destination:
// Attempt to deliver a webhook payload to its destination URL
async function deliverWebhook(webhook) {
  try {
    const response = await fetch(webhook.url, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: webhook.body,
      signal: AbortSignal.timeout(10000)
    });
    if (!response.ok) throw new Error(`Destination responded with ${response.status}`);
    return { success: true };
  } catch (error) {
    return { success: false, error: error.message };
  }
}

What does this code do?

The deliverWebhook function POSTs the webhook body to its URL. It returns a simple object: { success: true } or { success: false, error: '...' }.

The AbortSignal.timeout(10000) cancels the request if the destination takes longer than 10 seconds to respond. Without it, a hanging destination could stall your entire queue.

  • Save queue.js.
  • Verify the module loads without errors by running this command:
node -e "require('./queue'); console.log('queue module loaded')"

You should see queue module loaded printed with no errors.

Seeing a syntax error?

Check that queue.js is saved in the same folder as database.js. The require('./database') path is relative to the file's location.

Also check for missing closing braces or parentheses in deliverWebhook.

help me debug a syntax error in my queue.js file

Build the processing loop

The processing loop is the engine that makes retries happen automatically. Every 2 seconds, it checks for pending webhooks and decides what to do with each one: deliver, retry, or give up.

  • In queue.js, add the dead letter function below deliverWebhook:
// Move a permanently failed webhook to the dead letter queue
function moveToDeadLetters(webhook, error) {
  db.prepare(`INSERT INTO dead_letters (original_id, url, body, attempts, last_error) VALUES (?, ?, ?, ?, ?)`).run(webhook.id, webhook.url, webhook.body, webhook.attempts, error);
  db.prepare(`UPDATE webhooks SET status = 'failed' WHERE id = ?`).run(webhook.id);
}

What does this code do?

When a webhook exhausts all its retry attempts, moveToDeadLetters copies it into the dead_letters table and marks the original as failed. This lets you inspect permanently failed messages later and decide what to do with them.

  • Below moveToDeadLetters, add the main processing loop that ties everything together:
// Process all pending webhooks that are ready for their next attempt
async function processQueue(broadcast) {
  const pending = db.prepare(`SELECT * FROM webhooks WHERE status = 'pending' AND datetime(next_attempt) <= datetime('now')`).all();
  for (const webhook of pending) {
    db.prepare(`UPDATE webhooks SET status = 'delivering' WHERE id = ?`).run(webhook.id);
    broadcast({ type: 'webhook_update', data: { id: webhook.id, status: 'delivering', attempts: webhook.attempts } });
    const result = await deliverWebhook(webhook);
    if (result.success) {
      db.prepare(`UPDATE webhooks SET status = 'completed' WHERE id = ?`).run(webhook.id);
      broadcast({ type: 'webhook_update', data: { id: webhook.id, status: 'completed', attempts: webhook.attempts + 1 } });
      console.log(`[DELIVERED] Webhook #${webhook.id} succeeded on attempt ${webhook.attempts + 1}`);
    } else {
      const newAttempts = webhook.attempts + 1;
      if (newAttempts >= webhook.max_attempts) {
        // All retries exhausted - move to dead letters
        moveToDeadLetters({ ...webhook, attempts: newAttempts }, result.error);
        broadcast({ type: 'webhook_update', data: { id: webhook.id, status: 'failed', attempts: newAttempts, error: result.error } });
        console.log(`[DEAD LETTER] Webhook #${webhook.id} failed permanently after ${newAttempts} attempts`);
      } else {
        // Schedule next retry with exponential backoff
        const delay = calculateDelay(newAttempts);
        const nextAttempt = new Date(Date.now() + delay).toISOString();
        db.prepare(`UPDATE webhooks SET status = 'pending', attempts = ?, next_attempt = ?, last_error = ? WHERE id = ?`).run(newAttempts, nextAttempt, result.error, webhook.id);
        broadcast({ type: 'webhook_update', data: { id: webhook.id, status: 'pending', attempts: newAttempts, next_attempt: nextAttempt, error: result.error } });
        console.log(`[RETRY] Webhook #${webhook.id} attempt ${newAttempts} failed. Next retry in ${Math.round(delay / 1000)}s`);
      }
    }
  }
}

What does this code do?

The processQueue function is the heart of the retry engine. Every time it runs, it:

  • Queries the database for webhooks with status pending whose next_attempt time has passed.
  • Marks each one as delivering so it won't get picked up again by a concurrent run.
  • On success, marks it completed.
  • On failure, checks whether the max attempts (5) have been reached. If yes, it moves to dead letters. If no, it schedules the next retry using the exponential backoff delay.

The broadcast parameter is a function that will push updates to the live dashboard later. For now, it will be a no-op stub.

  • Save queue.js.
  • Verify the module still loads without errors by running:
node -e "require('./queue'); console.log('queue module loaded')"

You should see queue module loaded again with no errors.

  • Now add the function that starts the processing loop and the module exports at the bottom of queue.js:
// Start the queue processor on a 2-second polling interval
function startQueueProcessor(broadcast) {
  setInterval(() => processQueue(broadcast), 2000);
  console.log('[QUEUE] Processing loop started (checking every 2 seconds)');
}

module.exports = { startQueueProcessor, processQueue };

What does this code do?

The startQueueProcessor function kicks off a setInterval that calls processQueue every 2000 milliseconds (2 seconds). This is a simple polling loop. In a production system you might use a dedicated job queue like BullMQ with Redis, but polling is simpler and teaches the same underlying pattern.

  • Save queue.js.

✔️ Awesome, I've got everything!

Great. Confirm you have saved queue.js and move on to wiring it into your server.

ⓧ I'd like to double check the full code

Scroll up through your queue.js and confirm these sections appear in order:

  • The require('./database') import at the top.
  • The calculateDelay function.
  • The deliverWebhook async function.
  • The moveToDeadLetters function.
  • The processQueue async function.
  • The startQueueProcessor function and module.exports at the bottom.
Wire it up and test the retry flow

The queue engine is ready but it is not connected to anything yet. You need to update server.js so it imports the queue, starts the processor when the server boots, and queues webhooks instead of delivering them immediately.

  • In server.js, find the require lines at the top of the file:
const express = require('express');
const path = require('path');
const db = require('./database');
  • Add a new import for the queue module directly below those lines:
const { startQueueProcessor } = require('./queue');
  • Find the app.use(express.json()); line and add a broadcast stub function directly below it:
// Placeholder for live dashboard updates (built in the next step)
function broadcast(event) {}

Why is broadcast empty?

The queue processor calls broadcast() to push status updates to connected clients. You will build the real broadcasting system with Server-Sent Events in the next step. For now, an empty function prevents errors.

  • Now replace the entire app.post('/relay', async (req, res) => { ... }); route with this new version that queues instead of delivering immediately:
// Queue the webhook for async delivery instead of trying immediately
app.post('/relay', (req, res) => {
  const { url, body } = req.body;
  if (!url || !body) {
    return res.status(400).json({ error: 'Missing required fields: url and body' });
  }
  const result = db.prepare(`INSERT INTO webhooks (url, body, status, next_attempt) VALUES (?, ?, 'pending', datetime('now'))`).run(url, JSON.stringify(body));
  const webhook = { id: result.lastInsertRowid, url, body, status: 'pending', attempts: 0 };
  broadcast({ type: 'webhook_update', data: { id: Number(webhook.id), status: 'pending', attempts: 0 } });
  console.log(`[INGESTED] Webhook #${webhook.id} queued for delivery to ${url}`);
  res.status(202).json({ message: 'Webhook accepted and queued for delivery', id: Number(webhook.id) });
});

What changed in the relay route?

The old version tried to deliver the webhook immediately and marked it failed right away if the destination was down. The new version only inserts the webhook into the database and responds with 202 Accepted.

The actual delivery now happens asynchronously via processQueue. This separation between "accepting work" and "doing work" is a core job queue pattern.

  • Find the app.listen block at the bottom of server.js:
app.listen(PORT, () => {
  console.log(`Webhook Relay running on http://localhost:${PORT}`);
});
  • Replace it with this version that starts the queue processor on boot:
app.listen(PORT, () => {
  console.log(`Webhook Relay running on http://localhost:${PORT}`);
  startQueueProcessor(broadcast);
});
  • Save server.js.

✔️ Awesome, I've got everything!

Great. Confirm server.js is saved and move on to testing.

ⓧ I'd like to double check the full code

Scroll through your server.js and confirm these sections appear in order:

  • Four require lines at the top (express, path, database, queue).
  • The app and PORT declarations.
  • The express.json() middleware and broadcast stub.
  • The two test destination routes.
  • The updated POST /relay route (no async, no fetch).
  • The GET /health route.
  • The app.listen block with startQueueProcessor(broadcast) inside the callback.

Time to see the retry engine in action. Before you run the test, think about this: what do you expect to see when a webhook is sent to the broken destination?

  • First, delete the old relay.db file so you start with a clean database. The server will recreate it on boot:
rm relay.db
  • Start the server by running:
node server.js

You should see two lines:

Webhook Relay running on http://localhost:3000
[QUEUE] Processing loop started (checking every 2 seconds)
  • Open a second terminal window. Send a webhook to the broken destination by running:
curl -X POST http://localhost:3000/relay -H "Content-Type: application/json" -d "{\"url\": \"http://localhost:3000/test-destination-broken\", \"body\": {\"message\": \"test retry\"}}"

Switch back to the terminal running your server. Watch the logs over the next 30 to 60 seconds.

You will see retry attempts with increasing delays:

[INGESTED] Webhook #1 queued for delivery to http://localhost:3000/test-destination-broken
[RETRY] Webhook #1 attempt 1 failed. Next retry in 5s
[RETRY] Webhook #1 attempt 2 failed. Next retry in 9s
[RETRY] Webhook #1 attempt 3 failed. Next retry in 17s
[RETRY] Webhook #1 attempt 4 failed. Next retry in 30s
[DEAD LETTER] Webhook #1 failed permanently after 5 attempts

Why do the delay numbers vary?

The exact seconds you see will differ slightly from the example above. That is the jitter at work. Each retry adds a random 0 to 1000ms so that multiple failing webhooks do not all retry at exactly the same moment.

  • Now send a webhook to the working destination to confirm successful delivery still works:
curl -X POST http://localhost:3000/relay -H "Content-Type: application/json" -d "{\"url\": \"http://localhost:3000/test-destination\", \"body\": {\"message\": \"test success\"}}"

Within 2 seconds, the server log should show:

[INGESTED] Webhook #2 queued for delivery to http://localhost:3000/test-destination
[DELIVERED] Webhook #2 succeeded on attempt 1
  • Stop the server with Ctrl+C when you are done observing.

Not seeing any retry logs?

If the server starts but you never see [RETRY] lines, check that startQueueProcessor(broadcast) is inside the app.listen callback in server.js. If it is outside the callback, the function reference may not be ready.

If you see [INGESTED] but no retries after 4 seconds, make sure processQueue queries for status = 'pending' in its SQL. If the relay route accidentally marks the webhook with a different status, the processor won't find it.

help me debug why my webhook retry loop isn't running

Your relay now retries intelligently instead of giving up on the first failure. Next up, you will build a live dashboard so you can watch all of this happening in real time without reading terminal logs.

Build the Live Dashboard

Right now you can only monitor your webhook relay by reading terminal logs. That works for debugging, but it tells you nothing at a glance.

A production-quality system has built-in observability. In this step you will add a live dashboard that updates in real-time using Server-Sent Events (SSE). SSE is a browser-native protocol where the server pushes data to the page without the page needing to refresh or poll.

In this step, get ready to:
  • Set up an SSE endpoint that streams live updates to connected browsers.
  • Build a dashboard page with stat cards, test buttons, and a live event log.
  • Wire the relay and queue to the dashboard so every status change appears instantly.
Build the SSE endpoint

Your server.js currently has a stub broadcast function that does nothing. You need to replace it with a real implementation that streams events to every connected browser tab.

  • In server.js, find this code near the top of the file:
function broadcast(event) {}
  • Replace it with the following code that also adds the static file middleware and the clients tracking array:
// Serve static files (dashboard)
app.use(express.static(path.join(__dirname, 'public')));

// --- Server-Sent Events (SSE) ---
// Keep track of all connected dashboard clients
const clients = [];

function broadcast(event) {
  const data = `data: ${JSON.stringify(event)}\n\n`;
  clients.forEach(client => client.write(data));
}

What does this code do?

The clients array holds every open response object from browsers that are listening for updates.

The broadcast() function sends an SSE-formatted message to every connected client simultaneously. The data: ...\n\n format is the SSE protocol. The double newline tells the browser that one event is complete.

The express.static line tells Express to serve any file inside the public/ folder as a static asset. This is how the dashboard HTML page will be accessible in the browser.

  • Now add the SSE connection endpoint. Place this code directly after the broadcast function and before the test destination routes:
// SSE endpoint - clients connect here to receive live updates
app.get('/events', (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  // Send a comment to establish the connection
  res.write(':\n\n');

  clients.push(res);
  console.log(`[SSE] Client connected (${clients.length} total)`);

  req.on('close', () => {
    const index = clients.indexOf(res);
    if (index !== -1) clients.splice(index, 1);
    console.log(`[SSE] Client disconnected (${clients.length} total)`);
  });
});

What does this code do?

The three headers (text/event-stream, no-cache, keep-alive) tell the browser this is an SSE stream, not a normal response. The connection stays open indefinitely.

When a browser connects, its response object is pushed into the clients array. When the browser disconnects (tab closed, network drop), the close event fires and the client is removed. This prevents memory leaks.

  • Save server.js.
  • Stop your running server with Ctrl+C and restart it by running:
node server.js
  • Open a new browser tab and navigate to http://localhost:3000/events.

You should see a blank page that never finishes loading (the spinner keeps spinning). This is the SSE connection staying open, waiting for events. That confirms the endpoint works.

Page returns an error instead of staying open?

  • Make sure you placed the app.get('/events', ...) route before the app.listen call at the bottom of the file.
  • Check the terminal for error messages. A syntax error in the SSE headers will prevent the server from starting.

Why won't my SSE endpoint connect in the browser?

Create the dashboard HTML page

Now you need a front-end page that connects to the SSE stream and shows live webhook status. You will create an HTML file inside a new public/ folder.

  • Create the public folder inside your webhook-relay project by running:
mkdir public
  • Confirm the folder exists by running:
ls

You should see public listed alongside your other project files.

  • Create a new file called dashboard.html inside the public/ folder. In VS Code, right-click the public folder in the Explorer sidebar and select New File.
  • Name it dashboard.html and add the page skeleton with the layout styles:
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Webhook Relay Dashboard</title>
  <style>
    * { margin: 0; padding: 0; box-sizing: border-box; }
    body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #1a1a2e; color: #e0e0e0; padding: 20px; }
    h1 { color: #00d4aa; margin-bottom: 20px; }
    .stats { display: flex; gap: 15px; margin-bottom: 20px; flex-wrap: wrap; }
    .stat-card { background: #16213e; border-radius: 8px; padding: 15px 20px; min-width: 120px; }
    .stat-card .label { font-size: 12px; color: #888; text-transform: uppercase; }
    .stat-card .value { font-size: 28px; font-weight: bold; color: #00d4aa; }
    .stat-card.failed .value { color: #ff6b6b; }
    .stat-card.pending .value { color: #ffd93d; }
    .controls { margin-bottom: 20px; }
    button { background: #00d4aa; color: #1a1a2e; border: none; padding: 10px 20px; border-radius: 5px; cursor: pointer; font-weight: bold; margin-right: 10px; }
    button:hover { background: #00b894; }
    button.danger { background: #ff6b6b; color: white; }
    button.danger:hover { background: #e74c3c; }
    .section { background: #16213e; border-radius: 8px; padding: 20px; margin-bottom: 20px; }
    .section h2 { color: #00d4aa; margin-bottom: 10px; font-size: 16px; }
    .event-log { max-height: 400px; overflow-y: auto; }
    .event { padding: 8px 12px; border-bottom: 1px solid #2a2a4a; font-family: monospace; font-size: 13px; }

What do these styles do?

This sets up a dark theme with a navy background. The .stat-card classes create the status counter boxes (pending, delivering, completed, failed). The .section class wraps the event log and dead letters areas in rounded panels.

  • Directly below the code you just pasted, add the remaining styles, close the head, and add the body HTML structure:
    .event.completed { border-left: 3px solid #00d4aa; }
    .event.failed { border-left: 3px solid #ff6b6b; }
    .event.pending { border-left: 3px solid #ffd93d; }
    .event.delivering { border-left: 3px solid #74b9ff; }
    .event .time { color: #888; margin-right: 10px; }
    .dead-letter { padding: 10px; border-bottom: 1px solid #2a2a4a; display: flex; justify-content: space-between; align-items: center; }
    .dead-letter-info { flex: 1; }
    .dead-letter-info .url { color: #74b9ff; font-family: monospace; font-size: 12px; }
    .dead-letter-info .error { color: #ff6b6b; font-size: 12px; margin-top: 4px; }
    .connection-status { font-size: 12px; margin-bottom: 10px; }
    .connection-status.connected { color: #00d4aa; }
    .connection-status.disconnected { color: #ff6b6b; }
  </style>
</head>
<body>
  <h1>Webhook Relay Dashboard</h1>
  <div class="connection-status disconnected" id="connection-status">Disconnected</div>
  <div class="stats">
    <div class="stat-card pending"><div class="label">Pending</div><div class="value" id="stat-pending">0</div></div>
    <div class="stat-card"><div class="label">Delivering</div><div class="value" id="stat-delivering">0</div></div>
    <div class="stat-card"><div class="label">Completed</div><div class="value" id="stat-completed">0</div></div>
    <div class="stat-card failed"><div class="label">Failed</div><div class="value" id="stat-failed">0</div></div>
    <div class="stat-card failed"><div class="label">Dead Letters</div><div class="value" id="stat-dead-letters">0</div></div>
  </div>
  <div class="controls">
    <button onclick="sendTestWebhook(true)">Send Test (Working)</button>
    <button class="danger" onclick="sendTestWebhook(false)">Send Test (Broken)</button>
  </div>
  <div class="section">
    <h2>Live Event Log</h2>
    <div class="event-log" id="event-log"></div>
  </div>
  <div class="section">
    <h2>Dead Letters</h2>
    <div id="dead-letters-list"></div>
  </div>

What does this HTML structure do?

The body has four main sections. The stat cards show live counts for each webhook status. The controls section has two buttons that fire test webhooks (one to a working destination, one to a broken one). The event log shows a running feed of status changes. The dead letters section lists permanently failed webhooks.

  • Now add the JavaScript that connects to the SSE stream and handles incoming events. Paste this directly below the dead letters div:
  <script>
    function setupEventSource() {
      const eventSource = new EventSource('/events');
      eventSource.onopen = () => {
        document.getElementById('connection-status').textContent = 'Connected (live)';
        document.getElementById('connection-status').className = 'connection-status connected';
      };
      eventSource.onmessage = (event) => {
        const parsed = JSON.parse(event.data);
        if (parsed.type === 'webhook_update') {
          addEventToLog(parsed.data);
          refreshStats();
        }
      };
      eventSource.onerror = () => {
        document.getElementById('connection-status').textContent = 'Disconnected (reconnecting...)';
        document.getElementById('connection-status').className = 'connection-status disconnected';
      };
      return eventSource;
    }

What is EventSource?

EventSource is a built-in browser API for receiving server-sent events. You give it a URL and it opens a persistent connection. When the server pushes a message, the onmessage callback fires. If the connection drops, the browser automatically reconnects.

  • Continue pasting below. Add the functions that update the event log and refresh the stat counters:
    function addEventToLog(data) {
      const log = document.getElementById('event-log');
      const entry = document.createElement('div');
      entry.className = `event ${data.status}`;
      const time = new Date().toLocaleTimeString();
      let message = `#${data.id} - ${data.status.toUpperCase()}`;
      if (data.attempts) message += ` (attempt ${data.attempts})`;
      if (data.error) message += ` - ${data.error}`;
      if (data.retried) message += ' [retried from dead letters]';
      entry.innerHTML = `<span class="time">${time}</span>${message}`;
      log.insertBefore(entry, log.firstChild);
    }
    async function refreshStats() {
      try {
        const res = await fetch('/stats');
        const stats = await res.json();
        document.getElementById('stat-pending').textContent = stats.pending;
        document.getElementById('stat-delivering').textContent = stats.delivering;
        document.getElementById('stat-completed').textContent = stats.completed;
        document.getElementById('stat-failed').textContent = stats.failed;
        document.getElementById('stat-dead-letters').textContent = stats.dead_letters;
      } catch (e) {}
      refreshDeadLetters();
    }

What do these functions do?

The addEventToLog() function creates a new div for each webhook event, color-coded by status, and inserts it at the top of the log so the newest events appear first.

The refreshStats() function fetches the current counts from a /stats endpoint (you will build this next) and updates the stat card numbers. It also calls refreshDeadLetters() to update the dead letter list.

  • Finally, add the remaining functions and close the file. Paste this below the code you just added:
    async function refreshDeadLetters() {
      try {
        const res = await fetch('/dead-letters');
        const deadLetters = await res.json();
        const list = document.getElementById('dead-letters-list');
        if (deadLetters.length === 0) {
          list.innerHTML = '<div style="color: #888; padding: 10px;">No dead letters yet</div>';
          return;
        }
        list.innerHTML = deadLetters.map(dl => `<div class="dead-letter"><div class="dead-letter-info"><div class="url">${dl.url}</div><div class="error">${dl.last_error} (${dl.attempts} attempts)</div></div><button onclick="retryDeadLetter(${dl.id})">Retry</button></div>`).join('');
      } catch (e) {}
    }
    async function sendTestWebhook(working) {
      const url = working ? 'http://localhost:3000/test-destination' : 'http://localhost:3000/test-destination-broken';
      await fetch('/relay', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url: url, body: { message: 'Test webhook', timestamp: new Date().toISOString() } }) });
    }
    async function retryDeadLetter(id) {
      await fetch(`/dead-letters/${id}/retry`, { method: 'POST' });
      refreshStats();
    }
    setupEventSource();
    refreshStats();
  </script>
</body>
</html>

What do the remaining functions do?

The refreshDeadLetters() function fetches failed webhooks and renders them with a Retry button. The sendTestWebhook() function fires a webhook to either the working or broken destination. The two init calls at the bottom connect to the event stream and load the initial stats as soon as the page opens.

  • Save public/dashboard.html.
  • Open http://localhost:3000/dashboard.html in your browser.

You should see the dark-themed dashboard with five stat cards, two test buttons, and the connection status showing "Connected (live)" in green. The stat counters may show errors in the browser console because the /stats endpoint does not exist yet. You will fix that next.

Seeing a blank page or 404?

  • Make sure dashboard.html is inside the public/ folder (not the project root). The URL must be /dashboard.html, not /public/dashboard.html.
  • Confirm the express.static line is in server.js and that you restarted the server after saving.

Why is my dashboard page not loading?

Connect the relay to the dashboard

The dashboard front-end is ready, but it needs server endpoints to fetch stats and manage dead letters. Your existing broadcast() calls already push live events. Now you need the data endpoints that the dashboard fetches on load.

  • In server.js, add the following stats endpoint after the POST /relay route:
// --- Dashboard Data Endpoints ---
app.get('/stats', (req, res) => {
  const stats = {
    pending: db.prepare(`SELECT COUNT(*) as count FROM webhooks WHERE status = 'pending'`).get().count,
    delivering: db.prepare(`SELECT COUNT(*) as count FROM webhooks WHERE status = 'delivering'`).get().count,
    completed: db.prepare(`SELECT COUNT(*) as count FROM webhooks WHERE status = 'completed'`).get().count,
    failed: db.prepare(`SELECT COUNT(*) as count FROM webhooks WHERE status = 'failed'`).get().count,
    dead_letters: db.prepare(`SELECT COUNT(*) as count FROM dead_letters`).get().count
  };
  res.json(stats);
});

app.get('/dead-letters', (req, res) => {
  const deadLetters = db.prepare(`SELECT * FROM dead_letters ORDER BY failed_at DESC`).all();
  res.json(deadLetters);
});

What do these endpoints return?

The /stats endpoint counts webhooks in each status and returns the totals as JSON. The dashboard polls this after each SSE event to update the stat cards.

The /dead-letters endpoint returns all permanently failed webhooks so the dashboard can list them with retry buttons.

  • Directly below the /dead-letters GET route, add the retry endpoint that moves a dead letter back into the queue:
app.post('/dead-letters/:id/retry', (req, res) => {
  const { id } = req.params;
  const deadLetter = db.prepare(`SELECT * FROM dead_letters WHERE id = ?`).get(id);
  if (!deadLetter) {
    return res.status(404).json({ error: 'Dead letter not found' });
  }
  const result = db.prepare(`INSERT INTO webhooks (url, body, status, attempts, next_attempt) VALUES (?, ?, 'pending', 0, datetime('now'))`).run(deadLetter.url, deadLetter.body);
  db.prepare(`DELETE FROM dead_letters WHERE id = ?`).run(id);
  broadcast({ type: 'webhook_update', data: { id: Number(result.lastInsertRowid), status: 'pending', attempts: 0, retried: true } });
  console.log(`[RETRY] Dead letter #${id} moved back to queue as webhook #${result.lastInsertRowid}`);
  res.json({ message: 'Dead letter moved back to queue', new_id: Number(result.lastInsertRowid) });
});

What does the retry endpoint do?

When the dashboard sends a POST to /dead-letters/:id/retry, this endpoint copies the failed webhook back into the webhooks table with fresh attempts. It deletes the dead letter entry and broadcasts the change so the dashboard updates instantly.

  • Finally, update the app.listen callback at the bottom of server.js to log the dashboard URL. Find this line:
app.listen(PORT, () => {
  console.log(`Webhook Relay running on http://localhost:${PORT}`);
  startQueueProcessor(broadcast);
});
  • Replace it with:
app.listen(PORT, () => {
  console.log(`Webhook Relay running on http://localhost:${PORT}`);
  console.log(`Dashboard: http://localhost:${PORT}/dashboard.html`);
  startQueueProcessor(broadcast);
});
  • Save server.js.
  • Stop the server with Ctrl+C and restart it by running:
node server.js

You should see both the server URL and the dashboard URL logged in the terminal.

Before you test the dashboard, what do you think will happen when you click the "Send Test (Working)" button? Will the event log update without a page refresh?

  • Open http://localhost:3000/dashboard.html in your browser.
  • Click the Send Test (Working) button.

You should see a new entry appear in the Live Event Log showing the webhook going from PENDING to DELIVERING to COMPLETED. The stat cards update in real-time. No page refresh needed.

  • Click the Send Test (Broken) button.

Watch the event log show retry attempts with increasing delays. The pending counter flickers as the queue picks up and retries the webhook every few seconds.

Dashboard shows 0 for all stats?

  • Open the browser's developer console (F12) and check for 404 errors on /stats or /dead-letters. If you see them, the endpoints are missing from server.js.
  • Make sure you placed the /stats and /dead-letters routes after app.use(express.json()) and before app.listen.
  • Confirm you restarted the server after saving your changes.

Why are my dashboard stats not updating?

✔️ Awesome, I've got everything!

Great. Double check you have saved both server.js and public/dashboard.html.

ⓧ I'd like to double check the full code

Your server.js should contain all of these sections in order. Scroll through your file and confirm each one is present:

  • Require statements (express, path, db, startQueueProcessor).
  • App setup (express.json(), express.static).
  • SSE section (clients array, broadcast() function, GET /events route).
  • Test destinations (POST /test-destination, POST /test-destination-broken).
  • Webhook relay (POST /relay).
  • Dashboard data endpoints (GET /stats, GET /dead-letters, POST /dead-letters/:id/retry).
  • Health check (GET /health).
  • Server start (app.listen with both console.log lines and startQueueProcessor).

Your public/dashboard.html should contain these sections in order:

  • DOCTYPE and head with full CSS styles.
  • Body with connection status, stat cards, control buttons, event log div, dead letters div.
  • Script block with setupEventSource(), addEventToLog(), refreshStats(), refreshDeadLetters(), sendTestWebhook(), retryDeadLetter(), and the two init calls.

Your relay now has full observability. You can see every webhook enter the queue, get delivered or retried, and eventually succeed or fail. Next up, you will give dead letters a second chance by adding manual retry controls to the dashboard.

Add Dead Letter Management

Your retry engine from the previous step handles temporary failures automatically. But what about messages that fail permanently?

After five failed attempts, your queue moves the webhook into a dead letter queue. In production systems, a human investigates dead letters and decides whether to fix the destination and retry, or discard the message entirely. Your server and dashboard already have this management layer built in. In this step, you'll confirm the endpoints work and run the full dead letter lifecycle from failure through manual recovery.

In this step, get ready to:
  • Verify the dead letter API endpoints respond correctly.
  • Test the complete dead letter lifecycle on the dashboard.
Confirm the dead letter API endpoints

Your server.js already has two endpoints dedicated to dead letter management. Before you test the full flow visually, confirm they respond correctly.

  • Open a new browser tab.
  • Navigate to http://localhost:3000/dead-letters.

You should see a JSON array in the browser. If you tested the broken destination in the previous step and waited long enough for retries to exhaust, you'll see dead letter entries. If not, you'll see an empty array []. Both are correct at this point.

What do the dead letter endpoints do?

Your server exposes two dead letter management routes:

  • GET /dead-letters returns all permanently failed webhooks as JSON, sorted by most recent failure first. This is how a human inspects what went wrong.
  • POST /dead-letters/:id/retry moves a dead letter back into the main webhooks queue with a fresh attempt count. It deletes the dead letter entry and creates a new pending webhook, giving the message another chance.
  • Now navigate to http://localhost:3000/stats to see the overall queue statistics.

You should see a JSON object with counts for pending, delivering, completed, failed, and dead_letters. This confirms your API layer is responding and connected to the database.

Seeing an empty page or connection error?

  • Make sure your server is still running in the terminal. You should see the "Webhook Relay running on http://localhost:3000" message. If it crashed, restart it with node server.js.
  • Check that the URL is exactly http://localhost:3000/dead-letters with a hyphen, not an underscore.

Why can't I see my dead letter endpoints?

Test the full dead letter lifecycle on the dashboard

Now for the real test. You'll send a webhook to the broken destination, watch the retry engine exhaust all five attempts, see the message land in the dead letter section, and then manually retry it.

  • Return to your dashboard at http://localhost:3000/dashboard.html.
  • Confirm the connection status shows "Connected (live)" in green at the top of the page.
  • Click the red Send Test (Broken) button.

Watch the Live Event Log section. You'll see the webhook enter as PENDING, then cycle through DELIVERING and back to PENDING as each retry fails. The attempts counter increases each time.

How long do the retries take?

Your exponential backoff formula doubles the wait each time: roughly 4 seconds, then 8, then 16, then 30 (capped). Including the 2-second polling interval, the full cycle from first attempt to dead letter takes about 60 seconds.

Watch the terminal running your server for the [RETRY] log messages. They show the increasing delay between each attempt. This is the exponential backoff with jitter you built in the previous step, working exactly as designed.

  • Wait for approximately 60 seconds until you see a FAILED status appear in the live event log.
  • Check the Dead Letters stat card at the top. Its count should have increased by 1.
  • Scroll down to the Dead Letters section at the bottom of the dashboard. You should see the failed webhook listed with its destination URL, error message, and a Retry button.

Before you click Retry, think about what will happen. The dead letter's original destination was the broken endpoint. What status do you expect the retried webhook to end up with?

  • Click the Retry button next to the dead letter entry.

Watch the live event log. You'll see a new PENDING entry appear with the tag "[retried from dead letters]". The webhook re-enters the queue with a fresh attempt count, and the dead letter disappears from the Dead Letters section.

Why does the retried webhook keep the same destination?

In a production system, you'd typically fix the broken destination before retrying. The retry endpoint moves the message back to the queue with the same URL so the original delivery intent is preserved.

Since your test destination is still returning 500 errors, the retried webhook will cycle through five more attempts and land back in dead letters. That's the expected behavior. In real use, you'd only retry once the destination is healthy again.

Your terminal should show a [RETRY] Dead letter #... moved back to queue log message followed by new delivery attempts. This confirms the full dead letter lifecycle works: ingestion, retry exhaustion, dead letter storage, manual recovery, and re-delivery.

Dead letter not appearing after retries?

  • Make sure you waited long enough. The full retry cycle takes about 60 seconds due to exponential backoff. Check the terminal for [DEAD LETTER] in the log output.
  • If the Dead Letters section still shows "No dead letters yet" after 60 seconds, try refreshing the dashboard page. The SSE connection may have dropped.

Why aren't my dead letters showing up on the dashboard?

Your webhook relay is now feature-complete. It ingests webhooks, retries with exponential backoff, stores permanently failed messages in a dead letter queue, and lets you manually recover them. Next up, you'll add rate limiting in the Secret Mission to protect this system from abuse.

Secret mission

Add Rate Limiting

Right now, anyone can send unlimited requests to your relay. A single sender could flood your system with thousands of webhooks per second, overwhelming your queue and crashing your server. In this secret mission, you'll build a sliding window rate limiter from scratch using SQLite that protects your system from abuse.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine. There are no cloud resources, no subscriptions, and no ongoing costs.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The webhook-relay project folder on your Desktop (contains all source code, node_modules, and the relay.db database file).
  • A Node.js server process running on localhost:3000.

Keep everything running

No action needed. Choose this if you're still actively experimenting with the relay or planning to extend it.

  • Your project folder stays on your Desktop. The relay.db file is only a few KB.
  • The server only runs while your terminal is open. Close the terminal or press Ctrl+C to stop it at any time.
  • There are no background processes or daemons. Nothing runs when you're not actively using it.

Pause - I'll come back to this later

Stop the running server to free up port 3000. Your files stay on disk so you can pick up where you left off.

  • Switch to the terminal where your server is running.
  • Press Ctrl+C to stop the server.
  • When you're ready to resume, open a terminal in the webhook-relay folder and run node server.js again.

Delete - I don't want to use this again

Remove all project files. Nothing else to clean up since everything lives in one folder.

  • Switch to the terminal where your server is running and press Ctrl+C to stop it.
  • Delete the project folder by running one of these commands (pick your OS).

macOS

  • Run this command to delete the folder:
rm -rf ~/Desktop/webhook-relay

Windows

  • Run this command to delete the folder:
Remove-Item -Recurse -Force ~\Desktop\webhook-relay

That's it. The database, source code, and all dependencies lived inside that folder. Nothing else remains on your machine from this project.

Nice Work!

Nice Work!

You just built a production-grade webhook relay service from scratch. You can now explain exactly how job queues, retry logic, and failure recovery work because you wrote every line yourself.

Here's what you learned:

  • Built a webhook relay service that receives, queues, and delivers HTTP payloads with exponential backoff and jitter.
  • Built a real-time dashboard using Server-Sent Events with zero external libraries for live updates.
  • Created a dead letter queue with manual retry so permanently failed deliveries can be inspected and recovered.
  • Secret Mission: Implemented a sliding window rate limiter from scratch using SQLite that protects your system from abuse.

Ready to quiz yourself?