Event-Driven Microservices with BullMQ

Build a reliable order pipeline using BullMQ queues and Redis in Docker.

Introduction

30 Second Summary

Think about ordering food online. Behind that single "Place Order" button, your request passes through payment processing, inventory checks, and notification services. If any one of those services goes down for a moment, your order shouldn't vanish into thin air.

In this project, you will build an event-driven order processing pipeline with three independent microservices that communicate through BullMQ job queues. You'll see first-hand why simple messaging loses data and how persistent queues solve the problem.

What You'll Build

You'll place orders from your terminal and watch them flow through payment processing and notification delivery across three separate processes running in three terminal windows, with retries and failures happening in real time.

By the end of this project, you'll have:

  • A working multi-service pipeline where you place an order and watch it flow through payment processing, survive random failures via automatic retries, and trigger a delayed notification.
  • First-hand proof that plain Redis Pub/Sub loses messages when a subscriber disconnects, and the confidence to explain why production systems need a persistent queue instead.
  • A dead-letter queue where permanently failed jobs land after exhausting retries, ready for you to inspect via a status endpoint.
  • Secret Mission: A real-time Bull Board dashboard to visually monitor all queues, job states, and retry history in your browser.

Are there any prerequisites?

This project assumes you have experience building REST APIs with Node.js and basic Docker knowledge (running containers). You do not need prior experience with message queues or BullMQ.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters. You're going to experience first-hand why reliable inter-service communication is essential for backend systems that can't afford to silently lose data.

Set Up Docker Redis and Project Structure

Your microservices need a shared message broker to pass jobs between them. Redis will serve as the persistent backend for BullMQ's job queues, storing every job until a worker picks it up.

In this step, you'll scaffold a TypeScript project, spin up a local Redis instance inside Docker, and create the shared connection module that all your services will import.

In this step, get ready to:
  • Create the project directory, install dependencies, and configure TypeScript.
  • Spin up a Redis container with Docker Compose.
  • Create the shared Redis connection module and verify it connects.
Create the project and install dependencies
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir order-pipeline
cd order-pipeline

This project needs several packages: Express for the HTTP server, BullMQ for reliable job queues, and ioredis as the Redis client. You'll define all of these in a package.json file.

  • Create a file called package.json in your order-pipeline folder with the following content:
{
  "name": "order-pipeline",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "order-service": "tsx src/order-service-queue.ts",
    "payment-worker": "tsx src/payment-worker.ts",
    "notification-worker": "tsx src/notification-worker.ts",
    "naive-order": "tsx src/order-service.ts",
    "naive-sub": "tsx src/naive-subscriber.ts",
    "dead-letter": "tsx src/dead-letter.ts"
  },
  "dependencies": {
    "bullmq": "5.80.2",
    "express": "5.2.1",
    "ioredis": "5.11.1"
  },
  "devDependencies": {
    "@types/express": "^5.0.0",
    "tsx": "4.23.1",
    "typescript": "^5.5.0"
  }
}

What does this file do?

This package.json defines your project's identity and its dependencies.

  • "type": "module" enables ES module import/export syntax instead of CommonJS require().
  • The scripts section defines shortcuts to run each microservice. They reference files you'll create in later steps.
  • Dependencies are pinned to exact versions (5.80.2) to ensure everyone gets identical behaviour.

✔️ Awesome, I've got everything!

Make sure you've saved package.json before continuing.

ⓧ I'd like to double check the full code

{
  "name": "order-pipeline",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "order-service": "tsx src/order-service-queue.ts",
    "payment-worker": "tsx src/payment-worker.ts",
    "notification-worker": "tsx src/notification-worker.ts",
    "naive-order": "tsx src/order-service.ts",
    "naive-sub": "tsx src/naive-subscriber.ts",
    "dead-letter": "tsx src/dead-letter.ts"
  },
  "dependencies": {
    "bullmq": "5.80.2",
    "express": "5.2.1",
    "ioredis": "5.11.1"
  },
  "devDependencies": {
    "@types/express": "^5.0.0",
    "tsx": "4.23.1",
    "typescript": "^5.5.0"
  }
}
  • Install all the project dependencies by running:
npm install
  • You should see output listing the installed packages. Confirm by running:
ls node_modules/.package-lock.json

If the file exists, your dependencies installed correctly.

Seeing permission errors or network failures?

  • Check your internet connection. npm install downloads packages from the npm registry.
  • On macOS, if you see EACCES errors, you may need to fix npm permissions. Run sudo chown -R $(whoami) ~/.npm and try again.
  • Make sure your package.json has valid JSON syntax. A missing comma or quote will cause install to fail.

Still stuck? Why is npm install failing in my order-pipeline project?

Now you need a TypeScript configuration so tsx knows how to compile your code.

  • Create a file called tsconfig.json in your order-pipeline folder with the following content:
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "bundler",
    "esModuleInterop": true,
    "strict": true,
    "outDir": "dist",
    "skipLibCheck": true
  },
  "include": ["src"]
}

What does this configure?

  • "target": "ES2022" compiles to modern JavaScript that Node.js v22 supports natively.
  • "moduleResolution": "bundler" allows importing with .js extensions, which ESM requires.
  • "include": ["src"] tells TypeScript to only compile files inside the src/ directory.

✔️ Awesome, I've got everything!

Make sure you've saved tsconfig.json before continuing.

ⓧ I'd like to double check the full code

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "bundler",
    "esModuleInterop": true,
    "strict": true,
    "outDir": "dist",
    "skipLibCheck": true
  },
  "include": ["src"]
}
  • Verify your project structure by running:
ls

You should see node_modules, package.json, package-lock.json, and tsconfig.json listed.

Create the Docker Compose file and start Redis

BullMQ stores all jobs in Redis. You'll run Redis locally inside a Docker container so you don't need to install it directly on your machine.

  • Create a file called docker-compose.yml in your order-pipeline folder with the following content:
services:
  redis:
    image: redis:7.4-alpine
    container_name: redis-bullmq
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    command: redis-server --appendonly yes
    restart: unless-stopped

volumes:
  redis_data:

What does this Docker Compose file do?

  • redis:7.4-alpine uses Redis 7.4 on a lightweight Alpine Linux base. This version is verified compatible with BullMQ.
  • --appendonly yes enables append-only file persistence, so jobs survive if the container restarts.
  • Port 6379 is Redis's default port, mapped from the container to your machine.
  • The redis_data volume keeps data intact between container restarts.

Why Redis 7.4 instead of 8.x?

BullMQ requires Redis 5.0.0 at minimum and recommends 6.2.0 or above. Redis 8.x is very new and the ioredis library explicitly states compatibility with Redis 7.x. Using 7.4 gives you a stable, well-tested foundation.

✔️ Awesome, I've got everything!

Make sure you've saved docker-compose.yml before continuing.

ⓧ I'd like to double check the full code

services:
  redis:
    image: redis:7.4-alpine
    container_name: redis-bullmq
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    command: redis-server --appendonly yes
    restart: unless-stopped

volumes:
  redis_data:
  • Start the Redis container by running:
docker compose up -d
  • Verify the container is running by running:
docker ps

You should see a container named redis-bullmq with status Up in the output.

Container not starting?

  • Make sure Docker Desktop is running. Open it from your applications if it's not active in your system tray.
  • If you see a port conflict error, another process is using port 6379. Stop any existing Redis process or change the port mapping in docker-compose.yml.
  • Check that your docker-compose.yml indentation uses spaces, not tabs. YAML is whitespace-sensitive.

Still stuck? Why is my Docker Redis container not starting?

Create the shared Redis connection

Every service in your pipeline needs to connect to the same Redis instance. A shared connection module avoids duplicating configuration across files.

  • Create the src directory by running:
mkdir src
  • Create a file called connection.ts inside the src/ folder with the following content:
import IORedis from "ioredis";

export const connection = new IORedis({
  host: "localhost",
  port: 6379,
  maxRetriesPerRequest: null,
});

Why is maxRetriesPerRequest set to null?

BullMQ workers use blocking Redis commands that can wait indefinitely for new jobs. The default ioredis retry limit of 20 would cause the connection to throw an error during normal operation. Setting maxRetriesPerRequest: null removes that limit so workers can wait as long as needed.

BullMQ will throw an exception at startup if this value is not null.

✔️ Awesome, I've got everything!

Make sure you've saved src/connection.ts before continuing.

ⓧ I'd like to double check the full code

import IORedis from "ioredis";

export const connection = new IORedis({
  host: "localhost",
  port: 6379,
  maxRetriesPerRequest: null,
});

Before you run the next command, what do you think Redis will respond with if the connection is working?

  • Verify that Redis is reachable by running:
docker exec redis-bullmq redis-cli PING

You should see PONG printed in your terminal. This confirms Redis is running and accepting connections.

Seeing 'Error: No such container'?

  • Run docker ps -a to check if the container exists but stopped. If it shows Exited status, run docker compose up -d again.
  • Make sure you're running the command from inside the order-pipeline folder where your docker-compose.yml lives.

Still stuck? Why can't I connect to my Redis container?

Your project structure is ready and Redis is running. Next up, you'll build the Order Service using plain Redis Pub/Sub to see how basic messaging works.

Build the Order Service with Redis Pub/Sub

Your project structure is ready and Redis is running in Docker. Now it is time to build your first two microservices and have them talk to each other.

The most intuitive way for two services to communicate through Redis is Pub/Sub. In this step you will build an Order Service that publishes order events and a subscriber that receives them.

In this step, get ready to:
  • Create an Express API that publishes order events to a Redis channel.
  • Create a subscriber script that listens for those events and logs them.
  • Run both services in separate terminals and confirm messages flow between them.
Create the order service

The Order Service is an Express server with a single POST endpoint. When a request comes in, it creates an order object and publishes it to the orders channel using the Redis PUBLISH command.

  • Create a new file src/order-service.ts and paste the following code:
import express from "express";
import IORedis from "ioredis";

const app = express();
app.use(express.json());

const redis = new IORedis({ host: "localhost", port: 6379 });

app.post("/orders", async (req, res) => {
  const order = {
    id: crypto.randomUUID(),
    item: req.body.item || "Unknown Item",
    amount: req.body.amount || 0,
    timestamp: new Date().toISOString(),
  };

  await redis.publish("orders", JSON.stringify(order));
  console.log(`[Order Service] Published order ${order.id}`);

  res.status(201).json({ message: "Order published", order });
});

app.listen(3000, () => {
  console.log("[Order Service] Running on http://localhost:3000");
  console.log("[Order Service] Using Redis Pub/Sub (fire-and-forget)");
});

What does this code do?

  • It creates its own ioredis connection (separate from the shared connection.ts you made earlier) because this service only publishes. It does not need the maxRetriesPerRequest: null setting that BullMQ workers require.
  • Each order gets a unique ID via crypto.randomUUID(), a built-in Node.js function.
  • The redis.publish("orders", ...) call sends the serialized order to the orders channel. Any subscriber listening on that channel receives it instantly.
  • Save src/order-service.ts.

File not appearing in src/?

Make sure you created the file inside the src folder, not in the project root. The path should be order-pipeline/src/order-service.ts.

Help me check my file structure.

Create the subscriber

The subscriber is a standalone script that connects to Redis in subscriber mode and listens for messages on the orders channel. A Redis connection that calls subscribe() enters subscriber mode and can no longer issue regular commands.

  • Create a new file src/naive-subscriber.ts and paste the following code:
import IORedis from "ioredis";

const subscriber = new IORedis({ host: "localhost", port: 6379 });

subscriber.subscribe("orders", (err) => {
  if (err) {
    console.error("Failed to subscribe:", err);
    process.exit(1);
  }
  console.log("[Subscriber] Listening for orders on 'orders' channel...");
});

subscriber.on("message", (channel, message) => {
  const order = JSON.parse(message);
  console.log(`[Subscriber] Received order ${order.id}: ${order.item} - $${order.amount}`);
});

What does this code do?

  • It creates a dedicated Redis connection and immediately calls subscriber.subscribe("orders", ...). This puts the connection into subscriber mode.
  • The "message" event fires each time a message arrives on any subscribed channel. The callback receives both the channel name and the raw message string.
  • This subscriber is intentionally separate from the Order Service because a Redis connection cannot publish and subscribe at the same time.
  • Save src/naive-subscriber.ts.
  • Confirm both files exist by running:
ls src/

You should see connection.ts, naive-subscriber.ts, and order-service.ts listed.

Run both services and send a test order

You need three terminal windows for this test: one running the order service, one running the subscriber, and one to send requests.

  • In your first terminal, start the order service by running:
npm run naive-order

You should see two lines confirming it started:

[Order Service] Running on http://localhost:3000
[Order Service] Using Redis Pub/Sub (fire-and-forget)
  • Open a second terminal in the same order-pipeline folder and start the subscriber by running:
npm run naive-sub

You should see:

[Subscriber] Listening for orders on 'orders' channel...
  • Open a third terminal in the same folder. Before you send a request, think about what you expect to happen. If you POST an order, which of the two running processes will show output?
  • Send a test order by running:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"item": "Mechanical Keyboard", "amount": 89.99}'

Both terminals show the order event. Your first terminal (order service) prints a line like:

[Order Service] Published order <some-uuid>

Your second terminal (subscriber) prints:

[Subscriber] Received order <some-uuid>: Mechanical Keyboard - $89.99

Why does this work?

Redis delivered the message in real time because the subscriber was connected and listening at the moment the order was published. Both processes share the same Redis instance running in your Docker container.

This feels great for a first pass. But what happens if the subscriber goes down? You will find out in the next step.

Not seeing output in the subscriber terminal?

  • Confirm both processes are running (you should see their startup messages). If either exited, check for a Redis connection error and make sure your Docker container is still running with docker ps.
  • Make sure the subscriber was started before you sent the curl request. Redis Pub/Sub only delivers to subscribers that are already connected.
  • Check that both terminals are in the order-pipeline directory so npm run can find package.json.

Help me debug why my subscriber is not receiving messages from the order service.

✔️ Awesome, I've got everything!

Great. Make sure both files are saved. Keep both processes running for the next step.

ⓧ I'd like to double check the full code

Here is the full src/order-service.ts:

import express from "express";
import IORedis from "ioredis";

const app = express();
app.use(express.json());

const redis = new IORedis({ host: "localhost", port: 6379 });

app.post("/orders", async (req, res) => {
  const order = {
    id: crypto.randomUUID(),
    item: req.body.item || "Unknown Item",
    amount: req.body.amount || 0,
    timestamp: new Date().toISOString(),
  };

  await redis.publish("orders", JSON.stringify(order));
  console.log(`[Order Service] Published order ${order.id}`);

  res.status(201).json({ message: "Order published", order });
});

app.listen(3000, () => {
  console.log("[Order Service] Running on http://localhost:3000");
  console.log("[Order Service] Using Redis Pub/Sub (fire-and-forget)");
});

Here is the full src/naive-subscriber.ts:

import IORedis from "ioredis";

const subscriber = new IORedis({ host: "localhost", port: 6379 });

subscriber.subscribe("orders", (err) => {
  if (err) {
    console.error("Failed to subscribe:", err);
    process.exit(1);
  }
  console.log("[Subscriber] Listening for orders on 'orders' channel...");
});

subscriber.on("message", (channel, message) => {
  const order = JSON.parse(message);
  console.log(`[Subscriber] Received order ${order.id}: ${order.item} - $${order.amount}`);
});

Your two services are talking to each other through Redis. The subscriber picks up every order the moment it is published. But this setup has a critical weakness that only shows up when things go wrong. In the next step, you will break it on purpose and watch messages vanish.

Break It and Watch Messages Vanish

In the previous step, you confirmed that Redis Pub/Sub works perfectly when both publisher and subscriber are online at the same time.

But what happens when a subscriber crashes or restarts? In production, services go down all the time. This step reveals the critical flaw in plain Pub/Sub that makes it dangerous for real microservice communication.

In this step, get ready to:
  • Disconnect the subscriber and send orders while it's offline.
  • Restart the subscriber and check whether the missed orders arrive.
  • Understand why fire-and-forget messaging breaks production systems.
Stop the subscriber and send orders into the void

Your subscriber is currently listening for orders in one terminal, and the order service is running in another. You're about to simulate what happens when a consumer service goes offline.

  • Switch to the terminal where src/naive-subscriber.ts is running.
  • Press Ctrl+C to stop the subscriber process.

The subscriber is now offline. The order service is still running and has no idea anything changed.

  • Send two orders to the order service by running these commands in a free terminal:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"item": "Lost Order 1", "amount": 50}'
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"item": "Lost Order 2", "amount": 75}'

Look at the responses. Both return a 201 status with a success message. The order service terminal also shows two [Order Service] Published order ... log lines.

Notice anything suspicious?

The publisher received no error and no warning. It happily published both orders into the void. From the publisher's perspective, everything worked perfectly. Redis accepted the messages and returned success.

This is the dangerous part. There is zero feedback that nobody received those messages.

Restart the subscriber and check for the missed messages

You sent two orders while the subscriber was down. Before you restart it, take a moment to predict what will happen.

Prediction time

When you restart the subscriber, do you think it will receive the two orders that were published while it was offline? Or will those messages be gone?

  • Restart the naive subscriber by running this command:
npx tsx src/naive-subscriber.ts

You see the startup message:

[Subscriber] Listening for orders on 'orders' channel...

And then nothing. No orders arrive. Wait as long as you like. Those two orders are gone forever.

The messages are permanently lost

This is not a bug or a timing issue. Redis Pub/Sub delivered both messages to zero subscribers (because none were connected at publish time), and then discarded them immediately. There is no inbox, no buffer, and no replay mechanism.

Those two orders are gone with no trace. No error. No log. No way to recover them.

Understand why this breaks production systems

What you just experienced is the fundamental limitation of Redis Pub/Sub for microservice communication.

Why did this happen?

Redis Pub/Sub is fire-and-forget. When a message is published, Redis delivers it to every currently connected subscriber and then immediately forgets it. There is no persistence, no acknowledgment, and no replay.

Any subscriber that disconnects, even for a second, misses every message published during that window. The publisher never knows its messages went undelivered.

In production, services restart during deployments, scale down under low traffic, and crash from bugs. A messaging system that silently drops messages during these routine events is a recipe for lost orders, missed payments, and invisible data corruption.

What would a reliable system need?

  • Persistence: messages stored until a worker picks them up, not discarded on delivery.
  • Acknowledgment: a message is only removed after the consumer confirms it was processed.
  • Retry logic: if processing fails, the message is redelivered automatically.
  • Dead-letter handling: permanently failing messages are captured for inspection instead of silently lost.

This is exactly what a job queue provides. In the next step, you will replace this fragile Pub/Sub pattern with BullMQ, which gives you all four of these guarantees on top of the same Redis instance.

  • Stop the naive subscriber with Ctrl+C.
  • Leave the order service running for now. You will replace it in the next step.

You just proved that Pub/Sub silently loses messages the moment a subscriber disconnects. Next up, you will replace this brittle pattern with BullMQ and watch the same scenario produce zero lost messages.

Replace with BullMQ for Reliable Delivery

You just watched real messages disappear into thin air. Redis Pub/Sub has zero memory of what happens while a subscriber is offline.

BullMQ solves this by persisting every job in Redis. Workers pick up jobs whenever they connect, even if the job was added hours ago. If processing fails, BullMQ retries automatically with configurable exponential backoff.

In this step, get ready to:
  • Create a queue-based order service that persists jobs in Redis with BullMQ.
  • Build a payment worker that processes jobs with automatic retries and dead-letter handling.
  • Prove that jobs survive worker downtime and are processed on restart.
Create the queue-based order service

Instead of publishing to a channel that nobody might be listening to, you will now add jobs to a persistent BullMQ queue. The jobs stay in Redis until a worker is ready to process them.

  • Create a new file called src/order-service-queue.ts and paste the following code:
import express from "express";
import { Queue } from "bullmq";
import { connection } from "./connection.js";

const app = express();
app.use(express.json());

// Create persistent queues backed by Redis
const paymentQueue = new Queue("payments", { connection });
const notificationQueue = new Queue("notifications", { connection });

app.post("/orders", async (req, res) => {
  const order = {
    id: crypto.randomUUID(),
    item: req.body.item || "Unknown Item",
    amount: req.body.amount || 0,
    timestamp: new Date().toISOString(),
  };

  // Add job with retry config: 3 attempts, exponential backoff
  await paymentQueue.add("process-payment", order, {
    attempts: 3,
    backoff: { type: "exponential", delay: 1000 },
  });

  console.log(`[Order Service] Queued payment job for order ${order.id}`);
  res.status(201).json({ message: "Order queued for payment", order });
});

What does this code do?

Instead of redis.publish() which broadcasts into the void, paymentQueue.add() persists the job data in Redis as a list entry. It stays there until a worker picks it up.

The attempts: 3 and backoff: { type: "exponential", delay: 1000 } options tell BullMQ to retry up to 3 times if processing fails, waiting 1s, then 2s, then 4s between attempts.

  • Below the POST endpoint in the same file, add the dead-letter inspection endpoint and the server startup:
app.get("/dead-letter", async (req, res) => {
  const failedPayments = await paymentQueue.getFailed();
  const failedNotifications = await notificationQueue.getFailed();

  res.json({
    failedPayments: failedPayments.map((job) => ({
      id: job.id,
      data: job.data,
      failedReason: job.failedReason,
      attemptsMade: job.attemptsMade,
    })),
    failedNotifications: failedNotifications.map((job) => ({
      id: job.id,
      data: job.data,
      failedReason: job.failedReason,
      attemptsMade: job.attemptsMade,
    })),
  });
});

app.listen(3000, () => {
  console.log("[Order Service] Running on http://localhost:3000");
  console.log("[Order Service] Using BullMQ (reliable delivery)");
});

What is the dead-letter endpoint for?

The getFailed() method returns all jobs that exhausted their retry attempts and permanently failed. The GET /dead-letter endpoint gives you a JSON report of those failed jobs so you can inspect them later.

  • Save src/order-service-queue.ts.
  • Start the new order service by running:
npx tsx src/order-service-queue.ts

You should see [Order Service] Running on http://localhost:3000 and [Order Service] Using BullMQ (reliable delivery) printed in the terminal.

  • Open a second terminal and send a test order by running:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"item": "Mechanical Keyboard", "amount": 149}'

You see a JSON response with "message": "Order queued for payment". In the first terminal, the log shows the job was queued. The job is now sitting in Redis waiting for a worker.

Not seeing the queued message?

  • Make sure your Redis container is still running. Run docker ps and check for redis-bullmq.
  • Check that src/connection.ts has maxRetriesPerRequest: null set. BullMQ requires this setting.
  • Confirm you are importing from ./connection.js (with the .js extension). ESM imports require the full extension.

Help me debug my BullMQ order service connection

✔️ Awesome, I've got everything!

Great. Double check you have saved src/order-service-queue.ts and your service is responding to requests.

ⓧ I'd like to double check the full code

Scroll up and confirm you have these chunks in src/order-service-queue.ts, in order:

  • Imports (express, Queue from bullmq, connection) and queue declarations (paymentQueue, notificationQueue).
  • POST /orders endpoint that calls paymentQueue.add() with attempts: 3 and exponential backoff.
  • GET /dead-letter endpoint that calls getFailed() on both queues.
  • app.listen(3000) with both log lines.
Build the payment worker

The order service adds jobs to the queue. Now you need a separate process that pulls jobs off and processes them. This is the payment worker.

  • Stop the order service in the first terminal with Ctrl+C (you will restart it shortly).
  • Create a new file called src/payment-worker.ts and paste the core processor logic:
import { Worker } from "bullmq";
import { Queue } from "bullmq";
import { connection } from "./connection.js";

// Queue for chaining to the next pipeline stage
const notificationQueue = new Queue("notifications", { connection });

const worker = new Worker(
  "payments",
  async (job) => {
    console.log(`[Payment Worker] Processing payment for order ${job.data.id}...`);
    console.log(`[Payment Worker] Attempt ${job.attemptsMade + 1} of ${job.opts.attempts}`);

    // Simulate 30% random failure
    if (Math.random() < 0.3) {
      throw new Error("Payment gateway timeout - retrying...");
    }

    // Simulate processing time
    await new Promise((resolve) => setTimeout(resolve, 500));

    console.log(`[Payment Worker] Payment successful for order ${job.data.id} ($${job.data.amount})`);

What does this processor do?

The Worker class connects to the payments queue and processes jobs. For each job, it simulates a 30% chance of failure by throwing an Error. BullMQ catches that error and schedules a retry with exponential backoff.

The job.attemptsMade counter tracks which retry attempt is running. Combined with job.opts.attempts, you can see exactly where a job is in its retry lifecycle.

  • Still in src/payment-worker.ts, add the notification chaining logic below what you just pasted:
    // Chain: enqueue notification with 5-second delay
    await notificationQueue.add(
      "send-confirmation",
      {
        orderId: job.data.id,
        item: job.data.item,
        amount: job.data.amount,
      },
      { delay: 5000 }
    );

    console.log(`[Payment Worker] Notification scheduled (5s delay) for order ${job.data.id}`);
    return { status: "paid", orderId: job.data.id };
  },
  { connection, concurrency: 3 }
);

What does the chaining do?

On success, the worker adds a notification job to a separate notifications queue with a 5-second delay. This is how you build multi-stage pipelines where each stage is an independent process.

The concurrency: 3 option lets the worker process up to 3 jobs in parallel. The return value is stored by BullMQ so you can inspect completed job results later.

  • Finally, add the event listeners and startup log below the closing ); of the Worker constructor:
worker.on("completed", (job) => {
  console.log(`[Payment Worker] Job ${job.id} completed`);
});

worker.on("failed", (job, err) => {
  console.log(`[Payment Worker] Job ${job?.id} failed: ${err.message}`);
  if (job && job.attemptsMade >= (job.opts.attempts || 1)) {
    console.log(`[Payment Worker] Job ${job.id} moved to dead-letter (exhausted retries)`);
  }
});

worker.on("error", (err) => {
  console.error("[Payment Worker] Worker error:", err.message);
});

console.log("[Payment Worker] Waiting for payment jobs...");

What do the event listeners do?

The failed event fires on every failure, including retries. The if check detects when a job has used all its attempts and logs it as a dead-letter entry.

The error event catches connection-level problems (Redis going down). These three listeners give you full visibility into the worker lifecycle without needing an external dashboard.

  • Save src/payment-worker.ts.

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/order-service-queue.ts and src/payment-worker.ts.

ⓧ I'd like to double check the full code

Scroll up and confirm you have these chunks in src/payment-worker.ts, in order:

  • Imports (Worker, Queue, connection), the notificationQueue declaration, and the Worker constructor opening with the processor function (failure simulation, processing delay, success log).
  • The notification add() call with delay: 5000, the return statement, and the closing of the Worker constructor with { connection, concurrency: 3 }.
  • Three event listeners (completed, failed, error) and the final console.log startup message.
Test reliable delivery

Time to prove that BullMQ handles the exact scenario that broke Pub/Sub. You will start both services, send orders, watch retries, and then test message persistence.

  • Start the order service in your first terminal by running:
npx tsx src/order-service-queue.ts
  • Start the payment worker in a second terminal by running:
npx tsx src/payment-worker.ts

You see [Payment Worker] Waiting for payment jobs... in the second terminal. If the test order from earlier is still in the queue, the worker picks it up immediately.

  • In a third terminal, send an order by running:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"item": "Wireless Mouse", "amount": 59}'

Watch the payment worker terminal. You see the job being processed. Some jobs succeed on the first attempt. Others hit the 30% failure and retry automatically with backoff delays logged.

What should I see?

Successful jobs log Payment successful followed by Notification scheduled (5s delay). Failed attempts log Payment gateway timeout - retrying... and then the job reappears after the backoff period.

Now for the real test. Before you run this, what do you think happens if you stop the worker and send orders? Will they vanish like with Pub/Sub, or will something different happen?

  • Stop the payment worker with Ctrl+C in the second terminal.
  • Send two orders while the worker is down by running these commands in the third terminal:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"item": "USB Hub", "amount": 35}'
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"item": "Webcam", "amount": 89}'

Both requests return 201 with "Order queued for payment". The order service does not error or warn. The jobs are safely stored in Redis.

  • Restart the payment worker by running:
npx tsx src/payment-worker.ts

The worker immediately picks up and processes both queued jobs. No message lost. This is the difference between fire-and-forget Pub/Sub and a persistent job queue.

Why does this work when Pub/Sub didn't?

BullMQ stores jobs as Redis list entries. They persist on disk thanks to AOF in your Docker config. When a worker connects, it pulls from the list. No worker online? The list just grows until one shows up.

Pub/Sub broadcasts in real time with no storage. If nobody is listening, the message is gone forever. BullMQ decouples the producer from the consumer entirely.

Worker not picking up the queued jobs?

  • Verify the worker connects to the same queue name (payments). A typo means it listens to a different queue.
  • Check that the connection import path is ./connection.js in both files.
  • Run docker exec redis-bullmq redis-cli LLEN bull:payments:wait to check if jobs are sitting in the queue.

Help me debug why my payment worker isn't picking up queued jobs

Your services now communicate reliably through persistent queues. Next up, you will build a notification worker that processes delayed confirmation messages and inspect the dead-letter queue for permanently failed jobs.

Add Notification Worker with Delayed Jobs and Dead-Letter Handling

Your payment worker already schedules notification jobs with a 5-second delay after successful processing. But right now, those jobs sit in the notifications queue with no one to process them.

In this step, you will create a notification worker to pick up those delayed jobs and a dead-letter inspection tool. Together with the order service, these three processes form a complete pipeline you can watch in real time.

In this step, get ready to:
  • Create a notification worker that processes delayed confirmation jobs.
  • Create a dead-letter inspection script to query permanently failed jobs.
  • Run the full three-service pipeline and observe end-to-end message flow.
Create the notification worker

The payment worker enqueues a send-confirmation job to the notifications queue with a 5-second delay every time a payment succeeds. You need a BullMQ Worker listening on that queue to actually send the confirmation.

  • Create a new file src/notification-worker.ts and add the worker with its processor function:
import { Worker } from "bullmq";
import { connection } from "./connection.js";

const worker = new Worker(
  "notifications",
  async (job) => {
    console.log(`[Notification Worker] Sending confirmation for order ${job.data.orderId}...`);

    // Simulate sending email/SMS
    await new Promise((resolve) => setTimeout(resolve, 300));

    console.log(
      `[Notification Worker] Confirmation sent: "Your order for ${job.data.item} ($${job.data.amount}) has been confirmed!"`
    );

    return { status: "sent", orderId: job.data.orderId };
  },
  { connection, concurrency: 2 }
);

What does this code do?

  • The Worker listens on the notifications queue and processes each job it picks up.
  • The 300ms setTimeout simulates the time an email or SMS API would take to respond.
  • concurrency: 2 lets the worker handle two notification jobs in parallel if multiple arrive close together.
  • The job.data contains orderId, item, and amount passed from the payment worker when it enqueued the job.
  • Below the closing ); of the Worker constructor, add the event listeners and startup message:
worker.on("completed", (job) => {
  console.log(`[Notification Worker] Job ${job.id} completed`);
});

worker.on("failed", (job, err) => {
  console.error(`[Notification Worker] Job ${job?.id} failed: ${err.message}`);
});

worker.on("error", (err) => {
  console.error("[Notification Worker] Worker error:", err.message);
});

console.log("[Notification Worker] Waiting for notification jobs...");

What do the event listeners do?

  • completed fires after a job finishes successfully so you can track throughput.
  • failed fires when a job throws an error, letting you log the reason.
  • error catches worker-level problems like a lost Redis connection.
  • Save src/notification-worker.ts.
  • Start the notification worker by running:
npx tsx src/notification-worker.ts

You should see [Notification Worker] Waiting for notification jobs... printed in your terminal.

Not seeing the startup message?

  • Make sure your Redis Docker container is still running. Check with docker ps and look for redis-bullmq.
  • Confirm you saved the file as src/notification-worker.ts (not in the project root).
  • If you see a connection error, verify Redis is listening on port 6379. Help me debug my notification worker connection error.

✔️ Awesome, I've got everything!

Great. Keep this terminal running and leave it alone. You will open new terminals for the other services.

ⓧ I'd like to double check the full code

Your src/notification-worker.ts should have two sections in this order:

  • The Worker constructor (imports, new Worker("notifications", ...) with the processor function and concurrency: 2).
  • Three event listeners (completed, failed, error) followed by the console.log startup message.

Scroll up and confirm both chunks are present in order. Make sure the file is saved.

Create the dead-letter inspection script

When a job exhausts all its retries, BullMQ moves it to a "failed" state in Redis. You already have a GET /dead-letter endpoint in the order service that returns these as JSON. Now you will create a standalone script that gives you a formatted terminal report.

  • Create a new file src/dead-letter.ts and add the setup and query logic:
import { Queue } from "bullmq";
import { connection } from "./connection.js";

const paymentQueue = new Queue("payments", { connection });
const notificationQueue = new Queue("notifications", { connection });

async function inspectDeadLetter() {
  // Query all permanently failed jobs from both queues
  const failedPayments = await paymentQueue.getFailed();
  const failedNotifications = await notificationQueue.getFailed();

  console.log("\n=== Dead Letter Queue Report ===\n");

  if (failedPayments.length === 0 && failedNotifications.length === 0) {
    console.log("No failed jobs. All systems healthy.");
  }

What does this code do?

  • It creates Queue instances pointing at the same payments and notifications queues your workers use.
  • getFailed() returns all jobs that reached the permanently failed state after exhausting retries.
  • If both queues have zero failures, the script prints a healthy status.
  • Below the if block you just wrote, add the reporting logic, cleanup, and invocation:
  if (failedPayments.length > 0) {
    console.log(`Failed Payments (${failedPayments.length}):`);
    for (const job of failedPayments) {
      console.log(`  - Job ${job.id}: Order ${job.data.id}`);
      console.log(`    Reason: ${job.failedReason}`);
      console.log(`    Attempts: ${job.attemptsMade}`);
    }
  }

  if (failedNotifications.length > 0) {
    console.log(`\nFailed Notifications (${failedNotifications.length}):`);
    for (const job of failedNotifications) {
      console.log(`  - Job ${job.id}: Order ${job.data.orderId}`);
      console.log(`    Reason: ${job.failedReason}`);
      console.log(`    Attempts: ${job.attemptsMade}`);
    }
  }

  // Clean up connections before exiting
  await paymentQueue.close();
  await notificationQueue.close();
  await connection.quit();
}

inspectDeadLetter();

What does this code do?

  • Each if block iterates over failed jobs and prints the job ID, order data, failure reason, and attempt count.
  • paymentQueue.close() and connection.quit() shut down gracefully so the script exits cleanly instead of hanging.
  • The final inspectDeadLetter() call runs the function immediately when you execute the script.
  • Save src/dead-letter.ts.
  • Test the script by running it in a new terminal:
npx tsx src/dead-letter.ts

You should see the report header and No failed jobs. All systems healthy. (since you have not generated any permanent failures yet).

Script hangs without exiting?

  • Make sure you included both paymentQueue.close() and connection.quit() at the end of the function. Without these, the open Redis connection keeps the process alive.
  • Help me debug my dead-letter script that hangs without exiting.

✔️ Awesome, I've got everything!

Great. The dead-letter script is ready to inspect failures whenever you need it.

ⓧ I'd like to double check the full code

Your src/dead-letter.ts should have two sections in this order:

  • Imports, queue instantiation, inspectDeadLetter function opening with getFailed() calls and the "healthy" check.
  • Two if blocks that log failed payment and notification details, followed by .close() / .quit() calls, the closing brace, and inspectDeadLetter();.

Scroll up and confirm both chunks are present in order. Make sure the file is saved.

Run the full pipeline end-to-end

You now have all three services ready. Time to run them together and watch an order flow from creation through payment to notification.

  • Stop any running processes in your terminals with Ctrl+C.
  • In your first terminal, start the order service by running:
npx tsx src/order-service-queue.ts
  • In a second terminal, start the payment worker by running:
npx tsx src/payment-worker.ts
  • In a third terminal, start the notification worker by running:
npx tsx src/notification-worker.ts

You should see all three startup messages, one per terminal.

  • Open a fourth terminal and send an order by running:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"item": "Mechanical Keyboard", "amount": 149.99}'

Before you look at the other terminals, what do you expect to see? Think about the order: payment first, then a 5-second delay, then notification.

What should I see?

Watch the terminals in sequence:

  • Terminal 1 (order service): [Order Service] Queued payment job for order ....
  • Terminal 2 (payment worker): Processing payment... then either success or a retry. On success, Notification scheduled (5s delay).
  • Terminal 3 (notification worker): After a 5-second pause, Sending confirmation... followed by the confirmation message.

If the payment hits the 30% random failure, you will see retry attempts with exponential backoff (1s, 2s, 4s delays) before it succeeds or moves to failed.

  • Send a few more orders to generate some failures. Run this command 5-6 times:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"item": "USB-C Hub", "amount": 59.99}'

With a 30% failure rate and only 3 attempts per job, some orders will exhaust all retries and land in the failed state.

  • Check the dead-letter endpoint to see permanently failed jobs by running:
curl http://localhost:3000/dead-letter | python3 -m json.tool

Do you think you will see any failed jobs? It depends on how many hit the 30% failure three times in a row.

What does a dead-letter response look like?

If any jobs failed all 3 attempts, you will see JSON like this:

{
  "failedPayments": [
    {
      "id": "1",
      "data": { "id": "...", "item": "USB-C Hub", "amount": 59.99 },
      "failedReason": "Payment gateway timeout - retrying...",
      "attemptsMade": 3
    }
  ],
  "failedNotifications": []
}

If all jobs eventually succeeded, you will see empty arrays. Send more orders to increase the chance of a permanent failure.

  • You can also run the standalone dead-letter script from a separate terminal:
npx tsx src/dead-letter.ts

This prints the same data in a formatted terminal report instead of raw JSON.

Not seeing notifications arrive?

  • Make sure the notification worker terminal is still running. If you accidentally closed it, restart with npx tsx src/notification-worker.ts.
  • Remember the 5-second delay. Notifications appear 5 seconds after the payment succeeds, not immediately.
  • If payment keeps failing, that is expected. The 30% random failure can hit multiple times in a row. Send more orders to see some succeed.
  • Help me debug why my notification worker is not processing jobs.

Your three-service pipeline is fully operational. You can place orders, watch payments process with retries, see delayed notifications deliver, and inspect any permanently failed jobs. Next up, you will take on a secret mission to add a visual dashboard for monitoring all of this in the browser.

Secret mission

Add Bull Board Dashboard

Production systems need observability. In this bonus challenge, you'll integrate Bull Board into your order service to get a real-time visual dashboard showing queue states, job data, retry history, and failed job details. No more checking terminal logs to understand what's happening in your pipeline.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • A Docker container (redis-bullmq) running Redis on port 6379.
  • Local project folder order-pipeline on your Desktop with all source files and node_modules.

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with BullMQ queues.

  • Your Docker Redis container continues running in the background at no cost.
  • Your project files stay on your Desktop, ready to use anytime.
  • You can restart the services with npm run order-service, npm run payment-worker, and npm run notification-worker whenever you want.

Pause - I'll come back to this later

Stop the Docker container to free up memory while keeping your data and project files intact.

  • Stop any running Node.js processes in your terminals by pressing Ctrl+C in each one.
  • Stop the Redis container by running this command from your order-pipeline folder:
docker compose stop
  • When you're ready to come back, restart Redis with:
docker compose start

Your queued jobs and Redis data persist between stops.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the Docker container, its data volume, and your local project files.

  • Stop any running Node.js processes in your terminals by pressing Ctrl+C in each one.
  • Remove the Redis container and its data volume by running this from your order-pipeline folder:
docker compose down -v
  • Delete the project folder from your Desktop:

macOS

cd ~/Desktop
rm -rf order-pipeline

Windows

cd ~\Desktop
rmdir /s /q order-pipeline

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete event-driven order processing pipeline with three independent TypeScript microservices communicating reliably through BullMQ job queues.

You've learned how to:

  • Built three independent microservices (Order Service, Payment Worker, Notification Worker) that communicate through BullMQ job queues backed by Redis.
  • Experienced first-hand why Redis Pub/Sub loses messages when subscribers disconnect, and replaced it with persistent, reliable delivery.
  • Implemented automatic retries with exponential backoff, dead-letter queue handling for permanently failed jobs, and delayed job scheduling for timed notification delivery.
  • Built a production-style monitoring dashboard with Bull Board to visually observe queue state, job counts, and retry history in the browser (Secret Mission).

Ready to quiz yourself?