Build a Resilient Order Processing API

Build a production-grade API with circuit breakers, rate limiting, and async queues.

Introduction

30 Second Summary

When you order something online and the payment system goes down, the best services don't crash. They queue your order, serve a graceful fallback, and retry when the system recovers.

In this project, you will build an order processing API that handles downstream failures using a circuit breaker, queues work for async processing, and automatically recovers when services come back online.

What You'll Build

A TypeScript Express API that accepts incoming orders, rate-limits clients, processes orders asynchronously via BullMQ, and protects a simulated payment service with the circuit breaker pattern.

By the end of this project, you'll have:

  • A production-grade REST API with rate limiting and structured request tracing.
  • A working circuit breaker that trips during payment failures, serves fallback responses, and self-heals when the downstream service recovers.
  • Health check endpoints and graceful shutdown handling that preserves in-flight work and reflects circuit breaker state.
  • Secret Mission: A dead-letter queue system with admin endpoints to inspect and retry permanently failed orders.

Are there any prerequisites?

This project assumes you're familiar with Node.js, Express, and TypeScript fundamentals. You don't need prior experience with circuit breakers, Redis, or Docker.

Set Up Docker and Redis

Your resilient API needs a message queue to process orders asynchronously. Redis will serve as the backbone for that queue, storing jobs and coordinating work between the API server and the background worker.

In this step, you'll get Redis running inside a Docker container. This gives you an isolated, disposable Redis instance that's easy to start and stop without installing Redis directly on your machine.

In this step, get ready to:
  • Verify Docker Desktop is running.
  • Pull and run a Redis container.
  • Confirm Redis is reachable from your machine.
Verify Docker Desktop is running

Docker Desktop needs to be running in the background for any container commands to work. Before pulling the Redis image, confirm your terminal can reach it.

  • Open a terminal. Press Cmd+Space (macOS) or the Windows key (Windows), type Terminal (macOS) or Command Prompt (Windows), then press Enter.
  • Confirm Docker is accessible by running this command:
docker --version

You should see a version number printed, confirming Docker is installed and reachable from your terminal.

Not seeing a version number?

Make sure Docker Desktop is open and running. Look for the whale icon in your system tray (Windows) or menu bar (macOS). If Docker Desktop is not installed, download it from the official Docker website at docker.com/products/docker-desktop.

help me fix Docker not being recognized in my terminal

Pull and run the Redis container

Redis doesn't have official Windows binaries. Running it inside a container gives you a consistent experience across macOS and Windows.

  • Start a Redis container by running this command:
docker run --name redis-resilient -p 6379:6379 -d redis:latest

What does this command do?

  • --name redis-resilient gives the container a memorable name you can reference later.
  • -p 6379:6379 maps Redis's default port inside the container to the same port on your machine, so local apps can connect to 127.0.0.1:6379.
  • -d runs the container in the background (detached mode).
  • redis:latest pulls the official Redis image from Docker Hub.
  • You should see a long string of characters (the container ID) printed to your terminal. This means Docker pulled the image and started the container.
  • Confirm the container is running by running this command:
docker ps

You should see redis-resilient listed in the output with a status of Up.

Container not starting?

If you see an error about the name already being in use, a container with that name already exists. Remove it first:

docker rm redis-resilient

Then re-run the docker run command above.

If port 6379 is already in use, another process is occupying that port. Stop any other Redis instances or services listening on 6379 before trying again.

help me troubleshoot my Docker container not starting

Verify the Redis connection is reachable

Before writing any code, confirm Redis is accepting connections. The redis-cli tool inside the container can send a quick ping to check.

  • Test that Redis is responding by running this command:
docker exec redis-resilient redis-cli ping

You should see PONG printed to your terminal. This confirms Redis is running and accepting connections on port 6379.

Not seeing PONG?

Check that the container is still running:

docker ps

If redis-resilient doesn't appear in the list, restart it:

docker start redis-resilient

help me get Redis responding to my ping command

Redis is up and accepting connections. Next up, you'll scaffold the TypeScript project that connects to it.

Scaffold the TypeScript Project

Your Redis instance is running and ready to accept connections. Now it's time to build the application that talks to it.

In this step, you'll create a TypeScript project and install the libraries that power rate limiting, job queuing, and circuit breaker protection.

In this step, get ready to:
  • Initialize a Node.js project with TypeScript configuration.
  • Install production dependencies for Express, BullMQ, and opossum.
  • Install dev dependencies and configure build scripts.
Initialize a Node.js project with TypeScript configuration

Every Node.js project starts with a package.json file that tracks dependencies and scripts. You also need a tsconfig.json to tell TypeScript how to compile your code.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir resilient-order-api
cd resilient-order-api
  • Initialize a new Node.js project by running:
npm init -y
  • Confirm the folder and package.json exist by running:
ls

You should see package.json listed in the output.

Don't see package.json?

Make sure you ran npm init -y from inside the resilient-order-api folder, not from your Desktop.

Help me troubleshoot my npm init command.

  • Open the project in your editor. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Cursor (or VS Code) and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the resilient-order-api folder on your Desktop, then click Open.

Now create the TypeScript configuration file. This tells the TypeScript compiler how to process your code and where to output the compiled JavaScript.

  • Create a new file called tsconfig.json in the root of your resilient-order-api folder.
  • Paste the following configuration into tsconfig.json:
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "outDir": "dist",
    "rootDir": "src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "declaration": true
  },
  "include": ["src/**/*"]
}

What does this configuration do?

  • target: "ES2022" compiles to modern JavaScript that Node.js 20+ supports natively.
  • module: "NodeNext" enables ES module syntax with proper Node.js resolution.
  • rootDir: "src" tells TypeScript your source lives in a src/ folder, and outDir: "dist" is where compiled JavaScript goes.
  • strict: true enables all strict type-checking options for maximum safety.
  • Save tsconfig.json.
  • Create the source directory by running this in your terminal:
mkdir src
  • Confirm the src directory was created by running:
ls

You should see package.json, tsconfig.json, and src in the output.

Install production dependencies

Your API needs six libraries to handle HTTP requests, job queuing, Redis connections, circuit breaking, rate limiting, and unique ID generation.

  • Install all production dependencies by running:
npm install express@5.1.0 bullmq@5.78.0 ioredis@5.11.0 opossum@9.0.0 express-rate-limit@8.5.2 uuid@11.1.0

What are these packages?

  • express is the HTTP framework for your API routes.
  • bullmq is a Redis-based job queue for processing orders asynchronously.
  • ioredis is the Redis client that BullMQ and your app use to connect to Redis.
  • opossum implements the circuit breaker pattern to protect calls to external services.
  • express-rate-limit limits how many requests a single client can make in a time window.
  • uuid generates unique identifiers for orders and request tracing.

You should see output ending with "added X packages" confirming the install succeeded.

Seeing network or permission errors?

Make sure you have an internet connection. If you see permission errors on macOS, avoid using sudo. Instead, fix npm permissions by following npm's official guide.

Help me fix my npm install error.

Install dev dependencies and configure build scripts

Dev dependencies provide TypeScript type definitions and the development server. They aren't needed in production but are essential while you're building.

  • Install the dev dependencies by running:
npm install --save-dev @types/express@5.0.2 @types/node@22.15.21 @types/opossum@8.1.7 @types/uuid@10.0.0 tsx@4.19.4 typescript@5.8.3

What are the @types packages?

TypeScript needs type definitions to understand third-party libraries. The @types/ packages provide those definitions so your editor can offer autocompletion and catch type errors.

tsx lets you run TypeScript files directly during development without a manual compile step.

Now configure the build scripts so you can compile, run, and develop your project. You need to update package.json with the correct module type and scripts.

  • Open package.json in your editor.
  • Replace the entire file contents with the following:
{
  "name": "resilient-order-api",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "build": "tsc",
    "start": "node dist/index.js",
    "dev": "tsx watch src/index.ts"
  },
  "dependencies": {
    "bullmq": "5.78.0",
    "express": "5.1.0",
    "express-rate-limit": "8.5.2",
    "ioredis": "5.11.0",
    "opossum": "9.0.0",
    "uuid": "11.1.0"
  },
  "devDependencies": {
    "@types/express": "5.0.2",
    "@types/node": "22.15.21",
    "@types/opossum": "8.1.7",
    "@types/uuid": "10.0.0",
    "tsx": "4.19.4",
    "typescript": "5.8.3"
  }
}

What do these scripts do?

  • "type": "module" enables ES module import/export syntax instead of require().
  • "build": "tsc" compiles TypeScript to JavaScript in the dist/ folder.
  • "start": "node dist/index.js" runs the compiled production build.
  • "dev": "tsx watch src/index.ts" runs your TypeScript directly with hot-reload during development.
  • Save package.json.
  • Verify the TypeScript compiler is accessible by running:
npx tsc --version

You should see Version 5.8.3 printed in the terminal.

Not seeing the TypeScript version?

Make sure your terminal is inside the resilient-order-api folder. The npx command looks for locally installed packages in node_modules.

If you see a different version, re-run the dev dependency install command above to ensure typescript@5.8.3 is installed.

Help me fix my TypeScript installation.

✔️ Awesome, I've got everything!

Great. Double check you've saved both files before moving on.

ⓧ I'd like to double check the full code

Here is the complete package.json:

{
  "name": "resilient-order-api",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "build": "tsc",
    "start": "node dist/index.js",
    "dev": "tsx watch src/index.ts"
  },
  "dependencies": {
    "bullmq": "5.78.0",
    "express": "5.1.0",
    "express-rate-limit": "8.5.2",
    "ioredis": "5.11.0",
    "opossum": "9.0.0",
    "uuid": "11.1.0"
  },
  "devDependencies": {
    "@types/express": "5.0.2",
    "@types/node": "22.15.21",
    "@types/opossum": "8.1.7",
    "@types/uuid": "10.0.0",
    "tsx": "4.19.4",
    "typescript": "5.8.3"
  }
}

And the complete tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "outDir": "dist",
    "rootDir": "src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "declaration": true
  },
  "include": ["src/**/*"]
}

Your project skeleton is ready with all the tools you need. Next up, you'll build the Express API with rate limiting and start accepting orders.

Build the Express API with Rate Limiting

Your project structure is ready. Now you need the actual server that accepts incoming orders and protects itself from being overwhelmed.

In this step, you'll build the Express API layer piece by piece. You'll start with a Redis connection module and a job queue, then layer on request tracing, rate limiting, and an orders endpoint that enqueues work instead of processing it synchronously.

In this step, get ready to:
  • Create the Redis connection and BullMQ queue modules.
  • Build an Express server with request ID tracing and rate limiting.
  • Create a POST /orders endpoint that validates and enqueues orders for async processing.
Create the Express server with JSON parsing and request ID middleware

Before you can build the main server file, you need two supporting modules. The first connects to Redis (your message broker). The second defines the job queue where incoming orders will be placed.

  • Create a new file called src/redis.ts and add the following code:
import Redis from "ioredis";

// Create a shared Redis connection for the entire app
export const redisConnection = new Redis({
  host: "127.0.0.1",
  port: 6379,
  maxRetriesPerRequest: null,
});

// Log connection events for visibility
redisConnection.on("connect", () => {
  console.log(JSON.stringify({ level: "info", message: "Connected to Redis" }));
});

redisConnection.on("error", (err) => {
  console.log(JSON.stringify({ level: "error", message: "Redis connection error", error: err.message }));
});

What does this code do?

This module creates a single Redis connection that the rest of the app shares. It connects to 127.0.0.1:6379 where your Docker Redis container is running.

The maxRetriesPerRequest: null setting is required by BullMQ. Without it, BullMQ workers will throw unhandled timeout errors during blocking operations.

  • Save src/redis.ts.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

import Redis from "ioredis";

export const redisConnection = new Redis({
  host: "127.0.0.1",
  port: 6379,
  maxRetriesPerRequest: null,
});

redisConnection.on("connect", () => {
  console.log(JSON.stringify({ level: "info", message: "Connected to Redis" }));
});

redisConnection.on("error", (err) => {
  console.log(JSON.stringify({ level: "error", message: "Redis connection error", error: err.message }));
});
  • Create a new file called src/queue.ts and add the following code:
import { Queue } from "bullmq";
import { redisConnection } from "./redis.js";

// Define the order processing queue with retry logic
export const orderQueue = new Queue("orders", {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: "exponential",
      delay: 1000,
    },
    removeOnComplete: 100,
    removeOnFail: false,
  },
});

What does this code do?

This creates a BullMQ queue named orders. Every order the API receives will be placed here for asynchronous processing.

The defaultJobOptions configure retry behaviour. Failed jobs get 3 attempts with exponential backoff (1s, 2s, 4s). Completed jobs are pruned after 100 accumulate, but failed jobs are kept for inspection.

  • Save src/queue.ts.

Why .js extensions in TypeScript imports?

You'll notice the import uses "./redis.js" even though the file is redis.ts. Node.js ESM resolution requires the compiled output extension. TypeScript resolves .js imports back to .ts source files automatically.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

import { Queue } from "bullmq";
import { redisConnection } from "./redis.js";

export const orderQueue = new Queue("orders", {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: "exponential",
      delay: 1000,
    },
    removeOnComplete: 100,
    removeOnFail: false,
  },
});

Now you'll build the main server file. This is where Express is configured with JSON parsing, request tracing, and the rate limiter.

  • Create a new file called src/index.ts and add the following code:
import express from "express";
import { rateLimit } from "express-rate-limit";
import { v4 as uuidv4 } from "uuid";
import { orderQueue } from "./queue.js";
import { circuitBreaker } from "./circuit-breaker.js";
import { startWorker, stopWorker } from "./worker.js";
import { redisConnection } from "./redis.js";

const app = express();
const PORT = 3000;

// Parse incoming JSON request bodies
app.use(express.json());

// Request ID middleware
app.use((req, res, next) => {
  req.headers["x-request-id"] = req.headers["x-request-id"] || uuidv4();
  res.setHeader("x-request-id", req.headers["x-request-id"] as string);
  console.log(
    JSON.stringify({
      level: "info",
      requestId: req.headers["x-request-id"],
      method: req.method,
      path: req.path,
      timestamp: new Date().toISOString(),
    })
  );
  next();
});

What does this code do?

The imports pull in all the modules this server will use. Two of them (./circuit-breaker.js and ./worker.js) don't exist yet. You'll create them in the next steps.

The request ID middleware assigns a unique ID to every incoming request. If the client sends an x-request-id header, the server reuses it. Otherwise, it generates one with uuidv4(). This makes it possible to trace a single request through your entire system.

What is structured logging?

Instead of plain text like console.log("got a request"), the middleware outputs JSON with consistent fields. Production systems ingest structured logs into tools like Datadog or CloudWatch, where you can filter by requestId, method, or path to debug issues quickly.

  • Save src/index.ts.

Seeing TypeScript errors in your editor?

Your editor may show red underlines on the ./circuit-breaker.js and ./worker.js imports. That's expected. Those files will be created in Steps 4 and 5. The errors will disappear once those modules exist.

help me understand why my TypeScript imports are showing errors

Add rate limiting middleware with standard headers

Rate limiting prevents a single client from overwhelming your API. Without it, one misbehaving client could consume all your server's resources and deny service to everyone else.

  • In src/index.ts, add the following code below the request ID middleware:
// Rate limiting
const limiter = rateLimit({
  windowMs: 60 * 1000,
  limit: 20,
  standardHeaders: "draft-8",
  legacyHeaders: false,
  message: { error: "Too many requests, please try again later." },
});

app.use(limiter);

What does this code do?

This configures express-rate-limit to allow 20 requests per 60-second window per client. The standardHeaders: "draft-8" option adds standard RateLimit-Limit and RateLimit-Remaining response headers so clients can see their usage.

When a client exceeds 20 requests in a minute, the server responds with a 429 status and the JSON error message you defined.

  • Save src/index.ts.
Create the POST /orders endpoint that validates and enqueues orders

The orders endpoint is the core of this API. Instead of processing orders synchronously (which would block the server), it validates the request and drops the order into the BullMQ queue for a worker to handle later.

  • In src/index.ts, add the following code below app.use(limiter):
// POST /orders
app.post("/orders", async (req, res) => {
  const { customerId, items, total } = req.body;

  // Validate required fields
  if (!customerId || !items || !total) {
    res.status(400).json({ error: "Missing required fields: customerId, items, total" });
    return;
  }

  // Generate a unique order ID and enqueue for processing
  const orderId = uuidv4();
  await orderQueue.add("process-order", {
    orderId,
    customerId,
    items,
    total,
    createdAt: new Date().toISOString(),
  });

  console.log(
    JSON.stringify({
      level: "info",
      requestId: req.headers["x-request-id"],
      message: "Order queued",
      orderId,
    })
  );

  // 202 Accepted: the request is valid, processing happens later
  res.status(202).json({
    message: "Order accepted for processing",
    orderId,
    status: "queued",
  });
});

What does this code do?

The endpoint validates that customerId, items, and total are present. If any are missing, it returns a 400 error immediately.

For valid requests, it generates a unique orderId and places the order onto the BullMQ queue. The response is HTTP 202 (Accepted), which signals that the server acknowledged the request but hasn't finished processing it yet. The worker you build in Step 5 will pick up these queued jobs.

  • Save src/index.ts.

Now add the health check endpoints, server startup, and graceful shutdown logic to complete the file.

  • In src/index.ts, add the following code below the POST /orders endpoint:
// Health checks
app.get("/health/live", (req, res) => {
  res.status(200).json({ status: "alive", timestamp: new Date().toISOString() });
});

app.get("/health/ready", (req, res) => {
  const circuitState = circuitBreaker.status.stats;
  const isCircuitClosed = !circuitBreaker.opened;

  if (isCircuitClosed) {
    res.status(200).json({
      status: "ready",
      circuit: "closed",
      stats: {
        successes: circuitState.successes,
        failures: circuitState.failures,
        fires: circuitState.fires,
      },
    });
  } else {
    res.status(503).json({
      status: "not ready",
      circuit: "open",
      reason: "Payment service circuit breaker is open",
      stats: {
        successes: circuitState.successes,
        failures: circuitState.failures,
        fires: circuitState.fires,
      },
    });
  }
});

// Start server and worker
const server = app.listen(PORT, () => {
  console.log(
    JSON.stringify({
      level: "info",
      message: `Server running on port ${PORT}`,
      timestamp: new Date().toISOString(),
    })
  );
  startWorker();
});

// Graceful shutdown
const shutdown = async (signal: string) => {
  console.log(
    JSON.stringify({
      level: "info",
      message: `${signal} received, starting graceful shutdown`,
      timestamp: new Date().toISOString(),
    })
  );

  server.close(async () => {
    console.log(JSON.stringify({ level: "info", message: "HTTP server closed" }));
    await stopWorker();
    await orderQueue.close();
    await redisConnection.quit();
    console.log(JSON.stringify({ level: "info", message: "All connections closed, exiting" }));
    process.exit(0);
  });

  setTimeout(() => {
    console.log(JSON.stringify({ level: "error", message: "Forced shutdown after timeout" }));
    process.exit(1);
  }, 10000);
};

process.on("SIGTERM", () => shutdown("SIGTERM"));
process.on("SIGINT", () => shutdown("SIGINT"));

What does this code do?

The /health/live endpoint confirms the server process is running. The /health/ready endpoint reports whether the system can actually process work. It checks the circuit breaker state you'll build in Step 4.

The graceful shutdown handler listens for SIGTERM and SIGINT signals. When received, it stops accepting new connections, waits for in-flight work to finish, closes the queue and Redis connection, then exits cleanly. A 10-second timeout forces exit if cleanup stalls.

Why two health check endpoints?

In production, orchestrators like Kubernetes use liveness probes to restart crashed containers and readiness probes to stop sending traffic to unhealthy instances. Separating them lets you restart only when the process is truly dead, while temporarily routing traffic away when a downstream dependency fails.

  • Save src/index.ts.
  • You won't be able to run the dev server yet because src/circuit-breaker.ts and src/worker.ts don't exist. You'll create them in the next two steps.

✔️ Awesome, I've got everything!

Great. Double check you've saved all three files before moving on.

ⓧ I'd like to double check the full code

import express from "express";
import { rateLimit } from "express-rate-limit";
import { v4 as uuidv4 } from "uuid";
import { orderQueue } from "./queue.js";
import { circuitBreaker } from "./circuit-breaker.js";
import { startWorker, stopWorker } from "./worker.js";
import { redisConnection } from "./redis.js";

const app = express();
const PORT = 3000;

app.use(express.json());

// Request ID middleware
app.use((req, res, next) => {
  req.headers["x-request-id"] = req.headers["x-request-id"] || uuidv4();
  res.setHeader("x-request-id", req.headers["x-request-id"] as string);
  console.log(
    JSON.stringify({
      level: "info",
      requestId: req.headers["x-request-id"],
      method: req.method,
      path: req.path,
      timestamp: new Date().toISOString(),
    })
  );
  next();
});

// Rate limiting
const limiter = rateLimit({
  windowMs: 60 * 1000,
  limit: 20,
  standardHeaders: "draft-8",
  legacyHeaders: false,
  message: { error: "Too many requests, please try again later." },
});

app.use(limiter);

// POST /orders
app.post("/orders", async (req, res) => {
  const { customerId, items, total } = req.body;

  if (!customerId || !items || !total) {
    res.status(400).json({ error: "Missing required fields: customerId, items, total" });
    return;
  }

  const orderId = uuidv4();
  await orderQueue.add("process-order", {
    orderId,
    customerId,
    items,
    total,
    createdAt: new Date().toISOString(),
  });

  console.log(
    JSON.stringify({
      level: "info",
      requestId: req.headers["x-request-id"],
      message: "Order queued",
      orderId,
    })
  );

  res.status(202).json({
    message: "Order accepted for processing",
    orderId,
    status: "queued",
  });
});

// Health checks
app.get("/health/live", (req, res) => {
  res.status(200).json({ status: "alive", timestamp: new Date().toISOString() });
});

app.get("/health/ready", (req, res) => {
  const circuitState = circuitBreaker.status.stats;
  const isCircuitClosed = !circuitBreaker.opened;

  if (isCircuitClosed) {
    res.status(200).json({
      status: "ready",
      circuit: "closed",
      stats: {
        successes: circuitState.successes,
        failures: circuitState.failures,
        fires: circuitState.fires,
      },
    });
  } else {
    res.status(503).json({
      status: "not ready",
      circuit: "open",
      reason: "Payment service circuit breaker is open",
      stats: {
        successes: circuitState.successes,
        failures: circuitState.failures,
        fires: circuitState.fires,
      },
    });
  }
});

// Start server and worker
const server = app.listen(PORT, () => {
  console.log(
    JSON.stringify({
      level: "info",
      message: `Server running on port ${PORT}`,
      timestamp: new Date().toISOString(),
    })
  );
  startWorker();
});

// Graceful shutdown
const shutdown = async (signal: string) => {
  console.log(
    JSON.stringify({
      level: "info",
      message: `${signal} received, starting graceful shutdown`,
      timestamp: new Date().toISOString(),
    })
  );

  server.close(async () => {
    console.log(JSON.stringify({ level: "info", message: "HTTP server closed" }));
    await stopWorker();
    await orderQueue.close();
    await redisConnection.quit();
    console.log(JSON.stringify({ level: "info", message: "All connections closed, exiting" }));
    process.exit(0);
  });

  setTimeout(() => {
    console.log(JSON.stringify({ level: "error", message: "Forced shutdown after timeout" }));
    process.exit(1);
  }, 10000);
};

process.on("SIGTERM", () => shutdown("SIGTERM"));
process.on("SIGINT", () => shutdown("SIGINT"));
  • Confirm all three files exist by running:
ls src/

You should see index.ts, queue.ts, and redis.ts listed in the output.

Missing a file?

Make sure you saved each file inside the src/ directory, not in the project root. The filenames should be exactly redis.ts, queue.ts, and index.ts.

help me find where my files ended up

Your API skeleton is complete. You have Redis connectivity, a job queue with retry logic, request tracing, rate limiting, and an orders endpoint. Next up, you'll create the circuit breaker that protects your payment service calls from cascading failures.

Implement the Circuit Breaker and Payment Service

Your Express API accepts orders and queues them for processing. But what happens when the downstream payment service goes down?

Without protection, your worker would hammer the failing service with retries, wasting resources and cascading the failure upstream. In this step, you'll build a simulated payment service and wrap it in a circuit breaker that detects failures, stops the bleeding, and automatically recovers when the service comes back online.

In this step, get ready to:
  • Create a simulated payment service with controllable failure modes.
  • Wrap the payment service in an opossum circuit breaker with fallback responses.
  • Wire circuit breaker state changes to structured logging.
Create a simulated payment service that randomly fails

To test a circuit breaker, you need a service that fails in a predictable pattern. A real third-party payment API would be unpredictable and require credentials. Instead, you'll build a local simulation that toggles between healthy and failing states every 5 requests.

  • Create a new file called src/payment-service.ts by adding the following code:
let failureMode = false;
let requestCount = 0;

// Simulate a payment service that fails periodically
export async function processPayment(orderId: string, total: number): Promise<{ transactionId: string }> {
  requestCount++;

  // Toggle failure mode every 5 requests to simulate intermittent failures
  if (requestCount % 5 === 0) {
    failureMode = !failureMode;
  }

  // Simulate network latency
  await new Promise((resolve) => setTimeout(resolve, 100 + Math.random() * 200));

  if (failureMode) {
    throw new Error(`Payment service unavailable for order ${orderId}`);
  }

  return {
    transactionId: `txn_${Date.now()}_${orderId.slice(0, 8)}`,
  };
}

// Allow manual control for testing
export function setFailureMode(enabled: boolean): void {
  failureMode = enabled;
}

What does this code do?

This file simulates an external payment gateway that experiences intermittent outages.

  • requestCount tracks how many times the service has been called. Every 5th request, it flips failureMode on or off.
  • The setTimeout adds 100-300ms of artificial latency to mimic real network calls.
  • When failureMode is true, every request throws an error. When false, it returns a transaction ID.
  • setFailureMode lets you manually force failures during testing without waiting for the 5-request cycle.
  • Save src/payment-service.ts.

You won't see a change yet. The circuit breaker file you'll create next imports from this service.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

let failureMode = false;
let requestCount = 0;

// Simulate a payment service that fails periodically
export async function processPayment(orderId: string, total: number): Promise<{ transactionId: string }> {
  requestCount++;

  // Toggle failure mode every 5 requests to simulate intermittent failures
  if (requestCount % 5 === 0) {
    failureMode = !failureMode;
  }

  // Simulate network latency
  await new Promise((resolve) => setTimeout(resolve, 100 + Math.random() * 200));

  if (failureMode) {
    throw new Error(`Payment service unavailable for order ${orderId}`);
  }

  return {
    transactionId: `txn_${Date.now()}_${orderId.slice(0, 8)}`,
  };
}

// Allow manual control for testing
export function setFailureMode(enabled: boolean): void {
  failureMode = enabled;
}
Wrap the payment service call in an opossum circuit breaker with fallback

A circuit breaker sits between your application and an unreliable service. It tracks success and failure rates. When failures exceed a threshold, it "trips" and immediately returns a fallback response instead of calling the broken service.

What is a circuit breaker?

Think of a household circuit breaker. When too much current flows through a wire, the breaker trips to prevent a fire. It stays tripped until you manually reset it.

A software circuit breaker works the same way. It monitors calls to a service. When too many fail (crossing the error threshold), it trips open and stops making calls. After a cooldown period, it enters a half-open state and lets one test request through. If that succeeds, the circuit closes and traffic resumes normally.

Why opossum over other libraries?

Opossum is the standard Node.js circuit breaker library, maintained by Red Hat's nodeshift team. It directly implements the Fowler circuit breaker pattern with a clean event-driven API.

Alternatives like polly-ts are newer with minimal community adoption. Decorator-based libraries like resilia require experimental TypeScript features. Opossum gives you the clearest mental model with zero exotic configuration.

  • Create a new file called src/circuit-breaker.ts and add the circuit breaker setup with its fallback:
import CircuitBreaker from "opossum";
import { processPayment } from "./payment-service.js";

export const circuitBreaker = new CircuitBreaker(processPayment, {
  timeout: 5000,
  errorThresholdPercentage: 50,
  resetTimeout: 10000,
  volumeThreshold: 3,
});

// Fallback when circuit is open
circuitBreaker.fallback((orderId: string) => ({
  transactionId: `pending_${orderId.slice(0, 8)}`,
  degraded: true,
  message: "Payment queued for retry, circuit breaker is open",
}));

What does this code do?

  • new CircuitBreaker(processPayment, options) wraps the payment function so opossum can monitor every call.
  • timeout: 5000 means any single payment call that takes longer than 5 seconds is treated as a failure.
  • errorThresholdPercentage: 50 trips the circuit when half of recent calls have failed.
  • resetTimeout: 10000 keeps the circuit open for 10 seconds before attempting a test request (half-open state).
  • volumeThreshold: 3 requires at least 3 calls before the circuit can trip. This prevents a single early failure from opening the circuit.
  • The fallback function runs instead of the real payment call when the circuit is open. It returns a degraded response so the caller knows the payment is pending.
  • Save src/circuit-breaker.ts.

You won't see a change yet. The next chunk adds event logging so you can observe the circuit breaker's state transitions.

Seeing a type error on the import?

Make sure src/payment-service.ts exists and is saved. The import path uses ./payment-service.js (with the .js extension) because TypeScript's NodeNext module resolution requires file extensions in imports, even though the source file is .ts.

help me fix the import error in my circuit-breaker.ts file

Wire the circuit breaker events to structured logging

A circuit breaker that trips silently is useless in production. You need to know exactly when the circuit opens, when it tests recovery, and when service is restored.

  • In src/circuit-breaker.ts, add the following event listeners below the fallback call:
// Event logging
circuitBreaker.on("open", () => {
  console.log(
    JSON.stringify({
      level: "warn",
      message: "Circuit breaker OPENED - payment service is failing",
      timestamp: new Date().toISOString(),
    })
  );
});

circuitBreaker.on("halfOpen", () => {
  console.log(
    JSON.stringify({
      level: "info",
      message: "Circuit breaker HALF-OPEN - testing payment service",
      timestamp: new Date().toISOString(),
    })
  );
});

circuitBreaker.on("close", () => {
  console.log(
    JSON.stringify({
      level: "info",
      message: "Circuit breaker CLOSED - payment service recovered",
      timestamp: new Date().toISOString(),
    })
  );
});

circuitBreaker.on("fallback", () => {
  console.log(
    JSON.stringify({
      level: "warn",
      message: "Circuit breaker fallback executed",
      timestamp: new Date().toISOString(),
    })
  );
});

What does this code do?

Opossum emits events as the circuit transitions between states. Each listener outputs a structured JSON log line so you can trace the circuit's lifecycle in production logging systems.

  • open fires when failures cross the threshold. This is your alert that the downstream service is broken.
  • halfOpen fires after the resetTimeout expires. The breaker allows one test request through to check if the service recovered.
  • close fires when the test request succeeds. Traffic resumes normally.
  • fallback fires every time the fallback function executes instead of the real call.
  • Save src/circuit-breaker.ts.
  • Check your terminal. The app still won't fully start because src/worker.ts doesn't exist yet. You'll create that in the next step.

Seeing a module not found error?

This is expected. Your src/index.ts imports from ./worker.js which you haven't created yet. The app will compile and run after the next step.

help me understand why my app won't start after adding the circuit breaker

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

import CircuitBreaker from "opossum";
import { processPayment } from "./payment-service.js";

export const circuitBreaker = new CircuitBreaker(processPayment, {
  timeout: 5000,
  errorThresholdPercentage: 50,
  resetTimeout: 10000,
  volumeThreshold: 3,
});

// Fallback when circuit is open
circuitBreaker.fallback((orderId: string) => ({
  transactionId: `pending_${orderId.slice(0, 8)}`,
  degraded: true,
  message: "Payment queued for retry, circuit breaker is open",
}));

// Event logging
circuitBreaker.on("open", () => {
  console.log(
    JSON.stringify({
      level: "warn",
      message: "Circuit breaker OPENED - payment service is failing",
      timestamp: new Date().toISOString(),
    })
  );
});

circuitBreaker.on("halfOpen", () => {
  console.log(
    JSON.stringify({
      level: "info",
      message: "Circuit breaker HALF-OPEN - testing payment service",
      timestamp: new Date().toISOString(),
    })
  );
});

circuitBreaker.on("close", () => {
  console.log(
    JSON.stringify({
      level: "info",
      message: "Circuit breaker CLOSED - payment service recovered",
      timestamp: new Date().toISOString(),
    })
  );
});

circuitBreaker.on("fallback", () => {
  console.log(
    JSON.stringify({
      level: "warn",
      message: "Circuit breaker fallback executed",
      timestamp: new Date().toISOString(),
    })
  );
});

Your circuit breaker is ready to protect your system from cascading failures. Next up, you'll build the BullMQ worker that actually processes orders through this circuit breaker.

Build the BullMQ Order Worker

Your API accepts orders and queues them in Redis, but nothing is actually processing those jobs yet. The orders sit in the queue, waiting for a consumer to pick them up.

In this step, you will build the worker that pulls jobs from the queue, processes payments through the circuit breaker, and handles failures gracefully with automatic retries.

In this step, get ready to:
  • Create a BullMQ worker that processes orders from the queue.
  • Integrate the circuit breaker into the worker's processing logic.
  • Configure retry behaviour with exponential backoff.
Create the worker's order processing function

When a job arrives from the queue, the worker needs to extract the order data and attempt payment through the circuit breaker. If the circuit breaker is open and returns a degraded fallback, the worker should throw an error so the job gets retried later.

  • Create a new file called src/worker.ts in your resilient-order-api project folder.
  • Add the imports, type definition, and order processing function by pasting this code:
import { Worker, Job } from "bullmq";
import { redisConnection } from "./redis.js";
import { circuitBreaker } from "./circuit-breaker.js";

let worker: Worker | null = null;

interface OrderData {
  orderId: string;
  customerId: string;
  items: string[];
  total: number;
  createdAt: string;
}

async function processOrder(job: Job<OrderData>): Promise<void> {
  const { orderId, customerId, total } = job.data;

  console.log(
    JSON.stringify({
      level: "info",
      message: "Processing order",
      orderId,
      customerId,
      attempt: job.attemptsMade + 1,
      timestamp: new Date().toISOString(),
    })
  );

  const result = await circuitBreaker.fire(orderId, total);

  if (result && "degraded" in result && result.degraded) {
    throw new Error("Payment processed in degraded mode, retrying later");
  }

  console.log(
    JSON.stringify({
      level: "info",
      message: "Order processed successfully",
      orderId,
      transactionId: result.transactionId,
      timestamp: new Date().toISOString(),
    })
  );
}

What does this code do?

The OrderData interface defines the shape of job payloads coming from the queue. It matches the data structure your POST /orders endpoint adds to the queue.

The processOrder function is the processor that BullMQ calls for each job. It fires the circuit breaker with the order ID and total. If the circuit breaker is open, the fallback returns an object with degraded: true. The worker detects this and throws, which tells BullMQ to retry the job later.

Add the worker lifecycle functions

The worker needs a start function to begin consuming jobs and a stop function for graceful shutdown. These are already imported in src/index.ts from the previous step.

  • Add the startWorker and stopWorker functions below your processOrder function in src/worker.ts:
export function startWorker(): void {
  worker = new Worker("orders", processOrder, {
    connection: redisConnection,
    concurrency: 3,
  });

  worker.on("completed", (job) => {
    console.log(
      JSON.stringify({
        level: "info",
        message: "Job completed",
        jobId: job.id,
        orderId: job.data.orderId,
      })
    );
  });

  worker.on("failed", (job, err) => {
    console.log(
      JSON.stringify({
        level: "error",
        message: "Job failed",
        jobId: job?.id,
        orderId: job?.data.orderId,
        error: err.message,
        attempt: job?.attemptsMade,
      })
    );
  });

  console.log(JSON.stringify({ level: "info", message: "Order worker started" }));
}

export async function stopWorker(): Promise<void> {
  if (worker) {
    await worker.close();
    console.log(JSON.stringify({ level: "info", message: "Order worker stopped" }));
  }
}

What does this code do?

startWorker creates a new BullMQ Worker listening on the "orders" queue. The concurrency: 3 setting means it processes up to 3 jobs in parallel.

The completed and failed event listeners provide structured logging for each job's outcome. When a job fails, the log includes job.attemptsMade so you can track which retry attempt just failed.

stopWorker gracefully closes the worker, finishing any in-progress jobs before shutting down. Your src/index.ts already calls this during the SIGTERM/SIGINT shutdown sequence.

  • Save src/worker.ts.

Seeing TypeScript errors?

Make sure src/worker.ts is inside the src/ folder, not the project root. The tsconfig.json sets rootDir to src, so files outside that folder won't compile.

Also check that the import paths end with .js (not .ts). Node.js ESM requires the .js extension in import paths even for TypeScript files.

Help me debug TypeScript import errors in my BullMQ worker

Understand the retry configuration

The retry behaviour is already configured in src/queue.ts from the previous step. When a job fails, BullMQ automatically retries it based on the defaultJobOptions you set on the queue.

  • Switch back to src/queue.ts and find the defaultJobOptions block:
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: "exponential",
      delay: 1000,
    },
    removeOnComplete: 100,
    removeOnFail: false,
  },

How does exponential backoff work here?

attempts: 3 means each job gets 3 total processing attempts before BullMQ marks it as permanently failed.

The backoff setting controls the wait time between retries. With type: "exponential" and delay: 1000, the first retry waits 1 second, the second waits 2 seconds, and so on. This gives a failing downstream service time to recover without flooding it with requests.

removeOnFail: false keeps permanently failed jobs in Redis so you can inspect them later. This is essential for debugging production failures.

  • Confirm your dev server is still running. If you stopped it earlier, restart it by running:
npm run dev
  • You should see Connected to Redis, Server running on port 3000, and Order worker started in the terminal output.

Not seeing 'Order worker started'?

If you only see the server message but not the worker message, check that src/index.ts has import { startWorker, stopWorker } from "./worker.js" at the top and calls startWorker() inside the app.listen callback.

If you see a Redis connection error, verify your Redis container is running by opening a new terminal and running docker ps. You should see redis-resilient in the list.

Help me troubleshoot why my BullMQ worker isn't starting

✔️ Awesome, I've got everything!

Great. Make sure you have saved src/worker.ts and confirmed the three log messages appear when the server starts.

ⓧ I'd like to double check the full code

Here is the complete src/worker.ts file:

import { Worker, Job } from "bullmq";
import { redisConnection } from "./redis.js";
import { circuitBreaker } from "./circuit-breaker.js";

let worker: Worker | null = null;

interface OrderData {
  orderId: string;
  customerId: string;
  items: string[];
  total: number;
  createdAt: string;
}

async function processOrder(job: Job<OrderData>): Promise<void> {
  const { orderId, customerId, total } = job.data;

  console.log(
    JSON.stringify({
      level: "info",
      message: "Processing order",
      orderId,
      customerId,
      attempt: job.attemptsMade + 1,
      timestamp: new Date().toISOString(),
    })
  );

  const result = await circuitBreaker.fire(orderId, total);

  if (result && "degraded" in result && result.degraded) {
    throw new Error("Payment processed in degraded mode, retrying later");
  }

  console.log(
    JSON.stringify({
      level: "info",
      message: "Order processed successfully",
      orderId,
      transactionId: result.transactionId,
      timestamp: new Date().toISOString(),
    })
  );
}

export function startWorker(): void {
  worker = new Worker("orders", processOrder, {
    connection: redisConnection,
    concurrency: 3,
  });

  worker.on("completed", (job) => {
    console.log(
      JSON.stringify({
        level: "info",
        message: "Job completed",
        jobId: job.id,
        orderId: job.data.orderId,
      })
    );
  });

  worker.on("failed", (job, err) => {
    console.log(
      JSON.stringify({
        level: "error",
        message: "Job failed",
        jobId: job?.id,
        orderId: job?.data.orderId,
        error: err.message,
        attempt: job?.attemptsMade,
      })
    );
  });

  console.log(JSON.stringify({ level: "info", message: "Order worker started" }));
}

export async function stopWorker(): Promise<void> {
  if (worker) {
    await worker.close();
    console.log(JSON.stringify({ level: "info", message: "Order worker stopped" }));
  }
}

Your worker is now consuming jobs from the queue and processing payments through the circuit breaker. Next up, you will add health check endpoints and graceful shutdown so the system handles restarts without losing in-flight work.

Add Health Checks and Graceful Shutdown

Your API can accept orders, queue them, process payments through a circuit breaker, and retry failures. That covers the happy path and the unhappy path.

Production systems need two more capabilities: the ability to tell orchestrators whether they are healthy, and the ability to shut down without dropping in-flight work. In this step, you will review the health check endpoints and graceful shutdown handler already in your code, then test the entire system end-to-end.

In this step, get ready to:
  • Review the liveness and readiness health check endpoints.
  • Understand the graceful shutdown handler that preserves in-flight work.
  • Test the full flow: submit orders, observe the circuit breaker trip and recover, and shut down cleanly.
Review the health check endpoints

Orchestrators like Kubernetes or load balancers need a way to ask your service "are you alive?" and "are you ready to handle traffic?". Your src/index.ts already exposes two endpoints that answer those questions differently.

  • In src/index.ts, find the health check section. It starts with this code:
// Health checks
app.get("/health/live", (req, res) => {
  res.status(200).json({ status: "alive", timestamp: new Date().toISOString() });
});

What does /health/live do?

The liveness endpoint answers one question: is the process running? It always returns a 200 status with a simple JSON body. If this endpoint stops responding, the orchestrator knows the process has crashed and needs a restart.

  • Below the liveness endpoint, find the readiness check:
app.get("/health/ready", (req, res) => {
  const circuitState = circuitBreaker.status.stats;
  const isCircuitClosed = !circuitBreaker.opened;

  if (isCircuitClosed) {
    res.status(200).json({
      status: "ready",
      circuit: "closed",
      stats: {
        successes: circuitState.successes,
        failures: circuitState.failures,
        fires: circuitState.fires,
      },
    });
  } else {
    res.status(503).json({
      status: "not ready",
      circuit: "open",
      reason: "Payment service circuit breaker is open",
      stats: {
        successes: circuitState.successes,
        failures: circuitState.failures,
        fires: circuitState.fires,
      },
    });
  }
});

What does /health/ready do?

The readiness endpoint is smarter. It checks whether the circuit breaker is open or closed using circuitBreaker.opened.

When the circuit is closed (healthy), it returns 200. When the circuit is open (the payment service is failing), it returns 503. This tells the orchestrator to stop routing new traffic to this instance until the downstream service recovers.

Why two separate health endpoints?

Liveness tells the orchestrator the process is alive. Readiness tells it the process can do useful work. A service can be alive but not ready. If you collapse them into one, a tripped circuit breaker would cause restarts instead of temporary traffic diversion.

Understand the graceful shutdown handler

When a deployment rolls out a new version, the old process receives a termination signal. Without graceful shutdown, in-flight orders would be dropped mid-processing. Your shutdown handler gives the system time to finish current work before exiting.

  • In src/index.ts, scroll to the bottom and find the shutdown function:
// Graceful shutdown
const shutdown = async (signal: string) => {
  console.log(
    JSON.stringify({
      level: "info",
      message: `${signal} received, starting graceful shutdown`,
      timestamp: new Date().toISOString(),
    })
  );

  server.close(async () => {
    console.log(JSON.stringify({ level: "info", message: "HTTP server closed" }));
    await stopWorker();
    await orderQueue.close();
    await redisConnection.quit();
    console.log(JSON.stringify({ level: "info", message: "All connections closed, exiting" }));
    process.exit(0);
  });

  setTimeout(() => {
    console.log(JSON.stringify({ level: "error", message: "Forced shutdown after timeout" }));
    process.exit(1);
  }, 10000);
};

process.on("SIGTERM", () => shutdown("SIGTERM"));
process.on("SIGINT", () => shutdown("SIGINT"));

What does the shutdown handler do?

  • server.close() stops accepting new HTTP connections but lets in-flight requests finish.
  • stopWorker() tells the BullMQ worker to finish its current job, then stop polling for new ones.
  • orderQueue.close() and redisConnection.quit() cleanly disconnect from Redis.
  • The 10-second setTimeout is a safety net. If something hangs and the clean shutdown stalls, the process force-exits with code 1 so it does not linger indefinitely.

Why listen for both SIGTERM and SIGINT?

SIGTERM is what orchestrators (Docker, Kubernetes) send during deployments. SIGINT is what your terminal sends when you press Ctrl+C. Handling both means the same clean shutdown runs whether you are testing locally or deploying to production.

Test the full flow

Now that you understand every piece, it is time to prove the system works. You will submit orders, watch the circuit breaker trip, confirm the readiness endpoint reflects the state change, and then shut down cleanly.

  • Start the development server by running this command in your terminal:
npm run dev

You should see structured JSON logs confirming the server started on port 3000, the worker started, and Redis connected successfully.

  • Open a second terminal window. Test the liveness endpoint by running:
curl http://localhost:3000/health/live

You should see a JSON response with "status": "alive" and a timestamp.

  • Test the readiness endpoint by running:
curl http://localhost:3000/health/ready

You should see "status": "ready" and "circuit": "closed" since no failures have occurred yet.

What does the stats object show?

The stats field in the readiness response tracks how many times the circuit breaker has fired, succeeded, and failed. Right now all values are 0 because no orders have been processed.

  • Submit a test order by running this command:
curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -d '{"customerId": "cust-001", "items": ["widget-a", "widget-b"], "total": 49.99}'

What should I see?

The API returns a 202 response with an orderId and "status": "queued". In your first terminal (where the server is running), you will see logs showing the order was queued, then picked up by the worker, and processed against the payment service.

  • Submit several more orders rapidly to trigger the circuit breaker. Run this loop:
for i in $(seq 1 10); do
  curl -s -X POST http://localhost:3000/orders \
    -H "Content-Type: application/json" \
    -d "{\"customerId\": \"cust-00$i\", \"items\": [\"item-$i\"], \"total\": $((i * 10))}" &
done
wait

Windows alternative

If you are on Windows and not using Git Bash, run the single curl -X POST command from above 10 times manually, or use PowerShell:

1..10 | ForEach-Object { Start-Job { curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d ('{"customerId": "cust-' + $_ + '", "items": ["item"], "total": ' + ($_ * 10) + '}') } }
  • Watch the logs in your first terminal window.

After several requests, the simulated payment service starts failing. You should see log messages indicating the circuit breaker has opened, fallback responses are being served, and jobs are failing and being retried.

After 10 seconds (the resetTimeout value), you will see a "HALF-OPEN" log as the circuit breaker tests the service again. If the test succeeds, a "CLOSED" log appears and processing resumes normally.

  • Confirm the readiness endpoint reflects the circuit state by running:
curl http://localhost:3000/health/ready

If the circuit is currently open, you will see "status": "not ready" with a 503 status code. If it has recovered, you will see "status": "ready" again.

Timing matters

The payment service toggles failure mode every 5 requests, so the circuit breaker may open and close quickly. If you want to catch it in the open state, run the curl readiness check immediately after you see the "OPENED" log message.

  • Test graceful shutdown. In your first terminal (where the server is running), press Ctrl+C.

You should see a sequence of shutdown logs: "SIGINT received, starting graceful shutdown", then "HTTP server closed", "Order worker stopped", and finally "All connections closed, exiting". The process exits with code 0.

Server does not shut down cleanly?

If the process hangs after pressing Ctrl+C, a job may still be processing. Wait up to 10 seconds for the forced shutdown to kick in. If you see "Forced shutdown after timeout", that means a job or connection did not close in time.

If you see Redis connection errors after shutdown, make sure your Redis container is still running with docker ps. The container name should be redis-resilient.

Help me debug my graceful shutdown sequence.

You have just proven that your system handles the full lifecycle: accepting work, processing it with resilience patterns, reporting its own health, and shutting down without data loss. That is the foundation of a production-grade backend service.

Secret mission

Dead-Letter Queue with Admin Endpoint

Your circuit breaker handles transient failures gracefully, but what happens to orders that fail permanently? Right now, they sit in the failed jobs list with no way to inspect or recover them. Build a dead-letter queue (DLQ) that captures permanently failed orders, then expose admin endpoints to inspect and retry them.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker container redis-resilient running Redis on port 6379.
  • Project folder resilient-order-api on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend the project with the Secret Mission.

  • Your Redis container will continue running on port 6379 until you stop Docker Desktop or restart your machine.
  • Docker Desktop stays installed on your machine for future projects.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and container stay intact for later.

  • Stop the dev server by pressing Ctrl+C in the terminal where it is running.
  • Stop the Redis container by running:
docker stop redis-resilient
  • When you're ready to come back, restart the container by running:
docker start redis-resilient
  • Then start the dev server again with npm run dev from your project folder.

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

  • Stop the dev server by pressing Ctrl+C in the terminal if it is still running.
  • Stop and remove the Redis container by running:
docker stop redis-resilient
docker rm redis-resilient
  • Remove the Redis Docker image by running:
docker rmi redis:latest
  • Delete the project folder. On macOS/Linux, run:
cd ~/Desktop
rm -rf resilient-order-api
  • On Windows, run:
cd ~\Desktop
Remove-Item -Recurse -Force resilient-order-api
  • Docker Desktop remains installed on your machine. If you want to uninstall it, open Settings > Apps > Apps & features on Windows and uninstall Docker Desktop. On macOS, open Docker Desktop, click the Troubleshoot icon (top right), then click Uninstall.

Nice Work!

Nice Work!

You just built a production-grade resilient order processing API that handles real-world failure scenarios. Your system rate-limits clients, processes orders asynchronously, protects downstream calls with a circuit breaker, and shuts down gracefully without losing work.

You've learned how to:

  • Build a TypeScript Express API with rate limiting and structured request tracing.
  • Implement the circuit breaker pattern with opossum, including fallback responses and automatic recovery when the downstream service comes back online.
  • Use BullMQ for async job processing with retries and exponential backoff, plus health check endpoints that reflect circuit state and graceful shutdown that preserves in-flight work.
  • Build a dead-letter queue system with admin inspection and retry capabilities.

Ready to quiz yourself?