Order Processing Pipeline with BullMQ

Build a production event-driven order pipeline with job queues and retries.

Introduction

30 Second Summary

When you place an order online, you get an instant confirmation. But behind the scenes, the system is checking inventory, processing payment, and sending emails. All of that heavy work happens in the background so the website stays fast.

In this project, you will build a production-grade order processing pipeline that accepts webhook events, validates them, and processes them asynchronously using background job queues. You will separate the fast "accept the request" path from the slow "do the work" path, just like Shopify and Stripe do at scale.

What You'll Build

You will fire a simulated e-commerce webhook at your API, get back a response in milliseconds, then watch a separate worker process pick up the job and run it through inventory checks, payment processing, and order confirmation with full retry logic.

By the end of this project, you'll have:

  • A webhook API that validates HMAC signatures and returns a 200 response in under 50ms while a separate BullMQ worker handles the heavy processing in the background.
  • Automatic exponential backoff retries (2s, 4s, 8s delays) with structured logs showing attempt count, error context, and dead-letter handling for permanently failed jobs.
  • Two layers of idempotency protection that safely skip duplicate webhooks, plus graceful shutdown that lets in-flight jobs finish before the worker process exits.
  • Secret Mission: Add job priority levels so VIP customer orders jump the queue ahead of regular orders.

Are there any prerequisites?

This project assumes basic REST API experience with Express and MongoDB. You should be comfortable creating routes and saving documents to a database.

You will need a free Upstash Redis account (no credit card required) and a free MongoDB Atlas account.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for production backends.

Set Up the Project

Every production webhook system starts with the same foundation: a project structure that separates concerns, database connections that are ready before any request arrives, and a health endpoint to prove everything is wired up.

In this step, you will scaffold the project, connect to both MongoDB Atlas (for persistent order data) and Upstash Redis (for the job queue), and confirm the server responds before writing any business logic.

In this step, get ready to:
  • Initialize the Node.js project with production dependencies.
  • Create an Upstash Redis database and configure environment variables.
  • Wire up the Express server with database connections and a health check.
Initialize the project and install dependencies
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop and create the project folder by running these commands:
cd ~/Desktop
mkdir order-processing-pipeline
cd order-processing-pipeline
  • Confirm you are inside the project folder by running:
pwd

You should see a path ending in order-processing-pipeline.

  • Initialize a new Node.js project by running:
npm init -y

You should see output confirming a package.json file was created.

  • Install all production dependencies by running this command:
npm install express@5.2.1 mongoose@9.9.0 bullmq@5.81.3 ioredis@5.11.1 dotenv@17.4.2 helmet@8.3.0 cors@2.8.6 jsonwebtoken@9.0.3 bcryptjs@3.0.3

What are these packages?

  • express is the web framework that handles incoming HTTP requests.
  • mongoose connects to MongoDB and defines data models.
  • bullmq is the job queue library that manages background processing.
  • ioredis is the Redis client that BullMQ uses under the hood.
  • dotenv loads environment variables from a .env file.
  • helmet and cors add security headers and cross-origin support.

You should see output listing all installed packages with no errors.

Seeing permission errors?

On macOS, try prefixing the command with sudo. On Windows, close your terminal and reopen it with Run as administrator.

If you see network errors, check your internet connection and try again. Help me fix npm install errors

  • Now update package.json to enable ESM imports and add start scripts. Open package.json in VS Code and replace its entire contents with:
{
  "name": "order-processing-pipeline",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node src/server.js",
    "worker": "node src/worker.js"
  },
  "dependencies": {
    "bcryptjs": "3.0.3",
    "bullmq": "5.81.3",
    "cors": "2.8.6",
    "dotenv": "17.4.2",
    "express": "5.2.1",
    "helmet": "8.3.0",
    "ioredis": "5.11.1",
    "jsonwebtoken": "9.0.3",
    "mongoose": "9.9.0"
  }
}

What does type module do?

Setting "type": "module" tells Node.js to treat .js files as ES Modules. This lets you use import/export syntax instead of the older require().

  • Save package.json.
  • Create the project folder structure by running:
mkdir -p src/config src/queues src/workers src/routes src/models src/services src/middleware
  • Confirm the folders were created by running:
ls src

You should see config, queues, workers, routes, models, services, and middleware listed in the output.

Why this folder structure?

Each folder has a single responsibility. routes/ holds HTTP endpoints. queues/ holds queue definitions. workers/ holds job processors. services/ holds business logic.

This separation means your API server and worker process can import only what they need without pulling in each other's code.

Create an Upstash Redis database

BullMQ stores all job state (pending, active, completed, failed) in Redis. Upstash provides a free serverless Redis instance that works over TLS with no credit card required.

  • Open your browser and go to console.upstash.com.
  • Sign in to your Upstash account.
  • Click the + Create Database button.
  • Enter order-pipeline as the database name.
  • Select a region close to you (any region works for this project).
  • Click Create.

You will be redirected to the database details page.

  • On the database details page, find the connection string that starts with rediss://. Copy it to your clipboard.

Why rediss:// with two s's?

The rediss:// protocol means Redis over TLS (encrypted). Upstash enforces TLS on all connections, which is why the URL uses rediss instead of plain redis.

  • Back in VS Code, create a new file called .env in the root of your order-processing-pipeline folder. Paste the following contents:
PORT=3000
MONGODB_URI=[[MONGODB_URI="your MongoDB Atlas connection string"]]
UPSTASH_REDIS_URL=[[UPSTASH_REDIS_URL="your Upstash Redis connection string starting with rediss://"]]
WEBHOOK_SECRET=dev-secret-change-in-production

What do these variables do?

  • MONGODB_URI points to your Atlas cluster where order documents are stored.
  • UPSTASH_REDIS_URL is the connection string you just copied from Upstash.
  • WEBHOOK_SECRET is used to verify incoming webhook signatures. The development value is fine for now.
  • Replace your MongoDB Atlas connection string with your actual Atlas connection string from your MongoDB Atlas dashboard.
  • Replace your Upstash Redis connection string starting with rediss:// with the rediss:// URL you copied from Upstash.
  • Save .env.
  • Create a .gitignore file in the project root to keep secrets out of version control. Add this content:
.env
node_modules/
  • Save .gitignore.

Can't find your MongoDB Atlas connection string?

In your MongoDB Atlas dashboard, click Connect on your cluster, choose Drivers, and copy the connection string. Replace <password> with your database user's password and set the database name to order-pipeline.

Help me find my MongoDB Atlas connection string

Wire up the server and Redis connection

The server needs a shared Redis connection module that both the API and the worker process can import. You will create that first, then build the Express server on top of it.

  • Create a new file at src/queues/connection.js and add the following code:
import IORedis from 'ioredis';

// Shared Redis connection for both Queue and Worker processes
export const redis = new IORedis(process.env.UPSTASH_REDIS_URL, {
  maxRetriesPerRequest: null,
  enableOfflineQueue: false,
});

Why maxRetriesPerRequest: null?

BullMQ uses blocking Redis commands that wait indefinitely for new jobs. The default ioredis setting throws an error after 20 retries, which would crash your worker. Setting it to null disables that limit so BullMQ can wait as long as needed.

  • Save src/queues/connection.js.

✔️ Awesome, I've got everything!

Great. Make sure you have saved src/queues/connection.js before continuing.

ⓧ I'd like to double check the full code

import IORedis from 'ioredis';

// Shared Redis connection for both Queue and Worker processes
export const redis = new IORedis(process.env.UPSTASH_REDIS_URL, {
  maxRetriesPerRequest: null,
  enableOfflineQueue: false,
});
  • Now create src/server.js. Start by adding the imports, app configuration, and health endpoint:
import 'dotenv/config';
import express from 'express';
import helmet from 'helmet';
import cors from 'cors';
import mongoose from 'mongoose';
import { redis } from './queues/connection.js';

const app = express();
const PORT = process.env.PORT || 3000;

// Security and parsing middleware
app.use(helmet());
app.use(cors());
app.use(express.json());

// Health check endpoint for monitoring
app.get('/health', (req, res) => {
  res.json({ status: 'ok', timestamp: new Date().toISOString() });
});

What does this code do?

  • import 'dotenv/config' loads your .env variables into process.env before anything else runs.
  • Helmet sets secure HTTP headers automatically. CORS allows cross-origin requests.
  • The /health endpoint returns a JSON response so you can verify the server is alive.
  • Below the health endpoint in the same file, add the startup function and graceful shutdown handlers:
// Connect to databases and start listening
async function start() {
  await mongoose.connect(process.env.MONGODB_URI);
  console.log('MongoDB connected');

  await redis.ping();
  console.log('Redis connected');

  app.listen(PORT, () => {
    console.log(`Server running on port ${PORT}`);
  });
}

// Graceful shutdown on termination signals
process.on('SIGTERM', async () => {
  console.log('SIGTERM received. Shutting down gracefully...');
  await mongoose.disconnect();
  await redis.quit();
  process.exit(0);
});

process.on('SIGINT', async () => {
  console.log('SIGINT received. Shutting down gracefully...');
  await mongoose.disconnect();
  await redis.quit();
  process.exit(0);
});

start().catch((err) => {
  console.error('Failed to start server:', err);
  process.exit(1);
});

What does this code do?

  • start() connects to MongoDB, pings Redis to confirm the connection, then starts the Express server.
  • The SIGTERM and SIGINT handlers run when you stop the server (Ctrl+C or a deployment signal). They cleanly disconnect from databases before exiting.
  • start().catch(...) ensures any connection failure prints the error and exits with a non-zero code so you know something went wrong.
  • Save src/server.js.

✔️ Awesome, I've got everything!

Great. Confirm you have saved both src/queues/connection.js and src/server.js.

ⓧ I'd like to double check the full code

Your src/server.js should contain both chunks above in order. Scroll up and confirm you have:

  • The imports block (dotenv, express, helmet, cors, mongoose, redis).
  • App setup with helmet(), cors(), and express.json() middleware.
  • The /health GET endpoint.
  • The start() function connecting MongoDB and Redis.
  • SIGTERM and SIGINT shutdown handlers.
  • The start().catch(...) call at the bottom.

Before you run the server, what do you expect to see printed in the terminal? Think about what start() does in sequence.

  • Start the server by running:
node src/server.js

You should see three lines printed in order:

MongoDB connected
Redis connected
Server running on port 3000
  • Open your browser and go to http://localhost:3000/health.

You should see a JSON response containing {"status":"ok"} along with a timestamp.

Server not starting?

  • If you see a MongoDB connection error, double-check your MONGODB_URI in .env. Make sure you replaced <password> with your actual password and the database name is set to order-pipeline.
  • If you see a Redis connection error, confirm your UPSTASH_REDIS_URL starts with rediss:// (two s's) and was copied correctly from the Upstash dashboard.
  • If you see Cannot find module, make sure package.json has "type": "module" and your file paths match exactly.

Help me debug my server startup error

Your environment is ready. Both databases are connected and your server responds to requests in milliseconds. Next up, you will build the order API that accepts webhook payloads and stores them in MongoDB.

Build the Order API

In Step 1, you set up the project skeleton with MongoDB and Redis connections. Now it is time to build the actual endpoint that receives order data.

Before adding a job queue, you need a working order endpoint that accepts webhook payloads, verifies their HMAC signature, and stores them in MongoDB. This gives you the synchronous baseline where the response only returns after everything is saved.

In this step, get ready to:
  • Create a Mongoose Order model with validation and unique indexing.
  • Build a POST route with HMAC-SHA256 webhook signature verification.
  • Test the endpoint with Postman to confirm orders save to MongoDB.
Create the Order model

Every order that hits your API needs a consistent shape in the database. A Mongoose schema enforces which fields are required and what values they can hold.

  • Create the file src/models/Order.js by running:
touch src/models/Order.js
  • Open src/models/Order.js in VS Code and paste the following schema:
import mongoose from 'mongoose';

const orderSchema = new mongoose.Schema(
  {
    orderId: { type: String, required: true, unique: true, index: true },
    customerEmail: { type: String, required: true },
    items: [
      {
        name: { type: String, required: true },
        quantity: { type: Number, required: true },
        price: { type: Number, required: true },
      },
    ],
    total: { type: Number, required: true },
    status: {
      type: String,
      enum: ['pending', 'processing', 'completed', 'failed'],
      default: 'pending',
    },
    processedAt: { type: Date },
    attempts: { type: Number, default: 0 },
  },
  { timestamps: true }
);

export default mongoose.model('Order', orderSchema);

What does this schema do?

  • orderId has unique: true and index: true. The unique constraint prevents duplicate orders at the database level.
  • status uses an enum restricted to four values. An order always starts as pending and moves through a lifecycle.
  • timestamps: true automatically adds createdAt and updatedAt fields for audit trails.
  • Save src/models/Order.js.
  • Confirm the file exists by running:
ls src/models/

You should see Order.js listed in the output.

Create the orders route with webhook verification

Real webhook endpoints from services like Shopify sign every payload with an HMAC-SHA256 hash. Your route verifies this signature before trusting any data.

  • Create the file src/routes/orders.js by running:
touch src/routes/orders.js
  • Open src/routes/orders.js and paste the imports and verification function:
import { Router } from 'express';
import crypto from 'crypto';
import Order from '../models/Order.js';

export const orderRouter = Router();

// Verify webhook payload hasn't been tampered with
function verifyWebhookSignature(payload, signature, secret) {
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(JSON.stringify(payload))
    .digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expectedSignature)
  );
}

What does this verification do?

The function takes the request body, hashes it with the shared secret using HMAC-SHA256, and compares the result to the signature in the header.

crypto.timingSafeEqual performs a constant-time comparison. A regular === check leaks timing information that attackers can exploit to guess valid signatures character by character.

  • Below the verifyWebhookSignature function, paste the POST route handler:
orderRouter.post('/', async (req, res) => {
  // Extract signature from webhook header
  const signature = req.headers['x-webhook-signature'];

  if (!signature) {
    return res.status(401).json({ error: 'Missing webhook signature' });
  }

  // Verify the payload hasn't been tampered with
  const isValid = verifyWebhookSignature(req.body, signature, process.env.WEBHOOK_SECRET);
  if (!isValid) {
    return res.status(401).json({ error: 'Invalid webhook signature' });
  }

  // Validate required fields
  const { orderId, customerEmail, items, total } = req.body;

  if (!orderId || !customerEmail || !items || !total) {
    return res.status(400).json({
      error: 'Missing required fields',
      required: ['orderId', 'customerEmail', 'items', 'total'],
    });
  }

  // Save the order with pending status
  const order = await Order.create({
    orderId,
    customerEmail,
    items,
    total,
    status: 'pending',
  });

  res.status(201).json({ orderId: order.orderId });
});

What does this handler do?

  • It rejects requests missing the x-webhook-signature header with a 401.
  • It verifies the signature matches the payload. Invalid signatures also get a 401.
  • It validates required fields exist, then saves the order to MongoDB with status pending.
  • It returns 201 (created) with the orderId on success.
  • Save src/routes/orders.js.

Now you need to mount this route in your server so Express knows about it.

  • In src/server.js, find this line near the top of your imports:
import { redis } from './queues/connection.js';
  • Add this import directly above it:
import { orderRouter } from './routes/orders.js';
  • Next, find this line:
app.use(express.json());
  • Add this line directly below it:
app.use('/api/orders', orderRouter);
  • Save src/server.js.

Before you run the server, what do you expect to see if both MongoDB and Redis connections succeed?

  • Start the server by running:
node src/server.js

You should see the same three lines from Step 1: MongoDB connected, Redis connected, and Server running on port 3000. The server now has a /api/orders endpoint ready for requests.

Seeing an import error?

  • Check that the import path is exactly './routes/orders.js' with the .js extension. ESM requires full file extensions.
  • Make sure you exported orderRouter (not a default export) in src/routes/orders.js.
  • Confirm the src/models/Order.js file exists. The route imports it.

Why is my server failing to start?

Test the endpoint with Postman

Your route expects a valid HMAC signature in the x-webhook-signature header. You need to generate this signature from the request body using the same secret your server uses.

  • Open a second terminal (keep your server running in the first one).
  • Generate a valid signature for a test order by running this command:
node -e "const crypto = require('crypto'); const body = JSON.stringify({orderId:'ORD-001',customerEmail:'customer@example.com',items:[{name:'Widget',quantity:2,price:29.99}],total:59.98}); const sig = crypto.createHmac('sha256','dev-secret-change-in-production').update(body).digest('hex'); console.log(sig);"

What does this command do?

This one-liner hashes the same JSON payload you will send, using the same secret from your .env file. The output is the hex signature your server expects in the x-webhook-signature header.

  • Copy the hex string printed by the command. You will use it as a header value.
  • Open Postman.
  • Create a new request. Set the method to POST and the URL to http://localhost:3000/api/orders.
  • Click the Headers tab.
  • Add a header with key x-webhook-signature and paste the hex string you copied as the value.
  • Click the Body tab.
  • Select raw and change the dropdown to JSON.
  • Paste this exact JSON body:
{"orderId":"ORD-001","customerEmail":"customer@example.com","items":[{"name":"Widget","quantity":2,"price":29.99}],"total":59.98}

Matching the signature exactly

The JSON body you paste into Postman must be character-for-character identical to the body you hashed. If you add extra spaces or newlines, the signature will not match and you will get a 401.

Before you hit Send, what status code do you expect back? Remember that the signature matches and all required fields are present.

  • Click Send.

You should see a 201 Created response with {"orderId":"ORD-001"} in the body.

  • Now test the rejection path. Remove the x-webhook-signature header (uncheck it or delete the value).
  • Click Send again.

You should see a 401 Unauthorized response with {"error":"Missing webhook signature"}.

  • Open the MongoDB Atlas dashboard in your browser.
  • Click Browse Collections on your cluster.
  • Look in the order-pipeline database for the orders collection. You should see one document with orderId: "ORD-001" and status: "pending".

Getting a 401 even with the signature?

  • Make sure the JSON body in Postman has no extra whitespace, trailing commas, or formatting differences from the string you hashed.
  • Verify your WEBHOOK_SECRET in .env is exactly dev-secret-change-in-production (the same value you used in the node -e command).
  • Re-run the signature generation command with the exact body you are sending.

Why is my webhook signature verification failing?

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/models/Order.js, src/routes/orders.js, and src/server.js.

ⓧ I'd like to double check the full code

Here is src/models/Order.js:

import mongoose from 'mongoose';

const orderSchema = new mongoose.Schema(
  {
    orderId: { type: String, required: true, unique: true, index: true },
    customerEmail: { type: String, required: true },
    items: [
      {
        name: { type: String, required: true },
        quantity: { type: Number, required: true },
        price: { type: Number, required: true },
      },
    ],
    total: { type: Number, required: true },
    status: {
      type: String,
      enum: ['pending', 'processing', 'completed', 'failed'],
      default: 'pending',
    },
    processedAt: { type: Date },
    attempts: { type: Number, default: 0 },
  },
  { timestamps: true }
);

export default mongoose.model('Order', orderSchema);

Your src/routes/orders.js should contain both chunks you pasted above in order:

  • The imports (Router, crypto, Order), the orderRouter declaration, and the verifyWebhookSignature function.
  • The orderRouter.post('/', ...) handler that checks the signature, validates fields, creates the order, and returns 201.

Scroll up and confirm both chunks are present in your file with no gaps between them.

In src/server.js, confirm you added these two lines:

// Added after the mongoose import
import { orderRouter } from './routes/orders.js';

// Added after app.use(express.json())
app.use('/api/orders', orderRouter);

The signal handlers (SIGTERM and SIGINT) and start().catch(...) at the bottom remain unchanged from Step 1.

Your order API accepts webhooks, verifies their signatures, and persists orders. Right now the response is instant because you are not doing any processing yet. In the next step, you will add synchronous processing logic that blocks the response and creates the exact problem that job queues solve.

Add Synchronous Processing

In the previous step, you built a working webhook endpoint that validates signatures and saves orders to MongoDB. The response came back almost instantly because the endpoint did nothing except save a document.

Now you will add real processing logic: inventory checks, payment handling, and order confirmation. You will wire it directly into the route handler so the API response waits until all processing finishes. This deliberately creates the problem that background job queues exist to solve.

In this step, get ready to:
  • Build a multi-stage order processing service that simulates real-world dependencies.
  • Wire synchronous processing into the route handler so the response blocks until completion.
  • Experience the exact failure modes that make this architecture unsuitable for production.
Create the order processor service

Real order pipelines call external services: inventory databases, payment gateways, email providers. Each call takes time and can fail independently. You will simulate this with three processing stages that each take 1 second.

  • Create a new file src/services/orderProcessor.js and add the delay helper plus the first two processing stages:
function delay(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

async function checkInventory(orderData) {
  console.log(`  [${orderData.orderId}] Stage 1/3: Checking inventory...`);
  await delay(1000);
  console.log(`  [${orderData.orderId}] Stage 1/3: Inventory confirmed`);
  return { inventoryChecked: true };
}

async function processPayment(orderData) {
  console.log(`  [${orderData.orderId}] Stage 2/3: Processing payment ($${orderData.total})...`);
  await delay(1000);

  if (orderData.customerEmail.includes('fail')) {
    throw new Error('Payment declined: card_declined');
  }

  console.log(`  [${orderData.orderId}] Stage 2/3: Payment successful`);
  return { paymentProcessed: true };
}

What does this code do?

  • delay(ms) is a helper that pauses execution for a given number of milliseconds, simulating a network call to an external service.
  • checkInventory simulates a 1-second call to an inventory database to confirm items are in stock.
  • processPayment simulates a 1-second payment gateway call. If the customer email contains "fail", it throws an error to simulate a declined card.
  • Below the processPayment function in the same file, add the final stage and the orchestrator function:
async function sendConfirmation(orderData) {
  console.log(`  [${orderData.orderId}] Stage 3/3: Sending confirmation to ${orderData.customerEmail}...`);
  await delay(1000);
  console.log(`  [${orderData.orderId}] Stage 3/3: Confirmation sent`);
  return { confirmationSent: true };
}

export async function processOrder(orderData) {
  const results = {};
  results.inventory = await checkInventory(orderData);
  results.payment = await processPayment(orderData);
  results.confirmation = await sendConfirmation(orderData);
  return { success: true, orderId: orderData.orderId, stages: results };
}

What does this code do?

  • sendConfirmation simulates a 1-second email service call to notify the customer.
  • processOrder is the exported orchestrator that runs all three stages in sequence. Total: 3 seconds minimum per order.
  • Save src/services/orderProcessor.js.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before continuing.

ⓧ I'd like to double check the full code

Your src/services/orderProcessor.js should contain both chunks in order: the delay helper, checkInventory, processPayment, sendConfirmation, and the exported processOrder function. Scroll up and confirm each one is present, in order.

Wire processing into the route handler

Right now, the POST route saves the order and immediately returns. You will change it to call processOrder before responding, so the HTTP response is blocked until all three stages finish.

  • In src/routes/orders.js, find the import lines at the top of the file. Find this line:
import Order from '../models/Order.js';
  • Add a new import directly below it so the imports look like this:
import Order from '../models/Order.js';
import { processOrder } from '../services/orderProcessor.js';
  • Now scroll to the end of the POST handler. Find this code that currently creates the order and immediately responds:
  const order = await Order.create({
    orderId,
    customerEmail,
    items,
    total,
    status: 'pending',
  });

  res.status(201).json({ orderId: order.orderId });
});
  • Replace that entire block with the following code that processes the order synchronously before responding:
  const order = await Order.create({
    orderId,
    customerEmail,
    items,
    total,
    status: 'pending',
  });

  await Order.updateOne({ orderId }, { status: 'processing' });

  try {
    await processOrder(req.body);
    await Order.updateOne({ orderId }, { status: 'completed', processedAt: new Date() });
    res.status(200).json({ success: true, orderId: order.orderId });
  } catch (err) {
    await Order.updateOne({ orderId }, { status: 'failed' });
    res.status(500).json({ error: err.message });
  }
});

What does this code do?

  • After creating the order, it immediately updates status to "processing" so you can see the transition in MongoDB.
  • It then calls processOrder and waits for all three stages to complete before responding.
  • On success, it updates the order to "completed" with a timestamp. On failure, it updates to "failed" and returns a 500 error.
  • The HTTP response is held open for the entire 3+ seconds of processing. The caller cannot do anything else while waiting.
  • Save src/routes/orders.js.

Seeing a syntax error on save?

  • Make sure you replaced the entire block including the closing }); of the route handler. If you left the old res.status(201) line in place, you will have duplicate responses.
  • Check that the import { processOrder } line uses curly braces. It is a named export.
  • help me debug a syntax error in my orders route
Hit the wall with synchronous processing

Your API now processes orders inline before responding. Time to see why this is a problem in production.

  • Restart your server by pressing Ctrl+C in the terminal, then running:
node src/server.js

Before you fire the next request, think about this: your processing pipeline has three stages that each take 1 second. How long do you expect the API response to take now?

  • In Postman, send the same POST request to http://localhost:3000/api/orders with this body (use a new orderId since IDs are unique):
{
  "orderId": "ORD-100",
  "customerEmail": "customer@example.com",
  "items": [{"name": "Widget", "quantity": 2, "price": 29.99}],
  "total": 59.98
}

Watch the response time in Postman. It takes 3+ seconds to come back. In your terminal, you can see each processing stage logging one by one.

Why is this a problem?

Webhook senders like Shopify expect a 200 response within 5 seconds. If your endpoint takes 3+ seconds just for one order, adding any real database or payment gateway latency pushes you past the timeout.

The sender marks your endpoint as unhealthy and stops delivering events. Meanwhile, every incoming request holds an open connection and blocks your event loop.

Now test what happens when processing fails. What do you think happens to the order if the payment stage throws an error?

  • Send another POST request with an email that triggers the failure:
{
  "orderId": "ORD-101",
  "customerEmail": "fail@test.com",
  "items": [{"name": "Gadget", "quantity": 1, "price": 49.99}],
  "total": 49.99
}

You get a 500 response with the message Payment declined: card_declined. In your terminal, Stage 1 completes but Stage 2 throws.

  • Open your MongoDB Atlas Data Explorer and check the ORD-101 document.

The order sits in "failed" status permanently. There is no retry mechanism. No recovery path. No one gets notified.

The two problems you just proved

  • Slow responses: a 3-second minimum response time means webhook senders time out and stop delivering events. Concurrent requests serialize because each one holds the event loop.
  • No retry on failure: if payment declines or any stage throws, the order is permanently dead. In production, transient failures (network blips, gateway timeouts) are common and recoverable. This architecture treats every failure as permanent.

You have now felt the exact pain that every production backend team hits when they process work inline. Next up, you will fix both problems by decoupling the API from the processing with BullMQ: the API will respond instantly, and a separate worker process will handle retries automatically.

Decouple with BullMQ

In the last step, you felt both problems firsthand: the API blocked for 3+ seconds on every order, and failed orders got permanently stuck with no recovery path.

Now you fix both by separating "accepting the webhook" from "processing the order." The API will validate, persist, enqueue, and return 200 in under 50ms. A BullMQ worker running in a completely separate process handles the heavy lifting asynchronously with automatic exponential backoff retries.

In this step, get ready to:
  • Create the BullMQ queue module with production retry settings.
  • Build a separate worker process to handle order processing asynchronously.
  • Refactor the API route to the "fast path" pattern that returns instantly.
Create the queue module

The queue module is the bridge between your API and worker. It defines the job queue and its retry configuration.

  • Create the file src/queues/orderQueue.js by pasting the following code:
import { Queue } from 'bullmq';
import { redis } from './connection.js';

export const orderQueue = new Queue('order-processing', {
  connection: redis,
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 2000,
    },
    removeOnComplete: { count: 100 },
    removeOnFail: { count: 500 },
  },
});

What does this code do?

This creates a BullMQ Queue named order-processing connected to your Upstash Redis instance.

  • attempts: 3 means each job gets 3 tries before being marked as permanently failed.
  • backoff: { type: 'exponential', delay: 2000 } produces delays of 2s, 4s, 8s between retries. This avoids thundering herds when a downstream service recovers.
  • removeOnComplete: { count: 100 } and removeOnFail: { count: 500 } keep recent jobs for observability without unbounded storage growth.
  • Save src/queues/orderQueue.js.
  • Confirm the file exists by running:
ls src/queues/

You should see both connection.js and orderQueue.js listed.

Build the worker processor

The worker is a completely separate Node.js process that connects to the same Redis queue and picks up jobs. It never touches the Express server.

Why a separate process?

This is the production pattern: your API scales independently from your workers. A crash in processing never takes down your ability to accept new webhooks. Shopify, Stripe, and every serious SaaS company uses this architecture.

  • Create the file src/workers/orderWorker.js and paste the first part of the worker processor:
import { Worker } from 'bullmq';
import { redis } from '../queues/connection.js';
import { processOrder } from '../services/orderProcessor.js';
import Order from '../models/Order.js';

export const orderWorker = new Worker(
  'order-processing',
  async (job) => {
    const { orderId } = job.data;
    const startTime = Date.now();
    console.log(`[${orderId}] Processing started (attempt ${job.attemptsMade + 1}/${job.opts.attempts})`);

    const order = await Order.findOne({ orderId });
    if (!order) {
      throw new Error(`Order ${orderId} not found in database`);
    }

    if (order.status === 'completed') {
      console.log(`[${orderId}] Skipping duplicate - already completed at ${order.processedAt}`);
      return { skipped: true, orderId, reason: 'already_completed' };
    }

    await Order.updateOne({ orderId }, { status: 'processing' });

    const result = await processOrder(job.data);

What does this code do?

The processor function runs for every job BullMQ dequeues. It starts by logging the attempt number, then guards against common issues:

  • If the order doesn't exist in MongoDB, it throws immediately (BullMQ will retry).
  • If the order is already completed, it returns early to prevent duplicate processing.
  • Only after passing both guards does it update the status to "processing" and call your existing processOrder pipeline.
  • Directly below the code you just pasted, add the completion logic and Worker options:
    await Order.updateOne(
      { orderId },
      {
        status: 'completed',
        processedAt: new Date(),
        attempts: job.attemptsMade + 1,
      }
    );

    const duration = Date.now() - startTime;
    console.log(`[${orderId}] Completed successfully in ${duration}ms`);
    return { ...result, duration };
  },
  {
    connection: redis,
    concurrency: 3,
  }
);

What does this code do?

After processOrder completes successfully, this updates the order to "completed" in MongoDB with a timestamp and attempt count. The concurrency: 3 setting lets the worker process 3 jobs in parallel since each job is I/O-bound (waiting on simulated API calls), not CPU-bound.

  • Save src/workers/orderWorker.js.
  • Confirm the file was created by running:
ls src/workers/

You should see orderWorker.js listed.

  • Now add the event listeners below the closing ); of the Worker constructor in the same file:
orderWorker.on('completed', (job, result) => {
  if (result.skipped) {
    console.log(`[${job.data.orderId}] Job skipped: ${result.reason}`);
  }
});

orderWorker.on('failed', (job, err) => {
  const isFinalAttempt = job.attemptsMade >= job.opts.attempts;
  const level = isFinalAttempt ? 'ERROR' : 'WARN';
  console.log(
    `[${level}] [${job.data.orderId}] Attempt ${job.attemptsMade}/${job.opts.attempts} failed: ${err.message}`
  );
  if (isFinalAttempt) {
    console.log(`[ERROR] [${job.data.orderId}] Permanently failed. Dead-letter. Manual intervention required.`);
    Order.updateOne({ orderId: job.data.orderId }, { status: 'failed' }).catch(console.error);
  }
});

orderWorker.on('stalled', (jobId) => {
  console.log(`[WARN] Job ${jobId} stalled - exceeded lock timeout. Will be retried by another worker.`);
});

What do these event listeners do?

  • completed logs when a job was skipped (duplicate detection).
  • failed logs each failed attempt with context. On the final attempt, it marks the order as "failed" in MongoDB and logs a dead-letter warning for manual intervention.
  • stalled fires when a job exceeded its lock timeout (e.g., the worker crashed mid-processing). BullMQ will automatically retry it on another worker.
  • Save src/workers/orderWorker.js.
  • Now create the worker entry point at src/worker.js with the startup logic:
import 'dotenv/config';
import mongoose from 'mongoose';
import { orderWorker } from './workers/orderWorker.js';
import { redis } from './queues/connection.js';

async function start() {
  await mongoose.connect(process.env.MONGODB_URI);
  console.log('Worker: MongoDB connected');

  await redis.ping();
  console.log('Worker: Redis connected');
  console.log('Worker: Listening for jobs...');
}

What does this code do?

This is the entry point for your worker process. It loads environment variables, connects to MongoDB and Redis, and the import of orderWorker automatically starts the BullMQ Worker listening for jobs on the order-processing queue.

  • Below the start function, add the signal handlers and the startup call:
process.on('SIGTERM', async () => {
  console.log('Worker: SIGTERM received. Waiting for active jobs...');
  await orderWorker.close();
  console.log('Worker: Active jobs drained.');
  await mongoose.disconnect();
  await redis.quit();
  console.log('Worker: Shutdown complete.');
  process.exit(0);
});

process.on('SIGINT', async () => {
  console.log('Worker: SIGINT received. Waiting for active jobs...');
  await orderWorker.close();
  console.log('Worker: Active jobs drained.');
  await mongoose.disconnect();
  await redis.quit();
  console.log('Worker: Shutdown complete.');
  process.exit(0);
});

start().catch((err) => {
  console.error('Worker failed to start:', err);
  process.exit(1);
});

What does this code do?

The signal handlers implement graceful shutdown. When you press Ctrl+C or the orchestrator sends SIGTERM, orderWorker.close() tells BullMQ to stop picking new jobs and wait for active jobs to finish their current processing cycle. Only after in-flight jobs complete does it disconnect from MongoDB and Redis.

  • Save src/worker.js.
  • Test the worker process by running:
node src/worker.js

You should see three lines in sequence: Worker: MongoDB connected, Worker: Redis connected, and Worker: Listening for jobs....

Worker won't start?

  • If you see a Redis connection error, check that your UPSTASH_REDIS_URL in .env starts with rediss:// (double s for TLS).
  • If you see a MongoDB error, confirm your MONGODB_URI includes the database name at the end of the connection string.
  • If you see a module import error, verify the file paths match exactly: ./workers/orderWorker.js and ./queues/connection.js.

Why won't my worker process connect?

  • Press Ctrl+C to stop the worker for now. You will restart it alongside the server in the next substep.

✔️ Awesome, I've got everything!

Great. Double check you saved both src/workers/orderWorker.js and src/worker.js.

ⓧ I'd like to double check the full code

Both files exceed 30 lines, so verify each section is present in order:

src/workers/orderWorker.js should contain, in order:

  • Four import lines (Worker, redis, processOrder, Order).
  • The export const orderWorker = new Worker(...) block with the processor function (guards, processing, completion update, concurrency option).
  • Three event listeners: completed, failed, and stalled.

src/worker.js should contain, in order:

  • Four imports (dotenv/config, mongoose, orderWorker, redis).
  • The start() function that connects MongoDB and pings Redis.
  • SIGTERM and SIGINT handlers that call orderWorker.close() then disconnect and quit.
  • The start().catch(...) call at the bottom.
Refactor the route to the fast path

Now you transform the POST handler from blocking to non-blocking. The API's only job becomes: validate, persist, enqueue, respond.

  • In src/routes/orders.js, find this import line at the top:
import { processOrder } from '../services/orderProcessor.js';
  • Replace it with the queue import:
import { orderQueue } from '../queues/orderQueue.js';
  • Now find this section inside the POST handler, after the field validation:
  const order = await Order.create({
    orderId,
    customerEmail,
    items,
    total,
    status: 'pending',
  });

  await Order.updateOne({ orderId }, { status: 'processing' });

  try {
    await processOrder(req.body);
    await Order.updateOne({ orderId }, { status: 'completed', processedAt: new Date() });
    res.status(200).json({ success: true, orderId: order.orderId });
  } catch (err) {
    await Order.updateOne({ orderId }, { status: 'failed' });
    res.status(500).json({ error: err.message });
  }
  • Replace that entire block with this fast-path code:
  const existingOrder = await Order.findOne({ orderId });
  if (existingOrder) {
    return res.status(200).json({ received: true, message: 'Order already received', orderId });
  }

  const order = await Order.create({
    orderId,
    customerEmail,
    items,
    total,
    status: 'pending',
  });

  await orderQueue.add(orderId, req.body, { jobId: orderId });

  res.status(200).json({ received: true, orderId: order.orderId });

What changed here?

  • The idempotency check (Order.findOne) prevents re-processing if the same webhook arrives twice. It returns 200 so the sender stops retrying.
  • orderQueue.add(orderId, req.body, { jobId: orderId }) enqueues the job. The jobId: orderId ensures BullMQ also deduplicates at the Redis level.
  • The response returns immediately with { received: true }. No processOrder call, no try/catch, no 3-second wait.
  • Finally, add a queue status endpoint below the POST handler (still inside the same file):
orderRouter.get('/queue-status', async (req, res) => {
  const counts = await orderQueue.getJobCounts(
    'waiting',
    'active',
    'completed',
    'failed',
    'delayed'
  );
  res.json({ queue: 'order-processing', ...counts });
});

What does this endpoint do?

This gives you production observability. Hit /api/orders/queue-status to see how many jobs are waiting, active, completed, failed, or delayed at any moment.

  • Save src/routes/orders.js.

✔️ Awesome, I've got everything!

Great. Make sure the file is saved before moving on to the test.

ⓧ I'd like to double check the full code

Your refactored src/routes/orders.js should contain, in order:

  • Imports: Router, crypto, Order, and orderQueue (no processOrder import).
  • The verifyWebhookSignature function (unchanged).
  • The POST handler with: signature check, field validation, existingOrder check, Order.create, orderQueue.add, then res.status(200).
  • The GET /queue-status endpoint at the bottom.
Test the decoupled architecture

Now for the moment of truth. You will run two separate processes and watch the webhook get accepted instantly while processing happens in the background.

  • Open two terminal windows side by side.
  • In Terminal 1, start the API server by running:
node src/server.js
  • In Terminal 2, start the worker by running:
node src/worker.js

Terminal 1 shows Server running on port 3000. Terminal 2 shows Worker: Listening for jobs.... Two processes, connected only through Redis.

Before you send the first order, predict: how fast do you think the API response will be now? Remember, last step it took 3+ seconds.

  • In Postman, send the same POST request to http://localhost:3000/api/orders with a new orderId (e.g. ORD-101) and the same HMAC signature generation from the pre-request script.

The response comes back in under 50ms with { "received": true, "orderId": "ORD-101" }. Check Postman's response time in the bottom-right corner.

  • Now look at Terminal 2 (the worker). About 3 seconds after your POST, you see the processing stages complete:
[ORD-101] Processing started (attempt 1/3)
  [ORD-101] Stage 1/3: Checking inventory...
  [ORD-101] Stage 1/3: Inventory confirmed
  [ORD-101] Stage 2/3: Processing payment ($59.98)...
  [ORD-101] Stage 2/3: Payment successful
  [ORD-101] Stage 3/3: Sending confirmation to customer@example.com...
  [ORD-101] Stage 3/3: Confirmation sent
[ORD-101] Completed successfully in 3005ms

The API returned instantly. The heavy work happened asynchronously in the worker.

  • Now test the retry logic. Send a POST with customerEmail set to fail@test.com and a new orderId (e.g. ORD-102).

The API still returns 200 instantly. But watch Terminal 2 over the next 15 seconds:

  • Attempt 1 fails immediately with "Payment declined: card_declined".
  • After a 2-second delay, attempt 2 fails.
  • After a 4-second delay, attempt 3 fails.
  • Then you see "Permanently failed. Dead-letter. Manual intervention required."

What just happened?

BullMQ automatically retried the job 3 times with exponential backoff (2s, 4s delays between retries). After all attempts were exhausted, the failed event listener marked the order as permanently failed in MongoDB. In production, this is where you would trigger an alert for your on-call engineer.

Not seeing retries in Terminal 2?

  • Make sure you used a fresh orderId that does not already exist in your database. The idempotency check returns immediately for duplicates.
  • Verify the customerEmail field contains the word "fail" (e.g. fail@test.com). This triggers the payment failure in processPayment.
  • If you see no output at all in Terminal 2, confirm the worker is still running and shows Worker: Listening for jobs....

Why isn't my worker picking up jobs?

Your API now does zero processing. It validates, persists, enqueues, and responds in milliseconds. The worker handles the heavy work asynchronously with automatic retries. Next up, you will add idempotency and graceful shutdown to make this truly production-ready.

Add Idempotency and Graceful Shutdown

Production systems receive duplicate webhooks all the time. Shopify explicitly retries up to 8 times over 4 hours if your endpoint doesn't return a 200 fast enough.

Without idempotency, a retried "order placed" webhook would charge a customer twice. Without graceful shutdown, deploying a new version of your worker kills in-flight jobs mid-processing, leaving orders stuck in "processing" with no automatic recovery until the stall timeout fires.

In this step, get ready to:
  • Add idempotent processing to the worker so duplicate jobs are safely skipped.
  • Add idempotent webhook handling to the API route so duplicate deliveries never create duplicate orders.
  • Implement graceful shutdown so in-flight jobs finish before the process exits.
Add idempotency to the worker processor

A BullMQ job can be retried after the worker already completed it. This happens during Redis reconnection race conditions or when a job is marked stalled and re-queued while the original worker was still finishing.

  • Open src/workers/orderWorker.js in VS Code.
  • Find the processor function (the async (job) => callback inside the Worker constructor).
  • After fetching the order from MongoDB and checking it exists, add this idempotency guard:
    if (order.status === 'completed') {
      console.log(`[${orderId}] Skipping duplicate - already completed at ${order.processedAt}`);
      return { skipped: true, orderId, reason: 'already_completed' };
    }

What does this code do?

Before running the expensive 3-stage processing pipeline, the worker checks if this order has already been completed. If it has, the worker returns immediately with a skipped result instead of re-processing.

This makes the processor a pure idempotent function. Calling it N times with the same input always produces the same result with no additional side effects (no double charges, no duplicate emails).

  • Also confirm the completed event listener logs skipped jobs:
orderWorker.on('completed', (job, result) => {
  if (result.skipped) {
    console.log(`[${job.data.orderId}] Job skipped: ${result.reason}`);
  }
});

Why log skipped jobs?

In production, you want visibility into how often duplicate processing attempts happen. A spike in "skipped" logs signals a connectivity issue between your worker and Redis that's causing excessive retries.

  • Save src/workers/orderWorker.js.

Not sure where to place the check?

The idempotency guard goes immediately after const order = await Order.findOne({ orderId }) and its not-found check. It must come BEFORE await Order.updateOne({ orderId }, { status: 'processing' }).

Help me find where to place the idempotency check

Add idempotency to the API route

The worker-level check handles retried jobs. But you also need to prevent duplicate orders from being created and enqueued in the first place when a webhook sender retries the HTTP request.

  • Open src/routes/orders.js.
  • Inside the POST handler, after validating required fields and before calling Order.create, add a duplicate check:
  const existingOrder = await Order.findOne({ orderId });
  if (existingOrder) {
    return res.status(200).json({ received: true, message: 'Order already received', orderId });
  }

Why return 200 for a duplicate?

The webhook sender (Shopify, Stripe, etc.) only cares that you returned 200. Returning 200 tells it to stop retrying. Returning 409 or 400 would cause MORE retries, making the problem worse.

Combined with BullMQ's jobId deduplication (which prevents the same jobId from being enqueued twice at the Redis level), you now have two layers of idempotency protection: application-level (MongoDB unique constraint plus explicit check) and queue-level (BullMQ jobId).

  • Save src/routes/orders.js.
  • Restart the API server in your first terminal by pressing Ctrl+C then running:
node src/server.js
  • Restart the worker in your second terminal by pressing Ctrl+C then running:
node src/worker.js

Before you test, what do you think will happen if you send the exact same order payload twice with the same orderId?

  • In Postman, send your test order (with orderId ORD-001 or a new unique ID). Confirm you get a 200 response and the worker processes it.
  • Send the exact same request again with the same orderId.

The second request returns 200 with "message": "Order already received". The worker never picks up a second job because the route caught the duplicate before it reached the queue.

Getting a duplicate key error instead?

If you see a MongoDB duplicate key error, your Order.findOne({ orderId }) check might be placed after the Order.create call instead of before it. The check must come before the create.

Help me debug my idempotency check

Add graceful shutdown to both processes

When you deploy a new version of your worker, the orchestrator (Docker, Kubernetes, PM2) sends SIGTERM to the old process. Without graceful shutdown, in-flight jobs get killed mid-processing and are marked stalled.

  • Open src/worker.js and confirm it has signal handlers for both SIGTERM and SIGINT:
process.on('SIGTERM', async () => {
  console.log('Worker: SIGTERM received. Waiting for active jobs...');
  await orderWorker.close();
  console.log('Worker: Active jobs drained.');
  await mongoose.disconnect();
  await redis.quit();
  console.log('Worker: Shutdown complete.');
  process.exit(0);
});

process.on('SIGINT', async () => {
  console.log('Worker: SIGINT received. Waiting for active jobs...');
  await orderWorker.close();
  console.log('Worker: Active jobs drained.');
  await mongoose.disconnect();
  await redis.quit();
  console.log('Worker: Shutdown complete.');
  process.exit(0);
});

What does orderWorker.close() do?

Calling close() on a BullMQ Worker stops it from picking up new jobs and waits for all currently active jobs to finish their processing cycle. Only after in-flight jobs complete does the promise resolve.

The shutdown sequence is deliberate: drain the worker first, then disconnect MongoDB, then quit Redis. If you closed Redis first, the worker couldn't acknowledge completed jobs.

  • Open src/server.js and confirm it also has matching shutdown handlers:
process.on('SIGTERM', async () => {
  console.log('SIGTERM received. Shutting down gracefully...');
  await mongoose.disconnect();
  await redis.quit();
  process.exit(0);
});

process.on('SIGINT', async () => {
  console.log('SIGINT received. Shutting down gracefully...');
  await mongoose.disconnect();
  await redis.quit();
  process.exit(0);
});

Why doesn't the server call orderWorker.close()?

The server process only accepts HTTP requests. It doesn't run a worker, so there are no in-flight jobs to drain. It just needs to close its database and Redis connections cleanly.

  • Save both files.

Now test graceful shutdown. Before you press Ctrl+C, predict: will the in-flight job complete or get killed?

  • In Postman, send a new order (use a fresh orderId like ORD-SHUTDOWN-TEST).
  • Immediately switch to the worker terminal and press Ctrl+C while the processing stages are still running.

You should see the worker log each phase in sequence: the SIGINT message, then the remaining processing stages completing, then "Active jobs drained", and finally "Shutdown complete" before the process exits cleanly.

Worker exits immediately without draining?

If you press Ctrl+C twice, the second signal force-kills the process. Press it only once and wait for the logs to appear.

If the worker exits instantly with no "Waiting for active jobs" log, the signal handler might not be registered. Make sure process.on('SIGINT', ...) is at the top level of src/worker.js, not inside the start() function.

Help me debug my graceful shutdown

✔️ Awesome, I've got everything!

Great. Double check you've saved src/workers/orderWorker.js, src/routes/orders.js, src/worker.js, and src/server.js.

ⓧ I'd like to double check the full code

Since these files are long, scroll through each one and verify the key sections are present.

In src/workers/orderWorker.js, confirm you have:

  • The order.status === 'completed' check that returns { skipped: true } early, placed after the Order.findOne and not-found check.
  • The completed event listener that logs skipped jobs.
  • The failed event listener with final-attempt detection and dead-letter logging.
  • The stalled event listener with a warning log.

In src/routes/orders.js, confirm you have:

  • The Order.findOne({ orderId }) check placed AFTER field validation and BEFORE Order.create.
  • The early return with res.status(200).json({ received: true, message: 'Order already received', orderId }).

In src/worker.js, confirm you have:

  • SIGTERM and SIGINT handlers at the top level (not inside start()).
  • Each handler calls orderWorker.close() first, then mongoose.disconnect(), then redis.quit(), then process.exit(0).

In src/server.js, confirm you have:

  • SIGTERM and SIGINT handlers that call mongoose.disconnect() then redis.quit() then process.exit(0).

Your pipeline now has belt-and-suspenders protection: duplicate webhooks are caught at the API layer, duplicate job executions are caught at the worker layer, and graceful shutdown means zero-downtime deployments never leave orders stuck. This is how companies like Shopify handle millions of webhooks daily without data loss.

Secret mission

VIP Priority Queue

Real e-commerce platforms don't treat all orders equally. VIP customers expect their orders processed first. Can you add priority levels to your queue so high-value orders jump ahead without starving regular customers?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs on Upstash Redis and MongoDB Atlas free tiers, so there are no ongoing costs.

Resources you used:

  • Upstash Redis database ("order-pipeline") on the free tier (256MB data, 500K commands/month, $0).
  • MongoDB Atlas cluster on the free tier (512MB storage, $0).
  • Local project folder order-processing-pipeline on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively building or want to extend the project.

  • Both Upstash Redis and MongoDB Atlas free tiers have no time limit and cost nothing when idle.
  • Upstash archives free databases after 30 days of inactivity. You get warning emails before this happens, and no data is lost.
  • Your local project folder stays on your Desktop until you move or delete it.

Pause - I'll come back to this later

Shut down running processes to free up terminal sessions. Both free tiers cost $0 when idle, so no further action is needed on the cloud side.

  • Press Ctrl+C in both terminal windows (server and worker) to stop the processes.
  • Your Upstash Redis data and MongoDB Atlas data persist automatically on the free tier at no cost.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to fully clean up.

  • Delete the Upstash Redis database. Go to console.upstash.com, click your order-pipeline database, scroll to the bottom of the database details page, and click Delete Database. Confirm when prompted.
  • Delete the MongoDB Atlas cluster. Go to cloud.mongodb.com, select your project, click the three-dot menu on your cluster, and select Terminate. Type the cluster name to confirm deletion.
  • Delete the local project folder. Run the following command in your terminal:
rm -rf ~/Desktop/order-processing-pipeline

On Windows, run this instead:

Remove-Item -Recurse -Force ~\Desktop\order-processing-pipeline

Nice Work!

Nice Work!

You just built a production-grade order processing pipeline with separate API and worker processes connected through Redis. This is the exact architecture that companies like Shopify use to handle millions of webhooks without dropping a single one.

You've learned how to:

  • Decouple webhook acceptance from heavy processing using BullMQ, with HMAC-SHA256 signature verification, exponential backoff retries, and structured dead-letter handling for permanently failed jobs.
  • Build idempotent handlers with two layers of deduplication protection (MongoDB unique constraint and BullMQ jobId) so duplicate webhooks never produce duplicate side effects.
  • Implement graceful shutdown that lets in-flight jobs complete before the worker process exits, enabling zero-downtime deployments without data loss.
  • Secret Mission: Added priority queuing so VIP customer orders jump ahead of standard orders, plus a queue status endpoint for production observability.

Ready to quiz yourself?