Order Pipeline with Redis and Kubernetes

Build a production-grade order processing pipeline with autoscaling and monitoring.

Introduction

30 Second Summary

Every time you order something online, dozens of invisible steps fire behind the scenes. Payment gets validated, inventory is checked, and a shipping label is generated.

In this project, you will build a production-grade order processing pipeline. You'll create two Node.js microservices connected by a Redis message queue, containerize them with Docker, deploy to Kubernetes, and monitor the whole system with Prometheus and Grafana.

What You'll Build

An order processing pipeline for an e-commerce backend, with a REST API that validates and queues incoming orders into Redis and a separate Worker service that processes them with retry logic and failure handling.

By the end of this project, you'll have:

Two Node.js microservices connected through Redis and BullMQ: an Order API with validation, idempotency, rate limiting, and API key authentication, and an Order Worker with retries, exponential backoff, and a dead-letter queue.
Both services containerized with Docker and deployed to Kubernetes with liveness and readiness probes, rolling updates, resource limits, and a HorizontalPodAutoscaler for automatic scaling.
A Prometheus and Grafana monitoring stack with custom application metrics, alert rules, and a load test script that proves the system handles heavy traffic.
Secret Mission: Build a production operations dashboard in Grafana with panels for order throughput, p95 latency, failure rates, queue depth, and replica count.

Are there any prerequisites?

This project assumes basic JavaScript and Node.js knowledge. You should be comfortable with variables, functions, async/await, and HTTP concepts.

Set Up Your Development Environment

You understand the architecture of your order processing pipeline. An API accepts orders, a Worker processes them, and Redis connects the two through a message queue. Kubernetes runs everything in production, and Prometheus plus Grafana make it all observable.

Before you can build any of that, you need the right tools on your machine. In this step, you will install every tool the project depends on: Node.js for writing the services, Docker for containerizing them, and the Kubernetes toolchain (Minikube, kubectl, Helm) for deploying and monitoring them.

In this step, get ready to:

Install VS Code, Node.js 22, and Docker Engine on Ubuntu.
Install Minikube, kubectl, and Helm.
Start a Minikube cluster and verify it is running.

Install Node.js, Docker, and VS Code

You need three foundational tools before anything else: a code editor, a JavaScript runtime, and a container engine. VS Code is where you write code. Node.js runs the API and Worker services. Docker packages them into containers.

Install VS Code by running this command:
(If snap is not available on your system, see the VS Code Linux install guide for alternative methods.)

sudo snap install --classic code

What is snap?

Snap is a package manager built into most Ubuntu installations. The --classic flag gives VS Code full access to your filesystem, which it needs to function as a code editor.

Install Node.js 22 LTS using the NodeSource repository by running these commands:

# Download and run the NodeSource setup script for Node.js 22
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -

# Install Node.js
sudo apt-get install -y nodejs

What do these commands do?

The first command downloads a setup script from NodeSource that adds their package repository and GPG signing key to your system. This gives you access to the latest Node.js 22 LTS builds instead of the outdated version in Ubuntu's default repositories.

The second command installs Node.js (and npm, which comes bundled with it) from the newly added repository.

Verify the installation by running:

node --version

You should see a version number starting with v22.

✔️ I see a version number starting with v22

Node.js is ready. Move on to installing Docker.

ⓧ Command not found or wrong version

If node is not found, the NodeSource script may not have run successfully. Try running the two installation commands again.

If you see a version lower than v22, a different Node.js installation may be taking priority. Remove it first:

sudo apt remove nodejs
sudo apt autoremove

Then re-run the NodeSource setup and install commands above.

Install Docker Engine using Docker's official repository. First, set up the repository by running:

# Add Docker's official GPG key
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the Docker repository to apt sources
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt-get update

Install the Docker packages by running:

sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

What are all these Docker packages?

docker-ce is the Docker Engine daemon. docker-ce-cli is the command-line client you use to interact with it. containerd.io is the container runtime that Docker uses under the hood.

docker-buildx-plugin adds extended build capabilities. docker-compose-plugin gives you the docker compose command for running multi-container setups.

Add your user to the docker group so you can run Docker commands without sudo:

sudo groupadd docker
sudo usermod -aG docker $USER

Log out and back in now

Docker group changes require you to log out and log back in to take effect. Do this before continuing to the next command.

After logging back in, verify Docker works by running:

docker run hello-world

You should see a message that says "Hello from Docker!" confirming that Docker can pull images and run containers.

Getting a permission denied error?

If you see permission denied while trying to connect to the Docker daemon socket, your group membership has not taken effect yet. Make sure you fully logged out and back in (not just opened a new terminal).

As a quick workaround, run newgrp docker in your current terminal, then try docker run hello-world again.

Help me troubleshoot Docker permission errors on Ubuntu

Install Minikube, kubectl, and Helm

With Docker running, you now need the Kubernetes toolchain. Minikube creates a local Kubernetes cluster inside Docker. kubectl is the CLI you use to manage that cluster. Helm is a package manager for Kubernetes that installs complex applications (like the Prometheus monitoring stack) with a single command.

Download and install Minikube by running:

# Download the Minikube binary for Linux amd64
curl -LO https://github.com/kubernetes/minikube/releases/latest/download/minikube-linux-amd64

# Install it to /usr/local/bin and clean up
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64

Download and install kubectl by running:

# Download the latest stable kubectl binary
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Install it to /usr/local/bin
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Clean up the downloaded file
rm kubectl

Install Helm using the official install script by running:

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

What does this script do?

The Helm install script detects your operating system and CPU architecture, downloads the correct Helm binary, verifies its checksum, and installs it to /usr/local/bin/helm.

Verify all three tools are installed by running:

minikube version
kubectl version --client
helm version

Each command should print a version string. If any command returns "command not found", re-run the corresponding installation commands above.

Command not found for minikube, kubectl, or helm?

Ensure /usr/local/bin is on your PATH. Run echo $PATH to check. If it is missing, add it by running:

export PATH=$PATH:/usr/local/bin

Then try the version commands again. Help me fix PATH issues on Ubuntu for Minikube, kubectl, or Helm

Start Minikube and verify the cluster

The final piece of setup is starting an actual Kubernetes cluster on your machine. Minikube runs this cluster inside Docker containers, so you do not need a virtual machine. You will allocate extra memory and CPUs because the monitoring stack (Prometheus, Grafana, Alertmanager) needs room to run alongside your application.

Start a Minikube cluster with the Docker driver and increased resources by running:

minikube start --driver=docker --memory=6144 --cpus=3

Why 6144 MB of memory?

The default 2 GB is enough for simple applications. This project also deploys Prometheus, Grafana, and Alertmanager, which are memory-hungry. Allocating 6 GB ensures the cluster does not run out of resources under load.

This command takes 1-2 minutes. You should see output ending with a message like "Done! kubectl is now configured to use minikube cluster."

Enable the metrics-server addon, which the HorizontalPodAutoscaler needs to read CPU usage:

minikube addons enable metrics-server

What is the metrics-server?

The HorizontalPodAutoscaler (HPA) automatically scales your Worker pods when CPU gets high. It needs a metrics source to know the current CPU utilization. The metrics-server addon provides that source inside your Minikube cluster.

Verify the cluster is running by running these commands:

kubectl get nodes
minikube status

kubectl get nodes should show one node with status Ready. minikube status should show the host, kubelet, and apiserver all running.

Node not showing Ready?

If the node shows NotReady, give it another minute. Minikube needs time to finish bootstrapping all system components after the initial start.

If the status stays NotReady for more than 3 minutes, try deleting and recreating the cluster:

minikube delete
minikube start --driver=docker --memory=6144 --cpus=3

Help me troubleshoot a Minikube cluster that will not start

Your development environment is fully set up. Next, you will build the Order API with authentication, rate limiting, idempotency, and Prometheus metrics.

Build the Order API and Worker

Your development environment is ready. Now it's time to build the two core services that make up the order processing pipeline.

The API service receives orders from clients, validates them, checks for duplicates using idempotency keys, and queues them in Redis via BullMQ. A separate Worker service picks up queued orders and processes them through simulated steps like payment validation and inventory checks. When an order fails all retry attempts, it lands in a dead-letter queue for manual review.

In this step, get ready to:

Create the project structure and install all dependencies.
Build the Order API with authentication, rate limiting, idempotency, and Prometheus metrics.
Build the Order Worker with retries, exponential backoff, and dead-letter queue handling.

Create the project structure and install dependencies

The project has three subdirectories. The api/ directory holds the Order API service. The worker/ directory holds the Order Worker. The loadtest/ directory is for a load test script you will write in a later step.

Move to your Desktop by running this command:

cd ~/Desktop

Create the project directory and subdirectories by running these commands:

mkdir order-pipeline
cd order-pipeline
mkdir api worker loadtest

Confirm the directories exist by running:

ls

You should see api, loadtest, and worker listed in the output.

Each service needs its own package.json to track its Node.js dependencies independently. This keeps each service self-contained, which matters when you containerize them later.

Initialize the API project and install its dependencies by running these commands:

cd api
npm init -y
npm install express@4.21.2 bullmq@5.76.4 ioredis@5.6.1 prom-client@15.1.3 express-rate-limit@8.4.1 uuid@11.1.0

What are all these packages?

express is the web framework for handling HTTP requests.
bullmq is the message queue library that manages job queuing and retries through Redis.
ioredis is the Redis client that both Express and BullMQ use to talk to Redis.
prom-client instruments your app with Prometheus metrics (counters, histograms, gauges).
express-rate-limit adds IP-based rate limiting to protect the API from abuse.
uuid generates unique order IDs.

You should see a message like added 76 packages (the exact number may vary). A node_modules folder and a package-lock.json file now exist inside api/.

Seeing npm errors?

Make sure you are inside the api/ directory. Run pwd to check your current location.

If you see permission errors, try running with sudo or check that Node.js installed correctly by running node --version.

Help me debug my npm install errors.

Now initialize the Worker project and install its dependencies by running these commands:

cd ../worker
npm init -y
npm install bullmq@5.76.4 ioredis@5.6.1 express@4.21.2 prom-client@15.1.3

Why does the Worker need Express?

The Worker itself does not handle user-facing HTTP requests. It uses Express solely to serve a /metrics endpoint and health check endpoints so Prometheus and Kubernetes can monitor it.

Both services need a running Redis instance to connect to. You will start a temporary Redis container for local development.

Move back to the project root and start a Redis container by running these commands:

cd ..
docker run -d --name redis-dev -p 6379:6379 redis:7-alpine

What does this command do?

-d runs the container in the background (detached mode).
--name redis-dev gives it a recognizable name you can reference later.
-p 6379:6379 maps the container's Redis port to your machine so your local services can connect.
redis:7-alpine is a lightweight Redis 7 image.

Verify Redis is running by running:

docker ps

You should see a container named redis-dev with status Up and port 0.0.0.0:6379->6379/tcp in the output.

Container not starting?

If you see a "name already in use" error, a previous container with that name exists. Remove it first by running docker rm -f redis-dev and then try again.

If you see a "permission denied" error, make sure you logged out and back in after adding your user to the Docker group in Step 1.

Help me fix my Docker container issue.

Open the order-pipeline folder in VS Code. Press Ctrl+Shift+P to open the command palette.
Type File: Open Folder and press Enter.
Navigate to Desktop/order-pipeline and click Open.

You should see the api, loadtest, and worker folders in the VS Code Explorer sidebar on the left.

Write the Order API with auth, rate limiting, and idempotency

The Order API is the front door of the pipeline. Clients send HTTP requests to submit orders, check order status, and review failed orders.

The API includes several production-grade features: API key authentication blocks unauthorized requests, rate limiting prevents abuse, and idempotency keys stop duplicate orders from being processed twice.

In VS Code, right-click the api folder in the Explorer sidebar.
Click New File and name it server.js.
Add the imports, Express app setup, Redis connection, BullMQ queues, and Prometheus metrics by copying the following code into the file:

const express = require('express');
const { Queue } = require('bullmq');
const IORedis = require('ioredis');
const client = require('prom-client');
const { rateLimit } = require('express-rate-limit');
const { v4: uuidv4 } = require('uuid');

const app = express();
app.use(express.json());

const API_KEY = process.env.API_KEY || 'dev-api-key-12345';

// Redis connection
const connection = new IORedis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT) || 6379,
  maxRetriesPerRequest: null,
});

// BullMQ queues
const orderQueue = new Queue('order-queue', { connection });
const dlqQueue = new Queue('dead-letter-queue', { connection });

// Prometheus metrics
const register = new client.Registry();
client.collectDefaultMetrics({ register });

const ordersCreated = new client.Counter({
  name: 'orders_created_total',
  help: 'Total number of orders created',
  labelNames: ['status'],
  registers: [register],
});

const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 5],
  registers: [register],
});

const queueDepth = new client.Gauge({
  name: 'orders_queue_depth',
  help: 'Current number of orders waiting in the queue',
  registers: [register],
  async collect() {
    const counts = await orderQueue.getJobCounts('waiting', 'active', 'delayed');
    this.set(counts.waiting + counts.active + counts.delayed);
  },
});

What does this code do?

The IORedis connection uses environment variables so it works both locally (localhost) and inside Kubernetes (redis service name). The maxRetriesPerRequest: null setting is required by BullMQ.
Two BullMQ queues are created: order-queue for incoming orders and dead-letter-queue for permanently failed orders.
Three Prometheus metrics are registered: a Counter tracks how many orders are created, a Histogram measures HTTP request durations, and a Gauge reports the current queue depth.

You will not see a change yet. The next two code chunks add the middleware, endpoints, and server startup.

Below the Prometheus metrics in api/server.js, add the rate limiter, middleware, health endpoints, and authentication by copying the following code:

// Rate limiting
const limiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  limit: 100,
  standardHeaders: 'draft-8',
  legacyHeaders: false,
  message: { error: 'Too many requests. Please try again later.' },
});

// HTTP duration tracking middleware
app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on('finish', () => {
    end({ method: req.method, route: req.route ? req.route.path : req.path, status_code: res.statusCode });
  });
  next();
});

// Health check (liveness) - no auth required
app.get('/health', (req, res) => {
  res.json({ status: 'ok', service: 'order-api' });
});

// Readiness check - verifies Redis connectivity
app.get('/ready', async (req, res) => {
  try {
    await connection.ping();
    res.json({ status: 'ready', redis: 'connected' });
  } catch (err) {
    res.status(503).json({ status: 'not ready', redis: 'disconnected' });
  }
});

// Metrics endpoint - no auth required
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

// API key authentication middleware
function authenticate(req, res, next) {
  const key = req.headers['x-api-key'];
  if (!key || key !== API_KEY) {
    return res.status(401).json({ error: 'Invalid or missing API key' });
  }
  next();
}

// Apply rate limiting and auth to order routes
app.use('/orders', limiter, authenticate);

What does this code do?

The rate limiter caps each IP address to 100 requests per 15-minute window. Requests beyond that limit receive a 429 status.
The HTTP duration middleware records how long every request takes, feeding the Prometheus histogram.
/health is a liveness probe. Kubernetes uses it to detect if the process is stuck. /ready is a readiness probe. It pings Redis before accepting traffic.
The authenticate middleware checks the x-api-key header. Requests without a valid key get a 401 response.
app.use('/orders', limiter, authenticate) applies both protections to every route under /orders. The health, metrics, and readiness endpoints stay unprotected so Kubernetes and Prometheus can reach them.

Why API key auth instead of JWT?

In production you would use JWT or OAuth for authentication. API key auth teaches the same concept (rejecting unauthorized requests at the middleware layer) without adding the complexity of an auth server.

You will not see a change yet. The next chunk adds the order endpoints and starts the server.

Below the app.use('/orders', ...) line in api/server.js, add the order endpoints and server startup by copying the following code:

// Create a new order
app.post('/orders', async (req, res) => {
  const { customer, items, idempotencyKey } = req.body;

  if (!customer || !items || !Array.isArray(items) || items.length === 0) {
    return res.status(400).json({ error: 'Fields "customer" (string) and "items" (non-empty array) are required' });
  }

  // Idempotency check
  if (idempotencyKey) {
    const existing = await connection.get('idempotency:' + idempotencyKey);
    if (existing) {
      ordersCreated.inc({ status: 'duplicate' });
      return res.status(200).json({ orderId: existing, status: 'already_created', message: 'Duplicate request detected' });
    }
  }

  const orderId = uuidv4();

  const job = await orderQueue.add(
    'process-order',
    { orderId, customer, items, createdAt: new Date().toISOString() },
    {
      jobId: orderId,
      attempts: 5,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: { count: 1000 },
      removeOnFail: false,
    }
  );

  // Store idempotency key with 1 hour TTL
  if (idempotencyKey) {
    await connection.set('idempotency:' + idempotencyKey, orderId, 'EX', 3600);
  }

  ordersCreated.inc({ status: 'queued' });
  res.status(201).json({ orderId: job.id, status: 'queued', message: 'Order accepted for processing' });
});

// Check order status
app.get('/orders/:id', async (req, res) => {
  const job = await orderQueue.getJob(req.params.id);
  if (!job) {
    return res.status(404).json({ error: 'Order not found' });
  }
  const state = await job.getState();
  const response = { orderId: job.id, status: state, data: job.data, attemptsMade: job.attemptsMade };
  if (state === 'completed') response.result = job.returnvalue;
  if (state === 'failed') response.error = job.failedReason;
  res.json(response);
});

// Get dead-letter queue contents
app.get('/orders/failed/dlq', async (req, res) => {
  const jobs = await dlqQueue.getJobs(['waiting', 'completed'], 0, 20);
  const results = jobs.map(j => ({ id: j.id, data: j.data, failedReason: j.data.failedReason }));
  res.json({ count: results.length, orders: results });
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log('Order API running on port ' + PORT);
});

What does this code do?

POST /orders validates the request body, checks for a duplicate idempotencyKey in Redis, generates a unique order ID, and adds the order to the BullMQ queue with 5 retry attempts and exponential backoff starting at 2 seconds.
removeOnFail: false keeps failed jobs in the queue so the Worker can move them to the dead-letter queue after all retries are exhausted.
GET /orders/:id looks up an order by ID and returns its current state (waiting, active, completed, or failed) plus the result or error.
GET /orders/failed/dlq returns up to 20 orders from the dead-letter queue for manual review.

What is idempotency?

Idempotency means that sending the same request multiple times produces the same result as sending it once. If a client's network drops after submitting an order, the client might retry the request. Without idempotency, you would end up with two identical orders.

The API stores each idempotencyKey in Redis with a 1-hour TTL. If the same key arrives again within that window, the API returns the original order ID instead of creating a duplicate.

Save api/server.js.

✔️ Awesome, I've got everything!

Double check you have saved the file before moving on.

ⓧ I'd like to double check the full code

Here is the complete api/server.js file. Compare it with yours and fix any differences.

const express = require('express');
const { Queue } = require('bullmq');
const IORedis = require('ioredis');
const client = require('prom-client');
const { rateLimit } = require('express-rate-limit');
const { v4: uuidv4 } = require('uuid');

const app = express();
app.use(express.json());

const API_KEY = process.env.API_KEY || 'dev-api-key-12345';

// Redis connection
const connection = new IORedis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT) || 6379,
  maxRetriesPerRequest: null,
});

// BullMQ queues
const orderQueue = new Queue('order-queue', { connection });
const dlqQueue = new Queue('dead-letter-queue', { connection });

// Prometheus metrics
const register = new client.Registry();
client.collectDefaultMetrics({ register });

const ordersCreated = new client.Counter({
  name: 'orders_created_total',
  help: 'Total number of orders created',
  labelNames: ['status'],
  registers: [register],
});

const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 5],
  registers: [register],
});

const queueDepth = new client.Gauge({
  name: 'orders_queue_depth',
  help: 'Current number of orders waiting in the queue',
  registers: [register],
  async collect() {
    const counts = await orderQueue.getJobCounts('waiting', 'active', 'delayed');
    this.set(counts.waiting + counts.active + counts.delayed);
  },
});

// Rate limiting
const limiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  limit: 100,
  standardHeaders: 'draft-8',
  legacyHeaders: false,
  message: { error: 'Too many requests. Please try again later.' },
});

// HTTP duration tracking middleware
app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on('finish', () => {
    end({ method: req.method, route: req.route ? req.route.path : req.path, status_code: res.statusCode });
  });
  next();
});

// Health check (liveness) - no auth required
app.get('/health', (req, res) => {
  res.json({ status: 'ok', service: 'order-api' });
});

// Readiness check - verifies Redis connectivity
app.get('/ready', async (req, res) => {
  try {
    await connection.ping();
    res.json({ status: 'ready', redis: 'connected' });
  } catch (err) {
    res.status(503).json({ status: 'not ready', redis: 'disconnected' });
  }
});

// Metrics endpoint - no auth required
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

// API key authentication middleware
function authenticate(req, res, next) {
  const key = req.headers['x-api-key'];
  if (!key || key !== API_KEY) {
    return res.status(401).json({ error: 'Invalid or missing API key' });
  }
  next();
}

// Apply rate limiting and auth to order routes
app.use('/orders', limiter, authenticate);

// Create a new order
app.post('/orders', async (req, res) => {
  const { customer, items, idempotencyKey } = req.body;

  if (!customer || !items || !Array.isArray(items) || items.length === 0) {
    return res.status(400).json({ error: 'Fields "customer" (string) and "items" (non-empty array) are required' });
  }

  // Idempotency check
  if (idempotencyKey) {
    const existing = await connection.get('idempotency:' + idempotencyKey);
    if (existing) {
      ordersCreated.inc({ status: 'duplicate' });
      return res.status(200).json({ orderId: existing, status: 'already_created', message: 'Duplicate request detected' });
    }
  }

  const orderId = uuidv4();

  const job = await orderQueue.add(
    'process-order',
    { orderId, customer, items, createdAt: new Date().toISOString() },
    {
      jobId: orderId,
      attempts: 5,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: { count: 1000 },
      removeOnFail: false,
    }
  );

  // Store idempotency key with 1 hour TTL
  if (idempotencyKey) {
    await connection.set('idempotency:' + idempotencyKey, orderId, 'EX', 3600);
  }

  ordersCreated.inc({ status: 'queued' });
  res.status(201).json({ orderId: job.id, status: 'queued', message: 'Order accepted for processing' });
});

// Check order status
app.get('/orders/:id', async (req, res) => {
  const job = await orderQueue.getJob(req.params.id);
  if (!job) {
    return res.status(404).json({ error: 'Order not found' });
  }
  const state = await job.getState();
  const response = { orderId: job.id, status: state, data: job.data, attemptsMade: job.attemptsMade };
  if (state === 'completed') response.result = job.returnvalue;
  if (state === 'failed') response.error = job.failedReason;
  res.json(response);
});

// Get dead-letter queue contents
app.get('/orders/failed/dlq', async (req, res) => {
  const jobs = await dlqQueue.getJobs(['waiting', 'completed'], 0, 20);
  const results = jobs.map(j => ({ id: j.id, data: j.data, failedReason: j.data.failedReason }));
  res.json({ count: results.length, orders: results });
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log('Order API running on port ' + PORT);
});

Now test the API to confirm everything works. Open a new terminal in VS Code by pressing Ctrl+`.

Start the API server by running:

cd ~/Desktop/order-pipeline/api
node server.js

You should see Order API running on port 3000 in the terminal.

Open a second terminal by clicking the + icon in the terminal panel.
Test the health endpoint by running:

curl http://localhost:3000/health

You should see {"status":"ok","service":"order-api"}.

Submit a test order by running this curl command:

curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -H "x-api-key: dev-api-key-12345" \
  -d '{"customer":"Alice","items":[{"product":"Laptop","quantity":1}],"idempotencyKey":"test-001"}'

You should see a JSON response with "status":"queued" and a unique orderId.

Test idempotency by running the exact same curl command again.

This time you should see "status":"already_created" with the same orderId as before. The API detected the duplicate idempotencyKey and returned the existing order instead of creating a new one.

Test that authentication works by sending a request without the API key:

curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -d '{"customer":"Bob","items":[{"product":"Phone","quantity":1}]}'

You should see {"error":"Invalid or missing API key"} with a 401 status. The API correctly rejected the unauthorized request.

Seeing Connection refused?

Make sure the API server is running in your first terminal. Look for the Order API running on port 3000 message.

If you see a Redis connection error, verify the Redis container is running with docker ps. You should see redis-dev listed.

Help me debug my API connection issue.

Stop the API server by pressing Ctrl+C in the first terminal. You will restart it after writing the Worker.

Write the Order Worker with retries and dead-letter queue

The Worker is the second half of the pipeline. It picks up orders from the queue and processes them through multiple steps. Each step has a small chance of failure, simulating real-world unreliability like payment gateway timeouts or inventory service outages.

When a step fails, BullMQ automatically retries with exponential backoff. If all 5 attempts are exhausted, the Worker moves the order to a dead-letter queue so an operator can investigate.

In VS Code, right-click the worker folder in the Explorer sidebar.
Click New File and name it worker.js.
Add the imports, Redis connection, dead-letter queue reference, and Prometheus metrics by copying the following code into the file:

const { Worker, Queue } = require('bullmq');
const IORedis = require('ioredis');
const express = require('express');
const client = require('prom-client');

const connection = new IORedis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT) || 6379,
  maxRetriesPerRequest: null,
});

const dlqQueue = new Queue('dead-letter-queue', { connection });

// Prometheus metrics
const register = new client.Registry();
client.collectDefaultMetrics({ register });

const ordersProcessed = new client.Counter({
  name: 'orders_processed_total',
  help: 'Total orders processed',
  labelNames: ['status', 'failed_step'],
  registers: [register],
});

const orderDuration = new client.Histogram({
  name: 'order_processing_duration_seconds',
  help: 'Order processing duration in seconds',
  buckets: [0.1, 0.25, 0.5, 1, 2, 5, 10],
  registers: [register],
});

const dlqSize = new client.Gauge({
  name: 'orders_dlq_size',
  help: 'Number of orders in the dead-letter queue',
  registers: [register],
  async collect() {
    const counts = await dlqQueue.getJobCounts('waiting', 'completed');
    this.set(counts.waiting + counts.completed);
  },
});

What does this code do?

The Worker imports both Worker (to consume jobs) and Queue (to push failed orders into the dead-letter queue) from BullMQ.
ordersProcessed is a Counter that tracks how many orders were completed, retried, or sent to the dead-letter queue, broken down by which step failed.
orderDuration is a Histogram that measures how long each order takes to process end-to-end.
dlqSize is a Gauge that reports the current number of orders stuck in the dead-letter queue. It computes the value on each Prometheus scrape.

You will not see a change yet. The next chunk adds the core order processing logic.

Below the Prometheus metrics in worker/worker.js, add the simulated processing steps and the BullMQ Worker by copying the following code:

// Simulated processing steps
async function simulateStep(name, delayMs, failChance) {
  await new Promise(resolve => setTimeout(resolve, delayMs));
  if (Math.random() < failChance) {
    throw new Error('Step "' + name + '" failed: simulated transient error');
  }
}

// Order processor
const worker = new Worker(
  'order-queue',
  async (job) => {
    const end = orderDuration.startTimer();
    const { orderId, customer, items } = job.data;
    console.log('Processing order ' + orderId + ' for ' + customer + ' (attempt ' + job.attemptsMade + ')');

    await job.updateProgress(10);
    await simulateStep('validate-payment', 200, 0.05);
    await job.updateProgress(40);

    await simulateStep('check-inventory', 150, 0.03);
    await job.updateProgress(70);

    await simulateStep('generate-shipping-label', 100, 0.02);
    await job.updateProgress(100);

    end();
    ordersProcessed.inc({ status: 'completed', failed_step: 'none' });
    console.log('Order ' + orderId + ' completed successfully');

    return {
      orderId,
      status: 'shipped',
      trackingNumber: 'TRK-' + Date.now(),
      processedAt: new Date().toISOString(),
    };
  },
  { connection, concurrency: 3 }
);

What does this code do?

simulateStep waits a fixed delay then randomly throws an error based on the failChance (e.g. 0.05 = 5% chance of failure). This simulates real-world service unreliability.
The Worker processes each order through three steps: validate payment (200ms, 5% fail), check inventory (150ms, 3% fail), and generate shipping label (100ms, 2% fail).
concurrency: 3 means this Worker processes 3 orders in parallel per pod. This gives the autoscaler a meaningful load to scale against.
job.updateProgress() reports how far along processing has gotten. This is visible when querying order status through the API.

What is exponential backoff?

When a job fails, BullMQ waits before retrying. With exponential backoff and a 2000ms base delay, the waits grow: 2 seconds, 4 seconds, 8 seconds, 16 seconds, 32 seconds. This gives transient issues (like a brief network hiccup) time to resolve without hammering the failing service.

You will not see a change yet. The final chunk adds failure handling and the metrics server.

Below the Worker in worker/worker.js, add the event handlers and metrics server by copying the following code:

worker.on('failed', async (job, err) => {
  const step = err.message.includes('validate-payment') ? 'payment'
    : err.message.includes('check-inventory') ? 'inventory'
    : err.message.includes('generate-shipping') ? 'shipping'
    : 'unknown';

  if (job && job.attemptsMade >= job.opts.attempts) {
    console.error('Order ' + job.data.orderId + ' permanently failed after ' + job.attemptsMade + ' attempts. Moving to DLQ.');
    await dlqQueue.add('failed-order', {
      ...job.data,
      failedReason: err.message,
      failedAt: new Date().toISOString(),
      totalAttempts: job.attemptsMade,
    });
    ordersProcessed.inc({ status: 'dead_letter', failed_step: step });
  } else {
    console.warn('Order ' + (job ? job.data.orderId : 'unknown') + ' failed (attempt ' + (job ? job.attemptsMade : '?') + '): ' + err.message);
    ordersProcessed.inc({ status: 'retrying', failed_step: step });
  }
});

worker.on('error', (err) => {
  console.error('Worker error:', err);
});

// Metrics and health server
const metricsApp = express();

metricsApp.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

metricsApp.get('/health', (req, res) => {
  res.json({ status: 'ok', service: 'order-worker' });
});

metricsApp.get('/ready', async (req, res) => {
  try {
    await connection.ping();
    res.json({ status: 'ready', redis: 'connected' });
  } catch (err) {
    res.status(503).json({ status: 'not ready', redis: 'disconnected' });
  }
});

const METRICS_PORT = process.env.METRICS_PORT || 3001;
metricsApp.listen(METRICS_PORT, () => {
  console.log('Worker metrics on port ' + METRICS_PORT);
  console.log('Order Worker ready. Processing with concurrency: 3');
});

What does this code do?

The failed event fires every time a job attempt fails. It checks job.attemptsMade >= job.opts.attempts to see if all retries are exhausted.
When retries are exhausted, the handler adds the order data to the dead-letter-queue with the failure reason and timestamp. This is the dead-letter queue pattern. BullMQ does not have a built-in DLQ, so you implement it manually in the event handler.
The metrics Express server on port 3001 exposes /metrics, /health, and /ready endpoints for Prometheus and Kubernetes to scrape and monitor.

What is a dead-letter queue?

A dead-letter queue (DLQ) is a separate queue where messages go after they have failed all processing attempts. Instead of silently losing the order, the system parks it in the DLQ so an operator can inspect the failure, fix the root cause, and optionally reprocess the order.

Without a DLQ, permanently failed orders would sit in the main queue's failed set with no visibility. The DLQ makes failures explicit and actionable.

Save worker/worker.js.

✔️ Awesome, I've got everything!

Double check you have saved the file before moving on.

ⓧ I'd like to double check the full code

Here is the complete worker/worker.js file. Compare it with yours and fix any differences.

const { Worker, Queue } = require('bullmq');
const IORedis = require('ioredis');
const express = require('express');
const client = require('prom-client');

const connection = new IORedis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT) || 6379,
  maxRetriesPerRequest: null,
});

const dlqQueue = new Queue('dead-letter-queue', { connection });

// Prometheus metrics
const register = new client.Registry();
client.collectDefaultMetrics({ register });

const ordersProcessed = new client.Counter({
  name: 'orders_processed_total',
  help: 'Total orders processed',
  labelNames: ['status', 'failed_step'],
  registers: [register],
});

const orderDuration = new client.Histogram({
  name: 'order_processing_duration_seconds',
  help: 'Order processing duration in seconds',
  buckets: [0.1, 0.25, 0.5, 1, 2, 5, 10],
  registers: [register],
});

const dlqSize = new client.Gauge({
  name: 'orders_dlq_size',
  help: 'Number of orders in the dead-letter queue',
  registers: [register],
  async collect() {
    const counts = await dlqQueue.getJobCounts('waiting', 'completed');
    this.set(counts.waiting + counts.completed);
  },
});

// Simulated processing steps
async function simulateStep(name, delayMs, failChance) {
  await new Promise(resolve => setTimeout(resolve, delayMs));
  if (Math.random() < failChance) {
    throw new Error('Step "' + name + '" failed: simulated transient error');
  }
}

// Order processor
const worker = new Worker(
  'order-queue',
  async (job) => {
    const end = orderDuration.startTimer();
    const { orderId, customer, items } = job.data;
    console.log('Processing order ' + orderId + ' for ' + customer + ' (attempt ' + job.attemptsMade + ')');

    await job.updateProgress(10);
    await simulateStep('validate-payment', 200, 0.05);
    await job.updateProgress(40);

    await simulateStep('check-inventory', 150, 0.03);
    await job.updateProgress(70);

    await simulateStep('generate-shipping-label', 100, 0.02);
    await job.updateProgress(100);

    end();
    ordersProcessed.inc({ status: 'completed', failed_step: 'none' });
    console.log('Order ' + orderId + ' completed successfully');

    return {
      orderId,
      status: 'shipped',
      trackingNumber: 'TRK-' + Date.now(),
      processedAt: new Date().toISOString(),
    };
  },
  { connection, concurrency: 3 }
);

worker.on('failed', async (job, err) => {
  const step = err.message.includes('validate-payment') ? 'payment'
    : err.message.includes('check-inventory') ? 'inventory'
    : err.message.includes('generate-shipping') ? 'shipping'
    : 'unknown';

  if (job && job.attemptsMade >= job.opts.attempts) {
    console.error('Order ' + job.data.orderId + ' permanently failed after ' + job.attemptsMade + ' attempts. Moving to DLQ.');
    await dlqQueue.add('failed-order', {
      ...job.data,
      failedReason: err.message,
      failedAt: new Date().toISOString(),
      totalAttempts: job.attemptsMade,
    });
    ordersProcessed.inc({ status: 'dead_letter', failed_step: step });
  } else {
    console.warn('Order ' + (job ? job.data.orderId : 'unknown') + ' failed (attempt ' + (job ? job.attemptsMade : '?') + '): ' + err.message);
    ordersProcessed.inc({ status: 'retrying', failed_step: step });
  }
});

worker.on('error', (err) => {
  console.error('Worker error:', err);
});

// Metrics and health server
const metricsApp = express();

metricsApp.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

metricsApp.get('/health', (req, res) => {
  res.json({ status: 'ok', service: 'order-worker' });
});

metricsApp.get('/ready', async (req, res) => {
  try {
    await connection.ping();
    res.json({ status: 'ready', redis: 'connected' });
  } catch (err) {
    res.status(503).json({ status: 'not ready', redis: 'disconnected' });
  }
});

const METRICS_PORT = process.env.METRICS_PORT || 3001;
metricsApp.listen(METRICS_PORT, () => {
  console.log('Worker metrics on port ' + METRICS_PORT);
  console.log('Order Worker ready. Processing with concurrency: 3');
});

Now test the full pipeline with both services running together.

Start the API server in the first terminal by running:

cd ~/Desktop/order-pipeline/api
node server.js

Open a second terminal by clicking the + icon in the terminal panel.
Start the Worker in the second terminal by running:

cd ~/Desktop/order-pipeline/worker
node worker.js

You should see two messages: Worker metrics on port 3001 and Order Worker ready. Processing with concurrency: 3.

Open a third terminal by clicking the + icon again.
Submit a test order by running:

curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -H "x-api-key: dev-api-key-12345" \
  -d '{"customer":"Alice","items":[{"product":"Laptop","quantity":1}],"idempotencyKey":"pipeline-test-001"}'

In your curl output, you should see "status":"queued" with an orderId. Switch to the Worker terminal. You should see log messages showing the order being processed through each step and then completing successfully.

Send a few more orders to see the Worker handle multiple jobs. Run this a few times:

curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -H "x-api-key: dev-api-key-12345" \
  -d '{"customer":"Bob","items":[{"product":"Phone","quantity":2}]}'

What about the dead-letter queue?

The simulated failure chances are low (2-5% per step), so most orders complete successfully. In a later step, you will run a load test that sends hundreds of orders. With enough volume, some orders will fail all 5 retries and land in the dead-letter queue.

You can check the DLQ at any time by running:

curl -H "x-api-key: dev-api-key-12345" http://localhost:3000/orders/failed/dlq

Verify the Worker metrics endpoint is working by running:

curl http://localhost:3001/health

You should see {"status":"ok","service":"order-worker"}.

Worker not processing orders?

Make sure the Worker is connected to the same Redis instance as the API. Both services default to localhost:6379. Verify Redis is running with docker ps.

Check that the Worker terminal shows no error messages. If you see a Redis connection error, the redis-dev container may have stopped. Restart it with docker start redis-dev.

Help me debug my Worker connection issue.

Stop both services by pressing Ctrl+C in each terminal.

You now have two working microservices communicating through a Redis message queue. The API handles validation, authentication, rate limiting, and idempotency. The Worker handles retries, exponential backoff, and dead-letter queue routing. Next up, you will containerize both services with Docker and test the full stack with Docker Compose.

Containerize and Test with Docker Compose

Your API and Worker services are running and processing orders on your local machine. The problem is that running them requires having the right version of Node.js installed, all dependencies downloaded, and Redis running in the background.

Docker solves this by packaging your code and its dependencies into a container image that runs identically on any machine. Docker Compose takes it further by letting you start multiple containers together with a single command.

In this step, get ready to:

Write Dockerfiles for both the API and Worker services.
Create a Docker Compose file to orchestrate Redis, the API, and the Worker.
Write a load test script and test the full pipeline end-to-end.

Write Dockerfiles for both services

A Dockerfile is a blueprint that tells Docker how to build a container image. Each instruction in the file adds a layer: start from a base image, copy your code in, install dependencies, and define how to start the app.

Create a new file called Dockerfile inside your api/ folder and paste the following code:

FROM node:22-alpine

WORKDIR /app

# Copy dependency manifests first for better layer caching
COPY package*.json ./
RUN npm install --production

# Copy application source code
COPY . .

EXPOSE 3000

CMD ["node", "server.js"]

What does this Dockerfile do?

FROM node:22-alpine starts from a lightweight Node.js 22 image based on Alpine Linux. Alpine images are much smaller than full Linux images, which speeds up builds and reduces attack surface.
COPY package*.json copies only the dependency files first. Docker caches each layer, so if your dependencies haven't changed, Docker skips reinstalling them on the next build.
EXPOSE 3000 documents which port the API listens on. It doesn't actually publish the port. That happens in Docker Compose or Kubernetes.

Save api/Dockerfile.
You won't build this image yet. Docker Compose will use it in the next substep.

The Worker needs its own Dockerfile that is nearly identical but points at worker.js and exposes port 3001 instead.

Create a new file called Dockerfile inside your worker/ folder and paste the following code:

FROM node:22-alpine

WORKDIR /app

# Copy dependency manifests first for better layer caching
COPY package*.json ./
RUN npm install --production

# Copy application source code
COPY . .

EXPOSE 3001

CMD ["node", "worker.js"]

What changed from the API Dockerfile?

Only two lines differ. EXPOSE 3001 reflects the Worker's metrics port, and CMD ["node", "worker.js"] starts the Worker script instead of the API server.

Save worker/Dockerfile.
Confirm both Dockerfiles exist by running:

ls api/Dockerfile worker/Dockerfile

You should see both file paths printed in the output.

Don't see one of the files?

Make sure the file is named exactly Dockerfile with a capital D and no file extension. It should not be Dockerfile.txt or dockerfile.

Check that each file is inside the correct subfolder: api/Dockerfile and worker/Dockerfile. Help me find my missing Dockerfile.

✔️ Awesome, I've got everything!

Great. Make sure you have saved both files before moving on.

ⓧ I'd like to double check the full code

Here is the complete api/Dockerfile:

FROM node:22-alpine

WORKDIR /app

COPY package*.json ./
RUN npm install --production

COPY . .

EXPOSE 3000

CMD ["node", "server.js"]

Here is the complete worker/Dockerfile:

FROM node:22-alpine

WORKDIR /app

COPY package*.json ./
RUN npm install --production

COPY . .

EXPOSE 3001

CMD ["node", "worker.js"]

Create docker-compose.yml and test the full stack

Running each service manually gets tedious. Docker Compose lets you define all your services in a single file and start everything with one command.

Before you start the Compose stack, you need to remove the standalone Redis container you started earlier. Docker Compose will create its own Redis container, and having two Redis instances on the same port causes a conflict.

Stop and remove the temporary Redis container by running this command:

docker stop redis-dev && docker rm redis-dev

Verify the container is gone by running:

docker ps -a | grep redis-dev

You should see no output, confirming the container is removed.

Seeing "No such container"?

If the container was already stopped or removed, Docker prints an error. That is fine. It just means the container is already gone. Help me clean up my Redis container.

Now create the Compose file that defines the three services: Redis (the message broker), the API (the order entry point), and the Worker (the order processor).

Create a new file called docker-compose.yml in the root of your project folder (the same level as the api/ and worker/ folders) and paste the following code:

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    command: ["redis-server", "--maxmemory-policy", "noeviction"]

  api:
    build: ./api
    ports:
      - "3000:3000"
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - API_KEY=dev-api-key-12345
    depends_on:
      - redis

  worker:
    build: ./worker
    ports:
      - "3001:3001"
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - METRICS_PORT=3001
    depends_on:
      - redis

What does this Compose file do?

redis pulls the official redis:7-alpine image and starts it with --maxmemory-policy noeviction. This is critical for BullMQ. If Redis evicts keys under memory pressure, BullMQ's internal data structures break silently.
api builds the image from ./api using the Dockerfile you just wrote. The REDIS_HOST=redis environment variable tells the API to connect to the redis service by name. Docker Compose creates a network where services can reach each other by service name.
depends_on ensures Redis starts before the API and Worker. It does not wait for Redis to be fully ready, but it guarantees the container is created first.

Why noeviction for Redis?

BullMQ stores job metadata, queue state, and retry counters as Redis keys. If Redis runs low on memory and evicts those keys, jobs vanish mid-processing with no error. The noeviction policy tells Redis to reject new writes instead of silently deleting existing data.

Save docker-compose.yml.
You won't run Compose yet. First you will create a load test script to stress-test the pipeline after it is up.

Before starting the Compose stack, create a load test script that you will use later to bombard the API with orders. This script generates configurable bursts of HTTP requests so you can watch the system handle real traffic.

Create a new file called loadtest.js inside your loadtest/ folder and paste the configuration and request function:

// Load test configuration from environment variables or defaults
const TOTAL_ORDERS = parseInt(process.env.TOTAL_ORDERS) || 200;
const DURATION_SECONDS = parseInt(process.env.DURATION_SECONDS) || 30;
const API_URL = process.env.API_URL || 'http://localhost:3000';
const API_KEY = process.env.API_KEY || 'dev-api-key-12345';

// Sample data for generating realistic orders
const customers = ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'];
const products = ['Laptop', 'Phone', 'Headphones', 'Keyboard', 'Monitor', 'Mouse'];

// Send a single order to the API
async function sendOrder(index) {
  const order = {
    customer: customers[index % customers.length],
    items: [
      { product: products[Math.floor(Math.random() * products.length)], quantity: Math.floor(Math.random() * 5) + 1 },
    ],
    idempotencyKey: 'load-test-' + Date.now() + '-' + index,
  };

  try {
    const res = await fetch(API_URL + '/orders', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json', 'x-api-key': API_KEY },
      body: JSON.stringify(order),
    });
    const data = await res.json();
    if (res.ok) return { success: true, orderId: data.orderId };
    return { success: false, error: data.error || res.statusText };
  } catch (err) {
    return { success: false, error: err.message };
  }
}

What does this code do?

The four const lines at the top read configuration from environment variables. This lets you change the number of orders, duration, target URL, and API key without editing the file.
sendOrder() builds a random order payload (rotating through customers, picking random products and quantities) and sends it to the API with the required x-api-key header.
Each order gets a unique idempotencyKey built from the timestamp and index so the API does not reject them as duplicates.

Still in loadtest/loadtest.js, add the orchestrator function below the sendOrder function:

// Orchestrate the load test by spacing orders evenly over the duration
async function run() {
  console.log('Starting load test: ' + TOTAL_ORDERS + ' orders over ' + DURATION_SECONDS + 's');
  const interval = (DURATION_SECONDS * 1000) / TOTAL_ORDERS;
  let sent = 0;
  let succeeded = 0;
  let failed = 0;

  for (let i = 0; i < TOTAL_ORDERS; i++) {
    sendOrder(i).then(result => {
      if (result.success) succeeded++;
      else failed++;
    });
    sent++;
    if (sent % 20 === 0) console.log('Sent: ' + sent + '/' + TOTAL_ORDERS);
    await new Promise(r => setTimeout(r, interval));
  }

  // Wait a few seconds for in-flight responses to return
  await new Promise(r => setTimeout(r, 3000));
  console.log('Load test complete. Sent: ' + sent + ' Succeeded: ' + succeeded + ' Failed: ' + failed);
}

run();

What does the run function do?

run() calculates the delay between each request so all orders are spaced evenly across the configured duration (200 orders over 30 seconds = one order every 150 milliseconds).
It fires each request without waiting for a response (fire-and-forget), which creates realistic concurrent load on the API.
Every 20 orders it logs a progress update so you can see the test is running.

Save loadtest/loadtest.js.
You will not run the load test yet. You will use it in Step 5 to stress-test your Kubernetes deployment.

✔️ Awesome, I've got everything!

Great. Make sure you have saved all three files before moving on.

ⓧ I'd like to double check the full code

Here is the complete docker-compose.yml:

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    command: ["redis-server", "--maxmemory-policy", "noeviction"]

  api:
    build: ./api
    ports:
      - "3000:3000"
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - API_KEY=dev-api-key-12345
    depends_on:
      - redis

  worker:
    build: ./worker
    ports:
      - "3001:3001"
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - METRICS_PORT=3001
    depends_on:
      - redis

Here is the complete loadtest/loadtest.js:

const TOTAL_ORDERS = parseInt(process.env.TOTAL_ORDERS) || 200;
const DURATION_SECONDS = parseInt(process.env.DURATION_SECONDS) || 30;
const API_URL = process.env.API_URL || 'http://localhost:3000';
const API_KEY = process.env.API_KEY || 'dev-api-key-12345';

const customers = ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'];
const products = ['Laptop', 'Phone', 'Headphones', 'Keyboard', 'Monitor', 'Mouse'];

async function sendOrder(index) {
  const order = {
    customer: customers[index % customers.length],
    items: [
      { product: products[Math.floor(Math.random() * products.length)], quantity: Math.floor(Math.random() * 5) + 1 },
    ],
    idempotencyKey: 'load-test-' + Date.now() + '-' + index,
  };

  try {
    const res = await fetch(API_URL + '/orders', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json', 'x-api-key': API_KEY },
      body: JSON.stringify(order),
    });
    const data = await res.json();
    if (res.ok) return { success: true, orderId: data.orderId };
    return { success: false, error: data.error || res.statusText };
  } catch (err) {
    return { success: false, error: err.message };
  }
}

async function run() {
  console.log('Starting load test: ' + TOTAL_ORDERS + ' orders over ' + DURATION_SECONDS + 's');
  const interval = (DURATION_SECONDS * 1000) / TOTAL_ORDERS;
  let sent = 0;
  let succeeded = 0;
  let failed = 0;

  for (let i = 0; i < TOTAL_ORDERS; i++) {
    sendOrder(i).then(result => {
      if (result.success) succeeded++;
      else failed++;
    });
    sent++;
    if (sent % 20 === 0) console.log('Sent: ' + sent + '/' + TOTAL_ORDERS);
    await new Promise(r => setTimeout(r, interval));
  }

  await new Promise(r => setTimeout(r, 3000));
  console.log('Load test complete. Sent: ' + sent + ' Succeeded: ' + succeeded + ' Failed: ' + failed);
}

run();

Time to bring the full stack up. Docker Compose will build both images from your Dockerfiles, pull the Redis image, and start all three containers on a shared network.

Start the entire stack by running this command from your project root folder (the same level as docker-compose.yml):

docker compose up --build

What does this command do?

docker compose up starts all services defined in docker-compose.yml. The --build flag forces Docker to rebuild the API and Worker images from the Dockerfiles before starting. Without it, Docker reuses cached images.

You should see Docker pulling the Redis image, building the API and Worker images, and then interleaved logs from all three services.

Look for these lines in the output to confirm everything started successfully:

Order API running on port 3000 from the API service.
Worker metrics on port 3001 and Order Worker ready. Processing with concurrency: 3 from the Worker service.

Seeing connection errors or port conflicts?

If you see "port is already allocated" errors, make sure you stopped the redis-dev container from the previous step. Also check that you do not have the API or Worker running directly with node in another terminal.
If the API or Worker shows Redis connection errors, wait a few seconds. The depends_on directive starts Redis first, but it may not be fully ready when the API tries to connect. BullMQ retries the connection automatically.

Help me debug my Docker Compose startup.

With the stack running, open a new terminal to test the pipeline end-to-end. The logs from Docker Compose are streaming in the first terminal, so you need a separate one for curl commands.

Send a test order to the API by running this command:

curl -s -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -H "x-api-key: dev-api-key-12345" \
  -d '{"customer": "Alice", "items": [{"product": "Laptop", "quantity": 1}], "idempotencyKey": "docker-test-1"}'

You should see a JSON response with "status": "queued" and an orderId. In the Docker Compose terminal, you should also see the Worker pick up and process the order.

Verify the health endpoint by running:

curl -s http://localhost:3000/health

You should see {"status":"ok","service":"order-api"}.

Verify the readiness endpoint by running:

curl -s http://localhost:3000/ready

You should see {"status":"ready","redis":"connected"}. This confirms the API container can reach the Redis container over the Docker network.

Verify the Prometheus metrics endpoint by running:

curl -s http://localhost:3000/metrics | head -20

You should see Prometheus-formatted text with metrics like orders_created_total and http_request_duration_seconds. These are the custom metrics you instrumented in the API.

Not getting a response from curl?

Make sure the Docker Compose terminal still shows the services running. If you accidentally closed it, run docker compose up again from your project root.
Check that you are sending the x-api-key header in the order request. Without it, the API returns a 401 error.
Verify you are hitting localhost:3000 for the API and localhost:3001 for the Worker metrics.

Help me debug my Docker Compose curl requests.

Your entire pipeline is now running inside Docker containers. Next up, you will deploy these containers to a Kubernetes cluster with health checks, rolling updates, and autoscaling.

Deploy to Kubernetes with Production Configuration

Your Docker Compose stack proved that the API, Worker, and Redis work together as containers. The next challenge is running them in an environment that can self-heal, scale, and update without downtime.

Kubernetes provides all of this out of the box. In this step, you'll create Kubernetes manifests that configure health checks, rolling updates, resource limits, and autoscaling for your order pipeline.

In this step, get ready to:

Build Docker images inside Minikube's Docker daemon.
Create Kubernetes manifests for Redis, the API, and the Worker with production configuration.
Configure a HorizontalPodAutoscaler that scales the Worker under load.

Build images and create Kubernetes resources

Minikube runs its own Docker daemon inside a virtual machine. To use your local images without pushing to a registry, you need to build them directly inside Minikube's daemon.

If your Docker Compose stack is still running from the previous step, stop it by running:

docker compose down

Point your Docker CLI to Minikube's Docker daemon by running this command:

eval $(minikube docker-env)

What does this command do?

This command sets environment variables in your current terminal session so that docker build and docker images talk to Minikube's Docker daemon instead of your local one.

This only affects the current terminal session. If you open a new terminal later, you'll need to run this command again.

Build both Docker images inside Minikube by running these commands:

docker build -t order-api:v1 ./api
docker build -t order-worker:v1 ./worker

Verify both images exist inside Minikube by running:

docker images | grep order

You should see order-api and order-worker both tagged v1 in the output.

Not seeing the images?

Make sure you ran eval $(minikube docker-env) in the same terminal session before building. If you switched terminals, you need to run it again.

Also confirm you're running the build commands from the order-pipeline project root (the folder containing api/ and worker/ directories).

Help me debug my Docker image build inside Minikube

Now you'll create the Kubernetes manifest files. These YAML files describe the desired state of your cluster.

Create a k8s/ directory inside your project folder by running:

mkdir k8s

Create a file called namespace.yaml inside the k8s/ directory and paste the following code:

apiVersion: v1
kind: Namespace
metadata:
  name: orders

What is a Kubernetes Namespace?

A Namespace is a way to group related resources inside a cluster. All your order pipeline resources will live in the orders namespace, keeping them separate from system-level resources.

Save namespace.yaml.

Create a file called secret.yaml inside k8s/ and paste the following code:

apiVersion: v1
kind: Secret
metadata:
  name: api-key
  namespace: orders
type: Opaque
stringData:
  API_KEY: "production-api-key-secret"

What is a Kubernetes Secret?

A Secret stores sensitive data like API keys and passwords. Instead of hardcoding the API key in your Deployment manifest, you reference the Secret. Kubernetes injects it as an environment variable at runtime.

The stringData field lets you write the value in plain text. Kubernetes base64-encodes it automatically when storing it.

Save secret.yaml.

Next, you'll create the Redis deployment. Redis needs to be running inside the cluster before the API and Worker can connect to it.

Create a file called redis.yaml inside k8s/ and paste the following code:

# Deployment: runs the Redis container
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: orders
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          command: ["redis-server", "--maxmemory-policy", "noeviction"]
          ports:
            - containerPort: 6379
          resources:
            requests:
              memory: "64Mi"
              cpu: "50m"
            limits:
              memory: "128Mi"
              cpu: "200m"
---
# Service: makes Redis accessible by name within the cluster
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: orders
spec:
  selector:
    app: redis
  ports:
    - port: 6379
      targetPort: 6379

What does this manifest do?

This file defines two Kubernetes resources separated by ---.

The Deployment runs a single Redis replica with the noeviction memory policy. This is critical for BullMQ. If Redis evicts keys, BullMQ's internal data structures break silently.
The Service gives Redis a stable DNS name (redis) inside the cluster. Your API and Worker connect to this name instead of an IP address.
The resources section sets memory and CPU boundaries so Redis cannot consume all cluster resources.

Save redis.yaml.

Deploy the API with health checks, rolling updates, and resource limits

The API Deployment is where all the production reliability features come together. You'll configure health checks that let Kubernetes detect and recover from failures automatically.

Create a file called api.yaml inside k8s/ and paste the following code:

# Deployment: runs the Order API with production configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
  namespace: orders
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
    spec:
      containers:
        - name: order-api
          image: order-api:v1
          imagePullPolicy: Never
          ports:
            - containerPort: 3000
          env:
            - name: REDIS_HOST
              value: "redis"
            - name: REDIS_PORT
              value: "6379"
            - name: API_KEY
              valueFrom:
                secretKeyRef:
                  name: api-key
                  key: API_KEY
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "250m"
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 5
---
# Service: exposes the API within the cluster with labeled ports
apiVersion: v1
kind: Service
metadata:
  name: order-api
  namespace: orders
  labels:
    app: order-api
spec:
  selector:
    app: order-api
  ports:
    - port: 3000
      targetPort: 3000
      name: http

What do all these production features do?

livenessProbe hits /health every 10 seconds. If it fails, Kubernetes kills and restarts the pod. This catches stuck processes.
readinessProbe hits /ready every 5 seconds. This endpoint pings Redis. If Redis is down, the pod is removed from the Service so it stops receiving traffic.
The rolling update strategy with maxSurge: 1 and maxUnavailable: 0 means Kubernetes creates a new pod before terminating an old one during updates. This guarantees zero-downtime deployments.
The resources section defines memory and CPU boundaries. requests is the guaranteed minimum. limits is the maximum the container can use.
imagePullPolicy: Never tells Kubernetes to use the local image instead of trying to pull from a remote registry.
The API_KEY environment variable is injected from the Kubernetes Secret you created earlier, keeping the credential out of the manifest.

Save api.yaml.

Why two replicas for the API but one for the Worker?

The API handles incoming HTTP requests, so running 2 replicas means the Service can distribute traffic across both pods. If one pod restarts, the other continues serving.

The Worker starts with 1 replica because the HorizontalPodAutoscaler (which you'll create next) manages its replica count dynamically based on CPU usage.

✔️ Awesome, I've got everything!

Make sure you've saved api.yaml before continuing.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
  namespace: orders
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
    spec:
      containers:
        - name: order-api
          image: order-api:v1
          imagePullPolicy: Never
          ports:
            - containerPort: 3000
          env:
            - name: REDIS_HOST
              value: "redis"
            - name: REDIS_PORT
              value: "6379"
            - name: API_KEY
              valueFrom:
                secretKeyRef:
                  name: api-key
                  key: API_KEY
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "250m"
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: order-api
  namespace: orders
  labels:
    app: order-api
spec:
  selector:
    app: order-api
  ports:
    - port: 3000
      targetPort: 3000
      name: http

Deploy the Worker with autoscaling

The Worker Deployment follows the same production patterns as the API. The key difference is that a HorizontalPodAutoscaler will manage its replica count based on CPU usage.

Create a file called worker.yaml inside k8s/ and paste the following code:

# Deployment: runs the Order Worker with health checks and resource limits
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-worker
  namespace: orders
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: order-worker
  template:
    metadata:
      labels:
        app: order-worker
    spec:
      containers:
        - name: order-worker
          image: order-worker:v1
          imagePullPolicy: Never
          ports:
            - containerPort: 3001
          env:
            - name: REDIS_HOST
              value: "redis"
            - name: REDIS_PORT
              value: "6379"
            - name: METRICS_PORT
              value: "3001"
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "250m"
          livenessProbe:
            httpGet:
              path: /health
              port: 3001
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 3001
            initialDelaySeconds: 5
            periodSeconds: 5
---
# Service: exposes Worker metrics for Prometheus scraping
apiVersion: v1
kind: Service
metadata:
  name: order-worker
  namespace: orders
  labels:
    app: order-worker
spec:
  selector:
    app: order-worker
  ports:
    - port: 3001
      targetPort: 3001
      name: metrics

What does this manifest do?

The Worker Deployment mirrors the API's production features: liveness and readiness probes, resource requests/limits, and a rolling update strategy.

The Service exposes port 3001 with the name metrics. Prometheus will use this named port to scrape the Worker's metrics in the next step.

Save worker.yaml.

Now you'll create the autoscaler. The HPA watches CPU utilization on the Worker pods and adds or removes replicas to match demand.

Create a file called hpa.yaml inside k8s/ and paste the following code:

# HorizontalPodAutoscaler: scales Worker replicas based on CPU usage
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-worker-hpa
  namespace: orders
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-worker
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

What does this autoscaler do?

scaleTargetRef points at the order-worker Deployment. The HPA only controls this Deployment's replica count.
When average CPU utilization across Worker pods exceeds 60%, the HPA adds replicas (up to 5). When usage drops, it scales back down to 1.
The HPA requires the metrics-server addon you enabled in Step 1. Without it, the HPA has no CPU data to make scaling decisions.

Save hpa.yaml.

All six manifest files are ready. Time to apply them to the cluster.

Apply the namespace first by running:

kubectl apply -f k8s/namespace.yaml

Apply all remaining manifests by running:

kubectl apply -f k8s/

Why apply the namespace first?

Every other manifest references namespace: orders. If the namespace does not exist yet, those resources fail to create. Applying namespace.yaml first guarantees the namespace is ready. The second command re-applies it (which is harmless) along with everything else.

You should see output confirming each resource was created (namespace, secret, deployment, service, horizontalpodautoscaler).

Verify all pods are running by running:

kubectl get pods -n orders

You should see 4 pods: 1 Redis pod, 2 API pods, and 1 Worker pod. All should show Running status with 1/1 in the READY column.

Pods not ready yet?

Pods may take 15-30 seconds to pull images and start. If you see ContainerCreating, wait a moment and run the command again. You can also watch in real time:

kubectl get pods -n orders --watch

Press Ctrl+C to stop watching once all pods show Running.

Verify the HPA is active by running:

kubectl get hpa -n orders

You should see order-worker-hpa targeting the order-worker Deployment with min 1 and max 5 replicas. The TARGETS column may show <unknown>/60% initially. This is normal. It takes 1-2 minutes for the metrics-server to start reporting CPU data.

Now test the full pipeline running inside Kubernetes.

Port-forward the API Service to your local machine by running:

kubectl port-forward svc/order-api 3000:3000 -n orders

What does port-forward do?

This tunnels traffic from localhost:3000 on your machine to port 3000 on the order-api Service inside the cluster. You can now send requests to the API as if it were running locally.

Open a new terminal tab or window.
Send a test order to the API running inside Kubernetes by running:

curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -H "x-api-key: production-api-key-secret" \
  -d '{"customer":"Alice","items":[{"product":"Laptop","quantity":1}],"idempotencyKey":"k8s-test-1"}'

You should see a JSON response with "status":"queued" and an orderId. The API key is now production-api-key-secret (the value from your Kubernetes Secret), not the dev key you used earlier.

Getting a connection refused or 401 error?

If you see Connection refused, make sure the port-forward command is still running in your other terminal.
If you see a 401 error, check that you're using the production API key (production-api-key-secret) in your curl command, not the dev key.
If pods are crashing, check logs with kubectl logs -l app=order-api -n orders to see what went wrong.

Help me debug my Kubernetes deployment

Your order pipeline is running on Kubernetes with self-healing health checks, zero-downtime rolling updates, and autoscaling. Next up, you'll add monitoring with Prometheus and Grafana so you can see exactly what's happening inside the system.

Set Up Monitoring, Alerts, and Load Testing

Your order processing pipeline is deployed on Kubernetes with health checks, rolling updates, and autoscaling. But right now, you have no visibility into what is happening inside it.

In this step, you will deploy a full Prometheus + Grafana monitoring stack, wire it to your services with ServiceMonitors and alert rules, then hammer the system with a load test to watch autoscaling and monitoring in action.

In this step, get ready to:

Deploy Prometheus + Grafana and connect them to your services.
Run a load test and watch the Worker pods autoscale.
Explore Grafana dashboards, query custom metrics, and verify alert rules.

Deploy Prometheus + Grafana and connect to your apps

Both your API and Worker already expose a /metrics endpoint with Prometheus counters, histograms, and gauges. You need a Prometheus server to scrape those endpoints and a Grafana instance to visualize the data.

The kube-prometheus-stack Helm chart bundles Prometheus, Grafana, Alertmanager, and the Prometheus Operator into a single install.

Add the Prometheus community Helm repo and install the monitoring stack by running these commands:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false

What do these flags do?

By default, the kube-prometheus-stack only discovers ServiceMonitors and PrometheusRules that carry a release label matching the Helm release name. Your ServiceMonitors live in the orders namespace without that label.

Setting serviceMonitorSelectorNilUsesHelmValues=false and ruleSelectorNilUsesHelmValues=false tells Prometheus to discover all ServiceMonitors and PrometheusRules across all namespaces, regardless of labels.

The monitoring stack takes 2-3 minutes to fully start. Watch the pods come up by running:

kubectl get pods -n monitoring --watch

Wait until all pods show Running or Completed status, then press Ctrl+C to stop watching.

Pods stuck in Pending?

The monitoring stack needs significant memory. If pods stay in Pending, your Minikube cluster may not have enough resources. Verify your cluster was started with --memory=6144 --cpus=3 in Step 1.

If you need to recreate the cluster, run minikube delete and then minikube start --driver=docker --memory=6144 --cpus=3. You will need to re-apply all your Kubernetes manifests afterwards.

Help me troubleshoot pods stuck in Pending in my Minikube monitoring namespace

Now you need to tell Prometheus where to find your application metrics. A ServiceMonitor is a custom Kubernetes resource that tells Prometheus which Services to scrape and how often.

Create the file k8s/service-monitor.yaml and paste the following:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: order-api-monitor
  namespace: orders
spec:
  selector:
    matchLabels:
      app: order-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: order-worker-monitor
  namespace: orders
spec:
  selector:
    matchLabels:
      app: order-worker
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s

What are ServiceMonitors?

Instead of editing Prometheus configuration files directly, the Prometheus Operator watches for ServiceMonitor resources. Each ServiceMonitor uses label selectors to find matching Kubernetes Services and tells Prometheus which port and path to scrape.

The selector.matchLabels field must match the labels on the target Service. The port field references the named port in the Service spec (http for the API, metrics for the Worker). The interval of 15s means Prometheus scrapes every 15 seconds.

Save k8s/service-monitor.yaml.

Next, you will create alert rules. A PrometheusRule defines PromQL expressions that fire alerts when certain conditions are met.

Create the file k8s/prometheus-rules.yaml and paste the following:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: order-alerts
  namespace: orders
  labels:
    release: prometheus-stack
spec:
  groups:
    - name: order-processing
      rules:
        - alert: HighOrderFailureRate
          expr: rate(orders_processed_total{status="dead_letter"}[5m]) / rate(orders_processed_total[5m]) > 0.1
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "High order failure rate detected"
            description: "More than 10% of orders are permanently failing over the last 5 minutes."
        - alert: DeadLetterQueueNotEmpty
          expr: orders_dlq_size > 0
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: "Orders in dead-letter queue"
            description: "There are {{ $value }} orders in the dead-letter queue requiring manual review."

What do these alert rules do?

The HighOrderFailureRate alert fires when more than 10% of processed orders end up in the dead-letter queue over a 5-minute window. This is a critical alert because it means your system is permanently losing orders.

The DeadLetterQueueNotEmpty alert fires whenever the DLQ has any orders in it. This is a warning because even a single poisoned message needs manual attention.

The release: prometheus-stack label must match the Helm release name you used when installing the monitoring stack. This is how Prometheus knows to pick up these rules.

Save k8s/prometheus-rules.yaml.
Apply both new manifests by running:

kubectl apply -f k8s/service-monitor.yaml
kubectl apply -f k8s/prometheus-rules.yaml

Verify the resources were created by running:

kubectl get servicemonitors -n orders
kubectl get prometheusrules -n orders

You should see order-api-monitor and order-worker-monitor in the ServiceMonitors list, and order-alerts in the PrometheusRules list.

No resources found?

Make sure you are querying the orders namespace with -n orders. The ServiceMonitor and PrometheusRule CRDs are installed by the kube-prometheus-stack. If you see error: the server doesn't have a resource type, the monitoring stack may not have finished installing yet. Wait another minute and try again.

Help me debug why my ServiceMonitors are not showing up

✔️ Awesome, I've got everything!

Great. Double check you have saved both YAML files.

ⓧ I'd like to double check the full code

Here is the full k8s/service-monitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: order-api-monitor
  namespace: orders
spec:
  selector:
    matchLabels:
      app: order-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: order-worker-monitor
  namespace: orders
spec:
  selector:
    matchLabels:
      app: order-worker
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s

And the full k8s/prometheus-rules.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: order-alerts
  namespace: orders
  labels:
    release: prometheus-stack
spec:
  groups:
    - name: order-processing
      rules:
        - alert: HighOrderFailureRate
          expr: rate(orders_processed_total{status="dead_letter"}[5m]) / rate(orders_processed_total[5m]) > 0.1
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "High order failure rate detected"
            description: "More than 10% of orders are permanently failing over the last 5 minutes."
        - alert: DeadLetterQueueNotEmpty
          expr: orders_dlq_size > 0
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: "Orders in dead-letter queue"
            description: "There are {{ $value }} orders in the dead-letter queue requiring manual review."

Run a load test and watch the system respond

Your monitoring stack is deployed and wired up. Now it is time to see everything in action by flooding the API with orders and watching the HorizontalPodAutoscaler scale your Worker pods up in response.

You will need three separate terminal windows for this substep. One for the API port-forward, one for the HPA watch, and one for the load test.

In your first terminal, port-forward the API service by running:

kubectl port-forward -n orders svc/order-api 3000:3000

Why port-forward again?

If you still have the port-forward from Step 4 running, you can skip this. The load test script sends HTTP requests to http://localhost:3000, which needs to reach the API inside Kubernetes.

Open a second terminal. Watch the HPA respond in real time by running:

kubectl get hpa -n orders --watch

This shows the current CPU utilization and replica count for the Worker. Leave it running.

Open a third terminal. Navigate to your project folder (the one containing api/, worker/, and loadtest/).
Run the load test by executing this command:

API_KEY=production-api-key-secret node loadtest/loadtest.js

What does this command do?

The API_KEY=production-api-key-secret sets the API key to match the Kubernetes Secret value (not the local dev key). The load test sends 200 orders over 30 seconds to http://localhost:3000/orders, which routes through the port-forward to the API pods.

Each order includes a random product, quantity, and unique idempotency key. The script logs progress every 20 orders and a final summary of successes and failures.

Watch your second terminal (the HPA watcher). Within 1-2 minutes, you should see the TARGETS column show increasing CPU utilization. When it exceeds 60%, the REPLICAS column starts increasing from 1 toward the maximum of 5.

After the load test finishes, CPU usage drops. The HPA takes about 5 minutes to scale back down (Kubernetes uses a cooldown period to avoid thrashing).

HPA not scaling up?

Check that the metrics-server addon is enabled by running minikube addons list and looking for metrics-server: enabled. Without it, the HPA has no CPU data to act on.

Also verify your Worker Deployment has resources.requests.cpu set. The HPA calculates utilization as a percentage of the requested CPU. If no request is defined, it cannot compute a percentage.

Help me figure out why HPA is not scaling my Worker pods

Explore Grafana dashboards and verify alerts

The load test generated real traffic and your custom metrics are now flowing into Prometheus. Time to open Grafana and see what the data looks like.

Open a new terminal. Port-forward Grafana by running:

kubectl port-forward -n monitoring svc/prometheus-stack-grafana 3002:80

Why port 3002?

Grafana's Kubernetes Service listens on port 80 internally. The API is already forwarded to local port 3000, so you use 3002 to avoid conflicts.

Open your browser and go to http://localhost:3002.
Log in with username admin and password prom-operator.

Where do these credentials come from?

The kube-prometheus-stack Helm chart sets the default Grafana admin password to prom-operator. In a production environment, you would override this with a Kubernetes Secret.

Grafana comes with preconfigured Kubernetes dashboards. You can explore them by clicking the hamburger menu on the left, then Dashboards. Look for dashboards like Kubernetes / Compute Resources / Namespace (Pods) to see CPU and memory usage for the orders namespace.

The real power is querying your custom application metrics. Let's do that now.

In the left sidebar, click the compass icon to open Explore.
Make sure Prometheus is selected as the data source at the top.
In the query field, type orders_created_total and click Run query.

You should see a time series showing the total orders created, broken down by the status label (queued and possibly duplicate).

Try these additional queries one at a time:

Now check whether your alert rules were picked up.

In the left sidebar, click the bell icon to open Alerting.
Click Alert rules.
Search for HighOrderFailureRate or DeadLetterQueueNotEmpty. You should see both rules listed.

If the load test generated enough failures, the DeadLetterQueueNotEmpty alert may already be firing. The Worker has a small random failure chance on each processing step, so some orders likely exhausted their 5 retries and landed in the dead-letter queue.

To guarantee a DLQ alert fires, send another batch of orders by running the load test again from your project folder:

API_KEY=production-api-key-secret node loadtest/loadtest.js

After the test completes, wait about 1 minute and refresh the Alert rules page in Grafana. The DeadLetterQueueNotEmpty alert should show as Firing (yellow or red state).

Alerts not showing up in Grafana?

The release: prometheus-stack label on your PrometheusRule must match the Helm release name exactly. If you used a different name during helm install, update the label in k8s/prometheus-rules.yaml and re-apply with kubectl apply -f k8s/prometheus-rules.yaml.

Also check that your ServiceMonitors are scraping correctly. In Grafana Explore, query up{namespace="orders"}. You should see entries for your API and Worker. If they show 0, the scrape is failing.

Help me debug why my PrometheusRule alerts are not showing in Grafana

Your pipeline is now fully observable. You have metrics flowing from both services into Prometheus, dashboards in Grafana to visualize them, and alert rules that fire when things go wrong. In the Secret Mission, you will build a production operations dashboard that ties all of these metrics into a single view.

Secret mission

Build a Production Operations Dashboard in Grafana

You explored individual metrics by typing ad-hoc queries in Grafana's Explore tab. Real SRE teams go further. They build dashboards that combine multiple metrics into a single view so they can spot problems the moment they appear. In this mission, you'll create a 6-panel production operations dashboard from scratch that monitors your entire order processing pipeline in real time.

Before you go

Clean Up Your Resources

This project runs entirely on your local machine using Minikube, so there are no ongoing cloud costs. The Minikube cluster and Docker containers do consume system resources like RAM and CPU, so decide whether to keep them running, pause them, or delete them entirely.

Resources you used:

Minikube cluster with the orders namespace and all Kubernetes resources (Deployments, Services, Secrets, HPA, ServiceMonitors, PrometheusRules).
Docker images: order-api:v1 and order-worker:v1 built inside Minikube.
Prometheus and Grafana monitoring stack deployed via Helm (kube-prometheus-stack).
Local project files in the order-pipeline folder on your Desktop (api/, worker/, loadtest/, k8s/, docker-compose.yml).

Keep everything running

No action needed. Choose this if you are still actively building or want to keep experimenting with the monitoring stack and load tests.

Your Minikube cluster, all Kubernetes resources, and the Prometheus/Grafana stack persist across terminal sessions.
You can re-access Grafana at any time by port-forwarding again.
Minikube uses around 6 GB of RAM and 3 CPU cores while running.

Pause - I'll come back to this later

Stop the Minikube cluster to free up memory and CPU. Your cluster state is saved, so you can pick up right where you left off.

Stop Minikube by running:

minikube stop

When you are ready to come back, restart Minikube by running:

minikube start

All your Deployments, Services, and monitoring resources will still be there.

Delete - I don't want to use this again

Remove the Minikube cluster and all associated resources completely. This frees all system resources and cleans up Docker artifacts.

Delete the Minikube cluster by running:

minikube delete

Clean up unused Docker images and build cache by running:

docker system prune

Type y and press Enter when Docker asks for confirmation.
Optionally, delete the project folder from your Desktop if you no longer need the source code.

Mission accomplished

That's a Wrap!

Nice work. You just built a production-grade order processing pipeline from the ground up, complete with reliability, security, autoscaling, and full observability.

Here's what you learned:

Built two Node.js microservices communicating through Redis and BullMQ, with production reliability features like retries with exponential backoff, a dead-letter queue, idempotency keys, API key authentication, and rate limiting.
Containerized both services with Docker, deployed them to Kubernetes with liveness and readiness probes, a rolling update strategy, and a HorizontalPodAutoscaler that scales Worker pods based on CPU utilization.
Deployed a Prometheus + Grafana monitoring stack with custom application metrics, ServiceMonitor resources, PrometheusRule alerts, and a load test that proved the system handles traffic spikes.
Secret Mission: Built a production operations dashboard in Grafana with panels for order throughput, p95 latency, failure rates, queue depth, and replica count.

Ready to quiz yourself?