Event-Driven Order Processing System

Build a polyglot microservices system with RabbitMQ, TypeScript, and Python.

Introduction

30 Second Summary

Every time you place an order online, dozens of independent services spring into action behind the scenes. Your payment gets processed, inventory gets checked, and a confirmation email arrives, all triggered by one event: the order being placed.

In this project, you will build a local event-driven order processing system where a TypeScript API and a Python service communicate through RabbitMQ, a message broker that routes events between services asynchronously.

What You'll Build

A polyglot microservices system running locally where placing an order triggers an asynchronous inventory check across service boundaries, orchestrated by Docker Compose.

By the end of this project, you'll have:

  • A TypeScript Order API that accepts HTTP orders and publishes events to RabbitMQ, plus a Python Inventory Service that consumes those events and publishes confirmation or rejection results.
  • A Docker Compose configuration that orchestrates all services with health checks, dependency ordering, and restart policies.
  • A dead-letter queue setup for handling failed messages, with observable message flow through the RabbitMQ Management UI.
  • Secret Mission: Add a Notification Service that demonstrates fan-out messaging, where multiple consumers independently receive copies of the same event.

Are there any prerequisites?

This project assumes you have Docker, Node.js, and Python already installed on your machine. You should also be comfortable with basic TypeScript and Python syntax.

Set Up the Project Structure and Docker Compose

Every service in your event-driven system needs a way to send and receive messages without knowing about the other services directly. A message broker sits in the middle and handles routing, queuing, and delivery guarantees.

In this step, you'll configure Docker Compose to run RabbitMQ as your message broker. Docker Compose lets you define your entire multi-service system in a single file and start everything with one command.

In this step, get ready to:
  • Create the compose.yaml file with the RabbitMQ service configuration.
  • Start RabbitMQ using Docker Compose.
  • Verify the Management UI is accessible in your browser.
Write the Docker Compose configuration

Docker Compose uses a YAML file to define the services, networks, and volumes for your application. You'll start with just the RabbitMQ service and add more services in later steps.

  • In VS Code, create a new file called compose.yaml in the root of your project folder (at the same level as your order-api and inventory-service folders).
  • Paste the following configuration into compose.yaml:
# Define the services for the event-driven system
services:
  rabbitmq:
    # Official RabbitMQ image with management plugin pre-installed
    image: rabbitmq:4.3-management
    ports:
      - "5672:5672"    # AMQP protocol port
      - "15672:15672"  # Management UI port
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: password
    healthcheck:
      # Checks that all listener ports accept TCP connections
      test: rabbitmq-diagnostics -q check_port_connectivity
      interval: 5s
      timeout: 10s
      retries: 5
      start_period: 15s

What does this configuration do?

This file tells Docker Compose how to run your RabbitMQ service:

  • image: rabbitmq:4.3-management pulls the official RabbitMQ image with the Management UI plugin already installed.
  • Port 5672 is the AMQP protocol port where your services will send and receive messages.
  • Port 15672 exposes a web dashboard for monitoring queues, exchanges, and message flow.
  • The healthcheck block tells Docker Compose how to know when RabbitMQ is truly ready to accept connections.

Why use a health check?

RabbitMQ takes several seconds to boot up internally even after the container starts. The rabbitmq-diagnostics -q check_port_connectivity command verifies that all listener ports are accepting TCP connections.

In later steps, your Order API and Inventory Service will use depends_on: condition: service_healthy to wait for this check to pass before starting. This prevents connection failures during startup.

  • Save compose.yaml.

YAML indentation error?

YAML is whitespace-sensitive. Make sure you use spaces (not tabs) for indentation, and that each nested level is indented by exactly 2 spaces from its parent.

Help me fix my YAML indentation

✔️ Awesome, I've got everything!

Great. Double check you've saved compose.yaml and move on to starting RabbitMQ.

ⓧ I'd like to double check the full code

Here's the complete compose.yaml at this point:

services:
  rabbitmq:
    image: rabbitmq:4.3-management
    ports:
      - "5672:5672"
      - "15672:15672"
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: password
    healthcheck:
      test: rabbitmq-diagnostics -q check_port_connectivity
      interval: 5s
      timeout: 10s
      retries: 5
      start_period: 15s
Start RabbitMQ and verify the Management UI

With the configuration written, you can now start RabbitMQ in a container. Docker Compose will pull the image, create the container, and run the health check automatically.

  • Open a terminal in VS Code by pressing Ctrl+` (macOS and Windows).
  • Start only the RabbitMQ service by running this command:
docker compose up rabbitmq

Why start only RabbitMQ?

You haven't built the Order API or Inventory Service yet. Starting just rabbitmq lets you confirm the broker is healthy before adding services that depend on it.

  • Wait for the logs to show that RabbitMQ has started successfully. You should see output indicating that the broker is accepting AMQP connections on port 5672.
  • Open your browser and navigate to http://localhost:15672.
  • Log in with username admin and password password.

You should see the RabbitMQ Management overview dashboard. It shows node status, message rates, and connection counts.

Can't reach localhost:15672?

If the page doesn't load, the RabbitMQ container might still be starting up. The start_period: 15s in your health check gives RabbitMQ 15 seconds before Docker even begins checking. Wait about 30 seconds and try again.

If you see an error in the terminal logs mentioning port conflicts, make sure nothing else is already using port 5672 or 15672 on your machine.

Help me troubleshoot RabbitMQ connection issues

Your message broker is running and healthy. Next up, you'll build the Order API service that publishes order events to RabbitMQ.

Build the Order API Service

With RabbitMQ running and its Management UI accessible, you have a message broker ready to route messages between services. Now you need a service that actually sends messages to it.

The Order API is your system's front door. It accepts HTTP requests and converts them into AMQP messages that RabbitMQ can deliver to downstream services. You'll build it with Express for the HTTP layer and amqplib for communicating with RabbitMQ.

In this step, get ready to:
  • Initialize a TypeScript project with Express and amqplib.
  • Implement a publisher that connects to RabbitMQ with a retry loop and sends order events.
  • Containerize the service with Docker and add it to Docker Compose.
Initialize the Order API project

The Order API needs its own directory with a TypeScript configuration and the right dependencies. You'll set up the project structure first, then install packages.

  • Create the order-api directory and its src subfolder by running this command from your project root (the folder containing compose.yaml):
mkdir -p order-api/src
  • Create a new file called package.json inside the order-api folder and paste the following:
{
  "name": "order-api",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "build": "tsc",
    "start": "node dist/index.js"
  },
  "dependencies": {
    "amqplib": "2.0.1",
    "express": "4.21.2"
  },
  "devDependencies": {
    "@types/express": "5.0.2",
    "@types/node": "22.15.0",
    "typescript": "5.8.3"
  }
}

What does this configuration do?

  • "type": "module" enables ES module syntax (import/export) instead of CommonJS require().
  • amqplib 2.0.1 is the AMQP client for RabbitMQ. It bundles TypeScript types, so no separate @types/amqplib package is needed.
  • express 4.21.2 provides the HTTP server. @types/express adds TypeScript type definitions for it.
  • typescript is a devDependency because the compiled JavaScript is what runs in production.
  • Create another new file called tsconfig.json inside the order-api folder and paste the following:
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "Node16",
    "moduleResolution": "Node16",
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}

What does tsconfig.json control?

This tells TypeScript to compile files from ./src into ./dist as ES2022 JavaScript. The Node16 module resolution matches Node.js native ES module support.

  • Install the dependencies by navigating into order-api and running:
cd order-api
npm install
  • Confirm the install succeeded by checking that a node_modules folder appeared by running:
ls

You should see node_modules, package.json, package-lock.json, tsconfig.json, and src in the output.

Seeing npm errors?

Make sure you're inside the order-api directory, not the project root. Run pwd to check your current location.

Help me troubleshoot npm install

Implement the Express server and RabbitMQ publisher

This is the core of the Order API. The service connects to RabbitMQ on startup, then publishes an event every time someone places an order via HTTP.

  • Create a new file called index.ts inside order-api/src and paste the following connection setup code:
import express from "express";
import amqplib from "amqplib";

const app = express();
app.use(express.json());

// Connection URL from environment variable, with local fallback
const RABBITMQ_URL = process.env.RABBITMQ_URL || "amqp://admin:password@localhost:5672";
const EXCHANGE_ORDERS = "orders";
const EXCHANGE_RESULTS = "order-results";

// Channel reference shared across request handlers
let channel: amqplib.Channel | null = null;

// Retry loop handles the case where RabbitMQ is still booting
async function connectWithRetry(maxRetries = 5, delayMs = 3000): Promise<amqplib.Channel> {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const conn = await amqplib.connect(RABBITMQ_URL);
      const ch = await conn.createChannel();
      await ch.assertExchange(EXCHANGE_ORDERS, "direct", { durable: true });
      await ch.assertExchange(EXCHANGE_RESULTS, "fanout", { durable: true });
      console.log("Connected to RabbitMQ");
      return ch;
    } catch (err) {
      console.log(`RabbitMQ connection attempt ${attempt}/${maxRetries} failed. Retrying in ${delayMs}ms...`);
      if (attempt === maxRetries) throw err;
      await new Promise((resolve) => setTimeout(resolve, delayMs));
    }
  }
  throw new Error("Failed to connect to RabbitMQ");
}

What does this code do?

  • RABBITMQ_URL reads the connection string from an environment variable. Docker Compose will set this to use the rabbitmq service name as the hostname.
  • connectWithRetry attempts to connect up to 5 times with a 3-second delay between attempts. Even with Docker's service_healthy condition, there can be a brief window before RabbitMQ accepts AMQP connections.
  • assertExchange creates the exchanges if they don't exist. A direct exchange routes messages by exact routing key match. A fanout exchange broadcasts to all bound queues.
  • Add the POST /orders endpoint below the connectWithRetry function in the same file:
app.post("/orders", async (req, res) => {
  const { productId, quantity } = req.body;

  // Validate required fields
  if (!productId || !quantity) {
    res.status(400).json({ error: "productId and quantity are required" });
    return;
  }

  // Build the order event payload
  const order = {
    orderId: `ORD-${Date.now()}`,
    productId,
    quantity,
    timestamp: new Date().toISOString(),
  };

  // Reject if broker connection is down
  if (!channel) {
    res.status(503).json({ error: "Message broker not available" });
    return;
  }

  // Publish the order event to the orders exchange
  channel.publish(EXCHANGE_ORDERS, "order.placed", Buffer.from(JSON.stringify(order)), {
    contentType: "application/json",
    persistent: true,
  });

  console.log(`Order published: ${order.orderId}`);
  res.status(202).json({ message: "Order accepted", orderId: order.orderId });
});

What does the POST handler do?

  • It validates the request body, then builds an order object with a unique ID and timestamp.
  • channel.publish() sends the order as a JSON message to the orders exchange with routing key order.placed. Any queue bound to that key receives a copy.
  • persistent: true tells RabbitMQ to write the message to disk, so it survives a broker restart.
  • The 202 response tells the client the order was accepted for processing, not that it's complete. This is the async pattern in action.
  • Add the health check endpoint and server startup at the bottom of the file:
app.get("/health", (_req, res) => {
  res.json({ status: channel ? "healthy" : "degraded" });
});

// Start the server after connecting to RabbitMQ
async function main() {
  channel = await connectWithRetry();
  app.listen(3000, () => {
    console.log("Order API listening on port 3000");
  });
}

main().catch((err) => {
  console.error("Failed to start Order API:", err);
  process.exit(1);
});

What does the startup logic do?

  • The /health endpoint reports whether the RabbitMQ channel is connected. Orchestrators use this to monitor service status.
  • main() connects to RabbitMQ first, then starts listening for HTTP requests. If the connection fails after all retries, the process exits with code 1 so Docker knows to restart it.
  • Save order-api/src/index.ts.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

Here is the complete order-api/src/index.ts file:

import express from "express";
import amqplib from "amqplib";

const app = express();
app.use(express.json());

const RABBITMQ_URL = process.env.RABBITMQ_URL || "amqp://admin:password@localhost:5672";
const EXCHANGE_ORDERS = "orders";
const EXCHANGE_RESULTS = "order-results";

let channel: amqplib.Channel | null = null;

async function connectWithRetry(maxRetries = 5, delayMs = 3000): Promise<amqplib.Channel> {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const conn = await amqplib.connect(RABBITMQ_URL);
      const ch = await conn.createChannel();
      await ch.assertExchange(EXCHANGE_ORDERS, "direct", { durable: true });
      await ch.assertExchange(EXCHANGE_RESULTS, "fanout", { durable: true });
      console.log("Connected to RabbitMQ");
      return ch;
    } catch (err) {
      console.log(`RabbitMQ connection attempt ${attempt}/${maxRetries} failed. Retrying in ${delayMs}ms...`);
      if (attempt === maxRetries) throw err;
      await new Promise((resolve) => setTimeout(resolve, delayMs));
    }
  }
  throw new Error("Failed to connect to RabbitMQ");
}

app.post("/orders", async (req, res) => {
  const { productId, quantity } = req.body;

  if (!productId || !quantity) {
    res.status(400).json({ error: "productId and quantity are required" });
    return;
  }

  const order = {
    orderId: `ORD-${Date.now()}`,
    productId,
    quantity,
    timestamp: new Date().toISOString(),
  };

  if (!channel) {
    res.status(503).json({ error: "Message broker not available" });
    return;
  }

  channel.publish(EXCHANGE_ORDERS, "order.placed", Buffer.from(JSON.stringify(order)), {
    contentType: "application/json",
    persistent: true,
  });

  console.log(`Order published: ${order.orderId}`);
  res.status(202).json({ message: "Order accepted", orderId: order.orderId });
});

app.get("/health", (_req, res) => {
  res.json({ status: channel ? "healthy" : "degraded" });
});

async function main() {
  channel = await connectWithRetry();
  app.listen(3000, () => {
    console.log("Order API listening on port 3000");
  });
}

main().catch((err) => {
  console.error("Failed to start Order API:", err);
  process.exit(1);
});
Containerize and add to Docker Compose

The Order API needs its own Docker image so Docker Compose can manage its lifecycle alongside RabbitMQ. You'll use a multi-stage build to keep the final image small.

  • Create a new file called Dockerfile inside the order-api folder and paste the following:
FROM node:22-alpine AS builder
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm install
COPY tsconfig.json ./
COPY src ./src
RUN npm run build

FROM node:22-alpine
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm install --omit=dev
COPY --from=builder /app/dist ./dist
EXPOSE 3000
CMD ["node", "dist/index.js"]

What does this Dockerfile do?

This is a multi-stage build with two stages:

  • The builder stage installs all dependencies (including TypeScript), compiles the code, then gets discarded.
  • The production stage only copies the compiled JavaScript and production dependencies. No TypeScript compiler or dev tools ship in the final image.
  • Save order-api/Dockerfile.

Now you need to tell Docker Compose about this new service.

  • Return to compose.yaml in your project root. Add the order-api service below the rabbitmq service definition:
  order-api:
    build: ./order-api
    ports:
      - "3000:3000"
    environment:
      RABBITMQ_URL: amqp://admin:password@rabbitmq:5672
    depends_on:
      rabbitmq:
        condition: service_healthy
    restart: on-failure:3

What does this service configuration do?

  • build: ./order-api tells Docker Compose to build the image from the Dockerfile in that folder.
  • RABBITMQ_URL uses rabbitmq as the hostname. Docker Compose networking lets services resolve each other by service name.
  • depends_on with condition: service_healthy means Docker won't start the Order API until RabbitMQ's health check passes.
  • restart: on-failure:3 tells Docker to restart the container up to 3 times if it crashes. This prevents infinite crash loops.
  • Save compose.yaml.

✔️ Awesome, I've got everything!

Great. Double check that the order-api block is indented at the same level as rabbitmq under services:.

ⓧ I'd like to double check the full code

Here is the complete compose.yaml file:

services:
  rabbitmq:
    image: rabbitmq:4.3-management
    ports:
      - "5672:5672"
      - "15672:15672"
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: password
    healthcheck:
      test: rabbitmq-diagnostics -q check_port_connectivity
      interval: 5s
      timeout: 10s
      retries: 5
      start_period: 15s

  order-api:
    build: ./order-api
    ports:
      - "3000:3000"
    environment:
      RABBITMQ_URL: amqp://admin:password@rabbitmq:5672
    depends_on:
      rabbitmq:
        condition: service_healthy
    restart: on-failure:3

Time to build and run everything together.

  • Navigate back to the project root (the directory containing compose.yaml) by running:
cd ..
  • Build and start all services by running:
docker compose up --build

What does --build do?

The --build flag forces Docker Compose to rebuild images before starting containers. Without it, Compose reuses cached images and won't pick up your code changes.

Wait until you see Order API listening on port 3000 and Connected to RabbitMQ in the logs.

  • Open a second terminal window and send a test order by running:
curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -d '{"productId": "WIDGET-1", "quantity": 2}'

You should see a response like:

{"message":"Order accepted","orderId":"ORD-1719849600000"}
  • Open the RabbitMQ Management UI at http://localhost:15672 and log in with admin / password.
  • Click the Exchanges tab.
  • You should see the orders and order-results exchanges listed.

Order API not starting?

  • Check the Docker Compose logs for build errors. A missing semicolon or import typo in index.ts causes TypeScript compilation to fail during the Docker build.
  • Make sure your Dockerfile is in the order-api folder, not the project root.
  • If you see connection refused errors, the retry loop should handle them. If it exhausts all retries, confirm RabbitMQ's health check is passing by running docker compose ps and checking the status column.

Help me debug my Order API startup

Your Order API is live and publishing events to RabbitMQ. Next up, you'll build the Inventory Service in Python that consumes these order events and decides whether to confirm or reject them.

Build the Inventory Service

Your Order API is publishing events into RabbitMQ, but nothing is listening on the other end. Messages are flowing into the orders exchange with nowhere to go.

In this step, you'll build the Inventory Service in Python. It will consume order events, check stock availability, and publish confirmation or rejection results back through RabbitMQ.

In this step, get ready to:
  • Create the Inventory Service consumer using pika.
  • Implement inventory check logic with message acknowledgments.
  • Containerize the service and verify end-to-end message flow.
Create the Inventory Service consumer

The Inventory Service is a Python application that subscribes to the orders exchange and processes messages as they arrive. It uses pika, the canonical Python client for the AMQP protocol.

  • Create the inventory-service directory and its requirements.txt file by running these commands:
mkdir inventory-service
cd inventory-service
echo "pika==1.4.1" > requirements.txt
cd ..
  • Confirm the directory was created by running:
ls inventory-service

You should see requirements.txt in the output.

Directory not found?

Make sure you're running the command from the root of your event-driven-orders project folder (the same level as your compose.yaml).

Help me find my project root directory.

  • Create a new file called main.py inside inventory-service/ and add the imports, configuration constants, and connection logic:
import json
import os
import time

import pika

RABBITMQ_URL = os.environ.get("RABBITMQ_URL", "amqp://admin:password@localhost:5672")
EXCHANGE_ORDERS = "orders"
EXCHANGE_RESULTS = "order-results"
QUEUE_NAME = "inventory-check"
DLX_EXCHANGE = "orders-dlx"
DLQ_NAME = "inventory-check-dlq"

# Hardcoded stock levels for demonstration
STOCK = {
    "WIDGET-1": 100,
    "WIDGET-2": 50,
    "WIDGET-3": 0,
}


def connect_with_retry(max_retries=5, delay=3):
    for attempt in range(1, max_retries + 1):
        try:
            params = pika.URLParameters(RABBITMQ_URL)
            connection = pika.BlockingConnection(params)
            print("Connected to RabbitMQ")
            return connection
        except pika.exceptions.AMQPConnectionError:
            print(f"Connection attempt {attempt}/{max_retries} failed. Retrying in {delay}s...")
            if attempt == max_retries:
                raise
            time.sleep(delay)

What does this code do?

  • RABBITMQ_URL reads the connection string from an environment variable, falling back to localhost for local development.
  • The constants (EXCHANGE_ORDERS, QUEUE_NAME, etc.) define the AMQP topology this service participates in.
  • STOCK is a simple dictionary simulating an inventory database. WIDGET-3 has zero stock, which lets you test rejection later.
  • connect_with_retry handles the startup race condition where the service boots before RabbitMQ finishes initializing.

You won't see a change yet. The next chunk sets up the exchange and queue bindings.

  • Below the connect_with_retry function in main.py, add the function that declares all exchanges and queues:
def setup_exchanges_and_queues(channel):
    # Declare the exchanges this service interacts with
    channel.exchange_declare(exchange=EXCHANGE_ORDERS, exchange_type="direct", durable=True)
    channel.exchange_declare(exchange=EXCHANGE_RESULTS, exchange_type="fanout", durable=True)
    channel.exchange_declare(exchange=DLX_EXCHANGE, exchange_type="fanout", durable=True)

    # Dead-letter queue catches messages that fail processing
    channel.queue_declare(queue=DLQ_NAME, durable=True)
    channel.queue_bind(queue=DLQ_NAME, exchange=DLX_EXCHANGE)

    # Main queue with dead-letter routing configured
    channel.queue_declare(
        queue=QUEUE_NAME,
        durable=True,
        arguments={"x-dead-letter-exchange": DLX_EXCHANGE},
    )
    channel.queue_bind(queue=QUEUE_NAME, exchange=EXCHANGE_ORDERS, routing_key="order.placed")

What does this code do?

  • It declares three exchanges: orders (direct) for incoming order events, order-results (fanout) for publishing results, and orders-dlx (fanout) as the dead-letter exchange.
  • The inventory-check queue binds to the orders exchange with routing key order.placed. Only messages published with that key will arrive here.
  • The x-dead-letter-exchange argument tells RabbitMQ to route any nacked (rejected) messages to the dead-letter exchange automatically.

Why declare exchanges on both sides?

Both the Order API and the Inventory Service declare the same exchanges. In AMQP, declarations are idempotent. If an exchange already exists with matching settings, the declaration simply succeeds. This means services can start in any order without failing.

You won't see a change yet. The next chunk adds the message processing logic.

Implement the inventory check logic

When a message arrives, the service needs to parse the order, check stock, and publish a result. It must also decide whether to acknowledge the message (removing it from the queue) or reject it (routing it to the dead-letter queue).

  • Below setup_exchanges_and_queues in main.py, add the process_order callback function:
def process_order(ch, method, properties, body):
    try:
        order = json.loads(body)
        product_id = order["productId"]
        quantity = order["quantity"]
        order_id = order["orderId"]

        print(f"Processing order {order_id}: {quantity}x {product_id}")

        available = STOCK.get(product_id, -1)

        # Unknown product - reject and route to dead-letter queue
        if available == -1:
            print(f"Unknown product {product_id} - sending to DLQ")
            ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)
            return

        # Check if enough stock is available
        if available >= quantity:
            result = {
                "orderId": order_id,
                "status": "confirmed",
                "productId": product_id,
                "quantity": quantity,
            }
            STOCK[product_id] -= quantity
            print(f"Order {order_id} CONFIRMED. Remaining stock: {STOCK[product_id]}")
        else:
            result = {
                "orderId": order_id,
                "status": "rejected",
                "productId": product_id,
                "reason": f"Insufficient stock. Available: {available}, Requested: {quantity}",
            }
            print(f"Order {order_id} REJECTED. Insufficient stock.")

        # Publish the result to the order-results exchange
        ch.basic_publish(
            exchange=EXCHANGE_RESULTS,
            routing_key="",
            body=json.dumps(result),
            properties=pika.BasicProperties(content_type="application/json", delivery_mode=2),
        )
        ch.basic_ack(delivery_tag=method.delivery_tag)

    except Exception as e:
        print(f"Error processing message: {e}")
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)

What does this code do?

  • It parses the incoming JSON message to extract productId, quantity, and orderId.
  • If the product is unknown (not in STOCK), it calls basic_nack with requeue=False. This tells RabbitMQ to not retry the message but instead route it to the dead-letter exchange.
  • For valid products, it checks stock and publishes a confirmation or rejection result to the order-results fanout exchange.
  • basic_ack is called only AFTER successful processing. This guarantees that if the service crashes mid-processing, the message returns to the queue for redelivery.

You won't see a change yet. The next chunk adds the main entry point that ties everything together.

  • At the bottom of main.py, add the main function and the entry point:
def main():
    connection = connect_with_retry()
    channel = connection.channel()
    # Process one message at a time for fair distribution
    channel.basic_qos(prefetch_count=1)

    setup_exchanges_and_queues(channel)

    channel.basic_consume(queue=QUEUE_NAME, on_message_callback=process_order)

    print("Inventory Service waiting for orders...")
    channel.start_consuming()


if __name__ == "__main__":
    main()

What does this code do?

  • basic_qos(prefetch_count=1) tells RabbitMQ to send only one unacknowledged message at a time. This prevents a single consumer from hoarding messages.
  • basic_consume registers the process_order callback on the inventory-check queue.
  • start_consuming() blocks the main thread forever, waiting for messages. The service runs until stopped.
  • Save main.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

import json
import os
import time

import pika

RABBITMQ_URL = os.environ.get("RABBITMQ_URL", "amqp://admin:password@localhost:5672")
EXCHANGE_ORDERS = "orders"
EXCHANGE_RESULTS = "order-results"
QUEUE_NAME = "inventory-check"
DLX_EXCHANGE = "orders-dlx"
DLQ_NAME = "inventory-check-dlq"

STOCK = {
    "WIDGET-1": 100,
    "WIDGET-2": 50,
    "WIDGET-3": 0,
}


def connect_with_retry(max_retries=5, delay=3):
    for attempt in range(1, max_retries + 1):
        try:
            params = pika.URLParameters(RABBITMQ_URL)
            connection = pika.BlockingConnection(params)
            print("Connected to RabbitMQ")
            return connection
        except pika.exceptions.AMQPConnectionError:
            print(f"Connection attempt {attempt}/{max_retries} failed. Retrying in {delay}s...")
            if attempt == max_retries:
                raise
            time.sleep(delay)


def setup_exchanges_and_queues(channel):
    channel.exchange_declare(exchange=EXCHANGE_ORDERS, exchange_type="direct", durable=True)
    channel.exchange_declare(exchange=EXCHANGE_RESULTS, exchange_type="fanout", durable=True)
    channel.exchange_declare(exchange=DLX_EXCHANGE, exchange_type="fanout", durable=True)

    channel.queue_declare(queue=DLQ_NAME, durable=True)
    channel.queue_bind(queue=DLQ_NAME, exchange=DLX_EXCHANGE)

    channel.queue_declare(
        queue=QUEUE_NAME,
        durable=True,
        arguments={"x-dead-letter-exchange": DLX_EXCHANGE},
    )
    channel.queue_bind(queue=QUEUE_NAME, exchange=EXCHANGE_ORDERS, routing_key="order.placed")


def process_order(ch, method, properties, body):
    try:
        order = json.loads(body)
        product_id = order["productId"]
        quantity = order["quantity"]
        order_id = order["orderId"]

        print(f"Processing order {order_id}: {quantity}x {product_id}")

        available = STOCK.get(product_id, -1)

        if available == -1:
            print(f"Unknown product {product_id} - sending to DLQ")
            ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)
            return

        if available >= quantity:
            result = {
                "orderId": order_id,
                "status": "confirmed",
                "productId": product_id,
                "quantity": quantity,
            }
            STOCK[product_id] -= quantity
            print(f"Order {order_id} CONFIRMED. Remaining stock: {STOCK[product_id]}")
        else:
            result = {
                "orderId": order_id,
                "status": "rejected",
                "productId": product_id,
                "reason": f"Insufficient stock. Available: {available}, Requested: {quantity}",
            }
            print(f"Order {order_id} REJECTED. Insufficient stock.")

        ch.basic_publish(
            exchange=EXCHANGE_RESULTS,
            routing_key="",
            body=json.dumps(result),
            properties=pika.BasicProperties(content_type="application/json", delivery_mode=2),
        )
        ch.basic_ack(delivery_tag=method.delivery_tag)

    except Exception as e:
        print(f"Error processing message: {e}")
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)


def main():
    connection = connect_with_retry()
    channel = connection.channel()
    channel.basic_qos(prefetch_count=1)

    setup_exchanges_and_queues(channel)

    channel.basic_consume(queue=QUEUE_NAME, on_message_callback=process_order)

    print("Inventory Service waiting for orders...")
    channel.start_consuming()


if __name__ == "__main__":
    main()
Containerize and add to Docker Compose

The Inventory Service needs its own Docker image so Docker Compose can orchestrate it alongside RabbitMQ and the Order API.

  • Create a file called Dockerfile inside the inventory-service/ directory with this content:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py ./
CMD ["python", "main.py"]

What does this Dockerfile do?

  • It starts from python:3.12-slim, a lightweight Python image.
  • It copies requirements.txt first and installs dependencies. This layer gets cached, so rebuilds are fast when only main.py changes.
  • CMD runs the service when the container starts.
  • Save the Dockerfile.

You won't see a change yet. The next chunk registers this service in Docker Compose.

  • Open compose.yaml in the root of your project.
  • Find the order-api service block. Below it (after its restart: on-failure:3 line), add the new inventory-service definition:
  inventory-service:
    build: ./inventory-service
    environment:
      RABBITMQ_URL: amqp://admin:password@rabbitmq:5672
    depends_on:
      rabbitmq:
        condition: service_healthy
    restart: on-failure:3

What does this configuration do?

  • build: ./inventory-service tells Docker Compose to build the image from the Dockerfile in that directory.
  • depends_on with condition: service_healthy ensures the Inventory Service only starts once RabbitMQ's health check passes.
  • restart: on-failure:3 gives the service three restart attempts if it crashes, then stops trying.
  • Save compose.yaml.

✔️ Awesome, I've got everything!

Great. Make sure you've saved both the Dockerfile and compose.yaml.

ⓧ I'd like to double check the full code

services:
  rabbitmq:
    image: rabbitmq:4.3-management
    ports:
      - "5672:5672"
      - "15672:15672"
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: password
    healthcheck:
      test: rabbitmq-diagnostics -q check_port_connectivity
      interval: 5s
      timeout: 10s
      retries: 5
      start_period: 15s

  order-api:
    build: ./order-api
    ports:
      - "3000:3000"
    environment:
      RABBITMQ_URL: amqp://admin:password@rabbitmq:5672
    depends_on:
      rabbitmq:
        condition: service_healthy
    restart: on-failure:3

  inventory-service:
    build: ./inventory-service
    environment:
      RABBITMQ_URL: amqp://admin:password@rabbitmq:5672
    depends_on:
      rabbitmq:
        condition: service_healthy
    restart: on-failure:3
  • If you still have the previous docker compose up running, stop it with Ctrl+C.
  • Rebuild and start the full system by running:
docker compose up --build

You should see all three services start in sequence. Look for the log line Inventory Service waiting for orders... confirming the consumer is ready.

  • Open a new terminal window and test with an in-stock order by running:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"productId": "WIDGET-1", "quantity": 5}'

In the Docker Compose logs, you should see the Inventory Service print Order ORD-... CONFIRMED. Remaining stock: 95.

  • Test with an out-of-stock order by running:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"productId": "WIDGET-3", "quantity": 1}'

You should see Order ORD-... REJECTED. Insufficient stock. in the logs.

  • Open the RabbitMQ Management UI at http://localhost:15672 and navigate to the Exchanges tab.
  • Confirm you see the order-results exchange listed with recent message activity.

Inventory Service keeps restarting?

  • Check the logs for a syntax error in main.py. Python will show the exact line number.
  • Make sure requirements.txt contains exactly pika==1.4.1 with no typos.
  • Verify the RABBITMQ_URL in compose.yaml uses the service name rabbitmq (not localhost) as the hostname.

Help me debug why my Inventory Service container keeps restarting.

Your event-driven system is fully operational. Messages flow from the Order API through RabbitMQ to the Inventory Service and back. Next up, you'll explore what happens when messages fail processing and how the dead-letter queue captures them.

Add the Dead-Letter Queue and Observe Failure Handling

Your system can now process orders and reject them when stock is insufficient. But what happens when a message simply cannot be processed at all?

In production, messages fail for many reasons: invalid data, unknown product IDs, or unexpected exceptions. Silently dropping these messages means losing data with no way to investigate. A dead-letter queue gives failed messages a designated parking spot where they can be inspected, retried, or routed to an alerting system. Your Inventory Service already has this pattern built in. In this step, you'll trigger failures and watch messages route to the dead-letter queue in real time.

In this step, get ready to:
  • Review the dead-letter exchange and queue configuration already in your code.
  • Trigger a processing failure by ordering an unknown product.
  • Inspect the dead-lettered message in the RabbitMQ Management UI to see routing metadata.
Review the dead-letter queue configuration

Your inventory-service/main.py already contains a complete dead-letter queue configuration inside the setup_exchanges_and_queues function. Let's trace through it to understand how failed messages get routed.

  • In your editor, open inventory-service/main.py and find the setup_exchanges_and_queues function.
  • Locate the dead-letter exchange and queue declarations:
    channel.exchange_declare(exchange=DLX_EXCHANGE, exchange_type="fanout", durable=True)

    channel.queue_declare(queue=DLQ_NAME, durable=True)
    channel.queue_bind(queue=DLQ_NAME, exchange=DLX_EXCHANGE)

    channel.queue_declare(
        queue=QUEUE_NAME,
        durable=True,
        arguments={"x-dead-letter-exchange": DLX_EXCHANGE},
    )
    channel.queue_bind(queue=QUEUE_NAME, exchange=EXCHANGE_ORDERS, routing_key="order.placed")

What does this configuration do?

This sets up a three-part dead-letter routing chain:

  • orders-dlx is the dead-letter exchange. It receives messages that fail processing. It uses fanout type so any queue bound to it gets a copy of every failed message.
  • inventory-check-dlq is the dead-letter queue, bound to orders-dlx. Failed messages land here for inspection or retry.
  • The x-dead-letter-exchange argument on the inventory-check queue tells RabbitMQ where to route messages that are nacked without requeue.
  • Now find the failure handling in the process_order function:
        if available == -1:
            print(f"Unknown product {product_id} - sending to DLQ")
            ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)
            return

How does nack route to the dead-letter queue?

When a consumer calls basic_nack with requeue=False, RabbitMQ checks if the source queue has an x-dead-letter-exchange argument. If it does, the message routes there instead of being discarded. The original message body and headers are preserved, and RabbitMQ adds x-death metadata showing why the message was dead-lettered.

Trigger a processing failure with an unknown product

The STOCK dictionary only contains WIDGET-1, WIDGET-2, and WIDGET-3. Ordering any other product ID triggers the unknown-product path and sends the message to the dead-letter queue.

  • Send an order for a product that does not exist by running this command in a new terminal:
curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -d '{"productId": "UNKNOWN-ITEM", "quantity": 1}'

You should see a response confirming the order was accepted:

{"message":"Order accepted","orderId":"ORD-..."}

Why does the API still return 202 Accepted?

The Order API's job is to accept the order and publish it to the message broker. It does not know whether inventory exists. The Inventory Service discovers the failure asynchronously. This is the core trade-off of event-driven architecture: the producer is decoupled from the consumer's success or failure.

  • Check the Inventory Service logs by running:
docker compose logs inventory-service --tail=5

You should see a log line showing the unknown product being routed to the DLQ:

Processing order ORD-...: 1x UNKNOWN-ITEM
Unknown product UNKNOWN-ITEM - sending to DLQ

Not seeing the log message?

  • Make sure all services are still running. Run docker compose ps and confirm inventory-service shows a running state.
  • If the service restarted, it may take a few seconds to reconnect to RabbitMQ. Wait 10 seconds and retry the curl command.

Help me debug why my Inventory Service isn't processing messages from RabbitMQ.

Inspect the dead-lettered message in the Management UI

The message has been nacked and routed to the dead-letter queue. Now you can inspect it through the RabbitMQ Management UI to see what metadata RabbitMQ attaches.

  • Switch back to the RabbitMQ Management UI in your browser at http://localhost:15672.
  • Click the Queues and Streams tab in the top navigation.
  • Find and click inventory-check-dlq in the queue list. You should see it has 1 message ready.
  • Scroll down to the Get messages section.
  • Set Ack Mode to Nack message requeue true (this lets you peek without removing the message).
  • Click Get Message(s).

You should see the original order JSON in the message payload, along with an x-death header in the properties section.

What are x-death headers?

RabbitMQ automatically attaches x-death headers to dead-lettered messages. These headers contain the original queue name, the exchange the message came from, the reason it was dead-lettered (e.g. rejected), and a timestamp.

This metadata is what makes dead-letter queues useful for debugging. You can trace exactly which queue rejected the message and when it happened.

What would a dead-letter consumer look like in production?

In a production system, you would run a dedicated service that consumes from inventory-check-dlq. It could send an alert to an on-call engineer, write the failed message to a database for manual review, or implement a retry schedule with exponential backoff.

The pattern is the same consumer pattern you already built. The only difference is its purpose: triage and recovery instead of business logic.

  • Send a few more unknown orders to see multiple messages accumulate in the dead-letter queue by running:
curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -d '{"productId": "FAKE-PRODUCT", "quantity": 5}'
  • Return to the inventory-check-dlq queue in the Management UI and refresh the page.
  • Confirm the message count increased. Each failed order lands here with its full payload and x-death metadata intact.

Messages not appearing in the dead-letter queue?

  • Make sure the product ID you sent is not one of WIDGET-1, WIDGET-2, or WIDGET-3. Those are known products and will be processed normally.
  • If you rebuilt your services recently, the queue arguments may have changed. Run docker compose down -v and then docker compose up --build to recreate the queues with fresh arguments.

Help me figure out why messages aren't routing to my dead-letter queue in RabbitMQ.

Your dead-letter queue is catching failed messages and preserving them for inspection. Next up, you'll stress-test the whole system by stopping and restarting services to prove that event-driven architecture survives component failures without losing data.

Orchestrate, Break, and Heal the System

Your event-driven system is fully wired and processing orders end to end. But the real test of a distributed system is not whether it works when everything is healthy. It is whether it survives when things break.

In this step, you will deliberately destroy parts of your system and observe how RabbitMQ buffers messages during consumer downtime and how Docker Compose restart policies prevent infinite crash loops.

In this step, get ready to:
  • Stop the consumer and prove messages survive without being lost.
  • Break the broker and observe restart policy behavior.
  • Observe the full system lifecycle from startup ordering through graceful shutdown.
Stop the consumer and observe message buffering

The core promise of asynchronous messaging is decoupling. Producers should be able to publish events even when consumers are offline. The broker holds messages until a consumer returns to process them.

  • Make sure your full system is running. If it is not, bring it up by running:
docker compose up --build -d
  • Forcefully stop the Inventory Service by running:
docker compose stop inventory-service

The Inventory Service is now offline. RabbitMQ and the Order API are still running.

  • Send several orders to the Order API while the consumer is down by running these commands:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"productId": "WIDGET-1", "quantity": 2}'

curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"productId": "WIDGET-2", "quantity": 5}'

curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"productId": "WIDGET-1", "quantity": 3}'
  • Open the RabbitMQ Management UI at http://localhost:15672 and navigate to the Queues and Streams tab.
  • Look at the inventory-check queue. You should see 3 messages in the Ready column. These messages are safely stored in RabbitMQ, waiting for the consumer to return.

Why didn't the messages disappear?

RabbitMQ persists messages marked as persistent (delivery mode 2) to disk. Your Order API publishes with persistent: true, so even if the broker itself restarted, the messages would survive.

The inventory-check queue is also declared as durable, meaning the queue definition itself persists across broker restarts.

  • Now bring the Inventory Service back online by running:
docker compose start inventory-service
  • Watch the Inventory Service logs to see it drain the message backlog by running:
docker compose logs inventory-service --tail=20

You should see the Inventory Service reconnect and immediately process all three queued orders in rapid succession.

What just happened?

This is the decoupling guarantee of asynchronous messaging in action. The producer (Order API) and consumer (Inventory Service) have no direct knowledge of each other. The broker absorbs the time gap, and no messages are lost.

Not seeing all three messages processed?

Check the Management UI queue page. If messages still show as Ready, the Inventory Service may not have reconnected successfully. Run docker compose restart inventory-service to force a fresh connection.

Help me debug why my Inventory Service isn't draining queued messages after restarting.

Break the broker and observe restart policies

What happens when the broker itself goes down? Without RabbitMQ, the Order API cannot publish messages. Its connection will break, causing the process to crash. This is where Docker Compose restart policies come in.

  • Stop RabbitMQ while the system is running by executing:
docker compose stop rabbitmq
  • Watch the Order API's logs to observe its restart behavior by running:
docker compose logs order-api --tail=30

You should see the Order API crash when it loses the RabbitMQ connection. Docker restarts it up to 3 times (per the restart: on-failure:3 policy). Each restart attempt triggers the connectWithRetry loop, which fails because RabbitMQ is still down. After 3 failed restarts, Docker gives up.

Why not restart forever?

The restart: on-failure:3 policy creates a safety boundary. Without a cap, a service with a persistent configuration error would restart indefinitely. This consumes resources and floods logs with noise, making it harder to diagnose the root cause.

The retry loop inside your code (5 attempts with a 3-second delay) handles transient connection blips. The Docker restart policy handles longer outages. Together they create a two-layer resilience pattern.

  • Bring RabbitMQ back online by running:
docker compose start rabbitmq
  • Wait about 10 seconds for RabbitMQ's health check to pass, then restart the Order API by running:
docker compose restart order-api
  • Confirm the Order API is healthy again by running:
curl http://localhost:3000/health

You should see {"status":"healthy"} in the response. The system has fully recovered from a broker outage.

Seeing "degraded" or connection refused?

RabbitMQ may still be booting. The health check takes up to 15 seconds (start_period in your compose file). Wait 20 seconds after starting RabbitMQ, then restart the Order API again.

Help me figure out why my Order API won't reconnect to RabbitMQ after I restarted both services.

Observe the full system lifecycle

For the final test, you will tear everything down and bring it back up from scratch. This lets you observe Docker Compose's startup ordering, health check sequencing, and graceful shutdown all in one clean run.

  • Shut down the entire system by running:
docker compose down
  • Bring the full system up in attached mode so logs stream to your terminal by running:
docker compose up

Watch the log output carefully. You should observe three distinct phases.

  • RabbitMQ starts first and runs its health check (rabbitmq-diagnostics -q check_port_connectivity) every 5 seconds until it passes.
  • Once healthy, the Order API and Inventory Service start simultaneously (both have depends_on: rabbitmq: condition: service_healthy).
  • Both services connect to RabbitMQ, declare their exchanges and queues, and begin listening.
  • Open a second terminal window and send a test order to confirm end-to-end flow by running:
curl -X POST http://localhost:3000/orders -H "Content-Type: application/json" -d '{"productId": "WIDGET-2", "quantity": 1}'

In your first terminal (where logs are streaming), you should see the Order API publish the event, followed immediately by the Inventory Service consuming and confirming it.

  • Return to your first terminal and press Ctrl+C to initiate graceful shutdown.

Docker Compose sends a SIGTERM to each container and waits for them to exit cleanly. You should see each service stop in the logs.

What does this prove about your system?

You have now observed all three resilience properties of your event-driven architecture. Messages survive consumer downtime because the broker buffers them. Services self-heal on restart because of the retry logic and Docker restart policies. Startup ordering is deterministic because health checks gate dependent services.

  • If the system is still running, perform a final clean shutdown by running:
docker compose down
  • Bring the system back up in detached mode so it is ready for the next section by running:
docker compose up --build -d

Services start but immediately crash?

If you see repeated restart loops, RabbitMQ might not have passed its health check before the dependent services tried to connect. Run docker compose down and then docker compose up --build (without -d) to watch the startup sequence and confirm RabbitMQ's health check passes before the other services start.

Help me understand why my Docker Compose services keep restarting in a loop.

Your system has proven its resilience. Messages survive consumer downtime, services recover from broker outages, and Docker Compose coordinates the entire lifecycle. Next up, you will extend this architecture with a fan-out pattern in the Secret Mission.

Secret mission

Add a Notification Service (Fan-Out)

Your system currently has one consumer per event type. In production, multiple services often need to react to the same event independently. Can you add a Notification Service that receives the same order results as any other consumer, proving that fan-out messaging works?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker containers, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers for RabbitMQ, Order API, Inventory Service, and Notification Service.
  • Docker images built from your Dockerfiles.
  • A Docker network created by Compose.
  • Local project files on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively experimenting with your event-driven system or want to send more test orders.

  • Your containers continue running in the background.
  • The RabbitMQ Management UI stays accessible at http://localhost:15672.
  • The Order API remains available at http://localhost:3000.

Pause - I'll come back to this later

Stop all containers to free up memory and CPU while preserving their state. You can restart them later without rebuilding.

  • Stop all containers by running:
docker compose stop
  • When you are ready to resume, start everything again by running:
docker compose start

Your containers, images, and network remain intact on disk.

Delete - I don't want to use this again

Remove all Docker resources created by this project and delete the local files.

  • Remove all containers, volumes, networks, and locally-built images by running:
docker compose down -v --rmi local

What does this command do?

  • -v removes named volumes and anonymous volumes attached to the containers.
  • --rmi local removes images that were built locally by Compose (your Order API, Inventory Service, and Notification Service images).
  • Containers and the Compose network are removed automatically by docker compose down.
  • Delete the project folder from your Desktop by running:

macOS

cd ~/Desktop
rm -rf event-driven-orders

Windows

cd ~\Desktop
rmdir /s /q event-driven-orders

This removes all source code, Dockerfiles, and configuration files from your machine.

Nice Work!

Nice Work!

You just built a fully orchestrated polyglot event-driven system from scratch. Your TypeScript and Python services communicate asynchronously through RabbitMQ, handle failures gracefully, and self-heal after outages.

You've learned how to:

  • Build a polyglot event-driven system with a TypeScript Order API and Python Inventory Service communicating through RabbitMQ exchanges, routing keys, and message acknowledgments.
  • Orchestrate multi-service startup with Docker Compose health checks, dependency ordering, and restart policies to build a resilient local infrastructure.
  • Implement dead-letter queues to capture failed messages and observe system resilience as messages survive consumer downtime without data loss.
  • Secret Mission: Added fan-out messaging with a Notification Service so multiple consumers independently receive the same order result events.

Ready to quiz yourself?