Kafka vs RabbitMQ: Message Brokers

Run Kafka and RabbitMQ locally and experience the core architectural difference.

Introduction

30 Second Summary

When you order something online, dozens of systems spring into action. Your payment gets processed, a confirmation email fires off, and the warehouse starts picking your items. All of these services need to hear about your order, but they each need to hear about it differently.

In this project, you will build an e-commerce order processing system using both Apache Kafka and RabbitMQ. You'll publish real order messages through each broker and discover firsthand when to use one over the other.

What You'll Build

You'll run both brokers side by side in Docker, send order messages through each, and watch them behave in fundamentally different ways. One deletes messages the moment they're processed. The other keeps them forever so any new service can replay the full history.

By the end of this project, you'll have:

  • Sent orders through a RabbitMQ task queue, watched them get processed by a worker, and proved they vanish permanently after consumption.
  • Published the same orders to a Kafka event stream and replayed the entire order history from a brand-new consumer at any time.
  • Routed a single order to multiple specialized workers (email, warehouse) simultaneously using RabbitMQ's fanout exchange.
  • Secret Mission: Add Kafka consumer groups so two independent services process the same order stream at their own pace.

Are there any prerequisites?

You need Docker Desktop installed on your Mac. Everything else (Kafka, RabbitMQ, Python, and all libraries) runs inside Docker containers that this project sets up for you. No prior experience with message brokers is required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

You're going to run two completely different message brokers side by side and send the same order data through both. By the end, you'll know from firsthand experience when to reach for a message queue versus an event stream.

Set Up the Docker Environment

To compare Kafka and RabbitMQ side by side, you need both brokers running locally along with a Python environment that can talk to each one.

Docker Compose lets you define all three services in a single file and start them with one command. No Java, no Erlang, no manual broker configuration on your machine.

In this step, get ready to:
  • Create a project directory with a Dockerfile and Python dependencies file.
  • Write a Docker Compose configuration defining Kafka, RabbitMQ, and a Python app container.
  • Start all three containers and verify they are healthy.
Create the project directory and build files

Before writing any broker configuration, you need a project folder and the files that tell Docker how to build your Python container.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal, and pressing Enter.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and a scripts subdirectory by running these commands:
mkdir kafka-rabbitmq-faceoff
cd kafka-rabbitmq-faceoff
mkdir scripts

The scripts folder is where all your Python producer and consumer scripts will live throughout this project.

  • Create the requirements.txt file that lists the Python libraries your container needs by running:
cat > requirements.txt << 'EOF'
pika==1.4.2
confluent-kafka==2.15.0
EOF
  • Create the Dockerfile that builds your Python container by running:
cat > Dockerfile << 'EOF'
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

CMD ["tail", "-f", "/dev/null"]
EOF

What do these files do?

  • requirements.txt pins two libraries: pika (the official RabbitMQ Python client) and confluent-kafka (Confluent's production-grade Kafka Python client built on librdkafka).
  • The Dockerfile installs those libraries into a slim Python 3.12 image. The tail -f /dev/null command keeps the container alive so you can exec scripts into it later.
  • Confirm both files were created by running:
ls

You should see Dockerfile, requirements.txt, and the scripts directory in the output.

Missing a file?

Make sure you ran each cat > filename << 'EOF' command in full. If you see only one file, re-run the missing command from above.

help me troubleshoot missing project files after running cat heredoc commands

Write the Docker Compose configuration

Docker Compose defines your entire multi-container environment in one YAML file. You will configure three services: a Kafka broker, a RabbitMQ broker, and a Python app container.

  • Create docker-compose.yml with the Kafka service definition by running:
cat > docker-compose.yml << 'EOF'
services:
  kafka:
    image: apache/kafka:4.3.1
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: CONTROLLER://localhost:9091,HOST://0.0.0.0:9092,DOCKER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: HOST://localhost:9092,DOCKER://kafka:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,DOCKER:PLAINTEXT,HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9091
      KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    healthcheck:
      test: /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list || exit 1
      interval: 10s
      timeout: 5s
      retries: 5
EOF

What is KRaft mode?

Kafka 4.x uses KRaft mode by default, which eliminates the need for a separate ZooKeeper process. The KAFKA_PROCESS_ROLES variable tells this single node to act as both broker and controller.

The dual-listener setup (HOST on port 9092, DOCKER on port 9093) lets your host machine connect on 9092 while containers reach Kafka at kafka:9093.

  • Now append the RabbitMQ and app service definitions to the same file by running:
cat >> docker-compose.yml << 'EOF'

  rabbitmq:
    image: rabbitmq:4.3.4-management
    container_name: rabbitmq
    ports:
      - "5672:5672"
      - "15672:15672"
    healthcheck:
      test: rabbitmq-diagnostics -q ping
      interval: 10s
      timeout: 5s
      retries: 5

  app:
    build: .
    container_name: app
    depends_on:
      kafka:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
    volumes:
      - ./scripts:/app
    working_dir: /app
    command: tail -f /dev/null
EOF

What do these services do?

  • rabbitmq uses the -management image variant, which includes a web UI on port 15672 for inspecting queues and exchanges.
  • app builds from your local Dockerfile and mounts the scripts/ folder so any Python file you create locally appears instantly inside the container.
  • depends_on with service_healthy ensures the app container only starts after both brokers pass their health checks.
  • Validate that Docker Compose can parse your file by running:
docker compose config --services

You should see three service names printed, one per line: kafka, rabbitmq, and app.

Seeing a YAML error?

YAML is whitespace-sensitive. If you see a parsing error, check that the cat >> command (with two >>) was used for the second block. A single > overwrites the file instead of appending.

help me fix a YAML parsing error in my docker-compose.yml

✔️ Awesome, I've got everything!

Great. Make sure all four files are in your kafka-rabbitmq-faceoff directory before continuing.

ⓧ I'd like to double check the full code

Your docker-compose.yml was built in two chunks above. Scroll up and confirm both the Kafka service block (starting with services:) and the RabbitMQ + app block are present in order.

Here are the other two files for comparison:

pika==1.4.2
confluent-kafka==2.15.0
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

CMD ["tail", "-f", "/dev/null"]
Start the containers and verify everything is healthy

With all configuration files in place, one command brings up both brokers and your Python container.

  • Start all three services in detached mode by running:
docker compose up -d

What does this command do?

The -d flag runs containers in the background. Docker will pull the Kafka and RabbitMQ images (this may take a minute on the first run), build your app image from the Dockerfile, then start all three containers. The app container waits for both brokers to be healthy before starting.

You will see Docker pulling images and creating containers. This can take 1-2 minutes on the first run.

  • Once the command completes, verify all containers are running and healthy:
docker compose ps

You should see all three containers (kafka, rabbitmq, app) listed with a status of Up and health showing healthy.

  • Open the RabbitMQ management UI by navigating to http://localhost:15672 in your browser.
  • Log in with username guest and password guest.

You should see the RabbitMQ management dashboard showing an overview of your broker with 0 queues and 0 connections.

  • Before you run this next command, what do you think it will output given that no topics have been created yet?
  • Verify Kafka is responsive by listing topics inside the Kafka container:
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

The command completes with no output. That is correct. No topics exist yet, so the list is empty. The important thing is that it did not error out, proving Kafka is healthy and accepting connections.

Containers not starting?

  • If docker compose ps shows a container as unhealthy or restarting, check its logs with docker compose logs kafka or docker compose logs rabbitmq.
  • Make sure Docker Desktop is running. If you see a connection error, open Docker Desktop and wait for the engine to fully start.
  • If port 5672 or 9092 is already in use, stop the conflicting process or change the host port mapping in docker-compose.yml.

help me troubleshoot Docker containers that won't start or stay unhealthy

Your local broker environment is ready. Both Kafka and RabbitMQ are running side by side, and your Python app container is connected to both. Next up, you will send your first order messages through RabbitMQ and watch them get consumed.

Send Orders Through RabbitMQ

Your Docker environment is running with both brokers waiting for messages. Before comparing Kafka and RabbitMQ head-to-head, you need to see one broker working end-to-end.

RabbitMQ's simplest pattern is the work queue: a producer sends tasks to a queue, and a worker picks them up one at a time. You will build this pattern with real order data so you can see a message go in and a worker pick it up.

In this step, get ready to:
  • Write a producer script that sends order messages to RabbitMQ.
  • Write a consumer script that processes orders from the queue.
  • Run both scripts and verify the messages flow through.
Write the producer script

The producer connects to RabbitMQ, declares a queue, and publishes messages into it. Each message is a JSON object representing a customer order.

  • Create a new file called rabbitmq_producer.py inside your scripts/ folder and paste the following code:
import pika
import json

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host="rabbitmq", port=5672)
)
channel = connection.channel()

channel.queue_declare(queue="order_tasks", durable=True)

orders = [
    {"order_id": 1, "customer": "Alice", "item": "Laptop", "amount": 999.99},
    {"order_id": 2, "customer": "Bob", "item": "Headphones", "amount": 79.99},
    {"order_id": 3, "customer": "Charlie", "item": "Keyboard", "amount": 129.99},
    {"order_id": 4, "customer": "Diana", "item": "Monitor", "amount": 449.99},
    {"order_id": 5, "customer": "Eve", "item": "Mouse", "amount": 49.99},
]

for order in orders:
    channel.basic_publish(
        exchange="",
        routing_key="order_tasks",
        body=json.dumps(order),
        properties=pika.BasicProperties(delivery_mode=2),
    )
    print(f"Sent order #{order['order_id']} - {order['customer']} bought {order['item']}")

connection.close()
print("\nAll orders sent to RabbitMQ!")

What does this code do?

  • BlockingConnection opens a synchronous connection to the RabbitMQ container using the Docker hostname rabbitmq on port 5672.
  • queue_declare creates a durable queue called order_tasks. Durable means the queue survives a RabbitMQ restart.
  • delivery_mode=2 marks each message as persistent, so it is written to disk rather than only held in memory.
  • The loop publishes 5 order messages as JSON payloads to the order_tasks queue.
  • Save scripts/rabbitmq_producer.py.
Write the consumer script

The consumer subscribes to the same queue and waits for messages. When an order arrives, it processes the order and sends an acknowledgment back to RabbitMQ confirming it was handled.

  • Create a new file called rabbitmq_consumer.py inside your scripts/ folder and paste the following code:
import pika
import json
import sys

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host="rabbitmq", port=5672)
)
channel = connection.channel()

channel.queue_declare(queue="order_tasks", durable=True)

print("Waiting for orders. Press Ctrl+C to exit.\n")


def callback(ch, method, properties, body):
    order = json.loads(body)
    print(f"Processing order #{order['order_id']}: {order['customer']} - {order['item']} (${order['amount']})")
    ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue="order_tasks", on_message_callback=callback)

try:
    channel.start_consuming()
except KeyboardInterrupt:
    print("\nStopped consuming.")
    connection.close()

What does this code do?

  • callback is called once for each message that arrives. It parses the JSON body and prints the order details.
  • basic_ack tells RabbitMQ the message was successfully processed. Once acknowledged, the message is permanently removed from the queue.
  • basic_qos(prefetch_count=1) ensures the consumer processes one message at a time before asking for the next.
  • The try/except KeyboardInterrupt block lets you stop the consumer cleanly with Ctrl+C.
  • Save scripts/rabbitmq_consumer.py.

Why manual acknowledgment?

Without manual acknowledgment, RabbitMQ auto-acknowledges messages the moment they are delivered. If your worker crashes mid-processing, the message is lost forever. Manual basic_ack means the message stays in the queue until the worker confirms it is done.

✔️ Awesome, I've got everything!

Great. Make sure both files are saved in your scripts/ folder before continuing.

ⓧ I'd like to double check the full code

Here is the full rabbitmq_producer.py:

import pika
import json

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host="rabbitmq", port=5672)
)
channel = connection.channel()

channel.queue_declare(queue="order_tasks", durable=True)

orders = [
    {"order_id": 1, "customer": "Alice", "item": "Laptop", "amount": 999.99},
    {"order_id": 2, "customer": "Bob", "item": "Headphones", "amount": 79.99},
    {"order_id": 3, "customer": "Charlie", "item": "Keyboard", "amount": 129.99},
    {"order_id": 4, "customer": "Diana", "item": "Monitor", "amount": 449.99},
    {"order_id": 5, "customer": "Eve", "item": "Mouse", "amount": 49.99},
]

for order in orders:
    channel.basic_publish(
        exchange="",
        routing_key="order_tasks",
        body=json.dumps(order),
        properties=pika.BasicProperties(delivery_mode=2),
    )
    print(f"Sent order #{order['order_id']} - {order['customer']} bought {order['item']}")

connection.close()
print("\nAll orders sent to RabbitMQ!")

Here is the full rabbitmq_consumer.py:

import pika
import json
import sys

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host="rabbitmq", port=5672)
)
channel = connection.channel()

channel.queue_declare(queue="order_tasks", durable=True)

print("Waiting for orders. Press Ctrl+C to exit.\n")


def callback(ch, method, properties, body):
    order = json.loads(body)
    print(f"Processing order #{order['order_id']}: {order['customer']} - {order['item']} (${order['amount']})")
    ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue="order_tasks", on_message_callback=callback)

try:
    channel.start_consuming()
except KeyboardInterrupt:
    print("\nStopped consuming.")
    connection.close()
Run the producer and consumer

Both scripts run inside the app container, which already has pika installed and can reach the RabbitMQ container by hostname.

  • Send the 5 order messages to RabbitMQ by running this command:
docker compose exec app python rabbitmq_producer.py

You should see 5 lines confirming each order was sent, ending with "All orders sent to RabbitMQ!".

  • Open the RabbitMQ management UI at http://localhost:15672 in your browser.
  • Log in with username guest and password guest.
  • Click the Queues and Streams tab.

You should see the order_tasks queue with 5 messages ready. The messages are sitting in the queue waiting for a consumer.

Before you run the consumer, what do you think will happen to those 5 messages once the consumer processes and acknowledges them?

  • Run the consumer to process the queued orders:
docker compose exec app python rabbitmq_consumer.py

You should see all 5 orders printed with "Processing order #1", "Processing order #2", and so on. The consumer then hangs waiting for more messages.

  • Press Ctrl+C to stop the consumer.
  • Return to the RabbitMQ management UI and refresh the Queues and Streams page.

The order_tasks queue now shows 0 messages ready and 0 unacknowledged. Every message was consumed and acknowledged, so RabbitMQ deleted them from the queue.

Not seeing any output from the consumer?

  • Make sure you ran the producer first. The consumer only picks up messages that already exist in the queue.
  • Check that both files are saved inside the scripts/ folder (not the project root). The app container mounts scripts/ as its working directory.
  • If you see a connection error, confirm the RabbitMQ container is still running with docker compose ps.

Why isn't my RabbitMQ consumer receiving messages?

You have a working RabbitMQ pipeline. Orders go in, a worker picks them up, acknowledges them, and they disappear from the queue. Next, you will discover what happens when a second service wants to read those same orders.

Attempt to Replay — The Messages Are Gone

Attempt to Replay -- The Messages Are Gone

Your order processing system works. Messages go in, a worker picks them up, and they're processed. Now imagine a new requirement lands: an analytics dashboard needs to read the same order history to build reports.

In this step, you'll attempt to replay the orders you already processed. What you discover will reveal a fundamental architectural boundary of traditional message queues.

In this step, get ready to:
  • Run the consumer script a second time without sending new messages.
  • Confirm in the RabbitMQ management UI that acknowledged messages are permanently gone.
Run the consumer a second time

Your 5 orders were successfully consumed and acknowledged in the last step. A new analytics service just came online and needs to process those same orders to build a sales report.

Before you run the consumer again, take a moment to predict: what do you think will happen? Will the consumer see those 5 orders again, or will something else happen?

  • Run the consumer script again (without sending any new messages) by running this command:
docker compose exec app python rabbitmq_consumer.py

The terminal prints Waiting for orders. Press Ctrl+C to exit. and then... nothing. It just hangs there. No orders appear.

The 5 orders you sent earlier are gone. They were consumed and acknowledged by your first consumer, and RabbitMQ permanently deleted them from the queue. This is not a bug. This is exactly how a message queue is supposed to work.

  • Press Ctrl+C to stop the consumer and return to your terminal.

Why did the messages disappear?

When your consumer called basic_ack on each message, it told RabbitMQ: "I've finished processing this. You can delete it now." RabbitMQ obliged.

This is the core design of a message queue: each message is a task to be processed exactly once by one worker. Once acknowledged, the broker's job is done and the message is discarded.

Confirm the queue is empty in the management UI
  • Switch back to your browser with the RabbitMQ management UI at http://localhost:15672.
  • Click the Queues and Streams tab in the top navigation.
  • Find the order_tasks queue in the list.

The Ready column shows 0 and the Unacked column shows 0. The messages are permanently gone from the broker. No amount of new consumers connecting will bring them back.

When is this a problem?

For task distribution (send one confirmation email per order), this "consume once and forget" model is perfect. Each task should only happen once.

It becomes a problem when multiple independent services need access to the same event history. An analytics dashboard that comes online tomorrow cannot replay yesterday's orders. A fraud-detection service added next week cannot retroactively scan old transactions. This is the limitation you just hit.

You just hit the wall. RabbitMQ faithfully did its job: it delivered your messages to a worker and deleted them after acknowledgment. But your analytics service that arrived too late is out of luck. In the next step, you'll solve this problem with a completely different architecture: an event stream that never forgets.

Build a Kafka Event Stream

You just proved that RabbitMQ's queue model has a hard limit. Once a consumer acknowledges a message, it vanishes forever.

Kafka takes a fundamentally different approach. Instead of deleting messages after delivery, Kafka stores every event in a durable log. Any consumer can read from the beginning at any time, days or weeks later. In this step, you'll publish the same orders to Kafka and prove they survive consumption.

In this step, get ready to:
  • Write a Kafka producer that publishes orders to a durable event stream.
  • Write a Kafka consumer that reads orders and supports replaying from the beginning.
  • Prove messages persist by consuming them twice with different consumer groups.
Write the Kafka producer

The producer needs to do two things: create a topic to write to, and publish order messages into that topic.

  • Create a new file called kafka_producer.py inside your scripts folder.
  • Add the imports and topic creation logic by pasting this code:
import json
import sys
from confluent_kafka import Producer
from confluent_kafka.admin import AdminClient, NewTopic

# Connect to Kafka and create the "order_events" topic
admin = AdminClient({"bootstrap.servers": "kafka:9093"})
topic = NewTopic("order_events", num_partitions=1, replication_factor=1)
futures = admin.create_topics([topic])
for t, f in futures.items():
    try:
        f.result()
        print(f"Created topic: {t}")
    except Exception as e:
        # If the topic already exists, that's fine
        if "TOPIC_ALREADY_EXISTS" in str(e):
            print(f"Topic '{t}' already exists.")
        else:
            print(f"Failed to create topic '{t}': {e}")
            sys.exit(1)

What does this code do?

The AdminClient connects to Kafka on port 9093 (the DOCKER listener configured for container-to-container traffic).

NewTopic defines the topic with 1 partition and replication factor 1 (appropriate for a single-node setup). If the topic already exists, the script handles that gracefully instead of crashing.

  • Below the topic creation code, add the producer logic that publishes 5 orders:
# Create a producer instance
producer = Producer({"bootstrap.servers": "kafka:9093"})

# Same 5 orders as the RabbitMQ version
orders = [
    {"order_id": 1, "customer": "Alice", "item": "Laptop", "amount": 999.99},
    {"order_id": 2, "customer": "Bob", "item": "Headphones", "amount": 79.99},
    {"order_id": 3, "customer": "Charlie", "item": "Keyboard", "amount": 129.99},
    {"order_id": 4, "customer": "Diana", "item": "Monitor", "amount": 449.99},
    {"order_id": 5, "customer": "Eve", "item": "Mouse", "amount": 49.99},
]

# Callback that fires when Kafka confirms receipt of each message
def delivery_report(err, msg):
    if err:
        print(f"Delivery failed: {err}")
    else:
        print(f"Sent order to Kafka topic '{msg.topic()}' [partition {msg.partition()}]")

# Publish each order to the "order_events" topic
for order in orders:
    producer.produce(
        "order_events",
        key=str(order["order_id"]),
        value=json.dumps(order),
        callback=delivery_report,
    )

# Wait for all messages to be delivered
producer.flush()
print("\nAll orders sent to Kafka!")

What does this code do?

producer.produce() queues each message for asynchronous delivery. The delivery_report callback fires once Kafka confirms it stored the message.

producer.flush() blocks until all queued messages are confirmed. This ensures nothing is lost before the script exits.

  • Save kafka_producer.py.
  • Run the producer inside the app container by running this command:
docker compose exec app python kafka_producer.py

You should see 5 delivery confirmations followed by "All orders sent to Kafka!" printed in your terminal.

Seeing a connection error?

  • Make sure your Docker containers are still running. Run docker compose ps and confirm all three services show as healthy.
  • The script connects to kafka:9093 (the DOCKER listener). If you accidentally used localhost:9092, it won't resolve from inside the container.

Why can't my producer connect to Kafka inside Docker?

Write the Kafka consumer

Now you need a consumer that reads from the order_events topic. This consumer will support a --from-beginning flag that lets you replay all messages from the start of the log.

  • Create a new file called kafka_consumer.py inside your scripts folder.
  • Add the imports and argument parsing by pasting this code:
import json
import sys
from confluent_kafka import Consumer, KafkaException

# Default settings
group_id = "default-group"
from_beginning = False

# Parse command-line flags
args = sys.argv[1:]
for i, arg in enumerate(args):
    if arg == "--from-beginning":
        from_beginning = True
    elif arg == "--group" and i + 1 < len(args):
        group_id = args[i + 1]

What does this code do?

The script accepts two optional flags: --from-beginning (replay all messages from the start) and --group (set a custom consumer group ID). Without flags, it defaults to default-group and reads only new messages.

  • Below the argument parsing, add the consumer configuration and poll loop:
# Configure the consumer with group and offset settings
config = {
    "bootstrap.servers": "kafka:9093",
    "group.id": group_id,
    "auto.offset.reset": "earliest" if from_beginning else "latest",
}

consumer = Consumer(config)
consumer.subscribe(["order_events"])

print(f"Consuming from 'order_events' (group: {group_id}, from_beginning: {from_beginning})")
print("Press Ctrl+C to exit.\n")

# Poll for messages in a loop
try:
    while True:
        msg = consumer.poll(timeout=2.0)
        if msg is None:
            continue
        if msg.error():
            raise KafkaException(msg.error())
        order = json.loads(msg.value().decode("utf-8"))
        print(f"[{group_id}] Order #{order['order_id']}: {order['customer']} - {order['item']} (${order['amount']})")
except KeyboardInterrupt:
    print("\nStopped consuming.")
finally:
    consumer.close()

What does this code do?

auto.offset.reset controls where a new consumer group starts reading. Set to earliest, it replays from the very first message. Set to latest, it only reads messages that arrive after it connects.

The consumer.poll() call waits up to 2 seconds for a message. If nothing arrives, it loops and tries again until you press Ctrl+C.

  • Save kafka_consumer.py.
  • Run the consumer to read the orders you just published:
docker compose exec app python kafka_consumer.py --from-beginning

You should see all 5 orders printed with the [default-group] prefix. Press Ctrl+C to exit.

Not seeing any messages?

  • Make sure you included the --from-beginning flag. Without it, the consumer only reads NEW messages that arrive after it starts.
  • Confirm you ran the producer successfully first. If the producer printed delivery confirmations, the messages are in Kafka.

Help me debug my Kafka consumer not receiving messages.

Prove messages persist with replay

Here is the moment of truth. You already consumed all 5 orders. With RabbitMQ, those messages would be gone forever. Kafka is different.

Before you run this next command, think: do you expect to see the 5 orders again, or will the consumer get nothing like RabbitMQ?

  • Run a second consumer with a different group ID and the --from-beginning flag:
docker compose exec app python kafka_consumer.py --from-beginning --group replay-group

All 5 orders print again. The messages are still sitting in Kafka's log, untouched by the first consumer.

Why does this work?

Kafka never deletes messages after consumption. Each consumer group tracks its own position (called an offset) in the log. A brand-new group with auto.offset.reset: earliest starts reading from the very first message, regardless of how many other groups have already read the same data.

This is why Kafka is used for event sourcing, analytics pipelines, and audit logs. The data stays put, and any new service can catch up from the beginning.

  • Press Ctrl+C to exit the consumer.

What is the difference between RabbitMQ and Kafka?

  • RabbitMQ is a message queue. Messages are delivered to one consumer, acknowledged, and deleted. Great for task distribution.
  • Kafka is an event log. Messages are appended to a durable stream and retained indefinitely. Any number of consumers can read the same data at their own pace.

✔️ Awesome, I've got everything!

Great. Double check you saved both kafka_producer.py and kafka_consumer.py.

ⓧ I'd like to double check the full code

Here is the complete kafka_producer.py:

import json
import sys
from confluent_kafka import Producer
from confluent_kafka.admin import AdminClient, NewTopic

admin = AdminClient({"bootstrap.servers": "kafka:9093"})
topic = NewTopic("order_events", num_partitions=1, replication_factor=1)
futures = admin.create_topics([topic])
for t, f in futures.items():
    try:
        f.result()
        print(f"Created topic: {t}")
    except Exception as e:
        if "TOPIC_ALREADY_EXISTS" in str(e):
            print(f"Topic '{t}' already exists.")
        else:
            print(f"Failed to create topic '{t}': {e}")
            sys.exit(1)

producer = Producer({"bootstrap.servers": "kafka:9093"})

orders = [
    {"order_id": 1, "customer": "Alice", "item": "Laptop", "amount": 999.99},
    {"order_id": 2, "customer": "Bob", "item": "Headphones", "amount": 79.99},
    {"order_id": 3, "customer": "Charlie", "item": "Keyboard", "amount": 129.99},
    {"order_id": 4, "customer": "Diana", "item": "Monitor", "amount": 449.99},
    {"order_id": 5, "customer": "Eve", "item": "Mouse", "amount": 49.99},
]

And the second half of kafka_producer.py (continues directly below the orders list):


def delivery_report(err, msg):
    if err:
        print(f"Delivery failed: {err}")
    else:
        print(f"Sent order to Kafka topic '{msg.topic()}' [partition {msg.partition()}]")


for order in orders:
    producer.produce(
        "order_events",
        key=str(order["order_id"]),
        value=json.dumps(order),
        callback=delivery_report,
    )

producer.flush()
print("\nAll orders sent to Kafka!")

Here is the complete kafka_consumer.py:

import json
import sys
from confluent_kafka import Consumer, KafkaException

group_id = "default-group"
from_beginning = False

args = sys.argv[1:]
for i, arg in enumerate(args):
    if arg == "--from-beginning":
        from_beginning = True
    elif arg == "--group" and i + 1 < len(args):
        group_id = args[i + 1]

config = {
    "bootstrap.servers": "kafka:9093",
    "group.id": group_id,
    "auto.offset.reset": "earliest" if from_beginning else "latest",
}

And the second half of kafka_consumer.py (continues directly below the config dict):

consumer = Consumer(config)
consumer.subscribe(["order_events"])

print(f"Consuming from 'order_events' (group: {group_id}, from_beginning: {from_beginning})")
print("Press Ctrl+C to exit.\n")

try:
    while True:
        msg = consumer.poll(timeout=2.0)
        if msg is None:
            continue
        if msg.error():
            raise KafkaException(msg.error())
        order = json.loads(msg.value().decode("utf-8"))
        print(f"[{group_id}] Order #{order['order_id']}: {order['customer']} - {order['item']} (${order['amount']})")
except KeyboardInterrupt:
    print("\nStopped consuming.")
finally:
    consumer.close()

You just proved the core difference between a message queue and an event stream with your own hands. Next up, you'll see what RabbitMQ does better than Kafka: routing a single message to multiple specialized workers simultaneously.

Route Orders to Multiple Workers with RabbitMQ

You proved that Kafka retains events for replay by any new consumer. RabbitMQ cannot do that. But RabbitMQ has a strength Kafka does not match out of the box: intelligent message routing.

In a real e-commerce system, a single order triggers multiple downstream actions simultaneously. An email confirmation goes out, the warehouse gets notified, and payment is captured. RabbitMQ's fanout exchange broadcasts one message to every bound queue at once, giving each worker its own copy.

In this step, get ready to:
  • Write a fanout producer that publishes orders to an exchange.
  • Write a fanout consumer that binds a named queue to the exchange and processes orders.
  • Run two consumers simultaneously and confirm each receives all orders.
Write the fanout producer

Instead of publishing directly to a queue (like in Step 2), this producer sends messages to an exchange. The exchange then routes copies to every queue bound to it.

  • Create a new file called rabbitmq_fanout_producer.py inside your scripts folder, and paste the following code:
import pika
import json

# Connect to the RabbitMQ broker
connection = pika.BlockingConnection(
    pika.ConnectionParameters(host="rabbitmq", port=5672)
)
channel = connection.channel()

# Declare a fanout exchange (broadcasts to all bound queues)
channel.exchange_declare(exchange="order_exchange", exchange_type="fanout")

# Three new orders to publish
orders = [
    {"order_id": 101, "customer": "Frank", "item": "Tablet", "amount": 599.99},
    {"order_id": 102, "customer": "Grace", "item": "Charger", "amount": 29.99},
    {"order_id": 103, "customer": "Hank", "item": "Case", "amount": 39.99},
]

# Publish each order to the exchange with an empty routing key
for order in orders:
    channel.basic_publish(
        exchange="order_exchange",
        routing_key="",
        body=json.dumps(order),
    )
    print(f"Published order #{order['order_id']} to exchange 'order_exchange'")

connection.close()
print("\nAll orders published to fanout exchange!")

What does this code do?

  • exchange_declare creates a fanout exchange called order_exchange. Unlike a queue, an exchange never stores messages. It only routes them.
  • basic_publish sends each order to the exchange with an empty routing_key. Fanout exchanges ignore the routing key entirely.
  • Any queue bound to this exchange at the time of publish receives its own copy of the message.

Why is the routing key empty?

A fanout exchange broadcasts every message to all bound queues regardless of the routing key value. Setting it to an empty string makes this explicit: the exchange does the routing, not the key.

Write the fanout consumer

This consumer takes a queue name as a command-line argument, declares that queue, binds it to the exchange, and then listens for orders. Running multiple instances with different queue names simulates separate microservices.

  • Create a new file called rabbitmq_fanout_consumer.py inside your scripts folder, and paste the setup and binding logic:
import pika
import json
import sys

# Require a queue name as a command-line argument
if len(sys.argv) < 2:
    print("Usage: python rabbitmq_fanout_consumer.py <queue_name>")
    print("Example: python rabbitmq_fanout_consumer.py email_tasks")
    sys.exit(1)

queue_name = sys.argv[1]

# Connect to RabbitMQ
connection = pika.BlockingConnection(
    pika.ConnectionParameters(host="rabbitmq", port=5672)
)
channel = connection.channel()

# Declare the exchange, declare the queue, and bind them together
channel.exchange_declare(exchange="order_exchange", exchange_type="fanout")
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="order_exchange", queue=queue_name)

print(f"[{queue_name}] Waiting for orders. Press Ctrl+C to exit.\n")

What does this setup do?

  • The script reads sys.argv[1] to decide which queue to create. This lets you run the same script as different workers (email, warehouse, etc.).
  • exchange_declare is idempotent. Both producer and consumer declare it to guarantee it exists regardless of startup order.
  • queue_bind creates the binding between the named queue and the exchange. This is what tells the exchange to route messages to this queue.
  • Now add the message handling logic below the print statement in the same file:

def callback(ch, method, properties, body):
    order = json.loads(body)
    print(f"[{queue_name}] Order #{order['order_id']}: {order['customer']} - {order['item']}")
    ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_consume(queue=queue_name, on_message_callback=callback)

try:
    channel.start_consuming()
except KeyboardInterrupt:
    print(f"\n[{queue_name}] Stopped.")
    connection.close()

What does the callback do?

  • callback fires for each incoming message. It parses the JSON, prints the order details prefixed with the queue name, and sends an acknowledgment back to RabbitMQ.
  • basic_consume registers the callback, and start_consuming blocks until you press Ctrl+C.
  • Save rabbitmq_fanout_consumer.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved both rabbitmq_fanout_producer.py and rabbitmq_fanout_consumer.py inside the scripts folder.

ⓧ I'd like to double check the full code

Scroll up and confirm your rabbitmq_fanout_consumer.py contains both chunks in order:

  • Chunk 1: imports, argument check, connection, exchange/queue declaration, and binding (ending with the print statement).
  • Chunk 2: the callback function, basic_consume, and the try/except KeyboardInterrupt block.
Run both consumers, then publish orders

A fanout exchange only delivers messages to queues that are already bound at the moment of publishing. You need to start both consumers first so their queues exist, then send the orders.

  • Start the email worker by running this command:
docker compose exec app python rabbitmq_fanout_consumer.py email_tasks

You should see the line: [email_tasks] Waiting for orders. Press Ctrl+C to exit. The terminal is now blocked, waiting for messages.

  • Open a second terminal window in the same project directory.
  • Start the warehouse worker by running this command:
docker compose exec app python rabbitmq_fanout_consumer.py warehouse_tasks

You should see: [warehouse_tasks] Waiting for orders. Press Ctrl+C to exit. Both workers are now listening.

  • Open a third terminal window in the same project directory.
  • Before you run the producer, predict: will both consumers receive all 3 orders, or will the orders be split between them?
  • Publish the orders by running this command:
docker compose exec app python rabbitmq_fanout_producer.py

In the producer terminal, you see all 3 orders published. Switch to each consumer terminal. Both the email worker and the warehouse worker printed all 3 orders independently.

Why did both workers get all 3 orders?

The fanout exchange copied each message to every bound queue. email_tasks and warehouse_tasks each received their own independent copy. This is the publish/subscribe pattern: one message, many receivers.

Compare this to the simple work queue from Step 2, where each message was delivered to exactly one consumer. Exchanges give you control over how messages fan out across your system.

  • Press Ctrl+C in both consumer terminals to stop them.
  • Return to the RabbitMQ management UI at http://localhost:15672 (login with guest/guest if needed).
  • Click the Queues and Streams tab. You should see both email_tasks and warehouse_tasks listed with 0 messages (all consumed).
  • Click the Exchanges tab. You should see order_exchange listed as type fanout. Click it to see both queue bindings.

Consumers show no orders?

  • Make sure you started both consumers BEFORE running the producer. A fanout exchange discards messages if no queues are bound at publish time.
  • Confirm you ran the commands inside the app container (using docker compose exec app). Running python directly on your host will fail to connect to the rabbitmq hostname.
  • Check that the exchange name is order_exchange in both files. A typo means the consumer binds to a different exchange than the one the producer publishes to.

My consumers aren't receiving any orders from the fanout exchange.

You now have both brokers flexing their strengths. Kafka retains events for replay. RabbitMQ routes discrete tasks to specialized workers. Next up in the Secret Mission, you will push Kafka further with consumer groups that let multiple independent services read the same stream at their own pace.

Secret mission

Kafka Consumer Groups

You just saw RabbitMQ's fanout exchange deliver copies of messages to multiple queues. Kafka takes a completely different approach. Instead of copying data, multiple consumer groups read from the same immutable log using independent offsets. In this mission, you will run two independent services against a single Kafka topic and prove that each tracks its own position.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Docker containers, so there are no ongoing cloud costs.

Resources you used:

  • Three Docker containers: kafka, rabbitmq, and app (consuming local memory and CPU).
  • Docker volumes storing Kafka and RabbitMQ data.
  • Local project files: docker-compose.yml, Dockerfile, requirements.txt, and the scripts/ folder.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the brokers.

  • Your three containers continue running in the background.
  • The RabbitMQ management UI remains available at http://localhost:15672.
  • You can keep sending messages and testing new consumer patterns at any time.

Pause - I'll come back to this later

Shut down running containers to free up memory and CPU, but keep all data and files intact for later.

  • In your terminal, navigate to the folder containing your docker-compose.yml and run:
docker compose stop
  • This stops all three containers without removing them or their data.
  • When you are ready to resume, run:
docker compose start
  • Your containers restart with all Kafka topics, RabbitMQ queues, and consumer offsets intact.

Delete - I don't want to use this again

Remove all containers, volumes, and project files to free all disk space and memory.

  • In your terminal, navigate to the folder containing your docker-compose.yml and run:
docker compose down -v
  • This removes all three containers and their associated volumes (Kafka data, RabbitMQ data).
  • Delete the project folder from your Desktop. Open Finder, navigate to your Desktop, right-click the project folder, and select Move to Trash.
  • This removes docker-compose.yml, Dockerfile, requirements.txt, and the entire scripts/ folder.

Nice Work!

Nice Work!

You just built a local order processing system powered by two completely different message broker architectures. You now have firsthand experience with the core trade-off between message queues and event streams.

You've learned how to:

  • Build a Docker Compose environment running both Kafka and RabbitMQ side by side, processing real order messages through each broker's architecture.
  • Experience the fundamental architectural difference firsthand: RabbitMQ messages disappear after acknowledgment, while Kafka retains events for any new consumer to replay from the beginning.
  • Use RabbitMQ's fanout exchange to broadcast a single order to multiple specialized worker queues simultaneously, proving each broker has a distinct superpower.
  • Implement Kafka consumer groups where independent services consume the same stream at their own pace with independent offset tracking (Secret Mission).

Ready to quiz yourself?