Real-Time Order Pipeline with Kafka

Stream e-commerce orders through Kafka partitions and watch consumer groups rebalance.

Introduction

30 Second Summary

Every time you order something online, that purchase gets routed to one of many workers processing orders simultaneously. This is event streaming, and Apache Kafka is the technology powering it at companies like Uber, Netflix, and LinkedIn.

In this project, you will build a real-time order processing pipeline using Apache Kafka running locally in Docker. You'll stream simulated e-commerce orders through a partitioned topic and watch consumers automatically divide the workload.

What You'll Build

A local event-driven pipeline where a Python producer streams orders into a 3-partition Kafka topic, and multiple consumers in a consumer group process them in parallel.

By the end of this project, you'll have:

  • A Docker Compose stack running a Kafka broker in KRaft mode and Kafka UI for visual monitoring.
  • A working producer/consumer pipeline where order events flow through partitions and get processed by a consumer group in parallel.
  • Visual proof of partition rebalancing as you scale consumers up and down and watch partition reassignment in Kafka UI.
  • Secret Mission: add a dead letter queue to handle failed order processing gracefully.

Are there any prerequisites?

You'll need basic Python knowledge, Docker Desktop installed on your machine, and familiarity with running terminal commands. No prior Kafka experience is required.

Before We Start

Before We Start...

Set Up the Environment

Every real-time system starts with reliable infrastructure. Before you write a single line of application code, you need a local Kafka broker running and a Python environment ready to talk to it.

In this step, you'll create a project workspace, install the Kafka Python library, and spin up a local Kafka cluster using Docker Compose.

In this step, get ready to:
  • Create a project directory with a Python virtual environment.
  • Install the confluent-kafka Python package.
  • Launch a Kafka broker and Kafka UI with Docker Compose.
Create the project directory and install dependencies

You need an isolated Python environment for this project so your packages don't collide with other projects on your machine.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir kafka-orders
cd kafka-orders
  • Confirm the folder exists by running:
ls

You should see the kafka-orders folder listed in the Desktop output. You're now inside it.

  • Create a Python virtual environment and activate it by following the instructions for your operating system:

macOS

  • Create and activate the virtual environment by running these commands:
python3 -m venv venv
source venv/bin/activate

Your terminal prompt now shows (venv) at the beginning, confirming the environment is active.

Windows

  • Create and activate the virtual environment by running these commands:
python -m venv venv
venv\Scripts\activate

Your terminal prompt now shows (venv) at the beginning, confirming the environment is active.

Don't see (venv) in your prompt?

Make sure you ran the activation command (not just the python -m venv venv creation command). The creation and activation are two separate steps.

On Windows, if you see an execution policy error, run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned first, then try activating again.

Help me activate my Python virtual environment

  • Install the confluent-kafka package by running:
pip install confluent-kafka

What is confluent-kafka?

This is the official Python client for Apache Kafka, maintained by Confluent (the company founded by Kafka's creators). It provides the Producer, Consumer, and AdminClient classes you'll use throughout this project.

You should see output ending with Successfully installed confluent-kafka-2.15.0 (or a similar version number).

Installation failed?

Make sure your virtual environment is active (you see (venv) in your prompt). If you run pip install without an active venv, it may fail due to system permissions.

If you see a build error mentioning librdkafka, try upgrading pip first with pip install --upgrade pip and then retry the install.

Help me install confluent-kafka in my virtual environment

Create the Docker Compose file

Docker Compose lets you define multi-container applications in a single YAML file. Your stack needs two services: a Kafka broker and a web UI for inspecting topics and messages.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop and select the kafka-orders folder, then click Open.
  • In the Explorer sidebar on the left, hover over the KAFKA-ORDERS heading and click the new file icon.
  • Name the file docker-compose.yml and press Enter.
  • Paste the following configuration into docker-compose.yml:
services:
  kafka:
    image: apache/kafka:4.3.1
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: CONTROLLER://localhost:9091,HOST://0.0.0.0:9092,DOCKER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: HOST://localhost:9092,DOCKER://kafka:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,DOCKER:PLAINTEXT,HOST:PLAINTEXT
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9091
      KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  kafka-ui:
    image: kafbat/kafka-ui:main
    ports:
      - "8080:8080"
    environment:
      DYNAMIC_CONFIG_ENABLED: "true"
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9093
    depends_on:
      - kafka

What does this configuration do?

This file defines two services that Docker will run as containers:

  • kafka runs a single-node Kafka broker using KRaft mode (no ZooKeeper needed). It exposes port 9092 so your Python scripts on the host machine can connect. The three listener configuration (CONTROLLER, HOST, DOCKER) lets the broker handle internal coordination, host connections, and container-to-container communication separately.
  • kafka-ui gives you a browser-based dashboard at port 8080 for inspecting topics, messages, and consumer groups visually. It connects to Kafka on port 9093 (the internal Docker network port).
  • Save docker-compose.yml by pressing Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

services:
  kafka:
    image: apache/kafka:4.3.1
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: CONTROLLER://localhost:9091,HOST://0.0.0.0:9092,DOCKER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: HOST://localhost:9092,DOCKER://kafka:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,DOCKER:PLAINTEXT,HOST:PLAINTEXT
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9091
      KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  kafka-ui:
    image: kafbat/kafka-ui:main
    ports:
      - "8080:8080"
    environment:
      DYNAMIC_CONFIG_ENABLED: "true"
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9093
    depends_on:
      - kafka
Start the Kafka stack and verify it's running

With the Compose file saved, you can now start both containers with a single command.

  • Make sure Docker Desktop is running on your machine. You should see the Docker icon in your system tray or menu bar.
  • Open the integrated terminal in VS Code by pressing Ctrl+` (macOS) or Ctrl+` (Windows).
  • Start the Docker Compose stack by running:
docker compose up -d

What does this command do?

The -d flag runs the containers in detached mode (in the background). Docker pulls the Kafka and Kafka UI images, creates the containers, and starts them. The first run takes a minute or two while the images download.

You should see output showing both containers starting up. Wait about 10 seconds for Kafka to fully initialize.

  • Open your browser and navigate to http://localhost:8080.

You should see the Kafka UI dashboard load, showing the local cluster with 0 topics listed.

Page not loading at localhost:8080?

Make sure Docker Desktop is running. If you see "Cannot connect", the containers may still be starting up. Wait 15-20 seconds and refresh.

Run docker compose ps in your terminal to check if both containers show a running state. If the kafka container keeps restarting, check that port 9092 is not already in use by another application.

Help me get Kafka UI running in Docker

Your infrastructure is live. Next up, you'll create your first Kafka topic with multiple partitions and see how Kafka organizes data for parallel processing.

Create a Partitioned Topic

Your Kafka broker is running, but it has no topics yet. A topic is where messages live in Kafka, and partitions are how Kafka splits a topic's data to enable parallel processing.

In this step, you will create an "orders" topic with 3 partitions using a Python script. Then you will inspect it from the command line to see how Kafka distributes partitions across brokers.

In this step, get ready to:
  • Write a Python admin script that creates a partitioned topic.
  • Run the script and verify the topic in Kafka UI.
  • Inspect the topic using the Kafka CLI inside the Docker container.
Write the topic admin script

The confluent-kafka library includes an AdminClient that lets you manage Kafka resources programmatically. You will use it to create a topic with a specific number of partitions.

  • Create a new file called admin_topics.py in your project folder by running:
touch admin_topics.py
  • Open admin_topics.py in VS Code and paste in the following code:
from confluent_kafka.admin import AdminClient, NewTopic

admin = AdminClient({'bootstrap.servers': 'localhost:9092'})

topic = NewTopic('orders', num_partitions=3, replication_factor=1)

futures = admin.create_topics([topic])

for topic_name, future in futures.items():
    try:
        future.result()
        print(f"Topic '{topic_name}' created successfully")
    except Exception as e:
        print(f"Failed to create topic '{topic_name}': {e}")

What does this code do?

This script connects to your local Kafka broker and creates a topic called orders.

  • AdminClient({'bootstrap.servers': 'localhost:9092'}) connects to the Kafka broker running on your machine.
  • NewTopic('orders', num_partitions=3, replication_factor=1) defines a topic with 3 partitions. Each partition can be processed independently, which is how Kafka achieves parallelism.
  • admin.create_topics([topic]) sends the create request to Kafka and returns a dictionary of futures you can check for success or failure.
  • Save admin_topics.py.

File not saving?

Make sure you are saving the file inside your project directory (the same folder that contains docker-compose.yml). Help me find where to save my file.

Run the script and verify in Kafka UI

With the script saved, you can run it to create the topic on your running Kafka broker.

  • Run the admin script by executing:
python admin_topics.py
  • You should see this output in your terminal:
Topic 'orders' created successfully

Seeing a connection error?

Make sure your Docker Compose stack is still running. Switch back to the terminal where you ran docker compose up and confirm Kafka is healthy. If the stack stopped, restart it with docker compose up -d.

Also confirm your virtual environment is activated. You should see (venv) at the start of your terminal prompt. Help me fix my connection error.

  • Open your browser and navigate to http://localhost:8080 (Kafka UI).
  • Click on Topics in the left sidebar.
  • You should see the orders topic listed with 3 partitions.

What are partitions?

Partitions are how Kafka divides a topic's data into smaller, independent streams. With 3 partitions, up to 3 consumers can process messages from this topic in parallel.

Each partition maintains its own ordering guarantee. Messages within a single partition are always read in the order they were written.

Inspect the topic using the Kafka CLI

Kafka ships with command-line tools inside the Docker container. You can use these to inspect your topic's partition layout directly.

  • Describe the orders topic by running this command:
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
  • You should see output similar to this:
Topic: orders	TopicId: ...	PartitionCount: 3	ReplicationFactor: 1	Configs:
	Topic: orders	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: orders	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: orders	Partition: 2	Leader: 1	Replicas: 1	Isr: 1

What does this output mean?

Each line after the header represents one partition of the orders topic.

  • Leader: 1 means node 1 (your single broker) handles all reads and writes for that partition.
  • Replicas: 1 shows which brokers hold a copy of the partition data.
  • Isr: 1 lists the in-sync replicas. Since you have only one broker, all three values are the same.

In a multi-broker cluster, different partitions would have different leaders to spread the load. Your single-node setup keeps things simple for learning.

Command not found or connection refused?

If you see an error about the container not running, make sure your Docker Compose stack is up. Run docker compose ps to check that the kafka service shows as running.

If you get a "topic does not exist" message, re-run python admin_topics.py to create the topic first. Help me troubleshoot the kafka-topics.sh command.

✔️ Awesome, I've got everything!

Great. Double check you have saved admin_topics.py and confirmed the topic exists in Kafka UI.

ⓧ I'd like to double check the full code

from confluent_kafka.admin import AdminClient, NewTopic

admin = AdminClient({'bootstrap.servers': 'localhost:9092'})

topic = NewTopic('orders', num_partitions=3, replication_factor=1)

futures = admin.create_topics([topic])

for topic_name, future in futures.items():
    try:
        future.result()
        print(f"Topic '{topic_name}' created successfully")
    except Exception as e:
        print(f"Failed to create topic '{topic_name}': {e}")

Your topic is ready with 3 partitions waiting for messages. Next up, you will build a producer that streams order events into these partitions and see how Kafka distributes them.

Build the Order Producer

Your orders topic is ready with 3 partitions. Now you need something to feed data into it.

In this step, you'll write a Python producer that continuously streams simulated e-commerce orders into the topic. Each order gets a unique key that Kafka hashes to decide which partition it lands in.

In this step, get ready to:
  • Write a producer script that generates and sends order events.
  • Run the producer and observe partition routing in delivery reports.
  • Verify message distribution across partitions in Kafka UI.
Write the order producer script

A producer is a client application that publishes messages to a Kafka topic. Your producer will generate random order data and send each order as a JSON message.

  • Create a new file called producer.py in your project folder.
  • Add the imports and delivery report callback by pasting this code:
import json
import time
import random
from confluent_kafka import Producer

# Callback triggered when a message is successfully delivered (or fails)
def delivery_report(err, msg):
    if err is not None:
        print(f"Delivery failed: {err}")
    else:
        print(f"Order delivered to partition {msg.partition()} at offset {msg.offset()}")

What does this code do?

The delivery_report function is a callback that Kafka triggers after each message is either delivered or fails. It prints the partition number and offset so you can see exactly where each order lands.

The Producer import comes from the confluent-kafka library. It handles connecting to the broker and managing message delivery under the hood.

  • Below the callback, add the producer configuration and the main event loop:
# Connect to the local Kafka broker
producer = Producer({'bootstrap.servers': 'localhost:9092'})

# Sample data for generating random orders
order_items = ['laptop', 'phone', 'tablet', 'headphones', 'keyboard', 'monitor']
regions = ['north', 'south', 'east', 'west']

# Continuously produce orders until manually stopped
order_id = 0
try:
    while True:
        order_id += 1
        order = {
            'order_id': order_id,
            'item': random.choice(order_items),
            'quantity': random.randint(1, 5),
            'region': random.choice(regions),
            'timestamp': time.time()
        }

        # Send the order to the 'orders' topic with order_id as the partition key
        producer.produce(
            'orders',
            key=str(order_id),
            value=json.dumps(order),
            callback=delivery_report
        )
        # Trigger delivery report callbacks without blocking
        producer.poll(0)

        print(f"Produced order #{order_id}: {order['item']} x{order['quantity']} ({order['region']})")
        time.sleep(1)

except KeyboardInterrupt:
    print("\nStopping producer...")
finally:
    # Ensure all buffered messages are sent before exiting
    producer.flush()
    print("All messages flushed.")

What does this code do?

The producer.produce() call sends each order to the orders topic. The key=str(order_id) parameter is critical. Kafka hashes the key to determine which partition the message goes to. The same key always routes to the same partition, guaranteeing ordering per key.

The producer.poll(0) call triggers any pending delivery report callbacks without blocking the loop. The producer.flush() in the finally block makes sure all buffered messages reach the broker before the script exits.

  • Save producer.py.

Why use order_id as the message key?

In a real system, you'd typically key by customer_id or region so that all events for the same entity land in the same partition. This guarantees ordering for that entity. Here you're using order_id to spread messages evenly across partitions and make the distribution visible.

✔️ Awesome, I've got everything!

Great. Make sure you've saved producer.py before moving on.

ⓧ I'd like to double check the full code

import json
import time
import random
from confluent_kafka import Producer

def delivery_report(err, msg):
    if err is not None:
        print(f"Delivery failed: {err}")
    else:
        print(f"Order delivered to partition {msg.partition()} at offset {msg.offset()}")

producer = Producer({'bootstrap.servers': 'localhost:9092'})

order_items = ['laptop', 'phone', 'tablet', 'headphones', 'keyboard', 'monitor']
regions = ['north', 'south', 'east', 'west']

order_id = 0
try:
    while True:
        order_id += 1
        order = {
            'order_id': order_id,
            'item': random.choice(order_items),
            'quantity': random.randint(1, 5),
            'region': random.choice(regions),
            'timestamp': time.time()
        }

        producer.produce(
            'orders',
            key=str(order_id),
            value=json.dumps(order),
            callback=delivery_report
        )
        producer.poll(0)

        print(f"Produced order #{order_id}: {order['item']} x{order['quantity']} ({order['region']})")
        time.sleep(1)

except KeyboardInterrupt:
    print("\nStopping producer...")
finally:
    producer.flush()
    print("All messages flushed.")
Run the producer and observe partition routing

With the script ready, you can start streaming orders. The delivery report callback will show you which partition each message lands in.

  • Run the producer by executing this command:
python producer.py
  • Watch the terminal output. You should see two lines printed for each order: one from the print statement showing the order details, and one from the delivery_report callback showing the partition and offset.
  • Let the producer run for at least 10 seconds so messages spread across all 3 partitions.

What should I see?

Your terminal will print lines like:

Produced order #1: laptop x3 (north)
Order delivered to partition 2 at offset 0
Produced order #2: phone x1 (south)
Order delivered to partition 0 at offset 0
Produced order #3: headphones x5 (east)
Order delivered to partition 1 at offset 0

Notice that different order IDs route to different partitions. The partition assignment comes from hashing the message key. You'll see partitions 0, 1, and 2 all receiving messages.

Not seeing any output?

  • Make sure your Docker Compose stack is still running. If the Kafka container stopped, the producer can't connect to localhost:9092.
  • Check that your virtual environment is activated. If confluent-kafka isn't installed in the active environment, you'll see an ImportError.
  • Verify the orders topic exists. If you skipped Step 2, re-run python admin_topics.py first.

Why isn't my Kafka producer connecting?

Check Kafka UI to see messages distributed across partitions

The terminal output confirms delivery, but Kafka UI gives you a visual breakdown of how messages are distributed across partitions.

  • Switch to your browser with Kafka UI open at http://localhost:8080.
  • Click on **Topics** in the left sidebar.
  • Click on the orders topic to open its detail view.
  • Click the **Messages** tab to browse the messages that have been produced.
  • You should see messages listed with their partition number visible. Messages are spread across partitions 0, 1, and 2.

What am I looking for?

The Messages view shows each message's key, value, partition, and offset. You'll see that messages are not all in one partition. They're distributed across all three, confirming that Kafka's key-based partitioning is working.

  • Leave the producer running in your terminal. You'll need it active for the next step.

Kafka UI showing 0 messages?

  • Click the refresh icon or reload the page. Kafka UI does not always live-update automatically.
  • Make sure the producer is still running in your terminal. If it crashed or you pressed Ctrl+C, restart it.

Why can't I see messages in Kafka UI?

Your producer is streaming orders into all 3 partitions. Next up, you'll build a consumer group that processes these orders in parallel and witness rebalancing in action.

Build the Consumer Group and Observe Rebalancing

Your producer is streaming orders into the orders topic across 3 partitions. Now you need something on the other side to actually process those orders.

In a real system, a single consumer cannot keep up with high-volume traffic. Consumer groups solve this by letting multiple consumers share the workload automatically. Kafka assigns each partition to exactly one consumer in the group, and when consumers join or leave, it triggers a rebalance to redistribute the partitions. In this step, you will see this happen live.

In this step, get ready to:
  • Write a consumer script that joins a consumer group and processes orders.
  • Run multiple consumer instances and observe partition rebalancing in real time.
  • Visualize partition assignments in Kafka UI and watch recovery when a consumer stops.
Write the consumer script

The consumer needs to join a named group, subscribe to the topic, and continuously poll for new messages. Each message arrives with metadata about which partition it came from, which is exactly what you will observe.

  • Create a new file called consumer.py in your project folder.
  • Add the consumer setup code by pasting the following:
import json
import sys
from confluent_kafka import Consumer, KafkaError

# Allow group ID to be passed as a command-line argument
group_id = sys.argv[1] if len(sys.argv) > 1 else 'order-processors'

# Configure the consumer with broker address and group settings
consumer = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': group_id,
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': True
})

# Subscribe to the orders topic
consumer.subscribe(['orders'])

print(f"Consumer started (group: {group_id}). Waiting for orders...")

What does this code do?

This sets up the consumer's identity and connection to Kafka.

  • group.id tells Kafka which consumer group this consumer belongs to. All consumers with the same group ID share the partition workload.
  • auto.offset.reset set to earliest means new consumers start reading from the beginning of the topic if no previous offset is committed.
  • The sys.argv[1] trick lets you override the group ID from the command line, which is useful for testing different group configurations.
  • Below the setup code in consumer.py, add the main processing loop:
# Main loop: continuously poll for new messages
try:
    while True:
        msg = consumer.poll(timeout=1.0)

        if msg is None:
            continue
        if msg.error():
            if msg.error().code() == KafkaError._PARTITION_EOF:
                continue
            else:
                print(f"Error: {msg.error()}")
                break

        # Decode and print the order with its partition info
        order = json.loads(msg.value().decode('utf-8'))
        print(f"[Partition {msg.partition()}] Processing order #{order['order_id']}: "
              f"{order['item']} x{order['quantity']} ({order['region']})")

except KeyboardInterrupt:
    print("\nStopping consumer...")
finally:
    consumer.close()
    print("Consumer closed.")

What does this code do?

The loop polls Kafka once per second for new messages.

  • consumer.poll(timeout=1.0) waits up to 1 second for a message. If nothing arrives, it returns None and the loop continues.
  • KafkaError._PARTITION_EOF means the consumer has reached the end of a partition. This is normal and gets silently skipped.
  • Each message prints its partition number, so you can see exactly which consumer handles which partitions.
  • consumer.close() in the finally block ensures the consumer leaves the group cleanly, triggering an immediate rebalance for other members.
  • Save consumer.py.

Seeing a syntax error when you save?

Make sure the processing loop code starts at the same indentation level as the print statement above it (no extra indent). The try block should be at the top level of the file.

Check that the f-string on the last print line uses matching quotes. The outer quotes are double quotes and the inner dictionary keys use single quotes.

Help me fix a syntax error in my Kafka consumer.py file.

✔️ Awesome, I've got everything!

Great. Double check you have saved consumer.py before moving on.

ⓧ I'd like to double check the full code

import json
import sys
from confluent_kafka import Consumer, KafkaError

group_id = sys.argv[1] if len(sys.argv) > 1 else 'order-processors'

consumer = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': group_id,
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': True
})

consumer.subscribe(['orders'])

print(f"Consumer started (group: {group_id}). Waiting for orders...")

try:
    while True:
        msg = consumer.poll(timeout=1.0)

        if msg is None:
            continue
        if msg.error():
            if msg.error().code() == KafkaError._PARTITION_EOF:
                continue
            else:
                print(f"Error: {msg.error()}")
                break

        order = json.loads(msg.value().decode('utf-8'))
        print(f"[Partition {msg.partition()}] Processing order #{order['order_id']}: "
              f"{order['item']} x{order['quantity']} ({order['region']})")

except KeyboardInterrupt:
    print("\nStopping consumer...")
finally:
    consumer.close()
    print("Consumer closed.")
Run multiple consumers and observe rebalancing

With one consumer running, Kafka assigns all 3 partitions to it. When a second consumer joins the same group, Kafka triggers a rebalance and splits the partitions between them.

  • Make sure your producer is still running in its terminal from the previous step.
  • Open a new terminal tab or window. Activate your virtual environment again and start the first consumer by running:
python consumer.py

You should see output like this, with messages arriving from all 3 partitions:

Consumer started (group: order-processors). Waiting for orders...
[Partition 0] Processing order #42: laptop x2 (north)
[Partition 2] Processing order #43: phone x1 (east)
[Partition 1] Processing order #44: tablet x3 (west)

Why does one consumer get all 3 partitions?

Kafka assigns every partition in a topic to exactly one consumer within a group. With only one consumer in the group, it gets all the work. This is the baseline you will compare against after adding a second consumer.

  • Leave that consumer running. Open a second terminal tab or window.
  • Activate your virtual environment in the new terminal and start a second consumer by running:
python consumer.py

Watch both terminal windows. Within a few seconds, you will see the rebalance happen. The first consumer stops receiving messages from some partitions, and the second consumer picks them up.

For example, Consumer 1 might now only show messages from partition 0, while Consumer 2 handles partitions 1 and 2.

What is rebalancing?

When a consumer joins or leaves a group, Kafka redistributes partitions across all active members. This is called a rebalance. It happens automatically, with no code changes needed.

The key rule is that each partition is assigned to exactly one consumer. With 3 partitions and 2 consumers, one consumer gets 2 partitions and the other gets 1.

Not seeing messages split between consumers?

Give it 5-10 seconds for the rebalance to complete. The consumer group coordinator needs time to detect the new member and reassign partitions.

Make sure both consumers are using the same group ID. Running python consumer.py without arguments defaults to order-processors for both.

Help me debug why my consumers aren't splitting messages.

Visualize partition assignments in Kafka UI

Kafka UI gives you a visual dashboard of how partitions are distributed across your consumer group members. This is how teams monitor consumer health in production.

  • Switch to your browser with Kafka UI open at http://localhost:8080.
  • Click Consumers in the left sidebar.
  • Click on the order-processors group that appears in the list.

You should see a view showing 2 members in the group, with partitions distributed between them.

  • Now go back to one of your consumer terminals and press Ctrl+C to stop that consumer.

Watch the remaining consumer's terminal. Within seconds, it picks up the partitions that the stopped consumer was handling. It now shows messages from all 3 partitions again.

  • Refresh the Consumers page in Kafka UI. The group now shows 1 member with all 3 partitions assigned.

Why does this matter in production?

This automatic failover is what makes Kafka consumer groups so powerful. If a server crashes, its partitions get redistributed to surviving consumers within seconds. No manual intervention needed.

You just witnessed the same mechanism that keeps companies like Uber and Netflix processing millions of events per second without data loss.

Kafka UI not showing the consumer group?

The consumer group only appears after at least one consumer has connected and polled. Make sure your consumer is running and has printed at least one message before checking Kafka UI.

Try refreshing the page. Kafka UI polls the broker periodically and may take a moment to reflect the latest state.

Help me troubleshoot why Kafka UI isn't showing my consumer group.

You have now seen the core Kafka pattern that powers real-time systems at scale. Next up, you will extend this pipeline with a dead letter queue to handle failed orders gracefully.

Secret mission

Add a Dead Letter Queue

What happens when an order fails to process? In production, you can't just crash and lose the message. Your mission: route failed orders to a dedicated dead letter queue so nothing gets lost and the pipeline keeps running.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker containers, so there are no ongoing cloud costs. However, the Kafka broker and Kafka UI containers will continue using system memory and CPU while running. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers: Kafka broker and Kafka UI (running locally).
  • Docker volumes: Kafka data stored in container volumes.
  • Local project files: docker-compose.yml, admin_topics.py, producer.py, consumer.py, and the Python virtual environment.

Keep everything running

No action needed. Choose this if you're still actively experimenting with your Kafka pipeline or want to try different partition counts and consumer group configurations.

  • Your Docker containers will keep running in the background.
  • Kafka UI remains accessible at http://localhost:8080.
  • Your producer and consumer scripts are ready to run again anytime.

Pause - I'll come back to this later

Shut down the Docker containers to free up memory and CPU, but keep everything in place so you can resume quickly.

  • Stop the running producer and consumer scripts by pressing Ctrl+C in each terminal window.
  • Stop the Docker containers by running this command from your project folder:
docker compose stop

When you're ready to come back, restart the containers by running:

docker compose start

Your topics, messages, and consumer group offsets will still be there.

Delete - I don't want to use this again

Remove all Docker resources and local project files to start completely fresh.

  • Stop any running producer or consumer scripts by pressing Ctrl+C in each terminal window.
  • Remove the Docker containers and volumes by running this command from your project folder:
docker compose down -v

What does this command do?

The -v flag tells Docker Compose to remove the named volumes along with the containers. This wipes all Kafka data (topics, messages, offsets) permanently.

  • Confirm the containers are gone by running:
docker compose ps

You should see no containers listed in the output.

  • Deactivate the Python virtual environment by running:
deactivate
  • Navigate to your Desktop and remove the project folder by running:
cd ~/Desktop
rm -rf kafka-orders

Windows users

If you're on Windows, use this command instead to remove the project folder:

cd ~\Desktop
rmdir /s /q kafka-orders

That's a Wrap!

That's a Wrap!

Nice work! You just built a real-time order processing pipeline with Apache Kafka, complete with partitioned topics, parallel consumers, and automatic rebalancing.

You've learned how to:

  • Set up a modern Kafka broker in KRaft mode using Docker Compose and created a multi-partition topic that enables parallel processing.
  • Built a producer that streams order events with partition keys, guaranteeing message ordering per key across partitions.
  • Built a consumer group and witnessed automatic partition assignment and rebalancing as consumers joined and left the group.
  • Secret Mission: Implemented a dead letter queue pattern for graceful error handling, routing failed orders to a separate topic without interrupting the pipeline.

Ready to quiz yourself?