Real-Time Order Pipeline with Kafka

Build a local event-driven order pipeline using Apache Kafka and Python.

Introduction

30 Second Summary

Every time you place an order online, your purchase triggers a chain of events. Payment processing, inventory updates, shipping notifications. Behind the scenes, systems like Apache Kafka handle millions of these events per second without dropping a single one.

In this project, you will build a local event-driven order processing pipeline using Apache Kafka and Python. You'll produce order events, consume them in parallel, and witness how Kafka distributes work across multiple consumers.

What You'll Build

A local order processing pipeline where a Python producer generates realistic order events and publishes them to a partitioned Kafka topic, while consumers in a consumer group process orders in parallel.

By the end of this project, you'll have:

  • A running Kafka broker in KRaft mode inside a Podman container.
  • A Python producer script that generates and publishes realistic order events as JSON to a multi-partition topic.
  • Multiple consumers in a consumer group demonstrating partition-based parallel processing and rebalancing.
  • Secret Mission: Add a dead letter topic that captures malformed orders the consumer cannot process.

Are there any prerequisites?

You should be comfortable writing Python scripts and running basic terminal commands. This project also assumes you already have Podman, PyCharm, and pyenv installed on your machine.

Before We Start

Every e-commerce backend needs a way to handle orders as they arrive, in real time. Apache Kafka is the industry-standard tool for this. It lets services produce and consume streams of events without losing data.

Before you can build your pipeline, you need the right Python version, the Kafka container image, and the Python client library that talks to Kafka. This step gets all three in place.

In this step, get ready to:
  • Install Python 3.12 via pyenv and create a virtual environment.
  • Pull the Apache Kafka container image with Podman.
  • Install the confluent-kafka Python package.
Install Python 3.12 and create a virtual environment

This project requires Python 3.12 because the confluent-kafka library has explicit compatibility fixes for this version. You will use pyenv to install and manage it.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal, and pressing Enter.
  • Check which Python versions you already have by running this command:
pyenv versions

Look at the output and find the tab that matches what you see.

✔️ I see Python 3.12 listed

Python 3.12 is already installed. Move on to creating your project directory below.

ⓧ I see other versions but not 3.12

You have pyenv working but need to add Python 3.12. Install it by running:

pyenv install 3.12

This downloads and compiles the latest Python 3.12.x release. It may take a few minutes.

ⓧ pyenv: command not found

You need to install pyenv first. Follow the official installation instructions at:

https://github.com/pyenv/pyenv#installation

After installing pyenv, restart your terminal and run:

pyenv install 3.12

This downloads and compiles the latest Python 3.12.x release.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir kafka-pipeline
cd kafka-pipeline
  • Set Python 3.12 as the version for this directory by running:
pyenv local 3.12

What does pyenv local do?

This command creates a .python-version file in your project folder. Whenever you cd into this directory, pyenv automatically activates Python 3.12.

  • Create a virtual environment by running:
python -m venv venv
  • Activate the virtual environment by running:
source venv/bin/activate
  • Confirm your Python version by running:
python --version

You should see Python 3.12.x in the output, and your terminal prompt should show (venv) at the start.

Seeing an older Python version?

If python --version shows a version other than 3.12, make sure you ran pyenv local 3.12 inside the kafka-pipeline folder. Then delete the old venv and recreate it:

rm -rf venv
python -m venv venv
source venv/bin/activate

help me fix my Python version in pyenv

  • Open PyCharm by pressing Cmd+Space (macOS) or the Windows key (Windows), typing PyCharm, and pressing Enter.
  • Click Open on the Welcome screen.
  • Navigate to Desktop, select the kafka-pipeline folder, and click Open.
Pull the Apache Kafka container image

Podman needs the Kafka container image downloaded locally before you can run a broker. You will use the official Apache Kafka image in KRaft mode, which is the modern architecture that replaced ZooKeeper in Kafka 4.0.

  • Pull the Apache Kafka 4.2.1 image by running this command:
podman pull docker.io/apache/kafka:4.2.1

Why the docker.io prefix?

Podman supports short image names, but using the fully-qualified reference docker.io/apache/kafka:4.2.1 avoids ambiguity about which registry to pull from. This ensures a predictable download every time.

  • Confirm the image downloaded successfully by running:
podman images | grep kafka

You should see a row containing docker.io/apache/kafka with the tag 4.2.1.

Image pull failed?

Make sure Podman is running. Try podman info to confirm it can connect. If you see a network error, check your internet connection and retry the pull command.

help me fix my Podman image pull

Install the confluent-kafka package

The confluent-kafka library is the production-grade Python client for Kafka. It is maintained by Confluent (the company behind Kafka) and built on the high-performance librdkafka C library.

  • Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt.
  • Install the package by running:
pip install confluent-kafka

Why confluent-kafka over kafka-python?

The confluent-kafka client wraps the C library librdkafka, giving it significantly higher throughput than pure-Python alternatives. It is what production environments use, and knowing this client carries more weight in interviews.

  • Verify the installation by running:
pip show confluent-kafka

You should see package details including Version: 2.14.2 (or a nearby release) in the output.

Install failed with a build error?

The confluent-kafka package includes a compiled C extension. If the install fails, make sure you are running Python 3.12 (not an older version) and that your system has a C compiler available. On macOS, run xcode-select --install if prompted.

help me install confluent-kafka

Your environment is ready. Next up, you will launch a Kafka broker inside that container and create your first topic.

Launch Kafka with Podman

With your Python environment ready, you need something to produce messages to. That something is a Kafka broker.

In this step, you will start a Kafka broker inside a Podman container and create the topic your producer and consumer will use. The broker runs in KRaft mode, which means no ZooKeeper dependency.

In this step, get ready to:
  • Start a single-node Kafka broker in KRaft mode using Podman.
  • Verify the broker is running and healthy.
  • Create an "orders" topic with 4 partitions.
Start the Kafka broker container

A Kafka broker is the server that receives, stores, and serves messages. You will run one inside a Podman container configured entirely through environment variables.

  • Create a named volume to persist Kafka data across container restarts by running this command:
podman volume create kafka-data
  • Start the Kafka broker in KRaft mode by running this command:
podman run -d --name kafka-broker \
  -p 9092:9092 \
  -v kafka-data:/tmp/kraft-combined-logs \
  -e KAFKA_NODE_ID=1 \
  -e KAFKA_PROCESS_ROLES=broker,controller \
  -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
  -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
  -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
  -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \
  -e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
  docker.io/apache/kafka:4.2.1

What does this command do?

This is a single command with many environment variables. Here is what each section does:

  • -d runs the container in the background (detached mode).
  • --name kafka-broker gives your container a memorable name you can reference later.
  • -p 9092:9092 maps port 9092 inside the container to port 9092 on your machine, so your Python code can reach the broker at localhost:9092.
  • -v kafka-data:/tmp/kraft-combined-logs mounts the named volume to the directory where Kafka stores its log data, so data survives container restarts.
  • KAFKA_PROCESS_ROLES=broker,controller tells this single node to act as both the message broker and the cluster controller (combined mode).
  • The replication factor variables are all set to 1 because you only have one broker. In production you would have multiple brokers and higher replication.
  • Confirm the container is running by running:
podman ps

You should see a row for kafka-broker with a status of "Up" and port 9092 listed.

Container not showing as running?

  • Run podman logs kafka-broker to see the startup output. If you see a permission error about /opt/kafka/config, make sure your Podman version is up to date.
  • If port 9092 is already in use, stop whatever process is using it or change the host port mapping (e.g. -p 9093:9092).
  • If the image was not pulled in Step 1, Podman may be trying to pull it now. Wait for the download to complete and retry.

Help me troubleshoot my Kafka broker container not starting in Podman.

Verify the broker is healthy

A running container does not guarantee the broker is fully initialized. Kafka needs a few seconds to format its storage and elect itself as the controller.

  • Wait about 10 seconds after starting the container, then verify the cluster is ready by running:
podman exec -it kafka-broker /opt/kafka/bin/kafka-cluster.sh cluster-id --bootstrap-server localhost:9092

What does this command do?

This runs the kafka-cluster.sh script inside the running container. It asks the broker for its cluster ID. If you get a cluster ID back, the broker is fully operational and ready to accept connections.

You should see output similar to:

Cluster ID: 5L6g3nShT-eMCtK--X86sw

The actual ID value will differ. What matters is that you get a response instead of an error.

Seeing a connection error?

  • The broker may still be starting up. Wait 15-20 seconds and try the command again.
  • Run podman logs kafka-broker and look for a line containing "started" or "ready". If you see repeated exceptions, the environment variables may be misconfigured.

Help me fix a connection error when verifying my Kafka broker cluster ID.

Create the orders topic

Kafka organizes messages into topics. Each topic can be split into multiple partitions, which is how Kafka enables parallel processing. You will create an orders topic with 4 partitions so that multiple consumers can process orders simultaneously in later steps.

  • Create the orders topic by running this command:
podman exec -it kafka-broker /opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --partitions 4 \
  --replication-factor 1

What does this command do?

  • --create tells Kafka to make a new topic.
  • --topic orders names the topic. Your producer and consumer code will reference this exact name.
  • --partitions 4 splits the topic into 4 partitions. Each partition is an ordered log that can be read by a different consumer in parallel.
  • --replication-factor 1 means one copy of the data. With a single broker there is nowhere to replicate to.

You should see:

Created topic orders.
  • Confirm the topic was created correctly by running:
podman exec -it kafka-broker /opt/kafka/bin/kafka-topics.sh \
  --describe \
  --topic orders \
  --bootstrap-server localhost:9092

You should see the topic name, 4 partitions listed (0 through 3), a replication factor of 1, and the leader set to broker 1 for each partition.

Topic creation failed?

  • If you see "Topic 'orders' already exists", a previous attempt already created it. Run the --describe command above to confirm it has 4 partitions.
  • If you see a connection refused error, the broker may not be fully started yet. Wait a few more seconds and retry.

Help me fix a topic creation error in my Kafka broker container.

Why 4 partitions?

Four partitions gives you room to experiment with consumer groups later. With 4 partitions and 2 consumers, each consumer handles 2 partitions. Scale up to 3 consumers and one gets 2 while the others get 1.

This partition-to-consumer mapping is the core mechanism Kafka uses for parallel processing. You will see it in action in Step 5.

Your Kafka broker is running and your topic is ready. Next up, you will write a Python producer that generates order events and publishes them to this topic.

Build the Order Producer

Your Kafka broker is running and the orders topic is ready to receive messages. Now it's time to build the application that sends data into that topic.

In this step, you'll write a Python producer that generates fake e-commerce orders and publishes them as JSON messages. Each order lands in one of the 4 partitions, and a delivery callback confirms exactly where every message ended up.

In this step, get ready to:
  • Write helper functions that generate order events and report delivery status.
  • Configure and run a Kafka producer that publishes 20 orders to the orders topic.
  • Observe delivery callbacks confirming each message's partition and offset.
Write the producer helper functions

The producer needs two helper functions before you can wire up the main loop. One generates realistic order data. The other handles delivery confirmations from the broker.

  • In PyCharm, create a new file called producer.py in your project directory.
  • Add the imports and the delivery report callback by pasting this code:
import json
import random
import time
from confluent_kafka import Producer


# Called once per message to confirm delivery or report failure
def delivery_report(err, msg):
    if err is not None:
        print(f"Delivery failed: {err}")
    else:
        print(
            f"Order delivered to topic={msg.topic()} "
            f"partition={msg.partition()} offset={msg.offset()}"
        )

What does this code do?

The delivery_report function is a callback. Kafka's producer calls it automatically for every message to tell you whether delivery succeeded or failed.

On success, it prints the topic name, partition number, and offset (the message's position within that partition). On failure, it prints the error. This gives you per-message confirmation without blocking your code.

  • Below the delivery_report function, add the order generator:
# Build a randomized order with product, quantity, and price
def generate_order(order_id):
    products = ["Laptop", "Phone", "Tablet", "Keyboard", "Monitor", "Headphones"]
    return {
        "order_id": f"ORD-{order_id:04d}",
        "product": random.choice(products),
        "quantity": random.randint(1, 5),
        "price": round(random.uniform(9.99, 999.99), 2),
        "timestamp": time.time(),
    }

What does this code do?

The generate_order function creates a dictionary that looks like a real e-commerce order. It picks a random product, quantity, and price each time you call it.

The order_id field uses zero-padded formatting (ORD-1001, ORD-1002) so orders are easy to identify in the consumer output later.

  • Save producer.py.

Seeing a red underline on the import?

If PyCharm highlights from confluent_kafka import Producer with an error, make sure your project interpreter is set to the virtual environment where you installed confluent-kafka. Check under Settings > Project > Python Interpreter.

Help me fix my Python interpreter in PyCharm

Add the main function with producer configuration

The main function is where you create the Producer instance, connect it to the broker, and loop through 20 orders. Each order is serialized to JSON and published to the orders topic.

  • Below the generate_order function in producer.py, add the main function and entry point:
def main():
    # Connect to the local Kafka broker
    conf = {
        "bootstrap.servers": "localhost:9092",
        "client.id": "order-producer",
    }

    producer = Producer(conf)

    print("Starting order producer... (Ctrl+C to stop)")
    order_id = 1000

    try:
        for i in range(20):
            order_id += 1
            order = generate_order(order_id)
            # Publish the order as a JSON-encoded message
            producer.produce(
                topic="orders",
                key=order["order_id"],
                value=json.dumps(order).encode("utf-8"),
                callback=delivery_report,
            )
            # Trigger any pending delivery callbacks without blocking
            producer.poll(0)
            time.sleep(0.5)
    except KeyboardInterrupt:
        print("\nStopping producer...")
    finally:
        # Wait for all in-flight messages to be delivered
        remaining = producer.flush(timeout=5)
        print(f"Producer stopped. {remaining} messages still in queue.")


if __name__ == "__main__":
    main()

What does this code do?

  • The conf dictionary tells the producer where to find the broker (localhost:9092) and sets a client ID for identification.
  • The producer.produce() call is asynchronous. It queues the message for batching and transmission without waiting for the broker's response.
  • The producer.poll(0) call triggers any pending delivery report callbacks from previous messages. It returns instantly without blocking.
  • The producer.flush(timeout=5) in the finally block waits up to 5 seconds for all remaining messages to be delivered before the script exits.

Why use a message key?

The key=order["order_id"] parameter determines which partition the message goes to. Kafka hashes the key and assigns the message to a partition based on that hash. Orders with the same key always land in the same partition, preserving ordering for that key.

  • Save producer.py.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before running it in the next substep.

ⓧ I'd like to double check the full code

Here is the complete producer.py file. Compare your code against this:

import json
import random
import time
from confluent_kafka import Producer


def delivery_report(err, msg):
    if err is not None:
        print(f"Delivery failed: {err}")
    else:
        print(
            f"Order delivered to topic={msg.topic()} "
            f"partition={msg.partition()} offset={msg.offset()}"
        )


def generate_order(order_id):
    products = ["Laptop", "Phone", "Tablet", "Keyboard", "Monitor", "Headphones"]
    return {
        "order_id": f"ORD-{order_id:04d}",
        "product": random.choice(products),
        "quantity": random.randint(1, 5),
        "price": round(random.uniform(9.99, 999.99), 2),
        "timestamp": time.time(),
    }


def main():
    conf = {
        "bootstrap.servers": "localhost:9092",
        "client.id": "order-producer",
    }

    producer = Producer(conf)

    print("Starting order producer... (Ctrl+C to stop)")
    order_id = 1000

    try:
        for i in range(20):
            order_id += 1
            order = generate_order(order_id)
            producer.produce(
                topic="orders",
                key=order["order_id"],
                value=json.dumps(order).encode("utf-8"),
                callback=delivery_report,
            )
            producer.poll(0)
            time.sleep(0.5)
    except KeyboardInterrupt:
        print("\nStopping producer...")
    finally:
        remaining = producer.flush(timeout=5)
        print(f"Producer stopped. {remaining} messages still in queue.")


if __name__ == "__main__":
    main()
Run the producer and observe partition delivery

With the code complete, you can now run the producer and watch it publish 20 orders to the orders topic in real time.

  • Open the terminal in PyCharm (click Terminal at the bottom of the window).
  • Run the producer by executing:
python producer.py

You should see 20 delivery confirmations printed over about 10 seconds, each showing the topic, partition, and offset:

Starting order producer... (Ctrl+C to stop)
Order delivered to topic=orders partition=2 offset=0
Order delivered to topic=orders partition=0 offset=0
Order delivered to topic=orders partition=3 offset=0
Order delivered to topic=orders partition=1 offset=0
...
Producer stopped. 0 messages still in queue.

What should I see?

Each line confirms one order was successfully delivered. The partition numbers (0-3) will vary because Kafka assigns them based on the message key hash. The offsets increment within each partition as more messages arrive.

The final line says 0 messages still in queue. This means flush() confirmed every message was delivered before the script exited.

Seeing a connection error?

If you see a message about failing to connect to localhost:9092, make sure your Kafka container is still running. Check with:

podman ps

If kafka-broker is not listed, restart it with podman start kafka-broker.

Help me troubleshoot my Kafka producer connection error

Your producer is working. It just pushed 20 order events into Kafka, spread across multiple partitions. Next up, you'll build a consumer that reads those orders back out and processes them.

Build the Order Consumer

Your producer is publishing order events to the orders topic across 4 partitions. The messages are sitting in Apache Kafka, waiting to be read.

Now you need the other side of the pipeline. A consumer subscribes to a topic, pulls messages, and processes them. In this step, you will build a consumer that reads every order and displays which partition and offset each message came from.

In this step, get ready to:
  • Write a Python consumer that subscribes to the orders topic.
  • Process incoming order messages and display partition/offset metadata.
  • Observe how a single consumer reads from all 4 partitions.
Write a Python consumer that subscribes to the orders topic

The producer fires and forgets. The consumer is the opposite. It needs to keep running, continuously polling for new messages. It also needs to shut down gracefully when you press Ctrl+C, so Kafka knows to reassign its partitions to other consumers.

  • In PyCharm, create a new file called consumer.py in your project directory (the same folder as producer.py).
  • Add the imports and graceful shutdown handler by pasting this code into consumer.py:
import json
import signal
import sys
from confluent_kafka import Consumer, KafkaError


# Flag that controls the polling loop
running = True


def signal_handler(sig, frame):
    # Set running to False so the loop exits cleanly
    global running
    running = False

What does this code do?

The running flag controls whether the consumer keeps polling for messages. When you press Ctrl+C, Python sends a SIGINT signal.

The signal_handler function catches that signal and sets running = False, which stops the loop on the next iteration. This gives the consumer a chance to close its connection cleanly.

  • Below the signal handler, add the main() function with the consumer configuration and topic subscription:
def main():
    # Register the signal handler for graceful shutdown
    signal.signal(signal.SIGINT, signal_handler)

    # Consumer configuration
    conf = {
        "bootstrap.servers": "localhost:9092",
        "group.id": "order-processors",
        "auto.offset.reset": "earliest",
        "enable.auto.commit": True,
    }

    # Create and subscribe the consumer to the orders topic
    consumer = Consumer(conf)
    consumer.subscribe(["orders"])

    print("Starting order consumer... (Ctrl+C to stop)")
    print("Waiting for messages...\n")

What does this configuration do?

  • bootstrap.servers tells the consumer where to find the Kafka broker.
  • group.id assigns this consumer to a consumer group called order-processors. Kafka tracks which messages each group has already read.
  • auto.offset.reset set to earliest means the consumer starts reading from the very first message when it joins a topic for the first time.
  • enable.auto.commit set to True means Kafka automatically records the consumer's position (its offset) so it can resume where it left off after a restart.
Process incoming order messages and display partition/offset metadata

The consumer needs a loop that continuously polls for new messages. Each message arrives with metadata showing which partition it came from and its offset within that partition.

  • Below the print statements inside main(), add the polling loop and the if __name__ block:
    try:
        while running:
            # Poll for a message with 1-second timeout
            msg = consumer.poll(timeout=1.0)

            if msg is None:
                continue

            # Handle any errors from the broker
            if msg.error():
                if msg.error().code() == KafkaError._PARTITION_EOF:
                    print(
                        f"Reached end of partition {msg.partition()} "
                        f"at offset {msg.offset()}"
                    )
                else:
                    print(f"Consumer error: {msg.error()}")
                continue

            # Decode the JSON message and display order details
            order = json.loads(msg.value().decode("utf-8"))
            print(
                f"[Partition {msg.partition()}] "
                f"Offset {msg.offset()} | "
                f"{order['order_id']} - {order['product']} "
                f"x{order['quantity']} = ${order['price']:.2f}"
            )
    finally:
        # Always close the consumer to commit final offsets
        consumer.close()
        print("\nConsumer stopped.")


if __name__ == "__main__":
    main()

What does this polling loop do?

  • consumer.poll(timeout=1.0) waits up to 1 second for a message. If nothing arrives, it returns None and the loop tries again.
  • KafkaError._PARTITION_EOF is not a real error. It signals that the consumer has read all available messages in a partition. The loop continues polling for new ones.
  • For each valid message, the code decodes the JSON payload and prints the partition number, offset, and order details.
  • The finally block calls consumer.close() which commits final offsets and triggers a group rebalance so other consumers can pick up the partitions.
  • Save consumer.py.

Seeing a syntax error?

Make sure the try: block is indented inside main() (one level of indentation). The if __name__ block should be at the top level with no indentation.

help me fix a syntax error in my consumer.py file

✔️ Awesome, I've got everything!

Great. Double check you have saved consumer.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete consumer.py file:

import json
import signal
import sys
from confluent_kafka import Consumer, KafkaError


# Flag that controls the polling loop
running = True


def signal_handler(sig, frame):
    # Set running to False so the loop exits cleanly
    global running
    running = False


def main():
    # Register the signal handler for graceful shutdown
    signal.signal(signal.SIGINT, signal_handler)

    # Consumer configuration
    conf = {
        "bootstrap.servers": "localhost:9092",
        "group.id": "order-processors",
        "auto.offset.reset": "earliest",
        "enable.auto.commit": True,
    }

    # Create and subscribe the consumer to the orders topic
    consumer = Consumer(conf)
    consumer.subscribe(["orders"])

    print("Starting order consumer... (Ctrl+C to stop)")
    print("Waiting for messages...\n")

    try:
        while running:
            # Poll for a message with 1-second timeout
            msg = consumer.poll(timeout=1.0)

            if msg is None:
                continue

            # Handle any errors from the broker
            if msg.error():
                if msg.error().code() == KafkaError._PARTITION_EOF:
                    print(
                        f"Reached end of partition {msg.partition()} "
                        f"at offset {msg.offset()}"
                    )
                else:
                    print(f"Consumer error: {msg.error()}")
                continue

            # Decode the JSON message and display order details
            order = json.loads(msg.value().decode("utf-8"))
            print(
                f"[Partition {msg.partition()}] "
                f"Offset {msg.offset()} | "
                f"{order['order_id']} - {order['product']} "
                f"x{order['quantity']} = ${order['price']:.2f}"
            )
    finally:
        # Always close the consumer to commit final offsets
        consumer.close()
        print("\nConsumer stopped.")


if __name__ == "__main__":
    main()
Observe how a single consumer reads from all 4 partitions

When only one consumer exists in a consumer group, Kafka assigns it all available partitions. Your single consumer will read from all 4 partitions of the orders topic.

  • Run the consumer by executing this command in your terminal:
python consumer.py
  • Watch the output. You should see all 20 order messages printed, each prefixed with its partition number and offset.

Your output will look similar to this (partition assignments and order details will vary):

Starting order consumer... (Ctrl+C to stop)
Waiting for messages...

[Partition 0] Offset 0 | ORD-1005 - Laptop x3 = $499.99
[Partition 0] Offset 1 | ORD-1012 - Monitor x1 = $349.50
[Partition 2] Offset 0 | ORD-1001 - Phone x2 = $199.99
[Partition 3] Offset 0 | ORD-1003 - Tablet x1 = $89.99
...

What should I see?

Every line starts with [Partition X] followed by Offset Y. You should see messages from partitions 0, 1, 2, and 3. The offset numbers start at 0 for each partition and increase sequentially.

After all 20 messages are consumed, you will see "Reached end of partition" messages. This is normal. The consumer is waiting for new messages to arrive.

  • Press Ctrl+C to stop the consumer. You should see "Consumer stopped." printed as the signal handler triggers a graceful shutdown.

Not seeing any messages?

  • Make sure the Kafka broker is still running. Run podman ps and confirm kafka-broker is in the list.
  • Check that you ran producer.py in a previous step. The consumer needs messages in the topic to read.
  • Verify that group.id is set to order-processors and auto.offset.reset is set to earliest. If you previously ran the consumer and it committed offsets, Kafka will not re-read the same messages. Stop the consumer, then re-run producer.py to add fresh messages.

help me debug why my Kafka consumer is not receiving any messages

Your consumer reads from all 4 partitions on its own. Next up, you will run multiple consumers in the same group and watch Kafka divide the partitions between them.

Scale with Consumer Groups

In the previous step, your single consumer read from all 4 partitions of the orders topic. That works fine at low volume, but real systems process thousands of events per second.

Kafka solves this with consumer groups. Multiple consumers sharing the same group.id automatically split the partitions between them. In this step, you will run multiple instances of your consumer and watch Kafka distribute the work.

In this step, get ready to:
  • Run multiple consumer instances in the same consumer group.
  • Observe Kafka rebalancing partitions across the consumers.
  • Stop one consumer and watch the remaining consumer pick up its partitions.
Run multiple consumer instances in the same consumer group

Your consumer.py already uses the group ID order-processors. Every instance you start with the same group ID joins that consumer group. Kafka then splits the 4 partitions across however many consumers are active.

You will open three terminal tabs in PyCharm: two for consumers and one for the producer.

  • In PyCharm, click the + icon in the Terminal panel at the bottom to open a new terminal tab.
  • Repeat this to create a second new terminal tab (you should now have three terminal tabs total).

Why three terminal tabs?

You need two tabs for running two consumer instances simultaneously, and a third tab for the producer. This lets you watch both consumers processing messages at the same time.

Each new terminal tab needs the virtual environment activated so it can find the confluent-kafka package.

  • In the first new terminal tab, activate your virtual environment by running:
source .venv/bin/activate
  • Switch to the second new terminal tab and activate the virtual environment there too by running the same command:
source .venv/bin/activate

Now start the first consumer instance.

  • In the first new terminal tab, start a consumer by running:
python consumer.py
  • You should see Starting order consumer... (Ctrl+C to stop) followed by Waiting for messages... in the output.
  • Switch to the second new terminal tab and start a second consumer by running:
python consumer.py
  • You should see the same startup message in this tab. Both consumers are now running in the same order-processors group.

Consumer exits immediately?

Make sure you activated the virtual environment in that terminal tab. You should see (.venv) at the start of your prompt.

Also confirm the Kafka broker is still running. Check with podman ps in your original terminal tab.

Help me debug why my consumer exits immediately.

Observe Kafka rebalancing partitions across the consumers

With two consumers waiting, Kafka has already assigned partitions between them. Each consumer gets an exclusive subset of the 4 partitions. To see this in action, you need to send some messages.

  • Switch to your original terminal tab (where you previously ran the producer).
  • Run the producer to send 20 new orders by running:
python producer.py
  • While the producer runs, switch between your two consumer terminal tabs to observe their output.

You should see that each consumer only receives messages from specific partitions. For example, Consumer 1 might show [Partition 0] and [Partition 1] messages, while Consumer 2 shows [Partition 2] and [Partition 3] messages.

What is rebalancing?

When a consumer joins or leaves a group, Kafka triggers a rebalance. During a rebalance, Kafka redistributes partition ownership so that each partition is assigned to exactly one consumer in the group.

With 4 partitions and 2 consumers, Kafka assigns 2 partitions to each. If you had 4 consumers, each would get 1 partition. A 5th consumer would sit idle because there are no partitions left to assign.

Partition count limits parallelism

The number of partitions in a topic sets the maximum number of consumers that can read in parallel within a single consumer group. This is why choosing the right partition count matters in production.

Stop one consumer and watch the remaining consumer pick up its partitions

In production, consumers crash or get redeployed all the time. Kafka handles this gracefully. When a consumer leaves the group, its partitions are reassigned to the remaining consumers.

  • Switch to one of your consumer terminal tabs.
  • Press Ctrl+C to stop that consumer.
  • You should see Consumer stopped. printed in that tab.

Now send more messages so you can see the remaining consumer handling all partitions.

  • Switch to your original terminal tab and run the producer again:
python producer.py
  • Switch to the remaining consumer's terminal tab.
  • You should now see messages from all 4 partitions ([Partition 0] through [Partition 3]) appearing in this single consumer's output.

What just happened?

When you stopped the first consumer, Kafka detected it had left the order-processors group and triggered a rebalance. The remaining consumer was reassigned all 4 partitions.

This is exactly how Kafka provides fault tolerance in production. If a server goes down, the other consumers in the group automatically take over its workload with no manual intervention.

Still only seeing messages from 2 partitions?

The rebalance takes a few seconds to complete. Wait about 10 seconds after stopping the first consumer before running the producer again.

If you still see only 2 partitions, the stopped consumer may not have closed cleanly. Make sure you pressed Ctrl+C (which triggers the graceful consumer.close() call) instead of closing the terminal tab directly.

Help me figure out why my consumer isn't picking up all partitions after rebalancing.

  • Press Ctrl+C in the remaining consumer's terminal tab to stop it.

Why does this matter in interviews?

Consumer group rebalancing is one of the most common Kafka interview questions. You can now explain from hands-on experience how Kafka distributes partitions, scales processing horizontally, and handles consumer failures automatically.

You just witnessed one of Kafka's most powerful features in action. Your consumer group scaled up to share the work, then recovered automatically when one member went down. Next up, you will tackle a real-world production pattern in the Secret Mission.

Secret mission

Add a Dead Letter Topic

What happens when a consumer receives a message it cannot process? In production, you need a safety net. Build a dead letter queue that captures malformed orders so they can be inspected and reprocessed later.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. The Kafka container uses about 500MB of RAM while running. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Podman container kafka-broker (running Kafka broker on port 9092).
  • Podman volume kafka-data (stores Kafka log data).
  • Python virtual environment with confluent-kafka installed.
  • Project Python files (producer.py, consumer.py, producer_with_errors.py, consumer_with_dlq.py, dlq_consumer.py).

Keep everything running

No action needed. Choose this if you're still actively experimenting with your Kafka pipeline or want to try different producer/consumer configurations.

  • Your kafka-broker container continues running in the background using about 500MB of RAM.
  • Your virtual environment and Python files remain on your Desktop, ready to use.
  • The Kafka topics (orders and dead-letter-orders) retain all previously produced messages.

Pause - I'll come back to this later

Stop the Kafka container to free up memory. Your data and configuration are preserved so you can pick up where you left off.

  • Stop the Kafka container by running:
  • Deactivate your virtual environment by running:
podman stop kafka-broker
deactivate

When you're ready to resume, restart the container and reactivate your environment:

podman start kafka-broker

All your topics, partitions, and message data are preserved in the kafka-data volume.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the container, volume, and all Kafka data permanently.

  • Remove the Kafka container by running:
podman rm -f kafka-broker
  • Remove the Kafka data volume by running:
podman volume rm kafka-data
  • Remove the Kafka container image by running:
podman rmi docker.io/apache/kafka:4.2.1
  • Deactivate your virtual environment (if still active) by running:
deactivate
  • Delete your project folder from the Desktop by running:
cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]

On Windows, use this command instead:

cd ~\Desktop
rmdir /s /q [[PROJECT_FOLDER="your project folder name"]]

After these steps, all Kafka resources and project files are completely removed from your machine.

That's a Wrap!

That's a Wrap!

Nice work! You just built a real-time event-driven order processing pipeline using Apache Kafka and Python. You now have hands-on experience with the core distributed systems concepts that come up in backend and data engineering interviews.

You've learned how to:

  • Run a Kafka broker locally in KRaft mode using Podman, and create a multi-partition topic for structured JSON messages with delivery confirmation.
  • Build a producer and consumer with the confluent-kafka Python client, processing orders in parallel through consumer groups with automatic partition assignment.
  • Demonstrate consumer group rebalancing by scaling consumers up and down, watching Kafka redistribute partitions in real time.
  • Implemented a dead letter topic pattern that gracefully handles malformed messages and routes them to a separate topic with error metadata.

Ready to quiz yourself?