Build a Real-Time Order Tracker

Stream food delivery order events with Apache Kafka running locally in Docker.

Introduction

30 Second Summary

Every time you order food delivery, your phone tracks its journey from "order placed" to "out for delivery" to "arrived." Behind the scenes, each status update is an event that flows through a streaming system to reach your screen in real time.

In this project, you will build a real-time order tracking system using Apache Kafka. You'll produce order events from Python, consume them live in another terminal, and prove that Kafka never loses a message by stopping your consumer and watching it catch up.

What You'll Build

You'll run two terminals side by side: one firing off food delivery order events, the other catching them live as they stream in, and then you'll stop the consumer, let messages pile up, restart it, and watch it replay every single missed event.

By the end of this project, you'll have:

  • A local Kafka broker launched with a single Docker command. No config files, no setup headaches.
  • A Python producer that generates realistic order events with customer names, menu items, and delivery statuses streaming into a Kafka topic.
  • A Python consumer that processes events in real time and resumes exactly where it left off after a restart. This is Kafka's message retention in action.
  • Secret Mission: A second independent consumer group that tracks order analytics without affecting the main consumer.

Are there any prerequisites?

You'll need basic Python knowledge (functions, dictionaries, loops) and comfort running commands in the terminal. No prior Kafka or event streaming experience is required.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Environment

Before you can stream any events, you need two things: a running Kafka broker to hold your messages, and a Python environment with the right client library to talk to it.

Docker gives you a one-command Kafka server with zero configuration. Modern Kafka runs in KRaft mode, which means no separate ZooKeeper service, no YAML files to edit, no multi-container setups. One container does it all.

In this step, get ready to:
  • Start a Kafka broker in Docker.
  • Create a Python project with the Kafka client library.
  • Verify your broker and client are both working.
Start a Kafka broker in Docker

Kafka traditionally required a multi-service setup. With Docker and the official Apache Kafka image, you get a fully functional broker from a single command.

  • Start the Kafka container by running this command:
docker run -d --name=kafka -p 9092:9092 apache/kafka:4.3.1

What does this command do?

  • -d runs the container in the background so you keep your terminal free.
  • --name=kafka gives the container a name you can reference later.
  • -p 9092:9092 maps Kafka's default port to your machine so Python can connect to it.
  • apache/kafka:4.3.1 is the official Apache Kafka image. It runs in KRaft mode by default, so there is no ZooKeeper needed.

Docker prints a long container ID string. That confirms the container started successfully.

Before you run the next command, what do you expect to see if Kafka is running and accepting connections?

  • Verify the broker is running by running this command:
docker exec -ti kafka /opt/kafka/bin/kafka-cluster.sh cluster-id --bootstrap-server :9092

You should see output like Cluster ID: 5L6g3nShT-eMCtK--X86sw (your ID will be different). This confirms Kafka is running and accepting connections on port 9092.

Seeing an error?

  • If you see docker: Error response from daemon, make sure Docker Desktop is running. Check your system tray or menu bar for the Docker whale icon.
  • If you see a name conflict (name kafka is already in use), remove the old container first with docker rm -f kafka, then re-run the command above.
  • If the cluster-id command hangs, the broker may still be starting up. Wait 5 seconds and try again.

help me troubleshoot my Docker Kafka container

Set up the Python project

With the broker running, you need a Python project with the right client library to produce and consume messages.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir kafka-order-tracker
cd kafka-order-tracker

Your terminal prompt now shows kafka-order-tracker as the current directory.

  • Create and activate a virtual environment by running these commands:
python3 -m venv .venv
source .venv/bin/activate

Your terminal prompt now shows (.venv) at the beginning. This confirms you are inside the virtual environment.

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python, so different projects can use different library versions without conflicting.

  • Install the Kafka client library by running:
pip install confluent-kafka==2.15.0

You will see pip downloading and installing the package. When it finishes, you will see Successfully installed confluent-kafka-2.15.0 at the end of the output.

Why confluent-kafka?

The confluent-kafka library is built on librdkafka (a high-performance C library). It is the officially supported Python client from Confluent, and it is significantly faster than pure-Python alternatives.

Install failing?

  • If you see error: externally-managed-environment, you are not inside your virtual environment. Run source .venv/bin/activate first, then retry the install.
  • If you see a build error mentioning librdkafka, make sure your Python version is 3.8 or higher by running python3 --version.

help me fix my confluent-kafka pip install error

Verify the setup

You have a Kafka broker running in Docker and the Python client library installed. One quick test confirms everything is connected and ready.

What do you expect to see if confluent-kafka installed correctly and Python can import it?

  • Test the installation by running this command:
python3 -c "from confluent_kafka import Producer; print('confluent-kafka ready')"

You should see confluent-kafka ready printed in your terminal. This confirms Python can import the Kafka library and you are ready to start building.

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is active. You should see (.venv) in your terminal prompt.
  • If not, run source .venv/bin/activate and try the command again.

help me fix my confluent-kafka import error

Your Kafka broker is running and your Python environment is ready. Next up, you will use Kafka's built-in CLI tools to produce and consume your first messages without writing any code.

Produce Your First Message with the CLI

Your Kafka broker is running and your Python environment is ready. Now you want to see messages actually flowing through Kafka before you write any code.

The built-in CLI tools let you produce and consume messages instantly. In this step, you will create a topic, send messages into it, and read them back from a separate terminal to prove the broker works end-to-end.

In this step, get ready to:
  • Create a Kafka topic with 3 partitions.
  • Produce messages into the topic via the console producer.
  • Consume those messages from a second terminal and confirm they arrive instantly.
Create a topic for order events

In Kafka, a topic is a named channel where messages live. Producers write to a topic and consumers read from it. You need to create one before any messages can flow.

  • Create the order-events topic by running this command:
docker exec -ti kafka /opt/kafka/bin/kafka-topics.sh --create --topic order-events --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

What does this command do?

  • docker exec -ti kafka runs a command inside your running Kafka container.
  • kafka-topics.sh --create tells Kafka to create a new topic.
  • --partitions 3 splits the topic into 3 partitions so messages can be distributed.
  • --replication-factor 1 keeps one copy of each message (appropriate for a single-broker local setup).
  • Verify the topic was created by describing it:
docker exec -ti kafka /opt/kafka/bin/kafka-topics.sh --describe --topic order-events --bootstrap-server localhost:9092

You should see a summary line showing PartitionCount: 3 and ReplicationFactor: 1, followed by one line for each of the 3 partitions listing their leader and replicas.

Topic not created?

  • Make sure the Docker container named kafka is still running. Check with docker ps and look for it in the list.
  • If you see "broker not available", the container may still be starting up. Wait 10 seconds and try again.

Help me troubleshoot why my Kafka topic creation is failing.

Produce and consume messages via CLI

Now that your topic exists, you can push messages into it from one terminal and read them from another. This proves the full pipeline works before you write any Python.

  • Start the console producer by running this command:
docker exec -ti kafka /opt/kafka/bin/kafka-console-producer.sh --topic order-events --bootstrap-server localhost:9092

What happens when you run this?

This command takes over your terminal and shows a > prompt. Every line you type and press Enter becomes a message sent to the order-events topic. The terminal stays in this mode until you exit with Ctrl+C.

  • Type these messages, pressing Enter after each one:
Order 001 placed
Order 002 placed
Order 003 placed

Leave the producer running

Do not press Ctrl+C yet. Keep this terminal open with the producer active while you open a second terminal for the consumer.

  • Open a new terminal window. Press Cmd+Space (macOS) or the Windows key (Windows), type Terminal, and press Enter to open it.

Before you run the consumer, predict what you think will happen. Will you see the 3 messages you already typed, or will you only see new messages from this point forward?

  • Start the console consumer by running this command in your new terminal:
docker exec -ti kafka /opt/kafka/bin/kafka-console-consumer.sh --topic order-events --from-beginning --bootstrap-server localhost:9092

You should see all three messages appear immediately in this terminal. The --from-beginning flag tells the consumer to read every message retained in the topic, starting from offset 0.

  • Switch back to your first terminal (where the producer is still running).
  • Type another message, such as Order 004 preparing, and press Enter.
  • Switch to the consumer terminal and confirm the new message appeared instantly.

That is real-time event streaming. Your producer and consumer are completely independent processes connected only by the topic.

Why is this powerful?

The producer does not know or care that a consumer exists. The consumer does not need the producer to be running. Kafka sits in the middle, retaining all messages, so each side works at its own pace.

This decoupled architecture is what makes event streaming different from direct service-to-service calls.

Not seeing messages in the consumer?

  • Make sure you included the --from-beginning flag. Without it, the consumer only shows messages produced after it started.
  • Confirm both terminals are targeting the same topic name order-events. A typo creates a different topic silently.
  • Check that the Kafka container is still running with docker ps.

Help me figure out why my Kafka console consumer isn't showing messages.

  • Stop the consumer with Ctrl+C.
  • Stop the producer with Ctrl+C.

You have just proven that Kafka can accept, store, and deliver messages end-to-end. Next up, you will replace these CLI tools with Python code that generates realistic order events automatically.

Build the Python Producer

In the previous step, you proved the broker works end-to-end using CLI tools. You typed messages by hand and watched them appear in a consumer terminal.

Real systems don't type messages by hand. They publish structured data from application code. In this step, you'll build a Python producer that generates realistic food delivery order events as JSON and streams them into your order-events topic.

In this step, get ready to:
  • Create a Python producer script that publishes structured order events to Kafka.
  • Run the producer and observe delivery reports confirming each message's partition and offset.
Create the producer script

Your producer needs three pieces: a callback that confirms delivery, a function that generates order data, and a main loop that ties them together. You'll build each piece one at a time.

  • In your code editor, create a new file called producer.py inside the kafka-order-tracker folder.
  • Add the imports and delivery report callback by pasting this code:
import json
import time
import random
from confluent_kafka import Producer

def delivery_report(err, msg):
    # Called once per message to confirm delivery or report failure
    if err is not None:
        print(f"Delivery failed: {err}")
    else:
        print(f"Delivered to {msg.topic()} [{msg.partition()}] @ offset {msg.offset()}")

What does this code do?

The imports bring in JSON serialization, time utilities, randomness, and the confluent-kafka Producer class.

The delivery_report function is a callback. Kafka calls it for every message to tell you whether delivery succeeded or failed. On success, it prints the topic, partition number, and offset where the message landed.

  • Below the delivery_report function, add the order event generator:
def create_order_event(order_id):
    # Sample data pools for realistic order events
    customers = ["Alice", "Bob", "Charlie", "Diana", "Eve"]
    items = ["Pizza Margherita", "Pad Thai", "Burger Deluxe", "Sushi Platter", "Tacos Al Pastor"]
    statuses = ["placed", "preparing", "out_for_delivery", "delivered"]

    return {
        "order_id": f"ORD-{order_id:04d}",
        "customer": random.choice(customers),
        "item": random.choice(items),
        "status": random.choice(statuses),
        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }

What does this code do?

This function builds a dictionary representing one order event. Each call picks a random customer, menu item, and status from the sample pools.

The order_id uses zero-padded formatting (ORD-0001) so IDs sort naturally. The timestamp captures the current UTC time in ISO 8601 format.

  • Save producer.py.
  • Test that your order event function works by running this command in your terminal:
python3 -c "from producer import create_order_event; import json; print(json.dumps(create_order_event(1), indent=2))"

You should see a JSON object with all five fields: order_id, customer, item, status, and timestamp.

Seeing a SyntaxError or ImportError?

  • Make sure you saved the file before running the command. Python imports the file from disk.
  • Check that your terminal is in the kafka-order-tracker directory where producer.py lives.
  • Confirm your virtual environment is active. You should see (.venv) at the start of your terminal prompt.

Help me debug my producer.py import error

  • Now add the main loop that produces events to Kafka. Paste this below your create_order_event function in producer.py:
def main():
    # Connect to the local Kafka broker
    producer = Producer({"bootstrap.servers": "localhost:9092"})

    print("Producing order events...")
    for i in range(1, 11):
        event = create_order_event(i)
        # Send the event to Kafka with order_id as the key
        producer.produce(
            topic="order-events",
            key=event["order_id"],
            value=json.dumps(event),
            callback=delivery_report,
        )
        # Trigger delivery report callbacks
        producer.poll(0)
        time.sleep(0.5)

    # Wait for any outstanding messages to be delivered
    producer.flush()
    print("All events produced.")

if __name__ == "__main__":
    main()

What does this code do?

  • Producer({"bootstrap.servers": "localhost:9092"}) creates a producer connected to your local Kafka broker.
  • The loop generates 10 events and sends each to the order-events topic. The key determines which partition the message goes to.
  • producer.poll(0) triggers any pending delivery report callbacks without blocking.
  • producer.flush() blocks until every queued message has been delivered. This guarantees nothing is lost when the script exits.
  • Save producer.py.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file. You're about to run it.

ⓧ I'd like to double check the full code

Scroll up and confirm your producer.py contains these three chunks in order:

  • Imports (json, time, random, Producer) and the delivery_report function.
  • The create_order_event function with customer, item, and status pools.
  • The main() function and if __name__ == "__main__" block at the bottom.
Run the producer and observe delivery reports

Your producer is ready. Before you run it, think about what you expect to see. You have a topic with 3 partitions and you're sending 10 messages with different order IDs as keys.

  • Before running the script, predict: will all 10 messages land on the same partition, or will they spread across the 3 partitions? Why?
  • Run the producer by executing this command:
python3 producer.py

You should see 10 delivery reports printed one by one, each confirming the topic, partition, and offset. The events distribute across the 3 partitions because Kafka hashes the message key to determine partition assignment.

Why do messages spread across partitions?

Kafka uses the message key to decide which partition gets each message. Different keys hash to different partitions. Messages with the same key always land on the same partition, guaranteeing order for that key.

Since each order has a unique order_id as its key, Kafka spreads them across all 3 partitions. This is how Kafka scales: more partitions means more parallelism.

Seeing a connection error?

  • Verify the Kafka container is still running with docker ps. You should see a container named kafka in the list.
  • If the container stopped, restart it with docker start kafka and wait a few seconds before retrying.
  • Make sure your bootstrap.servers in the script is set to localhost:9092 with no extra spaces.

Help me debug my Kafka producer connection

Your producer is streaming structured order events into Kafka. Next up, you'll build a consumer that reads these events in real time and proves Kafka's message retention by surviving a restart.

Build the Python Consumer

Your producer is streaming order events into Kafka, but nobody is reading them yet. The consumer completes the pipeline by subscribing to the topic and processing each event as it arrives.

More importantly, this step reveals Kafka's killer feature: message retention. You'll stop the consumer, let dozens of orders pile up, restart it, and watch it resume exactly where it left off.

In this step, get ready to:
  • Create consumer.py that reads order events from Kafka in real time.
  • Demonstrate message retention by stopping and restarting the consumer.
Create the consumer script

A Kafka consumer connects to the broker, joins a consumer group, and subscribes to one or more topics. The broker tracks how far each consumer group has read using offsets, so it knows exactly where to resume if the consumer restarts.

  • Create a new file called consumer.py in your kafka-order-tracker folder and paste the following setup code:
import json
from confluent_kafka import Consumer, KafkaError

def main():
    # Connect to Kafka as part of the order-tracker consumer group
    consumer = Consumer({
        "bootstrap.servers": "localhost:9092",
        "group.id": "order-tracker-group",
        "auto.offset.reset": "earliest",
    })

    # Subscribe to the order-events topic
    consumer.subscribe(["order-events"])
    print("Listening for order events... (Ctrl+C to stop)")

What does this code do?

The Consumer class from confluent-kafka connects to your local broker at port 9092.

  • group.id tells Kafka which consumer group this consumer belongs to. Kafka tracks offsets per group, so it knows where each group last read.
  • auto.offset.reset set to earliest means if this group has no saved offset, it starts reading from the very first message in the topic.
  • subscribe() registers interest in the order-events topic. The broker assigns partitions to this consumer automatically.
  • Save consumer.py. Next, you'll add the message-reading loop that completes the script.
  • In consumer.py, add this poll loop directly below the print statement, still inside the main() function:
    # Poll for messages in a loop
    try:
        while True:
            msg = consumer.poll(1.0)

            # Skip if no message arrived within the timeout
            if msg is None:
                continue
            # Handle errors (PARTITION_EOF is benign)
            if msg.error():
                if msg.error().code() == KafkaError._PARTITION_EOF:
                    continue
                print(f"Consumer error: {msg.error()}")
                break

            # Decode and display the order event
            event = json.loads(msg.value().decode("utf-8"))
            print(
                f"[Partition {msg.partition()} | Offset {msg.offset()}] "
                f"Order {event['order_id']}: {event['customer']} - "
                f"{event['item']} ({event['status']})"
            )
    except KeyboardInterrupt:
        print("\nShutting down consumer...")
    finally:
        # Always close to commit final offsets
        consumer.close()

if __name__ == "__main__":
    main()

What does this code do?

consumer.poll(1.0) waits up to 1 second for a new message. If nothing arrives, it returns None and the loop continues.

  • KafkaError._PARTITION_EOF is a benign signal meaning the consumer has caught up to the end of a partition. You safely ignore it.
  • Each message is decoded from bytes to a Python dictionary, then printed with its partition number and offset.
  • consumer.close() in the finally block ensures offsets are committed back to Kafka when you stop the script with Ctrl+C.
  • Save consumer.py.
  • Run the consumer to read the messages your producer already sent earlier by running this command:
python3 consumer.py

You should see all the order events from your earlier producer run printed out, each showing its partition number and offset.

  • Press Ctrl+C to stop the consumer after you see the messages.

Not seeing any messages?

  • Make sure your Docker container named kafka is still running. Check with docker ps and restart it with docker start kafka if needed.
  • Confirm your virtual environment is activated (you should see (.venv) in your terminal prompt).
  • Check for typos in group.id or bootstrap.servers. The values must match exactly.

Why isn't my consumer receiving messages from Kafka?

✔️ Awesome, I've got everything!

Great. Double check you've saved consumer.py and move on to the next substep.

ⓧ I'd like to double check the full code

Your consumer.py should contain these sections in order:

  • The import lines for json, Consumer, and KafkaError.
  • The main() function starting with Consumer({...}) configuration.
  • consumer.subscribe(["order-events"]) and the print statement.
  • The try/while True poll loop with msg.error() handling and event decoding.
  • except KeyboardInterrupt and finally: consumer.close().
  • The if __name__ == "__main__": main() guard at the bottom.

Scroll up and confirm each chunk is present in your file, in order.

Demonstrate message retention and replay

Kafka does not delete messages after a consumer reads them. Messages stay in the topic for a configurable retention period (7 days by default). The broker tracks each consumer group's progress using offsets, so if a consumer disconnects and comes back, it picks up right where it left off.

You'll prove this by stopping the consumer, letting new messages pile up, and restarting it.

  • Open a second terminal window. Make sure your virtual environment is activated in the new terminal by running:
cd ~/Desktop/kafka-order-tracker && source .venv/bin/activate
  • In your first terminal, start the consumer:
python3 consumer.py
  • In your second terminal, run the producer:
python3 producer.py

Watch your consumer terminal. You should see the 10 new order events appearing in real time as the producer sends them, each printed with its partition and offset.

  • In the consumer terminal, press Ctrl+C to stop the consumer.
  • In the producer terminal, run the producer again to send 10 more events:
python3 producer.py

The consumer is offline. Those 10 new events are sitting in Kafka with nobody reading them.

Before you restart the consumer, think about this: will it start from the very beginning of the topic, or will it pick up only the messages it missed?

  • Restart the consumer in your first terminal:
python3 consumer.py

The consumer picks up only the 10 messages it missed. It does not re-read the events it already processed before you stopped it. Kafka tracked the consumer group's last committed offset, and the consumer resumed from that exact point.

Why didn't it start from the beginning?

The first time your consumer ran, auto.offset.reset: earliest told it to start from offset 0 because no saved offset existed for order-tracker-group.

After reading messages, consumer.close() committed the offsets back to Kafka. On restart, the broker says "your group already read up to offset X" and resumes from there. This is what makes Kafka reliable for real systems. Consumers can crash and restart without losing their place.

  • Press Ctrl+C to stop the consumer.

Consumer re-reading all messages from the start?

  • Make sure your group.id is exactly order-tracker-group. If you changed it between runs, Kafka treats it as a brand new group with no saved offsets.
  • Confirm consumer.close() is inside the finally block. Without it, offsets are never committed and the consumer starts fresh each time.

Why is my Kafka consumer re-reading all messages instead of resuming?

You now have a working producer-consumer pipeline with message retention built in. Next up, you'll extend this system with a second consumer group that independently processes the same events for analytics.

Secret mission

Add a Second Consumer Group for Analytics

In real systems, an analytics service and a notification service both need the same order events, but they process them at different speeds. Can you add a second consumer group that independently processes events and computes running statistics?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Docker, so there are no ongoing cloud costs.

Resources you used:

  • A Docker container named kafka running Apache Kafka 4.3.1 (uses minimal resources when idle, no cost).
  • The kafka-order-tracker/ project directory on your Desktop, including the Python virtual environment and scripts.

Keep everything running

No action needed. Choose this if you're still actively experimenting with Kafka or plan to extend the project soon.

  • The kafka container continues running in the background. It uses minimal CPU and memory when idle.
  • Your project files remain in kafka-order-tracker/ on your Desktop.
  • To re-run the producer or consumer later, activate the virtual environment again with source .venv/bin/activate from inside the project folder.

Pause - I'll come back to this later

Stop the Docker container to free up memory, but keep everything in place so you can restart later.

  • Stop the Kafka container by running:
docker stop kafka
  • When you're ready to pick up again, restart it by running:
docker start kafka
  • Your project files stay on disk. Kafka retains all messages from previous sessions.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the Kafka container, all stored messages, and your local project files.

  • Stop and remove the Kafka container by running:
docker stop kafka && docker rm kafka
  • Delete the project directory by running:
cd ~/Desktop
rm -rf kafka-order-tracker

On Windows?

Use this command instead to remove the project directory:

cd ~\Desktop
rmdir /s /q kafka-order-tracker
  • Verify everything is cleaned up by running:
docker ps -a | grep kafka
  • You should see no output. The container is gone, and all Kafka data is removed.

That's a Wrap!

That's a Wrap!

Nice work! You just built a real-time event streaming system from scratch using Apache Kafka and Python.

You've learned how to:

  • Run a Kafka broker locally with a single Docker command in KRaft mode, and produce and consume messages through the CLI to prove the pipeline works end-to-end.
  • Build a Python producer that streams structured JSON order events across 3 partitions with delivery confirmation callbacks.
  • Build a Python consumer that reads events in real time and resumes from exactly where it left off after a restart, proving Kafka's message retention and offset tracking.
  • Secret Mission: Prove Kafka's consumer group model by running two independent consumers that each get their own copy of every event.

Ready to quiz yourself?