Stream Order Events with Apache Kafka

Build a real-time order pipeline with Kafka producers and consumers in Java.

Introduction

30 Second Summary

Every time you order food, hail a ride, or stream a show, thousands of events fire behind the scenes. Something has to catch every single one, route it to the right system, and make sure nothing gets lost.

In this project, you will build a real-time order tracking pipeline using Apache Kafka and Java. You will produce, consume, and track order events to see how event streaming ensures nothing gets lost.

What You'll Build

You will run a producer in one terminal, watch two independent consumers process the same events in real time from separate terminals, and pull up a live dashboard showing every message, offset, and consumer group in your pipeline.

By the end of this project, you'll have:

  • A live Kafka UI dashboard where you can click into topics, scroll through JSON order payloads, and see exactly which partition and offset every message landed on.
  • A Java producer and consumer that you run from separate terminals. You produce 10 order events in one and watch them print in real time in the other.
  • Two independent consumer groups processing the same event stream for different purposes, demonstrating the fan-out pattern that powers real production systems.
  • An optional Secret Mission to push your skills further.

Are there any prerequisites?

You should have basic Java experience and the following tools already installed: JDK 17+, Docker Desktop, and VS Code. This project walks you through installing Apache Maven if you don't have it yet.

Before We Start

Before we start writing code, let's get clear on what you're building and why it matters.

You're about to create a real-time order tracking pipeline with Apache Kafka, and understanding the full picture now will make every step ahead click into place.

Set Up Your Project and Start Kafka

You know what you are building, but you have nothing running yet. No build tool, no project folder, and no Kafka broker to stream events through.

In this step, you will install Apache Maven (the Java build tool), create the project skeleton, and spin up a local Kafka cluster using Docker. By the end, you will have a live Kafka UI dashboard in your browser.

In this step, get ready to:
  • Install Apache Maven and verify your JDK.
  • Create the Maven project with Kafka dependencies.
  • Start a Kafka broker and Kafka UI dashboard with Docker Compose.
Verify your JDK and install Maven

Kafka's Java client library requires JDK 17 or higher. Maven is the build tool that will download Kafka's library and compile your Java code.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Confirm your JDK version by running this command:
java -version

You should see a version number starting with 17 or higher (for example, 17.0.12 or 21.0.4).

Now check whether Maven is already installed by running this command:

mvn -version

✔️ I see a version number

Maven is ready to go. Continue to the next substep.

ⓧ Command not found

Install Maven using the instructions for your operating system.

macOS

The simplest way to install Maven on macOS is through Homebrew. This is the method recommended on the official Maven installation page.

  • Install Maven by running this command:
brew install maven
Seeing 'brew: command not found'?

Homebrew is not installed on your Mac. Visit brew.sh and run the install command on their homepage. Then rerun brew install maven.

Windows

The simplest way to install Maven on Windows is through Scoop, a command-line package manager. This is the method recommended on the official Maven installation page.

  • If you have Scoop installed, install Maven by running this command in PowerShell:
scoop install main/maven

If you do not have Scoop, follow the manual approach instead.

  • Open https://maven.apache.org/download.cgi in your browser.
  • Under Apache Maven 3.9.16, download the Binary zip archive (the link ending in -bin.zip).
  • Extract the zip to a permanent location, such as C:\Program Files\apache-maven-3.9.16.
  • Press the Windows key and type Environment Variables.
  • Click Edit the system environment variables.
  • Click Environment Variables.
  • Under User variables, select Path and click Edit.
  • Click New and paste the path to Maven's bin folder (for example, C:\Program Files\apache-maven-3.9.16\bin).
  • Click OK on all open dialogs to save.
  • Close and reopen your terminal so the new PATH takes effect.
  • After installing, verify Maven by running this command in a new terminal:
mvn -version

You should see a version number and details about Maven's installation, including the Java version it detected.

What is Maven?

Apache Maven is a build tool for Java projects. It manages your project's dependencies (libraries like the Kafka client), compiles your source code, and packages it for execution.

You define your dependencies in a single file called pom.xml, and Maven downloads everything your project needs automatically.

Create the project structure and pom.xml

Maven projects follow a standard directory layout. You will create the folder structure now and then add the build configuration file that tells Maven which libraries to download.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and the Maven source directory by running these commands:

macOS

mkdir kafka-order-tracker
cd kafka-order-tracker
mkdir -p src/main/java/com/kafka/ordertracker

Windows

mkdir kafka-order-tracker
cd kafka-order-tracker
mkdir src\main\java\com\kafka\ordertracker
  • Confirm the folder was created by running:
ls src/main/java/com/kafka/ordertracker

The directory should exist and be empty. If you see an error, double-check the mkdir command above.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the kafka-order-tracker folder, and click Open.

Seeing a trust prompt?

If VS Code asks whether you trust the authors of the files in this folder, click Yes, I trust the authors.

  • Create a new file in the root of your kafka-order-tracker folder by clicking the New File icon in the VS Code Explorer sidebar.
  • Name the file pom.xml and press Enter.
  • Paste the following into pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.kafka</groupId>
    <artifactId>order-tracker</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <!-- Official Apache Kafka Java client library -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>4.3.1</version>
        </dependency>
        <!-- Simple logging implementation for Kafka's internal logs -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>2.0.16</version>
        </dependency>
    </dependencies>
</project>

What does this file do?

The pom.xml file is Maven's project configuration. It tells Maven two key things.

  • kafka-clients 4.3.1 is the official Apache Kafka Java library. This is what lets your Java code send and receive events from a Kafka broker.
  • slf4j-simple provides a basic logging backend. Without it, Kafka's internal logs would print a warning to your console.
  • Save pom.xml by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Open the VS Code integrated terminal by pressing Ctrl+` (both platforms) or clicking Terminal in the top menu bar, then New Terminal.
  • Download the project dependencies and verify the build by running:
mvn compile

Maven will download the Kafka client library and SLF4J. After a minute or so, you should see a line that reads BUILD SUCCESS.

Seeing BUILD FAILURE?

  • Make sure pom.xml is saved and sits in the root of your kafka-order-tracker folder, not inside a subfolder.
  • Check that your terminal's working directory is kafka-order-tracker. VS Code's integrated terminal opens there automatically if you opened the folder correctly.
  • If Maven cannot resolve dependencies, confirm you have an internet connection. Maven needs to download JARs from a remote repository.

Help me debug my Maven build failure.

✔️ Awesome, I've got everything!

Great. Double check you have saved pom.xml and your terminal shows BUILD SUCCESS.

ⓧ I'd like to double check the full code

Compare your pom.xml against this complete version:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.kafka</groupId>
    <artifactId>order-tracker</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>4.3.1</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>2.0.16</version>
        </dependency>
    </dependencies>
</project>
Start Kafka and Kafka UI with Docker Compose

Docker Compose lets you define multiple containers in a single file and start them with one command. You will spin up two containers: a Kafka broker and a Kafka UI dashboard.

  • Create a new file in the root of your kafka-order-tracker folder by clicking the New File icon in the Explorer sidebar.
  • Name the file docker-compose.yml and press Enter.
  • Paste the following into docker-compose.yml:
services:
  kafka:
    image: apache/kafka:4.3.1
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8080:8080"
    depends_on:
      - kafka
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
      DYNAMIC_CONFIG_ENABLED: "true"

What does this file do?

This Compose file defines two services that run as Docker containers on your machine.

  • kafka runs the official Apache Kafka 4.3.1 image in KRaft mode (no ZooKeeper required). It listens on port 9092 so your Java code on the host can connect.
  • kafka-ui provides a visual web dashboard at port 8080 where you can inspect topics, messages, and consumer groups in real time.

Why KRaft mode?

Kafka 4.x runs entirely in KRaft mode, which means it manages its own metadata without needing a separate ZooKeeper cluster. This simplifies your setup from three containers to just one Kafka container.

  • Save docker-compose.yml.
  • Make sure Docker Desktop is running on your machine.
  • Start both containers by running this command in your VS Code terminal:
docker compose up -d

Docker will pull the Kafka and Kafka UI images (this may take a minute on the first run) and start both containers in the background.

What does the -d flag do?

The -d flag runs the containers in detached mode. They run in the background so your terminal stays free for other commands.

Kafka needs about 15 seconds to fully initialize after the containers start. Before you check, what do you think you will see when you open the Kafka UI dashboard?

  • Wait about 15 seconds for Kafka to start up.
  • Open your browser and navigate to http://localhost:8080.

You should see the Kafka UI dashboard with a connected cluster named local. This is your visual control panel for everything happening inside Kafka.

Page not loading?

  • Make sure Docker Desktop is running. If it is not, start it and rerun docker compose up -d.
  • Check that both containers are running by running docker ps in your terminal. You should see two containers: kafka and kafka-ui.
  • If port 8080 is already in use by another application, stop that application first, then rerun the Docker command.
  • Wait a full 30 seconds and try refreshing the page. Kafka can take longer on slower machines.

Help me debug why Kafka UI is not loading at localhost:8080.

✔️ Awesome, I've got everything!

Great. Double check you have saved docker-compose.yml and both containers are running.

ⓧ I'd like to double check the full code

Compare your docker-compose.yml against this complete version:

services:
  kafka:
    image: apache/kafka:4.3.1
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8080:8080"
    depends_on:
      - kafka
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
      DYNAMIC_CONFIG_ENABLED: "true"

Your development environment is ready. You have a Java project wired up to the Kafka client library and a live Kafka broker waiting for events. Next up, you will write a producer that sends order events into Kafka and see them appear in the dashboard.

Produce Your First Order Events

Your Kafka broker is running and the Kafka UI dashboard is live. Right now, though, nothing is flowing through the pipeline.

In this step, you'll write your first Kafka producer in Java. It will send 10 order events into a Kafka topic, and you'll see them appear in the dashboard as real, inspectable messages.

In this step, get ready to:
  • Write a Java producer that sends order events to Kafka.
  • Compile and run the producer to stream 10 events.
  • Verify the events arrived by inspecting them in Kafka UI.
Write the order producer

A Kafka producer is any application that sends events into a Kafka topic. Your producer will generate 10 order events and push each one into a topic called order-events.

  • In the VS Code Explorer sidebar, navigate to src/main/java/com/kafka/ordertracker.
  • Right-click the ordertracker folder and select New File.
  • Name the file OrderProducer.java and press Enter.
  • Set up the producer's imports and Kafka connection by pasting this code into the new file:
package com.kafka.ordertracker;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;
import java.util.concurrent.ExecutionException;

public class OrderProducer {

    public static void main(String[] args) throws ExecutionException, InterruptedException {
        // Configure connection to the local Kafka broker
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    }
}

What does this code do?

  • BOOTSTRAP_SERVERS_CONFIG tells the producer where to find your Kafka broker. localhost:9092 matches the port you mapped in docker-compose.yml.
  • KEY_SERIALIZER_CLASS_CONFIG and VALUE_SERIALIZER_CLASS_CONFIG tell Kafka how to convert your Java strings into bytes for storage. Every Kafka message is stored as raw bytes, so a serializer is required.
  • Save OrderProducer.java.
  • Confirm the file appears in the Explorer sidebar under ordertracker. You should see OrderProducer.java listed.

The skeleton connects to Kafka, but it doesn't send anything yet. Time to add the producer loop that creates and sends 10 order events.

  • In OrderProducer.java, find the line that says:
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
  • Delete the two closing } braces below that line, then paste the following code in their place:
        // Create a producer and send 10 order events
        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            String[] items = {"Laptop", "Headphones", "Keyboard", "Mouse", "Monitor"};
            String[] statuses = {"PLACED", "CONFIRMED", "SHIPPED"};

            for (int i = 1; i <= 10; i++) {
                // Build a unique order ID and select an item and status
                String orderId = "ORD-" + String.format("%03d", i);
                String item = items[i % items.length];
                String status = statuses[i % statuses.length];
                // Format the order data as a JSON string
                String value = String.format(
                    "{\"orderId\":\"%s\",\"item\":\"%s\",\"quantity\":%d,\"status\":\"%s\"}",
                    orderId, item, (i % 3) + 1, status
                );

                // Send the order event to the "order-events" topic
                ProducerRecord<String, String> record =
                    new ProducerRecord<>("order-events", orderId, value);

                // Wait for Kafka to confirm delivery and print the result
                RecordMetadata metadata = producer.send(record).get();
                System.out.printf("Sent: %s -> partition %d, offset %d%n",
                    orderId, metadata.partition(), metadata.offset());
            }
            System.out.println("All orders sent successfully!");
        }
    }
}

What does this code do?

  • try (KafkaProducer...) creates a producer that automatically closes when the block finishes. This is Java's try-with-resources pattern.
  • The loop generates 10 orders, each with a unique ID like ORD-001, an item name, a quantity, and a status. These are formatted as a JSON string.
  • producer.send(record).get() sends the event and waits for Kafka to confirm it was stored. The .get() call makes this synchronous so you can see each event's partition and offset as it lands.
  • Save OrderProducer.java.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file, then move on to compile and run it.

ⓧ I'd like to double check the full code

Scroll up and confirm your OrderProducer.java has both chunks in order:

  • The package declaration, imports, class declaration, and Properties configuration (chunk 1).
  • The try (KafkaProducer...) block with the order loop, send logic, and closing braces (chunk 2).

Make sure the try block sits inside the main method, right after the three props.put() lines. The file should end with three closing } braces.

Compile and run the producer

Before you can run the producer, Maven needs to compile your Java code and gather the Kafka dependency JARs into a single folder.

  • In your terminal, make sure you are inside the kafka-order-tracker folder.
  • Compile the project and copy all dependency JARs by running this command:
mvn compile dependency:copy-dependencies
  • You should see BUILD SUCCESS at the end of the output.

Seeing BUILD FAILURE?

  • Check that your terminal's current directory is kafka-order-tracker (the folder containing pom.xml).
  • Look for a "cannot find symbol" error. This usually means a typo in an import or class name in OrderProducer.java. Double check your imports match the code above exactly.
  • Why is my Maven build failing?

Now it's time to run the producer and send events to Kafka. Before you run it, what do you think will happen? Will you see 10 separate confirmation lines, or one summary at the end?

  • Run the producer by running this command:

macOS

java -cp "target/classes:target/dependency/*" com.kafka.ordertracker.OrderProducer

Windows

java -cp "target\classes;target\dependency\*" com.kafka.ordertracker.OrderProducer

You should see 10 lines printed, each confirming a sent order with its partition and offset. The output ends with All orders sent successfully!.

What are partitions and offsets?

Every Kafka topic is split into one or more partitions. Each partition is an ordered, append-only log of events.

An offset is the position number of a specific event within its partition. The first event gets offset 0, the next gets offset 1, and so on. Kafka uses offsets to track exactly where each consumer left off.

Seeing a connection refused error?

  • Make sure your Docker containers are still running. Open a new terminal and run docker ps. You should see both kafka and kafka-ui listed.
  • If Kafka just started, it may still be initializing. Wait about 15 seconds and try running the producer again.
  • Why can't my Kafka producer connect to the broker?
Inspect events in Kafka UI

The terminal output confirms events were sent, but seeing them stored inside Kafka is the real proof. The Kafka UI dashboard lets you browse every message in every topic.

  • Switch back to your browser and open http://localhost:8080 (the Kafka UI from earlier).
  • Click Topics in the left sidebar.
  • Click the order-events topic in the list.
  • Click the Messages tab.

Before you look at the results, how many messages do you expect to see listed? What format will the values be in?

You should see all 10 order events listed. Each message shows its key (like ORD-001), its JSON value with the order ID, item, quantity, and status, its partition number, and its offset.

Why is this dashboard so useful?

In production, teams use dashboards like this to verify that events are flowing, debug message formats, and check that topics have the right number of partitions. You just used it to confirm your pipeline works before writing a single line of consumer code.

Don't see the order-events topic?

  • Make sure the producer ran successfully (you saw All orders sent successfully! in the terminal). If not, go back and re-run the producer.
  • Try refreshing the Kafka UI page. Topics sometimes take a moment to appear after they are created.
  • Why is my Kafka topic not showing up in the Kafka UI dashboard?

Your pipeline has its first data flowing. You have 10 order events sitting in Kafka, ready to be consumed. Next up, you'll build a consumer to read those events. But fair warning: your first consumer is going to miss every single one of them.

Build a Consumer and Watch It Miss Events

Your producer is working. 10 order events are stored in Kafka and visible in the dashboard.

Now you need the other half of the pipeline. A consumer reads events from a topic and processes them. In this step, you'll write your first Kafka consumer and point it at the order-events topic.

In this step, get ready to:
  • Write a Java consumer that subscribes to your order-events topic.
  • Run the consumer and observe what happens when it connects.
Write OrderConsumer.java

A Kafka consumer subscribes to one or more topics and polls for new records. Every consumer belongs to a consumer group, which Kafka uses to track what events have been read.

  • Create a new file called OrderConsumer.java in src/main/java/com/kafka/ordertracker/ (the same folder as OrderProducer.java).
  • Paste the following code to set up the consumer's imports and configuration:
package com.kafka.ordertracker;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class OrderConsumer {

    public static void main(String[] args) {
        // Configure the consumer connection settings
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-processing-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

What does this configuration do?

  • BOOTSTRAP_SERVERS_CONFIG connects to the same local broker your producer uses.
  • GROUP_ID_CONFIG assigns this consumer to a group called "order-processing-group". Kafka tracks reading progress per group.
  • StringDeserializer converts the raw bytes from Kafka back into strings. This is the mirror of the StringSerializer your producer uses.

The next chunk adds the polling loop that reads events from the topic.

  • Below the configuration code (after the last line you just pasted), add the subscribe and poll loop:
        // Subscribe to the order-events topic and start polling
        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("order-events"));
            System.out.println("Listening for orders on 'order-events'...");

            // Continuously poll for new events every second
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("[Partition %d | Offset %d] Key: %s, Value: %s%n",
                        record.partition(), record.offset(), record.key(), record.value());
                }
            }
        }
    }
}

What does this code do?

  • consumer.subscribe() tells Kafka this consumer wants to read from the order-events topic.
  • consumer.poll(Duration.ofMillis(1000)) asks Kafka for any new records, waiting up to 1 second before returning empty.
  • The while (true) loop keeps the consumer alive and polling continuously, printing every record's partition, offset, key, and value.
  • Save OrderConsumer.java.
  • Compile your project by running this command in the terminal:
mvn compile
  • You should see BUILD SUCCESS in the output.

Seeing a compilation error?

  • Make sure the file is saved in exactly src/main/java/com/kafka/ordertracker/OrderConsumer.java. The folder path must match the package declaration.
  • Check that the closing braces at the bottom of the file are correct. You should have three closing braces: one for the try block, one for the main method, and one for the class.

Help me fix my OrderConsumer.java compilation error.

✔️ Awesome, I've got everything!

Great. Make sure you've saved OrderConsumer.java and saw BUILD SUCCESS from the compile.

ⓧ I'd like to double check the full code

Scroll up and confirm your file contains both chunks in order:

  • Chunk 1: The package declaration, imports, class declaration, and the four props.put() configuration lines.
  • Chunk 2: The try block with consumer.subscribe(), the while (true) poll loop, and the three closing braces.
Run the consumer and observe the output

Your consumer is compiled and your order-events topic already has 10 messages sitting in it from Step 2.

Before you run the consumer, think about this: you produced 10 events, and this consumer subscribes to the same topic. Do you think it will print all 10 events?

  • Run the consumer using the command for your operating system:

macOS

java -cp "target/classes:target/dependency/*" com.kafka.ordertracker.OrderConsumer

Windows

java -cp "target\classes;target\dependency\*" com.kafka.ordertracker.OrderConsumer

The terminal prints Listening for orders on 'order-events'... and then nothing happens. No order events appear. The consumer just sits there, polling in silence.

Why doesn't the consumer see the 10 events?

This is not a bug. Kafka has a setting called auto.offset.reset that controls where a brand-new consumer group starts reading. The default value is "latest", which means the consumer only sees events produced AFTER it joins.

Your 10 events were already in the topic before this consumer group existed. From Kafka's perspective, those events are in the past. The consumer is waiting for future events that haven't arrived yet.

  • Switch to your browser and open the Kafka UI at http://localhost:8080.
  • Click Topics, then click order-events, then click Messages.

All 10 events are still right there in Kafka. The data is not lost. Your consumer just started reading from the wrong position.

  • Return to the terminal where the consumer is running and press Ctrl+C to stop it.

Consumer didn't connect at all?

  • Make sure your Kafka Docker container is still running. Run docker ps and check that the kafka container appears in the list.
  • If you see a connection refused error, Kafka may still be starting up. Wait 15 seconds and try again.

Help me debug my Kafka consumer connection issue.

You just hit one of the most common Kafka surprises. Your events are safely stored, but your consumer started reading from the wrong position. In the next step, you'll fix this with a single configuration change and learn how Kafka's offset tracking guarantees no event is ever lost.

Fix the Consumer and Master Offset Tracking

In the last step, your consumer connected to Kafka and sat there silently. 10 order events are sitting in the topic (you can see them in Kafka UI), but your consumer ignored every one of them.

The culprit is Kafka's auto.offset.reset setting, which defaults to "latest". That means a new consumer group only sees events produced after it joins. To fix this, you need to configure offset tracking, the mechanism Kafka uses to remember exactly where each consumer left off in a topic.

In this step, get ready to:
  • Fix your consumer's offset configuration so it reads events from the beginning of the topic.
  • Run the fixed consumer and see all 10 order events appear.
  • Prove that Kafka remembers your consumer's position by stopping, producing more events, and resuming without replaying old ones.
Fix OrderConsumer.java with auto.offset.reset

Your consumer's configuration is missing a key property. Without auto.offset.reset set explicitly, Kafka defaults to "latest", which tells a new consumer group to skip everything already in the topic.

  • In VS Code, switch back to OrderConsumer.java in the src/main/java/com/kafka/ordertracker directory.
  • Find the line that sets GROUP_ID_CONFIG:
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-processing-group");
  • Change "order-processing-group" to "order-processing-group-v2", so the line looks like this:
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-processing-group-v2");

Why a new group ID?

Your old consumer group ("order-processing-group") already committed its offset at the end of the topic. Even if you add auto.offset.reset, Kafka ignores that setting for groups that already have a stored offset.

By using a fresh group ID, Kafka treats this as a brand-new consumer group with no stored offsets. The "earliest" setting then kicks in and tells it to start reading from the very first event.

  • Now find the line that sets VALUE_DESERIALIZER_CLASS_CONFIG:
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
  • Add this new line directly below it:
// Tell Kafka to start reading from the very first event in the topic
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

What does auto.offset.reset do?

This property controls where a new consumer group starts reading when it has no previously committed offset.

  • "latest" (the default): skip all existing events and only read future ones.
  • "earliest": start from the very first event still stored in the topic.
  • Save OrderConsumer.java.

✔️ Awesome, I've got everything!

Great. Double check you have saved OrderConsumer.java with both changes before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm your OrderConsumer.java has both of these changes in the properties configuration section:

  • GROUP_ID_CONFIG is set to "order-processing-group-v2" (not the old "order-processing-group").
  • A new line AUTO_OFFSET_RESET_CONFIG set to "earliest" appears after the VALUE_DESERIALIZER_CLASS_CONFIG line.
Run the fixed consumer and see all events
  • Compile the updated consumer by running this command in your terminal (from the kafka-order-tracker folder):
mvn compile

Before you run the consumer, think about what you expect to see. You changed auto.offset.reset to "earliest" and gave the consumer a fresh group ID. Will you see all 10 events this time, or nothing again?

  • Run the consumer by running this command:

macOS

java -cp "target/classes:target/dependency/*" com.kafka.ordertracker.OrderConsumer

Windows

java -cp "target/classes;target/dependency/*" com.kafka.ordertracker.OrderConsumer

All 10 order events now print to the terminal with their partition numbers and offsets. Your consumer is reading from the very beginning of the topic because this is a brand-new consumer group with no stored offsets.

What should I see?

Your terminal shows 10 lines of output, each starting with a partition and offset, followed by the order ID key and the JSON payload. The consumer then continues polling, waiting for new events.

Still seeing no events?

  • Make sure you changed the group ID to "order-processing-group-v2". If you kept the old group ID, Kafka uses the already-committed offset and skips to the end.
  • Confirm you added AUTO_OFFSET_RESET_CONFIG set to "earliest". Without it, the default is "latest".
  • Check that you saved the file and re-ran mvn compile before running the consumer.

Why is my Kafka consumer still not showing any events?

  • Leave the consumer running for now. You will need it in the next substep.
Demonstrate stop-and-resume offset tracking

Reading all 10 events proves the "earliest" fix works. But does Kafka actually remember where your consumer left off? Time to prove it.

  • In the terminal where the consumer is running, press Ctrl+C to stop it.

What happens when you stop a consumer?

When the consumer shuts down, Kafka stores the consumer group's last committed offset. This is the position marker that tells Kafka exactly which event the consumer read last. When the consumer restarts, Kafka picks up from that exact spot.

  • Run the producer again to send 10 more events into the topic by running this command:

macOS

java -cp "target/classes:target/dependency/*" com.kafka.ordertracker.OrderProducer

Windows

java -cp "target/classes;target/dependency/*" com.kafka.ordertracker.OrderProducer

You should see 10 more "Sent: ORD-xxx" lines. The order-events topic now holds 20 messages total.

Now think about what will happen when you restart the consumer. The topic has 20 events. Your consumer already read the first 10. Will it replay all 20, or only the 10 new ones?

  • Restart the consumer by running this command:

macOS

java -cp "target/classes:target/dependency/*" com.kafka.ordertracker.OrderConsumer

Windows

java -cp "target/classes;target/dependency/*" com.kafka.ordertracker.OrderConsumer

Only the 10 NEW events appear. Kafka committed your consumer's offset after reading the first 10, so it resumed exactly where it left off. No event was replayed and no event was lost.

Why does this matter in production?

This is exactly how production systems guarantee reliable event processing. If a consumer crashes at 2 AM, it restarts and picks up right where it stopped. No duplicate processing, no skipped events.

  • Press Ctrl+C to stop the consumer.

Seeing all 20 events instead of just 10?

  • This can happen if you changed the group ID again between runs, or if the first run was too brief for Kafka to commit the offset. Make sure you used "order-processing-group-v2" for both the first run and the restart.
  • Confirm you let the consumer run long enough during the first session to finish processing all 10 events before pressing Ctrl+C.

Why did my Kafka consumer replay events it already processed?

Your consumer can now read every event in the topic and pick up exactly where it left off after a restart. Next up, you will add a second consumer group that independently processes the same events for analytics, proving that multiple teams can share one event stream.

Add a Second Consumer Group for Analytics

In the last step, you fixed offset tracking and proved that Kafka remembers exactly where a consumer left off. Your order processing consumer reads every event without missing one.

In real systems, multiple teams need the same event stream. The fulfillment team processes orders while the analytics team counts item popularity. In this step, you will prove that two independent consumer groups can read the same events without interfering with each other.

In this step, get ready to:
  • Write an AnalyticsConsumer that counts item popularity from the same event stream.
  • Run both consumers at the same time to see the fan-out pattern in action.
  • Verify independent consumer groups in Kafka UI.
Write AnalyticsConsumer.java

This consumer subscribes to the same order-events topic, but instead of printing raw events it parses each order's item name and maintains a running count. The key difference is its group.id.

What is a consumer group?

A consumer group is a label that tells Kafka which consumers share the same position in a topic. Consumers with the same group.id split the work. Consumers with different group IDs each get their own copy of every event.

Your OrderConsumer uses order-processing-group-v2. The new AnalyticsConsumer will use analytics-group. Because the group IDs are different, both consumers independently receive every single event.

  • Create a new file called AnalyticsConsumer.java in src/main/java/com/kafka/ordertracker/ (the same folder as your other Java files).
  • Paste the following imports and configuration into the new file:
package com.kafka.ordertracker;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;

public class AnalyticsConsumer {

    public static void main(String[] args) {
        // Connect to the local Kafka broker
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        // Use a different group ID so this consumer tracks offsets independently
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "analytics-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        // Track how many orders each item has received
        Map<String, Integer> itemCounts = new HashMap<>();

What does this code do?

  • GROUP_ID_CONFIG is set to "analytics-group". This is the crucial difference. A different group ID means Kafka tracks this consumer's position separately from OrderConsumer.
  • AUTO_OFFSET_RESET_CONFIG is set to "earliest" so this brand-new group reads from the very first event in the topic.
  • itemCounts is a HashMap that will store a running count of how many times each product (Laptop, Headphones, etc.) appears across all orders.
  • Add the polling loop directly below the code you just pasted, still inside the main method:
        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("order-events"));
            System.out.println("[Analytics] Listening for orders...");

            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
                for (ConsumerRecord<String, String> record : records) {
                    // Parse the item name from the JSON string
                    String value = record.value();
                    int itemStart = value.indexOf("\"item\":\"") + 8;
                    int itemEnd = value.indexOf("\"", itemStart);
                    if (itemStart > 7 && itemEnd > itemStart) {
                        String item = value.substring(itemStart, itemEnd);
                        // Increment the running count for this item
                        itemCounts.merge(item, 1, Integer::sum);
                        System.out.printf("[Analytics] Order for %s | Running totals: %s%n",
                            item, itemCounts);
                    }
                }
            }
        }
    }
}

What does this code do?

  • The loop polls the order-events topic every second, exactly like OrderConsumer does.
  • Instead of printing the raw JSON, it uses indexOf and substring to extract the item name from each event's JSON payload.
  • itemCounts.merge(item, 1, Integer::sum) increments the count for that item by 1. If the item is new, it starts at 1.
  • After each event, it prints the item name alongside the full running totals map so you can see popularity changing in real time.
  • Save AnalyticsConsumer.java.
  • Confirm the file exists by running:
ls src/main/java/com/kafka/ordertracker/

You should see AnalyticsConsumer.java listed alongside OrderConsumer.java and OrderProducer.java.

Don't see the file?

  • Make sure you created the file in src/main/java/com/kafka/ordertracker/, not in the project root.
  • Check that the filename is exactly AnalyticsConsumer.java (capital A, capital C).

Help me find my AnalyticsConsumer.java file in the project.

✔️ Awesome, I've got everything!

Great. Make sure you have saved AnalyticsConsumer.java before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm both chunks are in your file, in this order:

  • Chunk 1: the package declaration, all imports, the class and main method opening, the Properties config with analytics-group, and the itemCounts HashMap declaration.
  • Chunk 2: the try block with the KafkaConsumer, subscribe, the polling while (true) loop, the item parsing logic, itemCounts.merge, and the closing braces for the class.
Run both consumers and observe fan-out

You need three terminals for this: one for the producer, one for OrderConsumer, and one for AnalyticsConsumer. All three must be in the kafka-order-tracker folder.

  • In your existing terminal, compile the project (including the new file) by running:
mvn compile dependency:copy-dependencies

You should see BUILD SUCCESS at the end of the output.

Seeing a compilation error?

  • Check that the class name in the file matches the filename exactly: public class AnalyticsConsumer in AnalyticsConsumer.java.
  • Make sure the package line at the top says com.kafka.ordertracker and that the file is inside the matching directory path.

Help me fix a compilation error in AnalyticsConsumer.java.

  • Run the producer first to send a fresh batch of 10 order events:

macOS

java -cp "target/classes:target/dependency/*" com.kafka.ordertracker.OrderProducer

Windows

java -cp "target\classes;target\dependency\*" com.kafka.ordertracker.OrderProducer

You should see 10 Sent: ORD-xxx lines followed by All orders sent successfully!. The producer finishes and returns you to the command prompt.

  • Open a second terminal window.
  • Navigate to your kafka-order-tracker folder in the new terminal by running:
cd ~/Desktop/kafka-order-tracker
  • Run OrderConsumer in this second terminal:

macOS

java -cp "target/classes:target/dependency/*" com.kafka.ordertracker.OrderConsumer

Windows

java -cp "target\classes;target\dependency\*" com.kafka.ordertracker.OrderConsumer

OrderConsumer prints the 10 new order events with their raw JSON payloads. It only shows the new batch because Kafka remembered where this consumer group left off from Step 4.

  • Open a third terminal window.
  • Navigate to your kafka-order-tracker folder in the new terminal by running:
cd ~/Desktop/kafka-order-tracker
  • Run AnalyticsConsumer in this third terminal:

macOS

java -cp "target/classes:target/dependency/*" com.kafka.ordertracker.AnalyticsConsumer

Windows

java -cp "target\classes;target\dependency\*" com.kafka.ordertracker.AnalyticsConsumer

AnalyticsConsumer prints every event from the entire topic history. After each event you see the item name and the running totals map updating in real time.

Why does AnalyticsConsumer show more events than OrderConsumer?

OrderConsumer's group (order-processing-group-v2) has committed offsets from your earlier runs, so it only sees events produced after its last known position.

AnalyticsConsumer's group (analytics-group) is brand new with auto.offset.reset=earliest, so it reads from the very beginning. Both consumers processed independently without blocking each other. This is the fan-out pattern.

AnalyticsConsumer not showing any output?

  • Make sure your Docker containers are still running. Run docker ps and confirm you see both kafka and kafka-ui containers.
  • Check that the group.id in your AnalyticsConsumer is exactly "analytics-group". A typo would create a different group that might not trigger the expected behavior.
  • Wait about 5 seconds after starting the consumer. It polls every second, and the first poll may return empty before data arrives.

Help me debug my AnalyticsConsumer not printing events.

Verify independent consumer groups in Kafka UI

Both consumers are processing events, but how do you know Kafka is really tracking them separately? The Kafka UI dashboard shows you exactly what Kafka knows about each consumer group.

Before you check, how many consumer groups do you expect to see listed in Kafka UI?

  • Open http://localhost:8080 in your browser.
  • Click Consumers in the left sidebar.

You should see two consumer groups listed: order-processing-group-v2 and analytics-group. Each row shows the group's ID, number of members, number of topics, and consumer lag.

  • Click on order-processing-group-v2 to see its committed offset for the order-events topic.
  • Go back and click on analytics-group to see its own committed offset.

Each group tracks its position independently. This is exactly how production systems enable multiple downstream applications to consume the same event stream without stepping on each other.

Why is this called the fan-out pattern?

One stream of events "fans out" to multiple consumers, each doing different work. The fulfillment team processes orders. The analytics team counts popularity. A third team could build search indexes.

Each team runs its own consumer group, reading at its own pace, without Kafka duplicating a single byte of data. Companies like Netflix, Uber, and LinkedIn use this exact pattern to process millions of events per second across dozens of consumer groups.

  • Stop both consumers by pressing Ctrl+C in each terminal when you are done.

You now have a multi-consumer event streaming system running on your laptop. Two independent consumer groups read the same topic for completely different purposes. Up next, an optional Secret Mission to handle what happens when bad data sneaks into your pipeline.

Secret mission

Handle Bad Events with a Dead Letter Topic

In production, bad data is inevitable. A single malformed event can crash your entire consumer pipeline. Can you build a safety net that catches bad events and routes them to a dead letter topic, keeping your pipeline alive?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine using Docker containers. There are no cloud resources or ongoing costs, so decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers: kafka and kafka-ui (running locally, no cost).
  • Kafka topics: order-events and order-events-dlq (stored inside the Kafka container's local storage).
  • Project folder: kafka-order-tracker on your Desktop, containing all Java source files, pom.xml, and docker-compose.yml.

Keep everything running

No action needed. Choose this if you're still actively building or want to keep experimenting with your Kafka pipeline.

  • Your Docker containers (kafka and kafka-ui) will continue running in the background.
  • The Kafka UI dashboard remains accessible at http://localhost:8080.
  • All Kafka topics and messages are preserved.
  • Apache Maven remains installed on your system. You can continue using it for other Java projects.

Pause - I'll come back to this later

Stop the Docker containers to free up system memory. Your Kafka data and project files are preserved so you can pick up where you left off.

  • Open your terminal and navigate to the kafka-order-tracker folder on your Desktop.
  • Stop the containers by running:
docker compose stop
  • When you're ready to resume, navigate back to the kafka-order-tracker folder and restart the containers by running:
docker compose start
  • Your Kafka topics, messages, and consumer group offsets are all preserved inside the stopped containers.

Delete - I don't want to use this again

Remove all Docker containers, Kafka data, and your project folder. This deletes everything you built in this project.

  • Open your terminal and navigate to the kafka-order-tracker folder on your Desktop.
  • Stop and remove the Docker containers, networks, and all Kafka data by running:
docker compose down -v

What does the -v flag do?

The -v flag removes the volumes where Kafka stored your topics and messages. Without it, the data would persist even after the containers are gone.

  • Navigate up one level so you're no longer inside the project folder by running:
cd ~/Desktop
  • Delete the project folder. On macOS or Linux, run:
rm -rf kafka-order-tracker
  • On Windows, run:
rmdir /s /q kafka-order-tracker
  • This removes all Java source files, pom.xml, docker-compose.yml, and all compiled output. Apache Maven remains installed on your system.

Nice Work!

Nice Work!

You just built a real-time order tracking pipeline powered by Apache Kafka. From streaming order events to consuming them with multiple independent groups, you now have hands-on experience with the same event streaming patterns used in production systems around the world.

You've learned how to:

  • Set up a local Kafka broker in KRaft mode using Docker Compose, with a visual Kafka UI dashboard for inspecting topics, messages, and consumer groups in real time.
  • Built a Java producer and consumer using the official kafka-clients library, and mastered offset tracking by hitting the "missing events" problem firsthand and fixing it with auto.offset.reset.
  • Implemented the fan-out pattern with two independent consumer groups processing the same event stream for different purposes.
  • Secret Mission: Completed an optional challenge to push your event streaming skills further.

Ready to quiz yourself?