Real-Time Order Tracking with Kafka

Build a local event streaming pipeline with Kafka producers and consumers.

Introduction

30 Second Summary

When you order something online, you expect to see your order status update in real time. Behind the scenes, that instant tracking relies on event streaming, where every status change is published as an event and picked up by the services that need it.

In this project, you will build a real-time order tracking pipeline using Apache Kafka. You'll run a Kafka broker in Docker, then create a Node.js producer that publishes order events and a consumer that processes them as they arrive.

What You'll Build

A local event streaming pipeline with a Kafka broker running in Docker, a Node.js producer that simulates e-commerce orders, and a Node.js consumer that tracks the order lifecycle in real time.

By the end of this project, you'll have:

  • A running Kafka 4.3.0 broker in Docker using KRaft mode (no ZooKeeper required).
  • A Node.js producer that publishes structured order events with key-based partition routing.
  • A Node.js consumer that processes events in real time using consumer group semantics and graceful shutdown.
  • Secret Mission: Add a second consumer group to see how Kafka enables multiple independent subscribers to the same event stream.

Are there any prerequisites?

You'll need Docker Desktop installed on your machine and familiarity with Node.js and basic terminal commands. No prior Kafka experience is needed.

Set Up Kafka with Docker

Every real-time system needs a message broker at its core. Apache Kafka handles this by letting services publish and subscribe to streams of events without being directly connected to each other.

In this step, you'll spin up a Kafka broker inside Docker and set up a Node.js project with the Confluent JavaScript client. By the end, your broker will be running and ready to accept messages.

In this step, get ready to:
  • Run a Kafka 4.3.0 broker in Docker using KRaft mode.
  • Verify the broker is running by checking its logs.
  • Initialize a Node.js project with the Confluent Kafka client installed.
Run a Kafka broker in Docker

Kafka 4.3.0 runs in KRaft mode by default. This means there is no separate ZooKeeper process to manage.

The official Docker image handles all the broker configuration automatically, so you can start a full Kafka instance with a single command.

  • Pull and start the Kafka container by running this command:
docker run -d --name kafka -p 9092:9092 apache/kafka:4.3.0

What does this command do?

  • -d runs the container in the background so you can keep using your terminal.
  • --name kafka gives the container a friendly name you can reference later.
  • -p 9092:9092 maps the container's Kafka port to your machine's port 9092.
  • apache/kafka:4.3.0 is the official Apache Kafka image. It runs in KRaft mode with zero configuration.

Why Docker instead of installing Kafka directly?

A native Kafka install requires Java 17+ and manual configuration. Docker gives you a single-command setup with no Java dependency on your machine, easy cleanup, and a reproducible environment.

  • Wait about 10 seconds for Kafka to finish starting up.

Seeing 'port is already allocated'?

  • Another process is using port 9092. Run docker ps to check for other containers, then stop them.
  • If you see a permission error, make sure Docker Desktop is running.
  • help me fix my Docker container startup error
Verify Kafka is running

Before moving on, you need to confirm the broker started successfully. Kafka logs a specific message when it is ready to accept connections.

  • Check the container logs by running this command:
docker logs kafka
  • Look for a line containing Kafka Server started near the end of the output.

What should I see?

You'll see a wall of log output as Kafka initializes its internal topics and configurations. The key line to look for is Kafka Server started which confirms the broker is ready to accept connections on port 9092.

Don't see 'Kafka Server started'?

  • If the container just started, wait a few more seconds and run docker logs kafka again.
  • If the container exited, run docker ps -a to check its status. Remove it with docker rm kafka and try the run command again.
  • help me troubleshoot my Kafka container not starting
Initialize the Node.js project

Now that Kafka is running, you need a Node.js project to interact with it. The Confluent JavaScript client provides the producer and consumer APIs you'll use throughout this project.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir kafka-order-pipeline
cd kafka-order-pipeline
  • Confirm the folder was created by running:
ls ~/Desktop | grep kafka
  • You should see kafka-order-pipeline in the output.
  • Initialize a new Node.js project by running this command:
npm init -y
  • You should see a confirmation message showing the generated package.json contents.
  • Install the Confluent JavaScript client by running this command:
npm install @confluentinc/kafka-javascript
  • You should see output confirming packages were added. This may take a minute as it downloads prebuilt native binaries.

Install failing or hanging?

  • Make sure you are connected to the internet. The package downloads prebuilt native binaries for your platform.
  • If you see a Node.js version error, confirm you have Node.js 18 or higher by running node --version in your terminal.
  • help me fix my npm install error for @confluentinc/kafka-javascript

Now you need to update the generated package.json with your project's name, description, and convenience scripts.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the kafka-order-pipeline folder on your Desktop, then click Open.
  • In the Explorer sidebar, click on package.json to open it.
  • Replace the entire contents of package.json with the following:
{
  "name": "kafka-order-pipeline",
  "version": "1.0.0",
  "description": "Real-time order tracking pipeline with Apache Kafka",
  "scripts": {
    "admin": "node admin.js",
    "produce": "node producer.js",
    "consume": "node consumer.js"
  },
  "dependencies": {
    "@confluentinc/kafka-javascript": "^1.9.0"
  }
}

What did we add?

  • scripts provides shortcut commands. You can run npm run admin, npm run produce, or npm run consume instead of typing the full node commands later.
  • description gives the project a meaningful label instead of the default empty string.
  • dependencies lists @confluentinc/kafka-javascript which is the actively maintained Kafka client backed by Confluent.
  • Save package.json by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Confirm your dependencies are installed by checking that a node_modules folder appears in the Explorer sidebar.

Don't see node_modules in the sidebar?

  • Make sure you ran npm install @confluentinc/kafka-javascript inside the kafka-order-pipeline folder, not from your Desktop.
  • If you opened the wrong folder in VS Code, close it and reopen the correct kafka-order-pipeline folder from your Desktop.
  • help me fix my missing node_modules in my Kafka project

✔️ Awesome, I've got everything!

Great. Double check you've saved package.json and that node_modules is visible in your sidebar.

ⓧ I'd like to double check the full code

Here is what your package.json should look like:

{
  "name": "kafka-order-pipeline",
  "version": "1.0.0",
  "description": "Real-time order tracking pipeline with Apache Kafka",
  "scripts": {
    "admin": "node admin.js",
    "produce": "node producer.js",
    "consume": "node consumer.js"
  },
  "dependencies": {
    "@confluentinc/kafka-javascript": "^1.9.0"
  }
}

Your Kafka broker is running and your Node.js project is ready. Next up, you'll create a topic with multiple partitions to see how Kafka organizes messages.

Create a Topic with Multiple Partitions

Your Kafka broker is up and your Node.js project is ready. Before you can send or receive messages, you need to create a topic for them to flow through.

In Kafka, a topic is a named channel that messages flow through. Each topic is divided into partitions, which control how messages are distributed and ordered. You'll use the Confluent client's admin API to create a 3-partition topic for your order events.

In this step, get ready to:
  • Write an admin script that creates the order-events topic with 3 partitions.
  • Run the script and verify the topic exists in your Kafka broker.
Write the admin script

The Confluent JavaScript client exposes an admin API that lets you manage Kafka cluster resources programmatically. You'll use it to create a topic with a specific number of partitions.

  • Create a new file called admin.js in your project folder by running this command:
touch admin.js
  • Open admin.js in your editor and paste the following code:
const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS;

async function createTopics() {
  // Create a Kafka instance pointing to our local broker
  const kafka = new Kafka({
    "bootstrap.servers": "localhost:9092",
  });

  // Create and connect the admin client
  const admin = kafka.admin();
  await admin.connect();
  console.log("Admin connected to Kafka");

  // Create the order-events topic with 3 partitions
  const result = await admin.createTopics({
    topics: [
      {
        topic: "order-events",
        numPartitions: 3,
        replicationFactor: 1,
      },
    ],
  });

  // Log whether the topic was newly created or already existed
  if (result) {
    console.log("Topic 'order-events' created successfully with 3 partitions");
  } else {
    console.log("Topic 'order-events' already exists");
  }

  await admin.disconnect();
  console.log("Admin disconnected");
}

createTopics().catch(console.error);

What does this code do?

  • const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS imports the promisified API from the Confluent client. This gives you async/await access to Kafka operations.
  • bootstrap.servers tells the client where to find your Kafka broker. This matches the port you exposed in Docker.
  • numPartitions: 3 splits the topic into 3 partitions. Later, your producer will use message keys to route related events to the same partition.
  • replicationFactor: 1 means each partition has one copy. This is correct for a single-broker local setup.
  • createTopics() returns true if the topic was created, or false if it already exists.
  • Save admin.js. You'll run this script in the next substep to create the topic.

✔️ Awesome, I've got everything!

Great. Make sure you've saved admin.js before moving on.

ⓧ I'd like to double check the full code

Here is the complete admin.js file:

const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS;

async function createTopics() {
  const kafka = new Kafka({
    "bootstrap.servers": "localhost:9092",
  });

  const admin = kafka.admin();
  await admin.connect();
  console.log("Admin connected to Kafka");

  const result = await admin.createTopics({
    topics: [
      {
        topic: "order-events",
        numPartitions: 3,
        replicationFactor: 1,
      },
    ],
  });

  if (result) {
    console.log("Topic 'order-events' created successfully with 3 partitions");
  } else {
    console.log("Topic 'order-events' already exists");
  }

  await admin.disconnect();
  console.log("Admin disconnected");
}

createTopics().catch(console.error);
Run the admin script and verify the topic
  • Run the admin script by executing this command:
node admin.js

You should see output confirming the connection and topic creation:

Admin connected to Kafka
Topic 'order-events' created successfully with 3 partitions
Admin disconnected

What if I see "already exists"?

If you see Topic 'order-events' already exists, that means the topic was already created in a previous run. This is safe to continue with. The script handles this gracefully.

Seeing an error instead?

  • If you see a connection error, confirm your Kafka Docker container is still running. Run docker ps and check for the apache/kafka:4.3.0 container. If it's not listed, restart it.
  • If the script hangs without output, the broker might still be initializing. Wait 10 seconds and try again.
  • If you see a timeout error, make sure port 9092 is not being used by another process on your machine.

Still stuck? Help me debug why my admin.js script can't connect to Kafka on localhost:9092.

Your topic is live and ready to receive events. Next up, you'll build a producer that generates order events and sends them into this topic.

Build the Order Event Producer

Your order-events topic is ready and waiting for data. Now you need something to actually publish events into it.

In Kafka, a producer is the component that sends messages to a topic. Each message has a key and a value. Kafka hashes the key to decide which partition the message lands on, so messages with the same key always end up on the same partition in the same order.

In this step, get ready to:
  • Create a producer that connects to Kafka and sends structured order events.
  • Use the order ID as a message key to route related events to the same partition.
  • Observe delivery reports confirming which partition and offset each message received.
Create the producer file with connection setup

The producer needs to connect to the Kafka broker and define what order data it will simulate. You will set up the connection first, then define the order IDs and statuses that represent an e-commerce order lifecycle.

  • Create a new file called producer.js in your project folder by running:
touch producer.js
  • Confirm the file exists by running:
ls

You should see producer.js listed alongside admin.js and package.json.

  • Open producer.js in your editor and add the following connection setup code:
const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS;

async function produceOrders() {
  // Create a producer instance connected to the local Kafka broker
  const producer = new Kafka().producer({
    "bootstrap.servers": "localhost:9092",
  });

  await producer.connect();
  console.log("Producer connected to Kafka");

  // Define the 4 order IDs and lifecycle statuses to simulate
  const orderIds = ["ORD-1001", "ORD-1002", "ORD-1003", "ORD-1004"];
  const statuses = ["placed", "confirmed", "shipped", "delivered"];
}

produceOrders().catch(console.error);

What does this code do?

This sets up the skeleton of your producer script.

  • The import pulls the promisified Kafka class from the Confluent client, the same pattern you used in admin.js.
  • The producer connects to localhost:9092 where your Docker-based broker is running.
  • The orderIds array simulates 4 different e-commerce orders. The statuses array represents the lifecycle each order goes through.
  • Save producer.js.

Seeing a syntax error?

  • Make sure the require statement ends with .KafkaJS. Without it, you get the callback-based API instead of the promisified one.
  • Check that the closing } of the function and the produceOrders().catch(console.error) call are both present at the bottom.

Help me debug my producer.js connection setup

Add the event generation loop with delivery reports

Now you need to loop through each status for every order ID, creating 16 total events (4 orders x 4 statuses). Each event gets the order ID as its message key, which tells Kafka to always route that order's events to the same partition.

Why use the order ID as the message key?

Kafka hashes the message key to determine the target partition. By using the order ID as the key, all events for ORD-1001 (placed, confirmed, shipped, delivered) land on the same partition in the order they were sent.

This guarantees that a downstream consumer reading that partition sees the full lifecycle of each order in sequence. Without a key, events would scatter across partitions randomly.

  • In producer.js, find the closing } of the produceOrders function (the line just before produceOrders().catch(console.error)).
  • Replace that closing } with the following loop, disconnect logic, and new closing bracket:
  // Loop through each status, sending an event for every order ID
  for (const status of statuses) {
    for (const orderId of orderIds) {
      // Build the order event payload
      const event = {
        orderId,
        status,
        timestamp: new Date().toISOString(),
      };

      // Send the event with orderId as the key for partition routing
      const reports = await producer.send({
        topic: "order-events",
        messages: [
          {
            key: orderId,
            value: JSON.stringify(event),
          },
        ],
      });

      // Log the delivery report showing partition and offset
      const report = reports[0];
      console.log(
        `Sent: ${orderId} [${status}] -> partition ${report.partition}, offset ${report.offset}`
      );
    }
  }

  await producer.disconnect();
  console.log("\nProducer disconnected. 16 events sent.");
}

What does this code do?

  • The outer loop iterates through statuses (placed, confirmed, shipped, delivered). The inner loop iterates through all 4 order IDs for each status.
  • Each event is a JSON object with orderId, status, and a timestamp.
  • The key: orderId field tells Kafka which partition to route the message to. The value is the full event serialized as JSON.
  • The send() method returns an array of delivery reports. Each report contains the partition and offset where the message was stored.
  • Save producer.js.

Getting an unexpected token error?

  • Make sure you deleted the old closing } before pasting. You should have exactly one } at the end of the function, right before produceOrders().catch(console.error).
  • Check that the template literal backticks in the console.log line are backticks (`), not regular quotes.

Help me fix my producer.js event loop

✔️ Awesome, I've got everything!

Great. Double check you have saved producer.js before moving on.

ⓧ I'd like to double check the full code

Here is the complete producer.js file:

const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS;

async function produceOrders() {
  const producer = new Kafka().producer({
    "bootstrap.servers": "localhost:9092",
  });

  await producer.connect();
  console.log("Producer connected to Kafka");

  const orderIds = ["ORD-1001", "ORD-1002", "ORD-1003", "ORD-1004"];
  const statuses = ["placed", "confirmed", "shipped", "delivered"];

  for (const status of statuses) {
    for (const orderId of orderIds) {
      const event = {
        orderId,
        status,
        timestamp: new Date().toISOString(),
      };

      const reports = await producer.send({
        topic: "order-events",
        messages: [
          {
            key: orderId,
            value: JSON.stringify(event),
          },
        ],
      });

      const report = reports[0];
      console.log(
        `Sent: ${orderId} [${status}] -> partition ${report.partition}, offset ${report.offset}`
      );
    }
  }

  await producer.disconnect();
  console.log("\nProducer disconnected. 16 events sent.");
}

produceOrders().catch(console.error);
Run the producer and observe partition routing

Time to send your order events to Kafka. When you run the producer, the delivery reports will show exactly which partition each order ID was routed to.

  • Run the producer by executing this command:
node producer.js

You should see 16 lines of output, each showing an order ID, its status, and the partition and offset it was stored at. The key observation is that every event for the same order ID goes to the same partition number.

What should I see?

Your output will look something like this (partition numbers will vary based on Kafka's hash of each key):

Producer connected to Kafka
Sent: ORD-1001 [placed] -> partition 2, offset 0
Sent: ORD-1002 [placed] -> partition 0, offset 0
Sent: ORD-1003 [placed] -> partition 1, offset 0
Sent: ORD-1004 [placed] -> partition 2, offset 1
...
Sent: ORD-1001 [delivered] -> partition 2, offset 6

Producer disconnected. 16 events sent.

Notice that ORD-1001 always goes to the same partition number. The same is true for every other order ID. This is Kafka's key-based partition routing in action.

Seeing a connection error?

  • Make sure your Kafka Docker container is still running. Check with docker ps and look for the apache/kafka:4.3.0 image.
  • If the container stopped, restart it with docker start followed by the container name from docker ps -a.
  • Verify the order-events topic exists by re-running node admin.js. If you see the "already exists" message, the topic is fine.

Help me troubleshoot my Kafka producer connection

Your producer is publishing keyed events to Kafka with delivery confirmations. Next up, you will build a consumer that reads these events in real time and displays the full order lifecycle.

Build the Order Event Consumer

Your producer is publishing 16 order events to Kafka, each routed to a specific partition based on its order ID key. Now it's time to build the other half of the pipeline: the consumer that reads those events in real time.

A consumer subscribes to a topic and pulls messages as they arrive. In this step, you'll see Kafka's ordering guarantees in action as events for the same order appear sequentially on the same partition.

In this step, get ready to:
  • Create a consumer that connects to Kafka with a consumer group.
  • Process messages and log formatted order lifecycle events.
  • Observe partition-based ordering and graceful shutdown.
Create the consumer connection and subscription

A Kafka consumer needs three things: the broker address, a consumer group ID, and a starting offset policy. The group ID lets Kafka track which messages this consumer has already processed.

  • Create a new file called consumer.js in your kafka-order-pipeline folder and add the following code:
const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS;

async function consumeOrders() {
  // Create a consumer with group ID and offset policy
  const consumer = new Kafka().consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "order-tracker-group",
    "auto.offset.reset": "earliest",
  });

  await consumer.connect();
  console.log("Consumer connected (group: order-tracker-group)");

  // Subscribe to the order-events topic
  await consumer.subscribe({ topics: ["order-events"] });

What does this code do?

The consumer is configured with three key properties:

  • bootstrap.servers tells the consumer where to find the Kafka broker.
  • group.id assigns this consumer to the order-tracker-group. Kafka uses this to track which messages the group has consumed.
  • auto.offset.reset set to earliest means the consumer will start reading from the very first message if no prior offset exists for this group.

You won't see a change yet. The next chunk adds the message handler that brings this consumer to life.

Add the message handler and graceful shutdown

The consumer.run() method starts a polling loop that calls your eachMessage callback for every message Kafka delivers. You also need a way to stop the consumer cleanly when you press Ctrl+C.

  • In consumer.js, add the following code below the subscribe line:

  let messageCount = 0;

  // Start the consume loop - calls eachMessage for every event
  consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const event = JSON.parse(message.value.toString());
      messageCount++;
      console.log(
        `[Partition ${partition} | Offset ${message.offset}] Order: ${event.orderId} | Status: ${event.status} | Time: ${event.timestamp}`
      );
    },
  });

  // Graceful shutdown on Ctrl+C
  process.on("SIGINT", async () => {
    console.log(`\nShutting down... Processed ${messageCount} messages.`);
    await consumer.disconnect();
    process.exit(0);
  });
}

consumeOrders().catch(console.error);

What does this code do?

The eachMessage callback receives three fields: topic, partition, and message. You parse the JSON value, increment a counter, and log a formatted line showing exactly where each event landed.

The SIGINT handler catches Ctrl+C and calls consumer.disconnect() to cleanly leave the consumer group. This triggers a final rebalance so Kafka knows the consumer is gone and can reassign its partitions.

  • Save consumer.js.

Seeing a syntax error?

Make sure your closing braces match: the consumeOrders function needs a closing } after the process.on block. Check the full file below to compare.

help me fix a syntax error in my consumer.js file

✔️ Awesome, I've got everything!

Great. Double check you've saved consumer.js before running it in the next substep.

ⓧ I'd like to double check the full code

const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS;

async function consumeOrders() {
  const consumer = new Kafka().consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "order-tracker-group",
    "auto.offset.reset": "earliest",
  });

  await consumer.connect();
  console.log("Consumer connected (group: order-tracker-group)");

  await consumer.subscribe({ topics: ["order-events"] });

  let messageCount = 0;

  consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const event = JSON.parse(message.value.toString());
      messageCount++;
      console.log(
        `[Partition ${partition} | Offset ${message.offset}] Order: ${event.orderId} | Status: ${event.status} | Time: ${event.timestamp}`
      );
    },
  });

  process.on("SIGINT", async () => {
    console.log(`\nShutting down... Processed ${messageCount} messages.`);
    await consumer.disconnect();
    process.exit(0);
  });
}

consumeOrders().catch(console.error);
Run the consumer and observe partition ordering

Time to see the consumer in action. It will read all 16 events from the beginning and display the full order lifecycle grouped by partition.

  • Run the consumer by executing this command:
node consumer.js
  • Watch the terminal output. You should see 16 messages logged, each showing the partition, offset, order ID, and status.
  • Look at the partition numbers. Notice that all events for ORD-1001 appear on the same partition, and all events for ORD-1002 appear on the same partition. This is Kafka's key-based routing in action.
  • Press Ctrl+C to stop the consumer. You should see the shutdown message showing how many messages were processed.

What should I see?

Your output will look something like this (partition numbers may differ):

Consumer connected (group: order-tracker-group)
[Partition 0 | Offset 0] Order: ORD-1001 | Status: placed | Time: 2026-06-24T...
[Partition 0 | Offset 1] Order: ORD-1001 | Status: confirmed | Time: 2026-06-24T...
[Partition 2 | Offset 0] Order: ORD-1003 | Status: placed | Time: 2026-06-24T...
...

The key insight: events for the same order ID always share the same partition number, and they appear in the correct lifecycle order (placed, confirmed, shipped, delivered).

Why does ordering matter?

Kafka guarantees message ordering within a single partition. Because you used the order ID as the message key, Kafka hashes that key to determine the partition. Same key always maps to the same partition, so all events for one order arrive in the exact sequence they were produced.

This is how real systems ensure an order isn't marked "delivered" before "shipped" even under heavy load.

Not seeing any messages?

  • Make sure your Kafka container is still running. Check with docker ps and look for the apache/kafka:4.3.0 image.
  • Confirm you ran producer.js in the previous step. The consumer needs existing messages on the order-events topic.
  • If the consumer connects but shows no output, check that auto.offset.reset is set to earliest in your code. Without this, the consumer only reads new messages produced after it joined.

help me debug why my Kafka consumer is not receiving any messages

Your pipeline is complete. You've built both sides of the streaming equation: a producer that publishes keyed events and a consumer that reads them with ordering guarantees. Next up, the Secret Mission challenges you to add a second consumer group to prove that multiple subscribers can independently process the same stream.

Secret mission

Add a Second Consumer Group

Your order tracking pipeline has one consumer group reading events. In production, multiple teams need the same data simultaneously. Can you add an analytics consumer that independently processes the same event stream without affecting the existing order tracker?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally via Docker, so there are no ongoing cloud costs.

Resources you used:

  • Kafka Docker container (apache/kafka:4.3.0 image, port 9092).
  • Project folder kafka-order-pipeline on your Desktop (contains admin.js, producer.js, consumer.js, analytics-consumer.js, and node_modules).

Keep everything running

No action needed. Choose this if you're still actively experimenting with Kafka producers and consumers.

  • Your Kafka container uses minimal CPU and memory when idle.
  • You can run producer.js and consumer.js again any time without any setup.

Pause - I'll come back to this later

Stop the Kafka container to free up memory. Your project files and container state stay intact so you can restart later.

  • Stop the Kafka container by running:
docker stop $(docker ps -q --filter ancestor=apache/kafka:4.3.0)
  • When you're ready to come back, restart it by running:
docker start $(docker ps -aq --filter ancestor=apache/kafka:4.3.0)
  • Your order-events topic and all messages will still be there when you restart.

Delete - I don't want to use this again

Remove the Kafka container and delete all project files from your machine.

  • Remove the Kafka container by running:
docker rm -f $(docker ps -aq --filter ancestor=apache/kafka:4.3.0)
  • Delete the project folder by running:
cd ~/Desktop
rm -rf kafka-order-pipeline
  • Confirm the folder is gone by running:
ls ~/Desktop
  • You should no longer see kafka-order-pipeline in the output.
  • Optionally, remove the Docker image to reclaim disk space:
docker rmi apache/kafka:4.3.0

That's a Wrap!

That's a Wrap!

Nice work! You just built a real-time order tracking pipeline using Apache Kafka, the same event streaming platform that powers systems at Netflix and Uber.

You've learned how to:

  • Run a Kafka 4.3.0 broker in KRaft mode using Docker and create a partitioned topic with the admin API.
  • Build a producer that sends keyed JSON events with delivery reports, demonstrating partition-based ordering guarantees.
  • Build a consumer with consumer group semantics and graceful shutdown that processes events in real time.
  • Secret Mission: Ran two independent consumer groups on the same topic, demonstrating Kafka's publish-subscribe pattern where multiple subscribers each get the full event stream.

Ready to quiz yourself?