Your First Kafka Event Round Trip

Run Kafka locally and publish, store, and replay order events.

Introduction

30 Second Summary

When an online order changes from created to paid to shipped, every update needs to remain available even if nobody is watching. Losing one update can leave another part of the system with the wrong picture.

In this project, you will run Apache Kafka 4.3.1 inside Docker Desktop to build a local event stream. Its complete history remains replayable after the consumer goes offline or the container restarts.

What You'll Build

Your finished demo shows one order event moving from a producer into the demo topic before a consumer replays the full history after the broker restarts.

By the end of this project, you'll have:

  • A local Kafka broker you can launch in Docker Desktop without installing Java on your Mac.
  • A publishable order stream where Kafka's console producer sends events to the demo topic for the console consumer to read.
  • A replayable event history that returns every stored order event after an offline consumer or broker restart.
  • Secret Mission: Extend the stream into a six-event order lifecycle for two orders. Explain the producer, topic, broker, and consumer roles.

What do I need before I start?

You need a Mac running a Docker-supported macOS release with at least 4 GB of RAM. You do not need prior Apache Kafka, Java, Docker, or programming experience because the full setup is included.

Before We Start

Before the hands-on work begins, this is your moment to commit to what you are building and why it matters.

You are building a local order-event stream with Apache Kafka. It will show you how one order event moves from publication into storage.

Replaying the stream later reveals that stored event again. Your Mac needs at least 4 GB of RAM plus access to macOS Terminal.

No direct Kafka or Java installation is required. The broker will run inside a container later in the project.

Set Up Docker Desktop and Kafka

Your local order-event stream needs an Apache Kafka broker before it can carry any events. This step gives that stream a running home on your Mac.

Docker Desktop 4.91.0 runs Apache Kafka 4.3.1 inside a container.

The container includes Java. Your Mac stays free from a separate Java installation.

In this step, get ready to:
  • Confirm Docker Desktop is installed and running.
  • Pull the pinned Kafka image onto your Mac.
  • Start the Kafka broker on port 9092.
Check Docker Desktop

The Docker command-line client sends requests to the engine managed by Docker Desktop. A version check reveals whether both parts can communicate.

  • Press Cmd+Space to open Spotlight Search.
  • Type Terminal and press Return to open it.
  • Check whether the Docker client can reach the engine by running this command:
docker version

What does this check prove?

This command asks the Docker client and engine to report their versions. Seeing both sections proves that Docker Desktop is running.

  • The Client section confirms that your Terminal can use Docker commands.
  • The Server section confirms that the Docker engine is available.
  • In Finder, view the file information for /Applications/Docker.app if the app exists.

The listed app version helps you choose the matching outcome below.

✔️ Client and Server appear

Your Docker client can reach the engine. Docker Desktop 4.91.0 is ready to run the Kafka broker.

ⓧ I see an older version

Your Docker engine works. Updating Docker Desktop gives this project the pinned 4.91.0 environment.

Check the Docker Desktop licence

Personal and educational use has no Docker Desktop licence charge. Larger enterprises require a paid subscription. Government entities also require a paid subscription.

The two downloads support different Mac processors.

  • Select the Mac (Apple chip) or Mac (Intel chip) download that matches the chip you identified.

Your download should produce Docker.dmg.

  • Open Docker.dmg.
  • Drag Docker to Applications.

The Applications folder now contains Docker.app.

  • Launch Docker.app from Applications.
  • Accept the agreement.

Docker Desktop now shows its setup choices.

  • Select Use recommended settings.
  • Select Finish.

Docker Desktop may take a minute or two to start its engine. A short wait here is normal.

  • Return to the Terminal window from earlier.
  • Confirm that the updated engine is available by running the version check again:
docker version

What does this check prove?

The repeated check confirms that the updated Docker Desktop engine is ready.

  • The Client section confirms that the command-line client is available.
  • The Server section confirms that the engine has finished starting.

You should now see both the Client and Server sections.

ⓧ Command not found

Docker Desktop is not installed yet. The Mac installer provides both the command-line client and the Docker engine.

Check the Docker Desktop licence

Personal and educational use has no Docker Desktop licence charge. Larger enterprises require a paid subscription. Government entities also require a paid subscription.

The two downloads support different Mac processors.

  • Select the Mac (Apple chip) or Mac (Intel chip) download that matches the chip you identified.

Your download should produce Docker.dmg.

  • Open Docker.dmg.
  • Drag Docker to Applications.

The Applications folder now contains Docker.app.

  • Launch Docker.app from Applications.
  • Accept the agreement.

Docker Desktop now shows its setup choices.

  • Select Use recommended settings.
  • Select Finish.

Docker Desktop may take a minute or two to start its engine. A short wait here is normal.

  • Return to the Terminal window from earlier.
  • Confirm that the Docker installation is available by running the version check again:
docker version

What does this check prove?

The repeated check confirms that Docker Desktop installed successfully.

  • The Client section confirms that the command-line client is installed.
  • The Server section confirms that the Docker engine is running.

You should now see both the Client and Server sections.

Still missing the Server section?

Docker Desktop may still be starting its engine. The version check can show the client before the engine becomes available.

  • Confirm that Docker.app appears in Applications.
  • Give Docker Desktop one or two minutes to finish starting.
  • Rerun the version check shown above.

If both sections still do not appear, help me troubleshoot Docker Desktop on my Mac.

Pull the Kafka image

A Docker image contains the files needed to create a container. Pulling the pinned Kafka image makes that package available locally.

  • Download the Kafka image to your Mac by running this command:
docker pull apache/kafka:4.3.1

What does this command do?

This command downloads the pinned Apache Kafka image into Docker Desktop's local image storage.

  • The apache/kafka repository provides the official Kafka JVM image.
  • The 4.3.1 tag keeps your Kafka environment aligned with this project.

The first pull may show download progress for a few minutes. Docker is fetching the Kafka runtime during this wait.

  • Wait for the image download to complete.
  • Confirm that Terminal returns to its prompt.

Image download not completing?

Confirm that Docker Desktop is still running. Check that your Mac has an active internet connection.

If the pull still does not complete, help me troubleshoot the Kafka image download.

Start the Kafka broker

The downloaded image is ready to become a running container. Publishing port 9092 lets your Mac communicate with the Kafka broker.

Before you run this, predict whether Terminal returns to its prompt or stays attached to the broker.

  • Start the Kafka container with port 9092 exposed by running this command:
docker run -p 9092:9092 apache/kafka:4.3.1

What does this command do?

This command creates a container from the local Kafka image. Terminal stays attached while the broker runs.

  • The port mapping publishes container port 9092 to host port 9092.
  • The continuing startup logs show that the Kafka process owns this Terminal session.
  • Keep this Terminal window open.
  • Switch back to Docker Desktop from earlier.
  • Select Containers.
  • Select the running Kafka container.
  • Confirm that its port entry shows 9092 published on the host.

You should see a running container created from apache/kafka:4.3.1 with port 9092 exposed.

That is the setup complete. Your Kafka broker is live and ready to accept its first order event.

Container not staying running?

Check that the original Terminal window remains open. Confirm that another local process is not already using port 9092.

If the container stops or the port is missing, help me troubleshoot my Kafka container.

Your local Kafka broker is running. Next, you will publish one order event and read it back through a consumer.

Complete Your First Event Round Trip

The Apache Kafka broker you launched is active in Docker Desktop. Now you can trace one event through the complete path.

A producer publishes an event to a topic. A consumer reads that stored event back. Seeing this round trip makes each role observable.

In this step, get ready to:
  • Access Kafka's command-line clients through the container's shell.
  • Publish one order event to the demo topic.
  • Read the stored event from the beginning of the topic.
Open the container shell

The Kafka command-line scripts live inside the running container. Docker Desktop's Exec tab gives you a shell where those scripts are available.

  • Switch back to Docker Desktop.
  • Select Containers in the left sidebar.
  • Select the running Kafka container.
  • Select the Exec tab.

You'll see a shell inside the running container. This confirms that you can use the Kafka scripts stored in /opt/kafka/bin.

Why use the Exec tab?

The Exec shell runs commands inside the container. This keeps Kafka's command-line tools inside Docker instead of installing them on your Mac.

Publish your first order event

The console producer waits for lines of text. Each line you submit becomes a separate event in the selected topic.

  • Start the console producer in the Exec shell by running this command:
/opt/kafka/bin/kafka-console-producer.sh --bootstrap-server :9092 --topic demo

What does this command do?

  • The kafka-console-producer.sh script starts an interactive producer.
  • The --bootstrap-server :9092 option connects the producer to the broker on port 9092.
  • The --topic demo option directs the event to the demo topic.

The terminal waits for your event input. The producer is now connected to the broker.

Producer did not stay open?

Confirm that Docker Desktop still shows the Kafka container as running. Make sure you ran the command in that container's Exec shell.

Ask for help with the connection details: troubleshoot my Kafka console producer connection

  • Enter your first order event using this line:
order.created id=1001

What does this event mean?

The order.created text describes what happened. The id=1001 text identifies the order involved.

Kafka stores the complete line as one event. Later consumers can read the same fact from the topic.

  • Press Return to publish the event.
  • Press Control-C to stop the producer.

The Exec terminal returns to its shell after the producer stops. The demo topic now contains your submitted event.

Event input not accepted?

Start the producer command again if you returned to the shell before entering the event. Enter the complete event on one line before pressing Return.

Get help checking the producer session: help me publish my first Kafka event

Read the event from the beginning

The console consumer reads stored events from a topic. The --from-beginning option asks it to begin with the earliest available event.

Before you run the consumer, do you think it can retrieve the event after the producer has stopped?

  • Read the demo topic from its earliest stored event by running this command:
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic demo --from-beginning

What does this command do?

  • The kafka-console-consumer.sh script starts a consumer inside the container.
  • The consumer connects to the broker through --bootstrap-server :9092.
  • The consumer reads the demo topic from the beginning because the command includes --from-beginning.

You'll see order.created id=1001 in the Exec tab. This proves that Kafka stored the event for a consumer to retrieve.

  • Press Control-C to stop the consumer after the event appears.

You completed your first Kafka event round trip. The producer published a fact to demo. The consumer retrieved that fact from the broker.

Event missing from the consumer?

Stop the consumer with Control-C. Check that the topic name is demo in both commands.

Ask for help comparing the commands: help me find my missing Kafka event

Your local stream now holds its first order event. Next, you'll test what Kafka does when no consumer is running.

Publish Events While Consumers Are Offline

Your Apache Kafka broker is still running in Docker Desktop. Your first event completed a full trip from a producer to a consumer.

Now you'll publish two more order events while the consumer is offline. A missing live result creates a deliberate uncertainty: did the broker keep the new events?

In this step, get ready to:
  • Confirm that no console consumer is running.
  • Publish two order lifecycle events.
  • Stop the producer while the consumer remains offline.
Confirm the consumer is offline

The live consumer previously gave you immediate feedback. This test removes that feedback so you can see what publishing feels like without a reader.

  • In the container's Exec tab from earlier, confirm that the shell prompt is idle.
  • Confirm that no console consumer command is running.

Pressing Control-C in the previous step stopped the consumer client. The broker container continued running independently.

Publish two offline events

A console producer turns each submitted line into a separate event. These next lines describe two later stages for the same order in the demo topic.

Before you start the producer, do you expect either line to trigger visible consumer output?

  • Start the console producer inside the Exec shell by running this command:
/opt/kafka/bin/kafka-console-producer.sh --bootstrap-server :9092 --topic demo

What does this command do?

  • /opt/kafka/bin/kafka-console-producer.sh launches the interactive producer inside the container.
  • --bootstrap-server :9092 directs the producer to the local broker.
  • --topic demo sends each submitted line to the demo topic.
  • Type the first line from the block below into the producer.
  • Press Return to publish the first line.
  • Type the second line from the block below into the producer.
  • Press Return to publish the second line.
order.paid id=1001
order.shipped id=1001

What do these events record?

  • order.paid id=1001 records the payment fact for order 1001.
  • order.shipped id=1001 records the shipping fact for the same order.
  • Each line submitted with Return becomes a separate event.
  • Stop the console producer by pressing Control-C.

The Exec shell returns to its prompt. You see no consumer output.

That quiet result is intentional. The producer accepted both lines while no consumer was running.

Producer not waiting for input?

  • Confirm that the Kafka container shows a running status in Docker Desktop.
  • Run the producer command above again in the container's Exec tab.

Help me diagnose why the Kafka console producer is not accepting events in the container's Exec tab.

You have published two events without a live consumer to confirm them. Next, you'll test whether the complete order history survived that quiet period.

Replay the Stored Stream

Your Apache Kafka broker now holds three order events. The final two produced no visible output because nothing was reading them.

Replaying the topic resolves that uncertainty. A consumer can retrieve the stored history after the events were published.

In this step, get ready to:
  • Replay every event stored in the demo topic.
  • Verify that the three events appear in publication order.
  • Stop the consumer after checking the replay.
Replay the topic from the beginning

Kafka uses durable storage to preserve events independently of a running consumer. The --from-beginning flag requests the retained history from the start of the topic.

Before you run this, how many stored events do you expect the consumer to print?

  • Start the console consumer in the Exec tab by running this command:
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic demo --from-beginning

What does this command do?

  • The /opt/kafka/bin/kafka-console-consumer.sh script starts Kafka's bundled console consumer.
  • The --bootstrap-server :9092 option connects the consumer to your running broker.
  • The --topic demo option selects the stream you created earlier.
  • The --from-beginning flag requests every retained event from the start of the topic.

First, you'll see order.created id=1001.

Next, you'll see order.paid id=1001.

Finally, you'll see order.shipped id=1001.

Stop the consumer after the replay

The console consumer keeps waiting for future events after it prints the stored history. Stopping it returns control of the Exec shell without deleting anything from demo.

  • Confirm that order.created id=1001 appears first.
  • Confirm that order.paid id=1001 appears second.
  • Confirm that order.shipped id=1001 appears third.
  • Press Control-C to stop the consumer.

The Exec shell returns to its prompt. That resolves the uncertainty from the last step.

The demo topic still contains three retained events in publication order: order.created id=1001, order.paid id=1001, and order.shipped id=1001.

Missing an event from the replay?

  • Check that your command targets the demo topic.
  • Confirm that your command includes --from-beginning.
  • Verify that the Kafka container still shows a running status in Docker Desktop.

Help me diagnose why my Kafka console consumer is not replaying all three events.

You proved that Kafka retained the events published while no consumer was running. Next, you'll restart the same container to test whether the complete stream remains replayable.

Restart Kafka and Verify Retention

Your last replay showed all three order events in publication order. The next test asks whether the same stream survives a restart of the broker container in Docker Desktop.

Apache Kafka stores events inside a topic. A replay after restart tests whether that stored stream outlives the earlier terminal session.

In this step, get ready to:
  • Stop the same Kafka container.
  • Restart the container in Docker Desktop.
  • Replay the three retained events after the restart.
Stop the running Kafka container

The original Terminal remains attached to the broker process. The stopped container stays available for your restart test.

  • Switch back to the original Terminal window running Apache Kafka.
  • Press Control-C to stop the broker process.
  • Return to the Containers view in Docker Desktop.
  • Confirm that the same Kafka container now shows a stopped status.

You have reached the key checkpoint. The broker is stopped while the same container remains available.

Why use the same container?

A stopped Docker container keeps changes in its writable state. Starting that exact container lets Kafka access its existing topic data again.

A newly created container has separate writable state. This retention check depends on restarting the existing container.

Restart the same container

Restarting the existing container creates a fresh broker session against the same container state. The replay in the next substep reveals whether the topic data remains available.

  • Select the same stopped Kafka container in the Containers view.
  • Use the container's start control to restart it.
  • Wait until the container status changes to running.
  • Select the same running container.
  • Select the Exec tab again.

Good progress. The same Kafka container is running again with port 9092 exposed.

What did the restart preserve?

The restart reactivates the same container state. The next replay checks whether Kafka can still read the events stored in that state.

Replay the retained stream

A new consumer provides the proof. Reading demo from the beginning checks the complete stored history after the restart.

  • Predict whether all three events will return from demo after the restart.
  • Replay the topic from the beginning by running this command:
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic demo --from-beginning

What does this command do?

  • The kafka-console-consumer.sh script starts an interactive Kafka consumer.
  • The --bootstrap-server :9092 option connects the consumer to the restarted broker.
  • The --topic demo option selects the stored order-event stream.
  • The --from-beginning option requests every retained event from the start.

You'll see order.created id=1001 first. You'll see order.paid id=1001 next.

You'll see order.shipped id=1001 last. The original publication order remains intact.

  • Press Control-C to stop the consumer.

Events missing after restart?

  • Confirm that Docker Desktop shows the same Kafka container as running.
  • Check that your command includes --topic demo.
  • Check that your command includes --from-beginning.

Help me diagnose why my Kafka events are missing after restarting the same Docker container.

You have now proved that the stream in demo survives a stop and restart of the same container. Your first Kafka retention test is complete.

Retention checkpoint

The same Apache Kafka 4.3.1 container is running again in Docker Desktop. Host port 9092 remains mapped to container port 9092.

The restarted demo topic prints order.created id=1001 first. It prints order.paid id=1001 next.

It prints order.shipped id=1001 last. The complete stream remains in publication order.

Secret mission

Design an Order Lifecycle Stream

Add a second order lifecycle to the stream you already preserved through a container restart. Replay all six events in publication order and explain the producer, topic, broker, and consumer roles.

Clean Up Your Resources

Clean Up Your Resources

Your local Apache Kafka resources have no usage charge. Decide whether to keep them running, pause the container for later, or delete them entirely.

Resources you used:

  • Kafka container: The currently running Apache Kafka 4.3.1 broker with host port 9092 exposed.
  • Topic: The demo stream containing all six order events.
  • Local image: The apache/kafka:4.3.1 image stored on your Mac.

Keep everything running

No cleanup action is needed. Choose this if you want to keep producing or replaying events now.

  • Leave the Kafka container running in Docker Desktop.
  • Keep the Exec tab available for more producer or consumer commands.

Pause - I'll come back to this later

Stop the running container to free local memory. Its writable state remains available for your next session.

  • Select Containers in Docker Desktop.
  • Select the running Apache Kafka container.
  • Select Stop.

You'll see the container status change to stopped. That confirms your broker is paused while its six-event stream remains available for the next start.

Delete - I don't want to use this again

Removing the container permanently deletes the events stored inside it. You can pull the Kafka image again if you want to rebuild the stream later.

  • Select Containers in Docker Desktop.
  • Select the Apache Kafka container.
  • Select its Delete action.

You'll no longer see the container in Containers. Its demo topic and six stored events are deleted with it.

Docker Desktop can now remove the image because its associated container is gone.

  • Select Images in Docker Desktop.
  • Locate apache/kafka:4.3.1.
  • Select the bin icon for the image.

You'll no longer see apache/kafka:4.3.1 in Images.

Cleanup complete. Your local Kafka broker and six-event stream are removed from Docker Desktop.

Nice Work!

Nice Work!

You did it! You built a replayable local order-event stream with Apache Kafka 4.3.1 in Docker Desktop.

You've learned how to:

  • Run a local Kafka broker in Docker Desktop without installing Java on your Mac.
  • Complete an event round trip through the demo topic with Kafka's console producer and console consumer.
  • Prove durable storage by publishing events while the consumer was stopped. Replay the complete stream after restarting the broker container.
  • Secret Mission: Extend the stream into a six-event order lifecycle for orders 1001 and 1002. Explain the role of each Kafka component in the stream.

Ready to quiz yourself?