Kafka Event Streaming on AWS EC2

Run a Kafka-compatible lab on EC2 and monitor consumer lag in real time.

Introduction

30 Second Summary

Every time you order food through a delivery app, your taps fire off real-time events: browsing menus, adding items to your cart, completing payment. Apache Kafka is the industry-standard tool that powers this kind of real-time event streaming.

In this project, you will build a Kafka operations lab on an AWS EC2 instance using Redpanda, a Kafka-compatible broker. You will create topics, produce realistic order events, and monitor how consumers keep up in real time using CLI tools.

What You'll Build

Picture yourself in an interview, pulling up a live terminal where order events stream into partitions, consumers automatically split the work between them, and you can show exactly how far behind each consumer is falling.

By the end of this project, you'll have:

A running bash producer script that streams realistic JSON order events with customer-based message keys across multiple partitions.
A live consumer group where you add and remove consumers, watching Redpanda automatically rebalance partition assignments between them.
Real-time monitoring of offset lag per partition using rpk group describe, where you see lag climb when consumers stop and drop as they catch up.
Secret Mission: Diagnose a stuck consumer vs. a slow consumer using lag patterns.

Are there any prerequisites?

You will need an AWS account and an SSH client on your Windows machine (Windows Terminal, PowerShell, or PuTTY all work). The project walks you through everything else, including launching the EC2 instance and installing Docker.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. This step is your chance to commit to the goal before any hands-on work begins.

Launch Your EC2 Lab and Deploy Redpanda

Running a Kafka-compatible broker like Redpanda requires a dedicated Linux environment with Docker. A cloud instance gives you a clean, isolated lab that mirrors how production streaming systems actually run.

In this step, you'll launch an AWS EC2 instance, install Docker and Docker Compose, and bring up your Redpanda broker with its web console. By the end, you'll have a running Kafka-compatible cluster ready for event streaming.

In this step, get ready to:

Launch a t3.medium EC2 instance running Amazon Linux 2023 and SSH in.
Install Docker and Docker Compose on the instance.
Deploy the Redpanda broker and Console with Docker Compose.

Launch your EC2 instance and SSH in

You need a Linux server with enough memory to run Redpanda comfortably. A t3.medium instance provides 4 GiB of RAM on an x86_64 architecture, which is exactly what Redpanda's Docker image requires.

Sign in to the AWS Management Console.
Type EC2 in the top search bar and select EC2.
Click Launch instances.
Enter kafka-lab in the Name field.
Under Application and OS Images, confirm Amazon Linux 2023 AMI is selected.
Under Instance type, select t3.medium from the dropdown.

Why t3.medium?

Redpanda's Docker images are built for x86_64 (linux/amd64). The t3.medium provides 4 GiB RAM on x86_64, which comfortably runs both the broker and Console containers.

This instance type is not free-tier eligible and costs approximately $0.04/hour. The full project takes about an hour, so expect roughly $0.50 in total EC2 charges.

Under Key pair (login), click Create new key pair.
Enter kafka-lab-key as the Key pair name.
Keep RSA selected for the key pair type.
Keep .pem selected for the file format.
Click Create key pair.

Your browser downloads a kafka-lab-key.pem file. Note the folder where it saves. You will need the path to this file in a moment.

Under Network settings, click Edit.
Keep the existing SSH rule on port 22.
Click Add security group rule.
Set Type to Custom TCP.
Enter 8080 in the Port range field.
Set Source type to Anywhere.

Why open port 8080?

Redpanda Console is a web UI that runs on port 8080 inside your instance. Opening this port in the security group lets you view topics, messages, and consumer groups from your browser in a later step.

Click Launch instance.
Click View all instances.
Wait until the Instance state column shows Running.
Select your kafka-lab instance.
Copy the Public IPv4 address from the details panel below: your EC2 public IPv4 address

Now connect to your instance via SSH. Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), then type Terminal and press Enter.

Connect to your EC2 instance by running this command:

ssh -i [[PEMPATH="path to your .pem key file"]] ec2-user@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]

What does this command do?

The ssh command opens a secure shell connection to your EC2 instance. The -i flag points to your private key file. ec2-user is the default username on Amazon Linux.

When prompted with "Are you sure you want to continue connecting?", type yes and press Enter.

You should see a prompt that starts with [ec2-user@, confirming you are logged into your EC2 instance.

Can't connect via SSH?

If you see a "Permission denied" error on macOS or Linux, your key file permissions are too open. Run chmod 400 followed by the path to your .pem file, then try the SSH command again.
If you see "Connection timed out", make sure your security group allows SSH on port 22 and that you copied the correct Public IPv4 address.
If you see "No such file or directory" for the .pem file, double-check the download location. On Windows, it is typically in C:\Users\YourName\Downloads\. On macOS, it is typically in ~/Downloads/.

Having trouble with SSH?

Install Docker and Docker Compose

Your EC2 instance is a fresh Amazon Linux 2023 machine. It does not have Docker or Docker Compose pre-installed, so you need to add both before you can run any containers.

Install Docker and start the service by running these commands:

sudo yum install -y docker
sudo service docker start
sudo usermod -a -G docker ec2-user

What do these commands do?

The first line installs Docker from the Amazon Linux package repository.
The second line starts the Docker daemon so it can manage containers.
The third line adds ec2-user to the docker group. This lets you run Docker commands without sudo after you refresh your session.

Apply the new group membership to your current session by running:

newgrp docker

Verify Docker is working by running:

docker --version

You should see a version number printed, confirming Docker is installed and accessible.

Install Docker Compose as a CLI plugin by running these commands:

mkdir -p $HOME/.docker/cli-plugins
curl -SL https://github.com/docker/compose/releases/download/v5.5.0/docker-compose-linux-x86_64 -o $HOME/.docker/cli-plugins/docker-compose
chmod +x $HOME/.docker/cli-plugins/docker-compose

What do these commands do?

The first line creates the directory where Docker looks for CLI plugins.
The second line downloads Docker Compose v5.5.0 for x86_64 Linux.
The third line makes the downloaded binary executable.

Verify Docker Compose is working by running:

docker compose version

You should see Docker Compose version v5.5.0 printed in the output.

Seeing 'command not found' for docker compose?

Make sure you ran chmod +x on the binary. Without execute permissions, your shell cannot run it.
Confirm the file is in the right location by running ls $HOME/.docker/cli-plugins/. You should see docker-compose listed.

Still stuck on Docker Compose?

Create your Compose file and deploy Redpanda

With Docker and Docker Compose ready, you can now define and launch the Redpanda broker and its web console. You will create a docker-compose.yml file that describes both containers and their networking.

Open a new file in nano by running:

nano docker-compose.yml

Paste the following as the first section of your Compose file. This defines the network, volume, and the Redpanda broker service:

name: kafka-lab
networks:
  redpanda_network:
    driver: bridge
volumes:
  redpanda-0: null
services:
  redpanda-0:
    command:
      - redpanda
      - start
      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
      - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092
      - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
      - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082
      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
      - --rpc-addr redpanda-0:33145
      - --advertise-rpc-addr redpanda-0:33145
      - --mode dev-container
      - --smp 1
      - --default-log-level=info
    image: docker.redpanda.com/redpandadata/redpanda:v26.2.2
    container_name: redpanda-0
    volumes:
      - redpanda-0:/var/lib/redpanda/data
    networks:
      - redpanda_network
    ports:
      - 18081:18081
      - 18082:18082
      - 19092:19092
      - 19644:9644

What does this section do?

The redpanda-0 service runs the Redpanda broker in dev-container mode, which uses sensible defaults for a single-node development cluster.
The --kafka-addr and --advertise-kafka-addr flags configure internal (container-to-container) and external (host-accessible) Kafka API listeners.
Port 19092 is exposed for the external Kafka API. Port 19644 maps the Admin API for Console to use.
The --smp 1 flag limits Redpanda to a single CPU core, keeping resource usage low on your lab instance.

Place your cursor at the end of the last line you just pasted. Press Enter to start a new line, then paste this second section directly below:

  console:
    container_name: redpanda-console
    image: docker.redpanda.com/redpandadata/console:v3.11.0
    networks:
      - redpanda_network
    entrypoint: /bin/sh
    command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
    environment:
      CONFIG_FILEPATH: /tmp/config.yml
      CONSOLE_CONFIG_FILE: |
        kafka:
          brokers: ["redpanda-0:9092"]
        schemaRegistry:
          enabled: true
          urls: ["http://redpanda-0:8081"]
        redpanda:
          adminApi:
            enabled: true
            urls: ["http://redpanda-0:9644"]
    ports:
      - 8080:8080
    depends_on:
      - redpanda-0

What does this section do?

The console service runs Redpanda Console, a web UI for inspecting topics, messages, and consumer groups.
It connects to the broker using the internal address redpanda-0:9092 over the shared redpanda_network.
Port 8080 is exposed so you can access the Console from your browser.
The depends_on field ensures the broker starts before the Console.

Save and close the file by pressing Ctrl+O, then Enter, then Ctrl+X.
Confirm the file was created by running:

ls docker-compose.yml

You should see docker-compose.yml listed in the output.

✔️ Awesome, I've got everything!

Great. Make sure you saved the file. You're ready to start the containers.

ⓧ I'd like to double check the full code

The full docker-compose.yml is two sections pasted back-to-back. Scroll up and confirm both are present in order:

First section: starts with name: kafka-lab and ends with - 19644:9644 (the redpanda-0 service).
Second section: starts with console: (2 spaces indent) and ends with - redpanda-0 (the console service).

If anything looks off, open the file again with nano docker-compose.yml and correct it. YAML is indentation-sensitive, so make sure nested items line up with spaces (not tabs).

Start the Redpanda stack in the background by running:

docker compose up -d

Docker pulls the Redpanda and Console images, then starts both containers. This download may take a minute or two on the first run.

What does the -d flag do?

The -d flag runs containers in detached mode, meaning they run in the background. Your terminal stays free for the next commands.

Confirm both containers are running by checking:

docker compose ps

You should see two containers listed: redpanda-0 and redpanda-console. Both should show a status of running.

Containers not showing as running?

If a container exited immediately, check its logs by running docker compose logs redpanda-0 to see what went wrong.
If you see a YAML parsing error, open docker-compose.yml in nano and check for misaligned indentation or tabs instead of spaces.
If Docker reports a port conflict, another process may be using one of the mapped ports. Run docker compose down, then docker compose up -d to retry.

Containers still not running?

Before you run the final check, what do you think rpk cluster info will show? The Redpanda broker just started. Predict whether it will report a healthy broker, and what address you expect to see.

Verify the broker is healthy by running:

docker compose exec redpanda-0 rpk cluster info

You should see output listing a single broker with its Kafka API address. The broker section shows the node ID and the address localhost:19092 for the external listener. This confirms your Redpanda cluster is healthy and accepting connections.

What is rpk?

The rpk CLI is Redpanda's built-in command-line tool for managing clusters, topics, and consumer groups. It ships inside the Redpanda container, so you access it through docker compose exec. You will use rpk throughout this project for all Kafka operations.

Your Kafka lab environment is live. You have a Redpanda broker and Console running on your EC2 instance, ready to handle event streaming. Next up, you'll create your first topic and send events through the broker.

Create Topics and Send Your First Events

In the previous step, you deployed a Redpanda broker and confirmed it is healthy. The broker is running, but it has no data flowing through it yet.

Before you can build anything useful with event streaming, you need to see the basic produce/consume cycle work end-to-end with your own eyes. In this step, you will create a topic, send messages into it, and read them back out. This is the foundation every Kafka concept builds on.

In this step, get ready to:

Create an orders topic with 3 partitions.
Produce and consume messages interactively using rpk CLI tools.
Inspect topic metadata to see partitions, offsets, and high watermarks.

Create the orders topic

A topic is a named stream of events. Think of it like a log file that multiple programs can write to and read from at the same time.

You will create an orders topic split across 3 partitions. Partitions let multiple consumers read in parallel, which you will see firsthand in later steps.

Create the orders topic with 3 partitions by running this command:

docker compose exec redpanda-0 rpk topic create orders -p 3

What does this command do?

The command runs rpk topic create inside the redpanda-0 container. The -p 3 flag tells Redpanda to split the topic into 3 partitions.

You run rpk inside the container because that is where the CLI is installed and can reach the broker directly.

You should see output confirming the topic was created:

TOPIC   STATUS
orders  OK

Seeing an error instead of OK?

If you see a topic already exists error, the topic was already created. You can continue to the next substep.
If you see a connection error, confirm your containers are still running with docker compose ps. If they are not, run docker compose up -d again.
Help me troubleshoot this rpk topic create error

Produce and consume messages interactively

Now that the topic exists, you will send a few messages into it and read them back. This is the produce/consume cycle. Every event streaming system works this way: one process writes events, another reads them.

Start an interactive producer session by running this command:

docker compose exec redpanda-0 rpk topic produce orders

What does this command do?

This starts an interactive session where every line you type becomes a message sent to the orders topic. The rpk topic produce command reads from STDIN, so it waits for your input line by line.

Type each of these lines one at a time, pressing Enter after each one:

order-001 placed
order-002 shipped
order-003 delivered

After each line, you see a confirmation message showing which partition and offset the message was written to.

Press Ctrl+C to stop the producer session.

Now read those messages back. Before you run the next command, what do you think the output will look like? Will you see just the text, or additional metadata?

Consume all messages from the orders topic by running this command:

docker compose exec redpanda-0 rpk topic consume orders --offset start

Each message appears as a JSON object with the topic, value, partition, and offset fields. You can see exactly which partition each message landed on and its position in that partition's log.

Press Ctrl+C to stop the consumer. By default, rpk topic consume runs continuously waiting for new messages.

What is an offset?

An offset is a message's position number within a partition. The first message in a partition gets offset 0, the next gets 1, and so on. Offsets never go backwards. They are how Kafka tracks where each consumer has read up to.

Not seeing your messages?

Make sure you included --offset start in the consume command. Without it, the consumer starts at the most recent offset and only shows new messages.
If you see an empty output, double check that you typed messages and pressed Enter after each line during the produce step.
Help me figure out why rpk topic consume is not showing my messages

Inspect topic metadata with rpk topic describe

You have produced and consumed messages. Now take a look under the hood at how those messages are distributed across the 3 partitions.

View the partition-level details of the orders topic by running this command:

docker compose exec redpanda-0 rpk topic describe orders -p

What does the -p flag do?

The -p flag tells rpk topic describe to print the detailed partitions section. By default, rpk topic describe only shows the summary and configuration. Adding -p reveals per-partition data including PARTITION, LEADER, LOG-START-OFFSET, and HIGH-WATERMARK.

The HIGH-WATERMARK is the offset of the next message that will be written to that partition. If a partition's high watermark is 2, it means 2 messages (at offsets 0 and 1) have been written there so far.

You should see a table with 3 rows, one for each partition. The HIGH-WATERMARK column shows how many messages landed in each partition.

Seeing all messages in one partition?

Without message keys, Redpanda uses a round-robin or sticky strategy that may place multiple messages in the same partition. This is expected for a small number of messages. In Step 3, you will add keys to control which partition each message goes to.
Help me understand why my messages all went to the same partition

✔️ Awesome, I've got everything!

Great. Make sure you have pressed Ctrl+C to stop any running consumer before moving on.

ⓧ I'd like to double check the full code

This step did not create any files. Here is a summary of the commands you ran, in order:

Created the topic: docker compose exec redpanda-0 rpk topic create orders -p 3
Produced messages: docker compose exec redpanda-0 rpk topic produce orders
Consumed messages: docker compose exec redpanda-0 rpk topic consume orders --offset start
Described the topic: docker compose exec redpanda-0 rpk topic describe orders -p

You have seen the full produce-consume cycle with your own eyes. You created a topic, sent messages, read them back, and inspected the partition metadata. Next up, you will build a bash script that produces a continuous stream of realistic order events with message keys.

Build a Continuous Bash Event Producer

You've proven the produce-consume pipeline works by typing messages manually. That confirmed Redpanda is healthy, but real Kafka systems process thousands of events per second from automated sources.

In this step, you'll write a bash script that continuously generates realistic order events with customer-based message keys. This script becomes the engine that powers every consumer group experiment in the next steps.

In this step, get ready to:

Write a producer.sh bash script that generates JSON order events with customer-based keys.
Run the producer and verify events are flowing to all 3 partitions.
Observe events arriving in Redpanda Console.

Create the producer script

In event streaming, every message can carry a key alongside its value. The broker uses this key to decide which partition the message lands in. Messages with the same key always go to the same partition.

Your script will assign each order a customer ID as its message key. This means all orders from customer-1 always land on the same partition, keeping that customer's events in order.

Create the producer.sh script with its configuration section by running this command:

cat > producer.sh << 'EOF'
#!/bin/bash
# Continuous order event producer for Kafka lab
# Produces JSON order events with customer keys to the 'orders' topic
# Usage: ./producer.sh [interval_seconds]

TOPIC="orders"
INTERVAL=${1:-1}
COUNT=0

echo "Starting order producer (interval: ${INTERVAL}s)..."
echo "Press Ctrl+C to stop."
EOF

What does this code do?

cat > producer.sh << 'EOF' creates a new file called producer.sh and writes everything between the two EOF markers into it. The single quotes around 'EOF' prevent the shell from expanding variables in the heredoc, so characters like $ are written literally to the file.
TOPIC="orders" sets the target topic. INTERVAL=${1:-1} lets you optionally pass a custom interval in seconds when running the script. It defaults to 1 second.
COUNT=0 tracks how many events the script has produced so far.

Append the main event loop to the script by running this command:

cat >> producer.sh << 'EOF'

while true; do
  COUNT=$((COUNT + 1))
  CUSTOMER_ID="customer-$((RANDOM % 5 + 1))"
  ORDER_ID=$((COUNT + RANDOM % 1000))
  AMOUNT=$((RANDOM % 500 + 10))
  TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

  EVENT="{\"order_id\": ${ORDER_ID}, \"customer_id\": \"${CUSTOMER_ID}\", \"amount\": ${AMOUNT}, \"timestamp\": \"${TIMESTAMP}\"}"

  echo "${CUSTOMER_ID}:${EVENT}" | docker compose exec -T redpanda-0 rpk topic produce "${TOPIC}" -f '%k:%v\n'

  echo "[${COUNT}] Produced: ${CUSTOMER_ID} -> \$${AMOUNT}"
  sleep "${INTERVAL}"
done
EOF

What does this loop do?

CUSTOMER_ID="customer-$((RANDOM % 5 + 1))" picks a random customer ID from customer-1 through customer-5 for each event. This gives you 5 distinct message keys.
The EVENT variable builds a JSON payload with an order ID, customer ID, dollar amount, and UTC timestamp.
echo "${CUSTOMER_ID}:${EVENT}" pipes the key and value separated by a colon into rpk topic produce. The -f '%k:%v\n' flag tells rpk to parse the input as key:value pairs.
Because rpk hashes the key to determine the partition, all orders from the same customer always land on the same partition.

Make the script executable by running:

chmod +x producer.sh

Confirm the script was created by running:

ls -l producer.sh

You should see producer.sh listed with execute permissions (-rwxr-xr-x or similar).

✔️ Awesome, I've got everything!

Great. Double check you've saved the file, then move on.

ⓧ I'd like to double check the full code

Here is the complete producer.sh file. Compare it against your version:

#!/bin/bash
# Continuous order event producer for Kafka lab
# Produces JSON order events with customer keys to the 'orders' topic
# Usage: ./producer.sh [interval_seconds]

TOPIC="orders"
INTERVAL=${1:-1}
COUNT=0

echo "Starting order producer (interval: ${INTERVAL}s)..."
echo "Press Ctrl+C to stop."

while true; do
  COUNT=$((COUNT + 1))
  CUSTOMER_ID="customer-$((RANDOM % 5 + 1))"
  ORDER_ID=$((COUNT + RANDOM % 1000))
  AMOUNT=$((RANDOM % 500 + 10))
  TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

  EVENT="{\"order_id\": ${ORDER_ID}, \"customer_id\": \"${CUSTOMER_ID}\", \"amount\": ${AMOUNT}, \"timestamp\": \"${TIMESTAMP}\"}"

  echo "${CUSTOMER_ID}:${EVENT}" | docker compose exec -T redpanda-0 rpk topic produce "${TOPIC}" -f '%k:%v\n'

  echo "[${COUNT}] Produced: ${CUSTOMER_ID} -> \$${AMOUNT}"
  sleep "${INTERVAL}"
done

If you need to recreate the file, you can run rm producer.sh and then re-run both cat commands from above.

Don't see the file?

Make sure you ran both cat commands. The first uses > (create), the second uses >> (append). Running only one produces an incomplete file.
Check that each heredoc ends with EOF on its own line with no leading spaces.
help me troubleshoot my producer.sh script creation

Run the producer and verify partition distribution

The script is ready. Time to start streaming events and confirm they're spreading across all 3 partitions.

Start the producer by running:

./producer.sh

You should see numbered output lines scrolling in your terminal, each showing a customer ID and dollar amount.

What should I see?

Each line of output looks like [1] Produced: customer-3 -> $247. The customer ID changes randomly between customer-1 and customer-5. Leave this running.

The producer is now occupying your terminal. You need a second SSH session to run verification commands.

Open a new terminal window on your local machine.
SSH into your EC2 instance from this new terminal with port forwarding for Redpanda Console by running:

ssh -i [[PEMPATH="PATH TO YOUR .PEM FILE"]] -L 8080:localhost:8080 ec2-user@[[EC2="YOUR PUBLIC IPV4 DNS"]]

What does -L 8080:localhost:8080 do?

The -L flag creates an SSH tunnel. It forwards port 8080 on your local machine to port 8080 on the EC2 instance. This lets you access Redpanda Console from your browser without opening ports in the security group.

Before you run the next command, what do you think the HIGH-WATERMARK values will look like compared to when you last checked them in Step 2?

Check the partition offsets by running:

docker compose exec redpanda-0 rpk topic describe orders -p

You should see the HIGH-WATERMARK column with values climbing across all 3 partitions. Each partition's watermark is higher than it was during your interactive tests because the producer has been streaming events continuously.

All messages landing on one partition?

Check that your producer.sh includes the -f '%k:%v\n' flag. Without this flag, rpk ignores the key and distributes messages differently.
Verify the echo line sends ${CUSTOMER_ID}:${EVENT} with a colon separating the key from the value. The colon must match the %k:%v format.
help me debug why messages are not distributing across partitions

View events in Redpanda Console

The SSH tunnel you created in the previous substep forwards port 8080 to your local machine. You can now use your browser to inspect the events visually.

Open your web browser.
Navigate to http://localhost:8080 to open Redpanda Console.
Click on **Topics** in the left sidebar.
Click on the orders topic.

You should see messages appearing with their customer-based keys (like customer-3) and JSON payloads containing order_id, customer_id, amount, and timestamp fields. New messages keep arriving as the producer runs.

Can't load localhost:8080?

Make sure you connected to the EC2 instance with the -L 8080:localhost:8080 flag in your second SSH session. A regular SSH connection without -L does not forward the port.
Check that Redpanda Console is still running by running docker compose ps in your SSH session. You should see redpanda-console listed as running.
If port 8080 is already in use on your local machine, close the application using it and reconnect.
help me access Redpanda Console through an SSH tunnel

Your producer is streaming events continuously across all 3 partitions. In the next step, you'll start consuming these events with a consumer group and watch offset lag grow and shrink in real time.

Consumer Groups and Offset Lag Monitoring

Your producer script is streaming order events across all 3 partitions. Right now, nobody is reading those events in a tracked way.

Without a consumer group, the broker has no record of what has been read and no way to measure how far behind a reader has fallen. In this step, you will create a consumer group, stop it while events keep flowing, and use rpk group describe to watch offset lag grow and shrink in real time.

In this step, get ready to:

Start a consumer in the order-processor group and observe tracked consumption.
Stop the consumer and watch offset lag grow while the producer keeps running.
Restart the consumer and watch lag drop back toward zero.

Start a consumer group in a new terminal

When you consume with a group, Redpanda tracks which messages the group has already read by storing offsets per partition. This tracking is what makes lag measurement possible.

Open a new terminal window on your Windows machine.
SSH into your EC2 instance using the same SSH command from Step 1.

Start consuming from the orders topic as part of the order-processor consumer group by running this command:

docker compose exec redpanda-0 rpk topic consume orders --group order-processor

You should see JSON order events scrolling in the terminal as the consumer reads through messages.

What does --group do?

Without --group, rpk topic consume reads messages without storing any progress. The broker has no record of where you left off.

With --group order-processor, the broker tracks an offset for each partition. It knows exactly which messages this group has already read.

Let the consumer run for about 10 seconds so it catches up on existing messages.
Press Ctrl+C to stop the consumer.

The producer is still running in your other terminal, sending new events every second. The orders topic keeps filling up, but nobody is reading anymore.

Not seeing any messages?

Check that producer.sh is still running in your first terminal. If it stopped, restart it with ./producer.sh.
Confirm you are in the home directory where docker-compose.yml is located. Run cd ~ to return to it.
Why isn't my consumer receiving messages from the orders topic?

Observe offset lag growing

Your consumer is stopped, but the producer keeps sending events. Before you check the group status, what do you think happened to the messages that arrived while the consumer was offline?

Check the consumer group's status by running this command:

docker compose exec redpanda-0 rpk group describe order-processor

You should see a table with one row per partition, each showing three key numbers.

What do these fields mean?

CURRENT-OFFSET is the last message your consumer read in each partition. This number is frozen because the consumer is stopped.
LOG-END-OFFSET is the latest message the producer has written to each partition. This number keeps climbing because the producer is still running.
LAG is the gap between the two. It tells you how many unread messages are waiting in each partition.

Wait about 15 seconds for the producer to send more events.
Run the same command again:

docker compose exec redpanda-0 rpk group describe order-processor

The LAG numbers have grown since your last check. CURRENT-OFFSET stayed the same because the consumer is stopped, but LOG-END-OFFSET kept climbing.

Seeing LAG of 0 everywhere?

Make sure the consumer is actually stopped. If it is still consuming in the background, it reads events as fast as the producer sends them.
Check that producer.sh is still running. If the producer stopped, no new events arrive and lag stays at zero.
Wait 15-20 seconds and try the describe command again. The producer needs time to send enough events for lag to become visible.
Why is my consumer group lag showing zero even though the consumer is stopped?

Restart the consumer and watch lag drop

If you restart the consumer in the same order-processor group, do you think it will re-read everything from the beginning, or pick up where it left off? What do you expect to happen to the LAG values?

Restart the consumer in the same group by running this command:

docker compose exec redpanda-0 rpk topic consume orders --group order-processor

You should see the consumer racing through the backlog of unread messages that piled up while it was stopped.

Let the consumer run for about 15 seconds to catch up.
Press Ctrl+C to stop the consumer.
Check the group status one more time by running this command:

docker compose exec redpanda-0 rpk group describe order-processor

LAG is now at or near zero. The consumer picked up from its last committed offset, not from the beginning of the topic.

Why didn't the consumer start from the beginning?

The broker stored the order-processor group's offsets. When you restarted a consumer in the same group, it resumed from exactly where it left off.

This is why consumer groups are essential for production workloads. They guarantee that each message is processed once without re-reading the entire topic on every restart.

LAG still showing high numbers?

The producer may have sent many events while the consumer was stopped. Let the consumer run for 30 seconds or more to catch up, then stop and check again.
Make sure you used the same group name order-processor when restarting the consumer. A different group name creates a separate group with its own offsets.
Why is my consumer group lag not dropping to zero?

You can now read offset lag like a dashboard and understand exactly how far behind a consumer has fallen. In the next step, you will scale the consumer group by adding a second consumer and watch Redpanda automatically redistribute partitions between them.

Scale Consumers and Observe Rebalancing

Right now, a single consumer is reading all 3 partitions of the orders topic. In a production system, that is a bottleneck. If that one consumer slows down or crashes, the entire event pipeline stalls.

Consumer groups solve this by distributing partitions across multiple consumers. When a new consumer joins the group, the broker triggers a rebalance to redistribute partitions. When a consumer leaves, the remaining members pick up its partitions automatically. In this step, you will scale the order-processor group and watch rebalancing happen live.

In this step, get ready to:

Add a second consumer to the order-processor group and see partitions split between two members.
Remove a consumer and watch partition rebalancing to the surviving member.
Use rpk group list and rpk group describe -t to explore group-level summaries.

Start a second consumer in the same group

With 3 partitions and 2 consumers in the same group, the broker cannot split evenly. One consumer gets 2 partitions and the other gets 1. You are about to see this split happen in real time.

Make sure your first consumer is still running in one of your SSH terminals from Step 4. If you stopped it, restart it by running this command:

docker compose exec redpanda-0 rpk topic consume orders --group order-processor

You should see consumed messages scrolling in that terminal. Leave it running.

Open a new SSH session to your EC2 instance. This is your third terminal.
In your new terminal, start a second consumer in the same group by running this command:

docker compose exec redpanda-0 rpk topic consume orders --group order-processor

Both consumer terminals should now be scrolling with messages. Notice that each terminal shows different messages. The broker has split the partitions between them.

What is partition rebalancing?

When a consumer joins or leaves a group, the broker redistributes partitions among the remaining members. This process is called a rebalance. During a rebalance, consumption briefly pauses while the broker calculates new partition assignments.

With 3 partitions and 2 consumers, one consumer gets assigned 2 partitions and the other gets 1. If you added a third consumer, each would get exactly 1 partition. A fourth consumer would sit idle because there are no partitions left to assign.

Open one more SSH session to your EC2 instance. You will use this terminal for monitoring commands.

Before you check the partition assignments, how do you think the 3 partitions are split between your 2 consumers? Do you expect an even split?

In your monitoring terminal, check the partition assignments by running this command:

docker compose exec redpanda-0 rpk group describe order-processor

What should I see?

The output shows a row for each partition. Look at the CLIENT-ID column. You should see two different client IDs, confirming two consumers are active.

One client ID appears next to 2 partitions.
The other client ID appears next to 1 partition.
The LAG values should be low or zero since both consumers are actively reading.

Still seeing only one member?

The rebalance may take a few seconds to complete. Wait 5-10 seconds and run the rpk group describe command again.

Make sure both consumers are running the exact same command with --group order-processor. If the group names don't match, each consumer joins a different group.

Help me figure out why rpk group describe only shows one consumer member

Stop a consumer and observe the rebalance

In production, consumers go down all the time. Servers crash, deployments restart, and network issues disconnect members. The broker handles this automatically by reassigning orphaned partitions to the surviving consumers.

Switch to one of your two consumer terminals (not the producer terminal).
Press Ctrl+C to stop that consumer.

Before you check the partition assignments, what do you think happened to the partitions that belonged to the consumer you just stopped?

Wait about 10 seconds for the rebalance to complete.
In your monitoring terminal, check the partition assignments by running this command:

docker compose exec redpanda-0 rpk group describe order-processor

All 3 partitions now belong to the single surviving consumer. The broker detected the missing heartbeats from the stopped consumer and reassigned its partitions.

Why might LAG spike during a rebalance?

During a rebalance, consumption briefly pauses while the broker recalculates partition assignments. Meanwhile, your producer.sh keeps producing events. That gap creates a small burst of lag that the surviving consumer catches up on once the rebalance finishes.

Still seeing two members after stopping a consumer?

The broker needs time to detect the missing heartbeats. Wait 15-20 seconds and run rpk group describe order-processor again. The stale member should disappear once the session timeout expires.

Help me understand why the stopped consumer still appears in rpk group describe

Explore group-level commands

So far you have been inspecting one specific group. In a real system, you might manage dozens of consumer groups. Two more rpk commands give you the broader picture.

In your monitoring terminal, list all known consumer groups by running this command:

docker compose exec redpanda-0 rpk group list

You should see order-processor listed along with its state. This command is your starting point when you need to see which groups exist on a cluster.

Now view the aggregated lag per topic by running this command:

docker compose exec redpanda-0 rpk group describe order-processor -t

What does the -t flag do?

The -t flag (short for --print-lag-per-topic) rolls up the per-partition lag into a single number per topic. Instead of seeing lag for each of the 3 partitions individually, you see one total lag value for the entire orders topic.

This is useful when a group consumes from multiple topics. One glance tells you which topic is falling behind without scanning every partition row.

Before you verify the full picture, what do you expect to see when you run rpk group describe order-processor right now? How many members and partitions?

Confirm the final state by running this command:

docker compose exec redpanda-0 rpk group describe order-processor

You should see one member holding all 3 partitions. The LAG values should be low or zero, confirming the surviving consumer has caught up after the rebalance.

You can now confidently demonstrate how consumer groups scale and rebalance in real time. You have seen partitions split across members, watched them consolidate when a consumer drops, and used rpk to monitor it all. Up next, the Secret Mission challenges you to diagnose the difference between a stuck consumer and a slow consumer using the lag patterns you have been monitoring.

Secret mission

Diagnose a Stuck Consumer vs. a Slow Consumer

When a consumer falls behind in Kafka, it could be crashed or just slow. These two problems leave different fingerprints in lag metrics. Simulate both failure scenarios and build a diagnostic cheat sheet you can reference in interviews.

Clean Up Your Resources

This project runs on an AWS EC2 instance that incurs hourly charges. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Cost warning

Your t3.medium EC2 instance costs approximately $0.04 per hour while running. If you leave it running overnight or over a weekend, charges will add up quickly.

Stop or terminate the instance as soon as you are finished to avoid unexpected costs on your AWS bill.

Resources you used:

1 EC2 instance (t3.medium, Amazon Linux 2023) running Redpanda broker and Redpanda Console Docker containers.

Keep everything running

No action needed. Choose this if you are still actively experimenting with your Kafka lab and plan to keep running consumer group exercises.

Your t3.medium instance costs approximately $0.04 per hour while it is running.
Your producer.sh and slow-consumer.sh scripts may still be running in their SSH terminals.
All consumer groups (order-processor, order-audit, order-slow) and the orders topic remain available for further experimentation.

Pause - I'll come back to this later

Stop your EC2 instance to pause compute charges while keeping your files and configuration on disk. The instance retains its EBS volume at a minimal storage cost.

Press Ctrl+C in any terminals where producer.sh or slow-consumer.sh are still running.
In the AWS Console, search for EC2 and open the EC2 dashboard.
Click Instances in the left navigation pane.
Select your EC2 instance by clicking its checkbox.
Click Instance state.
Select Stop instance.
Click Stop to confirm.

Resuming your lab later

To restart, select your stopped instance in the EC2 console. Click Instance state, then select Start instance. Wait for the instance to reach the running state, then SSH back in.

Docker will not auto-start after the instance restarts. Bring your containers back online by running these commands:

sudo service docker start
docker compose up -d

Delete - I don't want to use this again

Remove all project resources permanently. Terminating the EC2 instance deletes it along with its default EBS volume. No further charges will apply.

Press Ctrl+C in any terminals where producer.sh or slow-consumer.sh are still running.
In the AWS Console, search for EC2 and open the EC2 dashboard.
Click Instances in the left navigation pane.
Select your EC2 instance by clicking its checkbox.
Click Instance state.
Select Terminate (delete) instance.
Click Terminate (delete) to confirm.

Once the instance status changes to terminated, all resources from this project are gone. Your docker-compose.yml, producer.sh, and slow-consumer.sh files are permanently deleted along with the instance.

Nice Work!

You just built a fully operational Kafka lab on AWS and used it to produce events, run consumer groups, and diagnose lag in real time.

You've learned how to:

Deploy a Kafka-compatible broker on an AWS EC2 instance using Docker Compose, create topics with multiple partitions, and produce and consume events using rpk CLI tools.
Build a continuous bash event producer that generates realistic order events with message keys distributed across partitions, and monitor offset lag per partition with rpk group describe in real time.
Scale a consumer group by adding and removing consumers, and observe partition rebalancing and its effect on lag.
Secret Mission: Diagnosed the difference between a stuck consumer (frozen offset, empty group state) and a slow consumer (advancing offset, stable group state) using lag patterns.

Ready to quiz yourself?