Real-Time Order Pipeline with Redis

Build a durable order pipeline using Redis Pub/Sub and Streams on AWS EC2.

Introduction

30 Second Summary

When you order something online, that order needs to reach the warehouse, the payment system, and the shipping tracker all at once. Behind the scenes, messaging systems handle this coordination in real time.

In this project, you will build a real-time order processing pipeline using Redis on AWS. You'll create two messaging systems side by side to discover firsthand why durable messaging matters for production backends.

What You'll Build

You fire off five orders from a producer script, watch each one appear in a subscriber's terminal the instant it arrives, then pull the subscriber offline and run the same test to prove which messages survive.

By the end of this project, you'll have:

  • A running Redis server on your own EC2 instance that you can query from the command line to store and retrieve order data.
  • A Pub/Sub notification system that delivers order events in real time. You'll also see what happens when the subscriber disconnects: those messages are gone forever.
  • A durable order pipeline built on Redis Streams with consumer groups that survives the same offline test. Stop the consumer, publish orders, restart it, and every order is still there.
  • Secret Mission: Recover orphaned messages from a crashed consumer using the pending entries list.

Are there any prerequisites?

You'll need an AWS account with free tier eligibility, basic Python knowledge, and familiarity with the AWS Console. Experience launching EC2 instances and connecting via SSH will help you move through Step 1 quickly.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Launch an EC2 Instance and Install Redis

You know what you're building. Now you need a server to build it on. Every messaging pattern in this project runs on Redis, and Redis needs a Linux machine to host it.

In this step, you'll launch a Linux server on AWS and install Redis yourself. Setting it up from scratch (instead of using a managed service) gives you a clear view of what is happening under the hood.

In this step, get ready to:
  • Launch an EC2 instance with Amazon Linux 2023.
  • Install and start Redis on the instance.
  • Install Python tools to connect to Redis from code.
Launch an EC2 instance in the AWS Console

An EC2 instance is a virtual server in AWS that you can configure and connect to remotely. You will launch one running Amazon Linux 2023, which comes with Python 3.9 pre-installed.

  • Open console.aws.amazon.com in your browser and sign in to the AWS Management Console.
  • In the search bar at the top, type EC2 and select EC2 from the results.
  • In the Launch instance pane on the dashboard, click Launch instance.
  • Under Name and tags, enter redis-messaging in the Name field.
  • Under Application and OS Images (Amazon Machine Image), choose Quick Start and select Amazon Linux.
  • Confirm that the selected AMI is marked Free Tier eligible.
  • Under Instance type, select t2.micro.

Which instance type is free for me?

AWS updated their free tier on July 15, 2025. If your account was created before that date, t2.micro is free for 750 hours per month for 12 months.

If your account was created on or after that date, t2.micro is not free tier eligible. Choose t3.micro instead, which is covered under the new free tier program. Either instance type works for this project.

  • Under Key pair (login), click Create new key pair.
  • Enter a name for your key pair (for example, redis-key).
  • Keep the key pair type as RSA and the file format as .pem.
  • Click Create key pair. The .pem file downloads to your computer automatically.

Keep your .pem file safe

This file is your only way to SSH into your instance. If you lose it, you will need to create a new instance. Note which folder it downloaded to (usually ~/Downloads).

  • Under Network settings, click Edit.
  • Keep Create security group selected.
  • In the SSH rule row, change Source type from Anywhere to My IP. This restricts access so only your computer can connect.
  • Review your configuration in the Summary panel on the right, then click Launch instance.

What is a security group?

A security group acts as a virtual firewall for your instance. By choosing My IP, you are telling AWS to only allow SSH connections from your current IP address. This prevents the rest of the internet from attempting to connect.

  • Click the instance ID link in the Success notification to open the Instances page.
  • Wait for your instance's Instance state to change from pending to running. This usually takes about 30 seconds.

You should see your redis-messaging instance listed with the state running.

Instance stuck in pending?

Give it up to two minutes. If it does not reach running, click the refresh icon above the instances table to reload the status.

If the instance terminates immediately, your account may have a quota or region issue. Try launching in a different region (e.g. us-east-1). Help me troubleshoot my EC2 instance not launching.

Connect to your instance and install Redis

Your server is up. Now you need to connect to it from your local computer using SSH and install Redis.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.

Before you can SSH in, you need to lock down your key file's permissions and find your instance's public address.

  • Restrict permissions on your .pem file by running this command:
chmod 400 [[PEMPATH="~/Downloads/your-key-name.pem"]]

What does chmod 400 do?

This command sets the file to read-only for you and no access for anyone else. SSH refuses to use a key file that other users on the machine could read.

  • Switch back to the AWS Console in your browser.
  • On the Instances page, click the checkbox next to your redis-messaging instance.
  • In the details panel below, find the Public IPv4 address field and copy the address. It looks something like 3.91.200.42.
  • Back in Terminal, connect to your instance by running:
ssh -i [[PEMPATH="~/Downloads/your-key-name.pem"]] ec2-user@[[EC2_IP="your EC2 public IPv4 address"]]

What are these parts?

  • -i tells SSH which key file to use for authentication.
  • ec2-user is the default username on Amazon Linux instances.
  • The address after @ is the public IP of your instance.
  • When prompted to confirm the connection, type yes and press Enter.

You should see a welcome message from Amazon Linux and a command prompt. You are now inside your EC2 instance.

Connection timed out or refused?

Double-check that you selected My IP for the SSH security group rule, not Anywhere or a custom range. If your IP address has changed since you launched the instance (e.g. you switched networks), update the security group rule in the EC2 Console under Security > Security groups.

Also confirm you are using the correct .pem file path. If SSH says "Permission denied", re-run the chmod 400 command on your key file. Help me troubleshoot my SSH connection to EC2.

Now install Redis on the instance. Amazon Linux 2023 provides Redis 6 as a ready-made package.

  • Install Redis by running:
sudo dnf install redis6 -y
  • Start Redis and configure it to start automatically on boot by running:
sudo systemctl enable --now redis6

What does enable --now do?

The enable part tells the system to start Redis automatically whenever the instance reboots. The --now flag also starts it immediately, so you do not have to run a separate start command.

Verify Redis and install Python tools

Redis should now be listening on port 6379 (its default port). Before moving on, you need to confirm it is actually running and install the Python library you will use to talk to Redis from code.

Before you run the next command, what do you think Redis will respond with if it is running correctly?

  • Test the Redis connection by running:
redis-cli ping

You should see PONG printed on the next line. That single word confirms Redis is installed, running, and accepting connections.

Don't see PONG?

If you see command not found, the redis6 package may not have installed correctly. Re-run sudo dnf install redis6 -y and check for errors in the output.

If you see Could not connect, the service may not be running. Re-run sudo systemctl enable --now redis6 and try the ping again. Help me get Redis running on my EC2 instance.

Now install the Python tools you will need for the rest of the project.

  • Install pip (the Python package manager) by running:
sudo dnf install python3-pip -y
  • Install the redis-py library, which lets Python talk to your Redis server, by running:
pip3 install redis==5.2.1
  • Confirm the library installed correctly by running:
python3 -c "import redis; print(redis.__version__)"

You should see 5.2.1 printed to the terminal. Your EC2 instance now has Redis running and a Python library ready to interact with it.

Seeing a ModuleNotFoundError?

If Python cannot find the redis module, pip3 may have installed it to a different Python path. Re-run pip3 install redis==5.2.1 and check that the output says "Successfully installed". Help me fix my redis-py installation.

Your server is ready and Redis is listening. Next up, you will explore Redis basics and build your first order store.

Explore Redis and Build an Order Store

Your Redis server is running on your EC2 instance. Before you build any messaging systems, you need to understand the basics: how data goes into Redis, how it comes back out, and how it can expire automatically.

This step gives you the foundation to appreciate what messaging patterns add on top of simple key-value operations. You will store real order data in Redis and retrieve it from the command line.

In this step, get ready to:
  • Experiment with basic Redis commands to store, retrieve, and expire data.
  • Create a Python script that stores order data as Redis Hashes.
  • Verify the stored data from the command line.
Experiment with basic Redis commands

Redis is a key-value store. You give it a name (the key) and a value, and it holds onto that pair in memory. Let's see this in action.

  • Store a value in Redis by running this command in your SSH terminal:
redis-cli SET greeting "hello redis"
  • Retrieve the value by running:
redis-cli GET greeting

You should see "hello redis" printed back. The data went in with SET and came back with GET.

Redis keys can also expire automatically. Let's set a 10-second countdown on your greeting key.

  • Set an expiration on the key by running:
redis-cli EXPIRE greeting 10
  • Check how many seconds remain on the key by running:
redis-cli TTL greeting

You should see a number showing the seconds left before the key disappears.

  • Wait about 10 seconds, then try to retrieve the key again by running:
redis-cli GET greeting

This time you should see (nil). The key expired and Redis automatically removed it.

What is a key-value store?

A key-value store works like a dictionary. You look up data by its key (the name) and get back the value. Redis holds everything in memory, which makes reads and writes extremely fast.

The EXPIRE command tells Redis to delete a key after a set number of seconds. TTL (Time To Live) shows how many seconds remain before that happens.

Seeing an error when running redis-cli?

If you see a connection refused message, Redis may not be running. Start it again by running sudo systemctl start redis6 and try again.

If TTL already returns -2, the key expired before you checked. Run the SET and EXPIRE commands again and check TTL more quickly.

Still stuck? Help me troubleshoot my redis-cli commands.

Create a Python script to store orders

Simple keys hold single values. For structured data like an order (customer name, item, quantity, price), Redis offers Hashes. A Hash stores multiple field-value pairs under one key, like a row in a database table.

  • Create a new file called store_orders.py by running:
nano store_orders.py

This opens the nano text editor. You will paste your Python code here, then save the file.

  • Copy the following code and paste it into the nano editor:
import redis

# Connect to the local Redis server
r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Define three sample orders with their details
orders = [
    {'order_id': '1001', 'customer': 'Alice', 'item': 'Laptop', 'quantity': '1', 'price': '999.99'},
    {'order_id': '1002', 'customer': 'Bob', 'item': 'Headphones', 'quantity': '2', 'price': '149.50'},
    {'order_id': '1003', 'customer': 'Carol', 'item': 'Keyboard', 'quantity': '1', 'price': '79.99'},
]

# Store each order as a Redis Hash and print confirmation
for order in orders:
    key = f"order:{order['order_id']}"
    r.hset(key, mapping=order)
    print(f"Stored {key}: {order['item']} for {order['customer']}")

print(f"\nTotal orders stored: {len(orders)}")

What does this code do?

  • redis.Redis(host='localhost', port=6379, decode_responses=True) creates a connection to your local Redis server. The decode_responses=True setting tells redis-py to return strings instead of raw bytes.
  • The orders list contains three dictionaries, each representing one order with its customer, item, quantity, and price.
  • r.hset(key, mapping=order) stores each order as a Redis Hash. The mapping parameter tells Redis to create a field for every key-value pair in the dictionary.
  • Each order gets a key like order:1001. The colon is a common Redis naming convention to group related keys.
  • Save the file by pressing Ctrl+O, then press Enter to confirm the filename.
  • Exit nano by pressing Ctrl+X.
  • Run the script by running:
python3 store_orders.py

You should see three confirmation lines, one for each order, followed by a total count.

What should I see?

Your terminal should print something like:

Stored order:1001: Laptop for Alice
Stored order:1002: Headphones for Bob
Stored order:1003: Keyboard for Carol

Total orders stored: 3

Seeing a ModuleNotFoundError?

If you see an error about the redis module not being found, redis-py may not be installed. Run pip3 install redis==5.2.1 and try again.

If you see a connection error, make sure Redis is running with sudo systemctl status redis6. If it says inactive, start it with sudo systemctl start redis6.

Still stuck? Help me debug my store_orders.py script on EC2.

✔️ Awesome, I've got everything!

Great. Double check you saved store_orders.py and that your output shows all three orders stored.

ⓧ I'd like to double check the full code

Here is the complete store_orders.py file. Compare it with yours line by line.

import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

orders = [
    {'order_id': '1001', 'customer': 'Alice', 'item': 'Laptop', 'quantity': '1', 'price': '999.99'},
    {'order_id': '1002', 'customer': 'Bob', 'item': 'Headphones', 'quantity': '2', 'price': '149.50'},
    {'order_id': '1003', 'customer': 'Carol', 'item': 'Keyboard', 'quantity': '1', 'price': '79.99'},
]

for order in orders:
    key = f"order:{order['order_id']}"
    r.hset(key, mapping=order)
    print(f"Stored {key}: {order['item']} for {order['customer']}")

print(f"\nTotal orders stored: {len(orders)}")
Verify the stored order data

The script ran successfully, but how do you know the data actually landed in Redis? Let's verify it directly from the command line.

Before you run the next command, what do you think you will see? Think about the fields you stored for each order.

  • Retrieve all the fields of the first order by running:
redis-cli HGETALL order:1001

You should see all five fields listed as alternating field names and values: order_id, customer, item, quantity, and price. Alice's Laptop order is sitting right there in Redis.

  • List all the order keys stored in Redis by running:
redis-cli KEYS "order:*"

You should see three keys listed: order:1001, order:1002, and order:1003.

What does HGETALL do?

HGETALL returns every field and value in a Redis Hash. Unlike GET (which works on simple string keys), HGETALL pulls out the full structure of a Hash, so you can see all the fields at once.

KEYS "order:*" uses a wildcard pattern to find every key that starts with order:. This is handy for quick inspection, though in production you would use SCAN instead to avoid blocking the server.

Seeing an empty list or (nil)?

If HGETALL order:1001 returns nothing, your script may not have run successfully. Run python3 store_orders.py again and check for errors in the output.

Make sure you are using HGETALL (for Hashes), not GET (for simple strings). Running GET order:1001 on a Hash key will return an error.

Need help? Help me verify my Redis order data.

You now have a working data layer. Three orders are stored in Redis, and you can retrieve them instantly. But there is a problem: if another service needs to know when a new order arrives, it has to keep asking Redis over and over. There is no way to get notified in real time.

In the next step, you will solve this with Redis Pub/Sub, a messaging pattern that pushes events to listeners the instant they happen.

Build a Pub/Sub Notification System

Your order store is working. You can save orders to Redis and pull them back out with redis-cli. But there is a problem: if another service needs to know when a new order arrives, it has to keep asking Redis over and over again.

That constant asking is called polling, and it wastes resources. Pub/Sub solves this by pushing events to subscribers the instant they happen. In this step, you will build a real-time notification listener that reacts to new orders without polling.

In this step, get ready to:
  • Create a subscriber script that listens for order events in real time.
  • Create a publisher script that broadcasts five order events.
  • Run both scripts across two terminals and watch messages arrive instantly.
Create the subscriber script

In a Pub/Sub system, a subscriber registers its interest in a channel and then waits. The moment any other client publishes a message to that channel, every active subscriber receives it instantly.

  • In your SSH terminal, create subscriber.py by running:
nano subscriber.py
  • Paste the following code into the editor:
import redis
import json

# Connect to Redis
r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Create a Pub/Sub object and subscribe to the order-events channel
p = r.pubsub()
p.subscribe('order-events')

print("Listening for order events on 'order-events' channel...")
print("Press Ctrl+C to stop.\n")

# Listen for incoming messages and print parsed order details
for message in p.listen():
    if message['type'] == 'message':
        order = json.loads(message['data'])
        print(f"[NOTIFICATION] Order {order['order_id']}: "
              f"{order['customer']} bought {order['item']} "
              f"for ${order['total']}")
  • Save and exit nano by pressing Ctrl+X, then Y, then Enter.

What does this code do?

  • r.pubsub() creates a Pub/Sub listener object connected to your Redis server.
  • p.subscribe('order-events') tells Redis you want every message published to the order-events channel.
  • p.listen() is a generator that blocks and waits for new messages. Each message is a dictionary with a type field. The script filters for 'message' types (ignoring subscription confirmations) and parses each one from JSON.
  • Test that the subscriber starts correctly by running:
python3 subscriber.py

You should see Listening for order events on 'order-events' channel... printed to your terminal. The script is now waiting for messages.

  • Press Ctrl+C to stop the subscriber for now. You will run it again shortly in its own dedicated terminal.

Seeing a ModuleNotFoundError?

  • Make sure you installed redis-py with pip3 install redis==5.2.1 in Step 1. Run the install command again if needed.
  • Check that the first line of your file says import redis (lowercase), not import Redis.
  • Help me debug my subscriber.py script

✔️ Awesome, I've got everything!

Great. Make sure you have saved subscriber.py before moving on.

ⓧ I'd like to double check the full code

import redis
import json

r = redis.Redis(host='localhost', port=6379, decode_responses=True)
p = r.pubsub()
p.subscribe('order-events')

print("Listening for order events on 'order-events' channel...")
print("Press Ctrl+C to stop.\n")

for message in p.listen():
    if message['type'] == 'message':
        order = json.loads(message['data'])
        print(f"[NOTIFICATION] Order {order['order_id']}: "
              f"{order['customer']} bought {order['item']} "
              f"for ${order['total']}")
Create the publisher script

The subscriber is ready to listen. Now you need something to send messages to it. The publisher will simulate five incoming orders and broadcast each one to the order-events channel.

  • Create publisher.py by running:
nano publisher.py
  • Paste the following code into the editor:
import redis
import json
import time

# Connect to Redis
r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Five sample order events to publish
orders = [
    {'order_id': '2001', 'customer': 'Diana', 'item': 'Monitor', 'total': '349.99'},
    {'order_id': '2002', 'customer': 'Eve', 'item': 'Mouse', 'total': '29.99'},
    {'order_id': '2003', 'customer': 'Frank', 'item': 'Webcam', 'total': '89.99'},
    {'order_id': '2004', 'customer': 'Grace', 'item': 'USB Hub', 'total': '24.99'},
    {'order_id': '2005', 'customer': 'Hank', 'item': 'Desk Lamp', 'total': '45.00'},
]

print("Publishing order events...\n")

# Publish each order as a JSON string to the order-events channel
for order in orders:
    r.publish('order-events', json.dumps(order))
    print(f"Published order {order['order_id']}: {order['item']} for {order['customer']}")
    time.sleep(0.5)

print(f"\nDone. Published {len(orders)} orders.")
  • Save and exit nano by pressing Ctrl+X, then Y, then Enter.

What does this code do?

  • r.publish('order-events', json.dumps(order)) converts each order dictionary to a JSON string and sends it to every client currently subscribed to order-events.
  • time.sleep(0.5) adds a half-second pause between each publish so you can watch orders arrive one by one in the subscriber terminal.
  • The publish call returns the number of subscribers that received the message. If no subscriber is connected, it returns 0 and the message is gone forever.

✔️ Awesome, I've got everything!

Great. Make sure you have saved publisher.py before moving on.

ⓧ I'd like to double check the full code

import redis
import json
import time

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

orders = [
    {'order_id': '2001', 'customer': 'Diana', 'item': 'Monitor', 'total': '349.99'},
    {'order_id': '2002', 'customer': 'Eve', 'item': 'Mouse', 'total': '29.99'},
    {'order_id': '2003', 'customer': 'Frank', 'item': 'Webcam', 'total': '89.99'},
    {'order_id': '2004', 'customer': 'Grace', 'item': 'USB Hub', 'total': '24.99'},
    {'order_id': '2005', 'customer': 'Hank', 'item': 'Desk Lamp', 'total': '45.00'},
]

print("Publishing order events...\n")

for order in orders:
    r.publish('order-events', json.dumps(order))
    print(f"Published order {order['order_id']}: {order['item']} for {order['customer']}")
    time.sleep(0.5)

print(f"\nDone. Published {len(orders)} orders.")
Test real-time delivery across two terminals

Pub/Sub requires the subscriber to be running before the publisher sends messages. To see this in action, you need two SSH sessions: one for each script.

  • Open a new terminal window on your Mac. You now have two terminal windows: your original SSH session and this new one.
  • In the new terminal, SSH into your EC2 instance using the same command from Step 1:
ssh -i [[PEMPATH="path to your .pem file"]] ec2-user@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]

You now have two SSH terminals connected to the same EC2 instance. Think of them as two separate workers in a real system.

  • In Terminal 1 (your original SSH session), start the subscriber by running:
python3 subscriber.py

You should see Listening for order events on 'order-events' channel... and the terminal sits waiting. Leave it running.

Before you run the publisher, what do you think will happen in the subscriber terminal the moment you send the first order?

  • Switch to Terminal 2 (your new SSH session). Run the publisher by entering:
python3 publisher.py

Watch both terminals at the same time. In Terminal 2, the publisher prints each order as it sends it. In Terminal 1, the subscriber prints each order the instant it arrives.

What should I see?

In your subscriber terminal, you should see five notification lines appear one by one, with a half-second gap between each:

[NOTIFICATION] Order 2001: Diana bought Monitor for $349.99
[NOTIFICATION] Order 2002: Eve bought Mouse for $29.99
[NOTIFICATION] Order 2003: Frank bought Webcam for $89.99
[NOTIFICATION] Order 2004: Grace bought USB Hub for $24.99
[NOTIFICATION] Order 2005: Hank bought Desk Lamp for $45.00

Each order appeared in the subscriber terminal the moment the publisher sent it. No polling, no delay. That is Pub/Sub in action.

Subscriber not printing anything?

  • Make sure you started subscriber.py in Terminal 1 before running publisher.py in Terminal 2. Pub/Sub only delivers to subscribers that are connected at the time of publish.
  • Confirm both terminals are SSH'd into the same EC2 instance. If one terminal is on your local Mac, it cannot reach Redis on localhost.
  • Check that the channel name order-events is spelled identically in both subscriber.py and publisher.py. Even a single character difference means they are on different channels.
  • Help me debug why my Pub/Sub subscriber is not receiving messages

This real-time delivery is powerful. Orders appear in the subscriber terminal the instant they are published. But here is a question worth sitting with: what happens if the subscriber is not running when the publisher sends those five orders?

In the next step, you will find out the answer firsthand. Spoiler: it reveals a critical gap that changes how you think about messaging systems.

Discover the Pub/Sub Limitation

In the last step, your Pub/Sub system worked flawlessly. The subscriber printed each order the instant it was published. Real-time messaging, exactly as promised.

But real-world services crash and restart constantly. What happens to orders that arrive while your subscriber is offline? If Pub/Sub can't handle that scenario, it's not safe for order processing. In this step, you will find out firsthand.

In this step, get ready to:
  • Test what happens when messages are published with no subscriber listening.
  • Confirm that Pub/Sub messages are permanently lost when no subscriber is connected.
Stop the subscriber and publish while it's offline

Your subscriber.py is still running from the previous step, happily waiting for messages. Time to pull the plug and see what Pub/Sub does when nobody is listening.

  • Switch to the SSH terminal where subscriber.py is running.
  • Press Ctrl+C to stop the subscriber.

You should see the terminal return to the command prompt. The subscriber is now offline.

  • Switch to your second SSH terminal.
  • Run publisher.py again to publish five new orders by running this command:
python3 publisher.py

The publisher prints its usual five orders. But this time, no subscriber is connected to receive them.

  • Confirm that zero subscribers were listening by running this command:
redis-cli PUBSUB NUMSUB order-events

You should see order-events followed by 0. Zero subscribers were connected when those five orders were published.

What does PUBSUB NUMSUB do?

PUBSUB NUMSUB returns the number of subscribers currently connected to a given channel. A result of 0 proves that no one was listening when the publisher sent those messages.

Now for the critical question. Before you restart the subscriber, what do you think will happen? Will it pick up the five orders that were just published, or are they gone?

  • Switch back to your first SSH terminal.
  • Restart subscriber.py by running this command:
python3 subscriber.py

The subscriber prints "Listening for order events on 'order-events' channel..." and then sits there. Nothing else arrives. The five orders published while it was offline are permanently gone. There is no replay, no backlog, no recovery.

  • Press Ctrl+C to stop subscriber.py.

Subscriber actually received messages?

If the subscriber printed order notifications, it was still running from the previous step when you ran the publisher. Make sure you stopped subscriber.py with Ctrl+C first, then ran publisher.py, then restarted subscriber.py. The order matters.

Still stuck? Help me figure out why my subscriber received messages when it should have missed them.

Understand the at-most-once limitation

What you just experienced is the defining characteristic of Pub/Sub. This is not a bug. It is how Pub/Sub is designed to work.

What is at-most-once delivery?

Pub/Sub uses at-most-once delivery. Messages are delivered only to subscribers connected at the exact moment of publishing. Redis does not store Pub/Sub messages. It does not queue them. It does not retry.

If zero subscribers are connected, the message is discarded immediately and permanently. This is sometimes called fire-and-forget messaging.

For notifications and alerts, fire-and-forget is fine. A missed desktop notification is no big deal. But for order processing, losing five customer orders because a service restarted is not acceptable.

Why does this matter in interviews?

The difference between at-most-once and at-least-once delivery is one of the most common messaging architecture interview questions. You can now explain it with a concrete example you built yourself, not just a textbook definition.

You have now hit the ceiling of what Pub/Sub can do. You need a system where messages persist, where consumers can pick up where they left off, and where nothing gets lost. Next up, you will build exactly that with Redis Streams.

Build a Durable Order Pipeline with Redis Streams

In the last step, you watched five orders vanish into thin air. Pub/Sub delivered every message perfectly while the subscriber was online. The moment it went offline, every published order was permanently lost.

Redis Streams fix this problem. A Stream is an append-only log where every message persists until you explicitly remove it. Consumer groups track which messages each worker has read and acknowledged, enabling at-least-once delivery. In this step, you will build an order pipeline where no message is ever lost.

In this step, get ready to:
  • Create a stream producer that appends order events to a durable Redis Stream.
  • Create a stream consumer with consumer groups and message acknowledgment.
  • Test durability by stopping the consumer, publishing orders, and proving nothing is lost.
Create the stream producer

Instead of broadcasting messages that disappear, you will now append orders to a persistent stream. Each entry gets a unique, auto-generated ID and stays in the log indefinitely.

  • In your first SSH terminal, create stream_producer.py by running:
nano stream_producer.py
  • Paste the following code into the file:
import redis

# Connect to the local Redis server
r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Five sample orders to add to the stream
orders = [
    {'order_id': '3001', 'customer': 'Ivy', 'item': 'Tablet', 'total': '499.99'},
    {'order_id': '3002', 'customer': 'Jack', 'item': 'Charger', 'total': '19.99'},
    {'order_id': '3003', 'customer': 'Karen', 'item': 'Case', 'total': '34.99'},
    {'order_id': '3004', 'customer': 'Leo', 'item': 'Stand', 'total': '59.99'},
    {'order_id': '3005', 'customer': 'Mia', 'item': 'Cable', 'total': '12.99'},
]

print("Adding orders to 'orders' stream...\n")

# Append each order to the 'orders' stream and print the generated ID
for order in orders:
    entry_id = r.xadd('orders', order)
    print(f"Added order {order['order_id']} -> stream ID: {entry_id}")

print(f"\nDone. Stream length: {r.xlen('orders')}")

What does this code do?

  • r.xadd('orders', order) appends each order as a new entry in the orders stream. Unlike Pub/Sub's publish(), this data is stored permanently.
  • Each xadd call returns a unique stream ID (like 1691234567890-0) that includes a timestamp. This means entries are automatically ordered by when they were added.
  • r.xlen('orders') reports the total number of entries in the stream.
  • Save the file by pressing Ctrl+O, then Enter, then Ctrl+X to exit nano.
  • Run the producer by running:
python3 stream_producer.py

You should see five orders added, each with a unique stream ID, and a final stream length of 5.

  • Verify the data is actually persisted by inspecting the stream length:
redis-cli XLEN orders

You should see 5 in the output. The orders are sitting in the stream, waiting to be read.

  • Inspect the actual entries in the stream by running:
redis-cli XRANGE orders - +

You should see all five entries with their auto-generated IDs and the order fields (order_id, customer, item, total). This data is durable. It will not disappear when a consumer disconnects.

Seeing zero entries or an error?

  • Make sure you saved stream_producer.py before running it. Unsaved changes in nano are not written to disk.
  • Check that your Redis connection uses host='localhost' and port=6379. Run redis-cli ping to confirm Redis is still running.
  • Still stuck? Help me debug why my stream_producer.py is not adding entries to the Redis stream.
Create the stream consumer with consumer groups

A consumer group is a coordination mechanism. It tracks which messages have been delivered to which workers and which ones have been acknowledged. This is what makes at-least-once delivery possible.

  • Create stream_consumer.py by running:
nano stream_consumer.py
  • Paste the following code to set up the connection and create the consumer group:
import redis

# Connect to the local Redis server
r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Configuration for the consumer group
group_name = 'order-processors'
consumer_name = 'worker-1'
stream_name = 'orders'

# Create the consumer group (start reading from the beginning of the stream)
try:
    r.xgroup_create(stream_name, group_name, id='0', mkstream=True)
    print(f"Created consumer group '{group_name}'")
except redis.exceptions.ResponseError as e:
    # BUSYGROUP means the group already exists, which is fine
    if 'BUSYGROUP' in str(e):
        print(f"Consumer group '{group_name}' already exists")
    else:
        raise

print(f"Consumer '{consumer_name}' listening on '{stream_name}'...")
print("Press Ctrl+C to stop.\n")

What does this code do?

  • r.xgroup_create(stream_name, group_name, id='0', mkstream=True) creates a consumer group called order-processors on the orders stream. The id='0' tells the group to start reading from the very beginning of the stream.
  • mkstream=True creates the stream automatically if it does not exist yet.
  • The try/except block handles the case where the group already exists. Redis raises a BUSYGROUP error if you try to create a group that is already there.
  • Now add the main processing loop directly below the code you just pasted:
# Main processing loop
try:
    while True:
        # Read one new message at a time, wait up to 5 seconds
        messages = r.xreadgroup(
            group_name, consumer_name,
            {stream_name: '>'},
            count=1,
            block=5000
        )

        if messages:
            for stream, entries in messages:
                for entry_id, fields in entries:
                    print(f"[PROCESSING] Order {fields['order_id']}: "
                          f"{fields['customer']} - {fields['item']} "
                          f"${fields['total']}")

                    # Acknowledge the message so it's removed from pending
                    r.xack(stream_name, group_name, entry_id)
                    print(f"  -> Acknowledged {entry_id}")
        else:
            print("  (waiting for new orders...)")
except KeyboardInterrupt:
    print("\nConsumer stopped.")

What does this loop do?

  • r.xreadgroup() reads messages from the stream on behalf of worker-1 in the order-processors group. The special ID '>' means "give me only messages that have not been delivered to this group yet."
  • block=5000 makes the call wait up to 5 seconds for a new message before returning empty. This avoids busy-polling.
  • r.xack() acknowledges each message after processing. This tells Redis "this message has been handled" and removes it from the pending entries list. Without acknowledgment, Redis considers the message unprocessed.
  • Save the file by pressing Ctrl+O, then Enter, then Ctrl+X to exit nano.
  • Run the consumer by running:
python3 stream_consumer.py

The consumer creates the group and immediately processes all five orders from the stream. You should see each order printed with [PROCESSING] followed by an acknowledgment line for each one.

Consumer not processing any orders?

  • Make sure you ran stream_producer.py first. The consumer needs entries in the stream to read.
  • Verify the stream has data by running redis-cli XLEN orders in your second terminal. If it shows 0, run the producer again.
  • Check that both code chunks are in stream_consumer.py. The group creation block and the main loop must both be present.
  • Still stuck? Help me debug why my stream consumer is not reading messages.

✔️ Awesome, I've got everything!

Great. Make sure you have saved both stream_producer.py and stream_consumer.py.

ⓧ I'd like to double check the full code

Here is the complete stream_producer.py:

import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

orders = [
    {'order_id': '3001', 'customer': 'Ivy', 'item': 'Tablet', 'total': '499.99'},
    {'order_id': '3002', 'customer': 'Jack', 'item': 'Charger', 'total': '19.99'},
    {'order_id': '3003', 'customer': 'Karen', 'item': 'Case', 'total': '34.99'},
    {'order_id': '3004', 'customer': 'Leo', 'item': 'Stand', 'total': '59.99'},
    {'order_id': '3005', 'customer': 'Mia', 'item': 'Cable', 'total': '12.99'},
]

print("Adding orders to 'orders' stream...\n")

for order in orders:
    entry_id = r.xadd('orders', order)
    print(f"Added order {order['order_id']} -> stream ID: {entry_id}")

print(f"\nDone. Stream length: {r.xlen('orders')}")

For stream_consumer.py, scroll up and confirm you have both chunks in order:

  • Chunk 1: the import, connection, consumer group creation, and the two print statements.
  • Chunk 2: the main processing loop with xreadgroup, the print statements, xack, and the KeyboardInterrupt handler.
Test the durability difference

Now for the real test. You are going to repeat the exact same experiment from Step 4: stop the consumer, publish orders while it is offline, then bring it back. With Pub/Sub, those orders were gone forever.

  • Stop stream_consumer.py by pressing Ctrl+C in the terminal where it is running.

The consumer is now offline. Any orders added to the stream from this point have no active worker reading them.

  • Run the producer again to add five more orders while the consumer is down:
python3 stream_producer.py

You should see five more orders added to the stream. The stream length should now be 10 (the original five plus five new ones).

Before you restart the consumer, think about what you expect to happen. When you ran this same test with Pub/Sub in Step 4, the subscriber printed nothing because the messages were gone. Do you think the stream consumer will find these five orders, or will they be lost?

  • Restart the consumer to find out:
python3 stream_consumer.py

The consumer immediately processes all five orders that were added while it was offline. No messages lost. This is the fundamental difference between fire-and-forget and durable messaging.

  • Stop the consumer again with Ctrl+C.
  • Inspect the consumer group state by running:
redis-cli XINFO GROUPS orders

You should see the order-processors group listed with its consumer count and the number of pending messages (which should be 0 since every message was acknowledged).

Why did Streams keep the messages but Pub/Sub lost them?

Pub/Sub is a broadcast system. It pushes messages to whoever is connected right now. If nobody is listening, the message is dropped. There is no log, no history, no replay.

Streams store every message in a persistent, append-only log. Consumer groups track a pointer to the last delivered message. When a consumer reconnects, it picks up right where it left off. This is at-least-once delivery: every message reaches a consumer at least one time.

Consumer not picking up the missed orders?

  • If the consumer prints "(waiting for new orders...)" instead of processing, the group may have already delivered those messages in a previous run. Delete the group with redis-cli XGROUP DESTROY orders order-processors, then run the producer and consumer again.
  • Make sure you stopped the consumer before running the producer. If the consumer was still running, it would have processed the orders immediately.
  • Need help? Help me figure out why my stream consumer is not picking up orders added while it was offline.

You now have two complete messaging systems side by side. Pub/Sub is fast and simple, perfect for real-time notifications where missing a message is acceptable. Streams give you durability, consumer groups, and acknowledgment for workloads where every message must be processed. Next up, the Secret Mission challenges you to handle what happens when a consumer crashes mid-processing.

Secret mission

Recover from a Consumer Crash with XPENDING and XCLAIM

Your Streams pipeline is durable, but what happens when a consumer reads a message and then crashes before acknowledging it? That message is stuck in limbo. In this secret mission, you will simulate a crash and use Redis's built-in recovery tools to rescue orphaned orders.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Your EC2 instance is covered by the AWS free tier, but running it 24/7 counts against your monthly free hours.

Cost warning

Your t2.micro instance uses free tier hours while it is running. If your 12-month free tier period has ended, or if your account's free tier does not cover t2.micro, you will be charged standard EC2 rates. The safest option is to terminate the instance when you are done.

Resources you used:

  • 1 EC2 t2.micro instance (contains Redis 6, all Python scripts, and all Redis data).
  • 1 security group allowing SSH access on port 22.
  • 1 EC2 key pair and the corresponding .pem file on your Mac.

Keep everything running

No action needed. Choose this if you are still actively building or want to keep experimenting with Redis messaging patterns.

  • Leave the instance running. Monitor your free tier usage in the AWS Billing dashboard to avoid surprise charges.
  • Your .pem file stays on your Mac for future SSH access.

Pause - I'll come back to this later

Stop the instance to pause compute charges. Your EBS volume and data persist so you can start the instance again later.

  • Close both SSH terminal sessions on your Mac.
  • Return to the EC2 Console in your browser.
  • In the navigation pane, click Instances.
  • Select your instance.
  • Click Instance state, then Stop instance.
  • Click Stop to confirm.

What about storage charges?

A stopped instance incurs no compute charges. The EBS root volume continues to use a small amount of storage, which stays within the free tier for most accounts.

When you are ready to resume, select the instance and click Instance state then Start instance. You will need to SSH in again with a new public IP address.

Delete - I don't want to use this again

Remove all project resources to ensure no charges. This is permanent and cannot be undone.

  • Close both SSH terminal sessions on your Mac.
  • Return to the EC2 Console in your browser.
  • In the navigation pane, click Instances.
  • Select your instance.
  • Click Instance state, then Terminate (delete) instance.
  • Click Terminate (delete) to confirm. The instance will show a terminated state and disappear after about an hour.

Next, delete the security group that was created for this project.

  • In the EC2 Console navigation pane, expand Network & Security and click Security Groups.
  • Select the security group you created for this project (the one allowing SSH on port 22).
  • Click Actions, then Delete security groups.
  • Confirm the deletion.

Now delete the key pair from AWS and your local machine.

  • In the EC2 Console navigation pane, click Key Pairs (under Network & Security).
  • Select the key pair you created for this project.
  • Click Actions, then Delete.
  • Confirm the deletion.
  • Delete the .pem file from your Mac by running this command (replace the filename with your key pair name):
rm ~/Downloads/[[KEYPAIR_FILE="your-key-pair-name.pem"]]

All project resources have been removed.

Nice Work!

Nice Work!

Nice work! You just built a real-time order processing pipeline on AWS that proves when messages survive a subscriber outage and when they vanish. You can now explain and demo the difference between Pub/Sub and Redis Streams to an interviewer, backed by a working system you built yourself.

You've learned how to:

  • Launch an EC2 instance, install Redis from scratch on Amazon Linux 2023, and store order data using Redis Hashes.
  • Build a real-time Pub/Sub notification system and experience its fire-and-forget limitation firsthand by watching messages vanish when the subscriber goes offline.
  • Build a durable order pipeline with Redis Streams, consumer groups, and message acknowledgment, proving that messages survive a consumer outage by running the same offline test that wiped out Pub/Sub messages.
  • Secret Mission: Recover orphaned messages from a crashed consumer using XPENDING and XAUTOCLAIM to simulate fault-tolerant message processing.

Ready to quiz yourself?