Build a Dead Letter Queue on AWS SQS

Build a message pipeline that automatically quarantines poison messages.

Introduction

30 Second Summary

Every online order you place travels through a chain of systems. Sometimes data arrives corrupted, malformed, or just plain broken. In production, these "poison" messages can jam up an entire pipeline if nothing catches them.

In this project, you will build a message processing pipeline on AWS SQS that automatically quarantines broken messages into a Dead Letter Queue. Then you will build a tool that inspects the quarantined messages and replays them.

What You'll Build

You will send order messages into a queue, watch a consumer chew through them, then deliberately break the pipeline with poison messages and see SQS quarantine them automatically.

By the end of this project, you'll have:

  • A live message pipeline where you send orders into a queue and watch a consumer validate and process each one in your terminal.
  • An automatic poison message trap that quarantines broken orders into a Dead Letter Queue after 3 failed processing attempts.
  • A DLQ inspector and replay tool that lets you peek at quarantined messages, understand why they failed, and send them back for reprocessing.
  • Secret Mission: Add error classification that distinguishes transient failures from permanent ones, and only replays messages that have a chance of succeeding.

Are there any prerequisites?

You need an AWS account, Python installed on your machine, and the AWS CLI installed (you will configure it in Step 1). A code editor like VS Code is also needed.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for your portfolio. Putting words to it now makes everything that follows click faster.

Set Up AWS Credentials and Install boto3

Your Python scripts need a way to authenticate with AWS before they can create queues, send messages, or do anything in SQS. That connection depends on two things: configured AWS credentials and the boto3 library.

In this step, you will verify that both pieces are in place and then write the script that creates your first SQS queue.

In this step, get ready to:
  • Verify your AWS CLI credentials are working.
  • Confirm boto3 is installed and importable.
  • Write the queue creation script in create_queue.py.
Verify your AWS credentials

Your AWS CLI is already configured with your Access Key, Secret Key, region, and output format. Before writing any code, confirm those credentials actually work.

  • Test your credentials by running this command:
aws sts get-caller-identity

You should see a JSON response containing your UserId, Account, and Arn. This proves your terminal can authenticate with AWS.

Seeing 'Unable to locate credentials'?

This means aws configure has not run successfully yet. Run aws configure and enter your Access Key ID, Secret Access Key, region us-east-1, and output format json.

If you do not have an Access Key, create one in the AWS Console under IAM > Users > your user > Security credentials > Create access key.

help me fix my AWS CLI credentials

Verify boto3 is installed

The boto3 library is the official AWS SDK for Python. Every script in this project uses it to call SQS APIs.

Before you run the verification command, what do you expect to see if boto3 is properly installed?

  • Verify boto3 works by running this command:
python -c "import boto3; print(boto3.__version__)"

You should see a version number printed without errors.

Seeing 'ModuleNotFoundError: No module named boto3'?

This means boto3 is not installed in the Python environment your terminal is using. Install it by running:

pip install boto3

Then re-run the verification command above.

help me install boto3

What is boto3?

boto3 is the official AWS SDK for Python. It translates your Python function calls into HTTP requests to AWS services like SQS, S3, and Lambda.

Every time you call something like sqs.send_message() in this project, boto3 handles authentication, request signing, and response parsing behind the scenes.

Write the queue creation script

With your credentials verified and boto3 ready, you can write the script that creates your SQS queue. This queue is where your producer will send order messages later.

  • Switch to create_queue.py in your editor and add the following code:
import boto3


def create_queue():
    # Create an SQS client to interact with the queue service
    sqs = boto3.client("sqs")

    # Create a standard queue with custom settings
    response = sqs.create_queue(
        QueueName="order-processing-queue",
        Attributes={
            "VisibilityTimeout": "30",
            "MessageRetentionPeriod": "345600",
        },
    )

    # Extract and display the queue URL
    queue_url = response["QueueUrl"]
    print(f"Queue created successfully!")
    print(f"Queue URL: {queue_url}")
    return queue_url


if __name__ == "__main__":
    create_queue()

What does this code do?

  • boto3.client("sqs") creates a client connected to the SQS service using the credentials you just verified.
  • create_queue() sends an API request to AWS asking it to create a new standard queue named order-processing-queue.
  • VisibilityTimeout controls how long a message stays hidden from other consumers after one consumer picks it up (30 seconds).
  • MessageRetentionPeriod is how long SQS keeps undelivered messages before deleting them (345600 seconds = 4 days).
  • Save create_queue.py.

Not sure the code is right?

Check that your Attributes dictionary keys are strings and the values are also strings (not integers). SQS expects all attribute values as string types.

help me check my create_queue.py code

✔️ Awesome, I've got everything!

Great. Double check you have saved create_queue.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete create_queue.py file:

import boto3


def create_queue():
    sqs = boto3.client("sqs")

    response = sqs.create_queue(
        QueueName="order-processing-queue",
        Attributes={
            "VisibilityTimeout": "30",
            "MessageRetentionPeriod": "345600",
        },
    )

    queue_url = response["QueueUrl"]
    print(f"Queue created successfully!")
    print(f"Queue URL: {queue_url}")
    return queue_url


if __name__ == "__main__":
    create_queue()

Your environment is locked in. AWS credentials are working, boto3 is ready, and your first script is written. Next up, you will run this script to create the queue and start sending messages through it.

Create Your Queue and Send Messages

Your AWS credentials are configured and boto3 is installed. Before tackling failure scenarios, you need a working pipeline first.

A message queue pipeline has two sides: a producer that sends messages and a consumer that reads and processes them. In this step, you will create an SQS queue, send order messages into it, and build a consumer that processes them.

In this step, get ready to:
  • Create an SQS queue and confirm it exists in AWS.
  • Send order messages (good and deliberately broken) into the queue.
  • Build a consumer that validates, processes, and deletes messages.
Create the SQS queue

Your create_queue.py script is ready to go. It creates a standard SQS queue with a 30-second visibility timeout and 4-day message retention.

  • Run the script to create your queue:
python create_queue.py

Before you run this, what do you expect the output to look like?

You should see two lines: a success message and a queue URL in the format https://sqs.us-east-1.amazonaws.com/ACCOUNT_ID/order-processing-queue.

Seeing an error?

If you see InvalidClientTokenId or SignatureDoesNotMatch, your AWS credentials are not configured correctly. Run aws configure again and double-check your Access Key ID and Secret Access Key.

If you see a region error, confirm your default region is set to us-east-1 by running aws configure get region.

Help me troubleshoot my SQS queue creation error.

Create the producer

A producer sends messages into the queue. Yours will send real order data alongside deliberately broken messages so you can test how your system handles bad data.

  • Create a new file called producer.py in your sqs-dead-letter-queue folder and paste this first section:
import boto3
import json


def get_queue_url():
    # Look up the queue URL by name so we don't hardcode it
    sqs = boto3.client("sqs")
    response = sqs.get_queue_url(QueueName="order-processing-queue")
    return response["QueueUrl"]


def send_messages():
    sqs = boto3.client("sqs")
    queue_url = get_queue_url()

    # Five well-formed order messages
    good_orders = [
        {"order_id": "1001", "product": "Wireless Mouse", "quantity": 2, "price": 29.99},
        {"order_id": "1002", "product": "USB-C Cable", "quantity": 5, "price": 12.50},
        {"order_id": "1003", "product": "Mechanical Keyboard", "quantity": 1, "price": 89.99},
        {"order_id": "1004", "product": "Monitor Stand", "quantity": 1, "price": 45.00},
        {"order_id": "1005", "product": "Webcam HD", "quantity": 3, "price": 59.99},
    ]

What does this code do?

  • get_queue_url() looks up your queue by name so you never need to hardcode the URL. SQS assigns the URL when the queue is created.
  • good_orders is a list of 5 order dictionaries, each with the fields a real order would have: order_id, product, quantity, and price.
  • Below the good_orders list, add the poison messages and the sending logic:
    # Deliberately broken messages to test error handling
    poison_messages = [
        "this is not valid json at all {{{{",
        json.dumps({"order_id": "9999"}),
    ]

    sent_count = 0

    # Send each good order as JSON
    for order in good_orders:
        sqs.send_message(
            QueueUrl=queue_url,
            MessageBody=json.dumps(order),
        )
        sent_count += 1

    # Send poison messages as-is
    for poison in poison_messages:
        sqs.send_message(
            QueueUrl=queue_url,
            MessageBody=poison,
        )
        sent_count += 1

    print(f"Sent {sent_count} messages ({len(good_orders)} good, {len(poison_messages)} poison)")


if __name__ == "__main__":
    send_messages()

What does this code do?

  • poison_messages contains two deliberately broken messages. One is garbled text that cannot be parsed as JSON. The other is valid JSON but is missing required fields.
  • The two for loops send each message to SQS using send_message(). Good orders are serialized as JSON. Poison messages are sent raw.
  • The final print confirms how many of each type were sent.
  • Save producer.py and run it:
python producer.py

You should see: Sent 7 messages (5 good, 2 poison).

Seeing 'QueueDoesNotExist'?

This means get_queue_url() could not find a queue named order-processing-queue. Make sure you ran create_queue.py successfully in the previous substep and that you are using the same AWS region.

Help me figure out why my SQS queue cannot be found.

Create the consumer

The consumer reads messages from the queue, validates them, and deletes the ones it successfully processes. Messages that fail validation stay in the queue for redelivery.

  • Create a new file called consumer.py in your sqs-dead-letter-queue folder and paste the helper functions:
import boto3
import json
import time


def get_queue_url():
    # Look up the queue URL by name
    sqs = boto3.client("sqs")
    response = sqs.get_queue_url(QueueName="order-processing-queue")
    return response["QueueUrl"]


def validate_order(message_body):
    # Check that the message has all required order fields
    required_fields = ["order_id", "product", "quantity", "price"]

    order = json.loads(message_body)

    missing_fields = [field for field in required_fields if field not in order]
    if missing_fields:
        raise ValueError(f"Missing required fields: {missing_fields}")

    if order["quantity"] <= 0:
        raise ValueError(f"Invalid quantity: {order['quantity']}")

    return order

What does this code do?

  • validate_order() is the gatekeeper. It tries to parse the message body as JSON, then checks that all four required fields exist and that the quantity is valid.
  • If anything is wrong, it raises an exception. This means the consumer will catch the error and leave the message in the queue instead of deleting it.
  • Below validate_order, add the main processing function that polls SQS:
def process_messages():
    sqs = boto3.client("sqs")
    queue_url = get_queue_url()

    print("Polling for messages...\n")

    # Receive up to 10 messages, wait up to 5 seconds
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=10,
        WaitTimeSeconds=5,
        AttributeNames=["ApproximateReceiveCount"],
    )

    messages = response.get("Messages", [])

    if not messages:
        print("No messages in queue.")
        return

    print(f"Received {len(messages)} messages:\n")

What does this code do?

  • receive_message() pulls messages from the queue. WaitTimeSeconds=5 enables long polling, which means the call waits up to 5 seconds for messages to arrive rather than returning immediately.
  • AttributeNames=["ApproximateReceiveCount"] asks SQS to include how many times each message has been received. This becomes important when you add the Dead Letter Queue later.
  • Below the print(f"Received...") line, add the message processing loop and the entry point:
    for message in messages:
        body = message["Body"]
        receipt_handle = message["ReceiptHandle"]
        receive_count = message.get("Attributes", {}).get("ApproximateReceiveCount", "?")

        try:
            order = validate_order(body)
            print(f"  [OK] Processed order #{order['order_id']}: "
                  f"{order['quantity']}x {order['product']} @ ${order['price']}")

            # Delete successfully processed messages from the queue
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=receipt_handle,
            )

        except (json.JSONDecodeError, ValueError) as e:
            # Failed messages are NOT deleted - SQS will redeliver them
            print(f"  [FAIL] Could not process message (receive #{receive_count}): {e}")
            print(f"         Body: {body[:80]}...")
            print(f"         Message NOT deleted - will be redelivered.\n")


if __name__ == "__main__":
    process_messages()

What does this code do?

  • For each message, the consumer attempts to validate it using validate_order(). If validation succeeds, it prints a confirmation and calls delete_message() to permanently remove it from the queue.
  • If validation fails (bad JSON or missing fields), the except block catches the error and prints a failure message. The message is NOT deleted, so SQS will redeliver it on the next poll.
  • Save consumer.py.

Before you run the consumer, what do you think will happen to the 5 good messages versus the 2 poison messages?

  • Run the consumer:
python consumer.py

You should see 5 lines starting with [OK] Processed order #1001 through #1005, and 2 lines starting with [FAIL] for the poison messages.

Why do poison messages say 'NOT deleted'?

When a message fails validation, the consumer leaves it in the queue on purpose. SQS will make it available again after the visibility timeout expires. This is the correct pattern because you might fix the consumer logic and want to retry.

The problem is that without a Dead Letter Queue, those poison messages will keep coming back forever. You will see this problem in action in the next step.

Only seeing some messages?

SQS does not guarantee all messages arrive in a single receive_message call. If you see fewer than 7 messages, run python consumer.py again. The remaining messages will appear on the next poll.

If you see No messages in queue, make sure you ran python producer.py before running the consumer.

Help me understand why I'm not seeing all my messages.

✔️ Awesome, I've got everything!

Great. Double-check you have saved both producer.py and consumer.py in your sqs-dead-letter-queue folder.

ⓧ I'd like to double check the full code

Scroll up and confirm these code chunks are in producer.py, in order:

  • Imports (boto3 and json) and the get_queue_url() helper function.
  • The send_messages() function with good_orders, poison_messages, both sending loops, and the print statement.
  • The if __name__ == "__main__" block calling send_messages().

Scroll up and confirm these code chunks are in consumer.py, in order:

  • Imports (boto3, json, time), get_queue_url(), and validate_order().
  • The process_messages() function with receive_message call, early return for empty queue, and the for message in messages loop.
  • The try/except block that validates, prints, and either deletes or skips each message.
  • The if __name__ == "__main__" block calling process_messages().

Your pipeline is working. Good messages flow through and get deleted. Poison messages fail and stay in the queue. But what happens if you run the consumer again? Those poison messages come right back. In the next step, you will see this problem in action.

Break It With Poison Messages

In Step 2, you proved your pipeline works end to end. Messages flow from producer to consumer, orders get processed, and the queue empties out cleanly.

But that was a best-case scenario. Real production queues receive garbage all the time: malformed JSON, missing fields, corrupted payloads. When your consumer cannot process a message, it leaves it in the queue. SQS then redelivers that message over and over, forever.

In this step, get ready to:
  • Send a mix of valid orders and poison messages into the queue.
  • Watch the consumer validate and reject bad messages.
  • Observe poison messages looping indefinitely in the queue.
Send the mixed message batch

Your producer.py sends 5 valid orders alongside 2 deliberately broken messages. One is invalid JSON that cannot be parsed at all. The other is valid JSON but missing required fields.

  • Send a fresh batch of messages to the queue by running:
python producer.py

You should see this output:

Sent 7 messages (5 good, 2 poison)

What are the 2 poison messages?

The first poison message is the string this is not valid json at all {{{{. It will crash any JSON parser.

The second is {"order_id": "9999"}. It is valid JSON, but it is missing the product, quantity, and price fields your consumer requires.

Seeing a QueueDoesNotExist error?

This means your queue from Step 2 was deleted or created in a different region. Run python create_queue.py again to recreate it, then retry the producer.

Help me fix this

Run the consumer and observe failures

Your consumer validates every message before processing it. Messages that pass validation get deleted from the queue. Messages that fail validation are left untouched.

  • Process the messages in the queue by running:
python consumer.py

You should see 5 orders processed successfully and 2 failures:

Polling for messages...

Received 7 messages:

  [OK] Processed order #1001: 2x Wireless Mouse @ $29.99
  [OK] Processed order #1002: 5x USB-C Cable @ $12.5
  [OK] Processed order #1003: 1x Mechanical Keyboard @ $89.99
  [OK] Processed order #1004: 1x Monitor Stand @ $45.0
  [OK] Processed order #1005: 3x Webcam HD @ $59.99
  [FAIL] Could not process message (receive #1): ...
  [FAIL] Could not process message (receive #1): ...

Why are failed messages left in the queue?

Look at the except block in consumer.py. When validation fails, the code prints an error but never calls delete_message. SQS only removes a message when the consumer explicitly deletes it.

A message that is received but not deleted becomes invisible for the queue's VisibilityTimeout period (30 seconds in your queue). After that, it reappears and SQS delivers it again.

Observe the poison message loop

The 5 good orders are gone. They were processed and deleted. But the 2 poison messages were never deleted, so SQS will keep delivering them.

Before you run the consumer again, what do you think you will see? Will the queue be empty, or will something come back?

  • Wait about 30 seconds for the visibility timeout to expire, then run the consumer again:
python consumer.py

The same 2 poison messages are back. No good orders remain. Only the failures reappear:

Polling for messages...

Received 2 messages:

  [FAIL] Could not process message (receive #2): ...
  [FAIL] Could not process message (receive #2): ...
         Message NOT deleted - will be redelivered.
  • Wait another 30 seconds, then run the consumer a third time:
python consumer.py

The same 2 messages are back again. Notice the receive count has incremented. These messages will loop forever because nothing can process them and nothing removes them.

Polling for messages...

Received 2 messages:

  [FAIL] Could not process message (receive #3): ...
  [FAIL] Could not process message (receive #3): ...
         Message NOT deleted - will be redelivered.

Why is this a problem?

These 2 messages will circle through your queue indefinitely. Every time your consumer polls, it wastes compute retrying messages that will never succeed.

In a production system with thousands of messages per minute, a handful of poison messages create noise that drowns out legitimate traffic. You lose visibility into what is actually failing, and your consumer spends cycles on hopeless retries instead of real work.

This is the exact problem a Dead Letter Queue solves. After a set number of failed delivery attempts, SQS automatically moves the poison message to a separate queue where you can inspect it without blocking your main pipeline.

Consumer shows no messages?

If you ran the consumer too quickly (within 30 seconds of the previous run), the poison messages are still invisible. Wait the full 30 seconds and try again.

If the queue is genuinely empty, run python producer.py again to send a fresh batch, then repeat the consumer runs.

Help me debug this

Your pipeline is stuck. Two unprocessable messages will loop through the queue forever, wasting resources and blocking visibility. In the next step, you will add a Dead Letter Queue that automatically quarantines these messages after 3 failed attempts.

Add the Dead Letter Queue

You just experienced the poison message problem firsthand. Two broken messages loop through your queue forever, blocking visibility and wasting compute every time you run the consumer.

A Dead Letter Queue solves this by automatically quarantining messages that fail processing after a set number of attempts. In this step, you will create a DLQ, attach it to your main queue, and watch SQS move the poison messages there on its own.

In this step, get ready to:
  • Create a Dead Letter Queue and attach it to your main queue with a redrive policy.
  • Test the full pipeline and watch SQS quarantine poison messages automatically after 3 failed attempts.
  • Write a DLQ inspector tool to view the quarantined messages.
Create the DLQ and attach a redrive policy

A redrive policy tells SQS how many times a message can be received before it gets moved to the DLQ. You will set maxReceiveCount to 3, meaning after 3 failed processing attempts, SQS quarantines the message automatically.

  • Create a new file called setup_dlq.py in your sqs-dead-letter-queue folder.
  • Add the imports and the first half of the setup function, which creates the DLQ and retrieves its ARN:
import boto3
import json


def setup_dead_letter_queue():
    sqs = boto3.client("sqs")

    print("Creating Dead Letter Queue...")
    dlq_response = sqs.create_queue(
        QueueName="order-processing-dlq",
        Attributes={
            "MessageRetentionPeriod": "1209600",
        },
    )
    dlq_url = dlq_response["QueueUrl"]
    print(f"DLQ created: {dlq_url}")

    dlq_attributes = sqs.get_queue_attributes(
        QueueUrl=dlq_url,
        AttributeNames=["QueueArn"],
    )
    dlq_arn = dlq_attributes["Attributes"]["QueueArn"]
    print(f"DLQ ARN: {dlq_arn}")

What does this code do?

  • create_queue creates a new SQS queue named order-processing-dlq with a 14-day retention period (1209600 seconds). This is longer than the main queue's 4-day retention, so you have more time to inspect failed messages.
  • get_queue_attributes retrieves the DLQ's ARN (Amazon Resource Name). You need the ARN to link the DLQ to your main queue in the next chunk.
  • Now add the second half of the function directly below, which attaches the redrive policy to your main queue:
    main_queue_url = sqs.get_queue_url(QueueName="order-processing-queue")["QueueUrl"]

    redrive_policy = {
        "deadLetterTargetArn": dlq_arn,
        "maxReceiveCount": "3",
    }

    sqs.set_queue_attributes(
        QueueUrl=main_queue_url,
        Attributes={
            "RedrivePolicy": json.dumps(redrive_policy),
            "VisibilityTimeout": "5",
        },
    )

    print(f"\nRedrive policy attached to main queue:")
    print(f"  maxReceiveCount: 3")
    print(f"  deadLetterTargetArn: {dlq_arn}")
    print(f"  VisibilityTimeout: 5 seconds (short for demo)")
    print(f"\nMessages that fail 3 times will automatically move to the DLQ.")


if __name__ == "__main__":
    setup_dead_letter_queue()

What does this code do?

  • redrive_policy is a JSON object that tells SQS where to send failed messages (deadLetterTargetArn) and how many receive attempts to allow before moving them (maxReceiveCount = 3).
  • set_queue_attributes updates your main queue with this redrive policy. It also sets VisibilityTimeout to 5 seconds so you can demo quickly without waiting 30 seconds between consumer runs.
  • Save setup_dlq.py.
  • Run the setup script by running this command:
python setup_dlq.py

You should see output confirming the DLQ was created, its ARN, and that the redrive policy was attached with maxReceiveCount: 3.

Seeing an error about the queue not existing?

  • Make sure order-processing-queue still exists from Step 2. If you deleted it, re-run python create_queue.py first.
  • Check that your AWS credentials are still configured by running aws sts get-caller-identity.

Help me troubleshoot my setup_dlq.py error

✔️ Awesome, I've got everything!

Great. Double check you have saved setup_dlq.py and the output confirmed the redrive policy is attached.

ⓧ I'd like to double check the full code

Your setup_dlq.py should have these sections in order:

  • The import boto3 and import json lines at the top.
  • The setup_dead_letter_queue() function that creates the DLQ, gets its ARN, builds the redrive policy, and attaches it to the main queue.
  • The if __name__ == "__main__": block at the bottom that calls setup_dead_letter_queue().

Scroll up and confirm each section matches the two code blocks you pasted.

Test the full pipeline

Now you will see the DLQ in action. You will send messages (including poison), then run the consumer multiple times to trigger the automatic quarantine.

  • Send a fresh batch of messages by running:
python producer.py

You should see Sent 7 messages (5 good, 2 poison) printed.

  • Run the consumer for the first time:
python consumer.py

You should see all 5 good orders processed and deleted, plus the 2 poison messages failing. The poison messages remain in the queue because your consumer does not delete them.

  • Wait 10 seconds, then run the consumer again:
python consumer.py

The 2 poison messages appear again. This is receive attempt #2.

  • Wait 10 seconds and run the consumer a third time:
python consumer.py

The poison messages appear once more. This is receive attempt #3, which hits the maxReceiveCount threshold. SQS now marks these messages for quarantine.

Why wait 10 seconds between runs?

You set VisibilityTimeout to 5 seconds in your setup script. After a message is received but not deleted, SQS hides it for 5 seconds before making it visible again. Waiting 10 seconds ensures the messages are visible for your next consumer run.

  • Before you run the consumer a fourth time, predict what you will see. Do you think the poison messages will appear again, or will SQS have moved them?
  • Wait 10 seconds, then run the consumer one more time:
python consumer.py

You should see No messages in queue. printed. The poison messages are gone from the main queue because SQS moved them to the DLQ after 3 failed receives.

Still seeing poison messages on the 4th run?

  • Make sure you waited at least 10 seconds between each consumer run. If you ran too quickly, the visibility timeout had not expired and the receive count did not increment.
  • Confirm your setup_dlq.py ran successfully earlier. Re-run it if you are unsure.

Help me troubleshoot my DLQ redrive policy

Inspect the DLQ

The poison messages did not disappear. They are sitting in your Dead Letter Queue, preserved for inspection. Now you will write a tool that reads them out and displays why they failed.

  • Create a new file called dlq_inspector.py in your sqs-dead-letter-queue folder.
  • Add the imports and a helper function that looks up any queue URL by name:
import boto3
import json
import sys


def get_queue_url(queue_name):
    sqs = boto3.client("sqs")
    response = sqs.get_queue_url(QueueName=queue_name)
    return response["QueueUrl"]
  • Below that, add the inspect_dlq function which reads messages from the DLQ and displays their details:
def inspect_dlq():
    sqs = boto3.client("sqs")
    dlq_url = get_queue_url("order-processing-dlq")

    print("Inspecting Dead Letter Queue...\n")

    response = sqs.receive_message(
        QueueUrl=dlq_url,
        MaxNumberOfMessages=10,
        WaitTimeSeconds=5,
        AttributeNames=["All"],
    )

    messages = response.get("Messages", [])

    if not messages:
        print("DLQ is empty. No failed messages.")
        return messages

    print(f"Found {len(messages)} dead-lettered messages:\n")

    for i, message in enumerate(messages, 1):
        body = message["Body"]
        attributes = message.get("Attributes", {})
        sent_timestamp = attributes.get("SentTimestamp", "unknown")
        receive_count = attributes.get("ApproximateReceiveCount", "unknown")

        print(f"  Message {i}:")
        print(f"    Body: {body[:100]}")
        print(f"    Original receive count: {receive_count}")
        print(f"    Message ID: {message['MessageId']}")
        print(f"")

    return messages

What does this function do?

  • receive_message with AttributeNames=["All"] fetches every available metadata field for each message, including how many times it was received before being quarantined.
  • The function prints each message's body (truncated to 100 chars), its receive count, and its unique Message ID. This tells you exactly what failed and how many times SQS tried before giving up.
  • Below the inspect_dlq function, add a replay_messages function that moves messages from the DLQ back to the main queue:
def replay_messages():
    sqs = boto3.client("sqs")
    dlq_url = get_queue_url("order-processing-dlq")
    main_queue_url = get_queue_url("order-processing-queue")

    print("Replaying messages from DLQ to main queue...\n")

    response = sqs.receive_message(
        QueueUrl=dlq_url,
        MaxNumberOfMessages=10,
        WaitTimeSeconds=5,
    )

    messages = response.get("Messages", [])

    if not messages:
        print("DLQ is empty. Nothing to replay.")
        return

    replayed = 0
    for message in messages:
        sqs.send_message(
            QueueUrl=main_queue_url,
            MessageBody=message["Body"],
        )

        sqs.delete_message(
            QueueUrl=dlq_url,
            ReceiptHandle=message["ReceiptHandle"],
        )

        replayed += 1
        print(f"  Replayed message: {message['Body'][:60]}...")

    print(f"\nReplayed {replayed} messages back to main queue.")

What does replay do?

Replay reads each message from the DLQ, sends a copy to the main queue, then deletes the original from the DLQ. This lets you reprocess messages after fixing the root cause of the failure. You will use this feature in the next step.

  • Finally, add the entry point at the bottom of the file that selects inspect or replay mode based on a command-line flag:


if __name__ == "__main__":
    if "--replay" in sys.argv:
        replay_messages()
    else:
        inspect_dlq()
  • Save dlq_inspector.py.
  • Run the inspector to see the quarantined messages:
python dlq_inspector.py

You should see Found 2 dead-lettered messages followed by each message's body, receive count, and Message ID. One message is the invalid JSON string, and the other is the order missing required fields.

DLQ showing empty?

  • SQS messages have their own visibility timeout. If you ran the inspector too quickly after another read, wait 30 seconds and try again.
  • Confirm you ran the consumer 4 times with gaps in the previous substep. If not, go back and complete those runs first.

Help me figure out why the DLQ is empty

✔️ Awesome, I've got everything!

Great. Double check you have saved dlq_inspector.py and confirmed 2 quarantined messages appeared in the output.

ⓧ I'd like to double check the full code

Your dlq_inspector.py should have these sections in order:

  • Imports (boto3, json, sys) and the get_queue_url helper function.
  • The inspect_dlq() function that receives and displays DLQ messages.
  • The replay_messages() function that moves messages back to the main queue.
  • The if __name__ == "__main__": block that checks for --replay in sys.argv.

Scroll up and confirm each section matches the four code blocks you pasted.

Your pipeline now handles failure automatically. Poison messages get quarantined after 3 attempts, your main queue stays clean, and you can inspect exactly what went wrong. Next up, you will replay those dead-lettered messages back to the main queue for reprocessing.

Replay Messages from the DLQ

Your DLQ has quarantined 2 poison messages safely out of the main pipeline. But dead-lettered messages are not dead.

In production, you fix the root cause (a code bug, a bad data format) and then move messages back to the main queue for reprocessing. This completes the full DLQ lifecycle: produce, fail, quarantine, inspect, replay.

In this step, get ready to:
  • Replay quarantined messages from the DLQ back to the main queue.
  • Confirm the replayed messages reappear in the consumer.
  • Write a cleanup script to delete all AWS resources.
Run the replay

Your dlq_inspector.py already includes a replay_messages() function that activates when you pass the --replay flag. The function reads each message from the DLQ, sends it to the main queue, and deletes it from the DLQ.

How does replay work?

The replay process has three steps for each message:

  • receive_message pulls the message from the DLQ.
  • send_message copies it into the main queue as a fresh message.
  • delete_message removes it from the DLQ so it is not replayed again.

The order matters. If you deleted first and the send failed, you would lose the message entirely.

Before you run this, what do you think will happen to the 2 poison messages sitting in the DLQ?

  • Run the inspector in replay mode:
python dlq_inspector.py --replay

You should see output confirming each message was replayed:

Replaying messages from DLQ to main queue...

  Replayed message: this is not valid json at all {{{{...
  Replayed message: {"order_id": "9999"}...

Replayed 2 messages back to main queue.

Seeing "DLQ is empty"?

SQS messages have a visibility timeout. If you ran the inspector in inspect mode recently, those messages may still be invisible. Wait 5 seconds and try again.

If the DLQ is truly empty, you may need to re-run the full flow: run python producer.py, then run python consumer.py four times with 10-second gaps to move poison messages back into the DLQ.

Help me figure out why my DLQ is empty when I try to replay

Confirm replayed messages with the consumer

The 2 poison messages are now back in the main queue. What do you think will happen when the consumer tries to process them again?

  • Run the consumer to pick up the replayed messages:
python consumer.py

You should see the 2 poison messages appear and fail again:

Polling for messages...

Received 2 messages:

  [FAIL] Could not process message (receive #1): ...
  [FAIL] Could not process message (receive #1): ...

Why do the replayed messages still fail?

Replay does not fix the data. It moves messages back so they can be reprocessed after you fix the root cause. Since we did not fix the poison data (invalid JSON, missing fields), they fail again.

In a real system, you would fix the bug in your consumer or correct the upstream data before replaying. The replay mechanism itself is working perfectly.

This is the full DLQ lifecycle in action: produce, fail, quarantine, inspect, replay. You have demonstrated every stage.

Write the cleanup script

Your project needs a way to tear down all AWS resources when you are done. A cleanup script makes this repeatable and safe to include in your GitHub repo.

  • Create a new file called cleanup.py in your sqs-dead-letter-queue folder and add the following code:
import boto3


def cleanup():
    sqs = boto3.client("sqs")

    # Both queues to remove
    queues_to_delete = ["order-processing-queue", "order-processing-dlq"]

    for queue_name in queues_to_delete:
        try:
            response = sqs.get_queue_url(QueueName=queue_name)
            queue_url = response["QueueUrl"]
            sqs.delete_queue(QueueUrl=queue_url)
            print(f"Deleted: {queue_name}")
        except sqs.exceptions.QueueDoesNotExist:
            print(f"Already gone: {queue_name}")

    print("\nAll queues cleaned up. No AWS resources remain.")


if __name__ == "__main__":
    cleanup()

What does this code do?

  • queues_to_delete lists both queues the project created.
  • The loop looks up each queue URL, then calls delete_queue to remove it from SQS.
  • The try/except handles the case where a queue was already deleted, so you can safely run this script multiple times.
  • Save cleanup.py.
  • Confirm the file exists by running:
ls

You should see cleanup.py listed alongside your other project files: create_queue.py, producer.py, consumer.py, setup_dlq.py, and dlq_inspector.py.

Do not run cleanup yet

You will use this script in the Before You Go section when you are ready to tear down your resources. For now, keep the queues alive in case you want to try the Secret Mission.

✔️ Awesome, I've got everything!

Great. Make sure you have saved cleanup.py and you are ready to move on.

ⓧ I'd like to double check the full code

Here is the complete cleanup.py for reference:

import boto3


def cleanup():
    sqs = boto3.client("sqs")

    queues_to_delete = ["order-processing-queue", "order-processing-dlq"]

    for queue_name in queues_to_delete:
        try:
            response = sqs.get_queue_url(QueueName=queue_name)
            queue_url = response["QueueUrl"]
            sqs.delete_queue(QueueUrl=queue_url)
            print(f"Deleted: {queue_name}")
        except sqs.exceptions.QueueDoesNotExist:
            print(f"Already gone: {queue_name}")

    print("\nAll queues cleaned up. No AWS resources remain.")


if __name__ == "__main__":
    cleanup()

You now have a complete message processing pipeline with automatic failure handling. Your project demonstrates the full DLQ lifecycle that production systems rely on every day. Next up, you can take on the Secret Mission to add intelligent error classification to your replay logic.

Secret mission

Error Classification and Selective Replay

Your current replay sends everything back to the main queue, even messages that will never succeed. Can you build a smart replay that distinguishes transient failures from permanent ones, and only retries messages that have a chance of working?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses the AWS SQS free tier only, so there are no ongoing charges as long as you stay under 1 million requests per month.

Resources you used:

  • order-processing-queue (SQS standard queue in us-east-1).
  • order-processing-dlq (SQS standard queue in us-east-1).
  • sqs-dead-letter-queue local project folder.

Keep everything running

No action needed. Choose this if you are still actively building or plan to push this project to GitHub as a portfolio piece.

  • Both SQS queues cost nothing within the free tier (1 million requests per month, always free).
  • Messages in the main queue expire after 4 days by default. Messages in the DLQ expire after 14 days.
  • Your local project folder and scripts remain on your machine ready for further development.

Pause - I'll come back to this later

Shut down any running processes to free up memory. Your queues and files stay intact for when you return.

  • Close any terminal sessions running your producer or consumer scripts.
  • Your SQS queues persist in AWS at no cost. Messages will expire on their own based on the retention period.
  • Your local files remain in the sqs-dead-letter-queue folder on your Desktop.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes both SQS queues from AWS and removes the local project files.

  • Delete both SQS queues by running this command from your project folder:
python cleanup.py

You should see:

Deleted: order-processing-queue
Deleted: order-processing-dlq

All queues cleaned up. No AWS resources remain.
  • Delete the local project folder. On macOS/Linux, run:
cd ~/Desktop
rm -rf sqs-dead-letter-queue

On Windows, run:

cd ~\Desktop
rmdir /s /q sqs-dead-letter-queue
  • Optionally uninstall boto3 if you no longer need it for other projects:
pip uninstall boto3

Nice Work!

Nice Work!

Nice work! You just built a production-grade message processing pipeline on AWS SQS, complete with automatic poison message quarantining and a replay tool.

You've learned how to:

  • Build a producer/consumer pipeline on AWS SQS that sends, validates, and processes order messages in real time.
  • Solve the poison message problem with a Dead Letter Queue and redrive policy that automatically quarantines messages after 3 failed attempts.
  • Inspect quarantined messages and replay them back to the main queue, completing the full DLQ lifecycle: produce, fail, quarantine, inspect, replay.
  • Secret Mission: Classified errors as transient vs permanent and built selective replay logic that only retries messages with a chance of succeeding.

Ready to quiz yourself?