Event-Driven Order Processing on AWS

Build a serverless order pipeline with DynamoDB, SQS, and Lambda.

Introduction

30 Second Summary

Every time you place an online order, a chain reaction kicks off behind the scenes. Your order flows from one system to another, getting validated, processed, and tracked without any single piece needing to know about the rest.

In this project, you will build an event-driven order processing pipeline on AWS where new orders automatically flow from a database through a message queue to a processor. When processing fails, the pipeline catches the failure and sends a real email alert to your inbox.

What You'll Build

You will place a test order, watch it cascade through six connected AWS services in real time, then deliberately crash the processor with a "bad" order and receive an email notification within minutes proving your failure-alerting pipeline works end-to-end.

By the end of this project, you'll have:

  • A fully decoupled pipeline where orders written to DynamoDB automatically arrive in an SQS queue within seconds, with no direct connection between the writer and the consumer.
  • A designed failure scenario where a high-value "poison pill" order crashes the processor, retries exhaust, and the message routes itself to a Dead-Letter Queue.
  • A real-time alerting system that delivers an email to your inbox the moment a failed order lands in the DLQ, proving you have full observability over processing failures.
  • Secret Mission: expose the pipeline via an API Gateway REST endpoint so you can submit orders over HTTP instead of the CLI.

Are there any prerequisites?

You need an AWS account with the AWS CLI already configured on your machine. Basic Python familiarity is helpful but not required.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

You're going to wire together six AWS services into an event-driven pipeline that processes orders automatically and alerts you the moment something goes wrong.

Create the DynamoDB Table and SQS Queues

Before any events can flow through your pipeline, you need two foundational pieces: a data store that captures orders, and message queues that buffer them for background processing.

In this step, you will create a DynamoDB table with DynamoDB Streams enabled, plus two SQS queues: a processing queue and a Dead-Letter Queue that catches failed messages.

In this step, get ready to:
  • Create the Orders DynamoDB table with Streams enabled.
  • Create two SQS queues with a redrive policy linking them together.
  • Verify the table stream and queue redrive policy are correctly configured.
Create the Orders DynamoDB table

Every order that enters your system starts as an item in DynamoDB. Enabling Streams on the table means every new item automatically generates a change event that downstream services can consume.

  • Create the Orders table with Streams enabled by running this command in your terminal:
aws dynamodb create-table \
  --table-name Orders \
  --attribute-definitions AttributeName=orderId,AttributeType=S \
  --key-schema AttributeName=orderId,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES

What does this command do?

  • Creates a table named Orders with a single partition key orderId of type String.
  • PAY_PER_REQUEST uses on-demand billing so you only pay for what you use and stay within the free tier.
  • NEW_AND_OLD_IMAGES means the stream event includes the full item data before and after the change, so downstream consumers can see the complete order.

You should see a JSON response containing TableDescription with TableStatus set to CREATING or ACTIVE.

  • Confirm the table is active and Streams are enabled by running:
aws dynamodb describe-table --table-name Orders

Look for StreamSpecification in the output. You should see StreamEnabled set to true and StreamViewType set to NEW_AND_OLD_IMAGES.

Table status still showing CREATING?

DynamoDB tables usually become active within seconds. Wait a few moments and run the describe-table command again.

If you see an error about the table already existing, it means you ran create-table twice. That is fine. Run describe-table to confirm the configuration is correct.

help me troubleshoot my DynamoDB create-table command

Create the SQS queues with a redrive policy

You need two queues: a processing queue where orders arrive for handling, and a Dead-Letter Queue where messages land after repeated processing failures. The DLQ must exist first because the processing queue references it in its redrive policy.

  • Create the Dead-Letter Queue by running:
aws sqs create-queue --queue-name OrdersDLQ

You should see a JSON response with a QueueUrl value. Copy this URL. You will need it in the next command.

  • Get the DLQ's ARN by running the following command, replacing the URL with the one you just received:
aws sqs get-queue-attributes --queue-url [[DLQ_URL="your OrdersDLQ QueueUrl from above"]] --attribute-names QueueArn

The output shows an Attributes object containing a QueueArn value. Copy this ARN. It follows the pattern arn:aws:sqs:REGION:ACCOUNT_ID:OrdersDLQ.

  • Create an attributes file for the processing queue. Create a new file called create-queue.json with the following content, replacing the ARN placeholder with your DLQ ARN:
{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"[[DLQ_ARN="your OrdersDLQ ARN from above"]]\",\"maxReceiveCount\":\"3\"}"
}

What is a redrive policy?

A redrive policy tells SQS what to do with messages that keep failing. The deadLetterTargetArn points to the DLQ where failed messages go.

The maxReceiveCount of 3 means SQS lets a consumer try processing a message 3 times. If all 3 attempts fail, the message moves to the DLQ automatically.

  • Create the processing queue with the redrive policy by running:
aws sqs create-queue --queue-name OrdersProcessingQueue --attributes file://create-queue.json

You should see a JSON response with a QueueUrl for your new processing queue.

Seeing an error about invalid attribute?

Make sure create-queue.json is saved in the same directory where you are running the command. The file:// prefix tells the CLI to look for the file relative to your current working directory.

Check that the ARN in your JSON file is correct and properly escaped. The backslashes before the quotes are required because the RedrivePolicy value is a JSON string inside a JSON file.

help me fix my SQS create-queue redrive policy error

Verify the redrive policy is attached

Both queues exist, but you need to confirm they are properly linked. The processing queue should know to send failed messages to the DLQ.

  • Before you run this next command, what do you expect to see in the RedrivePolicy attribute?
  • Check the processing queue's attributes by running the following command, replacing the URL with your OrdersProcessingQueue URL:
aws sqs get-queue-attributes --queue-url [[PROCESSING_QUEUE_URL="your OrdersProcessingQueue QueueUrl"]] --attribute-names RedrivePolicy

You should see RedrivePolicy in the output showing your DLQ ARN and a maxReceiveCount of 3. This confirms messages will move to the DLQ after 3 failed processing attempts.

RedrivePolicy is empty or missing?

If the attribute comes back empty, the queue was created without the redrive policy. Delete the queue and recreate it with the correct attributes file.

Run aws sqs delete-queue --queue-url YOUR_PROCESSING_QUEUE_URL then re-run the create-queue command above.

help me verify my SQS redrive policy configuration

Your data layer is ready. You have a DynamoDB table that streams every change event, a processing queue to buffer orders for background work, and a DLQ that catches failures. Next up, you will build a Lambda function that writes orders into this table.

Build and Test the Order Ingestion Lambda

Your DynamoDB table and SQS queues are in place. Now you need a way to actually create orders in the system.

In this step, you will build a Lambda function that accepts order data, validates it, and writes it to DynamoDB. Testing it immediately gives you the early visible win of seeing orders land in the table.

In this step, get ready to:
  • Create an IAM role and Lambda function for order ingestion.
  • Test with sample orders and verify they appear in DynamoDB.
  • Validate that incomplete orders are rejected with a 400 error.
Create the IAM role and Lambda function

Every Lambda function needs an IAM role that defines what AWS services it can access. Your ingestion function needs permission to write items to DynamoDB and send logs to CloudWatch.

  • Create a file called trust-policy.json with the following content. This tells AWS that Lambda is allowed to assume this role:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

What is a trust policy?

A trust policy defines which AWS service is allowed to "assume" (use) this role. By specifying lambda.amazonaws.com as the principal, you are saying only Lambda functions can use this role's permissions.

  • Create the IAM role by running this command:
aws iam create-role \
  --role-name OrderIngestionLambdaRole \
  --assume-role-policy-document file://trust-policy.json

You should see JSON output that includes the role's ARN. It looks like arn:aws:iam::123456789012:role/OrderIngestionLambdaRole. Copy this ARN and save it somewhere. You will need it shortly.

  • Now create a file called permissions-policy.json with the inline permissions your Lambda needs:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "dynamodb:PutItem",
      "Resource": "arn:aws:dynamodb:*:*:table/Orders"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

What does this policy allow?

The first statement grants dynamodb:PutItem on the Orders table. This is the minimum permission needed to write new orders.

The second statement grants CloudWatch Logs permissions so you can see the function's output and debug issues.

  • Attach the permissions policy to the role by running:
aws iam put-role-policy \
  --role-name OrderIngestionLambdaRole \
  --policy-name OrderIngestionPolicy \
  --policy-document file://permissions-policy.json

This command produces no output on success. That silence means it worked.

Seeing an access denied error?

Make sure your AWS CLI user has iam:CreateRole and iam:PutRolePolicy permissions. If you are using an admin account, this should not be an issue.

Also check that your trust-policy.json and permissions-policy.json files are in your current working directory.

Help me troubleshoot this IAM error

Now you will write the function code and deploy it to AWS Lambda.

  • Create a file called lambda_function.py with the handler, event parsing, and validation logic:
import json
import boto3
import os

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Orders')


def lambda_handler(event, context):
    # Handle both direct invocation and API Gateway proxy events
    if 'body' in event:
        body = json.loads(event['body']) if isinstance(event['body'], str) else event['body']
    else:
        body = event

    # Validate required fields
    required_fields = ['orderId', 'customerId', 'amount', 'item']
    missing = [f for f in required_fields if f not in body]
    if missing:
        error_msg = f"Missing required fields: {', '.join(missing)}"
        if 'body' in event:
            return {
                'statusCode': 400,
                'headers': {'Content-Type': 'application/json'},
                'body': json.dumps({'error': error_msg})
            }
        return {'statusCode': 400, 'body': error_msg}

What does this code do?

The function connects to the Orders DynamoDB table using boto3, the AWS SDK that comes pre-installed in every Python Lambda runtime.

It checks whether the event has a body key (which means it came from API Gateway) or is a direct invocation. Either way, it validates that four required fields are present before proceeding.

  • Add the DynamoDB write and response logic below the validation code in the same file:
    # Write order to DynamoDB
    order_item = {
        'orderId': body['orderId'],
        'customerId': body['customerId'],
        'amount': int(body['amount']),
        'item': body['item'],
        'status': 'RECEIVED'
    }

    table.put_item(Item=order_item)

    response_body = {
        'message': f"Order {body['orderId']} received successfully",
        'order': order_item
    }

    if 'body' in event:
        return {
            'statusCode': 200,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps(response_body, default=str)
        }

    return {'statusCode': 200, 'body': response_body}

What does this section do?

It builds an order item with a RECEIVED status and writes it to DynamoDB using table.put_item(). The response format adapts based on whether the call came from API Gateway or a direct invocation.

✔️ Awesome, I've got everything!

Great. Make sure your lambda_function.py is saved with both sections combined into one file.

ⓧ I'd like to double check the full code

Your lambda_function.py should have two main sections. Scroll up and confirm you have:

  • The imports, boto3 setup, handler signature, event parsing, and field validation (first code block).
  • The DynamoDB write and response logic (second code block).

Both sections go in the same lambda_function.py file. The second section continues inside the lambda_handler function body.

  • Zip the function code for deployment by running:
zip function.zip lambda_function.py
  • Confirm the zip file was created by running:
ls function.zip

You should see function.zip in the output.

  • Deploy the Lambda function by running this command. Replace the --role value with the ARN you copied earlier:
aws lambda create-function \
  --function-name OrderIngestionFunction \
  --zip-file fileb://function.zip \
  --handler lambda_function.lambda_handler \
  --runtime python3.13 \
  --role [[ROLE_ARN="arn:aws:iam::YOUR_ACCOUNT_ID:role/OrderIngestionLambdaRole"]]

You should see JSON output confirming the function was created, including its ARN and the python3.13 runtime.

Why python3.13?

Python 3.13 is the latest Lambda runtime running on Amazon Linux 2023. It includes the boto3 SDK pre-installed, so you can use AWS services without adding any dependencies to your zip file.

Seeing a role cannot be assumed error?

IAM roles take a few seconds to propagate. Wait 10 seconds and try the create-function command again. If the error persists, double-check that your trust policy has lambda.amazonaws.com as the service principal.

Help me fix this Lambda role error

Test the Lambda with sample orders

Your function is deployed. Time to see if it actually works by sending it some test orders.

  • Create a test payload file called test-order.json:
{
  "orderId": "ORD-001",
  "customerId": "CUST-1",
  "amount": 150,
  "item": "Wireless Keyboard"
}
  • Invoke the Lambda function with this test payload by running:
aws lambda invoke \
  --function-name OrderIngestionFunction \
  --cli-binary-format raw-in-base64-out \
  --payload file://test-order.json \
  output.json
  • Check the response by running:
cat output.json

You should see a response containing "statusCode": 200 and a message confirming Order ORD-001 received successfully.

  • Verify the order landed in DynamoDB by running:
aws dynamodb scan --table-name Orders

You should see your order item in the Items array with orderId set to ORD-001 and status set to RECEIVED.

  • Submit a second order. Create test-order-2.json:
{
  "orderId": "ORD-002",
  "customerId": "CUST-2",
  "amount": 75,
  "item": "USB-C Cable"
}
  • Invoke with this second order:
aws lambda invoke \
  --function-name OrderIngestionFunction \
  --cli-binary-format raw-in-base64-out \
  --payload file://test-order-2.json \
  output.json
  • Now create a high-value order with an amount over 500. This will matter in a later step. Create test-order-3.json:
{
  "orderId": "ORD-003",
  "customerId": "CUST-3",
  "amount": 750,
  "item": "Mechanical Keyboard"
}
  • Invoke with this high-value order:
aws lambda invoke \
  --function-name OrderIngestionFunction \
  --cli-binary-format raw-in-base64-out \
  --payload file://test-order-3.json \
  output.json

Before you scan the table, how many items do you expect to see?

  • Scan the table to confirm all three orders are present:
aws dynamodb scan --table-name Orders

You should see three items in the response. All three have status set to RECEIVED.

Only seeing one or two items?

Check the output.json from each invocation. If any returned a statusCode of 400, you likely have a typo in one of your test payload files. Verify each file has all four required fields: orderId, customerId, amount, and item.

Help me debug my missing orders

Test validation with an incomplete payload

You know the happy path works. Now confirm that the validation logic rejects bad data before it ever reaches DynamoDB.

  • Create a file called test-invalid.json with a missing amount field:
{
  "orderId": "ORD-BAD",
  "customerId": "CUST-1",
  "item": "Mystery Item"
}

Before you invoke, what do you think will happen? Will this order make it into DynamoDB, or will the function reject it?

  • Invoke the function with the invalid payload:
aws lambda invoke \
  --function-name OrderIngestionFunction \
  --cli-binary-format raw-in-base64-out \
  --payload file://test-invalid.json \
  output.json
  • Check the response:
cat output.json

You should see "statusCode": 400 and an error message that says Missing required fields: amount. The ingestion layer rejected bad data before it entered the system.

Why validate at the ingestion layer?

Catching invalid data here prevents malformed records from flowing through DynamoDB Streams into your downstream processing pipeline. Every service downstream can trust that orders in the table have all required fields.

Your ingestion function is live and writing orders to DynamoDB. Next up, you will wire DynamoDB Streams to SQS so that every new order automatically triggers background processing.

Connect DynamoDB Streams to SQS via EventBridge Pipes

Your ingestion Lambda is writing orders to DynamoDB, and DynamoDB Streams is capturing every change. But nothing downstream knows about those new orders.

You need automatic event flow so every new order triggers background processing without the ingestion Lambda needing to know about the processing queue. EventBridge Pipes connects sources to targets with built-in filtering, so you can route only INSERT events (new orders) to your SQS queue.

In this step, get ready to:
  • Create an IAM role that lets EventBridge Pipes read from the DynamoDB stream and send to SQS.
  • Create an EventBridge Pipe with event filtering in the AWS Console.
  • Test the decoupled event flow end-to-end.
Create the IAM role for EventBridge Pipes

EventBridge Pipes needs an execution role that grants it permission to read from the DynamoDB stream and send messages to the SQS queue. You will create this role using the CLI before building the pipe in the Console.

  • Retrieve the Stream ARN from your Orders table by running this command:
aws dynamodb describe-table --table-name Orders --query "Table.LatestStreamArn" --output text
  • Copy the output value. It looks like arn:aws:dynamodb:REGION:ACCOUNT_ID:table/Orders/stream/TIMESTAMP. You will need it shortly.
  • Retrieve the ARN of your processing queue by running:
aws sqs get-queue-attributes --queue-url $(aws sqs get-queue-url --queue-name OrdersProcessingQueue --query "QueueUrl" --output text) --attribute-names QueueArn --query "Attributes.QueueArn" --output text
  • Copy this ARN too. It looks like arn:aws:sqs:REGION:ACCOUNT_ID:OrdersProcessingQueue.

Why gather these ARNs first?

IAM policies should be scoped to the exact resources the role needs access to. By specifying the stream ARN and queue ARN in your policy, you follow the principle of least privilege.

  • Create the trust policy that allows EventBridge Pipes to assume the role by running:
aws iam create-role --role-name EventBridgePipesRole --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "pipes.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}'

What does this command do?

This creates an IAM role named EventBridgePipesRole with a trust policy that allows the pipes.amazonaws.com service principal to assume it. EventBridge Pipes uses this role when reading from your stream and writing to your queue.

  • Attach an inline policy granting the DynamoDB stream and SQS permissions by running:
aws iam put-role-policy --role-name EventBridgePipesRole --policy-name PipesPolicy --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:ListStreams"
      ],
      "Resource": "[[STREAM_ARN="your Orders table stream ARN from above"]]"
    },
    {
      "Effect": "Allow",
      "Action": "sqs:SendMessage",
      "Resource": "[[QUEUE_ARN="your OrdersProcessingQueue ARN from above"]]"
    }
  ]
}'

What permissions does this grant?

  • dynamodb:DescribeStream, dynamodb:GetRecords, dynamodb:GetShardIterator, and dynamodb:ListStreams let the pipe read change events from the DynamoDB stream.
  • sqs:SendMessage lets the pipe forward matching events to your processing queue.
  • Confirm the role was created by running:
aws iam get-role --role-name EventBridgePipesRole --query "Role.Arn" --output text

You should see the role ARN printed. Copy it for the next substep.

Role creation failed?

  • If you see an error about the role already existing, a previous attempt may have partially succeeded. Delete it with aws iam delete-role-policy --role-name EventBridgePipesRole --policy-name PipesPolicy followed by aws iam delete-role --role-name EventBridgePipesRole, then retry.
  • Make sure you replaced the placeholder ARN values with the actual ARNs you copied earlier.

Help me troubleshoot my EventBridgePipesRole creation

Create the EventBridge Pipe in the AWS Console

With the IAM role ready, you can now create the pipe that connects your DynamoDB stream to the SQS processing queue. You will use the AWS Console since it provides a guided workflow for configuring the source, filter, and target.

Now configure the source.

  • Under Source, select DynamoDB.
  • For DynamoDB stream, select the stream for your Orders table.
  • For Starting position, leave it set to Latest. This means the pipe only processes new events from this point forward.

Next, add a filter so only new orders flow through.

  • In the Filtering section, enter this event pattern:
{"eventName": ["INSERT"]}

What does this filter do?

DynamoDB Streams emits events for inserts, updates, and deletes. This filter tells the pipe to only pass INSERT events to the target. Updates and deletes are silently ignored.

This keeps your processing queue focused on new orders only. No update noise, no delete noise.

  • Click Next to proceed past the filtering step.
  • Click Next again to skip Enrichment (you do not need to transform events for this project).

Now configure the target.

  • For Target service, select Amazon SQS queue.
  • For Queue, select OrdersProcessingQueue.

Finally, set the execution role.

  • Under Permissions, choose Use existing role.
  • For Role name, select EventBridgePipesRole.
  • Click Create pipe.

The pipe detail page appears. Wait for the status to update to Running. This may take 10-20 seconds.

Pipe stuck in Creating or Failed?

  • If the pipe fails, check that your EventBridgePipesRole has the correct stream ARN in its policy. A mismatched ARN is the most common cause.
  • Make sure you selected Use existing role and chose the correct role name.

Help me debug my EventBridge Pipe

Test the decoupled event flow

The pipe is running. Time to prove that inserting a new order into DynamoDB automatically delivers a message to the SQS processing queue with no direct coupling between the two.

  • Create a new test order by invoking the ingestion Lambda:
aws lambda invoke --function-name OrderIngestionFunction --cli-binary-format raw-in-base64-out --payload '{"orderId": "ORD-050", "customerId": "CUST-3", "amount": 150, "item": "Keyboard"}' /dev/stdout
  • You should see a 200 response confirming the order was received.
  • Wait 10-15 seconds for the pipe to pick up the DynamoDB stream event and deliver it to SQS.

Before you check the queue, what do you expect to find there? Think about what the DynamoDB stream event looks like compared to the raw order JSON you submitted.

  • Poll the processing queue to see if the event arrived by running:
aws sqs receive-message --queue-url $(aws sqs get-queue-url --queue-name OrdersProcessingQueue --query "QueueUrl" --output text) --max-number-of-messages 1

You should see a message in the response. The message body contains the DynamoDB stream event, including an eventName of INSERT and a dynamodb field with the NewImage of your order.

What is in the message body?

The pipe forwards the raw DynamoDB stream record, not your original order JSON. The order data lives inside dynamodb.NewImage using DynamoDB's type descriptor format (e.g. {"S": "ORD-050"} for strings, {"N": "150"} for numbers).

Your processor Lambda in the next step will need to parse this format to extract the order values.

No messages in the queue?

  • Wait a full 30 seconds and try polling again. The pipe may take a moment to start processing after creation.
  • Check the pipe status in the EventBridge console. If it shows Failed, the IAM role permissions are likely misconfigured.
  • Make sure you created a NEW order (not one that already existed). The pipe is set to Latest starting position, so it only processes events that occur after the pipe was created.

Help me troubleshoot my empty SQS queue

You now have a fully decoupled event flow. Writing to DynamoDB triggers the stream, the pipe filters for new orders and routes them to SQS. The ingestion Lambda has no knowledge of the processing queue. Next up, you will build the processor Lambda that consumes these messages and handles them.

Build the Order Processor Lambda

Messages are flowing into the OrdersProcessingQueue, but nothing is consuming them. You need a processor that picks up orders from the queue and handles them.

In this step, you will build a processor Lambda that automatically reads from SQS and processes each order. The processor will intentionally crash on high-value orders (over $500) to simulate a real failure scenario. You will see what happens when failures go undetected.

In this step, get ready to:
  • Create an IAM role and Lambda function that processes SQS messages.
  • Connect the processing queue to the Lambda with an event source mapping.
  • Submit a high-value order and watch it fail silently into the Dead-Letter Queue.
Create the processor Lambda's IAM role

The processor Lambda needs permissions to read messages from SQS and write logs to CloudWatch. You will create a dedicated IAM role for it.

  • Create a trust policy file for the Lambda service by running this command:
cat > trust-policy-processor.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
  • Create the IAM role using the trust policy by running:
aws iam create-role \
  --role-name OrderProcessorLambdaRole \
  --assume-role-policy-document file://trust-policy-processor.json

You should see JSON output containing the role's ARN. Copy the ARN value from the output. You will need it shortly.

  • Create the permissions policy file that grants SQS and CloudWatch Logs access by running:
cat > processor-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:*:*:OrdersProcessingQueue"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}
EOF
  • Attach the policy to the role by running:
aws iam put-role-policy \
  --role-name OrderProcessorLambdaRole \
  --policy-name OrderProcessorPolicy \
  --policy-document file://processor-policy.json

What do these permissions allow?

  • sqs:ReceiveMessage lets the Lambda read messages from the queue.
  • sqs:DeleteMessage lets Lambda remove successfully processed messages so they don't get reprocessed.
  • sqs:GetQueueAttributes lets Lambda check queue metadata during polling.
  • The logs: permissions enable CloudWatch Logs for debugging.

Seeing an access denied error?

Double check that you used OrderProcessorLambdaRole as the role name (not the ingestion role name from earlier). Also confirm the JSON file has no trailing commas or typos.

help me debug an IAM role creation error for my Lambda processor

Write and deploy the processor Lambda function

Now you will write the processor code. This Lambda will parse SQS messages, extract order data from the DynamoDB stream event payload, and intentionally crash on orders over $500.

  • Create the processor Lambda code file by running:
cat > lambda_function_processor.py << 'EOF'
import json


def lambda_handler(event, context):
    for record in event['Records']:
        body = json.loads(record['body'])

        # Extract order data from DynamoDB stream event
        if 'dynamodb' in body:
            new_image = body['dynamodb'].get('NewImage', {})
            order_id = new_image.get('orderId', {}).get('S', 'UNKNOWN')
            amount_str = new_image.get('amount', {}).get('N', '0')
            amount = int(amount_str)
            customer_id = new_image.get('customerId', {}).get('S', 'UNKNOWN')
            item_name = new_image.get('item', {}).get('S', 'UNKNOWN')
        else:
            order_id = body.get('orderId', 'UNKNOWN')
            amount = int(body.get('amount', 0))
            customer_id = body.get('customerId', 'UNKNOWN')
            item_name = body.get('item', 'UNKNOWN')

        print(f"Processing order {order_id}: {item_name} for customer {customer_id}, amount ${amount}")

        # Simulate failure for high-value orders
        if amount > 500:
            error_msg = f"PROCESSING FAILED: Order {order_id} requires manual approval (amount ${amount} exceeds $500 limit)"
            print(error_msg)
            raise Exception(error_msg)

        print(f"Order {order_id} processed successfully")

    return {'statusCode': 200, 'body': 'Batch processed successfully'}
EOF

What does this code do?

The processor receives SQS messages that contain DynamoDB stream events. Each message wraps order data in a nested structure with type annotations (S for strings, N for numbers).

  • It loops through each SQS record, extracts order fields from the DynamoDB stream format.
  • For orders with amount > 500, it raises an exception to simulate a processing failure.
  • When Lambda raises an exception, it does NOT delete the message from SQS. SQS will retry delivery up to maxReceiveCount (3) times before sending it to the DLQ.
  • Zip the function code by running:
zip function-processor.zip lambda_function_processor.py
  • Deploy the Lambda function by running this command (replace the role ARN with the one you copied earlier):
aws lambda create-function \
  --function-name OrderProcessorFunction \
  --zip-file fileb://function-processor.zip \
  --handler lambda_function_processor.lambda_handler \
  --runtime python3.13 \
  --role arn:aws:iam::[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:role/OrderProcessorLambdaRole

You should see JSON output confirming the function was created, including its ARN and State: Active.

Seeing an InvalidParameterValueException?

If you see an error about the role, wait 10 seconds and try again. IAM roles can take a few seconds to propagate before Lambda can use them.

help me fix a Lambda create-function error for my order processor

Connect SQS to the processor with an event source mapping

The processor Lambda exists, but it is not yet connected to the queue. An event source mapping tells Lambda to poll SQS automatically and invoke your function with each message.

  • Get the ARN of your processing queue by running:
aws sqs get-queue-attributes \
  --queue-url https://sqs.[[AWS_REGION="your AWS region, e.g. us-east-1"]].amazonaws.com/[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]/OrdersProcessingQueue \
  --attribute-names QueueArn

Copy the QueueArn value from the output.

  • Create the event source mapping by running:
aws lambda create-event-source-mapping \
  --function-name OrderProcessorFunction \
  --batch-size 1 \
  --event-source-arn arn:aws:sqs:[[AWS_REGION="your AWS region, e.g. us-east-1"]]:[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:OrdersProcessingQueue

You should see JSON output with "State": "Creating" or "State": "Enabled". Lambda will now poll the queue and invoke your function for each message.

Why batch size 1?

Setting --batch-size 1 means Lambda passes exactly one message per invocation. This makes it easy to see which specific order fails. In production, a larger batch size improves throughput.

  • Verify the event source mapping is active by waiting 10-15 seconds, then submit a normal test order to confirm processing works:
aws lambda invoke \
  --function-name OrderIngestionFunction \
  --cli-binary-format raw-in-base64-out \
  --payload '{"orderId":"ORD-050","customerId":"CUST-7","amount":75,"item":"USB Cable"}' \
  response.json && cat response.json

This creates a new order that flows through the entire pipeline: DynamoDB, stream, EventBridge Pipe, SQS, then your processor Lambda.

  • Wait 15-20 seconds for the message to flow through the pipeline, then check CloudWatch Logs in the AWS Console.
  • Open the CloudWatch service in the AWS Console.
  • Click Log groups in the left sidebar.
  • Click on /aws/lambda/OrderProcessorFunction.
  • Click on the most recent log stream.

You should see a log entry containing Processing order ORD-050: USB Cable for customer CUST-7, amount $75 followed by Order ORD-050 processed successfully.

Don't see a log group for OrderProcessorFunction?

The log group only appears after the first invocation. Confirm the event source mapping is enabled by checking the Lambda console. Also verify the ingestion Lambda ran successfully (check its response output for a 200 status).

help me troubleshoot why my processor Lambda isn't being triggered by SQS

Submit a high-value order and watch it fail silently

Your pipeline processes normal orders without any issues. Now it is time to see what happens when an order exceeds the $500 threshold.

Before you run this next command, what do you think will happen when you submit an order with an amount of 999?

  • Submit a high-value order by running:
aws lambda invoke \
  --function-name OrderIngestionFunction \
  --cli-binary-format raw-in-base64-out \
  --payload '{"orderId":"ORD-060","customerId":"CUST-8","amount":999,"item":"Premium Laptop"}' \
  response.json && cat response.json

The ingestion Lambda succeeds (you see a 200 response). The order flows into DynamoDB, through the stream, through the pipe, and into SQS. Then the processor picks it up and crashes.

  • Wait 15-20 seconds, then return to CloudWatch Logs for /aws/lambda/OrderProcessorFunction.
  • Refresh the log streams and click the most recent one.

You should see PROCESSING FAILED: Order ORD-060 requires manual approval (amount $999 exceeds $500 limit) in the logs. The Lambda raised an exception.

What happens after the failure?

When Lambda throws an exception, it does NOT delete the message from SQS. SQS makes the message visible again after the visibility timeout expires, and Lambda picks it up again. This happens 3 times (your maxReceiveCount).

After 3 failed attempts, SQS automatically moves the message to the Dead-Letter Queue. You will wait for this to happen now.

  • Wait about 2-3 minutes for all 3 retry attempts to complete. You can watch additional error entries appear in CloudWatch Logs during this time.
  • Once you see 3 failed invocations in the logs, check the Dead-Letter Queue for the failed message by running:
aws sqs receive-message \
  --queue-url https://sqs.[[AWS_REGION="your AWS region, e.g. us-east-1"]].amazonaws.com/[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]/OrdersDLQ \
  --wait-time-seconds 5

You should see JSON output containing the failed message body with your ORD-060 order data nested inside a DynamoDB stream event structure.

The order failed, retried 3 times, and landed in the DLQ. But here is the problem: nobody was notified. You only know this happened because you manually checked. In production, this means lost revenue and frustrated customers with no alert going to your team.

DLQ shows no messages?

The retries take time. SQS uses a visibility timeout (default 30 seconds) between each retry. Wait 2-3 minutes total from when you submitted the high-value order, then try the receive-message command again. If you still see nothing, confirm the redrive policy is set on OrdersProcessingQueue with maxReceiveCount of 3.

help me figure out why messages aren't moving to my Dead-Letter Queue

Your pipeline now has a real problem: failures are completely silent. In the next step, you will add SNS alerting so your team gets an email the moment a failed order lands in the DLQ.

Add SNS Alerting for Dead-Letter Queue Failures

In the previous step, you watched a high-value order silently land in the Dead-Letter Queue. You only knew it failed because you manually checked.

In production, failed orders mean lost revenue and frustrated customers. Your team needs to know the instant a failure occurs. In this step, you will wire up SNS email alerts triggered by a CloudWatch Alarm, so failures surface automatically.

In this step, get ready to:
  • Create an SNS topic and subscribe your email to it.
  • Create a CloudWatch Alarm monitoring the DLQ for new messages.
  • Test the full end-to-end pipeline and receive an email alert.
Create the SNS topic and subscribe your email

Amazon SNS is a pub/sub messaging service. You will create a topic that acts as a broadcast channel. When the alarm fires, it publishes a message to this topic, and every subscriber (your email) gets notified.

  • Create the SNS topic by running this command:
aws sns create-topic --name OrdersDLQAlerts

What does this command do?

This creates a new SNS topic named OrdersDLQAlerts. The output contains a TopicArn value that you will need for the next commands.

  • Copy the TopicArn value from the output. It looks like arn:aws:sns:us-east-1:123456789012:OrdersDLQAlerts.
  • Subscribe your email address to the topic by running this command (replace the topic ARN and email with your own values):
aws sns subscribe \
    --topic-arn [[SNS_TOPIC_ARN="arn:aws:sns:REGION:ACCOUNT_ID:OrdersDLQAlerts"]] \
    --protocol email \
    --notification-endpoint your-email@example.com

What does this command do?

The --protocol email flag tells SNS to deliver messages via SMTP. The --notification-endpoint is the email address that receives alerts.

The output shows "SubscriptionArn": "pending confirmation" because email subscriptions require you to confirm before messages flow.

  • Check your email inbox for a message from AWS with the subject line containing your topic name.
  • Click the Confirm subscription link in the email.
  • You should see a page that says your subscription is confirmed.

Don't see the confirmation email?

  • Check your spam or junk folder.
  • Make sure you used the correct email address in the --notification-endpoint flag.
  • Run the subscribe command again if the email address had a typo.

Why isn't my SNS subscription confirmation email arriving?

Create the CloudWatch Alarm on the DLQ

CloudWatch collects SQS metrics automatically every minute. The metric ApproximateNumberOfMessagesVisible tracks how many messages are sitting in a queue waiting to be read.

You will create an alarm that fires the moment this metric reaches 1 or more on the DLQ. That means any failed message triggers an immediate notification.

  • Create the CloudWatch Alarm by running this command (replace the SNS topic ARN with your own):
aws cloudwatch put-metric-alarm \
    --alarm-name OrdersDLQAlarm \
    --alarm-description "Alert when messages appear in the Orders DLQ" \
    --metric-name ApproximateNumberOfMessagesVisible \
    --namespace AWS/SQS \
    --statistic Sum \
    --period 60 \
    --evaluation-periods 1 \
    --threshold 1 \
    --comparison-operator GreaterThanOrEqualToThreshold \
    --dimensions Name=QueueName,Value=OrdersDLQ \
    --alarm-actions [[SNS_TOPIC_ARN="arn:aws:sns:REGION:ACCOUNT_ID:OrdersDLQAlerts"]]

What does this alarm do?

  • It checks the OrdersDLQ every 60 seconds (--period 60).
  • If the number of visible messages is >= 1 for 1 evaluation period (--evaluation-periods 1), the alarm transitions to ALARM state.
  • When in ALARM state, it publishes a notification to your OrdersDLQAlerts SNS topic, which sends you an email.
  • Verify the alarm was created by running:
aws cloudwatch describe-alarms --alarm-names OrdersDLQAlarm
  • You should see output containing your alarm name, the metric ApproximateNumberOfMessagesVisible, and the AlarmActions field pointing to your SNS topic ARN.

Alarm not appearing?

  • Double-check that the --alarm-actions value is the full ARN of your SNS topic (starts with arn:aws:sns:).
  • Make sure you did not add quotes around the dimension value. The format is Name=QueueName,Value=OrdersDLQ with no spaces around the comma.

Why isn't my CloudWatch alarm showing up after running put-metric-alarm?

Test the full pipeline end-to-end

Everything is wired up. Time to prove it works from start to finish. You will submit a high-value order, and if all the plumbing is correct, an email will land in your inbox within a few minutes.

  • First, purge any existing messages from the DLQ so you start with a clean slate:
aws sqs purge-queue --queue-url [[DLQ_URL="https://sqs.REGION.amazonaws.com/ACCOUNT_ID/OrdersDLQ"]]
  • Before you invoke the next command, predict what will happen. Do you think the order will process successfully, or will it fail? Why?
  • Submit a high-value order (amount over 500) by invoking the ingestion Lambda:
aws lambda invoke \
    --function-name OrderIngestionFunction \
    --cli-binary-format raw-in-base64-out \
    --payload '{"orderId":"ORD-DLQ-TEST","customerId":"CUST-99","amount":999,"item":"Premium Laptop"}' \
    /tmp/ingestion-output.json
  • Check the output file to confirm ingestion succeeded:
cat /tmp/ingestion-output.json

You should see a 200 status confirming the order was received. Now the dominoes start falling: DynamoDB Streams captures the insert, EventBridge Pipes routes it to SQS, the processor Lambda picks it up, fails on the $999 amount, and SQS retries 3 times before sending the message to the DLQ.

  • Wait about 2-3 minutes for the retries to exhaust and the message to land in the DLQ.
  • Confirm the message reached the DLQ by running:
aws sqs get-queue-attributes \
    --queue-url [[DLQ_URL="https://sqs.REGION.amazonaws.com/ACCOUNT_ID/OrdersDLQ"]] \
    --attribute-names ApproximateNumberOfMessages

You should see ApproximateNumberOfMessages showing at least 1. That means the failed order landed in the DLQ.

  • Now check your email inbox. Within 1-2 minutes of the message appearing in the DLQ, CloudWatch should have triggered the alarm and SNS should have delivered an alert to your email.

What should I see in the email?

The email subject includes ALARM: "OrdersDLQAlarm" and the body contains details about the metric that triggered it, including the queue name and the threshold that was breached.

This proves your entire event-driven pipeline works end-to-end: order ingestion, stream capture, pipe routing, processing failure, DLQ catch, alarm trigger, and email delivery.

Not receiving the alert email?

  • Confirm your SNS subscription is active (not pending). Run aws sns list-subscriptions-by-topic --topic-arn <your-topic-arn> and check the status shows Confirmed.
  • The CloudWatch alarm may still be in INSUFFICIENT_DATA state if the DLQ was inactive for over 6 hours. Polling the DLQ with aws sqs get-queue-attributes activates metric reporting.
  • Check the alarm state by running aws cloudwatch describe-alarms --alarm-names OrdersDLQAlarm and look at the StateValue field.

Why isn't my CloudWatch alarm sending an SNS email when messages land in my DLQ?

You now have complete observability over your order processing failures. Any order that crashes the processor will automatically alert your team within minutes. Next up, the Secret Mission will let you submit orders via HTTP instead of the CLI.

Secret mission

Add an API Gateway REST Endpoint

Right now, you can only submit orders via the AWS CLI. What if you could POST orders over HTTP instead? In this secret mission, you will expose your pipeline through an API Gateway REST endpoint, turning your backend into something a frontend could actually talk to.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. All resources in this project fall within the AWS Always Free tier limits for learning-scale usage, so there are no ongoing costs expected.

Resources you used:

  • DynamoDB table: Orders (with Streams enabled).
  • SQS queues: OrdersProcessingQueue and OrdersDLQ.
  • Lambda functions: OrderIngestionFunction and OrderProcessorFunction.
  • EventBridge Pipe: OrdersPipe.
  • SNS topic: OrdersDLQAlerts (with email subscription).
  • CloudWatch Alarm: OrdersDLQAlarm.
  • API Gateway REST API: OrdersAPI (Secret Mission).
  • IAM roles: OrderIngestionLambdaRole, OrderProcessorLambdaRole, EventBridgePipesRole.
  • Lambda event source mapping (connecting OrdersProcessingQueue to OrderProcessorFunction).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the pipeline.

  • All resources fall within the AWS Always Free tier. DynamoDB gives you 25 GB storage and 25 read/write capacity units. SQS gives you 1 million requests per month. Lambda gives you 1 million requests per month.
  • SNS gives you 1,000 email deliveries per month. CloudWatch gives you 10 free alarms. As long as you are not sending thousands of orders through the pipeline, you will not incur charges.

Pause - I'll come back to this later

Serverless resources only incur costs when used. There is nothing to pause because Lambda, SQS, DynamoDB, and EventBridge Pipes do not run continuously.

  • If you want to stop new orders from flowing, disable the EventBridge Pipe. In the EventBridge Pipes console, select OrdersPipe, then click Stop. This prevents new orders from reaching SQS while keeping everything intact.
  • Your DynamoDB data, Lambda code, and queue configuration all remain in place. Resume by starting the pipe again.

Delete - I don't want to use this again

Remove all project resources and start fresh. Run deletion commands in reverse order of creation to avoid dependency errors.

Delete resources in this order:

  • Delete the API Gateway REST API (Secret Mission). First find your API ID by running:
aws apigateway get-rest-apis --query "items[?name=='OrdersAPI'].id" --output text

Copy the ID from the output, then delete the API by running:

aws apigateway delete-rest-api --rest-api-id your-api-id-here
  • Delete the CloudWatch Alarm by running:
aws cloudwatch delete-alarms --alarm-names OrdersDLQAlarm
  • Delete the SNS topic (this also removes all subscriptions) by running:
aws sns delete-topic --topic-arn $(aws sns list-topics --query "Topics[?ends_with(TopicArn,'OrdersDLQAlerts')].TopicArn" --output text)
  • Delete the Lambda event source mapping. First find its UUID:
aws lambda list-event-source-mappings --function-name OrderProcessorFunction --query "EventSourceMappings[0].UUID" --output text

Copy the UUID from the output, then delete the mapping:

aws lambda delete-event-source-mapping --uuid your-uuid-here
  • Delete the EventBridge Pipe by running:
aws pipes delete-pipe --name OrdersPipe
  • Delete both Lambda functions by running:
aws lambda delete-function --function-name OrderProcessorFunction
aws lambda delete-function --function-name OrderIngestionFunction
  • Delete both SQS queues. Delete the processing queue first, then the DLQ:
aws sqs delete-queue --queue-url $(aws sqs get-queue-url --queue-name OrdersProcessingQueue --query "QueueUrl" --output text)
aws sqs delete-queue --queue-url $(aws sqs get-queue-url --queue-name OrdersDLQ --query "QueueUrl" --output text)
  • Delete the DynamoDB table by running:
aws dynamodb delete-table --table-name Orders
  • Delete the IAM roles. For each role, you must first delete its inline policies, then delete the role itself. Start with OrderIngestionLambdaRole:
aws iam list-role-policies --role-name OrderIngestionLambdaRole --query "PolicyNames" --output text

For each policy name returned, delete it:

aws iam delete-role-policy --role-name OrderIngestionLambdaRole --policy-name your-policy-name-here

Then delete the role itself:

aws iam delete-role --role-name OrderIngestionLambdaRole

Repeat the same three commands for OrderProcessorLambdaRole and EventBridgePipesRole.

  • Delete any local .zip files you created for Lambda deployment packages.

Nice Work!

Nice Work!

Nice work! You just built a fully event-driven order processing pipeline on AWS, connecting six serverless services into a resilient system that handles failures gracefully.

You've learned how to:

  • Connect DynamoDB Streams to SQS via an EventBridge Pipe with event filtering, decoupling order ingestion from background processing without writing any routing code.
  • Implement deliberate failure handling using a Dead-Letter Queue that catches messages after retry exhaustion, proving your pipeline degrades gracefully instead of losing data.
  • Set up real-time failure alerting with CloudWatch Alarms and SNS so your team gets notified within minutes of a processing failure landing in the DLQ.
  • Secret Mission: Exposed the entire pipeline via an API Gateway REST endpoint, turning your backend into a complete HTTP-accessible service ready for a frontend.

Ready to quiz yourself?