Troubleshoot a Serverless Order Pipeline

Break and fix 3 production Lambda failures using CloudWatch and DLQs.

Introduction

30 Second Summary

When something breaks in a live system, the clock is ticking. The difference between a 5-minute fix and a 5-hour outage comes down to whether you can actually see what went wrong.

In this project, you will build a serverless order processing pipeline on AWS, deliberately break it in three production-realistic ways, and then add the observability toolkit that turns hours of debugging into minutes.

What You'll Build

You POST an order to your API endpoint, watch it flow through Lambda and EventBridge in real-time structured logs, then break the pipeline and pinpoint the exact root cause in seconds using access logs, Logs Insights queries, and Dead Letter Queue metadata.

By the end of this project, you'll have:

  • A serverless order pipeline where you POST an order via curl and trace it flowing through API Gateway, Lambda, and EventBridge in real-time.
  • A production troubleshooting toolkit with structured JSON logging, API Gateway access logs, EventBridge bus logging, and a Dead Letter Queue that surfaces failures the moment they happen.
  • Reusable CloudWatch Logs Insights queries that detect timeouts, measure latency percentiles, and calculate cold start percentages across your Lambda functions.
  • Secret Mission: Define the entire pipeline as Terraform infrastructure-as-code so you can rebuild the stack from scratch in any AWS account.

Are there any prerequisites?

You need an AWS account with the AWS CLI configured. Basic familiarity with Lambda and API Gateway (knowing what they are and having used them at least once) is helpful but not required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Create the Serverless Pipeline

To diagnose production failures effectively, you first need a working system that can break. In this step, you will build the complete serverless order pipeline from scratch.

The pipeline follows a common event-driven pattern: an API receives orders, a Lambda function validates and publishes them to EventBridge, and a second Lambda fulfills the order. Once this baseline is running, you will have something healthy to compare against when things go wrong.

In this step, get ready to:
  • Create two Lambda functions and their IAM roles, plus an EventBridge custom bus and rule.
  • Create an API Gateway HTTP API with a POST /orders route.
  • Deploy the initial handler code and test the end-to-end flow.
Create the Lambda functions and EventBridge resources

You will create two Lambda functions: one to process incoming orders and one to fulfill them. Both use the python3.13 runtime.

  • Navigate to the Lambda service in the AWS Console.
  • Click Create function.
  • Select Author from scratch.
  • For Function name, enter order-processor.
  • For Runtime, select Python 3.13.
  • Under Permissions, expand Change default execution role and leave Create a new role with basic Lambda permissions selected.
  • Click Create function.

You should see the function overview page for order-processor with a green success banner at the top.

  • On the order-processor function page, click the Configuration tab.
  • Click General configuration.
  • Click Edit.
  • Set Timeout to 0 min 10 sec.
  • Click Save.
  • Still in the Configuration tab, click Environment variables.
  • Click Edit, then click Add environment variable.
  • For Key, enter EVENT_BUS_NAME. For Value, enter order-processing-bus.
  • Click Save.

Why set the timeout to 10 seconds?

Lambda's default timeout is 3 seconds. Setting it to 10 gives the function enough headroom for EventBridge calls. Later, you will deliberately lower the timeout to simulate a production failure.

Now create the second function for order fulfillment.

  • Navigate back to the Lambda Functions list.
  • Click Create function.
  • Select Author from scratch.
  • For Function name, enter order-fulfillment.
  • For Runtime, select Python 3.13.
  • Leave Create a new role with basic Lambda permissions selected under Permissions.
  • Click Create function.

You should see the function overview page for order-fulfillment with a green success banner.

  • Click the Configuration tab.
  • Click General configuration.
  • Click Edit.
  • Set Timeout to 0 min 10 sec.
  • Click Save.

You should now see both order-processor and order-fulfillment in the Lambda Functions list.

Next, add the EventBridge publish permission to the order-processor's IAM role.

  • Open the order-processor function and click the Configuration tab.
  • Click Permissions.
  • Under Execution role, click the role name link (it starts with order-processor-role-). This opens the IAM console.
  • In the IAM console, click Add permissions.
  • Click Create inline policy.
  • Click the JSON tab and paste the following policy:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "events:PutEvents",
            "Resource": "arn:aws:events:*:*:event-bus/order-processing-bus"
        }
    ]
}
  • Click Next.
  • For Policy name, enter eventbridge-put-events.
  • Click Create policy.

What does this policy do?

This grants the order-processor Lambda permission to call events:PutEvents on the order-processing-bus custom event bus. Without it, the function would get an access denied error when trying to publish order events.

Now create the EventBridge custom bus and routing rule.

  • Navigate to the Amazon EventBridge service in the AWS Console.
  • In the left navigation, click Event buses.
  • Click Create event bus.
  • For Name, enter order-processing-bus.
  • Leave all other settings at their defaults and click Create.

You should see order-processing-bus listed in the Event buses page.

  • In the left navigation, click Rules.
  • In the Event bus dropdown, select order-processing-bus.
  • Click Create rule.
  • For Name, enter order-placed-rule.
  • Make sure Event bus shows order-processing-bus and Rule type is set to Rule with an event pattern.
  • Click Next.

On the Build event pattern page:

  • Scroll down to Event pattern.
  • Select Enter my own (or Custom pattern) and paste the following JSON:
{
  "source": ["com.orders.processor"],
  "detail-type": ["OrderPlaced"]
}
  • Click Next.

What does this event pattern match?

This pattern tells EventBridge to route any event whose source is com.orders.processor and whose detail-type is OrderPlaced to whatever target you configure next.

On the Select targets page:

  • For Target type, select AWS service.
  • For Select a target, choose Lambda function.
  • For Function, select order-fulfillment.
  • Click Next.
  • On the Tags page, click Next again.
  • Review the rule and click Create rule.

What about the Lambda invoke permission?

When you add a Lambda function as a target in the EventBridge console, AWS automatically adds a resource-based policy to the Lambda allowing EventBridge to invoke it. You do not need to add this permission manually.

Don't see order-fulfillment in the Function dropdown?

Make sure you created both functions in the same AWS region. Check the region selector in the top-right corner of the console.

Help me find my Lambda function in EventBridge

Create the API Gateway HTTP API

You need a public endpoint to send orders to. An API Gateway HTTP API provides a URL that forwards requests to your order-processor Lambda.

  • Navigate to the API Gateway service in the AWS Console.
  • Click Create API.
  • Under HTTP API, click Build.
  • Click Add integration and select Lambda.
  • For Lambda function, select order-processor.
  • For API name, enter order-processing-api.
  • Click Next.

On the Configure routes page:

  • Change the Method to POST.
  • Change the Resource path to /orders.
  • Make sure the Integration target shows order-processor.
  • Click Next.
  • On the Define stages page, leave the default $default stage and click Next.
  • Review and click Create.

After creation, you will see your API's Invoke URL on the details page. Copy this URL and save it somewhere accessible.

What is the Invoke URL?

The Invoke URL is the public HTTPS endpoint for your API. HTTP APIs with a $default stage auto-deploy, meaning your POST /orders route is live the moment you create it.

Don't see an Invoke URL?

Click Stages in the left sidebar of the API Gateway console and select the $default stage. The Invoke URL appears at the top of the stage details.

Help me find my API Gateway invoke URL

Deploy the handler code and test the pipeline

Both Lambda functions currently have default placeholder code. You will now replace it with the order processing logic.

  • Navigate to the order-processor function in the Lambda console.
  • Click the Code tab.
  • In the code editor, select all the existing code in lambda_function.py and replace it with the following imports and client setup:
import json
import boto3
import os

# EventBridge client for publishing order events
client = boto3.client('events')
EVENT_BUS_NAME = os.environ.get('EVENT_BUS_NAME', 'order-processing-bus')
  • Below the setup code, add the handler function:
def lambda_handler(event, context):
    # Parse the incoming order from the request body
    body = json.loads(event.get('body', '{}'))
    order_id = body.get('order_id')
    customer_name = body.get('customer_name')
    items = body.get('items', [])
    event_detail = {
        'order_id': order_id,
        'customer_name': customer_name,
        'items': items,
        'total': body.get('total', 0)
    }
    # Publish order event to EventBridge
    response = client.put_events(Entries=[{
        'Source': 'com.orders.processor',
        'DetailType': 'OrderPlaced',
        'Detail': json.dumps(event_detail),
        'EventBusName': EVENT_BUS_NAME
    }])
    event_id = response['Entries'][0].get('EventId', '')
    return {
        'statusCode': 200,
        'body': json.dumps({'message': 'Order placed', 'order_id': order_id, 'event_id': event_id})
    }
  • Click Deploy to save your changes.

What does this code do?

The handler parses the order JSON from the HTTP request body. It then publishes an OrderPlaced event to the order-processing-bus using the EventBridge put_events API.

If the publish succeeds, the function returns a 200 response with the event ID. This is the basic version without structured logging. You will add proper logging in a later step.

Now deploy the fulfillment handler.

  • Navigate to the order-fulfillment function in the Lambda console.
  • Click the Code tab.
  • Replace all existing code in lambda_function.py with the following:
import json

def lambda_handler(event, context):
    # Extract order details from the EventBridge event
    detail = event.get('detail', {})
    order_id = detail.get('order_id')
    return {
        'statusCode': 200,
        'body': json.dumps({'order_id': order_id, 'status': 'FULFILLED'})
    }
  • Click Deploy.

What does the fulfillment handler do?

When EventBridge invokes this function, the event object contains a detail field with the order payload. The handler extracts the order_id and returns a fulfilled status. This is a minimal version that proves the event was delivered.

Time to test the full pipeline. Before you run the next command, think about what you expect to happen. Will the request succeed? What status code should the API return?

  • Open your terminal and send a test order by running the following command (replace the URL with your API's Invoke URL):
curl -X POST [[API_URL="your API Gateway invoke URL"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-001", "customer_name": "Alice", "items": [{"name": "Widget", "quantity": 2}], "total": 29.99}'

You should see a response like this:

{"message": "Order placed", "order_id": "ORD-001", "event_id": "abc12345-..."}

A 200 response with an event_id confirms the order-processor successfully published to EventBridge.

  • Navigate to CloudWatch in the AWS Console.
  • Click Log groups in the left navigation.
  • Open the /aws/lambda/order-processor log group and check the most recent log stream. You should see entries showing the function executed.
  • Go back to Log groups and open /aws/lambda/order-fulfillment. You should see a log stream with execution entries for the same order.

Seeing logs in both log groups proves the entire pipeline is working: API Gateway invoked the processor, which published to EventBridge, which triggered the fulfillment function.

Seeing a 500 Internal Server Error?

Check the /aws/lambda/order-processor log group for error messages. The most common cause is a missing EventBridge permission. Verify that the inline policy eventbridge-put-events is attached to the order-processor's execution role in IAM.

Help me debug my order-processor Lambda returning 500

Processor works but no logs in order-fulfillment?

Verify that the order-placed-rule in EventBridge is Enabled (not disabled). Also confirm the event pattern matches the source com.orders.processor and detail-type OrderPlaced exactly. Typos in these values will silently drop events.

Help me debug EventBridge not triggering my fulfillment Lambda

✔️ Awesome, I've got everything!

Great. Make sure you have your API Gateway Invoke URL saved somewhere. You will use it in every step from here.

ⓧ I'd like to double check the full code

Here is the complete code for both Lambda functions. Confirm yours matches exactly.

order-processor (lambda_function.py):

import json
import boto3
import os

client = boto3.client('events')
EVENT_BUS_NAME = os.environ.get('EVENT_BUS_NAME', 'order-processing-bus')

def lambda_handler(event, context):
    body = json.loads(event.get('body', '{}'))
    order_id = body.get('order_id')
    customer_name = body.get('customer_name')
    items = body.get('items', [])
    event_detail = {
        'order_id': order_id,
        'customer_name': customer_name,
        'items': items,
        'total': body.get('total', 0)
    }
    response = client.put_events(Entries=[{
        'Source': 'com.orders.processor',
        'DetailType': 'OrderPlaced',
        'Detail': json.dumps(event_detail),
        'EventBusName': EVENT_BUS_NAME
    }])
    event_id = response['Entries'][0].get('EventId', '')
    return {
        'statusCode': 200,
        'body': json.dumps({'message': 'Order placed', 'order_id': order_id, 'event_id': event_id})
    }

order-fulfillment (lambda_function.py):

import json

def lambda_handler(event, context):
    detail = event.get('detail', {})
    order_id = detail.get('order_id')
    return {
        'statusCode': 200,
        'body': json.dumps({'order_id': order_id, 'status': 'FULFILLED'})
    }

Your serverless pipeline is live. Next up, you will send several orders through it and trace a single order across both log groups to establish what "healthy" looks like.

Verify the Happy Path

In the last step, you built the serverless pipeline and confirmed it works with a single test request. A single successful call proves the wiring is correct, but it doesn't tell you what "healthy" looks like over multiple invocations with different data.

Before you deliberately break things in the next step, you need to see the full event flow from API Gateway through EventBridge to the fulfillment Lambda with varied payloads. This baseline is what you will compare against when failures start appearing.

In this step, get ready to:
  • Send 2-3 test orders with different payloads and confirm 200 responses with unique event IDs.
  • Trace a single order across both Lambda log groups in CloudWatch and note the time cost of manual correlation.
Send multiple test orders with different payloads

To establish a meaningful baseline, you need more than one data point. Sending orders with different customers, items, and totals proves the pipeline handles varied input correctly.

  • Send your first test order by running this command:
curl -X POST [[API_URL="your API Gateway invoke URL from Step 1"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-101", "customer_name": "Alice Smith", "items": [{"name": "Mechanical Keyboard", "quantity": 1}], "total": 89.99}'

What does this command do?

This curl command POSTs a JSON order to your API Gateway endpoint. The order-processor Lambda receives it, publishes an event to EventBridge, and the order-fulfillment Lambda processes it downstream.

You should see a JSON response with a 200 status containing your order_id and a unique event_id.

  • Send a second order with a different customer and items:
curl -X POST [[API_URL="your API Gateway invoke URL from Step 1"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-102", "customer_name": "Bob Chen", "items": [{"name": "USB-C Monitor", "quantity": 1}, {"name": "HDMI Cable", "quantity": 2}], "total": 349.97}'

Again you should see a 200 response with a different event_id.

  • Send a third order with yet another customer:
curl -X POST [[API_URL="your API Gateway invoke URL from Step 1"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-103", "customer_name": "Carol Davis", "items": [{"name": "Standing Desk", "quantity": 1}], "total": 599.00}'

All three responses should return a 200 status with the message Order placed and a unique event_id for each.

Getting a connection error or 403?

  • Double-check your API URL ends with /orders and does not have a trailing slash after it.
  • Make sure your API Gateway URL is the invoke URL from the $default stage (it looks like https://xxxxxxxxxx.execute-api.us-east-1.amazonaws.com).
  • help me debug why my curl request to API Gateway is failing
Trace an order across CloudWatch log groups

Your three orders succeeded. Now try to follow a single order's journey across both Lambda functions using only the basic CloudWatch logs that Lambda auto-generates.

  • Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
  • In the left navigation pane, click **Log Management**.
  • Click the log group named /aws/lambda/order-processor.
  • Click Search log group to search across all log streams in this group.
  • In the filter pattern field, type ORD-102 and press Enter to search for your second order.

You should see log events that mention ORD-102 in the order-processor function. Take note of the timestamp.

  • Now go back to the log groups list by clicking Log groups in the breadcrumb or navigation pane.
  • Click the log group named /aws/lambda/order-fulfillment.
  • Click Search log group again.
  • Type ORD-102 in the filter pattern field and press Enter.

Before you check the results, do you expect to find matching entries here? Think about what the fulfillment Lambda receives from EventBridge.

You should see log events containing ORD-102 in the fulfillment function logs too. This confirms the order flowed end-to-end through EventBridge.

Why is this manual search painful?

Notice how many clicks that took. You had to navigate to one log group, search, remember the result, navigate back, open another log group, and search again.

With just 3 test orders and 2 Lambda functions, the process is manageable. In production, you might have dozens of functions, thousands of log streams, and no structured fields to search on. That is the problem you will solve in Step 4.

Not finding your order_id in the fulfillment logs?

  • EventBridge delivery can take a few seconds. Wait 10-15 seconds after sending the order, then search again.
  • Make sure you are searching /aws/lambda/order-fulfillment (not the processor log group).
  • Check that your time range in the search includes the last few minutes.
  • help me figure out why my order isn't showing up in the fulfillment Lambda logs

You now have a documented baseline of a healthy pipeline. Three orders flowed successfully from API Gateway through the processor Lambda, into EventBridge, and out to the fulfillment Lambda. Next up, you will deliberately break this pipeline in three production-realistic ways and feel the pain of diagnosing failures with these basic logs.

Break Things and Attempt Diagnosis

Your pipeline works perfectly right now. In production, that baseline rarely lasts. Lambda functions time out under load, permissions get revoked during deploys, and concurrency gets throttled without warning.

In this step, you will inject 3 production-realistic failures into your pipeline and try to diagnose them using only basic CloudWatch log streams. The frustration of scrolling through unstructured logs across multiple log groups is the problem that proper observability solves.

In this step, get ready to:
  • Inject a Lambda timeout that returns a useless 500 error from the API.
  • Inject a permission error that causes silent delivery failure with no logs.
  • Inject concurrency throttling that looks identical to the permission error from the outside.
Inject Failure 1: Lambda timeout

The most common synchronous Lambda failure is a timeout. When a function exceeds its allowed execution time, API Gateway returns a generic 500 error with zero diagnostic detail.

  • Open your local order_processor.py file (the one you deployed in Step 1).
  • Add import time at the top, and add time.sleep(5) as the first line inside the handler. The updated file should look like this:
import json
import boto3
import os
import time

client = boto3.client('events')
EVENT_BUS_NAME = os.environ.get('EVENT_BUS_NAME', 'order-processing-bus')

def lambda_handler(event, context):
    time.sleep(5)
    body = json.loads(event.get('body', '{}'))
    order_id = body.get('order_id')
    customer_name = body.get('customer_name')
    items = body.get('items', [])
    event_detail = {
        'order_id': order_id,
        'customer_name': customer_name,
        'items': items,
        'total': body.get('total', 0)
    }
    response = client.put_events(Entries=[{
        'Source': 'com.orders.processor',
        'DetailType': 'OrderPlaced',
        'Detail': json.dumps(event_detail),
        'EventBusName': EVENT_BUS_NAME
    }])
    event_id = response['Entries'][0].get('EventId', '')
    return {
        'statusCode': 200,
        'body': json.dumps({'message': 'Order placed', 'order_id': order_id, 'event_id': event_id})
    }

What did you change?

You added import time and time.sleep(5) to force the function to wait 5 seconds before doing any work. Combined with a 3-second timeout (which you will set next), this guarantees the function will always exceed its allowed execution time.

  • Zip the updated file and redeploy it by running:
zip order_processor.zip order_processor.py
aws lambda update-function-code \
    --function-name order-processor \
    --zip-file fileb://order_processor.zip
  • Now reduce the function timeout to 3 seconds by running:
aws lambda update-function-configuration \
    --function-name order-processor \
    --timeout 3

The function now sleeps for 5 seconds but only has 3 seconds to live. Every invocation will time out.

Before you send an order, what do you think the API will return? Will you get a helpful error message explaining what went wrong?

  • Send a test order to your API by running:
curl -X POST [[API_URL="your API Gateway endpoint URL"]]/orders \
    -H "Content-Type: application/json" \
    -d '{"order_id": "ORD-TIMEOUT-1", "customer_name": "Test User", "items": [{"name": "Widget", "quantity": 1}], "total": 9.99}'

You get back a completely unhelpful response:

{"message":"Internal Server Error"}

No order_id. No error code. No indication of what went wrong. This is exactly what your customers would see.

  • Now try to diagnose it. Open the CloudWatch console.
  • Navigate to **Logs**, then **Log groups**.
  • Click on /aws/lambda/order-processor.
  • Open the most recent log stream.

You will eventually find a line mentioning the timeout. But notice how many clicks it took and how many unrelated log entries you scrolled through to find it.

Why is this so painful?

The API Gateway returned a generic 500 with no detail about which Lambda timed out or why. To find the root cause, you had to: open CloudWatch, guess which log group, pick the right log stream, and manually scan unstructured text.

In production with thousands of requests, this manual approach is nearly impossible at scale.

Inject Failure 2: permission error

The timeout is useful for testing, but you need to isolate the next failure. You will revert the timeout so the processor works again, then break the connection between EventBridge and the fulfillment Lambda.

  • Set the order-processor timeout back to 10 seconds so it no longer times out by running:
aws lambda update-function-configuration \
    --function-name order-processor \
    --timeout 10

Why not remove the sleep too?

The time.sleep(5) is still in the code, but with a 10-second timeout the function completes successfully (5 seconds of sleep is well within 10 seconds). This saves you a redeploy and still proves the processor works.

  • First, find the statement ID of the EventBridge permission on your fulfillment Lambda by running:
aws lambda get-policy --function-name order-fulfillment

Look at the output for the Sid field in the policy statement. Note this value. It is the statement ID you will use in the next command.

  • Remove the EventBridge invoke permission from the fulfillment Lambda using the statement ID you found:
aws lambda remove-permission \
    --function-name order-fulfillment \
    --statement-id [[STATEMENT_ID="the Sid value from get-policy output"]]

This command produces no output on success. EventBridge can no longer invoke the fulfillment Lambda.

What do you think will happen when you send an order now? Will you see an error anywhere?

  • Send a test order by running:
curl -X POST [[API_URL="your API Gateway endpoint URL"]]/orders \
    -H "Content-Type: application/json" \
    -d '{"order_id": "ORD-PERM-1", "customer_name": "Test User", "items": [{"name": "Gadget", "quantity": 2}], "total": 19.99}'

The response comes back 200 with an event_id. The processor worked fine. But wait for 10-15 seconds, then check the fulfillment Lambda.

  • In the CloudWatch console, open the /aws/lambda/order-fulfillment log group.
  • Check for any new log streams since your test.

Nothing. No new log entries at all. The fulfillment Lambda was never invoked. The order was accepted, published to EventBridge, and then silently dropped.

Why is this failure so dangerous?

Your API returned success (200). From the customer's perspective, the order went through. But the fulfillment Lambda never ran. There is no error anywhere you can easily find. EventBridge silently failed to deliver the event because it no longer has permission to invoke the target.

Without a Dead Letter Queue or EventBridge bus logging, this order is simply lost.

Inject Failure 3: concurrency throttle

Now you will inject a third failure that produces identical symptoms to the permission error. This teaches the critical lesson that different root causes can look the same from the outside.

  • Restore the EventBridge permission so it can invoke the fulfillment Lambda again. Run:
aws lambda add-permission \
    --function-name order-fulfillment \
    --action lambda:InvokeFunction \
    --statement-id AllowEventBridgeInvoke \
    --principal events.amazonaws.com
  • Now throttle the fulfillment Lambda by setting its reserved concurrency to 0. This prevents all invocations by running:
aws lambda put-function-concurrency \
    --function-name order-fulfillment \
    --reserved-concurrent-executions 0

What does reserved concurrency of 0 mean?

Setting reserved concurrency to 0 means Lambda will reject every invocation with a TooManyRequestsException (429 status). The function literally cannot run. EventBridge will retry delivery for up to 24 hours before giving up.

  • Send multiple test orders by running:
curl -X POST [[API_URL="your API Gateway endpoint URL"]]/orders \
    -H "Content-Type: application/json" \
    -d '{"order_id": "ORD-THROTTLE-1", "customer_name": "Test User", "items": [{"name": "Doohickey", "quantity": 3}], "total": 29.99}'

Again you get a 200 response with an event_id. The processor published the event successfully.

  • Check the /aws/lambda/order-fulfillment log group in CloudWatch.

Nothing. No new invocations. From the outside, this looks exactly like Failure 2. The processor succeeded, the fulfillment Lambda never ran, and there is no obvious error trail.

Two failures, one symptom

Failure 2 (permission error) and Failure 3 (concurrency throttle) produce identical symptoms: orders accepted, fulfillment silent. With basic CloudWatch logs alone, you cannot distinguish between them.

In the next step, you will add a Dead Letter Queue that reveals the difference. Permission errors go to the DLQ immediately (RETRY_ATTEMPTS = 0). Throttled events are retried for hours before landing in the DLQ (RETRY_ATTEMPTS > 0). Same symptom, completely different metadata.

You have now experienced the core problem. Three different failures, three different root causes, and basic CloudWatch logs leave you scrolling, guessing, and losing time. Next up, you will add the production observability toolkit that turns these minutes of manual searching into seconds of targeted diagnosis.

Add the Production Observability Toolkit

In the previous step, you experienced the pain of diagnosing failures by manually scrolling through unstructured log streams. The API Gateway returned a generic 500 with no explanation. The fulfillment Lambda silently stopped firing with no evidence of what went wrong.

Manual log searching is slow and unreliable at scale. Structured logging, API Gateway access logs, EventBridge bus logging, and Dead Letter Queues are purpose-built for diagnosing the exact failures you just experienced. This step adds the toolkit that turns hours of debugging into minutes.

In this step, get ready to:
  • Update both Lambda functions with structured JSON logging.
  • Enable API Gateway HTTP API access logging with error context variables.
  • Enable EventBridge bus logging and create an SQS Dead Letter Queue.
Deploy structured JSON logging to both Lambda functions

Structured logging means every log entry is a valid JSON object with consistent fields like requestId, order_id, and timestamp. This makes logs machine-parseable so you can query them with CloudWatch Logs Insights instead of manually searching text.

  • In the AWS Console, navigate to the Lambda service.
  • Click on the order-processor function.
  • Select the Code tab.
  • Select all the existing code in the editor and delete it.
  • Replace it with the following code. Start by pasting this imports and setup section:
import json
import boto3
import logging
import os
from datetime import datetime, timezone

# Set up structured logger
logger = logging.getLogger()
logger.setLevel(logging.INFO)

events_client = boto3.client('events')

EVENT_BUS_NAME = os.environ.get('EVENT_BUS_NAME', 'order-processing-bus')
  • Continue by pasting this handler entry point and JSON parsing section directly below:
def lambda_handler(event, context):
    logger.info(json.dumps({
        "message": "Order received",
        "requestId": context.aws_request_id,
        "httpMethod": event.get("requestContext", {}).get("http", {}).get("method"),
        "path": event.get("rawPath"),
        "timestamp": datetime.now(timezone.utc).isoformat()
    }))

    try:
        body = json.loads(event.get('body', '{}'))
    except json.JSONDecodeError as e:
        logger.error(json.dumps({
            "message": "Invalid JSON in request body",
            "error": str(e),
            "requestId": context.aws_request_id
        }))
        return {
            'statusCode': 400,
            'body': json.dumps({'error': 'Invalid JSON in request body'})
        }
  • Continue with the validation section:
    order_id = body.get('order_id')
    customer_name = body.get('customer_name')
    items = body.get('items', [])

    if not order_id or not customer_name or not items:
        logger.error(json.dumps({
            "message": "Missing required fields",
            "requestId": context.aws_request_id,
            "order_id": order_id,
            "has_customer_name": bool(customer_name),
            "has_items": bool(items)
        }))
        return {
            'statusCode': 400,
            'body': json.dumps({'error': 'Missing required fields: order_id, customer_name, items'})
        }
  • Continue with the event publishing section:
    event_detail = {
        'order_id': order_id,
        'customer_name': customer_name,
        'items': items,
        'total': body.get('total', 0),
        'timestamp': datetime.now(timezone.utc).isoformat()
    }

    try:
        response = events_client.put_events(
            Entries=[
                {
                    'Source': 'com.orders.processor',
                    'DetailType': 'OrderPlaced',
                    'Detail': json.dumps(event_detail),
                    'EventBusName': EVENT_BUS_NAME
                }
            ]
        )
  • Continue with the response handling and exception block to finish the function:
        if response['FailedEntryCount'] > 0:
            error_entry = response['Entries'][0]
            logger.error(json.dumps({
                "message": "EventBridge put_events failed",
                "requestId": context.aws_request_id,
                "errorCode": error_entry.get('ErrorCode'),
                "errorMessage": error_entry.get('ErrorMessage')
            }))
            return {
                'statusCode': 500,
                'body': json.dumps({'error': 'Failed to publish order event'})
            }

        event_id = response['Entries'][0]['EventId']
        logger.info(json.dumps({
            "message": "Order event published successfully",
            "requestId": context.aws_request_id,
            "order_id": order_id,
            "eventId": event_id,
            "eventBus": EVENT_BUS_NAME
        }))
        return {
            'statusCode': 200,
            'body': json.dumps({
                'message': 'Order placed successfully',
                'order_id': order_id,
                'event_id': event_id
            })
        }

    except Exception as e:
        logger.error(json.dumps({
            "message": "Exception publishing to EventBridge",
            "requestId": context.aws_request_id,
            "error": str(e),
            "errorType": type(e).__name__
        }))
        return {
            'statusCode': 500,
            'body': json.dumps({'error': 'Internal server error'})
        }

What does this structured logging code do?

Every log entry is a json.dumps() call that outputs valid JSON. Each entry includes context.aws_request_id so you can correlate all logs from a single invocation.

The order_id field lets you trace a single order across both Lambda functions. The timestamp field uses UTC ISO format for consistent time ordering across log groups.

  • Click Deploy to save and deploy the function.
  • You should see a green "Successfully deployed" banner at the top of the code editor.

Now repeat the process for the fulfillment function.

  • Navigate back to the Lambda functions list.
  • Click on the order-fulfillment function.
  • Select the Code tab.
  • Delete all existing code and paste this first section:
import json
import logging
from datetime import datetime, timezone

# Set up structured logger
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def lambda_handler(event, context):
    logger.info(json.dumps({
        "message": "Order fulfillment triggered",
        "requestId": context.aws_request_id,
        "eventSource": event.get('source'),
        "detailType": event.get('detail-type'),
        "timestamp": datetime.now(timezone.utc).isoformat()
    }))

    detail = event.get('detail', {})
    order_id = detail.get('order_id')
    customer_name = detail.get('customer_name')
    items = detail.get('items', [])
    total = detail.get('total', 0)
  • Continue with the validation and processing section directly below:
    if not order_id:
        logger.error(json.dumps({
            "message": "Missing order_id in event detail",
            "requestId": context.aws_request_id,
            "detail": detail
        }))
        raise ValueError("Missing order_id in event detail")

    logger.info(json.dumps({
        "message": "Processing order",
        "requestId": context.aws_request_id,
        "order_id": order_id,
        "customer_name": customer_name,
        "item_count": len(items),
        "total": total,
        "status": "PROCESSING"
    }))
  • Finish by pasting the fulfillment loop and return:
    fulfilled_items = []
    for item in items:
        fulfilled_items.append({
            "name": item.get("name", "unknown"),
            "quantity": item.get("quantity", 1),
            "status": "FULFILLED"
        })

    logger.info(json.dumps({
        "message": "Order fulfilled successfully",
        "requestId": context.aws_request_id,
        "order_id": order_id,
        "customer_name": customer_name,
        "fulfilled_items": len(fulfilled_items),
        "status": "COMPLETED",
        "completedAt": datetime.now(timezone.utc).isoformat()
    }))

    return {
        'statusCode': 200,
        'body': json.dumps({
            'order_id': order_id,
            'status': 'FULFILLED',
            'items': fulfilled_items
        })
    }
  • Click Deploy to save and deploy the fulfillment function.

Both Lambda functions now emit structured JSON logs. Send a test order to verify the processor outputs structured entries.

  • In your terminal, send a test order by running:
curl -X POST [[API_URL="your API Gateway invoke URL"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "obs-001", "customer_name": "Test User", "items": [{"name": "Widget", "quantity": 2}], "total": 29.99}'

The request will still timeout (the processor still has the 3-second timeout with no sleep removed in the code, but the code no longer has time.sleep(5)). You should get a successful 200 response with a JSON body containing order_id and event_id.

  • In the AWS Console, navigate to CloudWatch > Log groups > /aws/lambda/order-processor.
  • Click the most recent log stream.
  • Confirm the log entries are valid JSON objects with fields like "message", "requestId", and "order_id".

Still seeing unstructured text logs?

Make sure you clicked Deploy after pasting the code. The green banner must say "Successfully deployed" before invoking the function.

If you see a syntax error, check that all the code sections are pasted in order with no gaps or duplicates. Help me fix my structured logging deployment.

Enable API Gateway HTTP API access logging

Without access logging, the API Gateway returns {"message":"Internal Server Error"} to clients with zero diagnostic detail. Access logs with $context.integrationErrorMessage reveal exactly why the gateway returned a 500.

  • Create the log group that will receive API Gateway access logs by running:
aws logs create-log-group --log-group-name /aws/apigateway/order-processing-api
  • Confirm the log group was created by running:
aws logs describe-log-groups --log-group-name-prefix /aws/apigateway/order-processing-api

You should see the log group listed in the output with its ARN.

  • In the AWS Console, navigate to the API Gateway service.
  • Click on order-processing-api.
  • In the left sidebar, click Monitor > Logging.
  • Click Select next to the $default stage.
  • Click Edit.
  • Toggle Access logging to enabled.
  • In the Log destination field, paste the ARN of the log group you created. You can find it in the output of the describe command you just ran (it looks like arn:aws:logs:REGION:ACCOUNT:/aws/apigateway/order-processing-api).
  • In the Log format field, select JSON and paste the following format:
{ "requestId": "$context.requestId", "ip": "$context.identity.sourceIp", "requestTime": "$context.requestTime", "httpMethod": "$context.httpMethod", "routeKey": "$context.routeKey", "status": "$context.status", "protocol": "$context.protocol", "responseLength": "$context.responseLength", "integrationErrorMessage": "$context.integrationErrorMessage", "errorMessage": "$context.error.message", "errorResponseType": "$context.error.responseType" }
  • Click Save.

Why these specific context variables?

The $context.integrationErrorMessage variable is the key one. When a Lambda times out, this field contains the actual timeout error message that the client never sees.

The $context.error.message and $context.error.responseType capture gateway-level errors like throttling or authorization failures.

  • Verify access logging is working by sending another test order:
curl -X POST [[API_URL="your API Gateway invoke URL"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "obs-002", "customer_name": "Access Log Test", "items": [{"name": "Gadget", "quantity": 1}], "total": 15.00}'
  • In the AWS Console, navigate to CloudWatch > Log groups > /aws/apigateway/order-processing-api.
  • Click the most recent log stream.
  • Confirm you see a JSON entry with your requestId, status, and integrationErrorMessage fields.

No entries in the API Gateway access log group?

Access logs can take 1-2 minutes to appear. Refresh the log group page after a short wait.

If nothing appears after 2 minutes, double-check that the log group ARN you pasted matches the one returned by the describe command exactly. Help me troubleshoot missing API Gateway access logs.

Enable EventBridge bus logging and create the Dead Letter Queue

EventBridge bus logging captures the event lifecycle from receipt through delivery. A Dead Letter Queue catches events that EventBridge cannot deliver to the target, preserving failure metadata you would otherwise lose.

  • Create the log group for EventBridge bus logging by running:
aws logs create-log-group --log-group-name /aws/events/order-processing-bus
  • In the AWS Console, navigate to the Amazon EventBridge service.
  • In the left sidebar, click Event buses.
  • Click on order-processing-bus.
  • Find the Logging section and click Configure (or Edit if logging is already shown).
  • Set the Log level to ERROR.
  • Under Log destinations, select CloudWatch Logs and choose the /aws/events/order-processing-bus log group.
  • Click Save or Update.

Why ERROR level and not TRACE?

The ERROR level captures delivery failures, throttled invocations, and DLQ events without flooding your log group with every successful delivery. For production troubleshooting, you only need to see what went wrong.

Now create the SQS Dead Letter Queue that will capture failed EventBridge deliveries.

  • Create the DLQ by running:
aws sqs create-queue --queue-name order-processing-dlq

The command returns the queue URL. Note it for the next step.

  • Get the queue ARN (you will need it shortly) by running:
aws sqs get-queue-attributes \
  --queue-url [[DLQ_URL="the QueueUrl from the previous command"]] \
  --attribute-names QueueArn
  • The DLQ needs a resource policy that allows EventBridge to send messages. Set the policy by running:
aws sqs set-queue-attributes \
  --queue-url [[DLQ_URL="the QueueUrl from create-queue"]] \
  --attributes '{"Policy":"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Action\":\"sqs:SendMessage\",\"Resource\":\"[[DLQ_ARN=\"your DLQ ARN from get-queue-attributes\"]]\"  }]}"}'

Why does the DLQ need its own policy?

When you attach a DLQ via the API (rather than the console), EventBridge does not automatically add permissions to the queue. Without this policy, EventBridge would silently drop failed events instead of sending them to the DLQ.

  • Finally, attach the DLQ to your EventBridge rule target by running:
aws events put-targets \
  --rule order-placed-rule \
  --event-bus-name order-processing-bus \
  --targets "Id"="order-fulfillment-target","Arn"="[[FULFILLMENT_ARN=\"your order-fulfillment Lambda ARN\"]]","DeadLetterConfig"={"Arn"="[[DLQ_ARN=\"your DLQ ARN\"]]"}

What does this command do?

The put-targets command updates the existing rule target to add the DeadLetterConfig. Any event that EventBridge cannot deliver to the fulfillment Lambda (due to permissions errors, throttling, or other failures) will now land in your SQS DLQ with metadata explaining why.

  • Verify the DLQ is attached by running:
aws events list-targets-by-rule \
  --rule order-placed-rule \
  --event-bus-name order-processing-bus

You should see your target with a DeadLetterConfig object containing the DLQ ARN.

Command returned an error about the target?

The Id field in put-targets must match the existing target ID if you want to update it. Run list-targets-by-rule first to see the current target ID, then use that exact ID in your put-targets command.

Help me attach a DLQ to my EventBridge rule target.

Before you move on, take stock of what you have now. You have structured JSON logging on both Lambdas, API Gateway access logs that reveal integration errors, EventBridge bus logging at ERROR level, and an SQS DLQ ready to catch delivery failures.

What do you think will happen when you re-inject the same 3 failures from the previous step? With these tools in place, each failure should now surface through a different observability channel. That is exactly what you will prove in the next step.

Re-diagnose with the Right Tools

In the previous step, you built the complete production observability toolkit. Structured logging, API Gateway access logs, EventBridge bus logging, and the Dead Letter Queue are all in place.

Now it is time to prove their value. You will re-run the same failure scenarios that frustrated you earlier and watch how each observability channel pinpoints the root cause in seconds instead of minutes.

In this step, get ready to:
  • Diagnose the timeout failure using API Gateway access logs and CloudWatch Logs Insights queries.
  • Diagnose the permission error and concurrency throttle using DLQ message attributes.
  • Fix all failures and confirm the pipeline is healthy with a cold start and duration query.
Diagnose the timeout with access logs and Logs Insights

The timeout failure is still active from Step 3. The order-processor Lambda has a 3-second timeout with a 5-second sleep in the handler. Previously, all you saw was a generic 500 error. Now you have access logs that reveal exactly what happened.

  • Send a test order to trigger the timeout by running this command:
curl -X POST [[API_URL="your API Gateway endpoint URL"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-TIMEOUT-1", "customer_name": "Test User", "items": [{"name": "Widget", "quantity": 2}], "total": 29.99}'

After about 3 seconds, you will see the familiar generic error response with no useful detail.

  • Open the CloudWatch console and navigate to Logs then Log groups.
  • Select the /aws/apigateway/order-processing-api log group.
  • Click on the most recent log stream and find the entry for your request.

The access log entry now contains the integrationErrorMessage field showing the Lambda timeout error. Instead of a generic 500, you can see exactly why the gateway returned an error.

What does the access log tell you?

The $context.integrationErrorMessage variable captures the actual error from the Lambda integration. For a timeout, it shows that the function exceeded the 3-second limit.

Without this field, all you get is a 500 status code and no way to tell whether the Lambda crashed, timed out, or returned a code error.

Now run a Logs Insights query to quantify timeout occurrences across the order-processor log group.

  • In the CloudWatch console, navigate to Logs then Logs Insights.
  • Select the /aws/lambda/order-processor log group from the dropdown.
  • Paste the following query and click Run query:
filter @message like /Task timed out/ | stats count() by bin(5m)

What does this query do?

This filters all log entries for the phrase "Task timed out" and groups the count into 5-minute time buckets. You will see exactly how many timeouts occurred and when they happened.

  • Now run a latency percentiles query to see the duration profile. Replace the previous query with:
filter @type = "REPORT" | stats avg(@duration), max(@duration), percentile(@duration, 95) by bin(5m)

The REPORT lines are Lambda's built-in performance summaries. This query shows average duration, max duration, and the 95th percentile. You will see values near or at 3000ms (the timeout ceiling), confirming the function is being killed before completion.

Save queries for production dashboards

In a real production environment, you would save these Logs Insights queries as dashboard widgets. A spike in the timeout count or a sudden jump in p95 latency is an immediate signal to investigate.

Diagnose delivery failures with DLQ metadata

The DLQ is where EventBridge sends events it cannot deliver. To test it, you need the processor to succeed (so events reach EventBridge) while the fulfillment Lambda fails to receive them.

This means fixing the timeout first, then removing the EventBridge permission so both the permission error and the existing throttle are active simultaneously.

  • Fix the timeout by updating the order-processor timeout back to 10 seconds. In the Lambda console, select order-processor, click Configuration then General configuration then Edit, and set Timeout to 10 seconds. Click Save.
  • Remove the time.sleep(5) line from the order-processor code. In the Lambda console Code tab, delete the sleep call and click Deploy.
  • Verify the timeout fix by sending a test order:
curl -X POST [[API_URL="your API Gateway endpoint URL"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-FIX-1", "customer_name": "Test User", "items": [{"name": "Widget", "quantity": 1}], "total": 9.99}'

You should now see a 200 response with an event_id. The processor is working again.

  • Re-activate the permission error by removing the EventBridge invoke permission from the fulfillment Lambda. Run:
aws lambda remove-permission \
  --function-name order-fulfillment \
  --statement-id AllowEventBridgeInvoke

Now both the permission error and the concurrency throttle (reserved concurrency = 0) are active on the fulfillment Lambda. Events will be accepted by the processor but fail to reach the fulfillment function through two different failure modes.

  • Send several test orders to generate DLQ messages by running:
curl -X POST [[API_URL="your API Gateway endpoint URL"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-DLQ-1", "customer_name": "Alice", "items": [{"name": "Gadget", "quantity": 1}], "total": 49.99}'
curl -X POST [[API_URL="your API Gateway endpoint URL"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-DLQ-2", "customer_name": "Bob", "items": [{"name": "Gizmo", "quantity": 3}], "total": 89.99}'

Both requests should return a 200 with an event_id. The processor is working. The events are reaching EventBridge. But the fulfillment Lambda will never fire.

Why wait before checking the DLQ?

EventBridge delivers the permission-error event to the DLQ immediately (zero retries). The throttled event gets retried before landing in the DLQ. Give it 1-2 minutes for the throttle failure to exhaust some retries and arrive in the queue.

You can check the order-processing-dlq queue in the SQS console. Look at Messages available to see when messages arrive.

  • After 1-2 minutes, read the DLQ messages with their attributes by running:
aws sqs receive-message \
  --queue-url [[DLQ_URL="your order-processing-dlq queue URL"]] \
  --attribute-names All \
  --message-attribute-names All \
  --max-number-of-messages 10

Examine the message attributes in the response. You will see metadata fields from EventBridge that tell you exactly why each delivery failed.

How to distinguish failure types from DLQ metadata

Each DLQ message includes attributes like ERROR_CODE, ERROR_MESSAGE, TARGET_ARN, and RETRY_ATTEMPTS. The critical difference between the two failure types:

  • Permission error: RETRY_ATTEMPTS = 0. EventBridge sent the event directly to the DLQ without retrying because the target denied invocation.
  • Concurrency throttle: RETRY_ATTEMPTS > 0. EventBridge retried delivery multiple times before giving up and sending to the DLQ.

This single metadata field lets you distinguish two failures that look identical from the outside (fulfillment never fires) but have completely different root causes.

  • Confirm the throttle by checking CloudWatch metrics. In the CloudWatch console, navigate to Metrics then All metrics then Lambda then By Function Name.
  • Select the Throttles metric for order-fulfillment. You will see a spike corresponding to EventBridge delivery attempts.

Not seeing messages in the DLQ?

If the DLQ is empty, the throttled event may still be in retry. EventBridge retries for up to 24 hours by default. Wait a few more minutes and check again. The permission-error message should appear immediately.

Verify the queue URL is correct. You can find it in the SQS console under order-processing-dlq details.

Still stuck? Help me figure out why my EventBridge DLQ is not receiving messages.

Fix all failures and confirm the pipeline is healthy

You have diagnosed all three failures using the observability toolkit. Time to fix everything and prove the pipeline works end-to-end.

  • Restore the EventBridge invoke permission on the fulfillment Lambda by running:
aws lambda add-permission \
  --function-name order-fulfillment \
  --statement-id AllowEventBridgeInvoke \
  --action lambda:InvokeFunction \
  --principal events.amazonaws.com \
  --source-arn [[RULE_ARN="your order-placed-rule ARN"]]
  • Remove the reserved concurrency throttle by running:
aws lambda delete-function-concurrency \
  --function-name order-fulfillment

What does delete-function-concurrency do?

This removes the reserved concurrency setting entirely. The function returns to using the account's shared unreserved concurrency pool, which means it can scale normally again.

Before you send a test order, predict what you expect to see. Should the order flow through the entire pipeline successfully now?

  • Send a test order to verify the full pipeline by running:
curl -X POST [[API_URL="your API Gateway endpoint URL"]]/orders \
  -H "Content-Type: application/json" \
  -d '{"order_id": "ORD-HEALTHY-1", "customer_name": "Charlie", "items": [{"name": "Pro Widget", "quantity": 1}], "total": 59.99}'

You should see a 200 response with Order placed successfully and an event_id.

  • Confirm the fulfillment Lambda executed by checking the /aws/lambda/order-fulfillment log group in CloudWatch. You should see a structured JSON entry with "message": "Order fulfilled successfully" for your order.
  • Run the cold start and duration health check query. In Logs Insights, select both /aws/lambda/order-processor and /aws/lambda/order-fulfillment log groups, then run:
filter @type = "REPORT" | stats sum(strcontains(@message, "Init Duration"))/count(*) * 100 as coldStartPct, avg(@duration) by bin(5m)

What does this query tell you?

This calculates the percentage of invocations that were cold starts and the average duration per 5-minute bucket. A healthy pipeline shows average durations well below the timeout limit.

If you see 100% cold starts, that is expected for functions that have not been invoked recently. The important signal is that average duration is healthy (well under 10 seconds).

Your pipeline is fully restored. Both Lambdas execute successfully, structured logs are flowing, and you have the queries to monitor health going forward.

Fulfillment Lambda still not executing?

If the fulfillment Lambda is not being invoked, verify the permission was added correctly by running:

aws lambda get-policy --function-name order-fulfillment

You should see a statement with AllowEventBridgeInvoke and principal events.amazonaws.com. If it is missing, re-run the add-permission command.

Also confirm the reserved concurrency was removed:

aws lambda get-function-concurrency --function-name order-fulfillment

If concurrency is unreserved, you will get an empty JSON response or no ReservedConcurrentExecutions field.

Need more help? Help me figure out why my fulfillment Lambda is not being invoked after restoring permissions.

You just proved the full power of production observability. The same failures that took minutes of frustrated log scrolling in Step 3 now take seconds to pinpoint with API Gateway access logs, Logs Insights queries, and DLQ metadata. These patterns apply to any event-driven architecture in production.

Secret mission

Infrastructure as Code with Terraform

Console-created resources are fragile and unreproducible. In this secret mission, you'll convert your entire serverless pipeline into declarative Terraform code using the hashicorp/aws provider. One command rebuilds everything from scratch in any AWS account.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. All resources in this project fall within the AWS Free Tier, so there are no ongoing costs while they sit idle.

Resources you used:

  • 2 Lambda functions (order-processor, order-fulfillment).
  • 1 API Gateway HTTP API (order-processing-api) with access logging.
  • 1 EventBridge custom event bus (order-processing-bus) with bus logging and 1 rule.
  • 1 SQS queue (order-processing-dlq).
  • 4 CloudWatch Log groups (/aws/lambda/order-processor, /aws/lambda/order-fulfillment, /aws/apigateway/order-processing-api, /aws/events/order-processing-bus).
  • 2 IAM roles (order-processor-role, order-fulfillment-role).
  • Local project directory with main.tf, order_processor.py, order_fulfillment.py, and Terraform state files.

Keep everything running

No action needed. Choose this if you are still actively experimenting with the pipeline or plan to revisit the troubleshooting exercises.

  • All resources are serverless and only incur charges when invoked. Leaving them idle costs nothing within the Free Tier.
  • Your Terraform state file remains in your project directory so you can manage the infrastructure later.

Pause - I'll come back to this later

Serverless resources (Lambda, API Gateway HTTP API, EventBridge, SQS) only cost money when they process requests. There is nothing to pause.

  • If you want to prevent accidental invocations, you can delete the API Gateway HTTP API route so no public endpoint exists. The remaining resources stay dormant at zero cost.
  • Keep your local project directory intact so you can redeploy with terraform apply when you return.

Delete - I don't want to use this again

Remove all project resources and start fresh. If you completed the Secret Mission and have a working Terraform configuration, this is the fastest path.

Option A: Terraform (recommended if you completed the Secret Mission)
  • Navigate to your project directory in the terminal and run:
terraform destroy
  • Terraform will show you a plan of everything it will delete. Type yes when prompted to confirm.
  • This removes the Lambda functions, API Gateway, EventBridge bus and rule, SQS queue, the Terraform-managed CloudWatch Log group, and IAM roles.

What Terraform does not delete

The auto-created Lambda log groups and the EventBridge bus logging log group were not created by Terraform. You need to delete these manually in the next step.

Delete remaining CloudWatch Log groups
  • Open the CloudWatch console and navigate to Logs then Log groups.
  • Select the following log groups: /aws/lambda/order-processor, /aws/lambda/order-fulfillment, and /aws/events/order-processing-bus.
  • Choose Actions, then Delete log group(s).
  • Choose Delete to confirm.
Option B: AWS Console (if you did not use Terraform)

Delete each resource manually in this order:

  • Open the API Gateway console. Select order-processing-api. Choose Actions, then Delete. Confirm.
  • Open the EventBridge console. Go to Event buses. Select order-processing-bus. Delete the rule order-placed-rule first, then delete the bus.
  • Open the Lambda console. Select order-processor. Choose Actions, Delete. Type confirm and choose Delete. Repeat for order-fulfillment.
  • Open the SQS console. Select order-processing-dlq. Choose Delete. Type confirm to confirm.
  • Open the CloudWatch console. Navigate to Logs then Log groups. Select all four log groups. Choose Actions, then Delete log group(s). Choose Delete to confirm.
  • Open the IAM console. Navigate to Roles. Select order-processor-role. Choose Delete. Enter the role name to confirm, then choose Delete. Repeat for order-fulfillment-role.
Delete local project files
  • Remove your project directory by running:

macOS

cd ~/Desktop
rm -rf order-processing-pipeline

Windows

cd ~\Desktop
rmdir /s /q order-processing-pipeline

That's a Wrap!

That's a Wrap!

Nice work. You just built, broke, and debugged a production-grade serverless order pipeline using the same observability tools that real engineering teams rely on every day.

You've learned how to:

  • Build a complete event-driven pipeline (API Gateway HTTP API to Lambda to EventBridge to Lambda) and inject 3 production-realistic failures: timeout, permission error, and concurrency throttling.
  • Configure a full observability toolkit including API Gateway access logs with $context.integrationErrorMessage, structured JSON logging in Lambda, EventBridge bus logging, and a Dead Letter Queue that distinguishes failure types by DLQ metadata (RETRY_ATTEMPTS = 0 for permission errors vs. > 0 for throttling).
  • Write CloudWatch Logs Insights queries to detect timeouts, measure latency percentiles, and calculate cold start percentages across your Lambda functions in seconds.
  • Secret Mission: Defined the entire serverless pipeline as Terraform infrastructure-as-code using the hashicorp/aws provider.

Ready to quiz yourself?