Event-Driven Image Processing on AWS

Build a serverless pipeline that auto-processes images using 8 AWS services.

Introduction

30 Second Summary

Every time you upload a photo to social media, a chain of automated processes fires behind the scenes to resize, catalog, and distribute your image across servers. This is event-driven architecture in action.

In this project, you will build a serverless image processing pipeline on AWS that automatically resizes images when they're uploaded to the cloud. You'll connect 8 AWS services into a production-style architecture that mirrors how real companies handle media at scale.

What You'll Build

An event-driven pipeline where uploading an image to S3 triggers automatic resizing via Lambda, with SQS buffering events, DynamoDB storing metadata, SNS sending email notifications, and API Gateway exposing a query endpoint.

By the end of this project, you'll have:

  • A working serverless pipeline that automatically resizes images uploaded to S3 and stores processing metadata in DynamoDB.
  • Hands-on experience wiring together 8 AWS services (S3, Lambda, SQS, DynamoDB, SNS, CloudWatch, API Gateway, and IAM) into a cohesive architecture.
  • A live REST API endpoint that returns JSON metadata about all processed images.
  • Secret Mission: Add image dimension extraction and a DELETE endpoint to remove processed images.

Are there any prerequisites?

You'll need an AWS account with the AWS CLI already configured (credentials set up and working). You'll also need Python 3 installed on your machine and basic familiarity with Python syntax.

Create IAM Roles

Every AWS service in your pipeline needs permission to talk to the others. Without proper permissions, your Lambda function could not read from S3, write to DynamoDB, or send notifications through SNS.

AWS uses IAM roles to control what each service can do. In this step, you will create two IAM roles: one for the image processor Lambda (which needs broad access to multiple services) and one for the query API Lambda (which only needs to read from DynamoDB).

In this step, get ready to:
  • Create the image processor Lambda execution role with permissions for S3, SQS, DynamoDB, SNS, and CloudWatch Logs.
  • Create the API Lambda execution role with permissions for DynamoDB and CloudWatch Logs.
Create the image processor Lambda execution role

The image processor Lambda needs to access five different AWS services. Before creating the role, you need a trust policy that tells AWS which service is allowed to assume this role.

  • In your terminal, make sure you are inside the image-pipeline directory from earlier.
  • Create the trust policy file by running this command:
cat > trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

What is a trust policy?

A trust policy defines WHO can assume a role. In this case, you are saying "only the Lambda service is allowed to use this role."

This is separate from the permissions policy, which defines WHAT the role can do once assumed.

  • Confirm the file was created by running:
ls trust-policy.json

You should see trust-policy.json listed in the output.

  • Create the image-processor-role by running this command:
aws iam create-role \
  --role-name image-processor-role \
  --assume-role-policy-document file://trust-policy.json

You should see JSON output containing the role details, including an Arn field.

Seeing an error about the trust policy file?

Make sure you are running the command from the same directory where trust-policy.json was created. The file:// prefix tells the CLI to read a local file.

If you see a "MalformedPolicyDocument" error, double check that the JSON in your trust policy file is valid. Run cat trust-policy.json to inspect it.

Help me troubleshoot this error

Now you need to define what this role is allowed to do. The image processor needs permissions for S3, SQS, DynamoDB, SNS, and CloudWatch Logs.

  • Create the permissions policy file by running this command:
cat > processor-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "dynamodb:PutItem",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
EOF

What does each permission do?

  • s3:GetObject and s3:PutObject let the function download the original image and upload the resized version.
  • sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:GetQueueAttributes let Lambda poll messages from the SQS queue.
  • dynamodb:PutItem lets the function write metadata about each processed image.
  • sns:Publish lets the function send you an email notification when processing completes.
  • The logs: permissions let Lambda write logs to CloudWatch for debugging.
  • Attach the permissions policy to the role by running this command:
aws iam put-role-policy \
  --role-name image-processor-role \
  --policy-name image-processor-policy \
  --policy-document file://processor-policy.json

This command produces no output on success. No news is good news.

  • Store the role ARN as an environment variable by running these commands:
PROCESSOR_ROLE_ARN=$(aws iam get-role --role-name image-processor-role --query 'Role.Arn' --output text)
echo $PROCESSOR_ROLE_ARN

You should see an ARN like arn:aws:iam::123456789012:role/image-processor-role printed to your terminal (with your account ID instead of 123456789012).

Seeing an empty line or error?

If echo $PROCESSOR_ROLE_ARN prints nothing, the aws iam get-role command may have failed silently. Run aws iam get-role --role-name image-processor-role by itself to check for errors. The most common cause is a typo in the role name.

Help me fix my empty PROCESSOR_ROLE_ARN

Create the API Lambda execution role

The query API Lambda only needs to read from DynamoDB and write logs. It uses the same trust policy you already created, so you can reuse that file.

  • Create the api-lambda-role by running this command:
aws iam create-role \
  --role-name api-lambda-role \
  --assume-role-policy-document file://trust-policy.json

You should see JSON output with the new role's details.

  • Create the API permissions policy file by running this command:
cat > api-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:Scan",
        "dynamodb:GetItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
EOF

Why does this role include DeleteItem?

The Secret Mission at the end of this project adds a DELETE endpoint to the API. Including dynamodb:DeleteItem now means you will not need to update this role later.

  • Attach the permissions policy to the role by running this command:
aws iam put-role-policy \
  --role-name api-lambda-role \
  --policy-name api-lambda-policy \
  --policy-document file://api-policy.json

This command produces no output on success.

  • Store the API role ARN as an environment variable by running these commands:
API_ROLE_ARN=$(aws iam get-role --role-name api-lambda-role --query 'Role.Arn' --output text)
echo $API_ROLE_ARN

You should see an ARN like arn:aws:iam::123456789012:role/api-lambda-role printed to your terminal.

Role not found?

If you see a "NoSuchEntity" error, verify the role name matches exactly: api-lambda-role. Role names are case-sensitive.

Help me fix this NoSuchEntity error

Both IAM roles are now ready. Next up, you will create the AWS infrastructure that these roles protect: S3 buckets, an SQS queue, a DynamoDB table, and an SNS topic.

Create AWS Infrastructure

With your IAM roles ready, it's time to create the actual resources your pipeline will use.

In this step, you'll provision the storage, messaging, and notification services that form the backbone of your event-driven architecture. Each service plays a specific role in the chain that carries an image from upload to processed output.

In this step, get ready to:
  • Create S3 buckets and a DynamoDB table for storage.
  • Set up SNS notifications so you get an email when processing completes.
  • Create an SQS queue and wire it to S3 event notifications.
Create the S3 buckets and DynamoDB table

Your pipeline needs two S3 buckets: one where you upload raw images, and another where the processed (resized) images land. You also need a DynamoDB table to store metadata about each processed image.

  • Set environment variables for your bucket names and table name by running these commands:
# Define resource names using your account ID for uniqueness
export SOURCE_BUCKET=image-pipeline-source-$ACCOUNT_ID
export DEST_BUCKET=image-pipeline-processed-$ACCOUNT_ID
export TABLE_NAME=image-metadata

Why include your account ID in bucket names?

S3 bucket names must be globally unique across all AWS accounts. Appending your account ID guarantees no naming collision with other learners.

  • Create the source bucket (where you'll upload images) by running:
aws s3api create-bucket \
    --bucket $SOURCE_BUCKET \
    --region $REGION \
    --create-bucket-configuration LocationConstraint=$REGION

What does this command do?

The --create-bucket-configuration LocationConstraint flag tells AWS which region to place the bucket in. This is required for any region other than us-east-1. If your region is us-east-1, remove that flag from the command.

  • Create the processed bucket (where resized images will land) by running:
aws s3api create-bucket \
    --bucket $DEST_BUCKET \
    --region $REGION \
    --create-bucket-configuration LocationConstraint=$REGION
  • Create the DynamoDB table to store image metadata by running:
aws dynamodb create-table \
    --table-name $TABLE_NAME \
    --attribute-definitions AttributeName=image_id,AttributeType=S \
    --key-schema AttributeName=image_id,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST

What does this command do?

This creates a table called image-metadata with a single partition key image_id of type String. The PAY_PER_REQUEST billing mode means you only pay when you read or write data. There is no provisioned capacity to manage.

  • Verify the resources exist by running:
aws s3 ls | grep image-pipeline
aws dynamodb describe-table --table-name $TABLE_NAME --query 'Table.TableStatus'

You should see both bucket names listed and a table status of ACTIVE.

Not seeing ACTIVE status?

DynamoDB tables take a few seconds to become active. Wait 5-10 seconds and run the describe-table command again. If you see an error about the table not existing, double-check that your TABLE_NAME environment variable is set correctly.

Help me troubleshoot my DynamoDB table creation

Create the SNS topic and subscribe your email

Amazon SNS (Simple Notification Service) sends you an email every time your pipeline finishes processing an image. You'll create a topic and subscribe your email address to it.

  • Create the SNS topic and store its ARN by running:
# Create the topic and capture the ARN
export SNS_TOPIC_ARN=$(aws sns create-topic \
    --name image-processing-notifications \
    --query 'TopicArn' \
    --output text)

echo $SNS_TOPIC_ARN

You should see an ARN printed in the format arn:aws:sns:<region>:<account-id>:image-processing-notifications.

  • Subscribe your email address to the topic by running:
aws sns subscribe \
    --topic-arn $SNS_TOPIC_ARN \
    --protocol email \
    --notification-endpoint [[YOUR_EMAIL="your-email@example.com"]]

What happens next?

AWS sends a confirmation email to the address you provided. The subscription only becomes active after you click the confirmation link in that email. Check your inbox (and spam folder) for an email from AWS Notifications.

  • Open your email inbox and click the Confirm subscription link in the AWS email.
  • You should see a confirmation page that says Subscription confirmed in your browser.

Don't see the confirmation email?

Check your spam or junk folder. The email comes from AWS Notifications. If you still don't see it after a few minutes, re-run the subscribe command with the correct email address.

Help me troubleshoot my SNS email subscription

Create the SQS queue and configure S3 event notifications

The SQS (Simple Queue Service) queue sits between S3 and your Lambda function. When an image lands in the source bucket, S3 sends a message to the queue. Lambda then picks messages off the queue at its own pace.

Why use SQS between S3 and Lambda?

You could trigger Lambda directly from S3. However, adding SQS in the middle provides built-in retry handling, dead-letter queue support, and the ability to batch-process messages. This is the production pattern most AWS architects use for reliable event-driven systems.

  • Create the SQS queue and store its URL and ARN by running:
# Create the queue
export SQS_QUEUE_URL=$(aws sqs create-queue \
    --queue-name image-processing-queue \
    --query 'QueueUrl' \
    --output text)

# Get the queue ARN
export SQS_QUEUE_ARN=$(aws sqs get-queue-attributes \
    --queue-url $SQS_QUEUE_URL \
    --attribute-names QueueArn \
    --query 'Attributes.QueueArn' \
    --output text)

echo $SQS_QUEUE_URL
echo $SQS_QUEUE_ARN

What does this command do?

The first command creates the queue and captures its URL (needed for sending and receiving messages). The second command retrieves the queue's ARN (needed for IAM policies and S3 configuration).

Next, you need to allow S3 to send messages to this queue. Without a resource policy on the queue, S3 will be denied access.

  • Create the queue policy file by running:
# Write the queue policy to a file
cat > queue-policy.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "SQS:SendMessage",
            "Resource": "$SQS_QUEUE_ARN",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:s3:::$SOURCE_BUCKET"
                }
            }
        }
    ]
}
EOF
  • Apply the policy to the queue by running:
# Set the queue policy using the file contents
aws sqs set-queue-attributes \
    --queue-url $SQS_QUEUE_URL \
    --attributes '{"Policy":"'$(cat queue-policy.json | tr -d '\n' | sed 's/"/\\"/g')'"}'

What does this policy do?

This resource policy grants the s3.amazonaws.com service permission to call SQS:SendMessage on your queue. The condition restricts it so only messages originating from your specific source bucket are allowed through.

Finally, you need to tell S3 to actually send events to the queue whenever an object is created in the source bucket.

  • Create the notification configuration file by running:
# Create the notification configuration file
cat > notification.json << EOF
{
    "QueueConfigurations": [
        {
            "QueueArn": "$SQS_QUEUE_ARN",
            "Events": ["s3:ObjectCreated:*"]
        }
    ]
}
EOF
  • Apply the notification configuration to the source bucket by running:
aws s3api put-bucket-notification-configuration \
    --bucket $SOURCE_BUCKET \
    --notification-configuration file://notification.json

What does this do?

This tells S3: whenever any object is created in this bucket, send a notification to the SQS queue. The s3:ObjectCreated:* event covers uploads via PUT, POST, COPY, and multipart upload.

  • Verify the notification is configured by running:
aws s3api get-bucket-notification-configuration --bucket $SOURCE_BUCKET

You should see a JSON response containing your QueueConfigurations with the SQS queue ARN and the s3:ObjectCreated:* event.

Seeing an Unable to validate error?

This means S3 cannot confirm it has permission to write to the SQS queue. Double-check that you ran the set-queue-attributes command successfully. Verify that the SOURCE_BUCKET and SQS_QUEUE_ARN variables are set correctly by echoing them.

Help me fix my S3 to SQS notification configuration

Your infrastructure is in place. Next up, you'll write the Lambda function that actually processes images when they appear in the queue.

Build and Deploy the Image Processor

Your infrastructure is ready and waiting. Now it's time to write the code that brings the pipeline to life.

The image processor is the core Lambda function in your pipeline. It reacts to upload events from SQS, downloads the original image from S3, resizes it using Pillow, stores the result in your processed bucket, records metadata in DynamoDB, and sends you a notification via SNS.

In this step, get ready to:
  • Write the image processor Lambda function in Python.
  • Package the function with Pillow for Lambda's Linux environment.
  • Connect the SQS queue to the Lambda function with an event source mapping.
Write the image processor Lambda function

The processor function needs to handle a specific event structure. When S3 sends an upload event to SQS, the event arrives as a nested JSON payload. Your function parses that payload, processes the image, and then updates multiple AWS services.

  • Create a new file called lambda_function.py in your image-pipeline directory. Add the following imports and client setup:
import boto3
import os
import json
import uuid
from datetime import datetime, timezone
from urllib.parse import unquote_plus
from PIL import Image

# Initialize AWS service clients
s3_client = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
sns_client = boto3.client('sns')

# Read configuration from Lambda environment variables
DEST_BUCKET = os.environ['DEST_BUCKET']
TABLE_NAME = os.environ['TABLE_NAME']
SNS_TOPIC_ARN = os.environ['SNS_TOPIC_ARN']

What does this code do?

  • boto3 is the AWS SDK for Python. You create separate clients for S3, DynamoDB, and SNS so the function can interact with each service.
  • PIL is Pillow's import name. You'll use it to open and resize images.
  • os.environ reads configuration values that you'll set when deploying the Lambda. This keeps your code flexible across environments.

You won't see a change yet. The next chunk adds the main handler function that uses these imports.

  • Below the environment variables in lambda_function.py, add the main handler function:
# Entry point that Lambda invokes when an SQS message arrives
def lambda_handler(event, context):
    for record in event['Records']:
        # Parse the S3 event from the SQS message body
        body = json.loads(record['body'])
        for s3_record in body['Records']:
            bucket = s3_record['s3']['bucket']['name']
            key = unquote_plus(s3_record['s3']['object']['key'])
            size = s3_record['s3']['object']['size']

            # Create unique temp file paths for download and upload
            tmp_key = key.replace('/', '')
            download_path = f'/tmp/{uuid.uuid4()}{tmp_key}'
            upload_path = f'/tmp/resized-{tmp_key}'

            # Download the original image from S3
            s3_client.download_file(bucket, key, download_path)

            # Resize the image to fit within 400x400 pixels
            with Image.open(download_path) as image:
                image.thumbnail((400, 400))
                image.save(upload_path)

            # Upload the resized image to the processed bucket
            dest_key = f'resized-{key}'
            s3_client.upload_file(upload_path, DEST_BUCKET, dest_key)

            # Get the file size of the processed image
            processed_obj = s3_client.head_object(Bucket=DEST_BUCKET, Key=dest_key)
            processed_size = processed_obj['ContentLength']

            # Store processing metadata in DynamoDB
            table = dynamodb.Table(TABLE_NAME)
            timestamp = datetime.now(timezone.utc).isoformat()
            table.put_item(Item={
                'image_id': str(uuid.uuid4()),
                'original_key': key,
                'processed_key': dest_key,
                'original_size': size,
                'processed_size': processed_size,
                'processed_at': timestamp,
                'status': 'completed'
            })

            # Send a completion notification via SNS
            sns_client.publish(
                TopicArn=SNS_TOPIC_ARN,
                Subject='Image Processing Complete',
                Message=f'Image "{key}" has been processed.\n\nOriginal size: {size} bytes\nProcessed size: {processed_size} bytes\nStored as: {dest_key}'
            )

    return {'statusCode': 200, 'body': 'Processing complete'}

What does this function do?

The handler processes each SQS record in a loop. Each record contains an S3 event with the bucket name and object key of the uploaded image.

  • unquote_plus decodes URL-encoded characters in the S3 key (spaces become + in S3 event payloads).
  • image.thumbnail((400, 400)) resizes the image to fit within a 400x400 pixel bounding box while preserving the aspect ratio.
  • head_object retrieves the processed file's metadata (including size) without downloading the file again.
  • The function records everything in DynamoDB and then publishes a human-readable notification to SNS.
  • Save lambda_function.py.
  • Confirm the file exists by running:
ls lambda_function.py

You should see lambda_function.py listed in the output.

Don't see the file?

  • Make sure you're in the image-pipeline directory. Run pwd to check your current location.
  • If you used a text editor, confirm you saved the file with the exact name lambda_function.py (no capitals, no extra extensions).

Help me figure out why my lambda_function.py file isn't showing up.

✔️ Awesome, I've got everything!

Great. Make sure you've saved lambda_function.py before moving on to packaging.

ⓧ I'd like to double check the full code

import boto3
import os
import json
import uuid
from datetime import datetime, timezone
from urllib.parse import unquote_plus
from PIL import Image

s3_client = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
sns_client = boto3.client('sns')

DEST_BUCKET = os.environ['DEST_BUCKET']
TABLE_NAME = os.environ['TABLE_NAME']
SNS_TOPIC_ARN = os.environ['SNS_TOPIC_ARN']


def lambda_handler(event, context):
    for record in event['Records']:
        body = json.loads(record['body'])
        for s3_record in body['Records']:
            bucket = s3_record['s3']['bucket']['name']
            key = unquote_plus(s3_record['s3']['object']['key'])
            size = s3_record['s3']['object']['size']

            tmp_key = key.replace('/', '')
            download_path = f'/tmp/{uuid.uuid4()}{tmp_key}'
            upload_path = f'/tmp/resized-{tmp_key}'

            s3_client.download_file(bucket, key, download_path)

            with Image.open(download_path) as image:
                image.thumbnail((400, 400))
                image.save(upload_path)

            dest_key = f'resized-{key}'
            s3_client.upload_file(upload_path, DEST_BUCKET, dest_key)

            processed_obj = s3_client.head_object(Bucket=DEST_BUCKET, Key=dest_key)
            processed_size = processed_obj['ContentLength']

            table = dynamodb.Table(TABLE_NAME)
            timestamp = datetime.now(timezone.utc).isoformat()
            table.put_item(Item={
                'image_id': str(uuid.uuid4()),
                'original_key': key,
                'processed_key': dest_key,
                'original_size': size,
                'processed_size': processed_size,
                'processed_at': timestamp,
                'status': 'completed'
            })

            sns_client.publish(
                TopicArn=SNS_TOPIC_ARN,
                Subject='Image Processing Complete',
                Message=f'Image "{key}" has been processed.\n\nOriginal size: {size} bytes\nProcessed size: {processed_size} bytes\nStored as: {dest_key}'
            )

    return {'statusCode': 200, 'body': 'Processing complete'}
Create the deployment package and deploy the Lambda function

Lambda runs on Amazon Linux, not macOS. You need to install Pillow's pre-compiled Linux binaries into your package so it works in Lambda's execution environment.

  • Install Pillow with the correct platform binaries by running:
pip install --platform manylinux2014_x86_64 --target=package --implementation cp --python-version 3.13 --only-binary=:all: --upgrade pillow boto3

Why these flags?

  • --platform manylinux2014_x86_64 downloads binaries compiled for Lambda's Linux environment instead of your local macOS.
  • --target=package installs the libraries into a package/ directory instead of your system Python.
  • --only-binary=:all: ensures pip uses pre-compiled wheels rather than trying to build from source.
  • --python-version 3.13 matches Lambda's python3.13 runtime so the wheels are compatible.
  • Confirm the Pillow library was installed by running:
ls package/PIL

You should see a list of .py files and .so compiled libraries inside the PIL directory.

Seeing 'No matching distribution found'?

  • Make sure you have pip version 21.3 or later. Run pip --version to check. Upgrade with pip install --upgrade pip if needed.
  • Check that you typed manylinux2014_x86_64 exactly (underscores, not dashes).

Help me troubleshoot my pip install command for Lambda packaging.

  • Zip the installed packages into a deployment archive by running:
cd package
zip -r ../processor.zip .
cd ..
  • Add your Lambda function code to the zip by running:
zip processor.zip lambda_function.py
  • Confirm the deployment package was created by running:
ls -lh processor.zip

You should see processor.zip with a size of approximately 10-20 MB (Pillow's compiled libraries make up most of the size).

  • Deploy the Lambda function by running:
aws lambda create-function \
  --function-name image-processor \
  --zip-file fileb://processor.zip \
  --handler lambda_function.lambda_handler \
  --runtime python3.13 \
  --timeout 10 \
  --memory-size 1024 \
  --role $PROCESSOR_ROLE_ARN \
  --environment "Variables={DEST_BUCKET=$DEST_BUCKET,TABLE_NAME=$TABLE_NAME,SNS_TOPIC_ARN=$SNS_TOPIC_ARN}"

What does this command do?

  • --handler lambda_function.lambda_handler tells Lambda to look in lambda_function.py and call the lambda_handler function.
  • --runtime python3.13 sets the execution environment to Python 3.13 on Amazon Linux 2023.
  • --timeout 10 gives the function 10 seconds to process each image. Image resizing is fast.
  • --memory-size 1024 allocates 1 GB of memory. Pillow needs enough memory to hold images during resizing.
  • --environment passes your bucket name, table name, and SNS topic ARN as environment variables the function reads at runtime.

You should see JSON output containing a FunctionArn field. This confirms your Lambda function was created successfully.

  • Save the function ARN as an environment variable for later use by running:
PROCESSOR_FUNCTION_ARN=$(aws lambda get-function --function-name image-processor --query 'Configuration.FunctionArn' --output text)
echo $PROCESSOR_FUNCTION_ARN

You should see an ARN like arn:aws:lambda:us-east-1:123456789012:function:image-processor printed in your terminal (with your own region and account ID).

Seeing 'An error occurred (InvalidParameterValueException)'?

  • If the error mentions the role, wait 10 seconds and try again. IAM roles can take a moment to propagate after creation.
  • Check that $PROCESSOR_ROLE_ARN is set by running echo $PROCESSOR_ROLE_ARN. If it's empty, re-export it from Step 1.
  • If the error mentions the zip file, confirm processor.zip exists in your current directory with ls processor.zip.

Help me fix my Lambda create-function error.

Create the SQS event source mapping

Your Lambda function exists but doesn't know about the SQS queue yet. An event source mapping is the bridge that tells Lambda to poll the queue and invoke your function whenever messages arrive.

  • Connect the SQS queue to your Lambda function by running:
aws lambda create-event-source-mapping \
  --function-name image-processor \
  --batch-size 5 \
  --event-source-arn $SQS_QUEUE_ARN

What is an event source mapping?

An event source mapping is a Lambda resource that reads from a source (like SQS) and invokes your function with batches of messages. Lambda handles the polling, retries, and message deletion automatically.

The --batch-size 5 setting means Lambda delivers up to 5 SQS messages per invocation. Your function's for record in event['Records'] loop handles each one.

You should see JSON output with a UUID field and "State": "Creating" or "State": "Enabled". This confirms the mapping is active.

Seeing an access denied error?

  • Your processor role needs the AWSLambdaSQSQueueExecutionRole managed policy attached. This policy grants permissions for ReceiveMessage, DeleteMessage, and GetQueueAttributes.
  • Verify that $SQS_QUEUE_ARN is set correctly by running echo $SQS_QUEUE_ARN. It should look like arn:aws:sqs:us-east-1:123456789012:image-processing-queue.

Help me fix the event source mapping error.

Your image processor is deployed and wired up. When an image lands in your source bucket, S3 sends a message to SQS, and Lambda picks it up automatically. Next, you'll build the query API so you can retrieve the processing results.

Build and Deploy the Query API

Your image processing pipeline can now resize images and store metadata in DynamoDB. But that metadata is locked inside the database with no way for other applications to access it.

In this step, you'll build a second Lambda function that queries DynamoDB and returns the metadata as JSON. Then you'll expose it to the world through an API Gateway HTTP API, giving you a live REST endpoint anyone can call.

In this step, get ready to:
  • Write a Lambda function that scans DynamoDB and returns image metadata as JSON.
  • Deploy the function and create an API Gateway HTTP API with a single command.
  • Grant permissions and test the live API endpoint.
Write the API Lambda function code

This Lambda function has a different job than the image processor. It reads from DynamoDB and returns a JSON response that API Gateway can forward to any HTTP client.

  • Create a new file called api_function.py in your project directory by running:
touch api_function.py
  • Confirm the file exists by running:
ls api_function.py

You should see api_function.py in the output.

  • Open api_function.py in your text editor and add the imports and helper class:
import boto3
import os
import json
from decimal import Decimal

# Connect to DynamoDB and read the table name from environment
dynamodb = boto3.resource('dynamodb')
TABLE_NAME = os.environ['TABLE_NAME']


# Custom JSON encoder to handle DynamoDB's Decimal type
class DecimalEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Decimal):
            return int(obj)
        return super().default(obj)

What does this code do?

DynamoDB stores numbers as Python Decimal objects, but Python's built-in json.dumps() cannot serialize Decimals.

The DecimalEncoder class solves this by converting any Decimal value to an integer before JSON serialization. Without it, your API would crash the moment it tries to return file sizes from DynamoDB.

  • Below the DecimalEncoder class, add the main handler function:

def lambda_handler(event, context):
    # Query all items from the metadata table
    table = dynamodb.Table(TABLE_NAME)

    response = table.scan()
    items = response['Items']

    # Return JSON response with CORS headers for browser access
    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin': '*'
        },
        'body': json.dumps({
            'count': len(items),
            'images': items
        }, cls=DecimalEncoder)
    }

What does this handler do?

The handler uses table.scan() to read every item from the DynamoDB table. It then wraps the results in the response format API Gateway expects: a dictionary with statusCode, headers, and body fields.

The Access-Control-Allow-Origin header enables CORS so browsers can call this API directly from a web page.

  • Save api_function.py.

Seeing a syntax error?

  • Make sure the DecimalEncoder class and lambda_handler function are separated by two blank lines (Python convention for top-level definitions).
  • Confirm the return statement's dictionary uses single quotes for keys, not backticks or curly quotes.
  • Help me fix a syntax error in my api_function.py

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete api_function.py file:

import boto3
import os
import json
from decimal import Decimal

dynamodb = boto3.resource('dynamodb')
TABLE_NAME = os.environ['TABLE_NAME']


class DecimalEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Decimal):
            return int(obj)
        return super().default(obj)


def lambda_handler(event, context):
    table = dynamodb.Table(TABLE_NAME)

    response = table.scan()
    items = response['Items']

    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin': '*'
        },
        'body': json.dumps({
            'count': len(items),
            'images': items
        }, cls=DecimalEncoder)
    }
Deploy the API Lambda and create an API Gateway HTTP API

Unlike the image processor, this function has no native dependencies like Pillow. The deployment package is just the single Python file zipped up.

  • Create the deployment package by running:
zip api.zip api_function.py
  • You should see output showing the file was added to the archive.
  • Deploy the Lambda function by running:
aws lambda create-function --function-name api-function \
  --zip-file fileb://api.zip \
  --handler api_function.lambda_handler \
  --runtime python3.13 \
  --timeout 10 \
  --memory-size 256 \
  --role $API_ROLE_ARN \
  --environment "Variables={TABLE_NAME=$TABLE_NAME}"

What does this command do?

  • --handler api_function.lambda_handler tells Lambda to call the lambda_handler function inside api_function.py.
  • --role $API_ROLE_ARN attaches the api-lambda-role you created earlier, which only has DynamoDB and CloudWatch permissions.
  • --memory-size 256 allocates 256 MB. This function only reads from DynamoDB and returns JSON, so it needs far less memory than the image processor.
  • You should see JSON output including a FunctionArn field. This confirms the function deployed successfully.
  • Store the function ARN in an environment variable by running:
API_FUNCTION_ARN=$(aws lambda get-function --function-name api-function --query 'Configuration.FunctionArn' --output text)
echo $API_FUNCTION_ARN
  • You should see an ARN like arn:aws:lambda:us-east-1:123456789012:function:api-function printed to the terminal.

Seeing an error during deployment?

  • If you see a "role cannot be assumed" error, wait 10 seconds and try again. IAM role propagation can take a moment.
  • If the function already exists, delete it first with aws lambda delete-function --function-name api-function then rerun the create command.
  • Help me debug my Lambda deployment error

Now you'll create the API Gateway HTTP API. HTTP APIs support "quick create" which sets up the API, integration, route, and stage in a single command.

  • Create the HTTP API and capture its ID by running:
API_ID=$(aws apigatewayv2 create-api --name image-api \
  --protocol-type HTTP \
  --target $API_FUNCTION_ARN \
  --query 'ApiId' --output text)
echo $API_ID

What is quick create?

The --target flag triggers quick create mode. It automatically creates a Lambda integration, a default route, and a $default stage. Without quick create, you'd need 5+ separate CLI commands to wire this up.

  • You should see a short alphanumeric API ID printed to the terminal.
  • Retrieve and store the API endpoint URL by running:
API_ENDPOINT=$(aws apigatewayv2 get-api --api-id $API_ID --query 'ApiEndpoint' --output text)
echo $API_ENDPOINT
  • You should see a URL like https://abc123xyz.execute-api.us-east-1.amazonaws.com. This is your live API endpoint.
Grant API Gateway permission and test the endpoint

API Gateway needs explicit permission to invoke your Lambda function. Without this, requests to your endpoint will fail with an internal server error.

  • Grant API Gateway invoke permission by running:
aws lambda add-permission --function-name api-function \
  --statement-id apigateway-invoke \
  --action lambda:InvokeFunction \
  --principal apigateway.amazonaws.com \
  --source-arn "arn:aws:execute-api:$REGION:$ACCOUNT_ID:$API_ID/*/*"

What does this permission do?

This adds a resource-based policy to your Lambda function. The /*/* at the end of the source ARN means "any HTTP method on any route." This matches the default catch-all route that quick create set up.

  • You should see JSON output confirming the statement was added.
  • Test your API by calling the endpoint:
curl $API_ENDPOINT
  • You should see a JSON response like {"count": 0, "images": []}. The count is 0 because you haven't processed any images through the pipeline yet.

Getting an Internal Server Error?

  • If you see a 500 error, the most common cause is the missing invoke permission. Confirm you ran the add-permission command above and saw a successful response.
  • If the error persists, check that your TABLE_NAME environment variable matches the actual DynamoDB table name. Run echo $TABLE_NAME to verify.
  • Wait 5-10 seconds after granting permission and try again. Permission propagation can take a moment.
  • Help me debug my API Gateway 500 error

Your query API is live. In the next step, you'll upload a test image and watch the entire pipeline fire, from S3 upload through to a populated API response.

Test the Complete Pipeline

You've built every piece of the pipeline. Now it's time to see them all working together.

In this step, you'll upload images to the source bucket and watch the entire chain fire automatically. S3 triggers SQS, SQS triggers Lambda, Lambda resizes the image and writes metadata to DynamoDB, then sends you an email via SNS.

In this step, get ready to:
  • Upload a test image and verify the pipeline processes it end-to-end.
  • Confirm email notifications arrive and the API returns metadata.
  • Upload a second image and validate the pipeline handles multiple files.
Upload a test image and verify the pipeline fires

You need any image file to trigger the pipeline. If you don't have one handy, you can create a simple test image using Python's Pillow library that's already installed in your project's package directory.

  • Create a small test image by running this command:
python3 -c "from PIL import Image; img = Image.new('RGB', (800, 600), color='blue'); img.save('test-image.jpg')"

What does this command do?

This uses Pillow to generate a simple 800x600 blue JPEG image. It's large enough that the Lambda function will resize it down to 400x400, giving you a meaningful test of the thumbnail logic.

  • Upload the test image to your source bucket by running:
aws s3 cp test-image.jpg s3://$SOURCE_BUCKET/test-image.jpg

What happens when you upload?

The moment this file lands in the source bucket, S3 sends an event notification to the SQS queue. The event source mapping picks up the message and invokes your image-processor Lambda function. The function downloads the image, resizes it, uploads the result, writes metadata, and sends an SNS notification.

  • Wait about 15 seconds for the pipeline to process, then check that the resized image appeared in the processed bucket by running:
aws s3 ls s3://$DEST_BUCKET/
  • You should see a file named resized-test-image.jpg in the output.
  • Check the CloudWatch logs to confirm the Lambda executed successfully by running:
aws logs tail /aws/lambda/image-processor --since 5m

What should I see?

You should see log entries showing the Lambda function starting and completing. Look for lines indicating the function was invoked and returned a success status. If you see error messages instead, the safety net below can help.

  • Verify the metadata was written to DynamoDB by running:
aws dynamodb scan --table-name $TABLE_NAME
  • You should see a JSON response with Count: 1 and an Items array containing one record with fields like image_id, original_key, processed_key, original_size, processed_size, processed_at, and status.

Don't see a resized image or DynamoDB record?

  • Check the CloudWatch logs for error details. Permission errors (AccessDenied) usually mean the IAM role is missing a policy.
  • If the log group doesn't exist yet, the Lambda may not have been invoked. Verify the SQS event source mapping is enabled by checking the Lambda console.
  • If you see a timeout error, the Lambda may need more time. The 10-second timeout should be sufficient for most images, but very large files may need more.

Help me troubleshoot why my image processing pipeline isn't working. I uploaded an image to S3 but nothing happened.

Verify the email notification and query the API

Two more pieces of the pipeline need verification. Your SNS subscription should have delivered an email, and the API Gateway endpoint should now return the metadata you just confirmed in DynamoDB.

  • Check your email inbox (including spam/junk) for a message with the subject line Image Processing Complete.
  • The email body should include the original filename, original size in bytes, processed size in bytes, and the destination key.

No email received?

  • Check your spam or junk folder. AWS SNS notifications often get filtered.
  • Confirm you clicked the confirmation link in the earlier subscription email. Unconfirmed subscriptions won't receive messages.
  • Verify the SNS_TOPIC_ARN environment variable on the Lambda matches your topic by running aws lambda get-function-configuration --function-name image-processor.

Help me figure out why I'm not receiving SNS email notifications from my Lambda function.

  • Query your API Gateway endpoint to see the metadata returned as JSON by running:
curl $API_ENDPOINT
  • You should see a JSON response with a count of 1 and an images array containing your processed image metadata.

What does this response prove?

This single curl command exercises the full read path. API Gateway receives your HTTP request, invokes the api-function Lambda, which scans DynamoDB and returns the metadata as JSON. If you see data here, every service in the query chain is wired correctly.

Getting an empty response or error?

  • If you see {"count": 0, "images": []}, the API Lambda can't read the DynamoDB table. Check that the TABLE_NAME environment variable on api-function matches your table name exactly.
  • If you get a connection error, verify the API_ENDPOINT variable is set correctly. Re-run the aws apigatewayv2 get-apis command to find the endpoint URL.

Help me debug my API Gateway endpoint that isn't returning the expected image metadata.

Upload a second image and confirm multiple files are handled

A pipeline that works once is promising. A pipeline that works consistently is production-ready. Uploading a second image confirms the event-driven pattern handles repeated invocations without issues.

  • Create a second test image by running:
python3 -c "from PIL import Image; img = Image.new('RGB', (1024, 768), color='red'); img.save('test-image-2.jpg')"
  • Upload the second image to the source bucket by running:
aws s3 cp test-image-2.jpg s3://$SOURCE_BUCKET/test-image-2.jpg
  • Wait about 15 seconds, then check the processed bucket again by running:
aws s3 ls s3://$DEST_BUCKET/
  • You should now see both resized-test-image.jpg and resized-test-image-2.jpg listed.
  • Query the API endpoint again to confirm both records appear by running:
curl $API_ENDPOINT
  • The response should now show count of 2 and two items in the images array, each with unique image_id values.
  • Check your email inbox again. You should have a second notification for test-image-2.jpg.

Why does this matter?

Each upload triggers a completely independent execution of the pipeline. SQS ensures messages are processed reliably. Lambda scales automatically to handle concurrent invocations. DynamoDB stores each record with a unique image_id generated by uuid.uuid4().

This is the core value of event-driven architecture. You don't manage servers, poll for changes, or orchestrate retries. The services handle all of that for you.

Second image not processing?

  • If the first image worked but the second didn't, check CloudWatch logs for a new invocation. The SQS visibility timeout may need to be higher if the first message is still being retried.
  • Run aws sqs get-queue-attributes --queue-url $SQS_QUEUE_URL --attribute-names ApproximateNumberOfMessages to see if messages are stuck in the queue.

Help me troubleshoot why my second image upload isn't being processed by the pipeline.

Your pipeline is fully operational. Every service in the chain is connected and processing images automatically. Next up, take on the Secret Mission to extend the pipeline with dimension extraction and a DELETE endpoint.

Secret mission

Add Dimension Extraction and a DELETE Endpoint

Your pipeline resizes images but doesn't capture their dimensions. You also have no way to remove processed images through the API. In this secret mission, you'll extract width and height from every image using Pillow and store them in DynamoDB. Then you'll add a DELETE endpoint to your API so you can remove images on demand.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. All services in this project fall within the AWS Free Tier for light usage, but exceeding free tier limits will incur charges.

Cost warning

AWS Free Tier covers this project's usage at low volume (1M Lambda requests/month, 5GB S3 storage, 25GB DynamoDB, 1M SQS requests/month, 1M API Gateway requests/month for 12 months). If you leave these resources in place and exceed these limits, charges will apply.

If you are not planning to continue experimenting with this pipeline, choose Pause or Delete below.

Resources you used:

  • S3 Buckets: image-pipeline-source-<ACCOUNT_ID> and image-pipeline-processed-<ACCOUNT_ID>.
  • Lambda Functions: image-processor and api-function.
  • DynamoDB Table: image-metadata.
  • SQS Queue: image-processing-queue.
  • SNS Topic: image-processing-notifications (with email subscription).
  • API Gateway HTTP API: image-api.
  • IAM Roles: image-processor-role and api-lambda-role.
  • Local project files: lambda_function.py, api_function.py, and deployment packages.

Keep everything running

No action needed. Choose this if you are still actively experimenting with the pipeline or plan to extend it further.

  • All services are serverless and only charge on use. If nobody uploads images, you pay nothing.
  • The SNS email subscription will continue sending you notifications for each processed image.
  • Your API Gateway endpoint remains publicly accessible.

Pause - I'll come back to this later

Shut down active triggers to stop the pipeline from processing, while keeping all resources in place for later.

  • Disable the SQS event source mapping so the Lambda stops receiving messages. First, find the mapping UUID by running:
aws lambda list-event-source-mappings --function-name image-processor
  • Copy the UUID from the output, then disable it by running:
aws lambda update-event-source-mapping --uuid [[ESM_UUID="your event source mapping UUID"]] --no-enabled
  • When you are ready to resume, re-enable it by running the same command with --enabled instead.
  • Your S3 buckets, DynamoDB table, Lambda functions, and API endpoint all remain in place. No data is lost.

Delete - I don't want to use this again

Remove all project resources permanently. Make sure your environment variables from Step 1 are still set (ACCOUNT_ID and REGION).

  • Delete the SQS event source mapping. Find the UUID first:
aws lambda list-event-source-mappings --function-name image-processor
  • Copy the UUID from the output and delete the mapping:
aws lambda delete-event-source-mapping --uuid [[ESM_UUID="your event source mapping UUID"]]
  • Delete both Lambda functions:
aws lambda delete-function --function-name image-processor
aws lambda delete-function --function-name api-function
  • Delete the API Gateway HTTP API. First, get the API ID:
aws apigatewayv2 get-apis
  • Find the ApiId for image-api in the output, then delete it:
aws apigatewayv2 delete-api --api-id [[API_ID="your image-api API ID"]]
  • Empty and delete both S3 buckets:
aws s3 rb s3://image-pipeline-source-$ACCOUNT_ID --force
aws s3 rb s3://image-pipeline-processed-$ACCOUNT_ID --force
  • Delete the SQS queue:
aws sqs delete-queue --queue-url https://sqs.$REGION.amazonaws.com/$ACCOUNT_ID/image-processing-queue
  • Delete the DynamoDB table:
aws dynamodb delete-table --table-name image-metadata
  • Delete the SNS topic (this also removes all subscriptions):
aws sns delete-topic --topic-arn arn:aws:sns:$REGION:$ACCOUNT_ID:image-processing-notifications
  • Delete the IAM roles. First, detach the managed policy and delete inline policies from the processor role:
aws iam detach-role-policy --role-name image-processor-role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole
aws iam delete-role-policy --role-name image-processor-role --policy-name image-processor-policy
aws iam delete-role --role-name image-processor-role
  • Delete the API Lambda role:
aws iam delete-role-policy --role-name api-lambda-role --policy-name api-lambda-policy
aws iam delete-role --role-name api-lambda-role
  • Delete your local project files:
cd ~/Desktop
rm -rf [[PROJECT_DIR="your image-pipeline project folder name"]]
  • Confirm all resources are gone by running:
aws lambda list-functions --query "Functions[?starts_with(FunctionName, 'image-') || starts_with(FunctionName, 'api-')]"

You should see an empty list, confirming all Lambda functions are deleted. Your pipeline is fully cleaned up.

That's a Wrap!

That's a Wrap!

Nice work! You've just built a production-grade event-driven image processing pipeline on AWS, connecting 8 services into a fully automated serverless architecture.

You've learned how to:

  • Built an event-driven image processing pipeline connecting S3, Lambda, SQS, DynamoDB, SNS, CloudWatch, API Gateway, and IAM into a cohesive serverless architecture.
  • Created Lambda deployment packages with native Python dependencies and wired SQS as a message buffer between S3 event notifications and Lambda processing.
  • Deployed a REST API with API Gateway HTTP API and configured least-privilege IAM roles for secure service-to-service communication.
  • Secret Mission: Added image dimension extraction using Pillow and a DELETE endpoint to remove processed images from S3 and DynamoDB.

Ready to quiz yourself?