Event-Driven Serverless File Pipeline

Wire S3, Lambda, DynamoDB, API Gateway, and SNS into one pipeline.

Introduction

30 Second Summary

Every time you upload a photo to a social media app, it gets resized, catalogued, and stored without anyone pressing a button. Behind the scenes, a chain of automated services is doing the heavy lifting.

In this project, you will build your own version of this automation on AWS using serverless services. Upload a file, and a pipeline of services will automatically extract its metadata, store it in a database, email you a notification, and serve the results through a live API.

What You'll Build

Drop a file into your storage bucket and within seconds, an email lands in your inbox confirming it was processed, metadata appears in your database, and a live API endpoint returns everything your pipeline has handled.

By the end of this project, you'll have:

  • A file processing pipeline that automatically extracts metadata from uploaded files and sends you an email notification through SNS within seconds of a file landing in S3.
  • A live REST API endpoint on API Gateway you can hit from a browser or terminal to see a JSON list of every file your pipeline has processed.
  • Five AWS services (S3, Lambda, DynamoDB, API Gateway, SNS) working together as one connected event-driven system you wired from scratch.
  • An optional Secret Mission to push your skills further.

Are there any prerequisites?

You'll need an AWS account and basic familiarity with Python. No prior AWS experience is required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

You're going to wire five AWS services together into an event-driven pipeline where uploading a single file kicks off an automatic chain of processing, storage, and notification.

Set Up Your S3 Bucket, DynamoDB Table, and SNS Topic

You have committed to building an event-driven file processing pipeline. Before any Lambda code can run, you need the AWS resources that your pipeline connects to.

In this step, you will create three foundational services. An S3 bucket to receive uploaded files. A DynamoDB table to store file metadata. An SNS topic to send email notifications when files are processed.

In this step, get ready to:
  • Create an S3 bucket and upload a test file.
  • Create a DynamoDB table to store file metadata.
  • Set up an SNS topic with an email subscription.
Create your S3 bucket

Your pipeline starts with S3. When a file lands in this bucket, it will kick off the entire processing chain. First, you need to create the bucket and verify it works by uploading a test file.

  • Sign in to the AWS Management Console.
  • In the top navigation bar, select a Region close to you (for example, us-east-1). Note which Region you choose. You will use the same Region for every resource in this project.
  • In the search bar at the top, type S3 and select S3 from the results.
  • In the left navigation pane, choose General purpose buckets.
  • Click Create bucket.
  • Under General configuration, confirm Bucket type is set to General purpose.
  • For Bucket name, enter a unique name following this pattern: file-pipeline- followed by your initials, a dash, and a few random digits. For example: file-pipeline-jd-8472.
  • Leave all other settings at their defaults. Block Public Access should remain enabled. Versioning stays off.
  • Scroll to the bottom and click Create bucket.

Why does my bucket name need to be globally unique?

S3 bucket names are shared across all AWS accounts worldwide. No two buckets anywhere on AWS can have the same name. Adding your initials and random digits ensures yours does not collide with anyone else's.

You should see your new bucket in the bucket list. Now upload a test file to confirm it works.

  • Click your bucket name to open it.
  • Click Upload.
  • Click Add files and choose any small file from your computer (an image, a text file, or any document).
  • Click Upload.

You should see a green success banner and your file listed on the Objects tab of your bucket.

Bucket name already exists?

If you see an error saying the bucket name is already taken, try adding more random digits or different initials. Bucket names must be globally unique across all AWS accounts.

help me fix my S3 bucket naming error

Create the FileMetadata DynamoDB table

Your Lambda function will write metadata about each processed file (name, size, content type, timestamp) into a DynamoDB table. You need to create this table before your function can use it.

  • In the search bar at the top of the console, type DynamoDB and select DynamoDB from the results.
  • In the left navigation pane, choose Tables.
  • Click Create table.
  • For Table name, enter FileMetadata.
  • For Partition key, enter fileKey and leave the type as String.
  • Leave the Sort key field empty.
  • Under Table settings, select Customize settings.
  • Under Read/write capacity settings, select On-demand.
  • Leave everything else at its defaults and click Create table.

What is a partition key?

A partition key is the primary identifier DynamoDB uses to locate each item in your table. Every item must have a unique partition key value. In this project, fileKey holds the S3 object path (like uploads/photo.jpg), which is naturally unique for each file.

On-demand capacity mode means DynamoDB scales read and write throughput automatically. You do not need to pre-configure capacity units. For a project with low, unpredictable traffic like this one, On-demand keeps things simple and stays within the free tier.

The table takes a few seconds to create. Wait until the Status column shows Active before moving on.

You should see FileMetadata listed in your tables with Status: Active.

Table stuck on Creating?

DynamoDB table creation is asynchronous. It usually finishes within seconds, but can take up to a minute. Click the refresh icon in the tables list to check the status again.

help me troubleshoot my DynamoDB table creation

Set up SNS email notifications

The last piece of your pipeline's foundation is an SNS topic. When your Lambda function finishes processing a file, it publishes a message to this topic. SNS then delivers that message as an email to every subscriber.

What is pub/sub messaging?

SNS uses a pub/sub (publish/subscribe) pattern. Publishers send messages to a topic without knowing who will receive them. Subscribers register their interest and receive every message published to that topic. This decouples the sender from the receiver.

  • In the search bar at the top, type SNS and select Simple Notification Service from the results.
  • In the left navigation pane, choose Topics.
  • Click Create topic.
  • For Type, select Standard.
  • For Name, enter file-processed-notifications.
  • Leave all other settings at their defaults.
  • Click Create topic.

You should now see the topic's detail page with its ARN displayed. Next, create an email subscription so you receive notifications.

  • On the file-processed-notifications topic page, click Create subscription.
  • For Protocol, choose Email.
  • For Endpoint, enter your email address.
  • Click Create subscription.

Confirm your email subscription now

AWS sends a confirmation email to the address you entered. Open your email inbox and look for a message from AWS Notifications. Click the Confirm subscription link in the email. You will not receive any pipeline notifications until you confirm.

Check your spam or junk folder if you do not see the email within a couple of minutes. Confirmation tokens are valid for two days.

  • After confirming, return to the SNS console.
  • In the left navigation pane, choose Subscriptions.
  • Verify that your subscription's Status column shows Confirmed.

You should see your email subscription listed with Status: Confirmed.

Subscription still showing PendingConfirmation?

Make sure you clicked the Confirm subscription link in the email from AWS. Check your spam folder. If the email never arrived, delete the subscription, double-check your email address, and create a new one.

help me confirm my SNS email subscription

Before moving on, take a moment to predict: do you have all three resources in place? An S3 bucket with a test file, a DynamoDB table showing Active, and an SNS subscription showing Confirmed?

  • Open the DynamoDB console and confirm the FileMetadata table shows Status: Active.
  • Open the SNS console and confirm the file-processed-notifications topic exists and your email subscription shows Confirmed.

Your three foundational resources are in place. Next up, you will write the Lambda function that ties them all together, processing files the moment they land in S3.

Build and Deploy the File Processor Lambda

In Step 1, you set up the three data stores and messaging channels your pipeline needs. Right now, files can sit in your S3 bucket, metadata can be stored in DynamoDB, and notifications can flow through SNS. But nothing connects them.

The missing piece is a Lambda function that reacts to S3 uploads and orchestrates the entire flow. In this step, you will build that function, give it the right permissions, and test it with a simulated S3 event so you can see the full processing chain before wiring any real triggers.

In this step, get ready to:
  • Create a Lambda function with the Python 3.13 runtime and configure its IAM permissions.
  • Write the file processor code that reads S3 metadata, writes to DynamoDB, and publishes to SNS.
  • Test the function with a simulated S3 event and verify results in DynamoDB and your email inbox.
Create the fileProcessor Lambda function

Every Lambda function needs two things: a runtime (the language it runs) and an execution role (the IAM permissions it has). You will create the function first, then expand its permissions so it can reach S3, DynamoDB, and SNS.

  • Open the Functions page of the Lambda console.
  • Click Create function.
  • Select Author from scratch.
  • Under Basic information, enter fileProcessor for the Function name.
  • For Runtime, choose Python 3.13.
  • For Architecture, choose x86_64.
  • Leave the Change default execution role section at its default setting. Lambda will create a new role with basic permissions for you.
  • Click Create function.

Lambda creates the function and drops you into the code editor. You should see a default lambda_function.py file with a Hello World handler.

What is an execution role?

An execution role is an IAM role that Lambda assumes when your function runs. It controls which AWS services your function can talk to. The default role only has permission to write logs to CloudWatch. You need to add permissions for S3, DynamoDB, and SNS.

The default role only lets your function write logs. You need to attach three additional policies so the function can read from S3, write to DynamoDB, and publish to SNS.

  • In the Lambda console, click the Configuration tab, then click Permissions.
  • Under Execution role, click the role name link. This opens the role in the IAM console in a new tab.
  • In the IAM console, click Add permissions, then click Attach policies.
  • Search for AmazonS3ReadOnlyAccess and select its checkbox.
  • Clear the search, then search for AmazonDynamoDBFullAccess and select its checkbox.
  • Clear the search again, then search for AmazonSNSFullAccess and select its checkbox.
  • Click Add permissions.

You should see all three policies listed under Permissions policies alongside the original AWSLambdaBasicExecutionRole policy. That is four policies total.

Don't see the policies listed?

Make sure you clicked Add permissions at the bottom of the page after selecting all three policies. If you only see one or two, repeat the search-and-select process for the missing ones.

help me attach IAM policies to my Lambda execution role

Write the file processor code

Now you will replace the default Lambda code with your file processor. This code does three things when triggered by an S3 event: it reads the uploaded file's metadata, stores that metadata in DynamoDB, and sends a notification through SNS.

  • Switch back to the Lambda console tab and click the Code tab.
  • In the code editor, click the lambda_function.py tab.
  • Select all the existing code and delete it.
  • Paste the following imports and client setup code:
import json
import boto3
import os
from datetime import datetime, timezone
from urllib.parse import unquote_plus

s3_client = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
sns_client = boto3.client('sns')

TABLE_NAME = os.environ['DYNAMODB_TABLE']
SNS_TOPIC_ARN = os.environ['SNS_TOPIC_ARN']

What does this code do?

  • The import lines bring in boto3 (the AWS SDK for Python, included in the Lambda runtime) and standard libraries for JSON, dates, and URL decoding.
  • Three boto3 clients are created at module level so they are reused across invocations. This is a Lambda performance best practice.
  • TABLE_NAME and SNS_TOPIC_ARN read from environment variables so the code stays reusable without hardcoded values.
  • Place your cursor at the end of the code you just pasted and add a blank line. Then paste the handler function below the existing code:
def lambda_handler(event, context):
    table = dynamodb.Table(TABLE_NAME)

    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = unquote_plus(record['s3']['object']['key'])

        # Get file metadata from S3
        response = s3_client.head_object(Bucket=bucket, Key=key)
        content_type = response['ContentType']
        file_size = response['ContentLength']
        timestamp = datetime.now(timezone.utc).isoformat()

        # Write metadata to DynamoDB
        item = {
            'fileKey': key,
            'bucket': bucket,
            'contentType': content_type,
            'fileSize': file_size,
            'processedAt': timestamp
        }
        table.put_item(Item=item)

        # Publish notification to SNS
        message = (
            f"File processed successfully!\n\n"
            f"File: {key}\n"
            f"Bucket: {bucket}\n"
            f"Content Type: {content_type}\n"
            f"Size: {file_size} bytes\n"
            f"Processed at: {timestamp}"
        )

        sns_client.publish(
            TopicArn=SNS_TOPIC_ARN,
            Subject='File Processed - Pipeline Notification',
            Message=message
        )

        print(f"Processed file: {key}, size: {file_size}, type: {content_type}")

    return {
        'statusCode': 200,
        'body': json.dumps('File processing complete')
    }

What does the handler do?

  • lambda_handler loops through each record in the S3 event. Each record represents one uploaded file.
  • unquote_plus decodes the object key because S3 event notifications URL-encode special characters (spaces become +).
  • head_object retrieves the file's ContentType and ContentLength without downloading the actual file. This keeps the function fast and cheap.
  • put_item writes the file metadata to DynamoDB, and sns_client.publish sends a summary email through your SNS topic.
  • In the DEPLOY section of the code editor, click Deploy.

You should see a green banner confirming that changes have been deployed.

✔️ Awesome, I've got everything!

Great. Make sure you clicked Deploy and saw the confirmation banner before moving on.

ⓧ I'd like to double check the full code

Your lambda_function.py should contain two sections in order: the imports and client setup at the top, then the lambda_handler function below. Scroll up in your code editor and confirm each block is present.

  • Lines 1-12: imports (json, boto3, os, datetime, unquote_plus), three boto3 clients, and two environment variable reads.
  • Lines 14 onward: the lambda_handler function that loops through S3 records, calls head_object, writes to DynamoDB with put_item, publishes to SNS, and returns a 200 response.

Seeing a syntax error after deploying?

Make sure you deleted all the original default code before pasting. Leftover lines from the Hello World template will cause syntax errors. The file should start with import json on line 1.

help me fix a syntax error in my Lambda function code

Configure environment variables and test the function

The code reads DYNAMODB_TABLE and SNS_TOPIC_ARN from environment variables. You need to set these before the function can run. Then you will create a test event that simulates an S3 upload to verify the full processing chain.

  • Click the Configuration tab, then click Environment variables in the left sidebar.
  • Click Edit.
  • Click Add environment variable.
  • For Key, enter DYNAMODB_TABLE. For Value, enter FileMetadata.
  • Click Add environment variable again.
  • For Key, enter SNS_TOPIC_ARN. For Value, you need the ARN of your SNS topic.

Find your SNS topic ARN

Open the SNS console in a new tab. Click Topics in the left sidebar, then click file-processed-notifications. Copy the ARN value shown at the top of the topic details page. It looks like arn:aws:sns:us-east-1:123456789012:file-processed-notifications.

  • Paste the copied ARN as the value for SNS_TOPIC_ARN.
  • Click Save.

You should see both environment variables listed: DYNAMODB_TABLE set to FileMetadata and SNS_TOPIC_ARN set to your topic's ARN.

Now you will create a test event that simulates what S3 sends to Lambda when a file is uploaded. This lets you verify the entire pipeline without actually wiring an S3 trigger yet.

  • Click the Test tab.
  • Under Test event, select Create new event.
  • For Event name, enter testS3Upload.
  • In the Event JSON box, replace the default JSON with the following test event. Replace the two bucket name placeholders with your actual bucket name, and replace the object key placeholder with the name of the test file you uploaded in Step 1:
{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "EXAMPLE"
      },
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "x-amz-request-id": "EXAMPLE123456789",
        "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "testConfigRule",
        "bucket": {
          "name": "[[S3_BUCKET_NAME=\"your-bucket-name\"]]",
          "ownerIdentity": {
            "principalId": "EXAMPLE"
          },
          "arn": "arn:aws:s3:::[[S3_BUCKET_NAME=\"your-bucket-name\"]]"
        },
        "object": {
          "key": "[[TEST_FILE_NAME=\"your-test-file-name.txt\"]]",
          "size": 1024,
          "eTag": "0123456789abcdef0123456789abcdef",
          "sequencer": "0A1B2C3D4E5F678901"
        }
      }
    }
  ]
}

What should I replace?

  • Replace both bucket name values with your actual S3 bucket name (the one that starts with file-pipeline-).
  • Replace the object key with the filename of the test file you uploaded in Step 1 (for example, test.txt or photo.jpg).
  • Update the awsRegion value if your bucket is in a different region than us-east-1.
  • Click Save.

Before you run the test, think about what should happen. Your function reads S3 metadata, writes to DynamoDB, and publishes to SNS. What do you expect to see if everything works correctly?

  • Click Test.

Under Execution result, you should see a green succeeded banner. Expand Details and you should see a response with statusCode 200.

Now verify the dominos actually fell. Check DynamoDB and your email to confirm end-to-end processing.

  • Open the DynamoDB console in a new tab.
  • Click Tables in the left sidebar, then click FileMetadata.
  • Click Explore table items. You should see a new item with your test file's name as the fileKey, along with bucket, contentType, fileSize, and processedAt fields.
  • Check your email inbox. You should have received an email from AWS Notifications with the subject line File Processed - Pipeline Notification containing your file's metadata.

Test succeeded but no DynamoDB item or email?

  • Double-check your environment variables. Click Configuration > Environment variables and confirm DYNAMODB_TABLE is exactly FileMetadata (case-sensitive) and SNS_TOPIC_ARN is a valid ARN.
  • Make sure you replaced the bucket name and object key in the test event with real values from your S3 bucket. The file must actually exist in the bucket for head_object to succeed.
  • Verify the three IAM policies (AmazonS3ReadOnlyAccess, AmazonDynamoDBFullAccess, AmazonSNSFullAccess) are attached to the execution role.
  • If you see an email error, confirm that your SNS subscription still shows Confirmed status in the SNS console.

my Lambda test succeeded but I don't see items in DynamoDB or emails from SNS

Your file processor is live and tested. You have a Lambda function that can extract metadata from any S3 file, store it in DynamoDB, and notify you by email. Next up, you will wire an actual S3 trigger so uploads automatically kick off this processing chain.

Wire the S3 Trigger and Watch It Fail

In the previous step, you built and tested the fileProcessor Lambda function with a manual test event. It successfully wrote metadata to DynamoDB and fired an email notification through SNS.

But right now, uploading a file to your S3 bucket does nothing. The Lambda function only runs when you click Test in the console. In this step, you will connect S3 to Lambda so uploads automatically trigger processing. Then you will discover a real-world problem with how you configured the trigger.

In this step, get ready to:
  • Add an S3 trigger to your fileProcessor Lambda function.
  • Upload files and verify the full pipeline works end-to-end.
  • Observe what happens when triggers are not scoped properly.
Add an S3 trigger to the fileProcessor Lambda

S3 event notifications let a bucket send a message to another AWS service whenever something happens to an object. You will configure your bucket to notify the fileProcessor Lambda function every time a new object is created.

  • Return to the Lambda console and click on your fileProcessor function.
  • In the Function overview pane at the top, click Add trigger.
  • Select S3 from the trigger source dropdown.
  • Under Bucket, select your file-pipeline- bucket.
  • Under Event types, confirm that All object create events is selected.
  • Leave the Prefix and Suffix fields empty. You are not adding any filters right now.
  • Under Recursive invocation, check the acknowledgment box.
  • Click Add.

What is recursive invocation?

AWS warns you about a scenario where a Lambda function writes back to the same bucket that triggers it. That creates an infinite loop of events. Your function does not write to S3, so this warning does not apply here. You are checking the box to acknowledge you understand the risk.

You should now see S3 listed as a trigger in the Function overview diagram, connected to your fileProcessor function.

Don't see the S3 trigger?

If you see an error about being unable to validate destination configurations, a previous event notification on the bucket may be blocking the new one. Open the S3 console, go to your bucket's Properties tab, scroll to Event notifications, and delete any stale notifications before retrying.

Help me troubleshoot my S3 trigger not appearing on the fileProcessor Lambda function.

Upload a file and verify end-to-end processing

The trigger is live. Every new object created in your bucket now fires the Lambda function automatically, without you clicking Test.

Before you upload, think about what you expect to happen. Do you think the Lambda will run, write to DynamoDB, and send an email, just like the manual test?

  • Switch to the S3 console and open your file-pipeline- bucket.
  • Click Upload.
  • Click Add files and choose a small file from your computer. Pick a different file than the test file you uploaded in Step 1.
  • Click Upload to upload the file.

Wait about 10 seconds for the trigger to fire and the Lambda to finish executing.

Now verify the pipeline worked by checking three places.

  • Open the CloudWatch console by searching for CloudWatch in the AWS search bar.
  • In the left sidebar, click Logs, then click Log groups.
  • Click the log group named /aws/lambda/fileProcessor.
  • Click the most recent log stream at the top of the list.

You should see log entries showing the file name, size, and content type from your print statement in the Lambda code.

  • Return to the DynamoDB console and open the FileMetadata table.
  • Click Explore table items.

You should see a new item with the file name you just uploaded as the fileKey, along with its content type, size, and processing timestamp.

  • Check your email inbox for a new message with the subject File Processed - Pipeline Notification.

The email body shows the file name, bucket, content type, size, and processed timestamp. This proves the entire pipeline ran automatically from a single S3 upload.

What just happened?

You uploaded a file to S3, and without clicking anything else, five AWS services worked together. S3 detected the new object and sent an event to Lambda. Lambda read the file's metadata from S3, wrote it to DynamoDB, and published a notification through SNS that arrived in your inbox.

This is event-driven architecture in action. Each service reacts to the output of the previous one, with no polling or manual triggers needed.

No email or DynamoDB entry?

Make sure you are checking CloudWatch in the same AWS Region where you created your Lambda function. Logs only appear in the Region the function runs in.

If CloudWatch shows no invocation at all, the trigger may not have attached correctly. Return to the Lambda console and confirm S3 appears in the Function overview diagram.

If the function ran but DynamoDB or email is missing, check that the environment variables DYNAMODB_TABLE and SNS_TOPIC_ARN are still set correctly under Configuration > Environment variables.

Help me debug why my S3 trigger is not invoking the fileProcessor Lambda function.

Watch the unscoped trigger create noise

The pipeline works. Every file you upload gets processed, logged, stored, and emailed. But there is a problem hiding in this setup that surfaces when you look closely at the trigger configuration.

Right now, the trigger fires for every single object created anywhere in the bucket. There is no filter to distinguish between files you intentionally want to process and anything else. What do you think happens when you upload two more files in quick succession?

  • Return to the S3 console and open your bucket.
  • Upload two different files one right after the other. Any small files will do.
  • Wait about 10 seconds, then return to the CloudWatch console.
  • Click Log groups in the left sidebar, then click /aws/lambda/fileProcessor.
  • Look at the log streams. You should see new streams for each invocation.

Every single object created in the bucket triggered your Lambda. Both files were processed, which is expected. But think about what this means at scale.

Why is this a problem?

Without a prefix filter, your Lambda processes every object in the bucket indiscriminately. In a real system, you might have log files, temporary uploads, configuration files, or output data sitting in the same bucket. All of them would trigger processing, wasting compute time and cluttering your DynamoDB table with entries you never wanted.

This is the core challenge of event-driven architecture: unscoped triggers create noise. The trigger needs a filter so it only reacts to the objects that matter.

Your pipeline works end-to-end, but the trigger is too broad. In the next step, you will fix this by scoping the trigger to a specific folder prefix and then build a REST API so all this processed metadata is actually queryable.

Scope the Trigger and Build the Query API

In Step 3, you wired S3 to Lambda and saw the problem firsthand. Without a prefix filter on the trigger, every object event in the bucket fires your function. That means noisy CloudWatch Logs and wasted invocations.

In this step, you fix the trigger by scoping it to an uploads/ prefix so only files placed in that folder trigger processing. Then you build a REST API so anyone can query the metadata your pipeline collects without logging into the DynamoDB console.

In this step, get ready to:
  • Scope the S3 trigger to the uploads/ prefix so only targeted uploads fire the pipeline.
  • Create a second Lambda function that queries DynamoDB and returns file metadata as JSON.
  • Deploy a REST API with API Gateway that serves the metadata over a public URL.
Scope the S3 trigger to the uploads/ prefix

The unscoped trigger you created in Step 3 fires on every object event in the bucket. You need to remove it and replace it with a scoped trigger that only reacts to files uploaded under the uploads/ prefix.

  • Return to the Lambda console and select your fileProcessor function.
  • In the Function overview pane, click the S3 trigger to expand its details.
  • Click Delete to remove the existing unscoped trigger.
  • Confirm the deletion when prompted.

The S3 trigger is now removed from your function. Time to add the scoped replacement.

  • In the Function overview pane, click Add trigger.
  • Select S3 as the trigger source.
  • Under Bucket, select your file-pipeline- bucket.
  • Under Event types, confirm All object create events is selected.
  • In the Prefix field, enter uploads/.
  • Under Recursive invocation, check the acknowledgment box.
  • Click Add.

What does the prefix filter do?

Setting the prefix to uploads/ tells S3 to only fire the Lambda trigger when an object key starts with that string. Files uploaded to the bucket root or under other folders are ignored completely.

This is the fix for the noisy invocations you saw in Step 3. In real-world pipelines, prefix and suffix filters are essential for controlling which events reach your processing logic.

Now verify the scoped trigger works by uploading a file into the uploads/ folder.

  • Open the S3 console and navigate to your bucket.
  • On the Objects tab, click Create folder.
  • Enter uploads as the folder name and click Create folder.
  • Click into the uploads/ folder.
  • Click Upload, then Add files. Choose any small file from your computer.
  • Click Upload to send the file.

Wait about 10 seconds, then check three places to confirm the pipeline ran:

  • Open the DynamoDB console. Select the FileMetadata table and click Explore table items. You should see a new item with a fileKey starting with uploads/.
  • Check your email inbox for an SNS notification about the processed file.

Don't see a new DynamoDB item?

  • Make sure you uploaded the file inside the uploads/ folder, not the bucket root. Files outside the prefix are now ignored.
  • Confirm the trigger shows uploads/ as the prefix in the Lambda Function overview pane.
  • Check CloudWatch Logs for the fileProcessor function to see if there are any errors.

Help me debug why my scoped S3 trigger isn't firing the Lambda function.

Create the fileMetadataApi Lambda function

Your pipeline writes metadata to DynamoDB, but the only way to read it right now is to log into the console. A second Lambda function will serve that data over HTTP so anyone with the URL can query it.

  • Open the Lambda console and click Create function.
  • Select Author from scratch.
  • For Function name, enter fileMetadataApi.
  • For Runtime, select Python 3.13.
  • For Architecture, keep x86_64.
  • Under Permissions, expand Change default execution role and choose Create a new role with basic Lambda permissions.
  • Click Create function.

Now attach the DynamoDB read permission to the execution role, just like you did for fileProcessor in Step 2.

  • Scroll down to the Configuration tab and click Permissions.
  • Click the role name link under Execution role to open it in the IAM console.
  • Click Add permissions, then Attach policies.
  • Search for AmazonDynamoDBReadOnlyAccess and check the box next to it.
  • Click Add permissions.

Why read-only access this time?

The fileProcessor function needed AmazonDynamoDBFullAccess because it writes items. This API function only reads data, so AmazonDynamoDBReadOnlyAccess follows the principle of least privilege. Each function gets only the permissions it actually needs.

Now add the DYNAMODB_TABLE environment variable so the function knows which table to scan.

  • Switch back to the Lambda console tab for fileMetadataApi.
  • Click the Configuration tab, then Environment variables.
  • Click Edit, then Add environment variable.
  • For Key, enter DYNAMODB_TABLE. For Value, enter FileMetadata.
  • Click Save.

Time to replace the default function code. Scroll up to the Code tab in the Lambda console.

  • Select all the default code in the editor and delete it.
  • Paste the following imports, client setup, and DecimalEncoder class:
import json
import boto3
import os
from decimal import Decimal

dynamodb = boto3.resource('dynamodb')
TABLE_NAME = os.environ['DYNAMODB_TABLE']

class DecimalEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Decimal):
            return int(obj) if obj % 1 == 0 else float(obj)
        return super().default(obj)

What does this code do?

  • It imports Decimal from Python's standard library because DynamoDB returns numbers as Decimal objects, which Python's json.dumps cannot serialize by default.
  • The DecimalEncoder class teaches json.dumps how to convert Decimal values into regular int or float values so they can be included in the JSON response.
  • Below the code you just pasted, add the handler function:
def lambda_handler(event, context):
    table = dynamodb.Table(TABLE_NAME)
    try:
        response = table.scan()
        items = response.get('Items', [])
        # Handle pagination if there are more items
        while 'LastEvaluatedKey' in response:
            response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
            items.extend(response.get('Items', []))
        return {
            'statusCode': 200,
            'headers': {
                'Content-Type': 'application/json',
                'Access-Control-Allow-Origin': '*'
            },
            'body': json.dumps({
                'count': len(items),
                'files': items
            }, cls=DecimalEncoder)
        }
    except Exception as e:
        print(f"Error scanning table: {str(e)}")
        return {
            'statusCode': 500,
            'headers': {
                'Content-Type': 'application/json',
                'Access-Control-Allow-Origin': '*'
            },
            'body': json.dumps({'error': 'Failed to retrieve file metadata'})
        }

What does this handler do?

  • It scans the entire FileMetadata table and collects all items. The while loop handles pagination so no items are missed if the table grows beyond one page of results.
  • It returns a response formatted for API Gateway's Lambda proxy integration: a dictionary with statusCode, headers (including CORS), and a JSON body.
  • The Access-Control-Allow-Origin: * header allows the API to be called from any origin, which is needed for browser-based requests.
  • Click Deploy to save and deploy the function.

✔️ Awesome, I've got everything!

Great. Make sure you clicked Deploy to save the function code.

ⓧ I'd like to double check the full code

Your lambda_function.py for fileMetadataApi should contain two blocks, in this order:

  • The imports, client setup, and DecimalEncoder class (first code block above).
  • The lambda_handler function with the try/except block (second code block above).

Scroll up and confirm both blocks are pasted in order with no extra code between them.

Test the function before wiring it to an API. Click Test in the Lambda console, accept the default test event (the event content does not matter since the function ignores it), and click Test.

You should see a green Execution result: succeeded banner. Expand the result to see a JSON body with a count and files array containing all the metadata your pipeline has collected so far.

Seeing an error about missing environment variable?

  • Make sure you added the DYNAMODB_TABLE environment variable with the value FileMetadata in the Configuration tab.
  • Confirm the AmazonDynamoDBReadOnlyAccess policy is attached to the function's execution role.

Help me debug why my fileMetadataApi Lambda function is failing.

Deploy the REST API with API Gateway

The Lambda function works, but it has no public URL. API Gateway puts a REST endpoint in front of the function so you can call it from a browser or curl.

  • Open the API Gateway console at https://console.aws.amazon.com/apigateway.
  • Click Create API.
  • Under REST API (not REST API Private), click Build.
  • For API name, enter FileProcessingAPI.
  • Keep API endpoint type set to Regional.
  • Click Create API.

Your API is created with a root / resource. Now create the /files resource.

  • Click Create resource.
  • For Resource name, enter files.
  • Click Create resource.

Now add a GET method on the /files resource that forwards requests to your Lambda function.

  • With the /files resource selected, click Create method.
  • For Method type, select GET.
  • For Integration type, select Lambda function.
  • Turn on Lambda proxy integration.
  • For Lambda function, enter fileMetadataApi and select it from the dropdown.
  • Click Create method.
  • If a permissions dialog appears asking to grant API Gateway permission to invoke your function, click OK.

What is Lambda proxy integration?

With proxy integration turned on, API Gateway forwards the entire HTTP request to your Lambda function as-is. Your function is responsible for returning a properly formatted response with statusCode, headers, and body. That is exactly why the fileMetadataApi code returns a dictionary in that shape.

Deploy the API so it gets a live URL.

  • Click Deploy API.
  • For Stage, select New stage.
  • For Stage name, enter prod.
  • Click Deploy.

Under Stage details, copy the Invoke URL. It looks something like this:

https://<api-id>.execute-api.<region>.amazonaws.com/prod

Before you test this, take a moment to predict what you expect to see. Do you think the response will include the files you uploaded in earlier steps?

  • Append /files to the invoke URL and open it in your browser. The full URL looks like:
https://<api-id>.execute-api.<region>.amazonaws.com/prod/files

You should see a JSON response with a count and a files array listing every file your pipeline has processed. Each item includes the fileKey, bucket, contentType, fileSize, and processedAt timestamp.

Seeing an Internal Server Error?

  • Confirm you turned on Lambda proxy integration when creating the GET method. Without it, API Gateway expects a different response format.
  • Verify the DYNAMODB_TABLE environment variable is set to FileMetadata on the fileMetadataApi function.
  • Make sure you appended /files to the invoke URL. Hitting the base URL without a resource path returns a different error.

Help me debug why my API Gateway endpoint is returning an Internal Server Error.

Now confirm the full pipeline still works end-to-end by uploading a new file and checking the API.

  • Go back to S3, navigate into the uploads/ folder, and upload a second test file.
  • Wait about 10 seconds, then refresh your API URL in the browser.
  • You should see the count increase by one, and the new file should appear in the files array.

Five AWS services are now working together as one connected system. Upload a file to S3, Lambda processes it, DynamoDB stores the metadata, SNS sends you an email, and API Gateway serves it all over a public URL. Next up, the Secret Mission challenges you to make the pipeline smarter by routing notifications based on file type.

Secret mission

Add File-Type Routing

Right now every file gets the same generic notification. Can you make the pipeline smart enough to recognize different file types and send tailored notifications for images vs. documents?

Clean Up Your Resources

Clean Up Your Resources

All five AWS services in this project are serverless, so they only incur charges when actively processing requests. If your usage stays within the AWS free tier limits, there are no ongoing costs.

Cost warning

Most free tier allowances for these services (Lambda, DynamoDB, SNS, S3) are ongoing. However, the API Gateway free tier of 1 million REST API calls per month expires after 12 months. If you are done with this project, delete your resources to avoid any future charges.

Resources you used:

  • S3 bucket file-pipeline-<initials>-<random> with uploaded test files in the uploads/ folder.
  • DynamoDB table FileMetadata storing processed file metadata.
  • Lambda function fileProcessor with S3 trigger and IAM role.
  • Lambda function fileMetadataApi with IAM role.
  • REST API FileProcessingAPI in API Gateway with the prod stage.
  • SNS topic file-processed-notifications with your email subscription.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment with additional file types and trigger configurations.

  • All five services are serverless. They only consume resources when a request is made, so idle costs are effectively zero within free tier limits.
  • Monitor your usage in the AWS Billing console to make sure you stay within free tier allowances.
  • The API Gateway free tier expires after 12 months. Set a calendar reminder to revisit and delete if you no longer need the API.

Pause - I'll come back to this later

Serverless resources do not have a running state to pause. They only activate when triggered by an event or API call, so there is nothing to shut down.

  • Your Lambda functions, DynamoDB table, API Gateway, and SNS topic all sit idle at no cost until they receive a request.
  • If you want to prevent the pipeline from processing new uploads while you are away, remove the S3 trigger from the fileProcessor function in the Lambda console. You can re-add it later with the same uploads/ prefix filter.
  • When you are ready to resume, re-add the S3 trigger and upload a new file to confirm everything still works.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow these steps in order to avoid dependency errors.

  • Delete the S3 bucket: Open the S3 console and select your file-pipeline-<initials>-<random> bucket. Click Empty, type permanently delete to confirm, and click Empty. Then go back to the bucket list, select the bucket again, click Delete, type the bucket name to confirm, and click Delete bucket.
  • Delete the Lambda functions: Open the Lambda console and click Functions. Select fileProcessor, click Actions, then Delete, and confirm. Repeat for fileMetadataApi.
  • Delete the API Gateway: Open the API Gateway console. Select FileProcessingAPI, click Actions, then Delete, and confirm.
  • Delete the DynamoDB table: Open the DynamoDB console and click Tables. Select FileMetadata, click Delete, confirm by typing delete, and click Delete table.
  • Delete the SNS topic: Open the SNS console and click Topics. Select file-processed-notifications, click Delete, and confirm. This also removes the email subscription.

Nice Work!

Nice Work!

Nice work! You just built a fully automated, event-driven file processing pipeline on AWS using five interconnected serverless services.

You've learned how to:

  • Built an event-driven pipeline where uploading a file to S3 automatically triggers a Lambda function that extracts metadata, stores it in DynamoDB, and sends an email notification through SNS.
  • Scoped S3 event notifications with prefix filters to eliminate noisy invocations, a critical pattern for production event-driven architectures.
  • Created a REST API with API Gateway that returns processed file metadata as JSON, giving your pipeline a queryable interface.
  • Tackled an optional Secret Mission to extend your pipeline even further.

Ready to quiz yourself?