Build a Serverless File Pipeline on AWS

Wire S3, Lambda, and DynamoDB into an event-driven file processing pipeline.

Introduction

30 Second Summary

Every time you upload a photo to social media or drop a document into cloud storage, something behind the scenes detects that new file and springs into action. That invisible automation is what cloud engineers call an event-driven pipeline.

In this project, you will build your own serverless file processing pipeline on AWS. When you drop a file into a storage bucket, a function fires automatically to extract the file's metadata and store it in a database.

What You'll Build

Drop any file into your storage bucket and, within seconds, see its name, size, and type appear as a queryable record in your database.

By the end of this project, you'll have:

  • Upload any file to your S3 bucket and watch a Lambda function fire automatically within seconds.
  • See the file's metadata (name, size, type, upload time) appear as a new record in your DynamoDB table.
  • Diagnose and fix a real IAM permission failure using CloudWatch Logs. This is the same troubleshooting loop cloud engineers use daily.
  • Secret Mission: Add a file type filter so only image uploads get processed, and enrich the metadata with the uploading user's identity.

Do I need any cloud experience?

Not at all. This project starts from scratch. You will create an AWS Free plan account in Step 1, and everything happens right in your web browser.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your AWS Account and Create an S3 Bucket

Every serverless pipeline starts with a place to store files. Amazon S3 is AWS's object storage service, and the bucket you create here is where your pipeline begins.

In this step, you will create an AWS Free plan account, secure it with MFA, and set up an S3 bucket to hold the files your pipeline will process.

In this step, get ready to:
  • Create an AWS Free plan account with MFA enabled.
  • Create an S3 bucket with a globally unique name.
  • Upload a test file to verify the bucket works.
Create an AWS Free plan account

The AWS Free plan gives you $100 in credits at signup, with up to $100 more as you explore services. You will not be charged unless you upgrade to a Paid plan.

  • Open https://aws.amazon.com/free/ in your browser.
  • Choose Create a free account.
  • Enter your email address, choose an account name, and choose Verify email address.
  • Check your inbox for the verification code, enter it, and choose Verify.
  • Create a strong password for your root user and choose Continue.
  • When prompted to choose an account plan, select Free plan.
  • Enter your contact information, accept the AWS Customer Agreement, and choose Continue.
  • Add a payment method to verify your identity. AWS will not charge your card on the Free plan.
  • Complete the phone verification step by entering the code AWS sends you.
  • On the Select a support plan page, choose Basic support - Free.
  • Choose Complete sign up.

Account activation usually takes a few minutes but can take up to 24 hours. Check your email for a confirmation message.

Why do I need a credit card for a free account?

AWS requires a valid payment method to verify your identity and prevent abuse. Your card will not be charged unless you upgrade to a Paid plan.

Account activation taking a while?

Activation can take up to 24 hours in rare cases. Check your email (including spam) for a confirmation message. You can sign in and start exploring the console while you wait.

Help me troubleshoot my AWS account activation

Once your account is activated, sign in to the AWS Management Console at https://console.aws.amazon.com/.

Enable MFA on your root account

MFA adds a second layer of security to your account. AWS requires all root users to register MFA within 35 days of their first sign-in.

  • Download an authenticator app on your phone if you do not already have one (e.g. Google Authenticator, Microsoft Authenticator, or Authy).
  • In the AWS Management Console, click your account name in the top-right corner of the navigation bar.
  • Choose Security credentials.
  • Scroll down to the Multi-Factor Authentication (MFA) section.
  • Choose Assign MFA device.
  • Enter a device name (e.g. my-phone), select Authenticator app, and choose Next.
  • Choose Show QR code and scan the QR code with your authenticator app.
  • Enter the six-digit code currently displayed in your authenticator app into the MFA code 1 box.
  • Wait for the code to refresh (up to 30 seconds), then enter the new six-digit code into the MFA code 2 box.
  • Choose Add MFA.

You should see a success banner confirming that the MFA device has been assigned to your root user.

What is MFA?

Multi-factor authentication requires two forms of identity when signing in: your password (something you know) and a code from your phone (something you have). This means that even if someone steals your password, they cannot access your account without your phone.

Having trouble adding MFA?

Make sure the time on your phone is set to sync automatically. Authenticator apps generate time-based codes, so a mismatch of even a few seconds can cause them to be rejected.

Help me fix my MFA setup issue

Create an S3 bucket and upload a test file

Your S3 bucket is where files will land to kick off the pipeline. Every bucket name must be globally unique across all of AWS.

  • In the AWS Management Console, type S3 into the search bar at the top and select S3 from the results.
  • In the left navigation pane, choose General purpose buckets.
  • Choose Create bucket.
  • In the Bucket name field, enter a globally unique name. Use the pattern file-pipeline-yourname-2026, replacing yourname with your own name (all lowercase, no spaces).
  • Record your bucket name here: file-pipeline-yourname-2026.
  • Leave all other settings as their defaults. Block all public access should be enabled and ACLs should be disabled.
  • Scroll to the bottom and choose Create bucket.

Why does the bucket name have to be globally unique?

S3 bucket names share a single global namespace across all AWS accounts. No two buckets in the world can have the same name. Adding your name and a year makes collisions unlikely.

Bucket names must be 3 to 63 characters long, use only lowercase letters, numbers, and hyphens, and start and end with a letter or number.

You should see your new bucket in the General purpose buckets list with 0 objects.

Now upload a test file to confirm everything works.

  • Click on your bucket name to open it.
  • Choose Upload.
  • Choose Add files and select any small file from your computer (a .txt file works perfectly).
  • Choose Upload.
  • Wait for the upload to complete, then choose Close.

Before you check the result, what do you expect to see in the bucket after the upload completes?

You should see your test file listed inside the bucket with its name, size, and upload date.

Don't see your bucket or file?

Make sure you are looking in the correct AWS Region. The region selector is in the top-right of the console navigation bar. Your bucket only appears in the region where you created it.

If bucket creation failed, the name might already be taken. Try a different name with additional numbers or characters.

Help me troubleshoot my S3 bucket

Your AWS account is secured and your S3 bucket is ready. Next up, you will create a Lambda function that automatically fires every time a file lands in this bucket.

Create a Lambda Function and Wire It to S3

Your S3 bucket is set up and ready to receive files. Right now, uploading a file just stores it quietly. Nothing reacts to the upload.

AWS Lambda is a serverless compute service. It runs your code only when triggered, and you pay nothing when it is idle. In this step, you will create a Lambda function and connect it to your bucket so every file upload automatically fires your code.

In this step, get ready to:
  • Create a Lambda function with the Python 3.13 runtime.
  • Write Python code that extracts metadata from S3 upload events.
  • Wire the function to your S3 bucket so uploads trigger it automatically.
Create the Lambda function

Each Lambda function has a name, a runtime (the language it runs), and an IAM execution role that controls which AWS services it can access.

  • In the AWS Management Console, type Lambda in the top search bar and select Lambda from the results.
  • Choose Create function.
  • Select Author from scratch.
  • In the Basic information pane, enter FileProcessor for the Function name.
  • For Runtime, choose Python 3.13.
  • Expand Change default execution role. Under Execution role, choose Create a new role with basic Lambda permissions.
  • Choose Create function.

After a few seconds, the Lambda console loads your new function's detail page.

What did that just set up?

Lambda created your function and an IAM execution role named something like FileProcessor-role-xxxxxxxx. This role has the AWSLambdaBasicExecutionRole managed policy attached.

That policy only grants permission to write logs to CloudWatch Logs. Your function cannot access S3, DynamoDB, or any other service yet. This is least-privilege in action.

Add the imports and initialization code

The Lambda console includes a built-in code editor with a default handler file called lambda_function.py. You will replace the default code with your own Python code that reads S3 event data.

  • Choose the Code tab in the Lambda console.
  • Click inside the code editor and select all the existing code (Cmd+A on macOS or Ctrl+A on Windows).
  • Delete the selected code, then paste in the following imports and initialization:
import json
import urllib.parse
import boto3
from datetime import datetime, timezone

# Log when Lambda first loads this module
print('Loading function')

# Create an S3 client for retrieving file details later
s3 = boto3.client('s3')

What does this code do?

These lines run once when Lambda first loads your function into memory.

  • json and urllib.parse handle parsing the S3 event data. S3 URL-encodes object keys, so urllib.parse.unquote_plus decodes them back to readable file names.
  • boto3 is the AWS SDK for Python. It comes pre-installed in the Lambda Python runtime, so you do not need to install it separately.
  • s3 = boto3.client('s3') creates a reusable S3 client. Putting it outside the handler means Lambda reuses it across invocations instead of creating a new one each time.
  • Below the initialization code you just pasted, add the handler function that extracts file metadata from the S3 event:
def lambda_handler(event, context):
    # Log the full S3 event for debugging
    print("Received event: " + json.dumps(event, indent=2))

    # Extract file details from the S3 event record
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(
        event['Records'][0]['s3']['object']['key'], encoding='utf-8'
    )
    size = event['Records'][0]['s3']['object'].get('size', 0)
    event_time = event['Records'][0]['eventTime']

    # Log the extracted metadata
    print(f"File: {key}, Bucket: {bucket}, Size: {size}, Time: {event_time}")

    # Return a success response
    return {
        'statusCode': 200,
        'body': json.dumps(f"Received {key}")
    }

What does the handler do?

lambda_handler is the entry point Lambda calls every time your function runs. It receives two arguments: event (the S3 notification data) and context (runtime metadata from Lambda).

  • The function first logs the entire event as JSON so you can inspect it in CloudWatch Logs.
  • It then pulls four fields from the event: the bucket name, the object key (file name), the file size, and the timestamp of the upload.
  • Finally, it prints the extracted metadata and returns a success response.
  • In the DEPLOY section of the code editor, choose Deploy.

You should see a green banner confirming your changes have been deployed.

Don't see the Deploy button?

Make sure you are on the Code tab of your Lambda function. The Deploy button appears in the DEPLOY section of the code editor. If the button is greyed out, try making a small edit (add a space, then remove it) to trigger a change.

help me deploy my Lambda function code

✔️ Awesome, I've got everything!

Great. Make sure you have saved by clicking Deploy and seeing the green confirmation banner.

ⓧ I'd like to double check the full code

Your lambda_function.py should contain the following code. Make sure the imports and initialization are at the top, followed by the lambda_handler function.

import json
import urllib.parse
import boto3
from datetime import datetime, timezone

print('Loading function')

s3 = boto3.client('s3')

def lambda_handler(event, context):
    # Log the full S3 event for debugging
    print("Received event: " + json.dumps(event, indent=2))

    # Extract file details from the S3 event
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(
        event['Records'][0]['s3']['object']['key'], encoding='utf-8'
    )
    size = event['Records'][0]['s3']['object'].get('size', 0)
    event_time = event['Records'][0]['eventTime']

    print(f"File: {key}, Bucket: {bucket}, Size: {size}, Time: {event_time}")

    return {
        'statusCode': 200,
        'body': json.dumps(f"Received {key}")
    }
Add the S3 trigger and test the pipeline

Your code is deployed, but it will not run until something triggers it. An S3 trigger tells Lambda to fire your function every time a new file lands in a specific bucket.

  • Scroll up to the Function overview pane at the top of your Lambda function page.
  • Choose Add trigger.
  • Select S3 from the trigger type list.
  • Under Bucket, select the bucket you created in the previous step.
  • Under Event types, confirm that All object create events is selected.
  • Under Recursive invocation, select the check box to acknowledge the warning.
  • Choose Add.

What is the recursive invocation warning?

AWS warns you that if a Lambda function writes back to the same bucket that triggers it, it creates an infinite loop. Your function only reads from S3 and logs to CloudWatch, so this is safe. The checkbox confirms you understand the risk.

You should see S3 appear as a trigger in the Function overview diagram.

Now test the trigger by uploading a file to your bucket.

  • Open the S3 console in a new browser tab.
  • Choose your bucket from the list.
  • Choose Upload.
  • Choose Add files, select any small file from your computer, then choose Upload.

Before you check the logs, what do you think you will see? Will the function have run successfully, or will there be an error?

  • Switch back to the Lambda console tab with your FileProcessor function.
  • Choose the Monitor tab.
  • Choose View CloudWatch logs.
  • Under Log streams, choose the most recent log stream.

You should see a log entry starting with Received event: followed by the full S3 event JSON, including your bucket name and the file's object key. Further down, you will see the extracted metadata line showing the file name, bucket, size, and timestamp.

No log stream appearing?

  • Wait at least 30 seconds after uploading. S3 event notifications typically arrive within seconds, but can occasionally take up to a minute.
  • Confirm you are viewing CloudWatch Logs in the same AWS region where you created your Lambda function. Check the region dropdown in the top-right corner of the console.
  • Go back to the Lambda Function overview and confirm the S3 trigger shows under your function's triggers. If it is missing, add it again following the steps above.

help me find my Lambda function's CloudWatch logs

Your Lambda function is alive and reacting to S3 uploads. Next up, you will create a DynamoDB table to store file metadata permanently, so it does not vanish when you close the console.

Create a DynamoDB Table and Update the Lambda Code

Your Lambda function processes uploads and logs the event details to CloudWatch Logs. But that metadata disappears the moment you close the console. To make this pipeline useful, you need a database that stores file metadata permanently.

DynamoDB is AWS's serverless NoSQL database. It stores data as items (similar to rows) that you can query any time. In this step, you'll create a DynamoDB table, update your Lambda code to write metadata to it, and run straight into your first IAM permission challenge.

In this step, get ready to:
  • Create a DynamoDB table to store file metadata.
  • Update the Lambda function to write metadata to DynamoDB.
  • Hit and diagnose an IAM permission error in CloudWatch Logs.
Create the FileMetadata table in DynamoDB

Your pipeline needs somewhere to store the file name, size, content type, and upload timestamp for every file that lands in S3. A DynamoDB table with the file name as its primary key gives each record a unique identity.

  • Open the DynamoDB console by searching for DynamoDB in the AWS console search bar and selecting it.
  • In the left navigation pane, choose Tables.
  • Choose Create table.
  • For Table name, enter FileMetadata.
  • For Partition key, enter FileName and leave the type set to String.
  • Under Table settings, select Customize settings.
  • Under Read/write capacity settings, select On-demand.
  • Leave all other settings as defaults and choose Create table.

Why on-demand capacity mode?

On-demand mode means you pay per request instead of pre-setting how much read and write capacity you need. For a learning project with occasional uploads, this avoids the complexity of capacity planning and stays well within the free tier's 25 GB storage limit.

  • Wait until the table status changes to Active. This usually takes a few seconds. You can see the status on the Tables list page.

Table stuck in Creating status?

Refresh the page after 10-15 seconds. DynamoDB tables typically become active within a few seconds. If the status does not change after a minute, check that you are viewing the correct AWS region in the top-right corner of the console.

help me troubleshoot my DynamoDB table creation

Update the Lambda function to write metadata to DynamoDB

Your Lambda function currently logs event details and stops. You need to update it to connect to DynamoDB, look up the file's content type from S3, and write the full metadata as an item in the FileMetadata table.

  • Return to the Lambda console. Choose your FileProcessor function.
  • Choose the Code tab. In the Code source pane, select the lambda_function.py file.
  • Select all the existing code in the editor and delete it.
  • Copy and paste the following code into the editor. This is the first half of your updated function, covering the imports, DynamoDB connection setup, and content type lookup:
import json
import urllib.parse
import boto3
from datetime import datetime, timezone

print('Loading function')

# Set up clients for S3 and DynamoDB
s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('FileMetadata')


def lambda_handler(event, context):
    print("Received event: " + json.dumps(event, indent=2))

    # Extract file details from the S3 event
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(
        event['Records'][0]['s3']['object']['key'], encoding='utf-8'
    )
    size = event['Records'][0]['s3']['object'].get('size', 0)
    event_time = event['Records'][0]['eventTime']

    # Get the file's content type from S3
    try:
        response = s3.head_object(Bucket=bucket, Key=key)
        content_type = response['ContentType']
    except Exception as e:
        print(f"Error getting object metadata for {key} from bucket {bucket}: {e}")
        content_type = 'unknown'

What does this code do?

  • dynamodb = boto3.resource('dynamodb') creates a connection to DynamoDB using the boto3 resource interface. This is pre-installed in every Lambda Python runtime.
  • table = dynamodb.Table('FileMetadata') points to the table you just created. All writes go through this reference.
  • s3.head_object() calls S3 to look up the file's content type (e.g. image/png, text/plain). The S3 event itself does not include content type, so the code fetches it separately.
  • Now paste the second half of the function directly below what you just added. This writes the metadata to DynamoDB:
    # Write file metadata to DynamoDB
    try:
        table.put_item(
            Item={
                'FileName': key,
                'FileSize': size,
                'ContentType': content_type,
                'UploadTimestamp': event_time,
                'BucketName': bucket,
                'ProcessedAt': datetime.now(timezone.utc).isoformat()
            }
        )
        print(f"Successfully stored metadata for {key}")
    except Exception as e:
        print(f"Error storing metadata for {key}: {e}")
        raise e

    return {
        'statusCode': 200,
        'body': json.dumps(f"Processed {key}")
    }

What does this code do?

  • table.put_item() writes a new item (row) to the FileMetadata table. Each item contains the file name, size, content type, upload timestamp, bucket name, and the time it was processed.
  • raise e re-raises the exception so the error appears in CloudWatch Logs. Without this, the function would silently swallow the failure.
  • The return statement sends a 200 response with a confirmation message. Lambda uses this as the function's output.
  • In the DEPLOY section of the code editor, choose Deploy to save your updated code.

You should see a confirmation banner that says your function was updated successfully.

✔️ Awesome, I've got everything!

Great. Make sure you see the success banner confirming your function was updated.

ⓧ I'd like to double check the full code

Your complete lambda_function.py should contain two sections you pasted above, in order. Scroll up and confirm:

  • The first block starts with import json and ends with content_type = 'unknown'.
  • The second block starts with the # Write file metadata to DynamoDB comment and ends with the return statement.
  • There is one blank line between the two blocks (no extra gaps or missing lines).

Deploy button greyed out?

Make sure you are editing the lambda_function.py file, not a different tab. If you accidentally created a new file, delete it and paste your code into the original lambda_function.py file.

help me deploy my updated Lambda function code

Upload a file and hit the permission error

Your code is deployed and ready. Time to test it by uploading a file to S3 and checking what happens. Before you run this test, think about what you expect to see in CloudWatch Logs.

Your Lambda function now tries to call table.put_item() to write to DynamoDB. But the function's IAM role only has AWSLambdaBasicExecutionRole attached. That policy only grants CloudWatch Logs write access. Do you think the DynamoDB write will succeed or fail?

  • Switch back to the S3 console. Open your bucket.
  • Choose Upload, select any small file (a text file, image, or PDF), and choose Upload.
  • Wait about 30 seconds for the Lambda function to trigger.
  • Return to the Lambda console. Select your FileProcessor function.
  • Choose the Monitor tab, then choose View CloudWatch logs.
  • Open the most recent log stream (the one at the top).

You will see an error message containing AccessDeniedException and text that says the function is not authorized to perform dynamodb:PutItem on the FileMetadata table.

This is exactly what you should see. This is not a bug in your code. This is least-privilege permissions working as designed.

Why did this happen?

When you created the Lambda function, AWS attached the AWSLambdaBasicExecutionRole managed policy to its execution role. This policy only grants permission to write to CloudWatch Logs. It does not include any DynamoDB permissions.

AWS follows the principle of least privilege. Your function starts with the minimum permissions it needs, and you add more only when required. This error-then-fix loop is the same troubleshooting pattern cloud engineers follow every day.

Your pipeline has all the pieces in place, but it is blocked by a missing permission. In the next step, you will fix the IAM role, re-test, and watch the full pipeline work end to end.

Fix the IAM Permissions and Complete the Pipeline

In the last step, your pipeline hit a wall. The Lambda function tried to write file metadata to DynamoDB, but CloudWatch Logs revealed an AccessDeniedException. The function's IAM role only has permission to write logs. It has zero permission to touch DynamoDB.

This is the core cloud engineering troubleshooting loop: attempt something, hit a permission error, read the logs, identify the missing permission, and fix it with the narrowest possible access. You'll add an inline policy that grants your Lambda role only the specific DynamoDB actions it needs, on only the specific table it uses.

In this step, get ready to:
  • Add an inline IAM policy granting DynamoDB write access to your Lambda execution role.
  • Upload a file and verify the pipeline completes without errors in CloudWatch Logs.
  • Confirm file metadata appears in the DynamoDB table.
Add an inline IAM policy for DynamoDB access

The error message in CloudWatch Logs told you exactly what's missing: permission to perform dynamodb:PutItem. You'll create an inline policy scoped to just the FileMetadata table.

  • Open the AWS Management Console search bar and type IAM.
  • Select IAM from the results.
  • Click Roles in the left navigation pane.
  • Search for FileProcessor in the search box.
  • Click on the role name (it looks like FileProcessor-role-xxxxxxxx with random characters at the end).

You should see the Permissions tab with AWSLambdaBasicExecutionRole listed. That's the only policy this role has right now.

  • Click the Add permissions dropdown.
  • Select Create inline policy.
  • Click the JSON tab in the policy editor.
  • Clear the default policy and paste the following JSON:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:GetItem"
            ],
            "Resource": "arn:aws:dynamodb:*:*:table/FileMetadata"
        }
    ]
}

What does this policy do?

This policy follows the principle of least privilege. It grants only two specific actions: dynamodb:PutItem (write a record) and dynamodb:GetItem (read a record). No delete, no scan, no admin access.

The Resource field scopes these permissions to only the FileMetadata table. Your Lambda function cannot touch any other table in your account.

Why an inline policy instead of a managed policy?

AWS offers broad managed policies like AmazonDynamoDBFullAccess that grant permissions to every DynamoDB table. An inline policy lets you scope access to exactly one table and exactly two actions. In production, this is how cloud engineers keep blast radius small.

  • Click Next.
  • Enter DynamoDBFileMetadataAccess in the Policy name field.
  • Click Create policy.

You should now see DynamoDBFileMetadataAccess listed alongside AWSLambdaBasicExecutionRole on the role's Permissions tab.

Don't see the new policy listed?

Make sure you clicked Create policy (not just Next). Refresh the role's page if needed. The inline policy should appear under the Permissions policies section.

If you see a syntax error, double check that you pasted valid JSON and that the Resource value matches arn:aws:dynamodb:*:*:table/FileMetadata exactly.

Help me troubleshoot my inline IAM policy.

Test the pipeline with a new upload

The permission gap is closed. Time to find out if the full pipeline works end-to-end.

  • Navigate back to the S3 console by searching for S3 in the AWS search bar.
  • Open your bucket (file-pipeline-yourname-2026).
  • Click Upload, then Add files.
  • Choose any small file from your computer (a text file, image, or PDF).
  • Click Upload.

Before you check the logs, take a moment to predict: do you think the upload will succeed this time, or will you see another AccessDeniedException? Why?

  • Navigate to the Lambda console by searching for Lambda in the AWS search bar.
  • Click on your FileProcessor function.
  • Click the Monitor tab.
  • Click View CloudWatch logs.
  • Click the most recent log stream (the one at the top of the list).

This time you should see Successfully stored metadata for followed by your file name. No AccessDeniedException in sight. The inline policy fixed the permission gap.

Still seeing AccessDeniedException?

IAM policy changes can take a few seconds to propagate. Wait 30 seconds and upload a new file to trigger a fresh Lambda invocation. Check the newest log stream after the wait.

If the error persists, return to the IAM console and confirm the inline policy DynamoDBFileMetadataAccess is attached to your FileProcessor role. Verify the Action array includes dynamodb:PutItem and the Resource matches arn:aws:dynamodb:*:*:table/FileMetadata.

Help me debug my Lambda permission error.

Verify metadata in DynamoDB

CloudWatch confirmed the write succeeded. Now check that the actual data landed in your DynamoDB table.

  • Open the DynamoDB console by searching for DynamoDB in the AWS search bar.
  • Click Tables in the left navigation pane.
  • Click on the FileMetadata table.
  • Click Explore table items.

In the Items returned section, you should see a new item with attributes including FileName, FileSize, ContentType, and UploadTimestamp. Your pipeline is fully wired.

What do these attributes mean?

  • FileName: The object key from S3 (your uploaded file's name).
  • FileSize: The file size in bytes, pulled from the S3 event.
  • ContentType: The MIME type of the file (e.g. text/plain, image/jpeg).
  • UploadTimestamp: The time the S3 event fired, recorded by AWS.
  • BucketName: Which S3 bucket the file was uploaded to.
  • ProcessedAt: The UTC timestamp when your Lambda function processed the event.

Now stress-test the pipeline with a few more files to confirm it handles different types.

  • Return to the S3 console and open your bucket.
  • Upload two or three more files of different types and sizes (for example, a .jpg image and a .pdf document).
  • Wait 30 seconds after each upload.
  • Return to the DynamoDB console, open the FileMetadata table, and click Explore table items.
  • Verify that each uploaded file has created a new item with the correct metadata.

You should see one item per file, each with its own FileName, FileSize, and ContentType values.

Not seeing new items in the table?

Click the refresh icon (circular arrow) in the Items returned section. DynamoDB items appear immediately after a successful PutItem call, but the console view may need a manual refresh.

If items still don't appear, check CloudWatch Logs for the latest invocation. Look for any error messages that indicate what went wrong.

Help me troubleshoot missing DynamoDB items.

Secret mission

Add File Type Filtering and User Identity Tracking

Your pipeline processes every file the same way. Real-world pipelines are smarter. In this mission, you'll add logic so only image uploads get processed, and each DynamoDB record tracks who uploaded the file.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses serverless AWS resources that fall within the Always Free tier, so there are no ongoing costs while they sit idle.

Resources you used:

  • S3 bucket (your file-pipeline bucket).
  • Lambda function (FileProcessor) with S3 trigger.
  • DynamoDB table (FileMetadata).
  • IAM role (FileProcessor-role-xxxxxxxx) with inline policy DynamoDBFileMetadataAccess.
  • CloudWatch Logs log group (/aws/lambda/FileProcessor).

Keep everything running

No action needed. Choose this if you want to experiment further or revisit this pipeline later.

Your S3 bucket, Lambda function, DynamoDB table, and CloudWatch Logs all fall within the AWS Always Free tier. No charges accrue while these serverless resources sit idle. On the Free plan, your account closes automatically after 6 months or when your $100 credits run out, whichever comes first.

Pause - I'll come back to this later

These are all serverless resources. They automatically scale to zero when not in use and only incur costs when triggered. There is nothing to pause.

If you want to prevent accidental invocations, you can remove the S3 trigger from the Lambda function so uploads no longer fire the pipeline. Your code and table remain intact for when you return.

  • Open the Lambda console and select the FileProcessor function.
  • Choose the Configuration tab, then choose Triggers.
  • Select the S3 trigger and choose Delete.

Delete - I don't want to use this again

Remove all project resources and start fresh. Delete them in this order to avoid dependency issues.

Delete the Lambda function
  • Open the Functions page of the Lambda console.
  • Select the FileProcessor function.
  • Choose Actions, then Delete.
  • Type confirm in the text field and choose Delete.
Empty and delete the S3 bucket
  • Open the Amazon S3 console.
  • In the left navigation pane, choose General purpose buckets.
  • Select your file-pipeline bucket and choose Empty.
  • Type permanently delete to confirm, then choose Empty.
  • Return to the bucket list, select your bucket again, and choose Delete.
  • Enter the bucket name in the text field and choose Delete bucket.
Delete the DynamoDB table
  • Open the DynamoDB console.
  • In the left navigation pane, choose Tables.
  • Select the FileMetadata table and choose Delete.
  • Enter delete in the text box and choose Delete table.
Delete the CloudWatch Logs log group
  • Open the Log groups page of the CloudWatch console.
  • Select the log group named /aws/lambda/FileProcessor.
  • Choose Actions, then Delete log group(s).
  • In the confirmation dialog, choose Delete.
Delete the IAM execution role
  • Open the Roles page of the IAM console.
  • Search for FileProcessor-role and select the execution role.
  • Choose Delete.
  • Enter the role name in the text field and choose Delete.

What about my AWS Free plan account?

Your Free plan account closes automatically after 6 months or when your $100 credits run out, whichever comes first. No charges are incurred unless you upgrade to a Paid plan.

Nice Work!

Nice Work!

Nice work! You just built a fully wired serverless pipeline on AWS. Drop a file into your bucket and its metadata appears in your database seconds later.

You've learned how to:

  • Wire an S3 bucket, a Lambda function, and a DynamoDB table into a working event-driven pipeline that processes file uploads automatically.
  • Diagnose a real IAM permission failure (AccessDeniedException) in CloudWatch Logs and fix it by adding a least-privilege inline policy.
  • Write Python code that runs serverlessly in Lambda, extracts file metadata from S3 events, and stores it in DynamoDB.
  • Secret Mission: Add file type filtering so only image uploads get processed, and enrich each record with the uploader's identity pulled from the S3 event.

Ready to quiz yourself?