Build an AWS Upload Audit Pipeline

Use Terraform and Python to record S3 uploads in DynamoDB.

Introduction

30 Second Summary

A file can enter a shared system in seconds. Without an automatic check-in trail, teams struggle to confirm what arrived or when it appeared.

In this project, you will build a serverless upload audit pipeline with Terraform and Python. Each upload to Amazon S3 triggers AWS Lambda to create an audit record in Amazon DynamoDB.

What You'll Build

Picture uploading sample.txt to your private bucket, then seeing its file name, size, event time, event type, bucket, and sequencer appear automatically in the audit table.

By the end of this project, you'll have:

  • A live upload audit that turns each file upload into a matching database record you can inspect.
  • A manual worker check that proves a realistic S3 event reaches the Python function before you connect the live trigger.
  • A mocked Terraform contract test that protects the approved Lambda runtime and DynamoDB billing mode without creating another AWS stack.
  • Secret Mission: Restrict the event trigger so only files uploaded under incoming/ create audit records.

Are there any prerequisites?

You need Terraform CLI 1.7.0 or newer. You also need AWS CLI v2 plus Python 3.10 or newer with pip.

Your active AWS identity must be able to create the S3, DynamoDB, Lambda, IAM, and CloudWatch Logs resources used by the pipeline.

Before We Start

This opening checkpoint connects your upload audit pipeline to a real organizational need, such as tracking invoices, employee documents, media assets, data files, or customer submissions. Automatic records provide traceability, compliance evidence, and operational visibility.

Verify and Initialize Your Local Environment

A serverless pipeline can create cloud resources quickly. An outdated Terraform binary can block the mocked tests used later.

An unintended AWS principal can place resources in the wrong account. Verifying the active identity now keeps the deployment attached to the account you expect.

You will also pin the Terraform provider versions before any infrastructure exists. Initialization then proves that your local environment can download those exact providers.

In this step, get ready to:
  • Confirm that your Terraform CLI is version 1.7.0 or newer.
  • Verify that the AWS CLI uses the intended account.
  • Initialize a Visual Studio Code project with pinned providers.
Verify your Terraform version and AWS identity

Terraform provider mocking requires version 1.7.0 or newer. The AWS identity check reveals the account that receives any resources you deploy.

  • Press Cmd+Space on macOS to open the search bar.
  • Type Terminal into the search bar.
  • Press Enter to open Terminal.
  • Check your installed Terraform version by running this command:
terraform version

What does this command show?

The command prints the installed Terraform binary version. It also lists any providers already installed in the current folder.

✔️ I see version 1.7.0 or higher

That check is clear. Your Terraform binary supports the mocked provider tests used later in the project.

ⓧ I see an older version

Your installed version needs an upgrade before this project can use provider mocking.

  • Open HashiCorp's official Terraform installation guide.
  • Choose the macOS upgrade method that matches your existing installation.
  • Restart Terminal after the upgrade completes.
  • Repeat the version check above.

ⓧ Command not found

Terminal cannot find a Terraform binary on your system.

  • Open HashiCorp's official Terraform installation guide.
  • Choose the macOS installation method for your machine.
  • Complete the installation.
  • Restart Terminal after the installation completes.
  • Repeat the version check above.
  • Reveal the AWS account used by your current CLI credentials by running this command:
aws sts get-caller-identity

What does this command show?

  • The UserId field identifies the current caller.
  • The Account field shows the AWS account number.
  • The Arn field names the active AWS principal.
  • The command reads identity metadata without creating resources.
  • Compare the Account value with the AWS account you intend to use.
  • Confirm that the Arn value names the intended principal.

You have removed the biggest account-level surprise. Every later deployment now has a confirmed destination.

Identity points to the wrong account?

Correcting the active AWS profile changes which saved credentials the CLI uses. It does not modify any cloud resources.

  • Switch the AWS CLI to the credentials for your intended account.
  • Repeat the identity check above.

Need help selecting the correct credentials? Help me verify my active AWS CLI identity.

Create the project folder and pin providers

A dedicated project folder keeps the Terraform configuration beside its generated files. Version constraints ensure that every initialization selects the same provider releases.

  • Return to your Desktop by running this command in Terminal:
cd ~/Desktop

What does this command do?

The command changes Terminal's current location to your Desktop. Your new project folder will be easy to find there.

  • Create the upload-audit-pipeline folder and move into it by running these commands:
mkdir upload-audit-pipeline
cd upload-audit-pipeline

What do these commands do?

  • The first command creates upload-audit-pipeline on your Desktop.
  • The second command moves Terminal into that folder.
  • Confirm that Terminal is inside the new folder by running this command:
pwd

What does this command show?

The command prints the full path of Terminal's current folder. The path should end with Desktop/upload-audit-pipeline.

Project folder path looks wrong?

  • Repeat the Desktop navigation command above.
  • Confirm that any existing upload-audit-pipeline folder is empty before continuing.
  • Repeat the folder creation commands above.

Still stuck? Help me create the project folder in the correct location.

  • Press Cmd+Space on macOS or the Windows key on Windows to open system search.
  • Type Visual Studio Code into system search.
  • Press Enter to open Visual Studio Code.
  • Select File from the menu bar.
  • Select Open Folder....
  • Select your Desktop folder.
  • Select the upload-audit-pipeline folder.
  • Click Open.

Your upload-audit-pipeline folder now appears at the top of the Explorer. The editor is ready for the root configuration.

  • Select the New File action in the Explorer.
  • Enter versions.tf as the file name.
  • Paste the following configuration into versions.tf:
terraform {
  required_version = ">= 1.7.0, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 6.66.0"
    }

    archive = {
      source  = "hashicorp/archive"
      version = "= 2.8.1"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

What does this configuration do?

  • The required_version constraint accepts Terraform from 1.7.0 up to the next major release.
  • The aws entry pins HashiCorp AWS Provider 6.66.0.
  • The archive entry pins HashiCorp Archive Provider 2.8.1.
  • The provider "aws" block reads the deployment Region from var.aws_region.
  • Press Cmd+S on macOS or Ctrl+S on Windows to save versions.tf.
  • Confirm that versions.tf appears in the Explorer.

Provider configuration showing an error?

  • Confirm that every brace in versions.tf has a matching closing brace.
  • Compare each provider source with the reference below.
  • Confirm that the file name ends in .tf.

Need another pair of eyes? Help me check my Terraform provider configuration.

✔️ Awesome, I've got everything!

Your versions.tf file is saved with the required Terraform and provider versions.

ⓧ I'd like to double check the full code

terraform {
  required_version = ">= 1.7.0, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 6.66.0"
    }

    archive = {
      source  = "hashicorp/archive"
      version = "= 2.8.1"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

Initialization creates local downloads and generated artifacts. A .gitignore file keeps those files out of Git history.

Dotfiles are easy to overlook because their names start with a period. Visual Studio Code keeps this one visible in the Explorer.

  • Select the New File action in the Explorer.
  • Enter .gitignore as the file name.
  • Paste the following rules into .gitignore:
.terraform/
*.tfstate
*.tfstate.*
tfplan
response.json
modules/upload_audit/lambda/function.zip
modules/upload_audit/lambda/package/*
!modules/upload_audit/lambda/package/handler.py

What do these rules protect?

  • The .terraform/ rule excludes downloaded provider files.
  • The state patterns exclude local Terraform state files.
  • The plan and response rules exclude generated command outputs.
  • The Lambda package rules exclude build artifacts while preserving handler.py.
  • Press Cmd+S on macOS or Ctrl+S on Windows to save .gitignore.
  • Confirm that .gitignore appears beside versions.tf in the Explorer.

Git ignore file missing?

  • Confirm that the file name begins with a period.
  • Remove any automatic text-file extension from the name.
  • Save the file inside upload-audit-pipeline.

Still cannot see the file? Help me fix my .gitignore file name and location.

✔️ Awesome, I've got everything!

Your .gitignore file now protects the generated Terraform and Lambda artifacts.

ⓧ I'd like to double check the full code

.terraform/
*.tfstate
*.tfstate.*
tfplan
response.json
modules/upload_audit/lambda/function.zip
modules/upload_audit/lambda/package/*
!modules/upload_audit/lambda/package/handler.py
Initialize Terraform and verify the providers

Initialization turns each provider pin into an installed plugin. Terraform also records the selected packages in a local dependency lock file.

  • Select View from the Visual Studio Code menu bar.
  • Select Terminal.

The integrated terminal starts inside upload-audit-pipeline. The first initialization can take about a minute while Terraform downloads both providers.

Before you initialize, which provider versions do you expect Terraform to install?

  • Initialize the project by running this command:
terraform init

What does this command do?

  • Terraform reads the provider constraints from versions.tf.
  • Terraform downloads the selected providers into .terraform/.
  • Terraform records provider checksums in .terraform.lock.hcl.
  • Initialization creates no AWS resources.

You should see Terraform install hashicorp/aws at version 6.66.0.

You should also see Terraform install hashicorp/archive at version 2.8.1.

That is the local foundation finished. Your CLI version, AWS identity, project files, and provider installations are now confirmed.

Initialization did not complete?

  • Confirm that your internet connection allows Terraform to download providers.
  • Compare versions.tf with the full reference above.
  • Repeat the initialization command after correcting the configuration.

Need help reading the output? Help me troubleshoot my Terraform initialization.

Your local environment is ready for infrastructure work. Next, you will deploy the storage foundation and test what happens before an audit worker exists.

Deploy the Storage Foundation

Your AWS identity is verified. Your Terraform environment is also initialized with the pinned providers.

A private Amazon S3 bucket gives uploaded files somewhere to land. An Amazon DynamoDB table gives audit records somewhere to live. Your first upload will reveal whether storage can create those records on its own.

In this step, get ready to:
  • Build a reusable Terraform module for the storage foundation.
  • Deploy a private upload bucket with an on-demand audit table.
  • Upload a sample file to test what the storage layer does by itself.
Wire the reusable module

A reusable module groups the resources that form one capability. This module represents the upload audit pipeline as a single unit that later steps can extend.

  • Create variables.tf in the open VS Code project folder using the file sidebar.
  • Paste the root input variables into variables.tf using this code:
variable "aws_region" {
  description = "AWS Region for the upload audit pipeline."
  type        = string
  default     = "us-east-1"
}

variable "project_name" {
  description = "Name prefix for the upload audit resources."
  type        = string
  default     = "upload-audit"
}

What do these variables control?

  • The aws_region variable keeps every resource in us-east-1.
  • The project_name variable supplies the shared upload-audit name prefix.
  • Save variables.tf.
  • Check the root configuration by running this command in the existing terminal:
terraform validate

What does this check prove?

Terraform checks that the existing provider configuration can now resolve var.aws_region. A successful validation confirms the root inputs are connected correctly.

Good progress. Your root provider now has the regional input it needs.

Configuration not valid?

  • Confirm that variables.tf sits beside versions.tf in the open project folder.
  • Check that the variable name is exactly aws_region.

Still stuck? Help me troubleshoot why Terraform cannot validate my root variables.

  • Create the nested folder path modules/upload_audit using the VS Code file sidebar.
  • Create versions.tf inside modules/upload_audit.
  • Paste the module provider requirements into modules/upload_audit/versions.tf using this code:
terraform {
  required_version = ">= 1.7.0, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 6.66.0"
    }

    archive = {
      source  = "hashicorp/archive"
      version = "= 2.8.1"
    }
  }
}

Why pin providers inside the module?

The module declares the Terraform range that it supports. It also declares exact provider versions so the module uses the same APIs as the root configuration.

  • Save modules/upload_audit/versions.tf.
  • Confirm that versions.tf is listed inside the modules/upload_audit folder in the file sidebar.
  • Create variables.tf inside modules/upload_audit using the VS Code file sidebar.
  • Paste the module input into modules/upload_audit/variables.tf using this code:
variable "project_name" {
  description = "Name prefix for resources in the upload audit pipeline."
  type        = string
  default     = "upload-audit"
}

How does the module receive its name?

The module accepts project_name as an input. Its default keeps resource names consistent when the module is used on its own.

  • Save modules/upload_audit/variables.tf.
  • Create main.tf in the open project folder using the file sidebar.
  • Paste the root module call into main.tf using this code:
module "upload_audit" {
  source = "./modules/upload_audit"

  project_name = var.project_name
}

What does the module call do?

The source value points Terraform to the local module folder. The root project_name value flows into that module.

  • Save main.tf.
  • Initialize the new module reference by running these commands:
terraform init
terraform validate

What happens during initialization?

Terraform discovers the local module. It keeps the installed AWS Provider at 6.66.0 and the Archive Provider at 2.8.1.

The validation command then checks the root configuration together with the module.

You should see initialization complete followed by a successful configuration validation.

Module not found?

  • Confirm that the module folder path is exactly modules/upload_audit.
  • Confirm that the root module source is exactly ./modules/upload_audit.
  • Return to the terminal opened in the project folder before running the commands again.

Need another pair of eyes? Help me diagnose why Terraform cannot initialize my local upload_audit module.

Define the storage resources

The upload bucket needs a generated name because S3 bucket names must be unique. The public-access settings keep the learning bucket private.

  • Create main.tf inside modules/upload_audit using the VS Code file sidebar.
  • Paste the shared tags with the private bucket resources into modules/upload_audit/main.tf using this code:
locals {
  common_tags = {
    Project   = var.project_name
    ManagedBy = "Terraform"
  }
}

resource "aws_s3_bucket" "uploads" {
  bucket_prefix = "${var.project_name}-uploads-"
  force_destroy = true

  tags = local.common_tags
}

resource "aws_s3_bucket_public_access_block" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

How is the bucket protected?

  • The bucket_prefix setting lets AWS add a unique suffix to the project prefix.
  • The four public-access settings block public policies and public access control lists.
  • The force_destroy setting lets Terraform remove the disposable learning bucket even when test files remain inside it.
  • The shared tags identify resources managed by this project.
  • Save modules/upload_audit/main.tf.
  • Format the configuration by running these commands:
terraform fmt -recursive
terraform validate

What do these checks cover?

The formatting command normalizes Terraform files throughout the project. The validation command confirms that the bucket references resolve correctly.

You should see a successful validation after Terraform checks the bucket configuration.

Bucket configuration rejected?

  • Check that both bucket resources use the identifier uploads.
  • Check that the public-access block references aws_s3_bucket.uploads.id.

Still seeing a validation problem? Help me compare my S3 bucket resources with the expected Terraform configuration.

The audit table uses on-demand request billing. This keeps the table aligned with a small event-driven workload that has no fixed traffic level.

  • Add the DynamoDB table below the public-access block in modules/upload_audit/main.tf using this code:
resource "aws_dynamodb_table" "audit" {
  name         = "${var.project_name}-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "event_id"

  attribute {
    name = "event_id"
    type = "S"
  }

  tags = local.common_tags
}

How is the audit table structured?

  • The table name resolves to upload-audit-events with the default project name.
  • The PAY_PER_REQUEST billing mode charges for actual requests without reserved throughput.
  • The string attribute event_id becomes the table's partition key.
  • Save modules/upload_audit/main.tf.
  • Format the updated module by running these commands:
terraform fmt -recursive
terraform validate

What does validation prove now?

Terraform checks the bucket and table as one module. A successful result confirms that the table key schema is internally consistent.

You should see a successful validation with the complete storage foundation in place.

Table validation failing?

  • Confirm that both key declarations use event_id.
  • Confirm that the attribute type is the uppercase value S.
  • Confirm that the billing mode is exactly PAY_PER_REQUEST.

Need help finding the mismatch? Help me troubleshoot my DynamoDB table configuration in Terraform.

Deploy and reveal the missing automation

Terraform outputs provide stable names for resources that receive generated identifiers. Later commands can read those values directly from state.

  • Create outputs.tf inside modules/upload_audit using the VS Code file sidebar.
  • Paste the module outputs into modules/upload_audit/outputs.tf using this code:
output "upload_bucket_name" {
  description = "Name of the S3 upload bucket."
  value       = aws_s3_bucket.uploads.id
}

output "audit_table_name" {
  description = "Name of the DynamoDB audit table."
  value       = aws_dynamodb_table.audit.name
}

Why expose module outputs?

These outputs expose the generated bucket name and the audit table name to the root configuration. The module keeps its internal resource references contained.

  • Save modules/upload_audit/outputs.tf.
  • Check the new module outputs by running this command:
terraform validate

What does this validation cover?

Terraform confirms that each module output refers to a resource declared inside the same module. A successful result proves that the output references are valid.

  • Create outputs.tf in the open project folder using the file sidebar.
  • Paste the root outputs into outputs.tf using this code:
output "aws_region" {
  description = "AWS Region containing the pipeline."
  value       = var.aws_region
}

output "upload_bucket_name" {
  description = "Name of the S3 upload bucket."
  value       = module.upload_audit.upload_bucket_name
}

output "audit_table_name" {
  description = "Name of the DynamoDB audit table."
  value       = module.upload_audit.audit_table_name
}

How do the root outputs help?

The root outputs give terminal commands access to the deployment region and resource names. This removes the need to copy generated identifiers by hand.

  • Save outputs.tf.
  • Format the project by running these commands:
terraform fmt -recursive
terraform validate

What should pass now?

Terraform formats every configuration file before validating the root outputs and module outputs together. You should see a successful validation.

  • Create sample.txt in the open project folder using the file sidebar.
  • Paste the sample upload content into sample.txt using this text:
NextWork Terraform upload audit demo.

What is this sample file for?

This small text file gives the new bucket a concrete upload to store. Its arrival becomes the first test of the audit workflow.

  • Save sample.txt.
  • Confirm that sample.txt appears beside the root Terraform files in the file sidebar.

Use the comparison tabs before deployment if you want to check every project file.

✔️ Awesome, I've got everything!

Great. Double-check that every file is saved before you create the plan.

ⓧ I'd like to double check the full code

.terraform/
*.tfstate
*.tfstate.*
tfplan
response.json
modules/upload_audit/lambda/function.zip
modules/upload_audit/lambda/package/*
!modules/upload_audit/lambda/package/handler.py

terraform {
  required_version = ">= 1.7.0, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 6.66.0"
    }

    archive = {
      source  = "hashicorp/archive"
      version = "= 2.8.1"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

variable "aws_region" {
  description = "AWS Region for the upload audit pipeline."
  type        = string
  default     = "us-east-1"
}

variable "project_name" {
  description = "Name prefix for the upload audit resources."
  type        = string
  default     = "upload-audit"
}

module "upload_audit" {
  source = "./modules/upload_audit"

  project_name = var.project_name
}

output "aws_region" {
  description = "AWS Region containing the pipeline."
  value       = var.aws_region
}

output "upload_bucket_name" {
  description = "Name of the S3 upload bucket."
  value       = module.upload_audit.upload_bucket_name
}

output "audit_table_name" {
  description = "Name of the DynamoDB audit table."
  value       = module.upload_audit.audit_table_name
}

NextWork Terraform upload audit demo.

terraform {
  required_version = ">= 1.7.0, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 6.66.0"
    }

    archive = {
      source  = "hashicorp/archive"
      version = "= 2.8.1"
    }
  }
}

variable "project_name" {
  description = "Name prefix for resources in the upload audit pipeline."
  type        = string
  default     = "upload-audit"
}

locals {
  common_tags = {
    Project   = var.project_name
    ManagedBy = "Terraform"
  }
}

resource "aws_s3_bucket" "uploads" {
  bucket_prefix = "${var.project_name}-uploads-"
  force_destroy = true

  tags = local.common_tags
}

resource "aws_s3_bucket_public_access_block" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "audit" {
  name         = "${var.project_name}-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "event_id"

  attribute {
    name = "event_id"
    type = "S"
  }

  tags = local.common_tags
}

output "upload_bucket_name" {
  description = "Name of the S3 upload bucket."
  value       = aws_s3_bucket.uploads.id
}

output "audit_table_name" {
  description = "Name of the DynamoDB audit table."
  value       = aws_dynamodb_table.audit.name
}

Applying this configuration creates AWS resources that can incur storage and request charges. This test uses only a small file, while the cleanup section shows how to remove the complete environment.

  • Create a saved deployment plan by running this command:
terraform plan -out="tfplan"

What does the saved plan contain?

The plan calculates the bucket and table changes before anything is created. Saving it as tfplan ensures the reviewed plan is the one Terraform applies.

  • Review the plan for the upload bucket resource.
  • Review the plan for the public-access block resource.
  • Review the plan for the audit table resource.
  • Apply the reviewed plan by running this command:
terraform apply "tfplan"

What did Terraform create?

Terraform creates the private bucket and its public-access controls. It also creates the upload-audit-events table in us-east-1.

That's a meaningful milestone. Your reusable storage foundation is now live in the intended AWS account.

Deployment did not complete?

  • Confirm that the terminal still uses the verified AWS identity from the previous step.
  • Check the plan output for a permission problem involving bucket or table creation.
  • Run the planning command again after correcting any configuration error.

Need help with the failed deployment? Help me diagnose why my Terraform storage plan or apply failed in AWS.

  • Upload sample.txt to the bucket root by running this command:
aws s3 cp sample.txt "s3://$(terraform output -raw upload_bucket_name)/sample.txt"

What does this upload command use?

The nested Terraform output command reads the generated bucket name from state. The AWS CLI copies sample.txt to the root of that bucket.

You should see confirmation that sample.txt was copied to the generated bucket.

Before you scan the table, do you expect the upload to create an audit record by itself?

  • Scan the audit table with a consistent read by running this command:
aws dynamodb scan --table-name "$(terraform output -raw audit_table_name)" --consistent-read --region "$(terraform output -raw aws_region)"

What did the scan reveal?

The scan reports Count as 0. The file reached S3, while no component reacted to the upload or wrote an audit item.

This empty table is the intended shortfall. The storage foundation needs a processor before uploads can become audit records.

Upload or scan failed?

  • Confirm that the Terraform apply completed before retrying the upload.
  • Confirm that sample.txt is saved in the open project folder.
  • Confirm that the terminal still uses the intended AWS account.

Still blocked? Help me troubleshoot my S3 upload or DynamoDB scan command.

Your upload bucket and audit table are deployed. Next, you'll package a Python worker that turns an upload event into a DynamoDB record.

Package and Run the Python Audit Worker

Your private Amazon S3 bucket can store uploads. Your Amazon DynamoDB table can store audit records. The empty table proved that neither service reacts to an upload by itself.

This step adds a Python worker on AWS Lambda. You will package its dependencies. You will give it permission to write audit records. A manual event will prove the worker works before you connect the live upload trigger.

In this step, get ready to:
  • Package the Python handler with a pinned Boto3 dependency.
  • Deploy the Lambda function with its execution role and log group.
  • Invoke the worker manually and confirm it writes an audit record.
Package the Python worker

A Lambda deployment package contains the handler plus every dependency it imports. You will package Boto3 version 1.43.102 so the function uses the version you selected.

Why package Boto3?

Lambda includes Boto3 in its runtime. Packaging your selected version keeps the function's dependency set under your control. It also reduces the risk of a runtime update creating a version mismatch.

  • In the Visual Studio Code sidebar, create a folder named lambda inside modules/upload_audit.
  • Inside modules/upload_audit/lambda, create a folder named package.
  • Inside modules/upload_audit/lambda, create a file named requirements.txt.
  • Add the pinned dependency to requirements.txt by copying this line:
boto3==1.43.102

What does this dependency file do?

The exact pin makes future package builds use the same Boto3 release. Reproducible dependencies keep the worker consistent across machines.

  • Save modules/upload_audit/lambda/requirements.txt.
  • Install the pinned dependency into the deployment package by running:
python3 -m pip install --target modules/upload_audit/lambda/package -r modules/upload_audit/lambda/requirements.txt

What does this command do?

  • The target option places installed files inside modules/upload_audit/lambda/package.
  • The requirements option reads the exact dependency pin from modules/upload_audit/lambda/requirements.txt.
  • The resulting directory becomes the root of the Lambda deployment archive.

The installation output should finish successfully. You should also see Boto3 dependency files inside modules/upload_audit/lambda/package.

Dependency installation failed?

Confirm your terminal is still in the project folder that contains modules. Check that requirements.txt contains the exact dependency pin.

If Python reports an environment-specific installation problem, help me troubleshoot the package installation.

✔️ Awesome, I've got everything!

Your dependency file is saved. Boto3 is installed in the deployment package.

ⓧ I'd like to double check the full code

boto3==1.43.102

This reference is the complete requirements.txt file for this step.

A realistic event lets you test the worker before S3 is allowed to invoke it. Its object key is URL encoded so the handler can prove that it decodes uploaded paths correctly.

  • In the Visual Studio Code sidebar, create a folder named events inside the project folder.
  • Inside events, create a file named manual-s3.json.
  • Add the manual S3 event to events/manual-s3.json by copying this code:
{
  "Records": [
    {
      "eventVersion": "2.6",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "2026-09-25T12:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "s3SchemaVersion": "1.0",
        "bucket": {
          "name": "manual-demo"
        },
        "object": {
          "key": "manual%2Fsample.txt",
          "size": 37,
          "sequencer": "000001"
        }
      }
    }
  ]
}

What does this event represent?

  • The event describes one object-created operation in a bucket named manual-demo.
  • The encoded key represents manual/sample.txt.
  • The sequencer gives this object event a stable identifier component.
  • Save events/manual-s3.json.
  • Confirm that manual-s3.json now appears inside the events folder.

✔️ Awesome, I've got everything!

Your manual event is ready to send to the Lambda worker.

ⓧ I'd like to double check the full code

{
  "Records": [
    {
      "eventVersion": "2.6",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "2026-09-25T12:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "s3SchemaVersion": "1.0",
        "bucket": {
          "name": "manual-demo"
        },
        "object": {
          "key": "manual%2Fsample.txt",
          "size": 37,
          "sequencer": "000001"
        }
      }
    }
  ]
}

This reference is the complete manual event for this step.

The handler reads event metadata only. It does not download the uploaded object. This keeps its permissions focused on writing the audit record.

  • Inside modules/upload_audit/lambda/package, create a file named handler.py.
  • Add the imports and DynamoDB connection setup to handler.py by copying this code:
import os
from urllib.parse import unquote_plus

import boto3

TABLE_NAME = os.environ["TABLE_NAME"]
DYNAMODB = boto3.resource("dynamodb")
TABLE = DYNAMODB.Table(TABLE_NAME)


def lambda_handler(event, context):
    if event.get("Event") == "s3:TestEvent":
        print("Received the S3 notification test event.")
        return {"statusCode": 200, "processed": 0, "event_ids": []}

    event_ids = []

How does the handler start?

  • The environment variable identifies the DynamoDB table selected by Terraform.
  • The Boto3 resource creates a table client that the invocation can reuse.
  • The first branch safely handles the separate test event that S3 can send during notification setup.
  • The event ID list tracks every upload record processed by one invocation.
  • Save modules/upload_audit/lambda/package/handler.py.
  • Confirm that handler.py appears beside the installed dependencies inside package.
  • Continue the lambda_handler() function below event_ids = [] by copying this code:
    for record in event.get("Records", []):
        bucket = record["s3"]["bucket"]["name"]
        object_data = record["s3"]["object"]
        object_key = unquote_plus(object_data["key"])
        sequencer = object_data.get("sequencer", "no-sequencer")
        event_id = f"{bucket}#{object_key}#{sequencer}"

        item = {
            "event_id": event_id,
            "bucket": bucket,
            "object_key": object_key,
            "size_bytes": object_data.get("size", 0),
            "event_name": record["eventName"],
            "event_time": record["eventTime"],
            "sequencer": sequencer,
            "status": "RECORDED"
        }

        TABLE.put_item(Item=item)
        event_ids.append(event_id)
        print(f"Recorded upload event: {event_id}")

    return {
        "statusCode": 200,
        "processed": len(event_ids),
        "event_ids": event_ids
    }

How does the worker create an audit record?

  • The loop reads each regular S3 record from the event.
  • The URL decoder converts the encoded object key back to its readable path.
  • The event ID combines the bucket name with the decoded object key and sequencer.
  • The table write stores the upload metadata before the handler reports how many records it processed.
  • Save modules/upload_audit/lambda/package/handler.py.
  • Confirm that the function ends with the response containing statusCode, processed, and event_ids.

✔️ Awesome, I've got everything!

Your deployment package now contains the handler plus its pinned dependencies.

ⓧ I'd like to double check the full code

import os
from urllib.parse import unquote_plus

import boto3

TABLE_NAME = os.environ["TABLE_NAME"]
DYNAMODB = boto3.resource("dynamodb")
TABLE = DYNAMODB.Table(TABLE_NAME)


def lambda_handler(event, context):
    if event.get("Event") == "s3:TestEvent":
        print("Received the S3 notification test event.")
        return {"statusCode": 200, "processed": 0, "event_ids": []}

    event_ids = []

    for record in event.get("Records", []):
        bucket = record["s3"]["bucket"]["name"]
        object_data = record["s3"]["object"]
        object_key = unquote_plus(object_data["key"])
        sequencer = object_data.get("sequencer", "no-sequencer")
        event_id = f"{bucket}#{object_key}#{sequencer}"

        item = {
            "event_id": event_id,
            "bucket": bucket,
            "object_key": object_key,
            "size_bytes": object_data.get("size", 0),
            "event_name": record["eventName"],
            "event_time": record["eventTime"],
            "sequencer": sequencer,
            "status": "RECORDED"
        }

        TABLE.put_item(Item=item)
        event_ids.append(event_id)
        print(f"Recorded upload event: {event_id}")

    return {
        "statusCode": 200,
        "processed": len(event_ids),
        "event_ids": event_ids
    }

This reference is the complete Lambda handler for this step.

Deploy the Lambda infrastructure

The worker now needs a deployment archive plus a trusted execution identity. Terraform will build the archive. It will also create the function and its supporting resources.

  • In modules/upload_audit/main.tf, find the existing local values:
locals {
  common_tags = {
    Project   = var.project_name
    ManagedBy = "Terraform"
  }
}

What are you locating?

This is the current local values block from the storage module. It already centralizes the tags shared by the deployed resources.

  • Replace that local values block with this version:
locals {
  lambda_function_name = "${var.project_name}-processor"
  common_tags = {
    Project   = var.project_name
    ManagedBy = "Terraform"
  }
}

Why add a local function name?

The shared local value keeps the Lambda function and its log group on the same generated name. With the current project name, Terraform creates upload-audit-processor.

  • Save modules/upload_audit/main.tf.
  • Format the updated local values by running:
terraform fmt -recursive

What does this check do?

Terraform formats the module files using its standard layout. A clean format makes later plan changes easier to review.

  • Below the local values block, add the archive data source:
data "archive_file" "processor" {
  type        = "zip"
  source_dir  = "${path.module}/lambda/package"
  output_path = "${path.module}/lambda/function.zip"
}

How is the deployment archive built?

The Archive provider compresses the complete package directory. It writes the resulting deployment file to modules/upload_audit/lambda/function.zip.

  • Below the archive data source, add the Lambda trust policy document:
data "aws_iam_policy_document" "lambda_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

What does the trust policy allow?

The trust policy allows the Lambda service to assume the worker's execution role. It does not grant access to DynamoDB yet.

  • Save modules/upload_audit/main.tf.
  • Format and validate the archive configuration by running:
terraform fmt -recursive
terraform validate

What do these checks prove?

Formatting normalizes the Terraform source. Validation checks that the initialized providers understand the new archive and policy document blocks.

You should see Terraform report that the configuration is valid. The archive data source can now package the worker during planning.

Archive validation failed?

Confirm that modules/upload_audit/lambda/package exists. Check that the installed dependencies and handler.py are inside that folder.

If Terraform still cannot build the archive, help me diagnose the archive data source.

The execution role controls what the worker can do after Lambda starts it. AWS Identity and Access Management will limit the worker to one table plus its own logs.

  • Below the existing DynamoDB table resource, add the execution role:
resource "aws_iam_role" "processor" {
  name               = "${var.project_name}-lambda-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json

  tags = local.common_tags
}

What does this role provide?

The role gives the Lambda function an AWS identity. The trust policy controls which service can assume that identity.

  • Below the execution role, add the log group:
resource "aws_cloudwatch_log_group" "processor" {
  name              = "/aws/lambda/${local.lambda_function_name}"
  retention_in_days = 7

  tags = local.common_tags
}

How are the worker logs managed?

The Amazon CloudWatch Logs group uses Lambda's standard naming convention. Seven-day retention prevents this learning project from keeping logs indefinitely.

  • Save modules/upload_audit/main.tf.
  • Format and validate the identity resources by running:
terraform fmt -recursive
terraform validate

What is Terraform checking?

Terraform checks the role's trust-policy reference. It also validates the log group configuration and its retention value.

You should see another successful validation. The role and log group are now represented in the module.

Least-privilege access gives the worker only the actions required by its code. The handler writes items and logs messages. It never reads the uploaded S3 object.

  • Below the log group, add the worker permissions document:
data "aws_iam_policy_document" "processor" {
  statement {
    sid       = "WriteAuditRecords"
    effect    = "Allow"
    actions   = ["dynamodb:PutItem"]
    resources = [aws_dynamodb_table.audit.arn]
  }

  statement {
    sid    = "WriteFunctionLogs"
    effect = "Allow"
    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["${aws_cloudwatch_log_group.processor.arn}:*"]
  }
}

Why are these permissions limited?

  • The table statement permits item writes only against this module's audit table.
  • The logging statement permits stream creation plus event writes only under this function's log group.
  • No S3 read action appears because the handler uses metadata already present in the event.
  • Below the permissions document, attach it to the execution role:
resource "aws_iam_role_policy" "processor" {
  name   = "${var.project_name}-lambda-policy"
  role   = aws_iam_role.processor.id
  policy = data.aws_iam_policy_document.processor.json
}

How is the policy applied?

The inline policy binds the generated permission document to the worker's role. The Lambda function receives those permissions when it assumes the role.

  • Save modules/upload_audit/main.tf.
  • Format and validate the least-privilege policy by running:
terraform fmt -recursive
terraform validate

What does this validation cover?

Terraform resolves the table ARN and log group ARN used by the policy. It also confirms that the role policy receives generated JSON.

You should see a successful validation. The worker now has a narrowly scoped permission model.

  • Below the role policy, add the Lambda function:
resource "aws_lambda_function" "processor" {
  function_name = local.lambda_function_name
  role          = aws_iam_role.processor.arn
  handler       = "handler.lambda_handler"
  runtime       = "python3.14"
  filename      = data.archive_file.processor.output_path
  code_sha256   = data.archive_file.processor.output_base64sha256
  memory_size   = 128
  timeout       = 10

  environment {
    variables = {
      TABLE_NAME = aws_dynamodb_table.audit.name
    }
  }

  depends_on = [
    aws_cloudwatch_log_group.processor,
    aws_iam_role_policy.processor
  ]

  tags = local.common_tags
}

How is the worker configured?

  • The handler setting maps the root handler.py file to lambda_handler().
  • The runtime uses Python 3.14.
  • The archive hash tells Terraform when packaged code has changed.
  • The environment map passes the deployed table name to the handler.
  • Save modules/upload_audit/main.tf.
  • Format and validate the function configuration by running:
terraform fmt -recursive
terraform validate

What does this validation prove?

Terraform confirms that the function references the archive, role, policy, log group, and table consistently. A successful result means the module is ready for planning.

You should see Terraform report that the configuration is valid. The package wiring and function settings now agree.

  • At the bottom of modules/upload_audit/outputs.tf, add the worker contract outputs:
output "lambda_function_name" {
  description = "Name of the Python audit Lambda function."
  value       = aws_lambda_function.processor.function_name
}

output "lambda_runtime" {
  description = "Configured Lambda runtime."
  value       = aws_lambda_function.processor.runtime
}

output "audit_billing_mode" {
  description = "Configured DynamoDB billing mode."
  value       = aws_dynamodb_table.audit.billing_mode
}

Why expose these module outputs?

The function name supports repeatable manual invocations. The runtime plus billing mode expose two design decisions that a later contract test can protect.

  • Save modules/upload_audit/outputs.tf.
  • At the bottom of the root outputs.tf, expose the new module outputs:
output "lambda_function_name" {
  description = "Name of the Python audit Lambda function."
  value       = module.upload_audit.lambda_function_name
}

output "lambda_runtime" {
  description = "Configured Lambda runtime, exposed for contract testing."
  value       = module.upload_audit.lambda_runtime
}

output "audit_billing_mode" {
  description = "Configured DynamoDB billing mode, exposed for contract testing."
  value       = module.upload_audit.audit_billing_mode
}

How do the root outputs help?

Root outputs make module values available to commands run from the project folder. You can invoke the generated function without copying its name from the AWS console.

  • Save the root outputs.tf file.
  • Format and validate all output changes by running:
terraform fmt -recursive
terraform validate

What do these final checks cover?

Terraform verifies that every root output maps to an output declared by the child module. A successful result confirms the complete configuration is internally consistent.

You should see a successful validation. The function name and contract values are now available at the project level.

✔️ Awesome, I've got everything!

The upload-audit module now defines the archive, execution role, log group, policy, and Lambda function.

ⓧ I'd like to double check the full code

locals {
  lambda_function_name = "${var.project_name}-processor"
  common_tags = {
    Project   = var.project_name
    ManagedBy = "Terraform"
  }
}

data "archive_file" "processor" {
  type        = "zip"
  source_dir  = "${path.module}/lambda/package"
  output_path = "${path.module}/lambda/function.zip"
}

data "aws_iam_policy_document" "lambda_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_s3_bucket" "uploads" {
  bucket_prefix = "${var.project_name}-uploads-"
  force_destroy = true

  tags = local.common_tags
}

resource "aws_s3_bucket_public_access_block" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "audit" {
  name         = "${var.project_name}-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "event_id"

  attribute {
    name = "event_id"
    type = "S"
  }

  tags = local.common_tags
}

resource "aws_iam_role" "processor" {
  name               = "${var.project_name}-lambda-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json

  tags = local.common_tags
}

resource "aws_cloudwatch_log_group" "processor" {
  name              = "/aws/lambda/${local.lambda_function_name}"
  retention_in_days = 7

  tags = local.common_tags
}

data "aws_iam_policy_document" "processor" {
  statement {
    sid       = "WriteAuditRecords"
    effect    = "Allow"
    actions   = ["dynamodb:PutItem"]
    resources = [aws_dynamodb_table.audit.arn]
  }

  statement {
    sid    = "WriteFunctionLogs"
    effect = "Allow"
    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["${aws_cloudwatch_log_group.processor.arn}:*"]
  }
}

resource "aws_iam_role_policy" "processor" {
  name   = "${var.project_name}-lambda-policy"
  role   = aws_iam_role.processor.id
  policy = data.aws_iam_policy_document.processor.json
}

resource "aws_lambda_function" "processor" {
  function_name = local.lambda_function_name
  role          = aws_iam_role.processor.arn
  handler       = "handler.lambda_handler"
  runtime       = "python3.14"
  filename      = data.archive_file.processor.output_path
  code_sha256   = data.archive_file.processor.output_base64sha256
  memory_size   = 128
  timeout       = 10

  environment {
    variables = {
      TABLE_NAME = aws_dynamodb_table.audit.name
    }
  }

  depends_on = [
    aws_cloudwatch_log_group.processor,
    aws_iam_role_policy.processor
  ]

  tags = local.common_tags
}

This reference is the complete module infrastructure file for this step.

✔️ Awesome, I've got everything!

The module exposes its function name, runtime, and DynamoDB billing mode.

ⓧ I'd like to double check the full code

output "upload_bucket_name" {
  description = "Name of the S3 upload bucket."
  value       = aws_s3_bucket.uploads.id
}

output "audit_table_name" {
  description = "Name of the DynamoDB audit table."
  value       = aws_dynamodb_table.audit.name
}

output "lambda_function_name" {
  description = "Name of the Python audit Lambda function."
  value       = aws_lambda_function.processor.function_name
}

output "lambda_runtime" {
  description = "Configured Lambda runtime."
  value       = aws_lambda_function.processor.runtime
}

output "audit_billing_mode" {
  description = "Configured DynamoDB billing mode."
  value       = aws_dynamodb_table.audit.billing_mode
}

This reference is the complete module outputs file for this step.

✔️ Awesome, I've got everything!

The root configuration exposes all values needed for deployment checks and manual invocation.

ⓧ I'd like to double check the full code

output "aws_region" {
  description = "AWS Region containing the pipeline."
  value       = var.aws_region
}

output "upload_bucket_name" {
  description = "Name of the S3 upload bucket."
  value       = module.upload_audit.upload_bucket_name
}

output "audit_table_name" {
  description = "Name of the DynamoDB audit table."
  value       = module.upload_audit.audit_table_name
}

output "lambda_function_name" {
  description = "Name of the Python audit Lambda function."
  value       = module.upload_audit.lambda_function_name
}

output "lambda_runtime" {
  description = "Configured Lambda runtime, exposed for contract testing."
  value       = module.upload_audit.lambda_runtime
}

output "audit_billing_mode" {
  description = "Configured DynamoDB billing mode, exposed for contract testing."
  value       = module.upload_audit.audit_billing_mode
}

This reference is the complete root outputs file for this step.

The next commands create billable AWS resources. This small test should use little capacity. You will remove the stack in the cleanup section if you do not want to keep it.

  • Create a saved deployment plan by running:
terraform plan -out="tfplan"

What should the plan include?

The plan should add the archive-backed Lambda function, execution role, inline policy, and seven-day log group. Your existing bucket plus table should remain deployed.

  • Review the saved plan for the new worker resources.
  • Apply the reviewed plan by running:
terraform apply "tfplan"

What does this deployment create?

Terraform uploads the generated archive and creates the worker infrastructure. It also adds the new root output values to the apply result.

You should see Terraform finish the apply successfully. The output should include the new function name, runtime, and billing mode.

That is the infrastructure work complete. Your audit worker is deployed with a dedicated role and a limited permission policy.

Lambda deployment failed?

Check the plan output for the first resource that failed. Confirm your AWS session still targets the account verified earlier.

If the failure mentions packaging or permissions, help me diagnose the Terraform apply failure.

Invoke and inspect the worker

The live S3 trigger still belongs to the next step. A direct invocation isolates the worker so you can prove its code and write permission independently.

Before you run the invocation, do you expect the manual event to create zero records or one record?

  • Invoke the deployed worker with the manual S3 event by running:
aws lambda invoke --function-name "$(terraform output -raw lambda_function_name)" --payload fileb://events/manual-s3.json --region "$(terraform output -raw aws_region)" response.json

What does this invocation test?

The command reads the deployed function name plus region from Terraform outputs. It sends the local event file to Lambda and saves the handler response in response.json.

  • In the Visual Studio Code sidebar, open response.json.

You should see a successful response with processed set to 1. Its event ID should be manual-demo#manual/sample.txt#000001.

Before you scan the table, do you expect its count to remain at zero or increase?

  • Scan the audit table with a consistent read by running:
aws dynamodb scan --table-name "$(terraform output -raw audit_table_name)" --consistent-read --region "$(terraform output -raw aws_region)"

What proves the worker succeeded?

The scan should return the manual audit item. Look for an object key of manual/sample.txt plus an event name of ObjectCreated:Put.

The item also records the bucket, event time, sequencer, size, and RECORDED status. Those fields prove the handler converted the event into an audit record.

You have closed the empty-table gap. The deployed worker can now turn S3 event metadata into a DynamoDB audit record.

No manual audit record?

Open response.json first. A failed response points to the handler or its table environment setting.

If the response succeeds but the item is missing, confirm that the scan uses the region and table name from Terraform outputs. Help me trace the missing audit item.

Your worker is packaged, deployed, and writing audit records on demand. Next up, you will connect the S3 upload event so the same result happens automatically.

Connect and Prove the Upload Pipeline

The manual event proved that your AWS Lambda worker can write an audit record to Amazon DynamoDB.

Manual invocation still leaves a person in the loop for every Amazon S3 upload. In this step, Terraform adds the automatic route from the bucket to the function.

In this step, get ready to:
  • Allow the project bucket to invoke the Lambda function.
  • Send object-created events to the audit worker.
  • Upload a file and confirm its automatic audit record.
Grant S3 invocation permission

A Lambda resource-based permission controls which service can invoke the function. The project bucket and current AWS account form the permission boundaries.

  • In the Explorer sidebar in Visual Studio Code, select modules/upload_audit/main.tf.
  • Find this existing data-source group near the top of the file:
data "archive_file" "processor" {
  type        = "zip"
  source_dir  = "${path.module}/lambda/package"
  output_path = "${path.module}/lambda/function.zip"
}

data "aws_iam_policy_document" "lambda_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

Why move these data sources?

The archive data source packages the worker. The policy document defines the function's trust relationship.

Placing both after the storage resources makes the completed module follow its deployment flow.

  • Move this entire data-source group below the closing brace of aws_dynamodb_table.audit.
  • Save modules/upload_audit/main.tf.
  • Confirm the reordered module remains valid by running this command:
terraform validate

What does this check prove?

Terraform checks the initialized configuration for syntax and internal consistency. A successful result proves that both moved blocks still belong to the module.

Does the reordered module fail validation?

Confirm that you moved both complete data blocks. Check that each block still has its closing brace.

If the structure remains unclear, ask for help checking the moved Terraform blocks.

The permission also needs the account that owns your active session. A caller identity data source reads that account without requiring a copied identifier.

  • Insert this data source immediately below the closing brace of locals by copying the code below:
data "aws_caller_identity" "current" {}

What does this data source provide?

The data source reads details about the current AWS caller. The permission uses its account ID as an additional boundary.

  • Save modules/upload_audit/main.tf.
  • Check the caller identity reference by running this command:
terraform validate

What has Terraform checked?

Terraform confirms that the caller identity data source is valid in the initialized module. The data becomes available to the permission resource you add next.

Is the caller identity data source invalid?

Confirm that the data source sits outside the locals block. It needs its own line after the closing brace.

If validation still fails, ask for help checking the data source placement.

  • Add this permission immediately below aws_lambda_function.processor by copying the code below:
resource "aws_lambda_permission" "allow_s3" {
  statement_id   = "AllowExecutionFromS3Bucket"
  action         = "lambda:InvokeFunction"
  function_name  = aws_lambda_function.processor.function_name
  principal      = "s3.amazonaws.com"
  source_arn     = aws_s3_bucket.uploads.arn
  source_account = data.aws_caller_identity.current.account_id
}

What does this permission allow?

  • The service principal allows Amazon S3 to invoke the function.
  • The source ARN limits invocation to the project upload bucket.
  • The source account limits invocation to the current AWS account.
  • Save modules/upload_audit/main.tf.
  • Verify the permission references by running this command:
terraform validate

What does this validation confirm?

Terraform confirms that the permission refers to the existing Lambda function and upload bucket. It also confirms that the caller identity attribute is available.

Does the permission have an invalid reference?

Compare the function name and bucket ARN references with the existing resources in the same file. Confirm that source_account uses data.aws_caller_identity.current.account_id.

For a deeper check, ask for help tracing the permission references.

Connect the bucket notification

The permission authorizes the invocation. A bucket notification defines which event sends that invocation.

One notification resource manages the bucket's complete notification configuration. Its dependency ensures that the invocation permission exists before S3 connects the function.

  • Add this notification immediately below aws_lambda_permission.allow_s3 by copying the code below:
resource "aws_s3_bucket_notification" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.processor.arn
    events              = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.allow_s3]
}

How does the notification work?

  • The bucket reference attaches the notification to the existing upload bucket.
  • The Lambda ARN identifies the destination function.
  • The event pattern covers object-created events.
  • The dependency makes Terraform create the permission first.
  • Save modules/upload_audit/main.tf.
  • Format the completed module by running this command:
terraform fmt -recursive

What does formatting change?

Terraform normalizes the layout of every Terraform file in the project. This keeps the new resources aligned with the existing module.

  • Validate the formatted event configuration by running this command:
terraform validate

What has Terraform checked?

Terraform checks the permission and notification together. A successful result confirms that the event route is internally consistent.

Does the notification fail validation?

Confirm that aws_s3_bucket_notification.uploads sits outside the permission resource. Check that both resources have separate closing braces.

If the configuration still fails, ask for help comparing the notification references.

Choose the first tab if your module matches the snippets. Use the second tab to compare the complete file.

✔️ Awesome, I've got everything

Your module now contains the caller identity data source and invocation permission. It also contains the bucket notification. Confirm that modules/upload_audit/main.tf is saved.

ⓧ I'd like to double check the full code

locals {
  lambda_function_name = "${var.project_name}-processor"
  common_tags = {
    Project   = var.project_name
    ManagedBy = "Terraform"
  }
}

data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "uploads" {
  bucket_prefix = "${var.project_name}-uploads-"
  force_destroy = true

  tags = local.common_tags
}

resource "aws_s3_bucket_public_access_block" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "audit" {
  name         = "${var.project_name}-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "event_id"

  attribute {
    name = "event_id"
    type = "S"
  }

  tags = local.common_tags
}

data "archive_file" "processor" {
  type        = "zip"
  source_dir  = "${path.module}/lambda/package"
  output_path = "${path.module}/lambda/function.zip"
}

data "aws_iam_policy_document" "lambda_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "processor" {
  name               = "${var.project_name}-lambda-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json

  tags = local.common_tags
}

resource "aws_cloudwatch_log_group" "processor" {
  name              = "/aws/lambda/${local.lambda_function_name}"
  retention_in_days = 7

  tags = local.common_tags
}

data "aws_iam_policy_document" "processor" {
  statement {
    sid       = "WriteAuditRecords"
    effect    = "Allow"
    actions   = ["dynamodb:PutItem"]
    resources = [aws_dynamodb_table.audit.arn]
  }

  statement {
    sid    = "WriteFunctionLogs"
    effect = "Allow"
    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["${aws_cloudwatch_log_group.processor.arn}:*"]
  }
}

resource "aws_iam_role_policy" "processor" {
  name   = "${var.project_name}-lambda-policy"
  role   = aws_iam_role.processor.id
  policy = data.aws_iam_policy_document.processor.json
}

resource "aws_lambda_function" "processor" {
  function_name = local.lambda_function_name
  role          = aws_iam_role.processor.arn
  handler       = "handler.lambda_handler"
  runtime       = "python3.14"
  filename      = data.archive_file.processor.output_path
  code_sha256   = data.archive_file.processor.output_base64sha256
  memory_size   = 128
  timeout       = 10

  environment {
    variables = {
      TABLE_NAME = aws_dynamodb_table.audit.name
    }
  }

  depends_on = [
    aws_cloudwatch_log_group.processor,
    aws_iam_role_policy.processor
  ]

  tags = local.common_tags
}

resource "aws_lambda_permission" "allow_s3" {
  statement_id   = "AllowExecutionFromS3Bucket"
  action         = "lambda:InvokeFunction"
  function_name  = aws_lambda_function.processor.function_name
  principal      = "s3.amazonaws.com"
  source_arn     = aws_s3_bucket.uploads.arn
  source_account = data.aws_caller_identity.current.account_id
}

resource "aws_s3_bucket_notification" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.processor.arn
    events              = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.allow_s3]
}

What should the complete module contain?

The permission restricts invocation to the project bucket and current account. The notification sends object-created events to the existing Lambda function.

Deploy and prove the automatic path

A saved plan shows the live changes before they reach AWS. This plan should add the invocation permission and bucket notification.

  • Create a saved plan by running this command:
terraform plan -out="tfplan"

What should the plan contain?

The plan should add the Lambda invocation permission and S3 bucket notification. The existing bucket and audit table remain in place.

The function and its supporting resources also remain in place. This update only adds the automatic event route.

Does the plan fail?

Run the validation command again if Terraform reports a configuration problem. Confirm that your authenticated account can manage Lambda permissions and S3 notifications.

If the cause remains unclear, ask for help reviewing the Terraform plan failure.

  • Review the plan for the permission addition.
  • Review the plan for the notification addition.

Applying the plan updates live AWS resources. The saved plan limits this apply to the changes you just reviewed.

  • Apply the reviewed plan by running this command:
terraform apply "tfplan"

What does this apply connect?

Terraform grants the bucket permission to invoke the function. It then configures object-created events to use that function.

The automatic link is live. Your bucket can now invoke the audit worker after an upload.

Did the notification apply fail?

Confirm that the active AWS account owns the deployed bucket and function. Confirm that both resources use the Region exposed by your Terraform output.

If the apply still fails, ask for help checking the permission order.

  • Upload sample.txt as incoming/sample.txt by running this command:
aws s3 cp sample.txt "s3://$(terraform output -raw upload_bucket_name)/incoming/sample.txt"

How does the upload find the bucket?

The root output supplies the generated bucket name. The destination key stores the file as incoming/sample.txt.

Creating the object emits the event handled by the notification configuration.

The upload command should confirm that sample.txt reached the generated bucket.

Did the upload fail?

Confirm that sample.txt is still in the project folder. Confirm that the active account can upload to the generated bucket.

If the upload still fails, ask for help checking the bucket output and access.

Before you scan the table, do you expect the manual record to remain the only item?

  • Scan the audit table with a consistent read by running this command:
aws dynamodb scan --table-name "$(terraform output -raw audit_table_name)" --consistent-read --region "$(terraform output -raw aws_region)"

What proves the pipeline worked?

Look for an item whose object_key is incoming/sample.txt.

That item should have an event_name value of ObjectCreated:Put.

You have proved the complete path. An S3 upload now reaches the audit table through Lambda without a manual invocation.

Is the automatic record missing?

Notification changes can take a few minutes to propagate. Wait briefly before uploading the file again.

Scan the table again after the new upload. If the item remains absent, ask for help tracing the event path.

Your upload pipeline now reacts to real files from end to end. Next, you'll protect its runtime and billing decisions with a mocked Terraform contract test.

Add a Mocked Terraform Contract Test

Your deployed pipeline now turns each Amazon S3 upload into an Amazon DynamoDB audit record through AWS Lambda. You proved that path with incoming/sample.txt.

A working deployment can still regress when its configuration changes. A fast mocked Terraform test protects the approved runtime and billing mode without creating another stack.

In this step, get ready to:
  • Create a Terraform test that uses mocked providers.
  • Protect the Lambda runtime and DynamoDB billing mode with plan assertions.
  • Run the contract test and confirm that every assertion passes.
Build the mocked test file

Mocked providers replace live provider calls during the test plan. This lets the contract inspect planned values without provisioning another AWS stack.

  • Return to the editor you used earlier for the upload audit project.
  • Create a tests folder beside main.tf using your editor's new-folder control.

You'll see tests alongside main.tf in the project file tree.

  • Create pipeline.tftest.hcl inside tests using your editor's new-file control.
  • Add the mocked providers and contract assertions by pasting this code into tests/pipeline.tftest.hcl:
mock_provider "aws" {}
mock_provider "archive" {}

run "pipeline_contract" {
  command = plan

  assert {
    condition     = output.lambda_runtime == "python3.14"
    error_message = "The pipeline must use the approved Python 3.14 Lambda runtime."
  }

  assert {
    condition     = output.audit_billing_mode == "PAY_PER_REQUEST"
    error_message = "The audit table must use PAY_PER_REQUEST billing."
  }
}

What does this contract protect?

  • mock_provider "aws" {} replaces live AWS provider behavior during the plan.
  • mock_provider "archive" {} supplies the mocked Archive provider needed by the package data source.
  • command = plan keeps the test focused on planned configuration.
  • output.lambda_runtime must stay equal to python3.14.
  • output.audit_billing_mode must stay equal to PAY_PER_REQUEST.
  • Save tests/pipeline.tftest.hcl.
  • Confirm pipeline.tftest.hcl remains visible inside tests in the project file tree.

Test file missing from the project tree?

  • Confirm that tests sits beside main.tf.
  • Confirm that the file path is exactly tests/pipeline.tftest.hcl.

Still missing the test file? Help me check the test folder and file path.

✔️ Awesome, I've got everything!

Your saved test file contains both mocked providers plus two plan assertions.

ⓧ I'd like to double check the full code

mock_provider "aws" {}
mock_provider "archive" {}

run "pipeline_contract" {
  command = plan

  assert {
    condition     = output.lambda_runtime == "python3.14"
    error_message = "The pipeline must use the approved Python 3.14 Lambda runtime."
  }

  assert {
    condition     = output.audit_billing_mode == "PAY_PER_REQUEST"
    error_message = "The audit table must use PAY_PER_REQUEST billing."
  }
}

What does the full file contain?

This reference combines the mocked provider declarations with the complete pipeline_contract plan run. It is the exact completed test file for this step.

Keep the contract consistent

Terraform's formatter normalizes the layout of test files alongside the rest of your configuration. Formatting now also confirms that Terraform can read the new file.

  • Format the project and its subdirectories by running this command:
terraform fmt -recursive

What does this command do?

Terraform formats every Terraform file in the current directory and its subdirectories. That includes the new contract inside tests.

  • Wait for the formatter to return control to the terminal.

You'll return to the terminal prompt without a formatting error.

Formatting command fails?

  • Confirm that the terminal still points to the folder containing versions.tf.
  • Compare tests/pipeline.tftest.hcl with the completed file above.

Need help finding the formatting problem? Help me diagnose my Terraform test file.

Prove the module contract

The two assertions turn exposed module values into a contract. Future configuration changes must preserve the approved Lambda runtime and DynamoDB billing mode.

Before you run the test, do you think both assertions will pass against the pipeline you deployed?

  • Run the contract against the mocked providers by running this command:
terraform test

What should you see?

Terraform reports tests/pipeline.tftest.hcl... pass. The final summary shows zero failed tests.

That's your regression guard in place. The mocked plan checks both contract values without creating additional AWS infrastructure.

Contract test not passing?

  • Compare tests/pipeline.tftest.hcl with the completed file above.
  • Confirm that root outputs.tf still exposes lambda_runtime.
  • Confirm that root outputs.tf still exposes audit_billing_mode.

Still seeing a failed assertion? Help me trace the Terraform contract failure.

Secret mission

Restrict Processing to an Incoming Prefix

Your pipeline currently audits every object-created event in its bucket. Add an incoming/ boundary and prove that uploads outside it stay out of the audit table.

Clean Up Your Resources

Clean Up Your Resources

Your pipeline uses pay-as-you-go AWS services, so storage, requests, table operations, and logs can create charges. Choose whether to keep the demo ready, pause new activity, or delete the stack.

Cost warning

AWS Lambda includes a monthly free allowance. Amazon S3, Amazon DynamoDB, and Amazon CloudWatch Logs can still create charges through storage or usage.

This small demo should use little capacity while idle. Deleting the stack is the safest way to prevent future costs.

Resources you used:

  • A Terraform-managed AWS stack containing a private S3 bucket with test objects, an upload notification, and an on-demand DynamoDB audit table.
  • A Python Lambda function with its deployment package, resource-based S3 permission, and seven-day CloudWatch Logs log group.
  • An AWS Identity and Access Management execution role with an inline policy for audit-table writes and function logging.
  • Local Terraform state, the .terraform/ provider cache, the tfplan saved plan, the response.json response, and generated Lambda package artifacts.

Keep everything running

No action is needed. Choose this if you want the upload audit pipeline available for more demonstrations.

  • Leave the Terraform-managed AWS stack unchanged.
  • Retain the initialized project folder with its local state.
  • Upload future demo files under incoming/ so the prefix filter sends them to the Lambda worker.
  • Review AWS Billing after future demonstrations to confirm that usage remains within your budget.

Pause - I'll come back to this later

Stop creating new upload events while you take a break. Stored S3 objects, DynamoDB records, and CloudWatch logs remain in AWS.

  • Stop uploading test files to the S3 bucket.
  • Leave the Terraform configuration unchanged.
  • Keep the local Terraform state because it connects the configuration to the deployed resources.
  • Resume the demonstration later by uploading a file under incoming/.

Delete - I don't want to use this again

Remove the AWS stack before deleting local state. Terraform needs that state to identify every managed resource.

  • Switch back to the terminal in the initialized project folder from earlier.
  • Preview every Terraform-managed removal by running this command:
terraform plan -destroy

What does this preview show?

Terraform compares the local state with the deployed AWS stack. The resulting plan shows which managed resources it proposes to remove.

The preview should cover the bucket, table, function, permissions, role, policy, notification configuration, and log group.

  • Review the plan for resources that belong to the upload audit pipeline.

Destroy cloud resources first

Deleting the stack is permanent. The bucket uses force_destroy = true, so Terraform also removes the test objects stored inside it.

Your source files remain local until you remove the generated artifacts separately.

  • Destroy the reviewed stack with Terraform's prompted approval by running this command:
terraform destroy

What does this command remove?

Terraform deprovisions every AWS object recorded in this project's state. This includes the resources that store files, audit records, permissions, and logs.

Terraform should finish after removing all managed resources from the intended AWS account.

Destroy did not finish?

Confirm that the terminal is still in the initialized project folder. Terraform needs the configuration and local state from that folder.

Check that the AWS CLI session still uses the intended account. An expired session or missing deletion permission can block the destroy operation.

Ask for help with the specific failure: Help me diagnose why Terraform could not destroy my upload audit pipeline resources.

Before you verify, do you expect Terraform to find any managed AWS resources left?

  • Check the emptied Terraform state by running the destroy preview again:
terraform plan -destroy

What confirms cleanup?

The new plan should propose no AWS resource removals. That confirms the deployed stack is no longer tracked as active infrastructure.

That closes the billable part of the project. Your upload audit resources are no longer running in AWS.

Local generated artifacts do not create AWS charges. Removing them clears provider downloads, state files, saved output, and the packaged Lambda directory from this project.

  • Remove the generated local artifacts from the initialized project folder by running this command:
rm -rf .terraform *.tfstate *.tfstate.* tfplan response.json modules/upload_audit/lambda/function.zip modules/upload_audit/lambda/package

What does this local cleanup remove?

The command removes the downloaded provider cache, local Terraform state, saved plan, invocation response, generated zip, and packaged Lambda directory.

Your remaining Terraform files, test file, sample event, requirements file, and sample upload file stay available as project source.

Nice Work!

Nice Work!

You pulled it off! Your serverless upload audit pipeline now turns Amazon S3 uploads under incoming/ into Amazon DynamoDB records through AWS Lambda.

You've learned how to:

  • Built a reusable Terraform module that deploys a private upload bucket. The same module provisions an on-demand audit table plus a Python Lambda worker.
  • Packaged a pinned Boto3 dependency with the worker. Connected upload events using least-privilege IAM permissions. Proved that each approved upload creates an audit record.
  • Added mocked Terraform contract tests that protect the Lambda runtime plus the DynamoDB billing mode without deploying another stack.
  • Secret Mission: Restricted processing to the incoming/ prefix. Contrasting uploads proved that files under ignored/ never reach the worker.

Ready to quiz yourself?