Serverless Document Intelligence Pipeline

Build a secure, AI-powered document pipeline on AWS with Terraform IaC.

Introduction

30 Second Summary

Every time you upload a sensitive document online, you trust that system to handle your data securely. Most breaches happen because encryption, access controls, or secret management were afterthoughts instead of built-in defaults.

In this project, you will build a serverless document intelligence pipeline on AWS that encrypts everything by default, processes PDFs with AI, and manages the entire infrastructure through Terraform code.

What You'll Build

SecureDocAI is a serverless pipeline that accepts PDF uploads via pre-signed URLs, stores them in a KMS-encrypted S3 bucket, and triggers AI-powered data extraction through SQS and Lambda.

By the end of this project, you'll have:

  • A Terraform-managed AWS infrastructure with remote state, KMS encryption, and least-privilege IAM roles.
  • An event-driven processing pipeline using SQS queues with dead-letter isolation to handle document uploads reliably.
  • A Gemini AI integration stored securely in Secrets Manager that extracts structured data from uploaded PDFs.
  • Secret Mission: Lock down your S3 bucket with a VPC Gateway Endpoint so uploads can only flow through your private network.

Are there any prerequisites?

You need an AWS account with the AWS CLI configured, Terraform 1.15.7 installed, and a Gemini API key.

This is Project 1 of 3 in the SecureDocAI series. No prior projects in the series are required.

Set Up Remote State and Project Structure

Before building any infrastructure, you need a reliable way to track what Terraform has deployed.

By default, Terraform stores its state in a local file. This works for experimentation, but falls apart in production. If the file gets deleted or corrupted, Terraform loses track of everything it created. Remote state solves this by storing the state file in an S3 bucket with versioning and encryption, and using a lockfile to prevent two people from modifying infrastructure at the same time.

In this step, get ready to:
  • Bootstrap a remote state backend by provisioning an S3 bucket with Terraform.
  • Create the main project structure with provider, backend, and variable configuration.
  • Verify the remote backend is active and state locking works.
Bootstrap the Terraform remote state backend

A remote state backend needs to exist before any other Terraform code can use it. You will create a small, self-contained bootstrap configuration whose only job is to provision the S3 bucket that holds your main project's state.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project directory with a nested bootstrap folder, then move into the bootstrap folder by running these commands:
mkdir -p securedocai/bootstrap
cd securedocai/bootstrap
  • Confirm you are inside the bootstrap directory by running:
pwd

You should see a path ending in securedocai/bootstrap.

  • Create the bootstrap configuration file by running:
touch main.tf
  • Open main.tf in your code editor and paste the following Terraform block and provider configuration:
terraform {
  required_version = ">= 1.15.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# Create the S3 bucket for storing Terraform state
resource "aws_s3_bucket" "terraform_state" {
  bucket_prefix = "securedocai-tfstate-"

  # Prevent accidental deletion of the state bucket
  lifecycle {
    prevent_destroy = true
  }
}

What does this code do?

  • required_version = ">= 1.15.0" ensures anyone running this config has a Terraform version that supports the S3 lockfile feature.
  • hashicorp/aws ~> 6.0 pins the AWS provider to version 6.x, allowing patch updates but not major version jumps.
  • bucket_prefix lets AWS generate a unique suffix so your bucket name never collides with anyone else's.
  • prevent_destroy = true makes Terraform refuse to delete this bucket, protecting your state from accidental destruction.
  • Now add the security settings below the bucket resource. Paste this code at the bottom of main.tf:
# Enable versioning so state history is preserved
resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

# Encrypt the state bucket with AWS-managed KMS
resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

# Block all public access to the state bucket
resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Output the bucket name so we can reference it in the main project
output "state_bucket_name" {
  value = aws_s3_bucket.terraform_state.id
}

What do these resources do?

These three resources harden the state bucket against common risks:

  • Versioning keeps every previous version of your state file. If a bad apply corrupts state, you can roll back.
  • SSE-KMS encryption encrypts the state at rest. State files often contain sensitive values like resource ARNs and connection strings.
  • Public access block ensures the bucket can never be accidentally exposed to the internet.

The output block prints the generated bucket name after apply so you can copy it into your main project's backend config.

  • Save bootstrap/main.tf.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before continuing.

ⓧ I'd like to double check the full code

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "terraform_state" {
  bucket_prefix = "securedocai-tfstate-"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

output "state_bucket_name" {
  value = aws_s3_bucket.terraform_state.id
}
  • Initialize Terraform in the bootstrap directory by running:
terraform init

You should see a message saying Terraform has been successfully initialized and the AWS provider was installed.

Seeing a provider error?

  • Make sure your required_providers block specifies source = "hashicorp/aws" exactly. A typo in the source string causes Terraform to look in the wrong registry.
  • Confirm you have internet access. Terraform needs to download the provider plugin during init.

Help me troubleshoot my terraform init failure.

  • Provision the state bucket by running:
terraform apply
  • Terraform will show you a plan of the resources it wants to create. Type yes and press Enter to confirm.

After the apply completes, you should see an output like state_bucket_name = "securedocai-tfstate-abc12345". The exact suffix will be unique to your account.

  • Copy the bucket name from the output. You will need it in the next substep.

Save your bucket name somewhere handy

Paste the bucket name into a temporary note or keep the terminal visible. You will reference this name in the main project's backend configuration next.

Apply failed with an access denied error?

  • Verify your AWS CLI is configured with credentials that have permission to create S3 buckets. Run aws sts get-caller-identity to confirm your identity.
  • If you see a region error, make sure your provider block specifies region = "us-east-1" and your AWS CLI default region matches.

Help me fix my terraform apply error.

Initialize the main project

With the state bucket provisioned, you can now create the main project configuration that uses it as its backend. This is where all your serverless pipeline resources will be defined in upcoming steps.

  • Move up to the main project directory by running:
cd ~/Desktop/securedocai
  • Create the main configuration files by running:
touch main.tf variables.tf outputs.tf terraform.tfvars.example
  • Confirm the files exist by running:
ls

You should see bootstrap, main.tf, variables.tf, outputs.tf, and terraform.tfvars.example in the output.

  • Open main.tf in your code editor and paste the following backend configuration:
terraform {
  required_version = ">= 1.15.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }

  # Remote state stored in the bootstrap bucket
  backend "s3" {
    bucket       = "[[STATE_BUCKET=\"securedocai-tfstate-XXXXX\"]]"
    key          = "securedocai/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
    encrypt      = true
  }
}

provider "aws" {
  region = var.aws_region
}

What does this backend block do?

  • bucket points to the S3 bucket you just created with the bootstrap config.
  • key is the path inside the bucket where the state file lives. This keeps the state organized if you ever store multiple projects in one bucket.
  • use_lockfile = true enables the modern S3-native lockfile mechanism. During operations, Terraform creates a .tflock file in the bucket to prevent concurrent writes.
  • encrypt = true ensures the state is encrypted at rest in S3.

Why use_lockfile instead of DynamoDB?

Older Terraform versions required a DynamoDB table for state locking. Since Terraform 1.10, the S3 backend supports native lockfile-based locking directly in the bucket. The DynamoDB approach is now deprecated.

Using use_lockfile = true means one fewer resource to manage and no DynamoDB costs.

  • Replace the placeholder bucket name with the actual bucket name from your bootstrap output.
  • Save main.tf.
  • Open variables.tf and paste the following variable declarations:
variable "aws_region" {
  description = "AWS region for all resources"
  type        = string
  default     = "us-east-1"
}

variable "project_name" {
  description = "Project name used for resource naming"
  type        = string
  default     = "securedocai"
}

variable "gemini_api_key" {
  description = "Google Gemini API key"
  type        = string
  sensitive   = true
}

What does this code do?

  • aws_region and project_name have defaults so you can run terraform plan without passing values every time.
  • gemini_api_key is marked sensitive = true so Terraform never prints it in logs or plan output. You will provide the value in a terraform.tfvars file that is never committed to version control.
  • Save variables.tf.
  • Open outputs.tf and paste the following output declarations:
output "document_bucket_name" {
  value = aws_s3_bucket.documents.id
}

output "presign_lambda_name" {
  value = aws_lambda_function.presign.function_name
}

output "processor_lambda_name" {
  value = aws_lambda_function.processor.function_name
}

output "processing_queue_url" {
  value = aws_sqs_queue.processing.url
}

output "dlq_url" {
  value = aws_sqs_queue.dlq.url
}

Why add outputs for resources that don't exist yet?

This file is scaffolding. The resources it references (the document bucket, Lambda functions, and SQS queues) will be created in Steps 2 through 4.

Defining outputs now keeps the project structure clean from the start. Terraform will only evaluate these outputs after the referenced resources exist.

  • Save outputs.tf.
  • Open terraform.tfvars.example and paste the following template:
aws_region     = "us-east-1"
project_name   = "securedocai"
gemini_api_key = "your-gemini-api-key-here"

What is this file for?

This is a template showing the expected variable format. You will create a real terraform.tfvars file (without .example) when you are ready to provide your actual Gemini API key in Step 4. The .example file is safe to commit to version control because it contains no real secrets.

  • Save terraform.tfvars.example.
  • Initialize the main project backend by running:
terraform init

You should see a message confirming Terraform has been successfully initialized with the S3 backend.

Getting a backend configuration error?

  • Double check the bucket value in your main.tf matches the exact bucket name from your bootstrap output. Even one character off will cause a "bucket does not exist" error.
  • Make sure you are running terraform init from the securedocai directory, not from inside bootstrap.

Help me fix my terraform init backend error.

✔️ Awesome, I've got everything!

Great. Double check you have saved all four files before continuing.

ⓧ I'd like to double check the full code

Here are all the main project files for this step:

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }

  backend "s3" {
    bucket       = "REPLACE_WITH_STATE_BUCKET_NAME"
    key          = "securedocai/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
    encrypt      = true
  }
}

provider "aws" {
  region = var.aws_region
}
variable "aws_region" {
  description = "AWS region for all resources"
  type        = string
  default     = "us-east-1"
}

variable "project_name" {
  description = "Project name used for resource naming"
  type        = string
  default     = "securedocai"
}

variable "gemini_api_key" {
  description = "Google Gemini API key"
  type        = string
  sensitive   = true
}
output "document_bucket_name" {
  value = aws_s3_bucket.documents.id
}

output "presign_lambda_name" {
  value = aws_lambda_function.presign.function_name
}

output "processor_lambda_name" {
  value = aws_lambda_function.processor.function_name
}

output "processing_queue_url" {
  value = aws_sqs_queue.processing.url
}

output "dlq_url" {
  value = aws_sqs_queue.dlq.url
}
aws_region     = "us-east-1"
project_name   = "securedocai"
gemini_api_key = "your-gemini-api-key-here"
Verify the remote backend

With initialization complete, confirm that Terraform is actually storing state in your S3 bucket and that the lockfile mechanism is active.

  • Run a plan to confirm the backend connection works:
terraform plan -var="gemini_api_key=placeholder"

Why pass a placeholder variable?

The gemini_api_key variable has no default, so Terraform would prompt for it. Passing a placeholder lets the plan run without needing your real key yet. The plan will show errors about the undefined output references in outputs.tf, which is expected since those resources do not exist yet.

The important confirmation is that Terraform communicates with the S3 backend. You should see output mentioning the backend initialization, not a local state warning.

  • Verify the state is empty by running:
terraform state list

This should return no output, confirming the state exists remotely but has no resources tracked yet.

How does the lockfile work?

During any Terraform operation that writes state, Terraform creates a .tflock file in the S3 bucket alongside your state file. The lock file for this project would be securedocai/terraform.tfstate.tflock.

If another process tries to modify state while the lock is held, it will fail with a lock conflict error. The lock is automatically removed when the operation completes.

State list shows a local state warning?

  • If Terraform mentions local state or asks to migrate, your backend "s3" block may not have been picked up during init. Re-run terraform init and type yes if prompted to migrate state.
  • Make sure you did not accidentally create a terraform.tfstate file locally. If one exists, delete it and re-run terraform init.

Help me verify my Terraform remote backend is working.

Your remote state backend is ready and locked down. Next up, you will create the KMS encryption key and the encrypted S3 bucket where uploaded documents will live.

Create KMS Key and Encrypted S3 Bucket

Your remote state backend is working. Now it is time to build the encryption foundation for your document pipeline.

Any pipeline that processes sensitive documents needs encryption at rest. AWS offers a default encryption key, but a Customer Managed Key (CMK) gives you full control over key rotation, access policies, and audit trails. In this step, you will create a KMS key and use it to encrypt an S3 bucket that stores uploaded documents.

In this step, get ready to:
  • Create a KMS Customer Managed Key with automatic rotation.
  • Provision an encrypted, versioned S3 bucket with public access blocked.
  • Verify encryption is active using the AWS CLI.
Create KMS CMK with alias

A KMS Customer Managed Key encrypts data at rest and gives you control over who can decrypt it. You will also create an alias so that humans can reference the key by name instead of by ARN.

  • Create a new file called kms.tf in your main project directory (the same folder as main.tf).
  • Add the KMS key and alias resources by pasting this code into kms.tf:
resource "aws_kms_key" "main" {
  description             = "SecureDocAI encryption key for S3 and Secrets Manager"
  deletion_window_in_days = 7
  enable_key_rotation     = true
}

resource "aws_kms_alias" "main" {
  name          = "alias/${var.project_name}-key"
  target_key_id = aws_kms_key.main.key_id
}

What does this code do?

The aws_kms_key resource creates a symmetric encryption key managed by AWS KMS. Setting enable_key_rotation = true means AWS automatically rotates the underlying key material every year.

The aws_kms_alias gives the key a human-readable name like alias/securedocai-key. This makes it easier to identify in the AWS console and in logs.

  • Save kms.tf.

Why a Customer Managed Key instead of the default?

AWS provides a free default key (aws/s3), but you cannot control its access policy or audit which principals used it. A CMK lets you restrict decrypt access to specific Lambda roles and log every usage in CloudTrail.

For a pipeline handling PII documents, this separation of concerns is critical. You will reference this same key from S3, Secrets Manager, and your Lambda roles later.

Create the encrypted versioned S3 bucket

The document bucket is where uploaded PDFs land. It needs four layers of protection: KMS encryption, versioning (so objects cannot be silently overwritten), blocked public access, and an event notification that triggers processing.

  • Create a new file called s3.tf in your main project directory.
  • Add the bucket, versioning, encryption, and public access block by pasting this code into s3.tf:
resource "aws_s3_bucket" "documents" {
  bucket_prefix = "${var.project_name}-docs-"
}

resource "aws_s3_bucket_versioning" "documents" {
  bucket = aws_s3_bucket.documents.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "documents" {
  bucket = aws_s3_bucket.documents.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.main.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_public_access_block" "documents" {
  bucket = aws_s3_bucket.documents.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

What does this code do?

  • aws_s3_bucket creates the document bucket with a generated suffix (e.g. securedocai-docs-a1b2c).
  • aws_s3_bucket_versioning enables versioning so that overwriting an object preserves previous versions. This protects against accidental deletions.
  • aws_s3_bucket_server_side_encryption_configuration forces all objects to be encrypted with your CMK. The bucket_key_enabled = true setting reduces KMS API calls (and cost) by caching a per-bucket data key.
  • aws_s3_bucket_public_access_block ensures no bucket policy or ACL can accidentally expose the bucket to the internet.
  • Now add the S3 event notification block at the bottom of s3.tf. This resource references the SQS queue you will create in the next step:
resource "aws_s3_bucket_notification" "documents" {
  bucket = aws_s3_bucket.documents.id

  queue {
    queue_arn     = aws_sqs_queue.processing.arn
    events        = ["s3:ObjectCreated:*"]
    filter_prefix = "inbox/"
  }
}

What does this notification do?

When any object is created under the inbox/ prefix, S3 sends a message to the SQS processing queue. This triggers the document processor Lambda without polling.

The aws_sqs_queue.processing resource does not exist yet. You will create it in Step 3. Terraform will not let you apply this block until SQS is defined, which is expected.

  • Save s3.tf.

✔️ Awesome, I've got everything!

Great. Double check you have saved both kms.tf and s3.tf.

ⓧ I'd like to double check the full code

Here is the complete kms.tf:

resource "aws_kms_key" "main" {
  description             = "SecureDocAI encryption key for S3 and Secrets Manager"
  deletion_window_in_days = 7
  enable_key_rotation     = true
}

resource "aws_kms_alias" "main" {
  name          = "alias/${var.project_name}-key"
  target_key_id = aws_kms_key.main.key_id
}

And the complete s3.tf:

resource "aws_s3_bucket" "documents" {
  bucket_prefix = "${var.project_name}-docs-"
}

resource "aws_s3_bucket_versioning" "documents" {
  bucket = aws_s3_bucket.documents.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "documents" {
  bucket = aws_s3_bucket.documents.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.main.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_public_access_block" "documents" {
  bucket = aws_s3_bucket.documents.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_notification" "documents" {
  bucket = aws_s3_bucket.documents.id

  queue {
    queue_arn     = aws_sqs_queue.processing.arn
    events        = ["s3:ObjectCreated:*"]
    filter_prefix = "inbox/"
  }
}
Apply and verify encryption

Your configuration references resources that do not exist yet (the SQS queue in s3.tf and the Lambda functions in outputs.tf). Terraform validates the entire configuration before applying, so you need to temporarily hide those forward references.

  • Comment out the aws_s3_bucket_notification block at the bottom of s3.tf by wrapping it in a block comment:
/*
resource "aws_s3_bucket_notification" "documents" {
  bucket = aws_s3_bucket.documents.id

  queue {
    queue_arn     = aws_sqs_queue.processing.arn
    events        = ["s3:ObjectCreated:*"]
    filter_prefix = "inbox/"
  }
}
*/
  • Rename outputs.tf temporarily so Terraform ignores it. Run this command:
mv outputs.tf outputs.tf.bak

Why hide these files?

Terraform performs static validation on every .tf file in the directory. If any file references a resource that is not declared anywhere, Terraform exits with a "Reference to undeclared resource" error. Renaming outputs.tf and commenting the notification block lets you apply the resources that are ready now.

You will restore both after SQS and Lambda are created in later steps.

  • Run terraform apply to create the KMS key and S3 bucket:
terraform apply -var="gemini_api_key=placeholder"

Why pass a placeholder for gemini_api_key?

The gemini_api_key variable has no default value, so Terraform requires it at every plan and apply. You are not creating any resource that uses this key yet, so a placeholder value is fine for now.

  • Terraform will show a plan to create 6 resources (KMS key, KMS alias, S3 bucket, versioning, encryption config, public access block). Type yes when prompted to approve.
  • After the apply completes, note the bucket name in the output. It will look like securedocai-docs-XXXXX with a random suffix.
  • Verify the encryption configuration by running this command (replace the bucket name with your actual bucket name):
aws s3api get-bucket-encryption --bucket [[BUCKET_NAME="your bucket name e.g. securedocai-docs-a1b2c"]]

You should see output showing SSEAlgorithm as aws:kms and a KMSMasterKeyID field with your CMK ARN.

Seeing a "Reference to undeclared resource" error?

  • Make sure you renamed outputs.tf to outputs.tf.bak. Any .tf file in the directory is parsed by Terraform.
  • Confirm the aws_s3_bucket_notification block in s3.tf is fully wrapped in /* ... */ comment markers.
  • Check that no other .tf file references aws_sqs_queue, aws_lambda_function, or any resource not yet created.

Help me fix a Terraform undeclared resource error

  • Now restore outputs.tf by running:
mv outputs.tf.bak outputs.tf
  • In s3.tf, remove the /* */ comment markers around the aws_s3_bucket_notification block so it is back to active code.
  • Save s3.tf.

These files will work again in Step 3

Once you create the SQS queue and Lambda functions in the next steps, all forward references will resolve. You will not need to comment or rename anything again.

Your encryption foundation is in place. Next up, you will create the SQS processing queue and dead-letter queue that form the retry backbone of your pipeline.

Build the SQS Processing Queue and Dead-Letter Queue

Your encrypted S3 bucket is ready to receive documents, but uploading a file is only the first step. You need a way to trigger processing automatically when a new document lands in the bucket.

In this step, you will build an event-driven processing layer using SQS (Simple Queue Service). When a file arrives in S3, an event notification places a message on a processing queue. If processing fails, a dead-letter queue catches the failed message so it does not block the pipeline.

In this step, get ready to:
  • Create the dead-letter queue and main processing queue with retry logic.
  • Wire the S3 bucket notification to the processing queue.
  • Test the full event flow by uploading a file and confirming a message arrives in SQS.
Create the DLQ and main processing queue

The processing queue receives S3 event notifications and feeds them to a Lambda function (built in the next step). The dead-letter queue captures messages that fail processing after 3 attempts, so you can investigate failures without losing data.

  • Create a new file called sqs.tf in your main project directory (the same folder as main.tf).
  • Add the DLQ and processing queue definitions by pasting this code:
resource "aws_sqs_queue" "dlq" {
  name                      = "${var.project_name}-dlq"
  message_retention_seconds = 1209600
}

resource "aws_sqs_queue" "processing" {
  name                       = "${var.project_name}-processing"
  visibility_timeout_seconds = 300
  message_retention_seconds  = 86400

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 3
  })
}

What does this code do?

The DLQ retains failed messages for 14 days (1,209,600 seconds), giving you time to investigate.

The processing queue has a visibility_timeout_seconds of 300 (5 minutes). This must be at least as long as the Lambda timeout to prevent duplicate processing while a message is being handled.

The redrive_policy tells SQS to move any message that fails 3 times (maxReceiveCount) to the DLQ automatically.

Add the queue policy allowing S3 notifications

SQS queues deny messages from other services by default. You need a resource-based policy that explicitly grants S3 permission to send event notification messages to your processing queue.

  • In sqs.tf, add the following code below the processing queue resource:
data "aws_caller_identity" "current" {}

resource "aws_sqs_queue_policy" "processing" {
  queue_url = aws_sqs_queue.processing.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "AllowS3Notification"
        Effect    = "Allow"
        Principal = { Service = "s3.amazonaws.com" }
        Action    = "sqs:SendMessage"
        Resource  = aws_sqs_queue.processing.arn
        Condition = {
          ArnLike = {
            "aws:SourceArn" = aws_s3_bucket.documents.arn
          }
        }
      }
    ]
  })
}

What does this policy do?

The Condition block uses ArnLike to scope the permission. Only your specific document bucket can send messages to this queue. This is IAM least-privilege applied at the resource policy level.

  • Save sqs.tf.

Seeing a syntax error?

Make sure every opening brace { has a matching closing brace }, and that the jsonencode() block wraps the entire JSON object including the outer braces.

help me fix a syntax error in my Terraform SQS queue policy

✔️ Awesome, I've got everything!

Great. Double check you have saved sqs.tf with both queue resources and the policy.

ⓧ I'd like to double check the full code

resource "aws_sqs_queue" "dlq" {
  name                      = "${var.project_name}-dlq"
  message_retention_seconds = 1209600
}

resource "aws_sqs_queue" "processing" {
  name                       = "${var.project_name}-processing"
  visibility_timeout_seconds = 300
  message_retention_seconds  = 86400

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 3
  })
}

data "aws_caller_identity" "current" {}

resource "aws_sqs_queue_policy" "processing" {
  queue_url = aws_sqs_queue.processing.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "AllowS3Notification"
        Effect    = "Allow"
        Principal = { Service = "s3.amazonaws.com" }
        Action    = "sqs:SendMessage"
        Resource  = aws_sqs_queue.processing.arn
        Condition = {
          ArnLike = {
            "aws:SourceArn" = aws_s3_bucket.documents.arn
          }
        }
      }
    ]
  })
}
Verify the S3 event notification wiring

In the previous step, your s3.tf already contains the aws_s3_bucket_notification resource that sends s3:ObjectCreated:* events to the processing queue. Until now, Terraform could not resolve that reference because the SQS queue did not exist yet.

  • Open s3.tf and confirm you have the following notification block at the bottom of the file:
resource "aws_s3_bucket_notification" "documents" {
  bucket = aws_s3_bucket.documents.id

  queue {
    queue_arn     = aws_sqs_queue.processing.arn
    events        = ["s3:ObjectCreated:*"]
    filter_prefix = "inbox/"
  }
}

Why filter on inbox/?

The filter_prefix ensures only files uploaded to the inbox/ path trigger processing. This keeps the pipeline from reacting to temporary files, logs, or output written to other paths in the same bucket.

Apply and test the event flow

Now that sqs.tf exists, Terraform can resolve all the cross-references between S3 and SQS. Time to provision the queues and verify the full event chain.

What about the Lambda outputs?

Your outputs.tf references aws_lambda_function.presign and aws_lambda_function.processor, which do not exist yet. Terraform will error on these during plan.

Comment out the two Lambda-related outputs in outputs.tf for now. You will uncomment them in the next step when you deploy the Lambda functions.

  • In outputs.tf, comment out the presign_lambda_name and processor_lambda_name output blocks by adding # at the start of each line.
  • Save outputs.tf.
  • Run terraform apply to provision the queues by running this command:
terraform apply
  • Terraform will show a plan to create the SQS queues, the queue policy, and the S3 bucket notification. Type yes when prompted to confirm.

Seeing reference errors?

If Terraform reports an error about aws_lambda_function not being defined, make sure you commented out both Lambda output blocks in outputs.tf. Each output block is 3 lines (the output declaration, value line, and closing brace).

help me fix Terraform reference errors when applying my SQS configuration

Once the apply completes, verify the queues exist and test the event notification end-to-end.

  • Create a small test file and upload it to the inbox path by running these commands:
echo "test document" > /tmp/test.pdf
aws s3 cp /tmp/test.pdf s3://[[DOCBUCKET="your document bucket name from terraform output"]]/inbox/test.pdf

Find your bucket name quickly

Run terraform output document_bucket_name to get the exact bucket name Terraform generated. It will look something like securedocai-docs-abc123.

  • Wait 5-10 seconds for the S3 event notification to fire, then check the SQS queue for the message by running:
aws sqs receive-message --queue-url $(terraform output -raw processing_queue_url) --max-number-of-messages 1 --wait-time-seconds 10
  • You should see a JSON response containing a Messages array. The Body field contains the S3 event payload with your uploaded object key (inbox/test.pdf).

No messages returned?

S3 event notifications can take a few seconds to deliver. Try the receive command again with --wait-time-seconds 20 for long polling.

If you still see no messages, confirm the upload path includes the inbox/ prefix. Files uploaded to the bucket root do not trigger the notification because of the filter_prefix setting.

help me debug why my S3 event notification is not delivering messages to SQS

Your event-driven plumbing is in place. Next up, you will build the Lambda functions that generate pre-signed upload URLs and process documents with Gemini AI.

Deploy Lambda Functions and Secrets Manager

Your encrypted storage layer and message queue are ready. Now you need the compute layer that actually does the work: one Lambda function to generate secure upload URLs, and another to process documents with AI.

Both functions need access to your KMS key and S3 bucket. The processor also needs to retrieve your Gemini API key at runtime. You will store that key in Secrets Manager (encrypted with your CMK) so it never appears in code or environment variables.

In this step, get ready to:
  • Store your Gemini API key in AWS Secrets Manager encrypted with your CMK.
  • Deploy a pre-signed URL generator Lambda with least-privilege IAM.
  • Deploy a document processor Lambda that calls the Gemini API and is triggered by SQS.
Store Gemini API key in Secrets Manager

Your processor Lambda needs your Gemini API key at runtime. Hardcoding it in an environment variable would expose it in the Terraform state in plain text. Secrets Manager stores the value encrypted with your CMK, and the Lambda retrieves it on demand.

  • Create a new file called secrets.tf in your main project directory by running:
touch secrets.tf
  • Add the following code to secrets.tf:
resource "aws_secretsmanager_secret" "gemini_api_key" {
  name        = "${var.project_name}/gemini-api-key"
  description = "Gemini API key for document extraction"
  kms_key_id  = aws_kms_key.main.arn
}

resource "aws_secretsmanager_secret_version" "gemini_api_key" {
  secret_id     = aws_secretsmanager_secret.gemini_api_key.id
  secret_string = var.gemini_api_key
}

What does this code do?

The aws_secretsmanager_secret resource creates the secret container. The kms_key_id argument encrypts it with your CMK instead of the default AWS-managed key.

The aws_secretsmanager_secret_version resource stores the actual key value. It reads from var.gemini_api_key, which is marked sensitive = true so Terraform never displays it in logs.

At runtime, your Lambda calls GetSecretValue to retrieve the key. This keeps the secret out of environment variables and Lambda configuration.

  • Save secrets.tf.
  • You won't see a change yet. The next chunks define the IAM roles and Lambda functions that reference this secret.

Next, you need to provide the actual Gemini API key value for Terraform to use. The standard approach is a .tfvars file.

  • Create a file called terraform.tfvars in your main project directory (not the bootstrap directory).
  • Add the following content, replacing the placeholder with your real Gemini API key:
aws_region     = "us-east-1"
project_name   = "securedocai"
gemini_api_key = "your-gemini-api-key-here"

Why a .tfvars file?

Terraform automatically loads any file named terraform.tfvars in the working directory. Because gemini_api_key is marked sensitive = true, Terraform redacts its value from all CLI output.

Never commit this file to version control. Add terraform.tfvars to your .gitignore.

Deploy the pre-signed URL generator Lambda

The presign Lambda generates temporary upload URLs so clients can upload PDFs directly to S3 without needing AWS credentials. It needs s3:PutObject scoped to the inbox/ prefix, plus KMS permissions to encrypt the uploaded objects.

  • Create a new file called iam.tf and add the presign Lambda IAM role:
# Presign Lambda Role
resource "aws_iam_role" "presign_lambda" {
  name = "${var.project_name}-presign-lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = { Service = "lambda.amazonaws.com" }
      }
    ]
  })
}

resource "aws_iam_role_policy" "presign_lambda" {
  name = "presign-policy"
  role = aws_iam_role.presign_lambda.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:PutObject"]
        Resource = "${aws_s3_bucket.documents.arn}/inbox/*"
      },
      {
        Effect   = "Allow"
        Action   = ["kms:GenerateDataKey", "kms:Decrypt"]
        Resource = aws_kms_key.main.arn
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "presign_lambda_logs" {
  role       = aws_iam_role.presign_lambda.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

What does this IAM role do?

This follows least-privilege. The role can only PutObject under the inbox/ prefix. It cannot read, delete, or list objects.

The kms:GenerateDataKey permission is required because SSE-KMS encryption needs to generate a data key for each uploaded object. The AWSLambdaBasicExecutionRole attachment grants CloudWatch Logs permissions so the Lambda can write logs.

  • Save iam.tf.

Now write the Python handler that generates pre-signed PUT URLs.

  • Create the Lambda source directory and file by running:
mkdir -p lambda/presign
  • Create lambda/presign/index.py and add the following code:
import json
import os
import uuid
import boto3

s3_client = boto3.client("s3")

BUCKET_NAME = os.environ["BUCKET_NAME"]
KMS_KEY_ID = os.environ["KMS_KEY_ID"]


def handler(event, context):
    filename = event.get("filename", f"{uuid.uuid4()}.pdf")
    object_key = f"inbox/{filename}"

    presigned_url = s3_client.generate_presigned_url(
        "put_object",
        Params={
            "Bucket": BUCKET_NAME,
            "Key": object_key,
            "ContentType": "application/pdf",
            "ServerSideEncryption": "aws:kms",
            "SSEKMSKeyId": KMS_KEY_ID,
        },
        ExpiresIn=300,
    )

    return {
        "statusCode": 200,
        "body": json.dumps({
            "upload_url": presigned_url,
            "object_key": object_key,
            "expires_in_seconds": 300,
        }),
    }

What does this handler do?

The handler reads the bucket name and KMS key ARN from environment variables (set by Terraform). If the caller provides a filename in the event, it uses that. Otherwise it generates a UUID filename.

The pre-signed URL is valid for 300 seconds (5 minutes). It includes SSE-KMS parameters so any upload through this URL is automatically encrypted with your CMK. No external dependencies are needed beyond boto3, which is bundled in the Lambda runtime.

  • Save lambda/presign/index.py.

Now define the Terraform resource that packages and deploys this Lambda.

  • Create a new file called lambda.tf and add the presign Lambda configuration:
data "archive_file" "presign" {
  type        = "zip"
  source_file = "${path.module}/lambda/presign/index.py"
  output_path = "${path.module}/.build/presign.zip"
}

resource "aws_lambda_function" "presign" {
  function_name    = "${var.project_name}-presign"
  role             = aws_iam_role.presign_lambda.arn
  handler          = "index.handler"
  runtime          = "python3.13"
  filename         = data.archive_file.presign.output_path
  source_code_hash = data.archive_file.presign.output_base64sha256
  timeout          = 10

  environment {
    variables = {
      BUCKET_NAME = aws_s3_bucket.documents.id
      KMS_KEY_ID  = aws_kms_key.main.arn
    }
  }
}

What does this configuration do?

The archive_file data source zips the Python file into .build/presign.zip automatically. Terraform uploads this zip when it creates the Lambda.

The source_code_hash argument means Terraform re-deploys the function whenever the Python code changes. The runtime is python3.13 which runs on Amazon Linux 2023.

  • Save lambda.tf.

Seeing errors about missing references?

If you run terraform plan now, it may fail because outputs.tf references the processor Lambda which does not exist yet. This is expected. You will add the processor in the next substep and apply everything together.

help me understand why Terraform fails when outputs reference resources that don't exist yet

Deploy the document processor Lambda

The processor Lambda is triggered by the SQS queue whenever a new document lands in the inbox/ prefix. It fetches the PDF from S3, base64-encodes it, sends it to the Gemini generateContent API, and logs the extracted data.

  • Open iam.tf and add the processor Lambda IAM role below the presign role:
# Processor Lambda Role
resource "aws_iam_role" "processor_lambda" {
  name = "${var.project_name}-processor-lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = { Service = "lambda.amazonaws.com" }
      }
    ]
  })
}

resource "aws_iam_role_policy" "processor_lambda" {
  name = "processor-policy"
  role = aws_iam_role.processor_lambda.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject"]
        Resource = "${aws_s3_bucket.documents.arn}/*"
      },
      {
        Effect   = "Allow"
        Action   = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
        Resource = aws_sqs_queue.processing.arn
      },
      {
        Effect   = "Allow"
        Action   = ["secretsmanager:GetSecretValue"]
        Resource = aws_secretsmanager_secret.gemini_api_key.arn
      },
      {
        Effect   = "Allow"
        Action   = ["kms:Decrypt"]
        Resource = aws_kms_key.main.arn
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "processor_lambda_logs" {
  role       = aws_iam_role.processor_lambda.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

Why four separate permission statements?

Each statement grants the minimum permissions for one responsibility. s3:GetObject reads the uploaded PDF. The SQS actions let Lambda poll and delete processed messages. secretsmanager:GetSecretValue retrieves the Gemini key. kms:Decrypt decrypts both the S3 object and the secret.

This is IAM least-privilege in action. If this Lambda were compromised, it could not write to S3, send SQS messages, or create new secrets.

  • Save iam.tf.

Now write the processor Lambda code. This function has three parts: a helper to fetch the API key, a function to call Gemini, and the main handler that processes SQS events.

  • Create the processor Lambda source directory by running:
mkdir -p lambda/processor
  • Create lambda/processor/index.py and add the imports, configuration, and Gemini integration function:
import json
import os
import base64
import urllib.request
import boto3

s3_client = boto3.client("s3")
secrets_client = boto3.client("secretsmanager")

SECRET_NAME = os.environ["SECRET_NAME"]
GEMINI_MODEL = "gemini-3.5-flash"
GEMINI_ENDPOINT = (
    f"https://generativelanguage.googleapis.com/v1beta/models/{GEMINI_MODEL}:generateContent"
)


def get_api_key():
    response = secrets_client.get_secret_value(SecretId=SECRET_NAME)
    return response["SecretString"]


def extract_document(pdf_base64, api_key):
    payload = {
        "contents": [
            {
                "parts": [
                    {
                        "text": (
                            "Extract all structured data from this document. "
                            "Return a JSON object with fields: document_type, "
                            "entities (list of name/value pairs found), dates, "
                            "amounts, and any PII detected (redacted). "
                            "Respond with valid JSON only."
                        )
                    },
                    {
                        "inlineData": {
                            "mimeType": "application/pdf",
                            "data": pdf_base64,
                        }
                    },
                ]
            }
        ]
    }

    headers = {
        "Content-Type": "application/json",
        "x-goog-api-key": api_key,
    }

    req = urllib.request.Request(
        GEMINI_ENDPOINT,
        data=json.dumps(payload).encode("utf-8"),
        headers=headers,
        method="POST",
    )

    with urllib.request.urlopen(req, timeout=120) as resp:
        result = json.loads(resp.read().decode("utf-8"))

    return result["candidates"][0]["content"]["parts"][0]["text"]

What does this code do?

The module sets up two clients (S3 and Secrets Manager) and reads the secret name from an environment variable. get_api_key() fetches the Gemini API key from Secrets Manager at runtime.

The extract_document() function builds a request to the Gemini generateContent API. It sends the PDF as base64 inline data with a prompt asking for structured extraction. The x-goog-api-key header authenticates the request. No external packages are needed since it uses Python's built-in urllib.request.

  • Now add the main handler function at the bottom of lambda/processor/index.py:
def handler(event, context):
    api_key = get_api_key()

    for record in event["Records"]:
        body = json.loads(record["body"])
        s3_event = body["Records"][0]
        bucket = s3_event["s3"]["bucket"]["name"]
        key = s3_event["s3"]["object"]["key"]

        print(f"Processing: s3://{bucket}/{key}")

        response = s3_client.get_object(Bucket=bucket, Key=key)
        pdf_bytes = response["Body"].read()
        pdf_base64 = base64.b64encode(pdf_bytes).decode("utf-8")

        extraction_result = extract_document(pdf_base64, api_key)
        print(f"Extraction result for {key}:")
        print(extraction_result)

    return {"statusCode": 200, "body": "Processing complete"}

What does the handler do?

The handler processes SQS records. Each SQS message body contains an S3 event notification (a nested JSON structure). The handler parses through to find the bucket name and object key.

It then fetches the PDF from S3, base64-encodes it, and passes it to extract_document(). The Gemini response (structured JSON) is printed to CloudWatch Logs for verification.

  • Save lambda/processor/index.py.

Finally, add the processor Lambda Terraform configuration and the SQS event source mapping.

  • Open lambda.tf and add the following below the presign Lambda block:
data "archive_file" "processor" {
  type        = "zip"
  source_file = "${path.module}/lambda/processor/index.py"
  output_path = "${path.module}/.build/processor.zip"
}

resource "aws_lambda_function" "processor" {
  function_name    = "${var.project_name}-processor"
  role             = aws_iam_role.processor_lambda.arn
  handler          = "index.handler"
  runtime          = "python3.13"
  filename         = data.archive_file.processor.output_path
  source_code_hash = data.archive_file.processor.output_base64sha256
  timeout          = 300
  memory_size      = 256

  environment {
    variables = {
      SECRET_NAME = aws_secretsmanager_secret.gemini_api_key.name
      REGION      = var.aws_region
    }
  }
}

resource "aws_lambda_event_source_mapping" "sqs_to_processor" {
  event_source_arn = aws_sqs_queue.processing.arn
  function_name    = aws_lambda_function.processor.arn
  batch_size       = 1
  enabled          = true
}

Why timeout 300 and batch size 1?

The Gemini API call can take up to 30 seconds for large PDFs. A 300-second timeout provides headroom for network variability. This matches the SQS visibility timeout (also 300 seconds) so messages are not reprocessed while the Lambda is still running.

Batch size 1 means each Lambda invocation processes one document at a time. This simplifies error handling. If extraction fails, only that one message returns to the queue (up to 3 retries before landing in the DLQ).

  • Save lambda.tf.

All the configuration files are ready. Deploy everything with a single apply.

  • Apply the Terraform configuration by running:
terraform apply

What will Terraform prompt?

Terraform will show a plan creating the Secrets Manager secret, both IAM roles, both Lambda functions, and the SQS event source mapping. Type yes when prompted to confirm. The gemini_api_key value appears as (sensitive value) in the output because it is marked sensitive.

  • Wait for the apply to complete. You should see output ending with resources added and zero errors.

Seeing a permissions or access denied error?

Make sure your AWS CLI credentials have permissions to create IAM roles, Lambda functions, Secrets Manager secrets, and SQS event source mappings. If you are using an IAM user, it needs the AdministratorAccess policy or equivalent permissions for this project.

help me debug a Terraform apply access denied error when creating Lambda functions and IAM roles

Now test the presign Lambda to confirm it generates valid upload URLs.

  • Invoke the presign Lambda by running:
aws lambda invoke \
  --function-name securedocai-presign \
  --cli-binary-format raw-in-base64-out \
  --payload '{"filename": "test.pdf"}' \
  response.json
  • You should see output containing "StatusCode": 200 in your terminal.
  • View the response by running:
cat response.json
  • You should see a JSON response with upload_url, object_key, and expires_in_seconds fields. The upload_url is a long pre-signed S3 URL.

StatusCode is 200 but response shows FunctionError?

A 200 StatusCode means Lambda ran the function. Check response.json for the actual error. Common causes: the BUCKET_NAME environment variable is empty (check lambda.tf references aws_s3_bucket.documents.id), or the IAM role is missing kms:GenerateDataKey permission.

help me debug my presign Lambda returning a FunctionError

✔️ Awesome, I've got everything!

Great. Double check you have saved all files: secrets.tf, iam.tf, lambda.tf, lambda/presign/index.py, and lambda/processor/index.py.

ⓧ I'd like to double check the full code

Here are the complete files created in this step:

resource "aws_secretsmanager_secret" "gemini_api_key" {
  name        = "${var.project_name}/gemini-api-key"
  description = "Gemini API key for document extraction"
  kms_key_id  = aws_kms_key.main.arn
}

resource "aws_secretsmanager_secret_version" "gemini_api_key" {
  secret_id     = aws_secretsmanager_secret.gemini_api_key.id
  secret_string = var.gemini_api_key
}
# Presign Lambda Role
resource "aws_iam_role" "presign_lambda" {
  name = "${var.project_name}-presign-lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = { Service = "lambda.amazonaws.com" }
      }
    ]
  })
}

resource "aws_iam_role_policy" "presign_lambda" {
  name = "presign-policy"
  role = aws_iam_role.presign_lambda.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:PutObject"]
        Resource = "${aws_s3_bucket.documents.arn}/inbox/*"
      },
      {
        Effect   = "Allow"
        Action   = ["kms:GenerateDataKey", "kms:Decrypt"]
        Resource = aws_kms_key.main.arn
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "presign_lambda_logs" {
  role       = aws_iam_role.presign_lambda.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

# Processor Lambda Role
resource "aws_iam_role" "processor_lambda" {
  name = "${var.project_name}-processor-lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = { Service = "lambda.amazonaws.com" }
      }
    ]
  })
}

resource "aws_iam_role_policy" "processor_lambda" {
  name = "processor-policy"
  role = aws_iam_role.processor_lambda.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject"]
        Resource = "${aws_s3_bucket.documents.arn}/*"
      },
      {
        Effect   = "Allow"
        Action   = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
        Resource = aws_sqs_queue.processing.arn
      },
      {
        Effect   = "Allow"
        Action   = ["secretsmanager:GetSecretValue"]
        Resource = aws_secretsmanager_secret.gemini_api_key.arn
      },
      {
        Effect   = "Allow"
        Action   = ["kms:Decrypt"]
        Resource = aws_kms_key.main.arn
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "processor_lambda_logs" {
  role       = aws_iam_role.processor_lambda.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}
data "archive_file" "presign" {
  type        = "zip"
  source_file = "${path.module}/lambda/presign/index.py"
  output_path = "${path.module}/.build/presign.zip"
}

resource "aws_lambda_function" "presign" {
  function_name    = "${var.project_name}-presign"
  role             = aws_iam_role.presign_lambda.arn
  handler          = "index.handler"
  runtime          = "python3.13"
  filename         = data.archive_file.presign.output_path
  source_code_hash = data.archive_file.presign.output_base64sha256
  timeout          = 10

  environment {
    variables = {
      BUCKET_NAME = aws_s3_bucket.documents.id
      KMS_KEY_ID  = aws_kms_key.main.arn
    }
  }
}

data "archive_file" "processor" {
  type        = "zip"
  source_file = "${path.module}/lambda/processor/index.py"
  output_path = "${path.module}/.build/processor.zip"
}

resource "aws_lambda_function" "processor" {
  function_name    = "${var.project_name}-processor"
  role             = aws_iam_role.processor_lambda.arn
  handler          = "index.handler"
  runtime          = "python3.13"
  filename         = data.archive_file.processor.output_path
  source_code_hash = data.archive_file.processor.output_base64sha256
  timeout          = 300
  memory_size      = 256

  environment {
    variables = {
      SECRET_NAME = aws_secretsmanager_secret.gemini_api_key.name
      REGION      = var.aws_region
    }
  }
}

resource "aws_lambda_event_source_mapping" "sqs_to_processor" {
  event_source_arn = aws_sqs_queue.processing.arn
  function_name    = aws_lambda_function.processor.arn
  batch_size       = 1
  enabled          = true
}
import json
import os
import uuid
import boto3

s3_client = boto3.client("s3")

BUCKET_NAME = os.environ["BUCKET_NAME"]
KMS_KEY_ID = os.environ["KMS_KEY_ID"]


def handler(event, context):
    filename = event.get("filename", f"{uuid.uuid4()}.pdf")
    object_key = f"inbox/{filename}"

    presigned_url = s3_client.generate_presigned_url(
        "put_object",
        Params={
            "Bucket": BUCKET_NAME,
            "Key": object_key,
            "ContentType": "application/pdf",
            "ServerSideEncryption": "aws:kms",
            "SSEKMSKeyId": KMS_KEY_ID,
        },
        ExpiresIn=300,
    )

    return {
        "statusCode": 200,
        "body": json.dumps({
            "upload_url": presigned_url,
            "object_key": object_key,
            "expires_in_seconds": 300,
        }),
    }
import json
import os
import base64
import urllib.request
import boto3

s3_client = boto3.client("s3")
secrets_client = boto3.client("secretsmanager")

SECRET_NAME = os.environ["SECRET_NAME"]
GEMINI_MODEL = "gemini-3.5-flash"
GEMINI_ENDPOINT = (
    f"https://generativelanguage.googleapis.com/v1beta/models/{GEMINI_MODEL}:generateContent"
)


def get_api_key():
    response = secrets_client.get_secret_value(SecretId=SECRET_NAME)
    return response["SecretString"]


def extract_document(pdf_base64, api_key):
    payload = {
        "contents": [
            {
                "parts": [
                    {
                        "text": (
                            "Extract all structured data from this document. "
                            "Return a JSON object with fields: document_type, "
                            "entities (list of name/value pairs found), dates, "
                            "amounts, and any PII detected (redacted). "
                            "Respond with valid JSON only."
                        )
                    },
                    {
                        "inlineData": {
                            "mimeType": "application/pdf",
                            "data": pdf_base64,
                        }
                    },
                ]
            }
        ]
    }

    headers = {
        "Content-Type": "application/json",
        "x-goog-api-key": api_key,
    }

    req = urllib.request.Request(
        GEMINI_ENDPOINT,
        data=json.dumps(payload).encode("utf-8"),
        headers=headers,
        method="POST",
    )

    with urllib.request.urlopen(req, timeout=120) as resp:
        result = json.loads(resp.read().decode("utf-8"))

    return result["candidates"][0]["content"]["parts"][0]["text"]


def handler(event, context):
    api_key = get_api_key()

    for record in event["Records"]:
        body = json.loads(record["body"])
        s3_event = body["Records"][0]
        bucket = s3_event["s3"]["bucket"]["name"]
        key = s3_event["s3"]["object"]["key"]

        print(f"Processing: s3://{bucket}/{key}")

        response = s3_client.get_object(Bucket=bucket, Key=key)
        pdf_bytes = response["Body"].read()
        pdf_base64 = base64.b64encode(pdf_bytes).decode("utf-8")

        extraction_result = extract_document(pdf_base64, api_key)
        print(f"Extraction result for {key}:")
        print(extraction_result)

    return {"statusCode": 200, "body": "Processing complete"}

Both Lambda functions are deployed and the presign function is generating valid upload URLs. Next up, you will run a full end-to-end test and document the architecture.

Run End-to-End Test and Write Documentation

Your entire infrastructure is deployed. Every piece of the pipeline exists in AWS. Now it is time to prove the system actually works end-to-end.

In this step, you will drive a document through the full pipeline and then write the portfolio documentation that demonstrates your architectural thinking.

In this step, get ready to:
  • Run a full integration test from pre-signed URL generation through Gemini extraction.
  • Write an Architecture Decision Record and cost log.
  • Create a STRIDE threat model for the pipeline.
Run a full flow integration test

The integration test exercises every component in sequence: the presign Lambda generates a URL, you upload a PDF through that URL, S3 fires an event to SQS, and the processor Lambda extracts structured data via Gemini.

  • Invoke the presign Lambda to generate a pre-signed upload URL by running this command:
aws lambda invoke \
    --function-name securedocai-presign \
    --cli-binary-format raw-in-base64-out \
    --payload '{"filename": "test-document.pdf"}' \
    response.json

What does this command do?

  • --function-name securedocai-presign targets your presign Lambda by its deployed name.
  • --cli-binary-format raw-in-base64-out is required for AWS CLI v2 when passing a JSON payload directly.
  • --payload sends a JSON event to the Lambda with a filename for the upload.
  • response.json is where the CLI writes the Lambda's response (the pre-signed URL).
  • View the response to get your pre-signed URL by running:
cat response.json

You should see a JSON body containing an upload_url, an object_key, and expires_in_seconds.

Create a test PDF quickly

If you do not have a PDF handy, you can create a simple one-page test file. Any small PDF works for this test. A scanned invoice or a simple text document is ideal.

  • Extract the upload URL from the response and upload a test PDF using curl by running:
# Parse the upload URL from the response
UPLOAD_URL=$(cat response.json | python3 -c "import sys,json; print(json.loads(json.load(sys.stdin)['body'])['upload_url'])")

# Upload a test PDF using the pre-signed URL
curl -X PUT \
    -H "Content-Type: application/pdf" \
    -H "x-amz-server-side-encryption: aws:kms" \
    --upload-file [[TEST_PDF_PATH="path/to/your/test.pdf"]] \
    "$UPLOAD_URL"

What is happening here?

The curl command performs a PUT request to the pre-signed URL. The Content-Type and encryption headers must match what the Lambda specified when generating the URL.

Once the upload completes, S3 automatically sends an event notification to your SQS processing queue. The processor Lambda picks up the message and calls the Gemini API.

  • Wait about 30 seconds for the SQS event to trigger the processor Lambda, then check the CloudWatch logs by running:
aws logs tail /aws/lambda/securedocai-processor --since 5m

You should see log entries showing Processing: s3:// followed by your bucket name and key, then the extracted JSON result from Gemini.

Not seeing any logs?

  • The processor Lambda may take up to 60 seconds to be triggered by SQS. Wait a bit longer and run the logs command again with --since 10m.
  • Verify the upload succeeded. If curl returned an XML error, the pre-signed URL may have expired (they last 5 minutes). Re-invoke the presign Lambda to get a fresh URL.
  • Check the DLQ for failed messages. If the processor Lambda is crashing, messages move to the dead-letter queue after 3 attempts.

Help me debug why my processor Lambda isn't being triggered by SQS

Write the ADR and cost log

Architecture Decision Records (ADRs) capture why you made specific technology choices. A cost log tracks the ongoing expense of your infrastructure. Both are essential portfolio documentation.

  • Create the documentation directory structure by running:
mkdir -p docs/adr
  • Confirm the directories exist by running:
ls docs

You should see adr listed in the output.

  • Create docs/adr/001-gemini-over-textract.md and paste the following content:
# ADR 001: Gemini API Over AWS Textract for Document Extraction

## Status

Accepted

## Context

We need an AI service to extract structured data from uploaded PDF documents.
The two primary options are AWS Textract (native AWS service) and the Google
Gemini API (external multi-modal LLM).

## Decision

We chose the Gemini API for document extraction.

## Reasons

- **Flexibility**: Gemini handles arbitrary extraction prompts, not just
  predefined form fields. We can ask for entities, dates, PII, and custom
  fields in a single call.
- **Cost**: Gemini 3.5 Flash offers a free tier for input and output tokens.
  Textract charges per page with no free tier for AnalyzeDocument.
- **Multi-model portability**: The integration pattern (Secrets Manager +
  HTTPS call) works with any LLM API. Swapping to Claude, GPT, or a
  future model requires changing one endpoint and one secret.
- **No AWS vendor lock-in for the AI layer**: Keeping the AI extraction
  external to AWS means the architecture remains portable across clouds.

## Trade-offs

- Textract offers native IAM integration and no external network calls.
- Textract results are deterministic; LLM output varies between calls.
- External API introduces a network dependency and potential latency.

## Consequences

- The processor Lambda must have internet access to reach the Gemini endpoint.
- We store the Gemini API key in Secrets Manager encrypted with our CMK.
- If Gemini is unavailable, documents queue in SQS and retry up to 3 times
  before moving to the DLQ.

What is an ADR?

An Architecture Decision Record documents a significant technology choice, the alternatives considered, and the reasoning. Interviewers and hiring managers use ADRs to assess your decision-making process, not just your implementation skills.

  • Create docs/cost-log.md and paste the following content:
# SecureDocAI Monthly Cost Log

## Resources and Monthly Costs

| Resource | Monthly Cost | Notes |
|----------|-------------|-------|
| KMS Customer Managed Key | $1.00/month | + $0.03 per 10,000 API requests (free tier covers 20,000/month) |
| Secrets Manager (1 secret) | $0.40/month | + $0.05 per 10,000 API calls |
| S3 Document Bucket | $0.00 | Free tier: 5 GB storage, 20,000 GET, 2,000 PUT/month |
| Lambda Functions (x2) | $0.00 | Free tier: 1,000,000 requests, 400,000 GB-seconds/month |
| SQS Queues (x2) | $0.00 | Free tier: 1,000,000 requests/month |
| Remote State S3 Bucket | ~$0.00 | Negligible (tiny state file) |
| Gemini API | $0.00 | Free tier covers input and output tokens |

## Total Estimated Monthly Cost

**~$1.40/month** (KMS key + Secrets Manager)

## Notes

- S3 free tier applies to accounts created before July 15, 2025 (12-month limit).
- Accounts created after July 15, 2025 receive up to $200 in credits for 6 months.
- All costs assume low-volume usage within free tier boundaries.
- The KMS key is the largest fixed cost. Delete it if not actively using the pipeline.

Why track costs?

A cost log demonstrates that you understand the financial impact of your architecture. It shows potential employers that you think about operational expenses, not just technical functionality.

  • Save both files.
  • Verify the files exist by running:
ls docs/adr/001-gemini-over-textract.md docs/cost-log.md

Both file paths should appear in the output without errors.

Write a threat model scaffold

A threat model identifies where your system is vulnerable to attack. The STRIDE framework categorises threats into six types, giving you a systematic way to analyse security.

  • Create docs/threat-model.md and paste the following content:
# SecureDocAI Threat Model (STRIDE)

## Overview

This document analyses the SecureDocAI pipeline using the STRIDE threat
modelling framework. Each category identifies a specific threat vector,
its impact, and the mitigation in place.

## S - Spoofing (Identity)

**Threat**: Pre-signed URL abuse. An attacker obtains a valid pre-signed
URL and uploads malicious content before the URL expires.

**Mitigation**:
- URLs expire after 5 minutes (short TTL).
- The presign Lambda is only invokable by authenticated IAM principals.
- S3 bucket only accepts objects in the `inbox/` prefix via the URL.

## T - Tampering (Data Integrity)

**Threat**: S3 object modification. An attacker with compromised credentials
could overwrite or modify documents after upload.

**Mitigation**:
- S3 versioning preserves all object versions (overwrites don't delete).
- KMS encryption ensures objects cannot be read without key access.
- Block public access prevents anonymous writes.

## R - Repudiation (Accountability)

**Threat**: CloudTrail logging gap. If CloudTrail is not enabled, there is
no audit trail of who uploaded what document and when.

**Mitigation**:
- Enable CloudTrail for S3 data events on the document bucket.
- Lambda invocation logs in CloudWatch provide processing audit trail.
- S3 access logging provides bucket-level access records.

## I - Information Disclosure (Confidentiality)

**Threat**: Unencrypted transit. Documents or API keys could be intercepted
if transmitted without TLS.

**Mitigation**:
- All S3 operations use HTTPS (enforced by SDK defaults).
- Gemini API calls use HTTPS.
- API key stored in Secrets Manager, never in environment variables as plaintext.
- KMS encryption at rest for S3 objects and secrets.

## D - Denial of Service (Availability)

**Threat**: SQS flooding. An attacker could flood the upload endpoint,
creating thousands of SQS messages that overwhelm the processor Lambda.

**Mitigation**:
- SQS visibility timeout prevents duplicate processing.
- Lambda concurrency limits can be configured to cap execution.
- DLQ isolates poison messages after 3 failed attempts.
- Pre-signed URLs require authenticated Lambda invocation to generate.

## E - Elevation of Privilege

**Threat**: IAM over-permission. If a Lambda role has excessive permissions,
a compromised function could access unrelated resources.

**Mitigation**:
- Each Lambda has its own dedicated IAM role.
- Permissions are scoped to specific resource ARNs (not wildcard).
- Presign Lambda can only PutObject to `inbox/*`, not read the bucket.
- Processor Lambda cannot write to S3, only read.

What is STRIDE?

STRIDE stands for Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege. It is a threat modelling framework developed at Microsoft that helps you systematically identify security weaknesses in a system.

Including a threat model in your project shows that you think about security proactively, not just reactively after a breach.

  • Save the file and verify it exists by running:
ls docs/threat-model.md

You should see docs/threat-model.md in the output.

File not found?

  • Make sure you are running the ls command from your project root directory (the same directory that contains main.tf).
  • Confirm the docs/ directory was created with mkdir -p docs/adr earlier in this step.

Help me find my documentation files

Your pipeline is fully tested and documented. You have proven it works end-to-end and captured the architectural reasoning behind every major decision. Next up, the Secret Mission challenges you to lock down the pipeline even further with VPC endpoint restrictions.

Secret mission

VPC Endpoint Lockdown

Your pipeline works, but uploads currently travel over the public internet to reach S3. What if you could enforce that all PutObject requests must arrive through a private VPC endpoint? In this mission, you will add a Gateway VPC Endpoint for S3 and a bucket policy that denies any upload not routed through it. This is defense-in-depth: even if IAM credentials leak, the network-level restriction blocks unauthorized uploads.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project costs approximately $1.40/month when left running (KMS key at $1.00/month and Secrets Manager at $0.40/month). All other resources fall within the AWS Free Tier.

Cost warning

The KMS Customer Managed Key costs $1.00/month and the Secrets Manager secret costs $0.40/month. These charges accrue whether or not you actively use the pipeline. If you have AWS Free Tier credits, these may be covered. Otherwise, run terraform destroy to stop all charges.

Resources you used:

  • KMS Customer Managed Key ($1.00/month).
  • Secrets Manager secret for Gemini API key ($0.40/month).
  • S3 document bucket (versioned, KMS-encrypted).
  • 2 Lambda functions (presign and processor).
  • 2 SQS queues (processing and dead-letter).
  • VPC Gateway Endpoint for S3 (Secret Mission).
  • S3 remote state bucket (bootstrap).
  • IAM roles and policies.
  • Local project files on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend this pipeline into Project 2 of the series.

  • Your monthly cost is approximately $1.40/month (KMS key + Secrets Manager secret).
  • All other resources (Lambda, SQS, S3 storage) are covered by the AWS Free Tier.
  • Your Terraform state is stored remotely in S3, so you can come back and run terraform plan any time to see the current state.

Pause - I'll come back to this later

Serverless resources do not run continuously, so there is nothing to "pause" in the traditional sense. Your Lambda functions only incur cost when invoked. The fixed monthly charges come from the KMS key and Secrets Manager secret.

  • If you are comfortable with the $1.40/month charge, treat this the same as "Keep everything running" above.
  • If you want to stop all charges but rebuild later, follow the "Delete" tab instead. Your local Terraform files are your blueprint to recreate everything with a single terraform apply.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow these steps in order.

  • Empty the document bucket first. Open the AWS S3 Console, find your bucket (its name starts with securedocai-docs-), select it, click Empty, type permanently delete to confirm, and click Empty. This removes all versioned objects so Terraform can delete the bucket.
  • Navigate to your main project directory in the terminal by running:
cd ~/Desktop/securedocai
  • Destroy all main project infrastructure by running:
terraform destroy
  • Type yes when prompted to confirm.
  • Next, destroy the bootstrap state bucket. First, open bootstrap/main.tf and remove the lifecycle block containing prevent_destroy = true from the aws_s3_bucket.terraform_state resource. Save the file.
  • Empty the state bucket in the AWS S3 Console. Find your bucket (its name starts with securedocai-tfstate-), select it, click Empty, and confirm.
  • Navigate to the bootstrap directory and destroy by running:
cd bootstrap
terraform destroy
  • Type yes when prompted to confirm.
  • Finally, delete the local project files by running:
cd ~/Desktop
rm -rf securedocai
  • On Windows, use this command instead:
cd ~\Desktop
rmdir /s /q securedocai
  • Confirm the folder is gone by running ls (macOS/Linux) or dir (Windows). You should no longer see a securedocai directory.

Nice Work!

Nice Work!

You just built a production-grade serverless document intelligence pipeline from scratch using Terraform. Every resource is encrypted, every role is least-privilege, and the entire architecture is reproducible with a single terraform apply.

You've learned how to:

  • Provision a fully Terraform-managed serverless pipeline with KMS encryption, versioned S3 storage, and remote state using the modern S3 lockfile mechanism.
  • Build an event-driven processing architecture with SQS queues, dead-letter isolation, and Lambda functions following IAM least-privilege.
  • Integrate Gemini AI for document extraction with secrets stored in Secrets Manager, and document your architecture with ADRs, a STRIDE threat model, and a running cost log.
  • Secret Mission: Lock down the S3 bucket to VPC-endpoint-only access with an S3 Gateway VPC Endpoint, proving defense-in-depth.

Ready to quiz yourself?