ECS vs EKS vs Lambda Architecture
Deploy the same workload to ECS, EKS, and Lambda to prove compute tradeoffs.
Introduction
30 Second Summary
When a payment platform goes down for 47 minutes, the fallout is measured in millions of dollars and lost enterprise contracts. Choosing the right compute architecture is the business decision that prevents it from happening again.
In this project, you will deploy the same containerized workload to three AWS compute targets, build a serverless event backbone, chaos-test recovery under load, and produce a board-ready presentation proving your architecture decisions with real data.
What You'll Build
A production-grade compute reference architecture proving ECS, EKS, and Lambda on the same workload, with automated deployment, chaos validation, and a director-ready leadership package.
By the end of this project, you'll have:
- A Terraform-managed container platform deploying the same Docker image to ECS-on-EC2, ECS-on-Fargate, and EKS behind a weighted ALB.
- A serverless event-driven backbone with API Gateway REST, Lambda SnapStart, Step Functions Express, EventBridge, SQS dead-letter queue, and DynamoDB.
- A chaos-validated deployment pipeline with CodeDeploy blue/green canary, ECR scan gate, and measured sub-30-second recovery.
- Secret Mission: Extend the platform with ECS Service Connect and EventBridge Pipes to position a Phase 2 board recommendation.
Are there any prerequisites?
This is a hard-difficulty project designed for learners with hands-on AWS, container, and serverless experience. You will need an AWS account with admin permissions, Docker Desktop installed, Cursor IDE with Composer 2 access, and prior Terraform CLI experience.
Before We Start
Meridian Payments just suffered a $1.2M outage from a bad SSH deploy. The board wants a proven compute architecture before approving $4M in annual cloud spend. Your job as Principal Solutions Architect is to prove ECS vs EKS vs Lambda on the same workload in 90 minutes.
Before you can build anything, you need your tools sharp and your strike team mobilized. In this step, you will install and verify every CLI tool, configure AWS access, scaffold the project directory, and launch five parallel Cursor Composer agents that will execute the build.
In this step, get ready to:
- Install and verify Terraform, AWS CLI, k6, and Docker.
- Configure AWS credentials and create the project directory structure.
- Launch all 5 Cursor Composer agents with their assigned roles.
Verify Terraform 1.15.3
This project relies on four CLI tools working together. Terraform provisions all AWS resources. The AWS CLI validates infrastructure state. Grafana k6 runs load tests. Docker builds and pushes the container image.
- Open your terminal and check your Terraform version by running:
terraform --version
✔️ I see version 1.15 or higher
Terraform is ready. Move on to verifying the AWS CLI below.
ⓧ I see an older version
You need Terraform 1.15.3 for this project. Upgrade using the method matching your platform.
macOS
- Upgrade Terraform by running:
brew tap hashicorp/tap
brew upgrade hashicorp/tap/terraform
- Verify the new version by running:
terraform --version
Windows
- Download the latest Terraform 1.15.3 Windows AMD64 ZIP from the official install page.
- Extract the ZIP and replace the existing terraform.exe in the folder that is on your system PATH.
- Open a new terminal and verify by running:
terraform --version
ⓧ Command not found
Terraform is not installed. Install version 1.15.3 using the method matching your platform.
macOS
- Install Terraform via Homebrew by running:
brew tap hashicorp/tap
brew install hashicorp/tap/terraform
- Verify the installation by running:
terraform --version
Windows
- Download the Terraform 1.15.3 Windows AMD64 ZIP from the official install page.
- Extract the ZIP to a folder (e.g. C:\terraform).
- Add that folder to your system PATH environment variable.
- Open a new terminal and verify by running:
terraform --version
Verify AWS CLI v2
- Check your AWS CLI version by running:
aws --version
✔️ I see version 2.x
AWS CLI v2 is ready. Move on to verifying k6 below.
ⓧ I see version 1.x
You have AWS CLI version 1 installed. This project requires version 2.
- Follow the official upgrade instructions at the AWS CLI install page to replace v1 with v2.
- After installing, open a new terminal and verify by running:
aws --version
ⓧ Command not found
AWS CLI is not installed. Install version 2 using the method matching your platform.
macOS
- Download and install the official AWS CLI v2 package by running:
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /
- Verify the installation by running:
aws --version
Windows
- Download and run the AWS CLI MSI installer for Windows (64-bit) from the AWS CLI install page.
- Follow the on-screen prompts to complete the installation.
- Open a new command prompt and verify by running:
aws --version
Verify Grafana k6 1.7.1
- Check your k6 version by running:
k6 version
✔️ I see version 1.7 or higher
k6 is ready. Move on to verifying Docker below.
ⓧ I see an older version
You need k6 1.7.1 for this project. Upgrade using the method matching your platform.
macOS
- Upgrade k6 by running:
brew upgrade k6
- Verify the new version by running:
k6 version
Windows
- Upgrade k6 using Windows Package Manager by running:
winget upgrade k6
- Alternatively, download the latest MSI installer from the k6 GitHub releases page.
- Verify by running:
k6 version
ⓧ Command not found
k6 is not installed. Install version 1.7.1 using the method matching your platform.
macOS
- Install k6 via Homebrew by running:
brew install k6
- Verify the installation by running:
k6 version
Windows
- Install k6 using Windows Package Manager by running:
winget install k6 --source winget
- Alternatively, download the MSI installer (k6-v1.7.1-windows-amd64.msi) from the k6 GitHub releases page.
- Verify by running:
k6 version
Verify Docker Desktop is running
- Confirm Docker is running by executing:
docker info
✔️ I see Docker engine details
Docker is running and ready to build images.
ⓧ Cannot connect to the Docker daemon
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Docker Desktop and press Enter to open it.
- Wait for Docker Desktop to fully start (the whale icon in your system tray stops animating).
- Run docker info again to confirm it connects.
Still seeing errors from any tool?
Make sure you opened a new terminal window after installing. PATH changes do not apply to terminals that were already open.
If Docker says the daemon is not running on Windows, ensure you have enabled the WSL 2 backend in Docker Desktop settings.
Help me troubleshoot my tool installations for this project.
Configure AWS and create the project structure
With tools verified, you need AWS credentials configured and a project directory ready to receive code from all five agents.
- Configure the AWS CLI with your admin credentials and us-east-1 as the default region by running:
aws configure
What does aws configure ask for?
The CLI will prompt for four values. Enter your IAM admin access key ID, secret access key, us-east-1 as the default region, and json as the default output format.
If you do not have an access key, create one in the AWS IAM console under your admin user's security credentials.
- Move to your Desktop by running this command:
cd ~/Desktop
- Create your project folder, subdirectories, and move into it by running these commands:
mkdir compute-platform
cd compute-platform
mkdir infra lambda tests docs presentation
What are these directories for?
- infra holds all Terraform modules and the root configuration.
- lambda holds the Python Lambda function code and requirements.
- tests holds k6 load scripts and bash validation scripts.
- docs holds ADRs, Mermaid diagrams, and validation results.
- presentation holds the board-ready HTML presentation.
- Confirm the directories exist by running:
ls
You should see docs, infra, lambda, presentation, and tests listed.
- Initialize a Git repository and make your first commit by running:
git init
git add .
git commit -m "project-init: scaffold directories $(date +%Y%m%d-%H%M%S)"
Why a timestamp commit?
The timestamp in your commit message marks the exact start of your 90-minute sprint. When you commit at the end, the Git history proves start-to-finish timing. This is how you demonstrate delivery speed to the board.
Git commit failing?
If Git asks you to set your identity, run git config user.email and git config user.name with your values first.
Help me fix my git commit error.
Launch all 5 agents simultaneously in Cursor Composer
A principal architect decomposes, parallelizes, and synthesizes. You will launch five named agents in Cursor. Each agent owns a distinct domain. Nobody waits. Everybody ships.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Cursor and press Enter to open it.
- In Cursor, click File in the top menu bar.
- Click Open Folder.
- Select the compute-platform folder on your Desktop.
- Click Open.
- Open the Agents Window by pressing Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows) to open the command palette.
- Type Open Agents Window and press Enter.
- Launch 5 parallel agent sessions. For each agent, start a new thread and paste the role prompt below.
What is the Agents Window?
The Agents Window is Cursor's agent-first interface for running multiple agents in parallel across your project. Each agent gets its own isolated context and can work independently on different parts of the codebase.
If the Agents Window is not available, you can also open the sidepanel with Cmd+I (macOS) or Ctrl+I (Windows) and create multiple chat threads there instead.
Use the following role assignments for your five agents:
- Foundation (Platform Engineer): Owns ECR, VPC, ALB, ECS-EC2, ECS-Fargate, EKS. All Terraform in /infra. Starts with ECR + VPC immediately.
- Pipeline (Release Engineer): Owns CodeDeploy blue/green, scan-gate script, deploy trigger. Pre-writes scan-gate.sh while waiting for ECR.
- Backbone (Event Architect): Owns API Gateway REST, Lambda, Step Functions Express, EventBridge, SQS, DynamoDB. Starts Lambda module immediately.
- Breaker (SRE / Chaos Engineer): Owns k6 load scripts, chaos-fargate.sh, dlq-test.sh, validation-results.md. Writes test contracts against expected interfaces now.
- Narrator (Technical Program Manager): Owns ADRs, Mermaid diagrams, FinOps report, implementation plan, HTML presentation. Starts ADR-001 immediately.
Why launch all five at once?
EKS clusters take 10-12 minutes to provision. ECR scans take 90 seconds. By launching all agents in parallel, Foundation starts the long-running EKS provision while other agents write code that will plug in once infrastructure is ready.
This is the principal architect pattern: decompose the problem, parallelize the work, and synthesize at the end. The dependency graph is designed so no agent blocks another.
- Verify all 5 agents have their first file committed by checking that each agent thread is actively writing output.
Cannot run 5 parallel agents?
Cursor's parallel agent capability requires a Pro or Business plan. If your plan limits concurrent agents, launch them sequentially in priority order: Foundation first, then Backbone, then Narrator, then Pipeline, then Breaker.
Help me configure Cursor for parallel agents.
Your environment is locked and loaded. Five agents are running in parallel. Next up, Foundation will lay the Terraform base with ECR and VPC while the rest of the team writes code in anticipation.
Set Up Terraform Foundation and Container Image
The 47-minute outage at Meridian happened because a bad deploy reached production unchecked. The first line of defense is the container registry itself.
In this step, you will create the ECR repository with scan-on-push enabled, build a compute-agnostic Docker image, and write a scan-gate script that blocks any image carrying CRITICAL CVEs from ever reaching production.
In this step, get ready to:
- Write the Terraform foundation (S3 backend, AWS provider, ECR repository with lifecycle policy).
- Build and push a Python Flask container image to ECR.
- Verify the ECR scan and write a scan-gate script that exits 1 on CRITICAL findings.
Write the Terraform foundation and create the ECR repository
Every resource in this project lives in Terraform state. The S3 backend stores that state remotely so it survives across machines and agents.
- Create an S3 bucket for your Terraform state by running this command:
aws s3 mb s3://[[TFSTATE_BUCKET="your-unique-terraform-state-bucket-name"]]-tfstate --region us-east-1
Why a unique bucket name?
S3 bucket names are globally unique across all AWS accounts. Include your name or a random suffix to avoid collisions.
- Verify the bucket was created by running:
aws s3 ls | grep tfstate
You should see your bucket name in the output.
- Have Foundation agent write the Terraform configuration in infra/main.tf with the following content:
terraform {
required_version = "~> 1.15"
backend "s3" {}
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "us-east-1"
}
# ECR repository with vulnerability scanning enabled on every push
resource "aws_ecr_repository" "compute_platform" {
name = "compute-platform"
image_scanning_configuration {
scan_on_push = true
}
tags = {
Project = "compute-platform"
Company = "Meridian"
CostCenter = "engineering"
Environment = "lab"
}
}
# Lifecycle policy: retain only the last 5 images to control storage costs
resource "aws_ecr_lifecycle_policy" "retain_five" {
repository = aws_ecr_repository.compute_platform.name
policy = jsonencode({
rules = [{
rulePriority = 1
description = "Retain last 5 images"
selection = {
tagStatus = "any"
countType = "imageCountMoreThan"
countNumber = 5
}
action = {
type = "expire"
}
}]
})
}
What does this code do?
The terraform block pins the CLI to ~> 1.15 and declares an empty S3 backend (partial configuration). The AWS provider is locked to ~> 5.0.
The aws_ecr_repository resource creates the container registry with scan_on_push = true. Every image pushed is automatically scanned for CVEs. The lifecycle policy keeps storage lean by expiring anything beyond the 5 most recent images.
Tags ensure every resource is traceable back to this project for FinOps and teardown.
- Initialize and apply the Terraform configuration by running these commands from the infra directory:
cd infra
terraform init \
-backend-config="bucket=[[TFSTATE_BUCKET="your-unique-terraform-state-bucket-name"]]-tfstate" \
-backend-config="key=compute-platform/terraform.tfstate" \
-backend-config="region=us-east-1"
terraform apply -auto-approve
What does terraform init do here?
The -backend-config flags supply the required S3 bucket name, key path, and region at init time. This is called partial configuration. It keeps sensitive or environment-specific values out of committed code.
The terraform apply -auto-approve then provisions the ECR repository and lifecycle policy in your AWS account.
- Save infra/main.tf.
- After the apply completes, verify the ECR repository exists by running:
aws ecr describe-repositories --repository-names compute-platform --query 'repositories[0].repositoryUri' --output text
You should see a URI like 123456789012.dkr.ecr.us-east-1.amazonaws.com/compute-platform printed in your terminal.
Seeing an access denied error?
Make sure your AWS CLI is configured with admin credentials and the region is set to us-east-1. Run the following to confirm you are authenticated:
aws sts get-caller-identity
If the S3 bucket creation failed, your bucket name may already be taken. Try a different unique name.
help me debug my Terraform init or apply error
Build and push the container image
This Docker image is the workload that will run on all three compute targets (ECS-EC2, ECS-Fargate, EKS). The deployment target is the variable. The code stays the same.
- Create handler.py in your project root with the following Flask application:
import os, socket
from flask import Flask, jsonify
# Minimal Flask app that identifies its compute target and container hostname
app = Flask(__name__)
@app.route('/')
def index():
# Returns JSON with target (set by env var), hostname (container ID), and version
return jsonify({
"target": os.environ.get("TARGET", "generic"),
"hostname": socket.gethostname(),
"version": "v1"
})
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
What does this app do?
The app returns a JSON payload with three fields: target (set by environment variable to identify ECS-EC2, Fargate, or EKS), hostname (the container ID), and version (for blue/green tracking).
This is how the ALB weighted routing proves traffic distribution later. Each response tells you exactly which compute target served it.
- Create requirements.txt in the project root:
flask
- Create the Dockerfile in the project root:
FROM python:3.13-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY handler.py .
# TARGET env var is overridden per compute target at deploy time
ENV TARGET=generic
CMD ["python", "handler.py"]
Why python:3.13-slim?
The slim variant keeps the image small (fewer layers to scan, faster pulls). Python 3.13 is the same runtime you will use for Lambda SnapStart later, keeping the stack consistent across container and serverless targets.
- Find your AWS account ID by running:
aws sts get-caller-identity --query Account --output text
Copy the 12-digit number from the output. You will use it in the next commands.
- Build the Docker image, authenticate to ECR, tag, and push by running:
# Build the image locally
docker build -t compute-platform:v1 .
# Authenticate Docker to your ECR registry
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [[AWS_ACCOUNT_ID="your-12-digit-aws-account-id"]].dkr.ecr.us-east-1.amazonaws.com
# Tag and push to ECR
docker tag compute-platform:v1 [[AWS_ACCOUNT_ID="your-12-digit-aws-account-id"]].dkr.ecr.us-east-1.amazonaws.com/compute-platform:v1
docker push [[AWS_ACCOUNT_ID="your-12-digit-aws-account-id"]].dkr.ecr.us-east-1.amazonaws.com/compute-platform:v1
- Verify the image was pushed successfully by running:
aws ecr describe-images --repository-name compute-platform --query 'imageDetails[0].imageTags' --output text
You should see v1 in the output.
Docker push failing?
Make sure Docker Desktop is running. Run docker info to confirm the Docker daemon is active.
If you see a "denied" or "authorization" error, the ECR login token may not have issued correctly. Re-run the aws ecr get-login-password pipeline command. The token is valid for 12 hours.
help me fix my Docker push to ECR
Verify the scan and write the scan-gate script
ECR scan-on-push runs automatically. The scan-gate script turns a scan result into a deploy-or-block decision.
At Meridian, this script is the difference between maintaining PCI-DSS L1 certification and losing it.
- Wait about 90 seconds for the scan to complete, then check the results by running:
aws ecr describe-image-scan-findings --repository-name compute-platform --image-id imageTag=v1 --query 'imageScanFindings.findingSeverityCounts' --output json
You should see a JSON object showing finding counts by severity. For python:3.13-slim, expect zero CRITICAL findings.
- Have Pipeline agent write the scan-gate script at tests/scan-gate.sh:
#!/bin/bash
# Scan gate: blocks deployment if CRITICAL CVEs are found in the image
REPO=$1
TAG=$2
FINDINGS=$(aws ecr describe-image-scan-findings --repository-name $REPO --image-id imageTag=$TAG --query 'imageScanFindings.findingSeverityCounts.CRITICAL' --output text)
if [ "$FINDINGS" != "None" ] && [ "$FINDINGS" -gt 0 ]; then
echo "CRITICAL CVEs found: $FINDINGS. Blocking deployment."
exit 1
fi
echo "Scan gate passed. No CRITICAL findings."
exit 0
What does this script do?
It queries the ECR scan findings for CRITICAL-severity CVEs. If any are found, it exits with code 1 (failure), blocking the deployment pipeline. If none are found, it exits 0 (success) and the deploy proceeds.
In a CI pipeline, this script sits between the image push and the CodeDeploy trigger. A non-zero exit stops the pipeline cold.
- Make the script executable and run it against your clean image:
chmod +x tests/scan-gate.sh
./tests/scan-gate.sh compute-platform v1
You should see: Scan gate passed. No CRITICAL findings..
- Commit all changes by running:
cd ..
git add .
git commit -m "foundation: ECR repo, container image v1, scan-gate script"
Scan gate returning an error about pending scan?
ECR scans take approximately 90 seconds to complete. If you see an error about the scan status being IN_PROGRESS, wait a minute and try again.
If the script returns None instead of a number, the scan completed cleanly with no CRITICAL findings. That is the expected success case.
help me troubleshoot my ECR scan-gate script
✔️ Awesome, I've got everything!
Great. Double check you have saved infra/main.tf, handler.py, Dockerfile, requirements.txt, and tests/scan-gate.sh.
ⓧ I'd like to double check the full code
Here are the complete files for this step:
terraform {
required_version = "~> 1.15"
backend "s3" {}
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "us-east-1"
}
resource "aws_ecr_repository" "compute_platform" {
name = "compute-platform"
image_scanning_configuration {
scan_on_push = true
}
tags = {
Project = "compute-platform"
Company = "Meridian"
CostCenter = "engineering"
Environment = "lab"
}
}
resource "aws_ecr_lifecycle_policy" "retain_five" {
repository = aws_ecr_repository.compute_platform.name
policy = jsonencode({
rules = [{
rulePriority = 1
description = "Retain last 5 images"
selection = {
tagStatus = "any"
countType = "imageCountMoreThan"
countNumber = 5
}
action = {
type = "expire"
}
}]
})
}
import os, socket
from flask import Flask, jsonify
app = Flask(__name__)
@app.route('/')
def index():
return jsonify({
"target": os.environ.get("TARGET", "generic"),
"hostname": socket.gethostname(),
"version": "v1"
})
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
FROM python:3.13-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY handler.py .
ENV TARGET=generic
CMD ["python", "handler.py"]
#!/bin/bash
REPO=$1
TAG=$2
FINDINGS=$(aws ecr describe-image-scan-findings --repository-name $REPO --image-id imageTag=$TAG --query 'imageScanFindings.findingSeverityCounts.CRITICAL' --output text)
if [ "$FINDINGS" != "None" ] && [ "$FINDINGS" -gt 0 ]; then
echo "CRITICAL CVEs found: $FINDINGS. Blocking deployment."
exit 1
fi
echo "Scan gate passed. No CRITICAL findings."
exit 0
Your foundation is laid. ECR is scanning, the image is clean, and the scan gate is armed. Next, you will build the networking layer and deploy this same image to three different compute targets behind a weighted ALB.
Build the Multi-Target Container Platform
You have a compute-agnostic Docker image sitting in ECR. Now you need to prove it runs on three different targets: ECS on EC2, ECS on Fargate, and EKS.
In this step, Foundation builds the networking and deploys all three container targets behind a single weighted ALB. The board will see traffic split 34/33/33 across the three targets. Same image, same workload, three architectures compared side by side.
In this step, get ready to:
- Build the VPC and ALB with weighted routing across 3 target groups.
- Deploy ECS-EC2 and ECS-Fargate services with per-task awsvpc isolation.
- Provision EKS and validate all 3 targets healthy.
Build the VPC and ALB networking foundation
Every container target needs a network to live in and a load balancer to route traffic to it. The VPC provides isolated networking across 2 Availability Zones. The ALB distributes incoming requests across all three targets using weighted routing.
Why awsvpc over bridge mode?
The awsvpc network mode gives each task its own ENI with a per-task security group. Bridge mode requires broad ephemeral port ranges because services are assigned random ports. You cannot create specific rules for one service to communicate with only one other specific service.
Fargate requires awsvpc. For ECS-on-EC2, awsvpc is the recommended mode. The SAP-C02 exam will present bridge as the "simpler" option. It is the wrong answer for per-task security isolation.
- Instruct your Foundation agent to create the VPC module at infra/modules/vpc/main.tf with this prompt:
Create infra/modules/vpc/main.tf:
- VPC with 2 AZs (us-east-1a, us-east-1b)
- Public subnets (one per AZ) with IGW route
- Private subnets (one per AZ) with NAT Gateway route
- Single NAT Gateway in first public subnet
- Internet Gateway attached to VPC
- Outputs: vpc_id, public_subnets (list), private_subnets (list)
- Tag all resources: Project=compute-platform, Company=Meridian
- Next, instruct Foundation to create the ALB module at infra/modules/alb/main.tf with this prompt:
Create infra/modules/alb/main.tf:
- Internet-facing ALB in public subnets
- ALB security group: allow port 80 from 0.0.0.0/0
- 3 target groups, all with target_type = "ip":
- TG-1: ecs-ec2-tg (port 5000)
- TG-2: ecs-fargate-tg (port 5000)
- TG-3: eks-tg (port 5000)
- Listener on port 80 with weighted forward action: 34/33/33 across TG-1/TG-2/TG-3
- Variables: vpc_id, public_subnets
- Outputs: alb_dns, alb_sg_id, tg1_arn, tg2_arn, tg3_arn
- Tag all resources: Project=compute-platform, Company=Meridian
What does this networking setup do?
- The VPC creates isolated networking with public subnets (for the ALB) and private subnets (for container tasks that should not be directly internet-accessible).
- The NAT Gateway lets private-subnet tasks pull images from ECR without exposing them to inbound internet traffic.
- The weighted listener at 34/33/33 distributes traffic across all three targets. This gives the board real data comparing each architecture under the same load.
- All target groups use target_type = "ip" because awsvpc network mode assigns each task its own IP. The instance type will not work with awsvpc.
- Verify that Foundation created both module directories. Run:
ls infra/modules/vpc/ infra/modules/alb/
You should see main.tf in both directories.
Module directory missing?
If the agent did not create the directory structure, create the directories manually with mkdir -p infra/modules/vpc infra/modules/alb and re-prompt the agent to write the module files.
Ask for help: My Foundation agent didn't create the VPC and ALB modules in the right directory structure. How do I fix this?
Deploy ECS-EC2 and ECS-Fargate with per-task isolation
Meridian's Java payment engine is being Replatformed. Same containerized workload, deployed to managed compute with no code changes. ECS-EC2 gives you control over the underlying instances. ECS-Fargate removes instance management entirely.
- Instruct Foundation to create the ECS-EC2 module at infra/modules/ecs-ec2/main.tf with this prompt:
Create infra/modules/ecs-ec2/main.tf:
- ECS cluster
- EC2 launch template: t3.micro, Amazon Linux 2023 ECS-optimized AMI
- Auto Scaling Group: min 1, max 1, desired 1
- ECS capacity provider linked to ASG
- Task definition: awsvpc network mode, container port 5000, ENV TARGET=ecs-ec2
- Image from variable: ecr_image
- Task security group: allow inbound port 5000 ONLY from ALB security group (var: alb_sg_id)
- ECS service: desired 1, registered to target group (var: target_group_arn)
- Variables: vpc_id, private_subnets, alb_sg_id, target_group_arn, ecr_image
- Tag all resources: Project=compute-platform
- Instruct Foundation to create the ECS-Fargate module at infra/modules/ecs-fargate/main.tf with this prompt:
Create infra/modules/ecs-fargate/main.tf:
- Fargate launch type (no EC2 instances to manage)
- Task definition: awsvpc network mode, 0.25 vCPU, 0.5 GB memory
- Container port 5000, ENV TARGET=ecs-fargate
- Image from variable: ecr_image
- Task security group: allow inbound port 5000 ONLY from ALB security group (var: alb_sg_id)
- ECS service: desired 1, launch type FARGATE, registered to target group (var: target_group_arn)
- Assign public IP: false (tasks in private subnets, NAT handles outbound)
- Variables: vpc_id, private_subnets, alb_sg_id, target_group_arn, ecr_image
- Tag all resources: Project=compute-platform
What's the difference between these two modules?
ECS-EC2 gives you a t3.micro instance that you control. You can SSH into it, tune OS settings, and run multiple tasks on one host. The tradeoff is you manage patching and capacity.
ECS-Fargate eliminates the instance entirely. AWS provisions exactly 0.25 vCPU and 0.5 GB per task. You pay per-second for what you use. The tradeoff is no host-level access.
Both use awsvpc network mode. Both get per-task security groups accepting traffic only from the ALB. The security posture is identical. That is the point you prove to the board.
- Verify that Foundation created both ECS module directories by running:
ls infra/modules/ecs-ec2/ infra/modules/ecs-fargate/
You should see main.tf in both directories.
Agent creating separate clusters for each module?
If the ECS-Fargate module creates its own cluster, re-prompt to share the cluster from ECS-EC2 or pass the cluster ARN as a variable. Both services should run on the same ECS cluster unless you explicitly need separate clusters.
Ask for help: Should my ECS-EC2 and ECS-Fargate modules share a cluster or have separate ones?
Provision EKS and validate all three targets
The third target is EKS. This proves to the board that Kubernetes is an option, while also demonstrating why ECS might be the better fit for Meridian's workload. EKS adds a $0.10/hr control plane cost that ECS does not have.
- Instruct Foundation to create the EKS module at infra/modules/eks/main.tf with this prompt:
Create infra/modules/eks/main.tf:
- EKS cluster version 1.32
- Managed node group: t3.small, min 1, max 1, desired 1
- IAM roles: cluster role + node role with managed policies
- Kubernetes Deployment: 1 replica of the ECR image, container port 5000, ENV TARGET=eks
- TargetGroupBinding CRD to register pods with ALB target group (var: target_group_arn)
- Install AWS Load Balancer Controller via Helm or use the elbv2.k8s.aws/v1beta1 CRD
- Variables: vpc_id, private_subnets, target_group_arn
- Tag all resources: Project=compute-platform
Why would you choose EKS over ECS?
Only if you need Kubernetes-native tooling: Helm charts, Custom Resource Definitions (CRDs), service mesh, or multi-cloud portability. Otherwise EKS adds $876/yr in control plane costs plus operational complexity for zero benefit over ECS.
The SAP-C02 exam tests this exact decision. Meridian does not need CRDs or a service mesh. ECS wins. But proving all three targets with real data is what gives the board confidence in the recommendation.
- Now update infra/main.tf to wire all modules together. Instruct Foundation to add the module blocks below the existing ECR resources:
# VPC
module "vpc" { source = "./modules/vpc" }
# ALB
module "alb" { source = "./modules/alb" vpc_id = module.vpc.vpc_id public_subnets = module.vpc.public_subnets }
# ECS EC2
module "ecs_ec2" { source = "./modules/ecs-ec2" vpc_id = module.vpc.vpc_id private_subnets = module.vpc.private_subnets alb_sg_id = module.alb.alb_sg_id target_group_arn = module.alb.tg1_arn ecr_image = "${aws_ecr_repository.compute_platform.repository_url}:v1" }
# ECS Fargate
module "ecs_fargate" { source = "./modules/ecs-fargate" vpc_id = module.vpc.vpc_id private_subnets = module.vpc.private_subnets alb_sg_id = module.alb.alb_sg_id target_group_arn = module.alb.tg2_arn ecr_image = "${aws_ecr_repository.compute_platform.repository_url}:v1" }
# EKS
module "eks" { source = "./modules/eks" vpc_id = module.vpc.vpc_id private_subnets = module.vpc.private_subnets target_group_arn = module.alb.tg3_arn }
What does this wiring do?
- Each module receives only the outputs it needs from the modules above it. VPC outputs feed into ALB and all compute modules.
- The alb_sg_id variable lets each ECS module create a task security group that accepts traffic ONLY from the ALB. No direct internet access to tasks.
- The ECR image reference uses the repository URL output from the existing ECR resource. All three targets pull the same v1 image.
- Deploy the entire platform by running these commands from your infra directory:
cd infra
terraform init
terraform apply -auto-approve
How long does this take?
ECS services provision in about 2 minutes. The EKS cluster takes 10-12 minutes. Terraform will not return until everything is healthy. This is expected.
While Terraform runs, your other agents (Narrator, Backbone, Pipeline, Breaker) should continue their work. Foundation started EKS at minute 0 so it finishes before Breaker needs it at minute 50. Nobody blocks.
- Once Terraform completes, validate that all 3 target groups show healthy targets. Get your target group ARNs from Terraform output, then run aws elbv2 describe-target-health against each one:
aws elbv2 describe-target-health --target-group-arn $(terraform output -raw tg1_arn) --query 'TargetHealthDescriptions[].TargetHealth.State' --output text
aws elbv2 describe-target-health --target-group-arn $(terraform output -raw tg2_arn) --query 'TargetHealthDescriptions[].TargetHealth.State' --output text
aws elbv2 describe-target-health --target-group-arn $(terraform output -raw tg3_arn) --query 'TargetHealthDescriptions[].TargetHealth.State' --output text
All three commands should return healthy. This means each target group has at least one registered target passing health checks.
Seeing 'unhealthy' or 'initial' state?
- If a target shows initial, wait 30 seconds and retry. ECS tasks need time to pass the ALB health check interval.
- If a target shows unhealthy, check that the task security group allows inbound on port 5000 from the ALB security group. A missing ingress rule is the most common cause.
- If EKS targets are not registering, confirm the AWS Load Balancer Controller is installed and the TargetGroupBinding CRD is applied to the cluster.
Ask for help: My target groups are not showing healthy. How do I debug the health check path for my ECS and EKS targets?
✔️ Awesome, I've got everything!
Great. Double check you have saved all your files and that Terraform state is clean.
ⓧ I'd like to double check the full code
Here is what your infra/main.tf should look like at this point:
terraform { required_version = "~> 1.15" backend "s3" {} required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } }
provider "aws" { region = "us-east-1" }
# ECR (from Step 1)
resource "aws_ecr_repository" "compute_platform" {
name = "compute-platform"
image_scanning_configuration { scan_on_push = true }
tags = { Project = "compute-platform", Company = "Meridian", CostCenter = "engineering", Environment = "lab" }
}
resource "aws_ecr_lifecycle_policy" "retain_five" {
repository = aws_ecr_repository.compute_platform.name
policy = jsonencode({ rules = [{ rulePriority = 1, description = "Retain last 5 images", selection = { tagStatus = "any", countType = "imageCountMoreThan", countNumber = 5 }, action = { type = "expire" } }] })
}
# VPC
module "vpc" { source = "./modules/vpc" }
# ALB
module "alb" { source = "./modules/alb" vpc_id = module.vpc.vpc_id public_subnets = module.vpc.public_subnets }
# ECS EC2
module "ecs_ec2" { source = "./modules/ecs-ec2" vpc_id = module.vpc.vpc_id private_subnets = module.vpc.private_subnets alb_sg_id = module.alb.alb_sg_id target_group_arn = module.alb.tg1_arn ecr_image = "${aws_ecr_repository.compute_platform.repository_url}:v1" }
# ECS Fargate
module "ecs_fargate" { source = "./modules/ecs-fargate" vpc_id = module.vpc.vpc_id private_subnets = module.vpc.private_subnets alb_sg_id = module.alb.alb_sg_id target_group_arn = module.alb.tg2_arn ecr_image = "${aws_ecr_repository.compute_platform.repository_url}:v1" }
# EKS
module "eks" { source = "./modules/eks" vpc_id = module.vpc.vpc_id private_subnets = module.vpc.private_subnets target_group_arn = module.alb.tg3_arn }
All three compute targets are live behind the weighted ALB. Traffic is flowing 34/33/33. Next up, you will build the serverless event-driven backbone that handles Meridian's webhook processing at 10K/sec.
Deploy the Serverless Event-Driven Backbone
Three container targets are live and healthy behind a weighted ALB. That proves the Replatform migration path for Meridian's Java payment engine. Now it is time for the second workload: the webhook dispatcher.
Meridian's webhook handler is a 2000-line Express.js monolith that peaks at 10K requests/sec on payroll day. The migration strategy here is Refactor. You will decompose it into independently scaling event-driven components where each piece handles one responsibility.
In this step, get ready to:
- Write the Lambda webhook processor and define the API Gateway + WAF + Lambda configuration.
- Configure EventBridge, Step Functions Express, DynamoDB, and the SQS dead-letter queue.
- Apply the serverless module and verify all resources are live.
Write the Lambda handler and API Gateway configuration
The Backbone agent owns this entire serverless module. You will create the Lambda function code first, then direct the agent to write the Terraform that wires it all together.
- Create the lambda directory inside your project root by running this command:
mkdir lambda
- Create lambda/handler.py with the following code:
import json, boto3, os
sfn = boto3.client('stepfunctions')
events = boto3.client('events')
def lambda_handler(event, context):
body = json.loads(event.get('body', '{}'))
# Put event to EventBridge
events.put_events(Entries=[{
'Source': 'meridian.webhook',
'DetailType': 'WebhookReceived',
'Detail': json.dumps(body),
'EventBusName': os.environ['EVENT_BUS_NAME']
}])
return {'statusCode': 200, 'body': json.dumps({'status': 'accepted'})}
What does this code do?
This is the Refactored webhook processor. Instead of handling all logic in one function, it does exactly one thing: route the incoming payload to EventBridge.
- events.put_events() publishes the webhook body to a custom event bus. EventBridge then routes it downstream.
- The EVENT_BUS_NAME environment variable is injected by Terraform so the function knows which bus to target.
- The function returns immediately with a 200. Processing happens asynchronously via Step Functions downstream.
- Save lambda/handler.py.
- Confirm the file exists by running:
ls lambda/
You should see handler.py in the output.
Now direct the Backbone agent to create infra/modules/serverless/main.tf. The module must include the following resources:
- aws_api_gateway_rest_api with a usage plan capped at 1000 requests/day and an API key for enterprise client authentication.
- aws_wafv2_web_acl with a rate-based rule limiting to 100 requests per 5 minutes per IP, associated to the API Gateway stage.
- aws_lambda_function named webhook-processor using python3.13 runtime, SnapStart enabled (snap_start { apply_on = "PublishedVersions" }), 256 MB memory, 30-second timeout, handler set to handler.lambda_handler.
- aws_lambda_alias named live with routing_config for canary traffic shifting.
Why REST API instead of HTTP API?
HTTP API is cheaper, and the SAP-C02 exam will tempt you with "lower cost" as the answer. But HTTP API does not support usage plans or API keys.
Meridian's enterprise clients require API keys for webhook authentication. REST API is the only option that supports usage plans, API keys, and request validation together.
What is SnapStart?
Lambda SnapStart takes a snapshot of the initialized execution environment and caches it. Subsequent invocations resume from the snapshot instead of cold-starting from scratch.
SnapStart supports Java 11+, Python 3.12+, and .NET 8+. Node.js is not supported. The project uses Python 3.13 to demonstrate sub-second cold starts for Meridian's payment callbacks.
Configure EventBridge, Step Functions, DynamoDB, and the DLQ
The Lambda function routes events to EventBridge. Now the Backbone agent adds the downstream pipeline that orchestrates, persists, and catches failures.
- Direct the Backbone agent to add these resources to infra/modules/serverless/main.tf:
- aws_cloudwatch_event_bus named meridian-events with a rule routing matched events to the Step Functions state machine.
- aws_sfn_state_machine named webhook-orchestrator using EXPRESS type. ASL definition must include retry with MaxAttempts: 3 and BackoffRate: 2.0. A catch block routes States.ALL errors to the SQS DLQ.
- aws_dynamodb_table named webhook-events with PAY_PER_REQUEST billing mode.
- aws_sqs_queue named webhook-dlq with messageRetentionSeconds set to 1209600 (14 days) and a redrive policy with maxReceiveCount: 3.
Why Express instead of Standard Step Functions?
Standard workflows are exactly-once but capped at 2,000 starts/sec. Meridian's webhook volume peaks 10K/sec on payroll day. Express handles 100,000 executions/sec with nearly unlimited state transitions.
The SAP-C02 exam will tempt you with "exactly-once guarantees." But Express combined with an idempotent DynamoDB PUT achieves the same result at 50x the throughput. Express workflows have a maximum duration of 5 minutes and are priced by executions + duration + memory.
Why SQS for the dead-letter queue?
SNS does not retain failed messages unless you configure a DLQ on the subscription. SQS retains by default for up to 14 days (configurable). For payment webhooks where every message must be accounted for, SQS DLQ is mandatory.
Agent writing the wrong Step Functions type?
If the Backbone agent defaults to STANDARD type, explicitly specify type = "EXPRESS" in the prompt. Standard workflows will bottleneck at 2,000/sec and fail under Meridian's payroll-day load.
help me fix the Step Functions type in my serverless module
Apply the serverless module and verify all resources
The module is written. Now wire it into the root configuration and deploy everything.
- In infra/main.tf, add the serverless module reference below the existing EKS module:
# Serverless Backbone
module "serverless" { source = "./modules/serverless" vpc_id = module.vpc.vpc_id }
What does this module reference do?
This single block connects the entire serverless backbone to your root Terraform configuration. The vpc_id input lets the module place resources inside the same VPC as your container infrastructure.
When you run terraform apply, Terraform resolves the module source, plans all nested resources, and deploys them in dependency order.
- Save infra/main.tf.
- Deploy the serverless backbone by running these commands:
cd infra
terraform init
terraform apply -auto-approve
Why terraform init again?
You added a new module (./modules/serverless). Terraform needs to initialize new module sources before it can plan them. Running terraform init ensures the new module is registered in the working directory.
- Verify the API Gateway REST API exists by running:
aws apigateway get-rest-apis
You should see your REST API listed in the JSON output with its endpoint URL.
- Verify the Lambda function exists by running:
aws lambda get-function --function-name webhook-processor
You should see the function configuration including python3.13 runtime, 256 MB memory, and SnapStart configuration.
- Verify the Step Functions state machine exists by running:
aws stepfunctions list-state-machines
You should see webhook-orchestrator listed with type EXPRESS.
Lambda deploy package too large?
Check that only handler.py is being zipped in the Lambda deployment package. The boto3 library is pre-installed in the Lambda runtime. You do not need to bundle it.
If you see a WAF WebACL limit error, confirm you have not exceeded the free tier allowance of 1 WebACL + 10 rules per account.
help me debug my serverless module deployment
✔️ Awesome, I've got everything!
Great. Double check you have saved both lambda/handler.py and infra/main.tf. All serverless resources should be live.
ⓧ I'd like to double check the full code
Here is the Lambda handler:
import json, boto3, os
sfn = boto3.client('stepfunctions')
events = boto3.client('events')
def lambda_handler(event, context):
body = json.loads(event.get('body', '{}'))
# Put event to EventBridge
events.put_events(Entries=[{
'Source': 'meridian.webhook',
'DetailType': 'WebhookReceived',
'Detail': json.dumps(body),
'EventBusName': os.environ['EVENT_BUS_NAME']
}])
return {'statusCode': 200, 'body': json.dumps({'status': 'accepted'})}
Here is the updated root infra/main.tf (showing the new addition at the bottom):
terraform { required_version = "~> 1.15" backend "s3" {} required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } }
provider "aws" { region = "us-east-1" }
# ECR (from Step 1)
resource "aws_ecr_repository" "compute_platform" {
name = "compute-platform"
image_scanning_configuration { scan_on_push = true }
tags = { Project = "compute-platform", Company = "Meridian", CostCenter = "engineering", Environment = "lab" }
}
resource "aws_ecr_lifecycle_policy" "retain_five" {
repository = aws_ecr_repository.compute_platform.name
policy = jsonencode({ rules = [{ rulePriority = 1, description = "Retain last 5 images", selection = { tagStatus = "any", countType = "imageCountMoreThan", countNumber = 5 }, action = { type = "expire" } }] })
}
# VPC
module "vpc" { source = "./modules/vpc" }
# ALB
module "alb" { source = "./modules/alb" vpc_id = module.vpc.vpc_id public_subnets = module.vpc.public_subnets }
# ECS EC2
module "ecs_ec2" { source = "./modules/ecs-ec2" vpc_id = module.vpc.vpc_id private_subnets = module.vpc.private_subnets alb_sg_id = module.alb.alb_sg_id target_group_arn = module.alb.tg1_arn ecr_image = "${aws_ecr_repository.compute_platform.repository_url}:v1" }
# ECS Fargate
module "ecs_fargate" { source = "./modules/ecs-fargate" vpc_id = module.vpc.vpc_id private_subnets = module.vpc.private_subnets alb_sg_id = module.alb.alb_sg_id target_group_arn = module.alb.tg2_arn ecr_image = "${aws_ecr_repository.compute_platform.repository_url}:v1" }
# EKS
module "eks" { source = "./modules/eks" vpc_id = module.vpc.vpc_id private_subnets = module.vpc.private_subnets target_group_arn = module.alb.tg3_arn }
# Serverless Backbone
module "serverless" { source = "./modules/serverless" vpc_id = module.vpc.vpc_id }
The serverless module (infra/modules/serverless/main.tf) should contain: API Gateway REST API with usage plan (1000/day) and API key, WAF WebACL with rate rule (100 req/5min/IP), Lambda webhook-processor (Python 3.13, SnapStart, 256 MB, 30s), alias live with canary routing, EventBridge bus meridian-events, Step Functions Express webhook-orchestrator with retry and catch-to-DLQ, DynamoDB webhook-events (PAY_PER_REQUEST), and SQS webhook-dlq (14-day retention, maxReceiveCount 3).
The Refactored architecture is live. API Gateway authenticates and throttles. Lambda transforms and routes. EventBridge fans out. Step Functions orchestrates with retry. DynamoDB persists. SQS catches failures. Next up, you will build the blue/green deployment pipeline that safely ships new versions without repeating the $1.2M outage.
Configure the Blue/Green Deployment Pipeline
Meridian's 47-minute outage happened because a manual SSH deploy pushed broken code directly to production with no rollback path. Infrastructure that makes bad deploys impossible is the solution.
In this step, you will create a CodeDeploy blue/green pipeline that shifts traffic gradually (10% canary for 5 minutes) and auto-rolls back on failure. This is the deployment pattern that would have prevented the $1.2M outage.
In this step, get ready to:
- Create the CodeDeploy Terraform module with canary configuration and auto-rollback.
- Write the blue/green deployment CLI script that registers a new task definition and triggers the deployment.
- Apply the infrastructure and verify the deployment group is configured with both target groups.
Create the CodeDeploy Terraform module
CodeDeploy for ECS blue/green requires four things: an ALB or NLB, two target groups, a production listener, and an ECS service with its deployment controller set to CODE_DEPLOY. Your ECS-EC2 service already has all of these from the prior step.
- Create the module directory by running this command:
mkdir -p infra/modules/codedeploy
- Confirm the directory exists by running:
ls infra/modules/codedeploy
You should see the empty directory listed without errors.
- Create infra/modules/codedeploy/main.tf and paste the following Terraform configuration. Tell the Pipeline agent to write this module:
# CodeDeploy application scoped to ECS compute platform
resource "aws_codedeploy_app" "compute_platform" {
name = "compute-platform"
compute_platform = "ECS"
}
# Deployment group: canary 10% for 5 min, auto-rollback on failure
resource "aws_codedeploy_deployment_group" "ecs_bluegreen" {
app_name = aws_codedeploy_app.compute_platform.name
deployment_group_name = "ecs-bluegreen"
service_role_arn = aws_iam_role.codedeploy.arn
deployment_config_name = "CodeDeployDefault.ECSCanary10Percent5Minutes"
# Auto-rollback if the deployment fails health checks
auto_rollback_configuration {
enabled = true
events = ["DEPLOYMENT_FAILURE"]
}
# Blue/green traffic shifting behavior
blue_green_deployment_config {
deployment_ready_option {
action_on_timeout = "CONTINUE_DEPLOYMENT"
}
terminate_blue_instances_on_deployment_success {
action = "TERMINATE"
termination_wait_time_in_minutes = 5
}
}
# ECS service that CodeDeploy will manage
ecs_service {
cluster_name = module.ecs_ec2.cluster_name
service_name = module.ecs_ec2.service_name
}
# Load balancer config: two TGs for blue/green swap
load_balancer_info {
target_group_pair_info {
prod_traffic_route {
listener_arns = [module.alb.listener_arn]
}
target_group {
name = module.alb.tg1_name
}
target_group {
name = module.alb.tg2_name
}
}
}
}
What does this code do?
This module creates the full CodeDeploy blue/green pipeline for your ECS service.
- The deployment config CodeDeployDefault.ECSCanary10Percent5Minutes shifts 10% of traffic to the green task set first. If it stays healthy for 5 minutes, the remaining 90% shifts over.
- Auto-rollback on DEPLOYMENT_FAILURE means CodeDeploy automatically routes traffic back to the blue (original) task set if health checks fail.
- The target_group_pair_info block tells CodeDeploy which two target groups to swap between (TG-1 blue, TG-2 green) and which listener routes production traffic.
- The terminate_blue_instances_on_deployment_success block cleans up the old blue task set 5 minutes after a successful shift.
Why canary instead of all-at-once?
All-at-once shifts 100% of traffic instantly to the new task set. If it fails, every request fails until rollback completes.
Canary 10% for 5 minutes means only 10% of Meridian's payment traffic hits the new code initially. If the green task set returns 5xx errors, CodeDeploy rolls back before 90% of clients ever saw the problem.
- Wire the module into your root infra/main.tf by adding this module block at the bottom of the file:
module "codedeploy" {
source = "./modules/codedeploy"
ecs_cluster_name = module.ecs_ec2.cluster_name
ecs_service_name = module.ecs_ec2.service_name
listener_arn = module.alb.listener_arn
tg1_name = module.alb.tg1_name
tg2_name = module.alb.tg2_name
}
What does this module block do?
This passes the outputs from your existing ALB and ECS modules into the CodeDeploy module. The ECS cluster name and service name tell CodeDeploy which service to manage. The listener ARN and target group names configure the traffic-shifting path.
- Save both infra/modules/codedeploy/main.tf and infra/main.tf.
Seeing a variable not found error?
Confirm that your ECS-EC2 module outputs cluster_name and service_name. Your ALB module must output listener_arn, tg1_name, and tg2_name.
Also verify that the module source path matches the directory you just created. The Pipeline agent should generate matching outputs automatically.
Help me debug my CodeDeploy Terraform module variable errors
Write the blue/green deployment CLI script
The Terraform module creates the deployment infrastructure. Now you need a script that actually triggers a deployment when you push a new image.
This script registers a new task definition with the updated image, builds the AppSpec JSON that CodeDeploy requires, and calls aws deploy create-deployment.
- Create tests/deploy-bluegreen.sh with the following deployment script:
#!/bin/bash
set -e
# Accept the new image URI as the first argument
NEW_IMAGE=$1
# Register a new task definition revision with the updated image
TASK_DEF=$(aws ecs register-task-definition \
--family compute-platform-task \
--container-definitions "[{\"name\":\"app\",\"image\":\"$NEW_IMAGE\",\"portMappings\":[{\"containerPort\":5000}],\"environment\":[{\"name\":\"TARGET\",\"value\":\"ecs-ec2\"}]}]" \
--network-mode awsvpc \
--requires-compatibilities EC2 \
--query 'taskDefinition.taskDefinitionArn' \
--output text)
# Build the AppSpec JSON that CodeDeploy uses to identify the new task def
APPSPEC=$(cat <<EOF
{"version":0.0,"Resources":[{"TargetService":{"Type":"AWS::ECS::Service","Properties":{"TaskDefinition":"$TASK_DEF","LoadBalancerInfo":{"ContainerName":"app","ContainerPort":5000}}}}]}
EOF
)
# Trigger the blue/green deployment
aws deploy create-deployment \
--application-name compute-platform \
--deployment-group-name ecs-bluegreen \
--revision "{\"revisionType\":\"AppSpecContent\",\"appSpecContent\":{\"content\":$(echo $APPSPEC | jq -Rs .)}}"
What does this script do?
- It takes a new ECR image URI as its argument (e.g., 123456789012.dkr.ecr.us-east-1.amazonaws.com/compute-platform:v2).
- It registers a new ECS task definition revision pointing to that image, using awsvpc network mode for per-task security group isolation.
- It builds the AppSpec JSON inline. The AppSpec tells CodeDeploy which task definition to deploy and which container/port to route traffic to.
- It calls aws deploy create-deployment which starts the canary rollout. CodeDeploy launches a green task set, shifts 10% of traffic, waits 5 minutes, then shifts the rest.
At Meridian, this script runs in CI after the scan gate passes. If the scan gate exits 1, this script never runs. Two layers of protection.
- Save tests/deploy-bluegreen.sh.
- Make the script executable by running:
chmod +x tests/deploy-bluegreen.sh
Script fails with jq not found?
The script uses jq to properly escape the AppSpec JSON for the CLI call.
Install it with your package manager: brew install jq on macOS, choco install jq on Windows, or apt-get install jq on Linux.
Help me install jq and fix my deploy script
Apply and verify the deployment group
Time to provision the CodeDeploy resources and confirm the deployment group is correctly configured with both target groups.
- Initialize and apply the Terraform changes by running:
cd infra
terraform init
terraform apply -auto-approve
- Verify the deployment group exists and is configured correctly by running:
aws deploy get-deployment-group --application-name compute-platform --deployment-group-name ecs-bluegreen
You should see a JSON response containing deploymentGroupInfo with both target group names listed under targetGroupPairInfoList and the deployment config set to CodeDeployDefault.ECSCanary10Percent5Minutes.
What should I see?
The response includes the deployment group configuration showing your canary config, auto-rollback enabled for DEPLOYMENT_FAILURE, both target groups (TG-1 and TG-2), and the production listener ARN.
This confirms CodeDeploy can orchestrate the blue/green traffic shift between your two target groups.
Deployment group does not exist?
Check that your ECS service's deployment controller is set to CODE_DEPLOY in the ECS-EC2 Terraform module. If the deployment controller is set to ECS (the default), CodeDeploy cannot manage that service.
Also verify that the IAM role was created before the deployment group by checking for errors in the terraform apply output. The role must exist before CodeDeploy can reference it.
Help me fix my CodeDeploy deployment group creation error
✔️ Awesome, I've got everything!
Great. Double check you have saved both infra/modules/codedeploy/main.tf and infra/main.tf, and that tests/deploy-bluegreen.sh is executable.
ⓧ I'd like to double check the full code
Here is the full infra/modules/codedeploy/main.tf:
resource "aws_codedeploy_app" "compute_platform" {
name = "compute-platform"
compute_platform = "ECS"
}
resource "aws_codedeploy_deployment_group" "ecs_bluegreen" {
app_name = aws_codedeploy_app.compute_platform.name
deployment_group_name = "ecs-bluegreen"
service_role_arn = aws_iam_role.codedeploy.arn
deployment_config_name = "CodeDeployDefault.ECSCanary10Percent5Minutes"
auto_rollback_configuration {
enabled = true
events = ["DEPLOYMENT_FAILURE"]
}
blue_green_deployment_config {
deployment_ready_option {
action_on_timeout = "CONTINUE_DEPLOYMENT"
}
terminate_blue_instances_on_deployment_success {
action = "TERMINATE"
termination_wait_time_in_minutes = 5
}
}
ecs_service {
cluster_name = module.ecs_ec2.cluster_name
service_name = module.ecs_ec2.service_name
}
load_balancer_info {
target_group_pair_info {
prod_traffic_route {
listener_arns = [module.alb.listener_arn]
}
target_group {
name = module.alb.tg1_name
}
target_group {
name = module.alb.tg2_name
}
}
}
}
Here is the updated infra/main.tf (showing the new module block added at the bottom):
terraform {
required_version = "~> 1.15"
backend "s3" {}
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "us-east-1"
}
# All prior modules (ECR, VPC, ALB, ECS-EC2, ECS-Fargate, EKS, Serverless) retained
module "codedeploy" {
source = "./modules/codedeploy"
ecs_cluster_name = module.ecs_ec2.cluster_name
ecs_service_name = module.ecs_ec2.service_name
listener_arn = module.alb.listener_arn
tg1_name = module.alb.tg1_name
tg2_name = module.alb.tg2_name
}
Here is the full tests/deploy-bluegreen.sh:
#!/bin/bash
set -e
NEW_IMAGE=$1
TASK_DEF=$(aws ecs register-task-definition --family compute-platform-task --container-definitions "[{\"name\":\"app\",\"image\":\"$NEW_IMAGE\",\"portMappings\":[{\"containerPort\":5000}],\"environment\":[{\"name\":\"TARGET\",\"value\":\"ecs-ec2\"}]}]" --network-mode awsvpc --requires-compatibilities EC2 --query 'taskDefinition.taskDefinitionArn' --output text)
APPSPEC=$(cat <<EOF
{"version":0.0,"Resources":[{"TargetService":{"Type":"AWS::ECS::Service","Properties":{"TaskDefinition":"$TASK_DEF","LoadBalancerInfo":{"ContainerName":"app","ContainerPort":5000}}}}]}
EOF
)
aws deploy create-deployment --application-name compute-platform --deployment-group-name ecs-bluegreen --revision "{\"revisionType\":\"AppSpecContent\",\"appSpecContent\":{\"content\":$(echo $APPSPEC | jq -Rs .)}}"
What happens if the green task set fails health checks during canary?
CodeDeploy automatically rolls back to the blue task set. Zero manual intervention. The 10% of traffic that was hitting the broken green set gets rerouted back to blue within seconds of the failure detection.
This is the SAP-C02 answer: CodeDeploy with auto-rollback on DEPLOYMENT_FAILURE provides automated recovery without operator action. The exam will present this exact scenario.
Your deployment pipeline is armed. Next up, you will chaos-test the platform to prove recovery time meets Meridian's SLA requirements.
Run Chaos Tests and Validate Recovery
Meridian's enterprise clients need proof that the platform can survive failure. Not a promise in a slide deck. Measured recovery, repeated on demand, with numbers attached.
In this step, the Breaker agent writes three validation scripts. A k6 load test proves traffic distributes evenly across all three compute targets. A chaos test kills a Fargate task and times recovery. A DLQ test proves no payment webhook is ever lost.
In this step, get ready to:
- Run k6 load tests against the ALB and API Gateway to prove weighted traffic distribution.
- Kill a Fargate task and validate sub-30-second auto-recovery.
- Send malformed payloads and confirm the SQS DLQ catches failed messages.
Write and run the k6 load test
The load test proves two things at once. First, that all three container targets (ECS-EC2, ECS-Fargate, EKS) receive roughly equal traffic through the weighted ALB listener. Second, that the API Gateway REST endpoint processes webhook payloads and persists them to DynamoDB.
- Create the k6 load test script by pasting the following code into tests/load.js:
import http from 'k6/http';
import { check } from 'k6';
export let options = { vus: 10, iterations: 100 };
export default function () {
// ALB load test
let res = http.get(__ENV.ALB_URL);
check(res, { 'ALB status 200': (r) => r.status === 200, 'has X-Target header': (r) => r.headers['X-Target'] !== undefined });
// API Gateway load test
let apiRes = http.post(__ENV.APIGW_URL + '/webhook', JSON.stringify({event: 'payment', amount: 100}), { headers: { 'Content-Type': 'application/json' } });
check(apiRes, { 'APIGW status 200': (r) => r.status === 200 });
}
What does this script do?
The script runs 100 iterations across 10 virtual users. Each iteration sends a GET to the ALB (checking for status 200 and the X-Target header that identifies which compute target served it) and a POST to API Gateway with a sample payment webhook.
The __ENV.ALB_URL and __ENV.APIGW_URL variables are passed at runtime via the -e flag. This keeps the script reusable across environments.
- Run the load test by passing your ALB DNS and API Gateway URL as environment variables:
k6 run -e ALB_URL=[[ALB_URL="your ALB DNS name (e.g. http://compute-platform-alb-123.us-east-1.elb.amazonaws.com)"]] -e APIGW_URL=[[APIGW_URL="your API Gateway endpoint URL (e.g. https://abc123.execute-api.us-east-1.amazonaws.com/prod)"]] tests/load.js
- Verify the k6 output. You should see all checks passing (100% for both ALB status 200 and APIGW status 200).
- In the k6 summary, check the X-Target header values across responses. You should see roughly 34/33/33 distribution across your three compute targets.
Seeing failed checks?
If the ALB check fails, confirm all three target groups are healthy by running aws elbv2 describe-target-health on each target group ARN. Unhealthy targets mean the container failed its health check.
If the API Gateway check fails, verify your URL includes the stage name (e.g. /prod). A missing stage returns 403.
Help me debug my k6 load test failures against ALB and API Gateway.
Write and run the chaos recovery test
A chaos test answers the critical SLA question: if a Fargate task dies unexpectedly, how fast does ECS replace it? The script kills a running task and polls until the service scheduler launches a replacement.
- Create the chaos test script by pasting the following code into tests/chaos-fargate.sh:
#!/bin/bash
set -e
CLUSTER=$1
SERVICE=$2
TASK=$(aws ecs list-tasks --cluster $CLUSTER --service-name $SERVICE --query 'taskArns[0]' --output text)
aws ecs stop-task --cluster $CLUSTER --task $TASK
echo "Task stopped. Waiting for recovery..."
for i in $(seq 1 6); do
sleep 5
COUNT=$(aws ecs describe-services --cluster $CLUSTER --services $SERVICE --query 'services[0].runningCount' --output text)
echo "Running count: $COUNT"
if [ "$COUNT" -ge 1 ]; then
echo "Recovery confirmed in $((i*5))s"
exit 0
fi
done
echo "Recovery failed within 30s"
exit 1
What does this script do?
The script grabs the first running task ARN from the Fargate service using aws ecs list-tasks. It then kills that task with aws ecs stop-task.
After stopping the task, it polls runningCount every 5 seconds for up to 30 seconds. The ECS service scheduler detects the running count dropped below desired count and launches a replacement. If recovery happens within 30 seconds, the script exits 0 (pass).
- Make the script executable and run it by passing your Fargate cluster name and service name:
chmod +x tests/chaos-fargate.sh
./tests/chaos-fargate.sh [[ECS_CLUSTER="your ECS Fargate cluster name"]] [[FARGATE_SERVICE="your Fargate service name"]]
- Watch the polling output. You should see the running count drop to 0 and then return to 1 within 30 seconds, followed by the message Recovery confirmed.
How does ECS handle a failed Fargate task?
The ECS service scheduler detects the task stopped and compares the running count to the desired count. When they do not match, it launches a replacement task. Recovery time depends on image pull time and health check interval. Sub-30-second recovery is typical for pre-cached images.
At Meridian, this is the SLA proof that renews enterprise contracts. The board needs to see a number, not a promise.
Recovery taking longer than 30 seconds?
If the image is not cached, the first pull adds 10-20 seconds. Run a warm-up task first to cache the image on the infrastructure.
If the health check grace period is too long, the task reports healthy after the script timeout. Reduce the grace period in your ECS service definition for testing purposes.
Help me troubleshoot why my Fargate task recovery is taking too long.
Write and run the DLQ retention test
For payment webhooks, every message must be accounted for. If Lambda cannot process a payload, the SQS dead-letter queue must catch it. This test sends deliberately malformed JSON and confirms the DLQ retains the failed message.
- Create the DLQ test script by pasting the following code into tests/dlq-test.sh:
#!/bin/bash
set -e
APIGW_URL=$1
DLQ_URL=$2
curl -s -X POST $APIGW_URL/webhook -H 'Content-Type: application/json' -d '{"malformed":'
echo "Malformed payload sent. Waiting 30s for DLQ..."
sleep 30
MSG=$(aws sqs receive-message --queue-url $DLQ_URL --query 'Messages[0].Body' --output text)
if [ -z "$MSG" ] || [ "$MSG" == "None" ]; then
echo "DLQ empty. Test failed."
exit 1
fi
echo "DLQ caught message: $MSG"
exit 0
What does this script do?
The script POSTs truncated JSON ({"malformed": with no closing brace) to the API Gateway webhook endpoint. Lambda attempts to parse it, fails, and throws an error.
Step Functions catches the error via the States.ALL catch block and routes the failed message to the SQS DLQ. After 30 seconds, the script checks the DLQ for the message. If found, the test passes.
SQS retains messages for up to 14 days (configured via messageRetentionSeconds). For payment webhooks where every message must be accounted for, this retention is mandatory.
- Make the script executable and run it by passing your API Gateway URL and SQS DLQ URL:
chmod +x tests/dlq-test.sh
./tests/dlq-test.sh [[APIGW_URL="your API Gateway endpoint URL (e.g. https://abc123.execute-api.us-east-1.amazonaws.com/prod)"]] [[DLQ_URL="your SQS DLQ URL (e.g. https://sqs.us-east-1.amazonaws.com/123456789012/webhook-dlq)"]]
- You should see the output DLQ caught message: followed by the message body. The script exits 0.
Why SQS DLQ instead of SNS?
SNS does not retain failed messages unless you explicitly configure a DLQ on the subscription. SQS retains by default for 4 days (configurable to 14). For payment webhooks where every transaction must be reconciled, SQS DLQ is the correct choice.
The SAP-C02 exam will present SNS fan-out as the simpler option. It is simpler, but it loses messages on failure. That is unacceptable for financial workloads.
DLQ empty after 30 seconds?
Check that your Lambda function throws the error instead of swallowing it. If the handler catches the JSON parse exception and returns 200, SQS has no reason to redrive.
Also confirm the Step Functions state machine has catch States.ALL routing to the DLQ. Without the catch block, failed executions simply terminate without forwarding the payload.
Help me figure out why my DLQ test is not catching the malformed message.
All three scripts should now exit 0. These results feed directly into the Narrator's presentation. Slide 5: Reliability Proof. Real numbers, not estimates.
✔️ Awesome, I've got everything!
Great. Confirm all three test files are saved in your tests/ directory and that each script exited 0 when you ran it.
ⓧ I'd like to double check the full code
Here is the complete code for all three test scripts:
import http from 'k6/http';
import { check } from 'k6';
export let options = { vus: 10, iterations: 100 };
export default function () {
// ALB load test
let res = http.get(__ENV.ALB_URL);
check(res, { 'ALB status 200': (r) => r.status === 200, 'has X-Target header': (r) => r.headers['X-Target'] !== undefined });
// API Gateway load test
let apiRes = http.post(__ENV.APIGW_URL + '/webhook', JSON.stringify({event: 'payment', amount: 100}), { headers: { 'Content-Type': 'application/json' } });
check(apiRes, { 'APIGW status 200': (r) => r.status === 200 });
}
#!/bin/bash
set -e
CLUSTER=$1
SERVICE=$2
TASK=$(aws ecs list-tasks --cluster $CLUSTER --service-name $SERVICE --query 'taskArns[0]' --output text)
aws ecs stop-task --cluster $CLUSTER --task $TASK
echo "Task stopped. Waiting for recovery..."
for i in $(seq 1 6); do
sleep 5
COUNT=$(aws ecs describe-services --cluster $CLUSTER --services $SERVICE --query 'services[0].runningCount' --output text)
echo "Running count: $COUNT"
if [ "$COUNT" -ge 1 ]; then
echo "Recovery confirmed in $((i*5))s"
exit 0
fi
done
echo "Recovery failed within 30s"
exit 1
#!/bin/bash
set -e
APIGW_URL=$1
DLQ_URL=$2
curl -s -X POST $APIGW_URL/webhook -H 'Content-Type: application/json' -d '{"malformed":'
echo "Malformed payload sent. Waiting 30s for DLQ..."
sleep 30
MSG=$(aws sqs receive-message --queue-url $DLQ_URL --query 'Messages[0].Body' --output text)
if [ -z "$MSG" ] || [ "$MSG" == "None" ]; then
echo "DLQ empty. Test failed."
exit 1
fi
echo "DLQ caught message: $MSG"
exit 0
Your platform is chaos-validated. Every number is real. Next up, the Narrator agent compiles these results into the leadership package: ADRs, topology diagrams, FinOps report, and the 8-slide board presentation that sells the architecture.
Build the Leadership Package and Board Presentation
Your platform is chaos-tested, your validation numbers are in, and every target group is healthy. Now you need to translate all of that into a narrative the board can act on.
The principal architect's job is not done when the infrastructure works. It is done when the VP of Engineering can walk into Friday's board meeting and say: "Architecture proven. Cost justified. Risk quantified." That requires ADRs, topology diagrams, a FinOps report, and a director-ready HTML presentation. The Narrator agent has been writing in parallel since minute 0. Now it ships.
In this step, get ready to:
- Write 6 Architecture Decision Records in MADR format capturing the WHY behind every decision.
- Create Mermaid topology diagrams and an implementation plan with the 5-agent dependency graph.
- Build a FinOps report and a director-ready 8-slide HTML presentation with real validation numbers.
Write Architecture Decision Records
An Architecture Decision Record (ADR) captures not just WHAT you decided, but WHY. At Meridian, each ADR maps to a business consequence the board cares about. The MADR format gives a consistent structure: Context, Decision, Consequences, Alternatives Rejected.
- Create the ADR directory by running this command:
mkdir -p docs/adr
- Direct the Narrator agent in Cursor Composer to create docs/adr/ADR-001-ecs-vs-eks-vs-lambda.md with the following content:
# ADR-001: ECS vs EKS vs Lambda
## Status: Accepted
## Context: Meridian needs to modernize Java payment engine and Node.js webhook dispatcher. Three compute options evaluated.
## Decision: ECS for Java payment engine (Replatform), Lambda+Step Functions for webhook (Refactor), EKS available but not primary.
## Consequences: EKS adds $876/yr control plane + ops overhead. ECS sufficient for Meridian's workload. Lambda handles 10K/sec webhook spikes.
## Alternatives Rejected: EKS-only (unnecessary complexity), EC2-only (status quo that caused outage).
What does this ADR communicate?
ADR-001 is the most critical record. It answers the board's first question: "Why three compute targets?" The answer maps to the 7 Rs migration strategy.
- Java payment engine: Replatform to ECS (containerize, no code changes).
- Node.js webhook: Refactor into Lambda + Step Functions + EventBridge.
- The $876/yr consequence gives finance a concrete number to weigh against EKS benefits.
- Create docs/adr/ADR-002-awsvpc-over-bridge.md with this content:
# ADR-002: awsvpc Network Mode
## Status: Accepted
## Context: PCI-DSS L1 requires network isolation between tasks.
## Decision: awsvpc for all ECS tasks and Fargate.
## Consequences: Per-task security groups. PCI audit finding avoided. Bridge mode would require broad ephemeral port ranges.
## Alternatives Rejected: bridge mode (fails PCI SG isolation requirement).
- Create docs/adr/ADR-003-rest-over-http-api.md with this content:
# ADR-003: REST API over HTTP API
## Status: Accepted
## Context: Enterprise clients require API key authentication for webhook endpoints.
## Decision: API Gateway REST API with usage plans and API keys.
## Consequences: Enterprise clients can authenticate. HTTP API is cheaper but does not support usage plans or API keys.
## Alternatives Rejected: HTTP API (no usage plan support, cannot authenticate enterprise clients).
- Create docs/adr/ADR-004-express-over-standard.md with this content:
# ADR-004: Step Functions Express over Standard
## Status: Accepted
## Context: Webhook volume peaks 10K/sec on payroll day.
## Decision: Step Functions Express state machine.
## Consequences: Handles 100K/sec. Standard limited to 2K starts/sec. Idempotent DynamoDB PUT achieves exactly-once semantics.
## Alternatives Rejected: Standard (2K/sec limit insufficient for payroll-day spikes).
- Create docs/adr/ADR-005-terraform-only.md with this content:
# ADR-005: Terraform-Only Infrastructure
## Status: Accepted
## Context: Finance needs one line item. Ops needs one runbook. Zero orphans required.
## Decision: All resources managed by Terraform. Single terraform destroy teardown.
## Consequences: Finance sees one tool. Ops has one runbook. No ghost resources.
## Alternatives Rejected: Mixed CDK+Terraform (two state systems, orphan risk).
- Create docs/adr/ADR-006-python-snapstart.md with this content:
# ADR-006: Python 3.13 with SnapStart
## Status: Accepted
## Context: Payment callbacks require P99 < 500ms. Lambda cold starts can exceed 1s.
## Decision: Python 3.13 Lambda with SnapStart enabled.
## Consequences: Sub-second cold start. P99 < 500ms for payment callbacks.
## Alternatives Rejected: Python 3.11 without SnapStart (cold start latency unacceptable for payment SLA).
Why 6 ADRs, not 1 giant document?
Each ADR is scoped to a single decision. When the CISO asks "why awsvpc?", the VP pulls up ADR-002. When finance asks "why not EKS?", they get ADR-001 with the $876/yr number. Atomic records serve different stakeholders without forcing them to read a monolith.
- Confirm all 6 ADRs exist by running:
ls docs/adr/
You should see 6 files: ADR-001-ecs-vs-eks-vs-lambda.md through ADR-006-python-snapstart.md.
Missing ADR files?
Make sure the Narrator agent created them inside docs/adr/ and not in the project root. If files landed in the wrong directory, move them with mv ADR-*.md docs/adr/.
Why aren't my ADR files being created in the right folder?
Create topology diagrams and implementation plan
The board cannot read Terraform modules. They need visual topology diagrams they can absorb in 5 seconds. Mermaid diagrams render from text, paste directly into mermaid.live, and version-control cleanly in Git.
- Direct the Narrator agent to create docs/diagrams.md with three Mermaid diagrams:
# Architecture Diagrams
## Container Topology (Mermaid)
```mermaid
graph TD
Internet --> ALB
ALB -->|34%| ECS_EC2[ECS EC2 t3.micro]
ALB -->|33%| ECS_Fargate[ECS Fargate 0.25vCPU]
ALB -->|33%| EKS[EKS v1.32 t3.small]
ECS_EC2 --> ECR
ECS_Fargate --> ECR
EKS --> ECR
```
## Serverless Event Flow (Mermaid)
```mermaid
graph LR
Client --> APIGW[API Gateway REST]
APIGW --> WAF
WAF --> Lambda[Lambda SnapStart]
Lambda --> EventBridge
EventBridge --> SFN[Step Functions Express]
SFN --> DDB[DynamoDB]
SFN -->|failure| DLQ[SQS DLQ]
```
## Blue/Green Sequence (Mermaid)
```mermaid
sequenceDiagram
Dev->>ECR: Push image v2
ECR->>ScanGate: Scan on push
ScanGate->>CodeDeploy: Exit 0 (clean)
CodeDeploy->>ALB: Shift 10% to green
ALB->>HealthCheck: Monitor 5 min
HealthCheck->>CodeDeploy: Healthy
CodeDeploy->>ALB: Shift 100% to green
```
What do these diagrams communicate?
- Container topology shows the weighted ALB splitting traffic 34/33/33 across all three targets. The board sees the comparison at a glance.
- Serverless event flow shows how a webhook enters, gets processed, and either persists to DynamoDB or catches in the DLQ. Every message is accounted for.
- Blue/green sequence shows the scan gate blocking before CodeDeploy starts. This is the deployment that would have prevented the $1.2M outage.
- Create docs/implementation-plan.md documenting the dependency graph and time budget:
# Implementation Plan
## Dependency Graph
- Foundation (min 0): ECR, VPC, ALB, ECS, EKS
- Backbone (min 10): needs VPC from Foundation
- Pipeline (min 20): needs ECR + ECS from Foundation
- Breaker (min 40): needs live endpoints from Foundation + Backbone
- Narrator (min 0-90): continuous, fills validation numbers at min 80
## Time Budget
- Min 0-15: Foundation ECR + VPC + ECS
- Min 10-50: Backbone serverless stack
- Min 15-45: Foundation EKS (background)
- Min 30-60: Pipeline CodeDeploy
- Min 45-75: Breaker chaos + load tests
- Min 0-90: Narrator ADRs + diagrams + presentation
Why document the implementation plan?
The board does not fund teams that got lucky. They fund teams that executed a plan. This document shows how 5 agents achieved 90-minute delivery through deliberate decomposition and parallelization, not ad-hoc improvisation.
- Confirm both files exist by running:
ls docs/diagrams.md docs/implementation-plan.md
You should see both files listed in the output.
Validate Mermaid renders
Copy any code block from docs/diagrams.md and paste it into mermaid.live to confirm it renders correctly. A broken diagram in the board presentation destroys credibility.
Mermaid syntax errors?
Check that node IDs do not contain spaces or special characters. Use underscores for multi-word labels like ECS_EC2 and bracket notation for display names like [ECS EC2 t3.micro].
Why is my Mermaid diagram not rendering correctly?
Build the FinOps report and HTML presentation
Finance needs to see exactly what this costs before approving $4M in annual cloud spend. The FinOps report breaks down every resource with free-tier mapping, and the HTML presentation packages everything into 8 slides the VP can open on their laptop.
- Create docs/finops-report.md with per-resource cost breakdown:
# FinOps Report
## Lab Cost (90 min): $0.50-1.50
- EKS control plane: $0.10/hr x 1.5hr = $0.15
- NAT Gateway: $0.045/hr x 1.5hr = $0.07
- t3.micro EC2: free tier
- t3.small EKS node: $0.023/hr x 1.5hr = $0.03
- Lambda, API GW, DDB, SQS: free tier
## Annualized Production Estimate
- ECS-only: $14,000-18,000/yr
- With EKS: $24,000-30,000/yr (+$876/yr control plane + ops overhead)
## Free Tier Map
- Lambda: 1M requests/mo free
- DynamoDB: 25 GB + 25 WCU/RCU free
- API Gateway: 1M calls/mo free (12 mo)
- ECR: 500 MB/mo free
What does the FinOps report communicate?
The report answers three questions the board will ask. "What did this proof-of-concept cost?" ($0.50-1.50). "What will production cost?" ($14-18K/yr ECS-only). "What if we add EKS?" ($24-30K/yr, and here is why you do not need it).
The free-tier map shows that Lambda, DynamoDB, SQS, and EventBridge cost $0 at Meridian's webhook volume during the proof phase.
- Create docs/validation-results.md capturing the real numbers from your test runs:
# Validation Results
## Load Test (k6)
- ALB: 100/100 requests returned 200
- X-Target distribution: ECS-EC2 ~34%, ECS-Fargate ~33%, EKS ~33%
- API Gateway: 50/50 requests returned 200
- DynamoDB item count increased by 50
## Chaos Test
- Fargate task stopped at T+0
- Recovery confirmed at T+<30s
- runningCount returned to 1
## DLQ Test
- Malformed JSON sent to API Gateway
- Payload confirmed in SQS DLQ within 30s
- Message retention: 14 days
Why capture validation results separately?
These numbers feed directly into the presentation slides. Slide 5 (Reliability) references the chaos recovery time. Slide 3 (Architecture) references the weighted distribution. Real data, not estimates.
- Create the presentation directory and build presentation/index.html as a React-style SPA with 8 navigable slides:
mkdir -p presentation
- Create presentation/index.html with the following HTML:
<!DOCTYPE html><html><head><title>Meridian Payments - Compute Modernization</title><style>body{font-family:sans-serif;margin:0;background:#0a0a1a;color:#fff}.slide{display:none;padding:60px;min-height:100vh}.slide.active{display:block}h1{color:#00d4ff}button{background:#00d4ff;color:#000;border:none;padding:10px 20px;cursor:pointer;margin:5px}</style></head><body><div id='slide1' class='slide active'><h1>Meridian Payments: Compute Modernization</h1><p>Principal Architect Briefing | Board Meeting Friday</p><p>ECS vs EKS vs Lambda — Proven Under Load</p></div><div id='slide2' class='slide'><h1>The $1.2M Problem</h1><p>47-minute outage. Manual SSH deploy. No rollback. No scan gate.</p><p>Three enterprise clients issued RFPs. Wall Street Journal coverage.</p></div><div id='slide3' class='slide'><h1>Architecture: Three Paths Proven</h1><p>ECS-EC2 | ECS-Fargate | EKS — same image, weighted ALB 34/33/33</p><p>Serverless: REST API + SnapStart Lambda + Express Step Functions + DLQ</p></div><div id='slide4' class='slide'><h1>FinOps: What It Costs</h1><p>Lab: $1.50 | Production ECS-only: $14-18K/yr | With EKS: $24-30K/yr</p></div><div id='slide5' class='slide'><h1>Reliability: Chaos-Tested Proof</h1><p>Fargate recovery: <30s | ALB distribution: 34/33/33 confirmed | DLQ: 100% catch rate</p></div><div id='slide6' class='slide'><h1>Security: Scan Gate + PCI Alignment</h1><p>ECR scan-on-push. Critical CVE blocks deploy. awsvpc per-task isolation. PCI-DSS L1 maintained.</p></div><div id='slide7' class='slide'><h1>Technical Debt & Remediation Timeline</h1><p>Phase 1 (complete): Containerize + serverless backbone. Phase 2: Service Connect + EventBridge Pipes + Karpenter.</p></div><div id='slide8' class='slide'><h1>Phase 2: What We Build Next</h1><p>ECS Service Connect (east-west, no ALB hop). EventBridge Pipes (eliminate Lambda glue). Karpenter (30-60% compute savings). CloudWatch Composite Alarms.</p></div><div style='position:fixed;bottom:20px;left:50%;transform:translateX(-50%)'><button onclick='prev()'>← Prev</button><button onclick='next()'>Next →</button></div><script>let cur=1;const total=8;function show(n){document.querySelectorAll('.slide').forEach(s=>s.classList.remove('active'));document.getElementById('slide'+n).classList.add('active')}function next(){if(cur<total){cur++;show(cur)}}function prev(){if(cur>1){cur--;show(cur)}}</script></body></html>
What are the 8 slides?
- Slide 1: Executive Summary (the ask and the answer).
- Slide 2: The $1.2M Problem (why we are here).
- Slide 3: Architecture (what we built and proved).
- Slide 4: FinOps (what it costs, with projections).
- Slide 5: Reliability (chaos-tested, measured recovery).
- Slide 6: Security (scan gate, PCI alignment proof).
- Slide 7: Technical Debt (what remains, when it gets fixed).
- Slide 8: Phase 2 (what we build next, positioning forward).
- Open presentation/index.html in your browser.
- Click the Next → button to navigate through all 8 slides.
You should see a dark-themed presentation with Meridian branding, cyan headings, and working navigation between all 8 slides.
Presentation not rendering?
Make sure you opened the file directly in a browser (double-click or drag into a browser tab). If slides are not navigating, check the browser console for JavaScript errors. The script relies on getElementById matching the div IDs slide1 through slide8.
Why is my HTML presentation not showing slides correctly?
- Commit the entire leadership package by running these commands:
git add docs/ presentation/
git commit -m "sprint-complete: validated + briefing shipped"
Your Git history now shows the complete sprint from infrastructure to board briefing.
✔️ Awesome, I've got everything!
Double check that all files are saved and the presentation opens in your browser with working navigation across all 8 slides.
ⓧ I'd like to double check the full code
Here are all the files from this step. Compare against your versions.
# ADR-001: ECS vs EKS vs Lambda
## Status: Accepted
## Context: Meridian needs to modernize Java payment engine and Node.js webhook dispatcher. Three compute options evaluated.
## Decision: ECS for Java payment engine (Replatform), Lambda+Step Functions for webhook (Refactor), EKS available but not primary.
## Consequences: EKS adds $876/yr control plane + ops overhead. ECS sufficient for Meridian's workload. Lambda handles 10K/sec webhook spikes.
## Alternatives Rejected: EKS-only (unnecessary complexity), EC2-only (status quo that caused outage).
# ADR-002: awsvpc Network Mode
## Status: Accepted
## Context: PCI-DSS L1 requires network isolation between tasks.
## Decision: awsvpc for all ECS tasks and Fargate.
## Consequences: Per-task security groups. PCI audit finding avoided. Bridge mode would require broad ephemeral port ranges.
## Alternatives Rejected: bridge mode (fails PCI SG isolation requirement).
# ADR-003: REST API over HTTP API
## Status: Accepted
## Context: Enterprise clients require API key authentication for webhook endpoints.
## Decision: API Gateway REST API with usage plans and API keys.
## Consequences: Enterprise clients can authenticate. HTTP API is cheaper but does not support usage plans or API keys.
## Alternatives Rejected: HTTP API (no usage plan support, cannot authenticate enterprise clients).
# ADR-004: Step Functions Express over Standard
## Status: Accepted
## Context: Webhook volume peaks 10K/sec on payroll day.
## Decision: Step Functions Express state machine.
## Consequences: Handles 100K/sec. Standard limited to 2K starts/sec. Idempotent DynamoDB PUT achieves exactly-once semantics.
## Alternatives Rejected: Standard (2K/sec limit insufficient for payroll-day spikes).
# ADR-005: Terraform-Only Infrastructure
## Status: Accepted
## Context: Finance needs one line item. Ops needs one runbook. Zero orphans required.
## Decision: All resources managed by Terraform. Single terraform destroy teardown.
## Consequences: Finance sees one tool. Ops has one runbook. No ghost resources.
## Alternatives Rejected: Mixed CDK+Terraform (two state systems, orphan risk).
# ADR-006: Python 3.13 with SnapStart
## Status: Accepted
## Context: Payment callbacks require P99 < 500ms. Lambda cold starts can exceed 1s.
## Decision: Python 3.13 Lambda with SnapStart enabled.
## Consequences: Sub-second cold start. P99 < 500ms for payment callbacks.
## Alternatives Rejected: Python 3.11 without SnapStart (cold start latency unacceptable for payment SLA).
# Architecture Diagrams
## Container Topology (Mermaid)
```mermaid
graph TD
Internet --> ALB
ALB -->|34%| ECS_EC2[ECS EC2 t3.micro]
ALB -->|33%| ECS_Fargate[ECS Fargate 0.25vCPU]
ALB -->|33%| EKS[EKS v1.32 t3.small]
ECS_EC2 --> ECR
ECS_Fargate --> ECR
EKS --> ECR
```
## Serverless Event Flow (Mermaid)
```mermaid
graph LR
Client --> APIGW[API Gateway REST]
APIGW --> WAF
WAF --> Lambda[Lambda SnapStart]
Lambda --> EventBridge
EventBridge --> SFN[Step Functions Express]
SFN --> DDB[DynamoDB]
SFN -->|failure| DLQ[SQS DLQ]
```
## Blue/Green Sequence (Mermaid)
```mermaid
sequenceDiagram
Dev->>ECR: Push image v2
ECR->>ScanGate: Scan on push
ScanGate->>CodeDeploy: Exit 0 (clean)
CodeDeploy->>ALB: Shift 10% to green
ALB->>HealthCheck: Monitor 5 min
HealthCheck->>CodeDeploy: Healthy
CodeDeploy->>ALB: Shift 100% to green
```
# Implementation Plan
## Dependency Graph
- Foundation (min 0): ECR, VPC, ALB, ECS, EKS
- Backbone (min 10): needs VPC from Foundation
- Pipeline (min 20): needs ECR + ECS from Foundation
- Breaker (min 40): needs live endpoints from Foundation + Backbone
- Narrator (min 0-90): continuous, fills validation numbers at min 80
## Time Budget
- Min 0-15: Foundation ECR + VPC + ECS
- Min 10-50: Backbone serverless stack
- Min 15-45: Foundation EKS (background)
- Min 30-60: Pipeline CodeDeploy
- Min 45-75: Breaker chaos + load tests
- Min 0-90: Narrator ADRs + diagrams + presentation
# FinOps Report
## Lab Cost (90 min): $0.50-1.50
- EKS control plane: $0.10/hr x 1.5hr = $0.15
- NAT Gateway: $0.045/hr x 1.5hr = $0.07
- t3.micro EC2: free tier
- t3.small EKS node: $0.023/hr x 1.5hr = $0.03
- Lambda, API GW, DDB, SQS: free tier
## Annualized Production Estimate
- ECS-only: $14,000-18,000/yr
- With EKS: $24,000-30,000/yr (+$876/yr control plane + ops overhead)
## Free Tier Map
- Lambda: 1M requests/mo free
- DynamoDB: 25 GB + 25 WCU/RCU free
- API Gateway: 1M calls/mo free (12 mo)
- ECR: 500 MB/mo free
# Validation Results
## Load Test (k6)
- ALB: 100/100 requests returned 200
- X-Target distribution: ECS-EC2 ~34%, ECS-Fargate ~33%, EKS ~33%
- API Gateway: 50/50 requests returned 200
- DynamoDB item count increased by 50
## Chaos Test
- Fargate task stopped at T+0
- Recovery confirmed at T+<30s
- runningCount returned to 1
## DLQ Test
- Malformed JSON sent to API Gateway
- Payload confirmed in SQS DLQ within 30s
- Message retention: 14 days
<!DOCTYPE html><html><head><title>Meridian Payments - Compute Modernization</title><style>body{font-family:sans-serif;margin:0;background:#0a0a1a;color:#fff}.slide{display:none;padding:60px;min-height:100vh}.slide.active{display:block}h1{color:#00d4ff}button{background:#00d4ff;color:#000;border:none;padding:10px 20px;cursor:pointer;margin:5px}</style></head><body><div id='slide1' class='slide active'><h1>Meridian Payments: Compute Modernization</h1><p>Principal Architect Briefing | Board Meeting Friday</p><p>ECS vs EKS vs Lambda — Proven Under Load</p></div><div id='slide2' class='slide'><h1>The $1.2M Problem</h1><p>47-minute outage. Manual SSH deploy. No rollback. No scan gate.</p><p>Three enterprise clients issued RFPs. Wall Street Journal coverage.</p></div><div id='slide3' class='slide'><h1>Architecture: Three Paths Proven</h1><p>ECS-EC2 | ECS-Fargate | EKS — same image, weighted ALB 34/33/33</p><p>Serverless: REST API + SnapStart Lambda + Express Step Functions + DLQ</p></div><div id='slide4' class='slide'><h1>FinOps: What It Costs</h1><p>Lab: $1.50 | Production ECS-only: $14-18K/yr | With EKS: $24-30K/yr</p></div><div id='slide5' class='slide'><h1>Reliability: Chaos-Tested Proof</h1><p>Fargate recovery: <30s | ALB distribution: 34/33/33 confirmed | DLQ: 100% catch rate</p></div><div id='slide6' class='slide'><h1>Security: Scan Gate + PCI Alignment</h1><p>ECR scan-on-push. Critical CVE blocks deploy. awsvpc per-task isolation. PCI-DSS L1 maintained.</p></div><div id='slide7' class='slide'><h1>Technical Debt & Remediation Timeline</h1><p>Phase 1 (complete): Containerize + serverless backbone. Phase 2: Service Connect + EventBridge Pipes + Karpenter.</p></div><div id='slide8' class='slide'><h1>Phase 2: What We Build Next</h1><p>ECS Service Connect (east-west, no ALB hop). EventBridge Pipes (eliminate Lambda glue). Karpenter (30-60% compute savings). CloudWatch Composite Alarms.</p></div><div style='position:fixed;bottom:20px;left:50%;transform:translateX(-50%)'><button onclick='prev()'>← Prev</button><button onclick='next()'>Next →</button></div><script>let cur=1;const total=8;function show(n){document.querySelectorAll('.slide').forEach(s=>s.classList.remove('active'));document.getElementById('slide'+n).classList.add('active')}function next(){if(cur<total){cur++;show(cur)}}function prev(){if(cur>1){cur--;show(cur)}}</script></body></html>
Your leadership package is shipped. 6 ADRs explaining every decision with business consequences. 3 topology diagrams the board can absorb in seconds. A FinOps report that answers "how much?" before finance asks. An 8-slide presentation with real chaos-tested numbers. The VP of Engineering is prepared for Friday.
Run the ECR Scan Gate Demo and Final Validation
Your infrastructure is live, chaos-tested, and the leadership package is written. Now you need the two pieces of evidence that close the loop for Meridian's board: a proven zero-downtime deployment and a scan gate that blocks vulnerable code from reaching production.
This final phase deploys a clean image v2 via blue/green, proves mixed traffic during canary with zero errors, then deliberately pushes a vulnerable image to demonstrate the scan gate rejecting it. The result is auditable evidence that the $1.2M outage cannot recur.
In this step, get ready to:
- Deploy image v2 via CodeDeploy blue/green and prove zero-downtime traffic shift.
- Demonstrate the scan gate blocking a vulnerable image with Critical CVEs.
- Complete the final validation sweep, update the presentation with real numbers, and commit.
Deploy clean image v2 via blue/green
The blue/green deployment is what would have prevented Meridian's $1.2M outage. Traffic shifts gradually, health checks gate progression, and rollback is automatic if anything fails.
- Instruct the Pipeline agent to trigger the blue/green deployment with the v2 image by running deploy-bluegreen.sh:
# Deploy v2 image via the blue/green script
ECR_URI=$(aws ecr describe-repositories --repository-names compute-platform --query 'repositories[0].repositoryUri' --output text)
bash tests/deploy-bluegreen.sh $ECR_URI:v2
What does this command do?
The deploy-bluegreen.sh script registers a new task definition with the v2 image, builds an AppSpec JSON, and calls aws deploy create-deployment. CodeDeploy starts a canary: 10% of traffic shifts to the green task set for 5 minutes. If health checks pass, the remaining 90% shifts automatically.
- Note the deployment ID from the output of the previous command.
- While the canary is active (during the first 5 minutes), run the k6 load test to prove mixed v1/v2 traffic with zero downtime:
# Run k6 during canary to prove mixed traffic
k6 run --env ALB_URL=http://$(terraform -chdir=infra output -raw alb_dns) --env APIGW_URL=$(terraform -chdir=infra output -raw apigw_url) tests/load.js
What should you see?
The k6 output shows 100% of requests returning 200 OK. The X-Target header in responses shows a mix of v1 and v2 targets during canary.
After the canary completes, all traffic shifts to v2. This is the deployment pattern that replaced Meridian's manual SSH deploy.
- Poll the deployment status until it shows Succeeded:
# Poll deployment status until Succeeded
aws deploy get-deployment --deployment-id [[DEPLOYMENT_ID="your deployment ID from the create-deployment output"]] --query 'deploymentInfo.status' --output text
- You should see Succeeded in the output. The canary phase takes 5 minutes by design.
Deployment stuck in InProgress?
The canary phase takes 5 minutes by design (CodeDeployDefault.ECSCanary10Percent5Minutes). Wait the full 5 minutes before checking again.
If it shows Failed, the green task set failed health checks and CodeDeploy auto-rolled back to blue. Check the ECS service events for health check failures. The most common cause is a port mapping mismatch between the task definition and the target group.
Why is my blue/green deployment failing?
Demonstrate the scan gate blocking a vulnerable image
At Meridian, this scan gate runs in CI. If it exits 1, the pipeline stops. The image never reaches ECS. The CVE never reaches production. PCI-DSS L1 certification is maintained.
- Instruct the Pipeline agent to build a deliberately vulnerable image using python:3.8-slim as the base (known Critical CVEs) by running these commands:
# Build a vulnerable image to demonstrate the scan gate
cat > /tmp/Dockerfile.insecure << 'EOF'
FROM python:3.8-slim
COPY lambda/handler.py /app/handler.py
CMD ["python", "/app/handler.py"]
EOF
# Build and tag as v-insecure
docker build -f /tmp/Dockerfile.insecure -t compute-platform:v-insecure .
# Tag and push to ECR
ECR_URI=$(aws ecr describe-repositories --repository-names compute-platform --query 'repositories[0].repositoryUri' --output text)
docker tag compute-platform:v-insecure $ECR_URI:v-insecure
docker push $ECR_URI:v-insecure
Why python:3.8-slim?
Python 3.8 reached end-of-life and its base image contains multiple known Critical CVEs in system libraries. This makes it a reliable test case for demonstrating the scan gate.
The ECR scan_on_push configuration triggers an automatic scan as soon as the image is pushed. No manual action needed.
- Wait for the ECR scan to complete by running this command (polls every 5 seconds, typically takes about 90 seconds):
# Wait for scan to complete
aws ecr wait image-scan-complete --repository-name compute-platform --image-id imageTag=v-insecure
- Run the scan gate script against the vulnerable image:
# Run scan-gate.sh - should exit 1 (BLOCKED)
bash tests/scan-gate.sh compute-platform v-insecure
What does scan-gate.sh do?
The script queries ECR scan findings for CRITICAL severity CVEs. If the count is greater than 0, it prints the finding count and exits with code 1. This non-zero exit code stops any downstream pipeline.
You should see output like: CRITICAL CVEs found: 12. Blocking deployment. followed by exit code 1.
- Capture the evidence in a log file for audit purposes by running:
# Write scan gate evidence log
CRITICAL_COUNT=$(aws ecr describe-image-scan-findings --repository-name compute-platform --image-id imageTag=v-insecure --query 'imageScanFindings.findingSeverityCounts.CRITICAL' --output text)
cat > docs/scan-gate-evidence.log << EOF
ECR Scan Gate Evidence Log
Timestamp: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
Image: python:3.8-slim based image tagged v-insecure
Repository: compute-platform
Scan Status: COMPLETE
CRITICAL findings: $CRITICAL_COUNT
scan-gate.sh exit code: 1
Result: BLOCKED - image did not reach ECS. PCI-DSS L1 maintained.
EOF
- Verify the evidence log was written by running:
cat docs/scan-gate-evidence.log
- You should see the full evidence log with a timestamp, CRITICAL finding count, and the BLOCKED result.
What does this evidence prove?
The CISO needs auditable proof that no image with Critical CVEs can reach production. This log shows the exact timestamp, finding count, and blocking result.
In Meridian's CI pipeline, this evidence is attached to every deployment attempt. Clean images pass (exit 0). Vulnerable images are blocked (exit 1). PCI-DSS L1 certification is maintained.
Scan findings show 0 CRITICAL?
ECR basic scanning may not detect as many CVEs as enhanced scanning (which uses Amazon Inspector). If you see 0 CRITICAL findings, the base image may have been patched. Try an even older base like python:3.6-slim or enable enhanced scanning on your repository.
Also confirm the scan completed. If the wait command timed out, the scan may still be in progress. Re-run the wait command before retrying the scan gate.
Why is my ECR scan not finding Critical CVEs?
Run the final validation sweep and update the presentation
Every stakeholder needs proof. The board needs real numbers in the presentation. The SRE team needs a validation report.
- Instruct the Breaker agent to run the final infrastructure health check across all targets:
# Final sweep: verify all 3 target groups are healthy
aws elbv2 describe-target-health --target-group-arn $(terraform -chdir=infra output -raw tg1_arn) --query 'TargetHealthDescriptions[0].TargetHealth.State' --output text
aws elbv2 describe-target-health --target-group-arn $(terraform -chdir=infra output -raw tg2_arn) --query 'TargetHealthDescriptions[0].TargetHealth.State' --output text
aws elbv2 describe-target-health --target-group-arn $(terraform -chdir=infra output -raw tg3_arn) --query 'TargetHealthDescriptions[0].TargetHealth.State' --output text
# Verify Lambda alias is live at 100%
aws lambda get-alias --function-name webhook-processor --name live --query 'FunctionVersion' --output text
# Verify Step Functions execution succeeded
aws stepfunctions list-executions --state-machine-arn $(terraform -chdir=infra output -raw sfn_arn) --status-filter SUCCEEDED --query 'executions[0].status' --output text
# Verify DynamoDB items exist
aws dynamodb scan --table-name compute-platform-events --select COUNT --query 'Count' --output text
# Verify DLQ caught failure
aws sqs get-queue-attributes --queue-url $(terraform -chdir=infra output -raw dlq_url) --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text
What are you checking?
Each command verifies one component of the platform. Three target groups must show healthy (container platform). Lambda alias returns a version number (serverless live). Step Functions returns SUCCEEDED (orchestration working).
DynamoDB returns a count greater than 0 (persistence confirmed). SQS returns a message count greater than 0 (failure handling proven).
- Instruct the Narrator agent to update docs/validation-results.md with the final results:
# Final Validation Results
## Infrastructure
- TG-1 (ECS-EC2): healthy
- TG-2 (ECS-Fargate): healthy
- TG-3 (EKS): healthy
## Blue/Green Deployment
- CodeDeploy deployment: Succeeded
- Canary 10% for 5 min: zero errors
- Traffic shifted 100%: zero downtime
## Load Test (k6)
- ALB: 100/100 requests 200 OK
- Distribution: ECS-EC2 34%, ECS-Fargate 33%, EKS 33%
- API Gateway: 50/50 requests 200 OK
- DynamoDB items created: 50
## Chaos Test
- Fargate recovery: <30s confirmed
## DLQ Test
- Malformed payload caught in DLQ: confirmed
## Scan Gate
- Clean image (v1, v2): exit 0
- Vulnerable image (python:3.8-slim): exit 1, BLOCKED
## Elapsed Time: ≤90 minutes
- Instruct the Narrator agent to update slide 5 and slide 6 in presentation/index.html with the real validation numbers.
- Slide 5 content should now read: Fargate recovery <30s confirmed, ALB distribution 34/33/33 confirmed, DLQ catch rate 100%, and k6 150/150 requests 200 OK, all with checkmarks.
- Slide 6 should include the scan gate evidence: Critical CVE blocks deploy (12 CVEs blocked, evidence logged).
- Make the final git commit to mark the sprint as complete:
git add -A
git commit -m "sprint-complete: validated + briefing shipped"
- Verify the final state by opening presentation/index.html in your browser.
- All 8 slides should have real numbers. No pending placeholders remain.
What the board sees on Friday
The VP of Engineering opens the HTML presentation on their laptop. Eight slides. Architecture proven with data. Cost justified at $14-18K/yr. Risk quantified with scan-gate evidence.
Reliability demonstrated with chaos test results. Phase 2 positioned. The $4M budget is approved. Your contract is renewed.
One of the validation checks returned unexpected output?
If a target group shows unhealthy, check that the ECS service and EKS deployment are still running. Long-running labs can hit spot interruptions or task eviction.
If DynamoDB shows 0 items, re-run the k6 load test against the API Gateway URL. The Lambda function writes to DynamoDB on each invocation.
Why is my final validation sweep showing failures?
✔️ Awesome, I've got everything!
Great. Double check you have saved all files and your final git commit is in place.
ⓧ I'd like to double check the full code
Here is the complete docs/scan-gate-evidence.log:
ECR Scan Gate Evidence Log
Timestamp: [run timestamp]
Image: python:3.8-slim based image tagged v-insecure
Repository: compute-platform
Scan Status: COMPLETE
CRITICAL findings: 12
scan-gate.sh exit code: 1
Result: BLOCKED - image did not reach ECS. PCI-DSS L1 maintained.
Here is the complete docs/validation-results.md:
# Final Validation Results
## Infrastructure
- TG-1 (ECS-EC2): healthy
- TG-2 (ECS-Fargate): healthy
- TG-3 (EKS): healthy
## Blue/Green Deployment
- CodeDeploy deployment: Succeeded
- Canary 10% for 5 min: zero errors
- Traffic shifted 100%: zero downtime
## Load Test (k6)
- ALB: 100/100 requests 200 OK
- Distribution: ECS-EC2 34%, ECS-Fargate 33%, EKS 33%
- API Gateway: 50/50 requests 200 OK
- DynamoDB items created: 50
## Chaos Test
- Fargate recovery: <30s confirmed
## DLQ Test
- Malformed payload caught in DLQ: confirmed
## Scan Gate
- Clean image (v1, v2): exit 0
- Vulnerable image (python:3.8-slim): exit 1, BLOCKED
## Elapsed Time: ≤90 minutes
Here is the final presentation/index.html with slides 5 and 6 updated:
<!DOCTYPE html><html><head><title>Meridian Payments - Compute Modernization</title><style>body{font-family:sans-serif;margin:0;background:#0a0a1a;color:#fff}.slide{display:none;padding:60px;min-height:100vh}.slide.active{display:block}h1{color:#00d4ff}button{background:#00d4ff;color:#000;border:none;padding:10px 20px;cursor:pointer;margin:5px}</style></head><body><div id='slide1' class='slide active'><h1>Meridian Payments: Compute Modernization</h1><p>Principal Architect Briefing | Board Meeting Friday</p><p>ECS vs EKS vs Lambda — Proven Under Load</p></div><div id='slide2' class='slide'><h1>The $1.2M Problem</h1><p>47-minute outage. Manual SSH deploy. No rollback. No scan gate.</p><p>Three enterprise clients issued RFPs. Wall Street Journal coverage.</p></div><div id='slide3' class='slide'><h1>Architecture: Three Paths Proven</h1><p>ECS-EC2 | ECS-Fargate | EKS — same image, weighted ALB 34/33/33</p><p>Serverless: REST API + SnapStart Lambda + Express Step Functions + DLQ</p></div><div id='slide4' class='slide'><h1>FinOps: What It Costs</h1><p>Lab: $1.50 | Production ECS-only: $14-18K/yr | With EKS: $24-30K/yr</p></div><div id='slide5' class='slide'><h1>Reliability: Chaos-Tested Proof</h1><p>Fargate recovery: <30s ✓ | ALB distribution: 34/33/33 confirmed ✓ | DLQ catch rate: 100% ✓ | k6: 150/150 requests 200 OK ✓</p></div><div id='slide6' class='slide'><h1>Security: Scan Gate + PCI Alignment</h1><p>ECR scan-on-push. Critical CVE blocks deploy (12 CVEs blocked, evidence logged). awsvpc per-task isolation. PCI-DSS L1 maintained.</p></div><div id='slide7' class='slide'><h1>Technical Debt & Remediation Timeline</h1><p>Phase 1 (complete): Containerize + serverless backbone. Phase 2: Service Connect + EventBridge Pipes + Karpenter.</p></div><div id='slide8' class='slide'><h1>Phase 2: What We Build Next</h1><p>ECS Service Connect (east-west, no ALB hop). EventBridge Pipes (eliminate Lambda glue). Karpenter (30-60% compute savings). CloudWatch Composite Alarms.</p></div><div style='position:fixed;bottom:20px;left:50%;transform:translateX(-50%)'><button onclick='prev()'>← Prev</button><button onclick='next()'>Next →</button></div><script>let cur=1;const total=8;function show(n){document.querySelectorAll('.slide').forEach(s=>s.classList.remove('active'));document.getElementById('slide'+n).classList.add('active')}function next(){if(cur<total){cur++;show(cur)}}function prev(){if(cur>1){cur--;show(cur)}}</script></body></html>
Clock stops. All validation passes. Presentation has real numbers in all 8 slides. Git history shows start-to-finish in 90 minutes or less.
Secret mission
Service Connect + EventBridge Pipes
The board loved the Phase 2 slide. Now prove it works. Add ECS Service Connect for east-west mesh discovery (eliminating the ALB hop between payment service and webhook dispatcher) and wire an EventBridge Pipe from SQS through Lambda enrichment to Step Functions. One Lambda eliminated. Latency reduced. "Do more with less" in its purest form.
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project provisions paid AWS resources (EKS control plane, NAT Gateway, ALB) that accrue hourly charges.
Cost warning
Leaving your infrastructure running costs approximately $0.19/hr ($4.50/day). The EKS control plane alone is $0.10/hr. The NAT Gateway adds $0.045/hr. If you are not actively using these resources, destroy them now to avoid unexpected charges on your AWS bill.
Resources you used:
- AWS cloud infrastructure (ECR, VPC, NAT Gateway, ALB, ECS cluster, ECS Fargate service, EKS cluster, CodeDeploy application).
- Serverless backbone (API Gateway REST API, WAF WebACL, Lambda function, Step Functions Express, EventBridge bus, SQS queue, DynamoDB table).
- Secret Mission resources (ECS Service Connect namespace, EventBridge Pipes).
- Local project files (infra/, lambda/, tests/, docs/, presentation/ directories).
- Installed tools (Terraform, AWS CLI, k6, Docker Desktop).
Keep everything running
No action needed. Choose this if you are still actively building or plan to return within the next hour.
- Your AWS resources continue to accrue charges (approximately $0.19/hr).
- All Terraform state is preserved in your local infra/ directory.
- Terraform, AWS CLI, k6, and Docker Desktop remain installed on your machine.
Pause - I'll come back to this later
Scale down running compute to reduce costs while keeping Terraform state intact for a quick rebuild.
- Scale ECS services to zero tasks by running:
- Scale the EKS node group to zero nodes in your Terraform variables, then run terraform apply.
- Note: The EKS control plane ($0.10/hr) and NAT Gateway ($0.045/hr) cannot be paused without destroying them. If you are away for more than a few hours, use the Delete tab instead.
- Your local project files and Terraform state remain untouched for when you return.
Rebuilding is fast
Since everything is Terraform-managed, you can always terraform destroy and terraform apply again later. The entire platform rebuilds in about 15 minutes.
Delete - I don't want to use this again
Remove all project resources completely. One command destroys everything in AWS. Then clean up local files.
- Navigate to your infra/ directory and destroy all AWS resources by running:
cd ~/Desktop/infra
terraform destroy -auto-approve
This removes every AWS resource tracked in Terraform state. ECR, VPC, NAT Gateway, ALB, ECS, EKS, Lambda, API Gateway, Step Functions, EventBridge, SQS, DynamoDB, CodeDeploy, Service Connect, and EventBridge Pipes are all destroyed.
- Verify nothing remains by running:
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=compute-platform
This should return an empty ResourceTagMappingList. If any resources appear, run terraform destroy -auto-approve again.
- Confirm Terraform state is empty by running:
terraform state list
This should return nothing.
- Delete the local project files by running:
cd ~/Desktop
rm -rf infra lambda tests docs presentation
- To uninstall Terraform, delete the terraform binary from the directory where you placed it on your PATH.
- To uninstall k6, open Settings, select Apps, then Apps & features. Find k6 and click Uninstall.
- To uninstall AWS CLI v2, open Settings, select Apps, then Apps & features. Find AWS Command Line Interface v2 and click Uninstall.
- To uninstall Docker Desktop, open Settings, select Apps, then Apps & features. Find Docker Desktop and click Uninstall.
Zero orphans confirmed
Because every AWS resource was Terraform-managed and tagged with Project=compute-platform, the tag API query is your proof that nothing lingers. No ghost NAT Gateway at $32/month. No EKS control plane at $72/month.
That's a Wrap!
That's a Wrap!
You just operated as Principal Solutions Architect at Meridian Payments. 90 minutes. 5 parallel agents. Under $2. A complete compute platform built, chaos-validated, and presented to the board.
You've learned how to:
- Deploy the same Docker image to three compute targets (ECS-on-EC2, ECS-on-Fargate, and EKS) behind a weighted ALB, then chaos-test recovery in under 30 seconds to prove your SLA.
- Build a serverless event-driven backbone (REST API + Lambda SnapStart + Express Step Functions + SQS DLQ) with a CodeDeploy blue/green pipeline and ECR scan gate that blocks Critical CVEs from production.
- Produce a complete leadership package for a board audience: 6 ADRs with business consequences, a FinOps report mapping every resource to cost, and an 8-slide HTML presentation proving architecture, reliability, and security.
- Secret Mission: Enable ECS Service Connect for east-west mesh traffic and wire EventBridge Pipes to eliminate Lambda glue, positioning Phase 2 for 2x throughput at 30% lower cost.
Ready to quiz yourself?