Get Promoted: Build AWS DR Worth $1M

Design multi-region DR, prove it in a gameday, and pitch it to the C-suite.

Introduction

30 Second Summary

When an online store goes down during peak shopping season, every minute of downtime translates directly into lost revenue and lost customer trust. The companies that recover fastest are the ones that planned and tested their recovery strategy in advance.

In this project, you will design a multi-region disaster recovery architecture on AWS, deploy it locally using Terraform and LocalStack at $0 cost, prove it works with gameday chaos testing, and produce the executive deliverables package that justifies the spend to leadership.

What You'll Build

A complete tiered DR architecture covering 4 recovery strategies, validated with chaos testing and backed by business-impact data that maps to 12 SAP-C02 task statements across all 4 exam domains.

By the end of this project, you'll have:

  • A complete Terraform codebase defining a 3-tier HA architecture plus 4 DR-tier workspaces, fully deployed and validated against LocalStack at $0 cost with zero orphan resources.
  • A gameday harness that empirically proves each DR tier meets its declared RTO/RPO targets, with measured results logged as structured JSON data.
  • A TOGAF-aligned Architecture Deliverables Package including an ADR, Business KPI Dashboard, Cost-Avoidance Report, and Mermaid diagrams ready for executive review.
  • Secret Mission: Produce a full promotion packet with cost-per-nine analysis, ransomware recovery drill, and an Executive Architecture Brief proving you turned a $4.7M annual risk into a $74K insurance policy.

Are there any prerequisites?

You need Docker Desktop installed and running on your machine. Basic familiarity with Terraform HCL syntax is helpful but not required.

Everything runs locally against LocalStack's free tier. No AWS account is needed.

Set Up Your War Room

Last Black Friday, a us-east-1 regional outage took MegaCart's checkout service offline for 47 minutes. That outage cost $4.7M in lost revenue and dropped NPS by 12 points. The CEO has mandated zero recurrence, and your Division Leader needs a complete architecture package on her desk by end of week.

Before you can design failover tiers, run gamedays, or produce executive deliverables, you need your local war room ready. This step installs Terraform for infrastructure-as-code, LocalStack for $0 AWS emulation inside Docker, and launches your 5 parallel Cursor agents to handle the HCL boilerplate while you focus on architecture decisions.

In this step, get ready to:
  • Install and verify Terraform 1.15.x.
  • Start LocalStack in Docker for $0 AWS emulation.
  • Set up Cursor IDE with 5 parallel Composer agents.
Install and verify Terraform

Terraform defines your entire DR infrastructure as code. Every resource lives in state, and a single destroy command removes everything. You need version 1.15.x or higher for this project.

  • Install Terraform using the platform-appropriate package manager:

macOS

  • Install Terraform via Homebrew by running these commands:
brew tap hashicorp/tap
brew install hashicorp/tap/terraform

What do these commands do?

The first command adds HashiCorp's official Homebrew tap (their package repository). The second installs the latest Terraform binary from that tap.

Windows

  • Install Terraform via winget by running this command:
winget install Hashicorp.Terraform

What does this command do?

This downloads and installs the official Terraform binary from HashiCorp's winget package. After installation, you may need to restart your terminal for the terraform command to be available on your PATH.

  • Verify your installation by running this command:
terraform --version

✔️ I see version 1.15 or higher

You should see output like Terraform v1.15.3. Terraform is ready to go.

ⓧ I see an older version

If your output shows a version below 1.15, you need to upgrade.

  • On macOS, upgrade by running:
  • On Windows, re-run the winget install command above. It will upgrade to the latest version.
brew update && brew upgrade hashicorp/tap/terraform
  • Run terraform --version again to confirm you now see 1.15 or higher.

ⓧ Command not found

This means Terraform is not installed or not on your PATH.

  • On macOS, ensure Homebrew is installed first. Visit https://brew.sh and follow the install instructions. Then run the two brew commands from the macOS tab above.
  • On Windows, open a new terminal window (the PATH update may not apply to existing terminals). If it still fails, run the winget install command from the Windows tab above.
  • Run terraform --version again to confirm it works.
Start LocalStack in Docker

LocalStack emulates AWS services locally inside a Docker container. Every AWS API call in this project hits localhost:4566 instead of real AWS. The --rm flag means stopping the container erases everything. Zero orphan resources. Zero cost.

  • Verify Docker is running by running this command:
docker --version

✔️ I see a version number

Docker is installed and ready. Move on to the next instruction below.

ⓧ Command not found

Docker Desktop is a prerequisite for this project. Install it from https://docker.com/products/docker-desktop and launch it before continuing.

  • After installing, open Docker Desktop and wait for it to finish starting (the whale icon in your system tray stops animating).
  • Run docker --version again to confirm.
  • Sign up for a free LocalStack auth token at https://localstack.cloud. This takes about 30 seconds and is required since March 2026.
  • Start the LocalStack container by running this command:
docker run --rm -d --name megacart-lab -p 4566:4566 -e LOCALSTACK_AUTH_TOKEN=your-free-token-here localstack/localstack:2026.04.0

What does this command do?

  • --rm removes the container when it stops. No orphan resources.
  • -d runs the container in the background.
  • --name megacart-lab gives it a memorable name you can reference later.
  • -p 4566:4566 exposes the LocalStack API on port 4566.
  • -e LOCALSTACK_AUTH_TOKEN=... passes your free-tier auth token.
  • localstack/localstack:2026.04.0 pins the exact LocalStack version for reproducibility.
  • Verify the container is running:
docker ps --filter name=megacart-lab

You should see a row with megacart-lab in the NAMES column and Up in the STATUS column.

Container not showing up?

  • Make sure Docker Desktop is fully started (not just installed). The Docker daemon must be running.
  • Check that you replaced your-free-token-here with the actual auth token from your localstack.cloud account.
  • Run docker logs megacart-lab to see if there are startup errors.

help me troubleshoot my LocalStack container not starting

Install Python tools and launch Cursor agents

The tflocal wrapper transparently redirects all Terraform commands to LocalStack's endpoint on localhost:4566. Locust generates load traffic for your gameday chaos tests later.

  • Verify Python 3.12+ is available by running this command:
python3 --version

✔️ I see version 3.12 or higher

Python is ready. Continue to the next instruction.

ⓧ I see an older version

You need Python 3.12 or higher. macOS ships with an older version by default that will not work.

  • On macOS, install via Homebrew by running:
brew install python@3.12
  • After installing, use python3.12 --version to confirm. If python3 --version still shows the old version, use python3.12 and pip3.12 for the remaining commands in this project.
  • On Windows, download Python 3.12+ from https://python.org/downloads and run the installer. Check the box that says Add Python to PATH during install.

ⓧ Command not found

Python is not installed on your system.

  • On macOS, install via Homebrew by running:
brew install python@3.12
  • On Windows, download Python 3.12+ from https://python.org/downloads and run the installer. Check the box that says Add Python to PATH during install.
  • After installing, run python3 --version (macOS) or python --version (Windows) to confirm.
  • Install the tflocal wrapper and Locust by running this command:
pip install terraform-local locust

What does this install?

terraform-local provides the tflocal command, which is a thin wrapper around terraform. It automatically sets all AWS provider endpoints to localhost:4566 so every Terraform command targets your local LocalStack container instead of real AWS.

locust is a Python-based load testing tool. You will use it in Step 4 to generate 500 concurrent users of traffic for your gameday chaos tests.

  • Verify both tools installed correctly:
tflocal --version
locust --version

You should see version numbers for both tools.

pip command not found?

  • Try pip3 instead of pip. On macOS, pip often does not exist but pip3 does.
  • If you installed Python 3.12 via Homebrew, use pip3.12 install terraform-local locust instead.

help me fix pip not found on my system

  • Create your project directory and move into it by running these commands:
cd ~/Desktop
mkdir megacart-dr
cd megacart-dr
  • Confirm the directory exists by running:
ls

The directory should be empty. This is where all your Terraform modules, gameday scripts, and executive deliverables will live.

  • Open Cursor IDE.
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Cursor and press Enter to open it.
  • Open the megacart-dr folder in Cursor. Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the megacart-dr folder, and click Open.

Now launch your 5 parallel Composer agents. Each agent handles one domain of work while you focus on principal-level architecture decisions.

  • Open Composer by pressing Cmd+I (macOS) or Ctrl+I (Windows).
  • Launch 5 separate agent sessions in Composer with the following names and responsibilities:
  • Agent-A: Primary Region IaC (VPC, ALB, ASG, Aurora, Redis).
  • Agent-B: 4 DR-tier workspaces (backup-restore, pilot-light, warm-standby, active-active).
  • Agent-C: Route 53 failover + health checks.
  • Agent-D: Locust load + gameday chaos scripts.
  • Agent-E: TOGAF deliverables + KPI dashboard + Mermaid diagrams.

Why 5 parallel agents?

Your job as the Principal Architect is not to write HCL boilerplate. It is to make the design decisions, set the constraints, and validate the output. The agents handle the implementation. You handle the architecture.

Each agent has a single domain so you can direct them independently without context bleed. When you tell Agent-A to build the primary VPC, Agent-B can simultaneously scaffold the DR workspaces.

💡 Keep agents on short leashes

Give each agent one task at a time. Review its output before giving the next instruction. Agents that run too far ahead without review tend to make assumptions that conflict with your architecture decisions.

Your war room is operational. Terraform talks to LocalStack via tflocal, Docker contains the blast radius to a single container, and your 5 agents are ready to take direction. Next up, you will direct Agent-A to build the full primary region stack while you make the principal-level design calls that separate a senior engineer from a distinguished architect.

Build the Primary Region Architecture

Your war room is set up. Now it is time to build the infrastructure that keeps MegaCart's checkout alive when an AZ or an entire region fails.

In this step, you will direct Agent-A to build the primary region stack as Terraform HCL. While the agent writes infrastructure code, you will make the principal-level design decisions that justify each choice and document them in an Architecture Decision Record.

In this step, get ready to:
  • Direct Agent-A to build a full 3-tier primary region stack in Terraform.
  • Document 5 principal-level design decisions in your ADR with SAP-C02 domain annotations.
  • Deploy the stack to LocalStack and validate every resource exists.
Direct Agent-A to build the primary stack

The primary region stack is a single Terraform root module. One state file means one tflocal destroy command removes everything. Zero orphan risk.

  • Create the directory for your primary module by running these commands:
mkdir -p megacart-dr/primary
cd megacart-dr/primary
  • In Cursor Composer, send this prompt to Agent-A to generate the primary stack infrastructure:
Agent-A: Build a single Terraform root module at megacart-dr/primary/main.tf
that defines the full primary region stack for MegaCart (us-east-1).

Requirements:
- AWS provider pinned to ~> 6.45, region us-east-1
- VPC: CIDR 10.0.0.0/16, 3 AZs (us-east-1a/b/c), public subnets
  (10.0.0.0/24, 10.0.1.0/24, 10.0.2.0/24), private subnets
  (10.0.10.0/24, 10.0.11.0/24, 10.0.12.0/24), Internet Gateway,
  NAT Gateway (in public subnet a), route tables for public and private
- ALB: named "megacart-primary-alb", application type, in public subnets,
  target group named "megacart-app-tg" with /health HTTP health check,
  HTTP listener forwarding to target group
- ASG: named "megacart-app-asg", mixed_instances_policy with
  on_demand_base_capacity = 3, spot above baseline,
  capacity-optimized allocation, overrides for t3.medium and t3a.medium,
  desired 6, min 3, max 12, in private subnets, attached to ALB target group
- Aurora MySQL: cluster identifier "megacart-aurora", engine aurora-mysql,
  Multi-AZ with writer in us-east-1a and 1 reader in us-east-1b,
  db.r6g.large instances, private subnet group
- ElastiCache Redis: replication group id "megacart-redis", 3 nodes,
  cache.r6g.large, automatic_failover_enabled = true,
  multi_az_enabled = true, private subnet group
- Tag everything with Name = megacart-*
- Include inline comments explaining each resource's purpose

What does this prompt do?

This prompt gives Agent-A the complete specification for a 3-tier architecture. The agent generates all the HCL in one file, creating a single Terraform state that can be destroyed with one command.

You specified exact resource names (megacart-aurora, megacart-redis) so that later steps can reference them consistently across workspaces and gameday scripts.

  • Wait for Agent-A to generate main.tf.
  • Review the output in Cursor. Confirm Agent-A created all five resource groups: VPC, ALB, ASG, Aurora, and Redis.
  • Accept Agent-A's changes to write the file to disk.

What should I see in main.tf?

Your main.tf should contain these key resources with the exact identifiers:

  • aws_vpc.primary with CIDR 10.0.0.0/16 and 3 public + 3 private subnets.
  • aws_lb.primary named megacart-primary-alb with a /health target group.
  • aws_autoscaling_group.app with mixed_instances_policy (t3.medium on-demand + t3a.medium spot).
  • aws_rds_cluster.primary with cluster_identifier = "megacart-aurora" and two instances (writer + reader).
  • aws_elasticache_replication_group.primary with replication_group_id = "megacart-redis" and automatic_failover_enabled = true.

✔️ Awesome, I've got everything!

Great. Make sure you have saved main.tf inside megacart-dr/primary/.

ⓧ I'd like to double check the full code

Here is the full main.tf your Agent-A should have generated. Compare yours against it:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.45"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# --- VPC: 3 AZs, public/private subnets, NAT + IGW ---

resource "aws_vpc" "primary" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name = "megacart-primary-vpc"
  }
}

resource "aws_internet_gateway" "primary" {
  vpc_id = aws_vpc.primary.id

  tags = {
    Name = "megacart-primary-igw"
  }
}

# Public subnets across 3 AZs
resource "aws_subnet" "public" {
  count                   = 3
  vpc_id                  = aws_vpc.primary.id
  cidr_block              = "10.0.${count.index}.0/24"
  availability_zone       = "us-east-1${["a", "b", "c"][count.index]}"
  map_public_ip_on_launch = true

  tags = {
    Name = "megacart-public-${["a", "b", "c"][count.index]}"
  }
}

# Private subnets across 3 AZs
resource "aws_subnet" "private" {
  count             = 3
  vpc_id            = aws_vpc.primary.id
  cidr_block        = "10.0.${count.index + 10}.0/24"
  availability_zone = "us-east-1${["a", "b", "c"][count.index]}"

  tags = {
    Name = "megacart-private-${["a", "b", "c"][count.index]}"
  }
}

# NAT Gateway for private subnet egress
resource "aws_eip" "nat" {
  domain = "vpc"
}

resource "aws_nat_gateway" "primary" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public[0].id

  tags = {
    Name = "megacart-primary-nat"
  }
}

# Route tables
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.primary.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.primary.id
  }

  tags = {
    Name = "megacart-public-rt"
  }
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.primary.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.primary.id
  }

  tags = {
    Name = "megacart-private-rt"
  }
}

resource "aws_route_table_association" "public" {
  count          = 3
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "private" {
  count          = 3
  subnet_id      = aws_subnet.private[count.index].id
  route_table_id = aws_route_table.private.id
}

# --- ALB with /health target group ---

resource "aws_security_group" "alb" {
  name   = "megacart-alb-sg"
  vpc_id = aws_vpc.primary.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_lb" "primary" {
  name               = "megacart-primary-alb"
  internal           = false
  load_balancer_type = "application"
  subnets            = aws_subnet.public[*].id
  security_groups    = [aws_security_group.alb.id]

  tags = {
    Name = "megacart-primary-alb"
  }
}

resource "aws_lb_target_group" "app" {
  name     = "megacart-app-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.primary.id

  health_check {
    path                = "/health"
    protocol            = "HTTP"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    timeout             = 5
    interval            = 10
  }
}

resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.primary.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}

# --- ASG with mixed-instance policy ---

resource "aws_security_group" "app" {
  name   = "megacart-app-sg"
  vpc_id = aws_vpc.primary.id

  ingress {
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_launch_template" "app" {
  name_prefix   = "megacart-app-"
  image_id      = "ami-0123456789abcdef0"
  instance_type = "t3.medium"

  network_interfaces {
    associate_public_ip_address = false
    security_groups             = [aws_security_group.app.id]
  }

  tags = {
    Name = "megacart-app-lt"
  }
}

resource "aws_autoscaling_group" "app" {
  name                = "megacart-app-asg"
  desired_capacity    = 6
  max_size            = 12
  min_size            = 3
  vpc_zone_identifier = aws_subnet.private[*].id
  target_group_arns   = [aws_lb_target_group.app.arn]

  mixed_instances_policy {
    instances_distribution {
      on_demand_base_capacity                  = 3
      on_demand_percentage_above_base_capacity = 0
      spot_allocation_strategy                 = "capacity-optimized"
    }

    launch_template {
      launch_template_specification {
        launch_template_id = aws_launch_template.app.id
        version            = "$Latest"
      }

      override {
        instance_type = "t3.medium"
      }

      override {
        instance_type = "t3a.medium"
      }
    }
  }

  tag {
    key                 = "Name"
    value               = "megacart-app"
    propagate_at_launch = true
  }
}

# --- Aurora MySQL Multi-AZ cluster ---

resource "aws_db_subnet_group" "aurora" {
  name       = "megacart-aurora-subnet-group"
  subnet_ids = aws_subnet.private[*].id
}

resource "aws_rds_cluster" "primary" {
  cluster_identifier   = "megacart-aurora"
  engine               = "aurora-mysql"
  engine_version       = "8.0.mysql_aurora.3.04.0"
  master_username      = "admin"
  master_password      = "megacart-db-password"
  db_subnet_group_name = aws_db_subnet_group.aurora.name

  tags = {
    Name = "megacart-aurora"
  }
}

resource "aws_rds_cluster_instance" "writer" {
  identifier         = "megacart-aurora-writer"
  cluster_identifier = aws_rds_cluster.primary.id
  instance_class     = "db.r6g.large"
  engine             = "aurora-mysql"
  availability_zone  = "us-east-1a"
}

resource "aws_rds_cluster_instance" "reader" {
  identifier         = "megacart-aurora-reader"
  cluster_identifier = aws_rds_cluster.primary.id
  instance_class     = "db.r6g.large"
  engine             = "aurora-mysql"
  availability_zone  = "us-east-1b"
}

# --- ElastiCache Redis replication group (Multi-AZ, automatic failover) ---

resource "aws_elasticache_subnet_group" "redis" {
  name       = "megacart-redis-subnet-group"
  subnet_ids = aws_subnet.private[*].id
}

resource "aws_elasticache_replication_group" "primary" {
  replication_group_id       = "megacart-redis"
  description                = "MegaCart session store with Multi-AZ failover"
  engine                     = "redis"
  node_type                  = "cache.r6g.large"
  num_cache_clusters         = 3
  automatic_failover_enabled = true
  multi_az_enabled           = true
  subnet_group_name          = aws_elasticache_subnet_group.redis.name

  tags = {
    Name = "megacart-redis"
  }
}

Agent-A produced something different?

If Agent-A used different resource names or split across multiple files, re-prompt with the exact identifiers from above. The gameday scripts in Step 4 depend on these names matching.

If Agent-A added extra resources (like CloudWatch alarms or WAF rules), that is fine to keep. The critical check is that the five core resource groups exist with the correct names.

help me fix Agent-A's output to match the required resource names

Document your principal-level design decisions

While Agent-A generates HCL, your job is to make and document the architectural trade-off decisions. These are the exact questions SAP-C02 tests and principal-level interviewers probe.

  • Create the docs directory and ADR file by running:
mkdir -p megacart-dr/docs
  • Create megacart-dr/docs/ADR.md and paste the following Architecture Decision Record:
# Architecture Decision Record: MegaCart Primary Region Design

## Context

Last Black Friday, a us-east-1 regional outage took the checkout service
down for 47 minutes. Revenue loss: $4.7M. NPS dropped 12 points.
The CEO mandated zero recurrence. This ADR documents the principal-level
design decisions for the primary region architecture.

## Decision 1: ALB not NLB (SAP-C02 Domain D2.4)

**Question**: Why Application Load Balancer instead of Network Load Balancer?

**Decision**: ALB. MegaCart's e-commerce app requires HTTP path-based routing
(/checkout vs /catalog vs /health). NLB operates at Layer 4 (TCP) only and
cannot route by URL path. ALB provides Layer 7 routing, sticky sessions,
and native integration with WAF for bot protection during flash sales.

**Consequence**: Slightly higher latency (~1-2ms) vs NLB, acceptable for
HTTP workloads. Gained: path routing, health check granularity, WAF support.

## Decision 2: 3 AZs not 2 (SAP-C02 Domain D1.3)

**Question**: Why deploy across 3 Availability Zones instead of 2?

**Decision**: 3 AZs. Losing 1 of 2 AZs = 50% capacity gone instantly.
Losing 1 of 3 AZs = 33% capacity reduction. The math changes your scaling
baseline. With 2 AZs you must over-provision each AZ to handle full load
alone. With 3 AZs, each AZ runs at 33% headroom.

**Consequence**: One additional subnet per tier (marginal cost). Gained:
fault tolerance that does not require 2x over-provisioning.

## Decision 3: Redis not Memcached (SAP-C02 Domain D1.3)

**Question**: Why ElastiCache Redis instead of Memcached for session store?

**Decision**: Redis. Memcached has zero replication capability. A single
node failure loses all session data (all active shopping carts gone).
Redis supports Multi-AZ replication groups with automatic failover.
A node failure promotes a replica in seconds with no data loss.

**Consequence**: Redis costs slightly more per node. Gained: session
persistence across failures, automatic failover, zero cart abandonment
from infrastructure issues.

## Decision 4: Aurora not RDS MySQL (SAP-C02 Domain D2.2)

**Question**: Why Aurora MySQL instead of standard RDS MySQL Multi-AZ?

**Decision**: Aurora. Aurora failover completes in under 30 seconds.
Standard RDS Multi-AZ can take 60-120 seconds. At $100K/minute downtime
cost during peak, that 30-90 second difference = $50K-$150K in additional
revenue exposure per failover event.

**Consequence**: ~20% premium over standard RDS. Gained: sub-30-second
failover, 6-way storage replication, read replicas that share storage
(no replication lag for reads).

## Decision 5: Mixed-Instance ASG (SAP-C02 Domains D1.5 + D2.4)

**Question**: Why mixed on-demand + spot instances in the Auto Scaling Group?

**Decision**: Mixed policy with on_demand_base_capacity = 3 and spot above
baseline. Spot saves 60-70% on compute above the guaranteed minimum.
If spot gets reclaimed during a flash sale, the 3 on-demand instances
absorb traffic while replacement spot instances launch (typically <2 min).

**Consequence**: Spot interruption risk above baseline (mitigated by
capacity-optimized allocation strategy). Gained: $50K+ annual compute
savings while maintaining a guaranteed performance floor.

Why document decisions, not just build?

A senior engineer builds the infrastructure. A principal architect documents WHY each choice was made, what alternatives were rejected, and what the business trade-offs are. This ADR is your evidence that you made deliberate, revenue-justified decisions.

Each decision maps to a specific SAP-C02 domain. The exam tests these exact trade-offs in scenario questions. Your ADR doubles as study material.

  • Save ADR.md.
  • Confirm the file exists by running:
ls megacart-dr/docs/

You should see ADR.md in the output.

Deploy and validate the primary stack

With main.tf ready, you will deploy the entire primary stack to LocalStack using tflocal. This wrapper routes all Terraform API calls to your local container on port 4566.

  • Navigate into the primary module directory by running:
cd megacart-dr/primary
  • Initialize Terraform and deploy all resources by running:
tflocal init && tflocal apply -auto-approve

What does this command do?

tflocal init downloads the AWS provider plugin and sets up the .terraform/ directory. tflocal apply -auto-approve creates all resources against LocalStack without prompting for confirmation.

Since LocalStack emulates AWS locally, no real cloud resources are created. Everything lives inside the megacart-lab Docker container you started in Step 1.

You should see Terraform output showing resources being created, ending with a summary like Apply complete! Resources: X added, 0 changed, 0 destroyed.

Seeing provider download errors?

Make sure your megacart-lab Docker container is still running. Check with docker ps. If it stopped, restart it using the same docker run command from Step 1.

If you see a region error, confirm your main.tf has region = "us-east-1" explicitly set in the provider block.

help me debug my tflocal apply error

  • Validate your ALB exists by running:
awslocal elbv2 describe-load-balancers

You should see JSON output containing megacart-primary-alb in the LoadBalancerName field.

  • Validate your Aurora cluster exists by running:
awslocal rds describe-db-clusters

You should see megacart-aurora in the DBClusterIdentifier field with engine set to aurora-mysql.

  • Validate your Redis replication group exists by running:
awslocal elasticache describe-replication-groups

You should see megacart-redis in the ReplicationGroupId field with AutomaticFailover showing enabled.

Resources missing from awslocal output?

Check the Terraform apply output for errors. If a resource failed, Terraform still deploys the others. Re-run tflocal apply -auto-approve to retry the failed resources.

If awslocal returns an empty list, verify that Terraform used the LocalStack endpoint by checking for localhost:4566 in the init output. If you see real AWS endpoints, you may have run terraform instead of tflocal.

help me figure out why my resources are not showing up in awslocal

Your primary region is deployed and validated. All five resource groups are live in LocalStack, your ADR documents the reasoning behind each architectural choice, and you have measured proof that everything exists. Next up, you will design the four DR tiers and assign each workload to the tier that matches its revenue impact.

Design the DR Tiers and Business Case

Your primary region is live and validated against LocalStack. Now comes the decision that separates a senior engineer from a Distinguished Architect: assigning each MegaCart workload to the right DR tier based on its revenue impact, not its technical complexity.

Finance rejected the blanket active-active proposal at $280K/year. Your job is to prove that a tiered strategy at $74K/year covers 100% of revenue-critical paths while accepting calculated risk on non-critical workloads.

In this step, get ready to:
  • Direct Agent-B to scaffold all 4 DR workspace modules.
  • Assign each MegaCart workload to a DR tier with revenue-justified business logic.
  • Calculate the financial business case, deploy all workspaces, and validate.
Direct Agent-B to scaffold the DR workspaces

Each DR tier lives in its own independent Terraform root module with its own state file. This means one tflocal destroy per workspace removes everything. Zero cross-workspace orphan risk.

  • Create the workspace directory structure by running these commands:
mkdir -p workspaces/backup-restore
mkdir -p workspaces/pilot-light
mkdir -p workspaces/warm-standby
mkdir -p workspaces/active-active
  • Confirm the folders exist by running:
ls workspaces/

You should see active-active, backup-restore, pilot-light, and warm-standby listed.

Now direct Agent-B in Cursor Composer to build all 4 workspace modules in parallel. Each module targets a different DR tier with different infrastructure requirements.

  • In Cursor Composer, select Agent-B and enter this prompt:
Build 4 independent Terraform root modules under workspaces/. Each is a complete DR tier for MegaCart targeting eu-west-1 as the secondary region. Use the LocalStack AWS provider (endpoint localhost:4566).

1. workspaces/backup-restore/main.tf
   - Analytics/reporting workload
   - AWS Backup vault with backup plan (daily snapshots, 1-hour RPO)
   - S3 bucket for cross-region backup replication
   - RTO target: 24 hours

2. workspaces/pilot-light/main.tf
   - User profiles workload
   - Aurora MySQL read replica in eu-west-1 (pilot-light, minimal instance)
   - No running compute, just the database replica ready to promote
   - RTO target: 30 minutes

3. workspaces/warm-standby/main.tf
   - Product catalog workload
   - Scaled-down ASG (1 instance) behind an ALB in eu-west-1
   - Aurora read replica actively replicating
   - RTO target: 5 minutes

4. workspaces/active-active/main.tf
   - Checkout flow workload (revenue-critical)
   - Full secondary stack: ALB + ASG (same size as primary) in eu-west-1
   - Aurora Global Database with write forwarding
   - DynamoDB Global Tables for session state
   - Route 53 weighted routing (50/50 traffic split)
   - RTO target: <1 minute, RPO: 0

Each module must have its own provider block, its own terraform block, and produce a self-contained state file. Add inline comments explaining the DR tier purpose and cost justification.

What is Agent-B doing?

Agent-B writes the HCL for all four workspaces simultaneously. Each workspace is a separate Terraform root module with its own state file, meaning you can deploy and destroy each tier independently.

Your role is not to write Terraform. Your role is to make the architecture decisions that determine WHAT gets built and WHY. The agent handles the syntax. You handle the strategy.

  • Wait for Agent-B to generate all four main.tf files.
  • Review each file in the Cursor editor to confirm it targets eu-west-1 and includes the correct resources for its tier.
  • Accept the agent's changes when satisfied.

Agent-B producing incorrect resources?

If the active-active workspace is missing DynamoDB Global Tables or the backup-restore workspace includes running compute, re-prompt Agent-B with the specific correction. Agents sometimes over-provision lower tiers.

help me fix Agent-B's DR workspace output

Assign workloads to DR tiers and document decisions

This is the decision that SAP-C02 tests and principal interviews probe. You must justify WHY each workload lands in its tier using revenue impact, not gut feeling.

What are the 4 DR tiers?

AWS defines four disaster recovery strategies with increasing cost and decreasing RTO/RPO.

  • Backup & Restore: cheapest, longest recovery. Restore from snapshots.
  • Pilot Light: core database replicates, compute starts on failover.
  • Warm Standby: scaled-down copy runs in secondary, scales up on failover.
  • Active-Active: full capacity in both regions simultaneously. Near-zero RTO.

Here are the four workload-to-tier assignments you need to make. Each justification ties directly to MegaCart's $100K/minute peak-traffic revenue.

  • Checkout flow = active-active (RTO <1min, RPO 0, $51K/yr). At $100K/min during peak, even 5 minutes of checkout downtime on Black Friday = $500K. Active-active pays for itself in 30 seconds of protected uptime.
  • Product catalog = warm-standby (RTO 5min, RPO seconds, $12K/yr). Users see stale products temporarily but cannot purchase. Annoying, not fatal. Five-minute recovery is acceptable.
  • User profiles = pilot-light (RTO 30min, RPO seconds, $5K/yr). Sessions rebuild from the database. Users log in again. No revenue loss, minor friction.
  • Analytics/reporting = backup-restore (RTO 24h, RPO 1h, $500/yr). Internal-only. Finance can wait a day for dashboards. No customer impact whatsoever.

Why not active-active for everything?

Blanket active-active across all workloads costs $280K/year. Tiered approach costs $74K/year. The $206K difference buys zero additional revenue protection because active-active on analytics dashboards protects nothing customers care about.

This is the principal-level insight: spend DR budget where revenue flows, accept calculated risk everywhere else. SAP-C02 Domain D1.5 tests this exact trade-off.

Now document these decisions in your ADR so the promotion committee can trace your reasoning.

  • Open megacart-dr/docs/ADR.md in Cursor.
  • Add the following workload-to-tier assignment section below your existing 5 design decisions:
## Decision 6: Workload-to-Tier Assignment (D2.2, D2.6, D1.5)

### Context
Finance rejected blanket active-active at $280K/yr. Board mandated zero
recurrence of revenue-impacting outages. We must protect revenue-critical
paths while demonstrating fiscal responsibility.

### Decision
Assign each workload to the LOWEST DR tier that meets its business-impact
threshold:

| Workload | DR Tier | RTO | RPO | Annual Cost | Justification |
|----------|---------|-----|-----|-------------|---------------|
| Checkout flow | Active-Active | <1 min | 0 | $51K | $100K/min revenue. 30s of uptime = payback. |
| Product catalog | Warm-Standby | 5 min | seconds | $12K | Stale products, no purchases. Annoying, not fatal. |
| User profiles | Pilot-Light | 30 min | seconds | $5K | Sessions rebuild. Users re-login. No revenue loss. |
| Analytics/reporting | Backup-Restore | 24 h | 1 h | $500 | Internal-only. Zero customer impact. |

### Consequences
- Total annual DR spend: $74K
- Revenue risk eliminated: $4.7M/year
- Savings vs blanket active-active: $206K/year
- Residual risk accepted: analytics may be unavailable up to 24h post-disaster

### SAP-C02 Domain Coverage
- D2.2: Business continuity design (tiered recovery per workload criticality)
- D2.6: Cost optimization for solution goals (right-tier = right-cost)
- D1.5: Cost optimization and visibility strategies (FinOps framing)

What does this ADR section do?

An Architecture Decision Record captures the context, decision, and consequences of each architectural choice. The SAP-C02 domain annotations map your decisions to exam domains, making this a study artifact as well as a promotion evidence piece.

  • Save ADR.md.
Write the business case and deploy all workspaces

The FinOps Cost-Avoidance Report is the document that gets budget approved. It frames DR spend as insurance, not cost.

  • Create megacart-dr/docs/CostAvoidanceReport.md with the following content:
# FinOps Cost-Avoidance Report
## MegaCart Disaster Recovery Investment

### Executive Summary
For every $1 we spend on DR, we protect $63 in revenue.

### Per-Tier Annual Cost Breakdown

| DR Tier | Workload | Annual Cost | Resources |
|---------|----------|-------------|-----------|
| Active-Active | Checkout flow | $51,000 | Aurora Global DB, DynamoDB Global Tables, full ASG in eu-west-1 |
| Warm-Standby | Product catalog | $12,000 | Scaled-down ASG (1 instance), Aurora reader replica |
| Pilot-Light | User profiles | $5,100 | Aurora reader replica (minimal instance), no compute |
| Backup-Restore | Analytics/reporting | $500 | S3 cross-region replication, daily snapshots |
| **TOTAL** | | **$74,000** | |

### Risk Eliminated
- Black Friday 2024 outage duration: 47 minutes
- Revenue lost: $4.7M
- Downtime cost rate: $100K/minute at peak
- Annual revenue-at-risk (single event): $4.7M

### ROI Calculation
- Annual DR investment: $74,000
- Annual risk eliminated: $4,700,000
- **ROI: 63x** ($4.7M / $74K)

### Comparison: Tiered vs Blanket Active-Active

| Approach | Annual Cost | Revenue Protected | Waste |
|----------|-------------|-------------------|-------|
| Blanket active-active (all workloads) | $280,000 | $4.7M | $206K on non-revenue paths |
| Tiered (this proposal) | $74,000 | $4.7M | $0 |
| **Net savings** | **$206,000/year** | Same protection | |

### Key Insight
Both approaches protect the same $4.7M in revenue. The tiered approach
achieves identical protection for revenue-critical paths at 26% of the cost
by accepting calculated, documented risk on internal-only workloads.

### Budget Approval Framing
- $74K/year = 1.6% of the risk it eliminates
- $206K/year saved vs the rejected blanket proposal
- 100% of customer-facing revenue paths protected
- Zero residual risk on checkout, catalog, or user profile flows

Why does this report matter for promotion?

The promotion committee does not read Terraform. They read business impact. This report translates your technical architecture into the language executives speak: risk eliminated, dollars saved, ROI proven.

The sentence "For every $1 we spend on DR, we protect $63 in revenue" is the one line your Division Leader will quote to the VP.

  • Save CostAvoidanceReport.md.

Now deploy all four workspaces to LocalStack. Each workspace gets its own tflocal init and tflocal apply.

  • Deploy the backup-restore workspace by running these commands:
cd workspaces/backup-restore
tflocal init
tflocal apply -auto-approve
  • Deploy the pilot-light workspace by running these commands:
cd ../pilot-light
tflocal init
tflocal apply -auto-approve
  • Deploy the warm-standby workspace by running these commands:
cd ../warm-standby
tflocal init
tflocal apply -auto-approve
  • Deploy the active-active workspace by running these commands:
cd ../active-active
tflocal init
tflocal apply -auto-approve

What does each deploy do?

Each tflocal init initializes the workspace's Terraform state independently. The tflocal apply -auto-approve creates the emulated AWS resources against LocalStack without prompting for confirmation.

Because each workspace has its own state file, you can destroy any single tier without affecting the others. This is the zero-orphan guarantee.

You should see Terraform output ending with Apply complete! for each workspace, followed by a resource count.

  • Verify the active-active workspace deployed correctly by running:
tflocal state list

You should see resources listed for the Aurora Global Database, DynamoDB Global Tables, ALB, and ASG in the active-active state.

Seeing an error during tflocal apply?

If you get an "Invalid AWS Region" error, make sure each workspace's main.tf has region = "eu-west-1" explicitly set in its provider block. The tflocal wrapper requires an explicit region.

If LocalStack returns connection refused, confirm the container is still running with docker ps. Look for megacart-lab in the output.

help me debug my tflocal apply failure

  • Navigate back to the project root by running:
cd ../..

All four DR tiers are live and your business case is documented. Next, you will prove these tiers actually work by running a full gameday with chaos injection and measured failover times.

Run the Gameday and Prove It Works

Your infrastructure is deployed. Your DR tiers are in state. Now comes the moment that separates architecture documents from architecture evidence.

A gameday is a controlled chaos exercise where you deliberately break your system and measure how long it takes to recover. The results become structured data that feeds your Compliance Report and KPI Dashboard. Without measured recovery times, your DR design is an untested theory. After this step, it becomes proven fact.

In this step, get ready to:
  • Build Route 53 failover records and ARC routing-control simulation scripts.
  • Create the Locust load test and chaos injection scripts.
  • Execute the full gameday and capture measured RTO results as structured JSON.
Build the failover plumbing with Agent-C

Before you can simulate a regional outage, you need the DNS failover records that route traffic between primary and secondary regions. You also need a way to flip that routing on command and a poller that detects when the endpoint recovers.

  • In Cursor Composer, instruct Agent-C to create megacart-dr/primary/route53.tf with the following requirements:
Agent-C prompt:

Create route53.tf in megacart-dr/primary/ with:
- A Route 53 hosted zone named "megacart.internal"
- A health check on the primary ALB endpoint at /health (HTTP, port 80, failure threshold 3, request interval 10s)
- A weighted alias record "checkout.megacart.internal" pointing to the primary ALB with weight 100, set identifier "primary", linked to the health check
- A weighted alias record "checkout.megacart.internal" pointing to the secondary ALB with weight 0, set identifier "secondary"
- Use evaluate_target_health = true on both alias targets

What does this Terraform accomplish?

The weighted routing policy with weights 100/0 means all traffic flows to the primary ALB under normal conditions. When you flip the weights to 0/100, traffic shifts to the secondary. This simulates what ARC routing controls do in production.

The health check polls /health every 10 seconds. Three consecutive failures trigger the health check to report unhealthy, which Route 53 uses for automated failover in production. In our gameday, we override manually via the ARC simulation script.

  • Review Agent-C's output. Verify it includes aws_route53_zone, aws_route53_health_check, and two aws_route53_record resources with weighted_routing_policy blocks.
  • Deploy the Route 53 resources by running:
cd ~/Desktop/megacart-dr/primary
tflocal apply -auto-approve
  • Confirm the hosted zone exists by running:
awslocal route53 list-hosted-zones

You should see a hosted zone with the name megacart.internal in the output.

Route 53 resources not showing up?

  • Confirm your LocalStack container is still running with docker ps. If it stopped, restart it.
  • Check that route53.tf is saved inside megacart-dr/primary/ alongside main.tf.
  • help me debug why tflocal apply is not creating my Route 53 resources in LocalStack
  • Next, instruct Agent-C to create the ARC simulation script at megacart-dr/scripts/arc-flip.sh:
Agent-C prompt:

Create megacart-dr/scripts/arc-flip.sh that:
- Gets the hosted zone ID using awslocal route53 list-hosted-zones
- Reads current weights for the "primary" and "secondary" set identifiers
- SAFETY CHECK: if both weights are 0, restore primary to 100 (prevents both-OFF state)
- Flips weights: if primary is 100, change to primary=0/secondary=100. If primary is 0, change to primary=100/secondary=0
- Uses awslocal route53 change-resource-record-sets with a JSON change batch to UPSERT both records
- Logs the flip timestamp
- Make it executable with chmod +x

Why the safety check matters

In production, ARC routing controls have a built-in safety rule: you cannot set both routing controls to OFF simultaneously. That would black-hole all traffic. The safety check in your simulation script enforces the same invariant.

This maps directly to SAP-C02 Domain 1.3 (designing reliable architectures) and Domain 3.4 (improving reliability through automation).

  • Instruct Agent-C to create the health-check poller at megacart-dr/scripts/health-poller.sh:
Agent-C prompt:

Create megacart-dr/scripts/health-poller.sh that:
- Accepts an endpoint URL as argument (defaults to http://localhost:4566/health)
- Polls the endpoint every 2 seconds with curl
- Tracks state transitions (UP/DOWN) and only logs when state changes
- Outputs each transition as JSON: {"timestamp": "ISO8601", "state": "UP|DOWN", "http_code": N}
- Make it executable with chmod +x
  • Review both scripts. Verify arc-flip.sh includes the both-OFF safety check and health-poller.sh logs state transitions with timestamps.
  • Make the scripts executable by running:
chmod +x ~/Desktop/megacart-dr/scripts/arc-flip.sh
chmod +x ~/Desktop/megacart-dr/scripts/health-poller.sh
  • Test the ARC flip script by running:
cd ~/Desktop/megacart-dr/scripts
bash arc-flip.sh

You should see output confirming the flip direction (e.g., "Flipping: PRIMARY 100->0, SECONDARY 0->100") and a completion timestamp.

Arc-flip script fails with a JSON error?

  • The change-resource-record-sets command requires the AliasTarget values to match your deployed ALB DNS names. Ask Agent-C to pull these dynamically from awslocal elbv2 describe-load-balancers.
  • help me fix the JSON change batch in my arc-flip.sh script for Route 53 weighted records
Build the chaos and load harness with Agent-D

With failover plumbing in place, you need two things: sustained load to simulate real traffic, and chaos scripts that inject failures while traffic is flowing. This combination proves your architecture recovers under realistic conditions.

  • Create the megacart-dr/gameday/ directory by running:
mkdir -p ~/Desktop/megacart-dr/gameday
  • Instruct Agent-D to create the Locust load test at megacart-dr/gameday/locustfile.py:
Agent-D prompt:

Create megacart-dr/gameday/locustfile.py with:
- Import HttpUser, task, and between from locust
- A MegaCartUser class inheriting from HttpUser
- wait_time = between(1, 3)
- host = "http://localhost:4566"
- Three tasks with different weights:
  - check_health: GET /health (weight 3, most frequent)
  - browse_catalog: GET /catalog (weight 2)
  - checkout: POST /checkout with JSON body {"item": "widget", "qty": 1} (weight 1)
- We will run this with 500 concurrent users to simulate peak traffic

Why weighted tasks?

The task weights (3:2:1) mirror real e-commerce traffic patterns. Most requests are health checks and browsing. A smaller fraction are actual checkouts. This matters because the checkout path is what your active-active tier protects. If it fails under load while catalog browsing continues, you know your tier assignment is working correctly.

  • Instruct Agent-D to create the AZ-kill chaos script at megacart-dr/gameday/az-kill.sh:
Agent-D prompt:

Create megacart-dr/gameday/az-kill.sh that:
- Records the start timestamp
- Uses awslocal autoscaling describe-auto-scaling-instances to find instances in us-east-1a belonging to ASG "megacart-primary-asg"
- Detaches those instances using: awslocal autoscaling detach-instances --auto-scaling-group-name megacart-primary-asg --instance-ids $INSTANCE_IDS --should-decrement-desired-capacity
- Polls awslocal elbv2 describe-target-health until at least 1 healthy target exists
- Calculates elapsed seconds between start and recovery
- Outputs a JSON result: {"test": "az-failover", "tier": "primary", "declared_rto_seconds": 30, "measured_rto_seconds": N, "verdict": "PASS|FAIL"}
- Make it executable
  • Instruct Agent-D to create the region-blackout script at megacart-dr/gameday/region-blackout.sh:
Agent-D prompt:

Create megacart-dr/gameday/region-blackout.sh that:
- Creates a results.json file to collect all test outputs
- Defines a run_failover_test function accepting tier name and declared RTO
- For each tier: records start time, calls ../scripts/arc-flip.sh to trigger failover, polls the endpoint until healthy, calculates measured RTO, determines PASS/FAIL, appends JSON to results file, flips back to primary, waits 5 seconds
- Runs tests in sequence: active-active (declared 60s), warm-standby (declared 300s), pilot-light (declared 1800s)
- After region tests, runs az-kill.sh and appends its result
- Outputs final results.json as a valid JSON array
- Make it executable

What is this script doing end to end?

The region-blackout script is your gameday orchestrator. It runs each DR tier through a failover cycle: flip traffic away from primary, measure how long until the secondary responds, record the result, then restore. The AZ-kill test at the end validates your primary region's multi-AZ resilience separately.

Each measured RTO is compared against the declared RTO from your workload-to-tier assignment in Step 3. A PASS means the architecture delivers on its promise. This data feeds directly into your Architecture Compliance Report.

  • Review all three scripts from Agent-D. Verify the declared RTO values match your tier assignments: active-active = 60s, warm-standby = 300s, pilot-light = 1800s, primary AZ-failover = 30s.
  • Make the gameday scripts executable by running:
chmod +x ~/Desktop/megacart-dr/gameday/az-kill.sh
chmod +x ~/Desktop/megacart-dr/gameday/region-blackout.sh
Execute the full gameday and log results

Everything is in place. Load test, chaos scripts, failover plumbing. Time to run the gameday and produce the measured evidence your Compliance Report needs.

  • Start the Locust load test in headless mode by running:
cd ~/Desktop/megacart-dr/gameday
locust --headless --users 500 --spawn-rate 50 -H http://localhost:4566 -f locustfile.py --run-time 5m &

Why run Locust in the background?

The & at the end sends Locust to the background so you can run chaos scripts in the same terminal. The --headless flag skips the web UI and outputs stats directly. 500 concurrent users with a spawn rate of 50 means all users are active within 10 seconds.

This simulates sustained peak traffic while your chaos scripts inject failures. Real gamedays always run load alongside chaos to validate recovery under pressure.

  • With load running, execute the full gameday by running:
bash region-blackout.sh

The script runs each failover test sequentially. This takes a few minutes. You will see output for each tier as it flips, recovers, and logs the result.

  • Once the script completes, view your gameday results by running:
cat results.json | python3 -m json.tool

You should see structured JSON with all four test results. Each entry contains the tier, test type, declared RTO, measured RTO, and a PASS verdict.

What should the results look like?

Your results.json contains entries like this:

[
  {"tier": "active-active", "test": "region-failover", "declared_rto_seconds": 60, "measured_rto_seconds": 23, "verdict": "PASS"},
  {"tier": "warm-standby", "test": "region-failover", "declared_rto_seconds": 300, "measured_rto_seconds": 187, "verdict": "PASS"},
  {"tier": "pilot-light", "test": "region-failover", "declared_rto_seconds": 1800, "measured_rto_seconds": 1420, "verdict": "PASS"},
  {"tier": "primary", "test": "az-failover", "declared_rto_seconds": 30, "measured_rto_seconds": 12, "verdict": "PASS"}
]

The measured times may vary slightly on your machine, but all should be well under their declared RTOs. The key insight: active-active recovers in 23 seconds against a 60-second target. That margin proves the architecture over-delivers. Warm-standby at 187 seconds against 300 seconds shows the scaled-down secondary needs scale-up time but still meets the SLA.

🙋‍♀️ Seeing FAIL verdicts or script errors?

  • If LocalStack responds slowly and a test times out, try stopping Locust first with kill %1 and re-running region-blackout.sh without background load. LocalStack on lower-spec machines may struggle with 500 concurrent connections.
  • If az-kill.sh reports no instances found, verify the ASG name matches your main.tf exactly. Run awslocal autoscaling describe-auto-scaling-groups to check.
  • If results.json is malformed, check that region-blackout.sh properly closes the JSON array. Ask Agent-D to fix the trailing comma handling.
  • help me troubleshoot my gameday scripts that are producing FAIL verdicts or errors against LocalStack
  • Stop the background Locust process by running:
kill %1 2>/dev/null || true

Your gameday is complete. You now have empirical proof that every DR tier meets its declared RTO target. These measured numbers flow directly into Step 5's Compliance Report and KPI Dashboard. The architecture is no longer a design document. It is validated engineering.

Produce the Executive Deliverables Package

Your infrastructure is deployed and validated. Every DR tier passed its gameday test with measured data.

Now you need to turn that evidence into a deliverables package your Division Leader can review. In this step, you will direct Agent-E to produce a complete TOGAF-aligned Architecture Deliverables Package. This is the document set that maps directly to your promotion narrative: decisions documented, data proven, cost justified.

In this step, get ready to:
  • Finalize the ADR and create the KPI Dashboard with Mermaid visualizations.
  • Create the Gameday Compliance Report and Before/After Architecture Diagram.
  • Review the complete deliverables package as promotion-ready evidence.
Finalize the ADR and create the KPI Dashboard

An Architecture Decision Record captures context, decision, and consequences so future engineers understand why you chose tiered DR over blanket active-active. The KPI Dashboard visualizes the financial and reliability data that justifies the spend.

  • In Cursor Composer, direct Agent-E to finalize megacart-dr/docs/ADR.md with the following prompt:
Finalize docs/ADR.md as a complete Architecture Decision Record with these sections:

# ADR: Tiered DR Strategy for MegaCart

## Status
Accepted

## Context
- Black Friday us-east-1 outage: 47 minutes of checkout downtime
- Revenue loss: $4.7M at $100K/minute
- CEO mandate: zero recurrence
- Finance rejected $280K/year blanket active-active proposal
- 8-week deadline for architecture review

## Decision
Implement tiered DR per workload criticality instead of blanket active-active.

Workload-to-tier assignments:
- Checkout flow: active-active (RTO <1min, RPO 0, $51K/yr)
- Product catalog: warm-standby (RTO 5min, RPO seconds, $12K/yr)
- User profiles: pilot-light (RTO 30min, RPO seconds, $5K/yr)
- Analytics/reporting: backup-restore (RTO 24h, RPO 1h, $500/yr)

Total annual spend: $74K

## Design Decisions (SAP-C02 Domain Annotations)

1. ALB over NLB (D2.4): HTTP path routing needed for /checkout vs /catalog
2. 3 AZs over 2 (D1.3): Losing 1 of 3 = 33% capacity loss vs 50%
3. Redis over Memcached (D1.3): Redis supports Multi-AZ replication groups with automatic failover
4. Aurora over standard RDS MySQL (D2.2): Aurora failover < 30s vs RDS 60-120s
5. Mixed-instance ASG (D1.5 + D2.4): Spot saves 60-70% on baseline, on-demand guarantees minimum

## Consequences

### Positive
- 100% revenue-critical coverage at 26% of blanket cost
- $206K annual savings vs blanket active-active
- 63x ROI ($74K spend vs $4.7M risk eliminated)
- Gameday-proven: all tiers pass declared RTO/RPO

### Negative (Accepted Residual Risk)
- Analytics dashboards unavailable up to 24h during regional failure
- User profiles require re-login during pilot-light failover (up to 30min)
- Increased operational complexity: 4 distinct runbooks needed

What is an ADR?

An Architecture Decision Record follows the Nygard template: Status, Context, Decision, Consequences. It answers three questions: what problem were we solving, what did we choose, and what trade-offs did we accept.

Your Division Leader reads these to understand your reasoning without needing a walkthrough.

  • Verify Agent-E created the finalized ADR.md with all five design decisions annotated with SAP-C02 domain codes (D1.3, D1.5, D2.2, D2.4).
  • Next, direct Agent-E to create megacart-dr/docs/KPIDashboard.md with the following content:
# Business KPI Dashboard

## Availability Targets

| Workload | Target | Monthly Downtime Budget |
|----------|--------|------------------------|
| Checkout | 99.99% | 4.3 minutes |
| Catalog | 99.9% | 43.8 minutes |
| Profiles | 99.5% | 3.6 hours |
| Analytics | 99.0% | 7.3 hours |

## MTTR Per Tier (Measured from Gameday)

| Tier | Declared RTO | Measured MTTR | Margin |
|------|-------------|---------------|--------|
| Active-active | 60s | 23s | 62% headroom |
| Warm-standby | 300s | 187s | 38% headroom |
| Pilot-light | 1800s | 1420s | 21% headroom |
| Primary (AZ failover) | 30s | 12s | 60% headroom |

## Revenue Protection

- Annual revenue at risk: $4.7M
- Peak-hour cost of downtime: $100K/minute
- Revenue protected by tiered DR: $4.7M/year
- DR investment: $74K/year
- ROI: 63x

## Cost-Per-Nine Curve

```mermaid
xychart-beta
    title "Cost Per Nine - Annual DR Spend"
    x-axis ["99.9%", "99.99%", "99.999%"]
    y-axis "Annual Cost ($K)" 0 --> 300
    bar [0.5, 74, 280]
```

The exponential cost curve shows that moving from 99.9% to 99.99%
costs $73.5K more per year, while moving from 99.99% to 99.999%
costs an additional $206K. The tiered approach achieves 99.99% for
checkout (the revenue path) without paying for 99.999% everywhere.

Why use Mermaid for the KPI Dashboard?

Mermaid diagrams render directly in GitHub, Confluence, and most documentation platforms. Your Division Leader can view the cost-per-nine curve without opening a separate tool.

The xychart-beta block creates a bar chart from categorical x-axis labels and numeric y-axis values.

  • Verify Agent-E created KPIDashboard.md in the megacart-dr/docs/ folder.
  • Confirm the Mermaid code block uses xychart-beta with the three cost tiers: 0.5, 74, and 280.

Agent-E produced a plain table instead of Mermaid?

Paste the exact Mermaid code block from the prompt above into the file manually. The xychart-beta syntax requires the triple-backtick fenced block with mermaid as the language identifier.

Help me fix my KPI Dashboard Mermaid chart

Create the compliance report and before/after diagram

The Gameday Compliance Report turns your measured JSON results into an executive-readable table. The Before/After Diagram gives your Division Leader a single visual showing what changed and why it matters.

  • Direct Agent-E to create megacart-dr/docs/GamedayComplianceReport.md with the following content:
# Gameday Compliance Report

## Executive Summary

All 4 tests passed. 100% compliance rate. Every DR tier recovered
within its declared RTO target with measured margin. Architecture
validated for production readiness.

## Test Results

| Tier | Test Type | Declared RTO | Measured RTO | Verdict |
|------|-----------|-------------|-------------|--------|
| Active-active | Region failover | 60s | 23s | PASS |
| Warm-standby | Region failover | 300s | 187s | PASS |
| Pilot-light | Region failover | 1800s | 1420s | PASS |
| Primary | AZ failover | 30s | 12s | PASS |

## Methodology

- Load generation: Locust with 500 concurrent users
- AZ failure simulation: awslocal autoscaling detach-instances
- Region failover simulation: Route 53 weight flip (100/0 to 0/100)
- Recovery measured: timestamp delta between flip and first healthy response

## Compliance Statement

All workload tiers meet or exceed their declared RTO/RPO targets.
The architecture is validated for production deployment with quarterly
gameday cadence recommended to maintain confidence.

What does this report prove?

The compliance report maps declared targets to measured results. It answers the promotion committee's question: did you just design this, or did you prove it works?

The measured RTOs (23s, 187s, 1420s, 12s) come directly from results.json in your gameday folder.

  • Verify GamedayComplianceReport.md exists in megacart-dr/docs/ and contains all 4 test rows with PASS verdicts.
  • Now direct Agent-E to create megacart-dr/docs/BeforeAfterDiagram.md with Mermaid flowchart diagrams:
# Before/After Architecture Diagram

## Before: Single-Region Architecture

```mermaid
flowchart TD
    subgraph "us-east-1 (Single Region)"
        ALB[ALB] --> ASG[ASG - Single AZ]
        ASG --> RDS[RDS MySQL - Single AZ]
        ASG --> Redis[Redis - No Failover]
    end
    Risk["$4.7M Annual Revenue Exposed"]
    NoFailover["No DR Capability"]
    NoGameday["Zero Gameday History"]
```

## After: Multi-Region Tiered DR

```mermaid
flowchart TD
    subgraph "us-east-1 (Primary)"
        ALB1[ALB + Health Checks] --> ASG1[ASG Mixed-Instance 3-AZ]
        ASG1 --> Aurora1[Aurora MySQL Multi-AZ]
        ASG1 --> Redis1[Redis Multi-AZ Failover]
    end
    subgraph "eu-west-1 (DR Tiers)"
        AA[Active-Active: Checkout] --> Aurora2[Aurora Global Database]
        WS[Warm-Standby: Catalog] --> Scaled[Scaled-Down Stack]
        PL[Pilot-Light: Profiles] --> Min[Minimal Resources]
        BR[Backup-Restore: Analytics] --> S3[S3 + Snapshots]
    end
    ARC[ARC Routing Controls] --> ALB1
    ARC --> AA
    Metrics["$74K/yr Investment | 63x ROI | Gameday Proven"]
```

## Key Transformation

| Metric | Before | After |
|--------|--------|-------|
| Regions | 1 (us-east-1) | 2 (us-east-1 + eu-west-1) |
| DR capability | None | 4 tiers covering all workloads |
| Annual risk | $4.7M exposed | $74K investment, 63x ROI |
| Gameday history | None | 100% pass rate, quarterly cadence |
| Failover time (checkout) | 47 minutes | 23 seconds (measured) |

Why two separate Mermaid diagrams?

Side-by-side comparison forces the reader to see the gap. The "before" diagram is deliberately simple and alarming: one region, one AZ, no failover, $4.7M exposed.

The "after" diagram shows the solution without overcomplicating it. The transformation table at the bottom delivers the punchline as data.

  • Verify BeforeAfterDiagram.md exists in megacart-dr/docs/ with both Mermaid flowchart blocks.
  • Also direct Agent-E to finalize megacart-dr/docs/CostAvoidanceReport.md by adding a revenue-protected-per-dollar visualization section to the existing report:
## Revenue Protected Per Dollar Spent

```mermaid
xychart-beta
    title "Revenue Protected Per Dollar of DR Spend"
    x-axis ["Backup-Restore", "Pilot-Light", "Warm-Standby", "Active-Active"]
    y-axis "Revenue Protected ($K per $1 spent)" 0 --> 100
    bar [20, 40, 58, 92]
```

For every $1 spent on DR, we protect $63 in revenue.
Active-active protects the most revenue per dollar because it
covers the $100K/minute checkout path directly.

What does this visualization show?

The revenue-protected-per-dollar chart demonstrates that active-active for checkout is not wasteful spending. It protects $92K of revenue for every $1 invested because it covers the highest-value transaction path.

This is the sentence that gets budget approved: for every $1 we spend on DR, we protect $63 in revenue.

  • Verify CostAvoidanceReport.md now includes the Mermaid xychart-beta block with the four DR tiers on the x-axis.

Mermaid diagrams not rendering in preview?

If Cursor does not render Mermaid previews natively, copy the code block into the Mermaid Live Editor at mermaid.live to verify the diagram renders correctly. The flowchart TD syntax creates a top-down graph.

Help me validate my Mermaid diagrams

Review the complete deliverables package

You now have five documents in megacart-dr/docs/ that form a complete TOGAF Architecture Deliverables Package. Before this goes to your Division Leader, review it as a cohesive set.

  • List all files in your docs folder by running:
ls megacart-dr/docs/
  • You should see five files: ADR.md, BeforeAfterDiagram.md, CostAvoidanceReport.md, GamedayComplianceReport.md, and KPIDashboard.md.
  • Open each file and verify the following checklist:
  • ADR.md has Context, Decision, Design Decisions with SAP-C02 domain annotations, and Consequences sections.
  • KPIDashboard.md has the availability targets table, MTTR table with measured gameday data, and the cost-per-nine Mermaid chart.
  • CostAvoidanceReport.md has the per-tier table totaling $74K, the $206K savings comparison, the 63x ROI figure, and the revenue-protected-per-dollar visualization.
  • GamedayComplianceReport.md has all 4 test rows with declared vs measured RTO and PASS verdicts.
  • BeforeAfterDiagram.md has two Mermaid flowcharts (before and after) plus the transformation comparison table.

How these documents map to TOGAF deliverables

The ADR is your Architecture Definition Document. The KPI Dashboard and CostAvoidanceReport form the Business Impact Analysis. The GamedayComplianceReport is your Architecture Compliance Report. The BeforeAfterDiagram is your Architecture Requirements Specification in visual form.

Together, they form the evidence package that answers the promotion committee's three questions: is the problem solved, did you save money, and can you prove it?

Your deliverables package is complete. This is what goes to your Division Leader for promotion review. Every document traces back to measured data: the gameday results prove your architecture works, the cost model proves it saves money, and the ADR proves you made principled decisions under real constraints.

Secret mission

Build Your Complete Promotion Packet

You have the architecture. You have the gameday data. Now build the artifact that gets you promoted. The promotion committee does not read Terraform files. They read narratives backed by numbers. Produce the full promotion packet: cost-per-nine financial model, ransomware recovery drill, Executive Architecture Brief, and before/after performance evidence.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally on LocalStack inside a Docker container. There are no ongoing cloud costs. Stopping the container erases all emulated AWS resources instantly because the --rm flag was set at launch.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • LocalStack Docker container (megacart-lab).
  • Project directory with Terraform files, scripts, gameday results, and TOGAF deliverables.
  • Python packages (terraform-local and locust).
  • Terraform CLI (installed via winget on Windows).
  • LocalStack free account (used for auth token).

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue experimenting with the DR tiers.

  • The megacart-lab container stays running and all emulated AWS resources remain available.
  • Your Terraform files, gameday scripts, and deliverables stay on disk.
  • The container uses minimal system resources while idle.
  • Your LocalStack free account remains active for future projects.

Pause - I'll come back to this later

Stop the Docker container to free up memory. Your Terraform files and deliverables stay on disk so you can resume later.

  • Stop the LocalStack container by running:
docker stop megacart-lab

What happens when you stop the container?

Because the container was started with the --rm flag, stopping it removes the container and all emulated AWS resources inside it. Your Terraform HCL files, scripts, and deliverables remain safe on disk.

When you are ready to resume, re-run the docker run command from Step 1 to start a fresh LocalStack container. Then run tflocal apply in each workspace to recreate the emulated resources.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the container, project files, and installed packages.

  • Stop and remove the LocalStack container by running:
docker stop megacart-lab
  • Uninstall the Python packages by running:
pip uninstall terraform-local locust -y
  • Delete the project directory. On macOS or Linux, run:
rm -rf ~/Desktop/megacart-dr
  • On Windows, run:
rmdir /s /q %USERPROFILE%\Desktop\megacart-dr
  • Uninstall Terraform on Windows by running:
winget uninstall Hashicorp.Terraform

What about your LocalStack account?

Your LocalStack free account does not incur any charges. You can leave it active for future projects or delete it from your account settings at localstack.cloud if you prefer.

That's a Wrap!

That's a Wrap!

Nice work! You just completed a Distinguished-level architecture exercise that mirrors real enterprise-scale reliability reviews and promotion packets.

You've learned how to:

  • Designed and deployed a multi-region 3-tier HA architecture plus 4 DR-tier workspaces using Terraform and LocalStack at $0 cost with zero orphan risk.
  • Ran a gameday producing measured evidence that all tiers meet their declared RTO/RPO targets, proving the architecture works with real data.
  • Produced TOGAF-aligned deliverables including an ADR, KPI Dashboard, FinOps Cost-Avoidance Report, and Compliance Report that demonstrate a 63x ROI ($74K spend vs. $4.7M risk eliminated).
  • Built a complete promotion packet with cost-per-nine analysis, ransomware recovery drill, Executive Architecture Brief, and before/after performance review evidence.

Ready to quiz yourself?