Surviving Regional Failures on AWS

Deploy a multi-region app and prove zero-downtime failover with live chaos testing.

Introduction

30 Second Summary

Imagine you're on call and an entire US cloud region just went dark. Fifty thousand users across the country are hitting your app, but every request is timing out.

In this project, you will deploy a multi-region application across two AWS regions and configure CloudFront to automatically route around failures. You'll simulate a live regional outage, measure your exact recovery time, and generate a professional Incident Dashboard Report from the real data.

What You'll Build

You'll break your primary AWS region on command and watch traffic seamlessly shift to a secondary region in seconds, all captured by a heartbeat monitor measuring your exact recovery time on screen.

By the end of this project, you'll have:

  • A live multi-region application deployed across two AWS regions, where you can break the primary region and watch traffic seamlessly shift to the secondary in real time.
  • A chaos injection toolkit and heartbeat monitor that simulates regional outages on demand and measures your exact recovery time down to the second.
  • A professional AI-generated Incident Dashboard Report (HTML) with real failover data, timeline visualization, and production-readiness recommendations you can present to stakeholders.
  • Secret Mission: Perform a full cross-region disaster recovery cutover by promoting the RDS read replica and scaling the secondary region to handle production traffic.

Are there any prerequisites?

You'll need an AWS account with admin-level access and basic familiarity with VPC, EC2, and IAM concepts. Step 1 walks you through installing the AWS CLI and cloning the pre-built workshop toolkit.

Before We Start

You're about to build infrastructure that survives the kind of failures that take down major services. Before we dive in, take a moment to lock in what you're building and why it matters when an entire AWS region goes dark.

Set Up and Deploy Infrastructure

Before you can test how your application survives a regional failure, you need the infrastructure running in two regions. In this step, you'll get your tools configured and deploy the full multi-region stack using CloudFormation.

The workshop toolkit handles the heavy lifting. Pre-built templates spin up a VPC, ALB, Auto Scaling Group, and RDS instance in each region, plus a cross-region read replica. Your job is to get credentials configured, clone the repo, and let the deploy script do its thing.

In this step, get ready to:
  • Install and configure the AWS CLI with your credentials.
  • Clone the workshop toolkit and pass pre-flight checks.
  • Deploy CloudFormation stacks to us-east-1 and us-west-2.
Install and configure prerequisites

You need the AWS CLI to talk to AWS from your terminal, and Git to clone the workshop repo.

  • Check if the AWS CLI is already installed by running:
aws --version

✔️ I see a version number

The AWS CLI is installed and ready. Move on to configuring your credentials below.

ⓧ Command not found

Install the AWS CLI using the instructions for your operating system.

macOS

  • Install the AWS CLI via Homebrew by running:
brew install awscli
  • After installation finishes, confirm it worked by running aws --version again.

Windows

  • Download and run the MSI installer from:

https://awscli.amazonaws.com/AWSCLIV2.msi

  • Follow the installer prompts to completion.
  • Close and reopen your terminal, then confirm it worked by running aws --version.

Linux

  • Download and install the AWS CLI by running these commands:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
  • Confirm it worked by running aws --version again.

Before you run aws configure, you need your Access Key ID and Secret Access Key from the IAM console.

  • Select the Security credentials tab.
  • In the Access keys section, click Create access key.
  • For the use case, select Command Line Interface (CLI), then click Next.
  • Set Description tag value to "AWS CLI"
  • Click Create access key.
  • Click Show to reveal your Secret Access Key, or click Download .csv file to save both keys to your computer.

Why do I need to save the secret key now?

AWS only shows the Secret Access Key once, at the moment you create it. If you close the page without copying or downloading it, you cannot retrieve it again and will need to create a new key.

  • Configure the AWS CLI with your credentials by running:
aws configure

What will aws configure ask for?

The command prompts you for four values, one at a time:

  • AWS Access Key ID and AWS Secret Access Key from your IAM user credentials.
  • Default region name: enter us-east-1.
  • Default output format: enter json.
  • Verify your credentials work by running:
aws sts get-caller-identity

Seeing an error about invalid credentials?

  • Double-check that you pasted the Access Key and Secret Key without extra spaces.
  • Make sure your IAM user has programmatic access enabled.
  • If your account uses MFA, you may need to generate temporary credentials first.

Help me troubleshoot my AWS CLI credentials

  • Check if Git is installed by running:
git --version

✔️ I see a version number

Git is ready. Move on to cloning the workshop repo.

ⓧ Command not found

  • Download and install Git from the official site:

https://git-scm.com/downloads

  • Follow the installer for your operating system, then close and reopen your terminal.
  • Run git --version again to confirm it installed successfully.
Clone the workshop toolkit and run pre-flight checks

The workshop toolkit repo contains everything you need: a Flask app, CloudFormation templates, deployment scripts, a chaos injection script, and a heartbeat monitor.

  • In the same Terminal, move to your Desktop by running this command:
cd ~/Desktop
  • Clone the workshop repo and move into it by running these commands:
git clone https://github.com/Nik-1019/aws-multi-region-resilience-workshop.git
cd aws-multi-region-resilience-workshop
  • Confirm you are inside the project folder by running:
ls
  • Now that aws configure is complete and your credentials are verified, run the pre-flight validation script to check the rest of your environment:
./scripts/preflight.sh

What does the pre-flight script check?

The script validates several things before you spend time deploying:

  • Your AWS credentials are valid and can reach the API.
  • Both us-east-1 and us-west-2 regions are enabled on your account.
  • VPC and Elastic IP quotas have room for the new stacks.
  • No conflicting CloudFormation stacks already exist with the same name.
  • The workshop repo URL is reachable and supports anonymous clone (catches typos, private repos, and SSH URLs before deploy).

✔️ All checks pass green

Your environment is validated. Move on to deploying the infrastructure below.

ⓧ A check failed

The script tells you which check failed. Find your specific error below.

Credentials invalid

  • Run aws sts get-caller-identity to confirm your credentials are still working.
  • If using temporary credentials (from AWS Academy, SSO, or a CloudShell export), they may have expired. Re-run your credential export command.
  • If using long-term IAM user credentials, re-run aws configure and paste your Access Key and Secret Key again without extra spaces.

Region not enabled

  • Sign in to the AWS Management Console.
  • Click your account name in the top-right corner and select Account.
  • Scroll down to the AWS Regions section.
  • Find the region the script reported (likely us-west-2) and click Enable.
  • Wait 1-2 minutes for the region to activate, then re-run ./scripts/preflight.sh.

Insufficient IAM permissions

  • The workshop requires permissions for EC2, VPC, RDS, CloudFormation, CloudFront, Route 53, and Elastic Load Balancing.
  • Open the IAM console, find your user, and attach the AdministratorAccess managed policy for the duration of the workshop.
  • If your organization restricts full admin access, attach these managed policies at minimum: AmazonEC2FullAccess, AmazonRDSFullAccess, AmazonVPCFullAccess, CloudFrontFullAccess, AmazonRoute53FullAccess, and AWSCloudFormationFullAccess.
  • Re-run ./scripts/preflight.sh after updating permissions.

Repo URL unreachable

  • The script probes whether the workshop repo URL supports anonymous clone. If it reports the URL is unreachable, private, or still the placeholder default:
  • Confirm the URL uses HTTPS format (e.g. https://github.com/org/repo.git). SSH URLs (git@github.com:...) cannot be cloned by the EC2 instances because they have no SSH key.
  • Open the URL in your browser. If GitHub shows a 404 or asks you to sign in, the repo is private or the URL has a typo.
  • If the URL is correct but private, ask the workshop facilitator for the public HTTPS URL.
  • Re-run ./scripts/preflight.sh after correcting the URL.

Help me fix my pre-flight check failure

ⓧ Quota limits exceeded

The workshop creates 1 VPC, 1 NAT Gateway, and 1 Elastic IP in each region. If your account already uses these resources, you may hit the default limits.

  • Open the Service Quotas console at https://console.aws.amazon.com/servicequotas/.
  • In the left navigation, select AWS services, then search for Amazon VPC.
  • Search for the specific quota the script flagged (e.g. VPCs per Region, NAT gateways per Availability Zone, or EC2-VPC Elastic IPs).
  • Compare the Applied quota value against your current usage.

You have two options to resolve this:

  • Free up headroom: delete unused VPCs, release unattached Elastic IPs, or remove idle NAT Gateways in the affected region. This is the fastest fix.
  • Request an increase: click the quota name, then click Request increase at account level. Most VPC-related increases are approved within minutes.

Check quotas in both us-east-1 and us-west-2 using the region selector in the top-right corner. Re-run ./scripts/preflight.sh after freeing resources or receiving the increase.

Help me resolve a service quota limit for my AWS workshop

EC2 write permission warning is safe to ignore

If all other checks pass green but you see a warning about EC2 write permissions being "inconclusive", that's safe to ignore. The script tests write permissions using a dry-run API call that some account types (AWS Academy, sandbox accounts) don't support. Your deployment will still work.

🙋‍♀️ Credentials fail in the script but work directly?

If aws sts get-caller-identity works in your terminal but the pre-flight script reports sts:GetCallerIdentity failed, the script may have a JMESPath query syntax issue on line 109.

  • Open ./scripts/preflight.sh and find line 109.
  • Look for --query 'Account,Arn' (without square brackets).
  • Change it to --query '[Account,Arn]' (with square brackets). AWS CLI v2 requires brackets for multi-field JMESPath queries.
  • Save the file and re-run ./scripts/preflight.sh.

Help me fix the pre-flight script's JMESPath query error

Deploy the multi-region infrastructure

The deploy script creates two full CloudFormation stacks: one in us-east-1 (primary) and one in us-west-2 (secondary), plus a cross-region RDS read replica.

What does each regional stack create?

Primary (us-east-1) handles all production traffic under normal conditions. The stack creates a VPC with public and private subnets, a NAT Gateway, an ALB that receives user requests, an Auto Scaling Group running EC2 instances with the Flask app, and the primary RDS MySQL database.

Secondary (us-west-2) runs a reduced-capacity copy of the same architecture plus a cross-region RDS read replica that continuously replicates data from the primary.

This is the pilot light disaster recovery pattern. Instead of running two full production environments (which doubles cost), you keep the secondary region at reduced capacity with live data replication. When the primary fails, the secondary scales up and takes over without rebuilding from scratch.

  • Start the deployment by running:
./scripts/deploy.sh

What will the deploy script ask for?

The script will prompt you for a database password. Choose something you can remember for the workshop (it does not need to be highly complex for this temporary environment).

After you enter the password, the script deploys CloudFormation templates to both regions. While deploying, it displays the architecture being created: VPC, ALB, ASG, RDS in each region, plus the cross-region read replica.

Deployment takes 10-12 minutes.

Both stacks deploy in parallel, but creating the VPCs, NAT Gateways, RDS instances, and cross-region replication takes time. Leave the terminal running and wait for the script to report completion.

The deployment takes 15 minutes or more while CloudFormation creates VPCs, NAT Gateways, RDS instances, and the cross-region read replica. Leave the terminal running and use this time productively.

✔️ I'll just wait

No problem. Keep the terminal open and wait for the script to print CREATE_COMPLETE for both stacks. You can grab a coffee or stretch your legs.

ⓧ Let me explore the codebase while I wait

You just ran a single script and it is creating an entire multi-region architecture for you. That is Infrastructure as Code in action. Let's open the project in your IDE and walk through what the toolkit contains so you understand exactly what is being built.

Open the project in your IDE
  • Open Kiro, VS Code, or whichever code editor you prefer.
  • Click File in the top menu bar, then click Open Folder.
  • Navigate to your Desktop and select the aws-multi-region-resilience-workshop folder, then click Open.

In the Explorer sidebar on the left, you should see the full project structure. Take note of these key files and folders:

  • app.py — the Flask application that renders the region dashboard.
  • cloudformation/ — the IaC templates that define every AWS resource.
  • scripts/ — automation scripts for deploy, chaos injection, monitoring, and cleanup.
  • userdata.sh — the bootstrap script each EC2 instance runs on first boot.
  • templates/index.html — the dashboard UI (HTML + CSS + JS, no build step).
Read the primary CloudFormation template
  • In the Explorer sidebar, expand the cloudformation folder.
  • Click primary.yaml to open it in the editor.

Scroll through the file. This single YAML file declares every resource in your us-east-1 region. Look for these sections:

  • VPC and Subnets — a 10.10.0.0/16 VPC with 2 public subnets (for the ALB) and 2 private subnets (for the app and database).
  • NAT Gateway — allows instances in private subnets to reach the internet (to clone the repo on boot) without being publicly accessible.
  • Application Load Balancer (ALB) — sits in the public subnets, receives user traffic on port 80, and forwards it to healthy EC2 instances.
  • Auto Scaling Group (ASG) — manages 1-2 t3.micro instances running the Flask app behind the ALB.
  • RDS MySQL instance — a db.t3.micro database in Read-Write mode. Notice BackupRetentionPeriod: 1 — this enables automated backups, which is required for cross-region replication.Compare the secondary template
  • In the same cloudformation folder, click secondary.yaml to open it.

This template is almost identical to primary.yaml, but with two key differences:

  • The VPC uses 10.20.0.0/16 instead of 10.10.0.0/16 — a different CIDR block so the two VPCs don't clash.
  • Instead of creating a fresh RDS instance, it creates a cross-region read replica that references the primary database's ARN. This replica continuously replicates data from us-east-1 and serves it in Read-Only mode.

Why is this called a pilot light pattern?

The secondary region runs at reduced capacity with live data replication, like a pilot light on a gas stove. When the primary fails, you scale the secondary up and promote the replica to Read-Write. You avoid paying for two full production environments but still recover in minutes instead of hours.

Understand the deploy script logic
  • In the Explorer sidebar, expand the scripts folder.
  • Click deploy.sh to open it in the editor.

The script follows a specific order that matters:

  1. Prompts for a database password (or generates a random one).
  2. Deploys primary.yaml to us-east-1 and waits for CREATE_COMPLETE.
  3. Reads the primary RDS ARN from the stack outputs (the secondary template needs this ARN to create the replica).
  4. Deploys secondary.yaml to us-west-2 with the primary RDS ARN passed as a parameter.
  5. Writes .workshop-config — a local file with ALB URLs, security group IDs, and database endpoints that other scripts reference.

Why does the secondary deploy after the primary?

The cross-region read replica needs the primary database to exist and have backups enabled before replication can start. Deploying in parallel would fail because the secondary template references an ARN that doesn't exist yet.

Check what happens on each EC2 instance at boot
  • Back in the Explorer sidebar, click userdata.sh at the top level of the project.

This is what every EC2 instance executes when it first launches. It clones the workshop repo, installs Python dependencies from requirements.txt, and starts the Flask app with gunicorn on port 80.

Preview the chaos and monitoring scripts
  • In the Explorer sidebar, expand the scripts folder (if it's not already open).
  • Click through each script file to preview its contents. Here's what each one does:
  • preflight.sh — validates your account before deploying (you already ran this).
  • deploy.sh — deploys both CloudFormation stacks (running right now in your terminal).
  • heartbeat.sh — polls your app once per second and prints a live table showing which region is responding, latency, and database status.
  • chaos.sh — simulates a regional outage by revoking inbound HTTP rules on the primary ALB's security group. You'll use this in a later step.
  • cleanup.sh — deletes everything in reverse order when you're done with the workshop.

How does chaos.sh simulate a regional failure?

It doesn't destroy any resources. Instead, it revokes the inbound HTTP/HTTPS rules on the primary ALB's security group. From the internet's perspective, the primary region instantly becomes unreachable. This mimics a network partition without touching the actual instances or database.

Running chaos.sh restore adds the rules back. The effect is instant in both directions.

Switch back to your deploy terminal and check if both stacks have finished. If the script is still running, give it another few minutes.

Before you check the result, what do you expect the script to output when both stacks finish successfully?

When deployment completes, you should see both stacks report CREATE_COMPLETE status. The script also outputs both ALB URLs and writes a .workshop-config file that other scripts will reference.

  • Copy the primary ALB URL from the deploy output and open it in your browser.
  • Make sure you use http:// (not https://).

You should see the Multi-Region Resilience dashboard with a green PRIMARY REGION banner. The page displays the region (us-east-1), availability zone, instance ID, and a database section showing Connected with mode Read-Write.

  • Now copy the secondary ALB URL from the deploy output and open it in a new browser tab.

You should see the same dashboard but with a blue SECONDARY REGION banner. The region shows us-west-2 and the database section shows Connected with mode Read-Only (replica). This confirms data is replicating from the primary.

✔️ I see the app with region us-east-1

Your app is running and the primary region is serving traffic. Move on to the next step.

ⓧ I see a 503 Service Unavailable error

A 503 means the ALB is reachable but no healthy targets are behind it. This usually happens when the app failed to install on the EC2 instances (most commonly because the repo URL in the CloudFormation template is incorrect or the repo is private).

Diagnose the issue

  • Open the EC2 console and switch to the us-east-1 region.
  • Click Target Groups in the left navigation under Load Balancing.
  • Select the workshop target group and check the Targets tab. If all targets show unhealthy, the app never started.

Check the instance user-data logs

  • Select one of the unhealthy instances and click Connect.
  • Choose Session Manager and click Connect.
  • In the shell session, view the user-data log by running:
sudo cat /var/log/cloud-init-output.log | tail -50
  • Look for fatal: or Could not resolve host messages. These indicate the repo URL is wrong or unreachable from the instance.

Fix the repo URL

  • Confirm the repo URL is correct and publicly accessible (open it in your browser to verify).
  • If the URL is wrong, delete both CloudFormation stacks, update the GitRepoURL parameter in .workshop-config or the template, and re-run ./scripts/deploy.sh.

Help me debug why my EC2 instances aren't serving traffic after a successful CloudFormation deploy

ⓧ The page won't load at all

If the browser spins indefinitely or shows a connection timeout (not a 503), the ALB itself isn't reachable.

  • Wait 1-2 minutes. ALB DNS names take a moment to propagate after creation. Try refreshing.
  • Check the URL format. Make sure you're using http:// (not https://). The workshop ALBs are HTTP-only.
  • Verify the security group. Open the EC2 console, go to Load Balancers, select your ALB, and check the Security tab. Inbound rules should allow HTTP (port 80) from 0.0.0.0/0.

Check if chaos.sh was accidentally run

If the security group shows no inbound HTTP rule, someone (or a previous test) may have run ./scripts/chaos.sh break. Run ./scripts/chaos.sh restore to re-add the rules.

Help me figure out why my ALB page won't load at all

Your multi-region infrastructure is live. Next up, you'll explore both regional endpoints and discover what happens when one of them goes dark with no protection in place.

Explore Your Multi-Region Infrastructure

Your infrastructure is deployed across two AWS regions. Before you can protect against failure, you need to confirm both regional stacks are independently healthy and understand what each one does.

Then you'll experience what happens today when a region fails with no protection in place. This is the problem you'll solve in the next step.

In this step, get ready to:
  • Verify both regional endpoints are serving traffic independently.
  • Simulate a regional outage and observe its impact on users.
Visit both regional endpoints

Right now you have two completely independent application stacks. Each one has its own ALB, its own compute layer, and its own database connection. No failover mechanism connects them.

  • Copy the primary ALB URL from your deploy output (or from .workshop-config) and paste it here:
[[PRIMARY_ALB_URL="your-primary-alb-url.us-east-1.elb.amazonaws.com"]]
  • Open this URL in your browser (make sure you use http://, not https://).

The app displays the region us-east-1, an EC2 instance ID, and database connectivity status with latency.

Find your ALB URLs

Your ALB URLs were printed at the end of the deploy script and saved to the .workshop-config file in your project folder. You can also find them in the EC2 console under Load Balancers in each region.

  • Open the secondary ALB URL in a new browser tab.
  • This shows the same app, but displays region us-west-2 and reports database connected in read-only replica mode.

Both regions are healthy and serving traffic independently. If you bookmark one ALB URL and that region goes down, your users get nothing.

Why are there two separate stacks?

Each region runs its own VPC, Auto Scaling Group, and RDS instance. The secondary region's database is a cross-region read replica that continuously replicates data from the primary.

This is the pilot light disaster recovery pattern. The secondary region stays warm with minimal resources, ready to take over if the primary fails.

Experience the failure

You've confirmed both regions work. Now you'll see what happens when the primary region goes dark and there's no failover protection in front of it.

  • Start the heartbeat monitor against your primary ALB by running this command:
./scripts/heartbeat.sh [[PRIMARY_ALB_URL="your-primary-alb-url.us-east-1.elb.amazonaws.com"]]

What does the heartbeat script do?

The heartbeat script sends continuous HTTP requests to the target URL and displays the response status, region, and latency for each one. You'll see green lines scrolling with 200 OK from us-east-1. This gives you a real-time view of what your users experience.

Is this how production monitoring works?

No. The heartbeat script fires one request per second so you get instant visual feedback when failover happens. It's a workshop tool, not a production pattern.

In production, you'd use ALB health checks (every 5–30 seconds, built into the load balancer), synthetic monitoring like CloudWatch Synthetics or Datadog (every 1–5 minutes from global locations), and real user monitoring that passively tracks actual traffic. Nobody hammers their own endpoint once a second from a single terminal.

Think of the heartbeat script as a stethoscope held to your app's chest. Production monitoring is more like an ICU dashboard that aggregates thousands of real signals into graphs and alerts.

Leave the heartbeat running. You'll watch it live as you inject failure.

  • Open a second terminal tab or window.
  • Navigate back to the aws-multi-region-resilience-workshop folder in this new terminal.

Before you run the next command, make a prediction. You're about to kill inbound traffic to your primary ALB. What do you think will happen to the heartbeat monitor?

  • Inject a regional outage by running this command:
./scripts/chaos.sh break

Switch back to your heartbeat terminal. The green lines stop. Requests start failing with 504 timeouts.

The app is completely down for anyone hitting the primary ALB directly. Your secondary region in us-west-2 is perfectly healthy, but no one knows to go there.

What did the chaos script actually do?

The script revoked inbound HTTP rules on the primary ALB's security group. This simulates a network partition at the regional level. The EC2 instances and RDS database are still running, but no traffic can reach them through the load balancer.

This is the problem. A single-region deployment offers zero protection when that region becomes unavailable. Fifty thousand users refreshing and getting nothing.

  • Restore the primary region by running this command:
./scripts/chaos.sh restore
  • Watch the heartbeat terminal. Green lines return as the primary ALB starts accepting traffic again.
  • Stop the heartbeat script by pressing Ctrl+C.

Heartbeat still showing failures after restore?

Security group changes take effect almost instantly, but your ALB health checks need time to mark targets as healthy again. Wait 10-15 seconds and the green lines should return.

If failures persist after 30 seconds, confirm the restore worked by opening the primary ALB URL in your browser. If the page loads, the restore succeeded and the heartbeat script may need restarting.

help me debug why my primary ALB isn't recovering after running chaos.sh restore

You just felt the problem firsthand. A healthy secondary region exists, but without a failover mechanism, users have no way to reach it. Next up, you'll build the shield that makes this invisible to your users.

Build Your Failover Shield

In the previous step, you watched your primary ALB go dark and take all users with it. That failure exposed the core problem: a single-region endpoint gives you zero protection.

Now you'll place a global shield in front of both regions. CloudFront Origin Groups solve this by sitting at the edge layer and automatically routing requests to your secondary region when the primary returns errors. No manual intervention, no DNS propagation delay.

In this step, get ready to:
  • Create a CloudFront distribution with two origins pointing to your regional ALBs.
  • Configure an Origin Group that fails over from primary to secondary on server errors.
  • Set up Route 53 health checks to monitor both regional endpoints.
Create a CloudFront distribution with two origins

Your two ALBs are healthy but completely independent. A CloudFront distribution with an Origin Group ties them together at the edge, giving you a single URL that automatically fails over between regions.

STEP 1

  • Select the Free plan ($0/month) and click Next.

STEP 2

  • For Distribution name, enter multi-region-workshop-your_name .
  • Under Distribution type, confirm Single website or app is selected (it should be by default).
  • Leave the Route 53 managed domain field empty.
  • Click Next.

STEP 3

  • For Origin domain, Select Elastic Load Balancer. paste your primary ALB DNS name (the us-east-1 ALB URL from your deploy output, without the http:// prefix).
  • Under Settings, confirm Use recommended origin settings is selected (it should be by default).
  • Click Next.

STEP 4

  • Skip the WAF page and click Next.
  • Click Create distribution.
  • Click on your new distribution to open it.
  • Choose the Origins tab.
  • Click Create origin.
  • For Origin domain, paste your secondary ALB DNS name (the us-west-2 ALB URL from your deploy output, without the http:// prefix).
  • Under Settings, confirm Use recommended origin settings is selected.
  • Click Create origin.

What is WAF?

WAF (Web Application Firewall) is an AWS service that filters malicious web traffic like SQL injection and cross-site scripting. It's a paid add-on and not related to failover, so you can safely skip it for this project.

Your distribution now has two origins. Time to group them into a failover pair.

  • On the Origins tab, scroll down to the Origin groups pane.
  • Click Create origin group.
  • Select both origins and use the arrows to set the us-east-1 ALB as primary (top) and the us-west-2 ALB as secondary (bottom).
  • Enter a name for the origin group (e.g. multi-region-failover).
  • For the failover criteria, select these HTTP status codes: 500, 502, 503, 504.
  • Leave Origin selection criteria set to Default.
  • Click Create origin group.

What is an Origin Group?

An Origin Group pairs a primary and secondary origin. When the primary returns one of your configured error codes (500, 502, 503, 504), CloudFront automatically retries the request against the secondary origin.

This happens at the edge layer. No DNS record needs to change, no TTL needs to expire. The failover is invisible to the user.

One last step: you need to point the default cache behavior at your new Origin Group instead of a single origin.

  • Choose the Behaviors tab.
  • Select the default behavior (Default (*)) and click Edit.
  • Under Origin and origin groups, select your new Origin Group from the dropdown.
  • Click Save changes.

CloudFront will now deploy your updated distribution. This takes 5-10 minutes.

Use the wait time wisely

While CloudFront deploys, you'll set up Route 53 health checks in the next substep. The distribution doesn't need to finish deploying before you can continue.

🙋‍♀️ Don't see your ALB in the Origin domain dropdown?

The dropdown only shows resources in the same account. If your ALB doesn't appear, paste the DNS name directly into the field. You can find it in your .workshop-config file or in the EC2 console under Load Balancers.

Make sure you're pasting just the DNS hostname (e.g. my-alb-123456.us-east-1.elb.amazonaws.com) without http:// in front.

help me troubleshoot my CloudFront origin setup

Configure Route 53 health checks

CloudFront Origin Group failover is driven by HTTP status codes from the origin. Route 53 health checks give you independent monitoring so you can see at a glance which regions are healthy.

Configure the first health check for your primary region:

  • Choose Specify endpoint by: Domain name.
  • For the protocol, select HTTP.
  • Enter your primary ALB DNS name in the Domain name field.
  • Set Port to 80.
  • Set Path to /health.
  • Click Next.
  • For the health check name, enter Primary-Region-Health.
  • Click Create health check.

Now create the second health check for your secondary region:

  • Click Create health check again.
  • Choose Specify endpoint by: Domain name.
  • For the protocol, select HTTP.
  • Enter your secondary ALB DNS name in the Domain name field.
  • Set Port to 80.
  • Set Path to /health.
  • Click Next.
  • For the health check name, enter Secondary-Region-Health.
  • Click Create health check.

Wait 30-60 seconds, then check the Status column in the health checks list. Both health checks should show Healthy.

Why set up health checks if CloudFront handles failover?

CloudFront Origin Group failover is driven by HTTP status codes from the origin, not by Route 53 health checks. These health checks serve a different purpose: they give you an independent dashboard showing regional health at a glance.

During an incident, you'll be able to confirm failover happened by checking the Route 53 console. The primary health check flipping to "Unhealthy" is your confirmation signal.

🙋‍♀️ Health check stuck on unknown status?

Route 53 considers a new health check healthy until enough data arrives. If the status column shows no data after 60 seconds, verify your ALB's security group allows inbound HTTP on port 80 from any source (Route 53 health checkers use various IP ranges).

help me debug my Route 53 health check that isn't showing a healthy status

Test CloudFront routing

Your CloudFront distribution should be deployed by now (or close to it). Time to confirm it's routing traffic to your primary region.

  • Return to the CloudFront console and open your distribution.
  • Check the Last modified field. If it still shows Deploying, wait a few more minutes until it changes to a date and time.
  • Copy the distribution domain name (the *.cloudfront.net URL shown in the distribution details).

Before you open this URL, what do you think you'll see? Think about which origin CloudFront routes to by default.

  • Open the CloudFront distribution URL in your browser.

The app displays "us-east-1" with your instance details. CloudFront is successfully routing to the primary origin.

Finally, update your workshop config so the other scripts can use this URL:

  • Open .workshop-config in your text editor and add your CloudFront distribution URL to the file.

What does the .workshop-config update do?

The heartbeat and chaos scripts read this file to know which URL to target. By adding the CloudFront URL, you're telling those scripts to test through the failover layer instead of hitting the ALBs directly.

🙋‍♀️ Seeing a CloudFront error page instead of your app?

If you see a CloudFront error (e.g. 502 Bad Gateway), check that your Origin Group's default cache behavior is set to the Origin Group, not a single origin. Also confirm the ALB DNS name was entered without http:// and that the protocol is set to HTTP only.

help me troubleshoot a 502 error from my CloudFront distribution

Your failover shield is live. A single CloudFront URL now fronts both regional ALBs, with automatic failover on server errors. Next up, you'll put this shield under fire and measure exactly how long recovery takes.

Simulate the Incident

Your CloudFront Origin Group is deployed, routing traffic through the primary origin in us-east-1 with a secondary in us-west-2 standing by. On paper, the failover shield is complete. But you haven't proven it actually works under pressure.

In this step, you'll simulate a real regional outage and watch what happens. More importantly, you'll measure exactly how long recovery takes. This is the difference between "we think we're resilient" and "we measured our RTO at X seconds."

In this step, get ready to:
  • Start a heartbeat monitor against your CloudFront distribution and inject a simulated regional outage.
  • Observe live failover from us-east-1 to us-west-2 and measure the exact recovery time.
  • Restore the primary region and save your session data for the incident report.
Start monitoring and inject chaos

You need two terminal windows for this substep. One runs the heartbeat monitor continuously, and the other injects the failure. The heartbeat script polls your CloudFront URL and reports which region is responding.

  • Open a second terminal window alongside your existing one.
  • In the first terminal, start the heartbeat monitor against your CloudFront URL by running this command:
./scripts/heartbeat.sh <cloudfront-url>

What does this command do?

The heartbeat script sends repeated HTTP requests to your CloudFront distribution and parses the /api/status JSON endpoint. Each line shows the responding region, HTTP status code, and response latency. It highlights the moment failover occurs and calculates the total recovery time.

  • Watch the heartbeat terminal. You should see green lines scrolling with 200 OK responses from us-east-1.

Not seeing 200 OK responses?

Make sure you're using the CloudFront distribution URL (*.cloudfront.net), not the ALB URL directly. Check that your .workshop-config file has the correct CloudFront URL from Step 3.

If you see 403 errors, your CloudFront distribution may still be deploying. Wait a few minutes and try again.

Help me troubleshoot my CloudFront heartbeat issue

Now for the moment of truth. Before you inject the failure, think about what you expect to happen.

In Step 2, you broke the primary ALB directly and traffic failed completely. Now you have a CloudFront Origin Group sitting in front of both regions. Do you think requests will fail the same way, or will something different happen?

  • Switch to your second terminal window.
  • Inject the simulated outage by running this command:
./scripts/chaos.sh break

What does the chaos script do?

The chaos script revokes the inbound HTTP security group rules on the primary ALB in us-east-1. This instantly blocks all HTTP traffic to your primary region, simulating a network partition. The effect is immediate.

  • Switch back to your heartbeat terminal and watch what happens.

After a brief pause, you'll see the heartbeat output shift. Instead of complete failure like in Step 2, traffic redirects to us-west-2. The heartbeat script highlights the moment with a message like: "FAILOVER DETECTED - Recovery time: Xs".

Why does failover take about 30 seconds?

By default, CloudFront attempts to connect to your primary origin 3 times, with a 10-second timeout for each attempt. That's up to 30 seconds of waiting before CloudFront gives up on the primary and routes the request to the secondary origin in the Origin Group.

You'll optimize this in the next step to bring recovery time down dramatically.

Observe and measure the failover

The failover is live. Your primary region is down and CloudFront is routing traffic to the secondary. Time to confirm the failover from multiple angles.

  • Note the recovery time displayed by your heartbeat script. With default settings, expect approximately 30 seconds.
  • Open the Route 53 console in your browser.
  • Click Health checks in the left sidebar.
  • Confirm that Primary-Region-Health now shows Unhealthy status. This confirms Route 53 detected the outage independently.
  • Open the CloudFront console.
  • Click on your distribution, then select the Origins tab.
  • Confirm that your Origin Group shows the failover was triggered.
  • Open your CloudFront distribution URL in a browser tab.
  • The app now displays us-west-2 as the responding region, along with a SECONDARY REGION banner. Your users would see the same app, just served from a different region.

This is the payoff. In Step 2, breaking the primary region meant total downtime. Now, with the CloudFront Origin Group in place, users experienced a brief pause but zero errors on their end.

Restore the primary region

You've confirmed the failover works. Time to bring the primary region back online and confirm that traffic returns automatically.

  • In your second terminal, restore the primary region by running:
./scripts/chaos.sh restore
  • Switch back to the heartbeat terminal and watch. After a short period, traffic returns to us-east-1. CloudFront stops failing over once the primary origin starts returning healthy responses again.
  • Stop the heartbeat script by pressing Ctrl+C.

The script prints a session summary with total requests, success rate, failover count, and recovery time. This is the data you'll feed into your Incident Dashboard Report in the next step.

  • Copy and save the session summary output. Paste it into a text file or keep the terminal open for reference.

What should I see in the session summary?

Your summary should show: total requests sent during the session, a success rate close to 100% (requests that got a valid response from either region), one failover event, and a recovery time of approximately 30 seconds.

The 30-second recovery time is the baseline measurement. In Step 5, you'll tune CloudFront's connection settings to dramatically reduce this.

🙋‍♀️ Traffic not returning to us-east-1 after restore?

CloudFront may cache the failover state briefly. Wait 60 seconds after running the restore command, then check again. You can also open a fresh browser tab to your CloudFront URL to see the current routing.

If the heartbeat still shows us-west-2 after a full minute, confirm the restore command succeeded by checking your primary ALB URL directly in your browser.

Help me troubleshoot why traffic isn't returning to my primary region

  • Return to the Route 53 console and confirm that Primary-Region-Health is back to Healthy status.

You've just proven your architecture survives a regional outage with measured data. Thirty seconds of recovery time is solid, but it's not great. In the next step, you'll tune CloudFront to cut that recovery time down to under 5 seconds, discover a critical limitation of this approach, and generate a polished Incident Dashboard Report from your session data.

Harden, Break Again, and Present Your Findings

In the last step, you proved your failover shield works. Traffic shifted from us-east-1 to us-west-2 automatically.

But 30 seconds of recovery time is slow. In production, that means half a minute of degraded experience for every user hitting your app. Your shield also has a blind spot you haven't discovered yet. In this step, you'll optimize failover speed, uncover a critical limitation of CloudFront Origin Groups, and use AI to generate a polished Incident Dashboard Report from your real measured data.

In this step, get ready to:
  • Optimize CloudFront failover from ~30 seconds to under 5 seconds by tuning connection settings.
  • Discover a critical limitation of Origin Group failover that leaves write operations unprotected.
  • Generate a professional AI-powered Incident Dashboard Report with real failover metrics.
Optimize CloudFront failover timing

The default failover timing is 30 seconds because CloudFront makes 3 connection attempts with a 10-second timeout each. You can tune both of those values to cut recovery time dramatically.

  • Return to the CloudFront console in AWS.
  • Select your distribution, then click the Origins tab.
  • Select your primary origin (the us-east-1 ALB) and click Edit.
  • Set Connection timeout to 2 seconds.
  • Set Connection attempts to 1.
  • Click Save changes.

The distribution status changes to Deploying. Wait 2-3 minutes for it to return to Enabled.

What did you just change?

With 1 attempt and a 2-second timeout, CloudFront now waits at most 2 seconds before failing over to your secondary origin. The tradeoff is less tolerance for slow but healthy responses from the primary. If your primary takes longer than 2 seconds to respond (even if it is healthy), CloudFront will route to the secondary.

In production, you would tune these values based on your app's normal latency profile. For this workshop, 2 seconds is aggressive enough to demonstrate the improvement clearly.

Before you run the next chaos test, what recovery time do you expect to see now that you have set the timeout to 2 seconds with 1 attempt?

  • Start the heartbeat monitor against your CloudFront URL by running this command:
./scripts/heartbeat.sh [[CLOUDFRONT_URL="your-distribution.cloudfront.net"]]

You should see green lines scrolling with 200 OK responses from us-east-1.

  • In a second terminal, inject chaos by running:
./scripts/chaos.sh break

Watch the heartbeat terminal. After a brief pause of about 2 seconds, traffic shifts to us-west-2. The heartbeat script highlights the failover moment and prints the measured recovery time.

Your recovery time should now be under 5 seconds. Compare that to the ~30 seconds you measured in the previous step.

  • Stop the heartbeat monitor by pressing Ctrl+C.
  • Save the session summary output. You now have two summaries: the baseline (~30 seconds) and the optimized (under 5 seconds).

Why keep both session summaries?

You will feed both summaries into an AI prompt in the final substep. The before-and-after comparison makes your Incident Dashboard Report compelling because it shows a measurable improvement backed by real data.

🙋‍♀️ Recovery time still showing 30 seconds?

Make sure the distribution finished redeploying before you ran chaos. The status must show Enabled (not Deploying) for the new settings to take effect.

Also confirm you edited the correct origin (the primary us-east-1 ALB origin, not the secondary).

Help me troubleshoot my CloudFront failover timing. I changed the connection timeout to 2 seconds and attempts to 1, but recovery is still slow.

Discover the POST limitation

Your primary is still broken from the chaos injection. Traffic is flowing to us-west-2 for all the GET requests the heartbeat was making. But what about write operations?

What do you think will happen if you send a POST request through CloudFront while the primary is down?

  • Test a POST request against your CloudFront URL by running:
curl -X POST [[CLOUDFRONT_URL="your-distribution.cloudfront.net"]]/health

The POST request fails. You get an error response even though the secondary region is healthy and serving GET traffic just fine.

Why does the POST fail?

CloudFront Origin Groups only fail over GET, HEAD, and OPTIONS requests. POST, PUT, DELETE, and other methods are not failed over. They are sent to the primary origin and fail if it is unavailable.

This means login forms, checkout flows, API writes, and any other non-GET traffic all break during a regional outage, even with Origin Group failover configured. CloudFront alone does not protect write operations.

💡 How would you protect write traffic?

Three common approaches exist. Route 53 DNS failover can route the API layer separately based on health checks. Application-level retry logic can redirect failed writes to a secondary endpoint. A separate write-path architecture can decouple reads from writes entirely.

In production, most teams combine CloudFront Origin Group failover for read traffic with Route 53 DNS failover for the API/write layer.

  • Restore the primary region by running:
./scripts/chaos.sh restore

Your heartbeat (if still running) shows traffic returning to us-east-1. Both regions are healthy again.

Generate your Incident Dashboard Report

You now have real failover data: a baseline session with ~30 second recovery, an optimized session with sub-5 second recovery, and a documented limitation around POST requests. Time to turn these findings into a presentation-ready artifact.

The workshop repo includes a prompt template designed for this purpose. You will feed it your real data and an AI will generate a self-contained HTML dashboard.

  • Open Claude or ChatGPT in your browser.
  • Open the prompt template file in the workshop repo at prompts/incident-dashboard.md.
  • Copy the full contents of the prompt template.
  • Paste it into your AI chat.
  • Below the prompt, paste your two heartbeat session summaries: the baseline (~30 second recovery) and the optimized (under 5 seconds).
  • Send the message and wait for the AI to generate the HTML output.

What does the prompt ask the AI to generate?

The prompt instructs the AI to generate a complete, self-contained HTML file styled as a professional incident dashboard. It includes an executive summary, a color-coded incident timeline, key metrics cards showing RTO before and after optimization, an architecture diagram, a What Worked / What Didn't section, the POST limitation finding, and production readiness recommendations.

This is the same format engineering teams present to leadership after real incidents.

  • Copy the generated HTML from the AI response.
  • Save it as report.html in your workshop-resilience-app folder by running:
# Paste the AI-generated HTML into this file
# Use your preferred method: pbpaste on macOS, or open in a text editor
cat > report.html << 'EOF'
# Paste your HTML content here, then type EOF on a new line
EOF

Easier way to save the file

You can also create report.html in any text editor. Open a new file, paste the AI-generated HTML, and save it as report.html in your workshop-resilience-app folder.

What do you expect to see when you open this file in your browser?

  • Open report.html in your browser by running:
open report.html

Your Incident Dashboard Report opens in the browser. You should see a professional, styled page with your real failover metrics, the timeline of events, and actionable recommendations.

Report looks broken or unstyled?

Make sure you copied the complete HTML output from the AI, including the opening <html> tag and closing </html> tag. Partial copies often lose the CSS styling.

If the report is missing your data, re-check that you pasted both heartbeat session summaries into the prompt before sending it.

Help me fix my AI-generated incident dashboard report. It's not rendering correctly in the browser.

Secret mission

Cross-Region Disaster Recovery Cutover

CloudFront Origin Groups handle temporary blips. But what if us-east-1 is down for hours or days? The read replica in us-west-2 is read-only. In this mission, you'll perform a full disaster recovery cutover: promote the replica, scale up compute, and make Region B fully self-sufficient.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project deployed infrastructure across two AWS regions, and several resources bill by the hour even when idle.

Cost warning

Your multi-region stacks include NAT Gateways ($0.045/hour each), RDS instances, Application Load Balancers, and EC2 instances. Together, these resources cost approximately $1-2/hour across both regions.

If left running overnight, expect $25-50/day in charges. NAT Gateways are the biggest surprise cost driver. Prioritize cleanup immediately after completing the workshop.

Resources you used:

  • CloudFormation stack in us-east-1 (VPC, subnets, NAT Gateways, Internet Gateway, ALB, ASG with EC2 instances, RDS primary instance).
  • CloudFormation stack in us-west-2 (VPC, subnets, NAT Gateways, Internet Gateway, ALB, ASG with EC2 instances, RDS read replica or standalone instance if you completed the Secret Mission).
  • CloudFront distribution with Origin Group.
  • 2 Route 53 health checks (Primary-Region-Health and Secondary-Region-Health).
  • Local workshop repository folder (workshop-resilience-app) including report.html.

Keep everything running

No action needed. Choose this only if you are actively building on top of these stacks right now and plan to clean up within the next hour or two.

  • Your two CloudFormation stacks remain deployed across us-east-1 and us-west-2.
  • Your CloudFront distribution continues routing traffic to your Origin Group.
  • Route 53 health checks remain active.
  • NAT Gateways bill $0.045/hour each regardless of traffic. Do not leave these running overnight.

Pause - I'll come back to this later

Delete the AWS resources now and keep the local repository. Since deployment is fully automated via CloudFormation, you can redeploy the entire stack anytime by running ./scripts/deploy.sh again.

  • Run the cleanup script to delete all AWS resources:
  • Keep the workshop-resilience-app folder on your machine. Your report.html and heartbeat session summaries are saved there.
  • When you return, run ./scripts/deploy.sh to recreate both regional stacks from scratch.
./scripts/cleanup.sh

Delete - I don't want to use this again

Remove all project resources completely. This deletes your AWS infrastructure and the local project folder.

First, delete all AWS resources by running the cleanup script:

./scripts/cleanup.sh

What does this script do?

The cleanup script deletes resources in the correct dependency order. It disables and deletes the CloudFront distribution first (this includes a wait period), then removes the Route 53 health checks, then deletes the secondary CloudFormation stack (us-west-2), and finally deletes the primary stack (us-east-1).

After the script completes, delete the local project folder. On macOS or Linux:

cd ~/Desktop
rm -rf workshop-resilience-app

On Windows:

cd ~\Desktop
rmdir /s /q workshop-resilience-app
  • Verify in the AWS CloudFormation console that both stacks show DELETE_COMPLETE in us-east-1 and us-west-2.
  • Verify in the CloudFront console that your distribution is gone.
  • Verify in Route 53 that both health checks are removed.

Nice Work!

Nice Work!

You just survived a regional outage. You deployed a multi-region application stack on AWS, configured automated edge-level failover, injected chaos to prove it works, and walked away with a professional Incident Dashboard Report backed by real measured data.

You've learned how to:

  • Deploy a multi-region application stack across us-east-1 and us-west-2 using CloudFormation, then configure a CloudFront Origin Group that automatically fails over to a secondary origin on 5xx errors with zero manual intervention.
  • Simulate a regional outage using chaos injection, measure exact recovery time with a heartbeat monitor, and optimize failover from 30 seconds down to under 5 seconds by tuning CloudFront connection settings.
  • Generate a presentation-ready Incident Dashboard Report using AI from real failover data, complete with timeline visualization, metrics cards, and production readiness recommendations.
  • (Secret Mission) Perform a full disaster recovery cutover by promoting an RDS read replica to a standalone primary, scaling up secondary compute, and achieving near-zero RPO with measurable RTO.

Ready to quiz yourself?