Simulate Multi-Region Failover on AWS

Introduction

30 Second Summary

When your favorite app goes down, it is usually because the servers in one location failed and nothing was ready to take over. The best platforms prevent this by running backup servers in other locations that take over automatically.

In this project, you will build a multi-region failover system on AWS. You will deploy web servers in two regions, wire them together with health checks, then stop the primary and watch traffic reroute to the backup on its own.

What You'll Build

You will stop your primary web server mid-project, wait a couple of minutes, and watch your domain start serving traffic from a completely different region with zero manual intervention.

By the end of this project, you'll have:

• Two web servers running in separate AWS regions, each displaying which region is currently serving your request.
• Route 53 health checks that continuously monitor each server and automatically detect when one goes down.
• Failover routing records that redirect traffic from the unhealthy primary to the healthy secondary without any manual steps.
• Secret Mission: CloudWatch alarms and SNS email notifications that alert you the moment a failover event occurs.

What does this look like in the real world?

Production SaaS platforms take the failover pattern you are building and layer it into a much larger stack. A typical multi-tenant architecture routes users through Route 53 and CloudFront at the edge, filters traffic through a WAF and Application Load Balancer, then distributes requests across Fargate containers and EC2 instances backed by RDS databases and S3 storage.

Each tenant gets its own isolated infrastructure (separate subnets, databases, and storage), while shared services like Cognito for authentication, KMS for encryption, and GuardDuty for threat detection sit across the whole platform. The Route 53 failover routing you build in this project is the same mechanism that keeps those enterprise systems available when an entire region goes dark.

💡 Are there any prerequisites?

You will need an AWS account and the AWS CLI installed and configured on your machine before starting this project.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Launch Web Servers in Two AWS Regions

Before you can wire up any failover logic, you need actual endpoints in two separate AWS regions. Route 53 needs something to point at, and that means running web servers.

In this step, you will launch two EC2 instances, one in us-east-1 (N. Virginia) and one in us-west-2 (Oregon). Each instance will run a simple web server that identifies which region is serving the request.

In this step, get ready to:

• Launch a primary EC2 instance in us-east-1 with a startup script that serves a web page.
• Launch a secondary EC2 instance in us-west-2 with a matching setup.
• Confirm both web servers respond in the browser.

Launch the primary EC2 instance in us-east-1

• In your browser, navigate to console.aws.amazon.com.
• Sign in with your AWS account credentials.
• In the top-right corner, click the region dropdown and select US East (N. Virginia) us-east-1.
• In the search bar at the top, type EC2 and select EC2 from the results.
• On the EC2 dashboard, click Launch instance.
• Under Name and tags, for Name, enter primary-server.
• Under Application and OS Images (Amazon Machine Image), click Quick Start and choose Amazon Linux.
• From the Amazon Machine Image (AMI) dropdown, select an AMI marked Free tier eligible that says Amazon Linux 2023.

Which instance type is free tier eligible?

Under Instance type, look for the Free tier eligible label. If your account was created before July 15, 2025, both t2.micro and t3.micro qualify under the 12-month free tier.

If your account was created on or after July 15, 2025, t3.micro and several other types are eligible for 6 months. Select whichever type shows the Free tier eligible label in the console.

• Under Instance type, select a free-tier-eligible type (t2.micro or t3.micro).
• Under Key pair (login), select an existing key pair. If you do not have one, click Create new key pair, give it a name like failover-key, keep the defaults, and click Create key pair.

Save your key pair file

When you create a new key pair, the browser downloads a .pem file. Save it somewhere you can find it later. You will need this file to SSH into your instances in a later step.

Now configure the network rules that control which traffic can reach your instance.

• Under Network settings, click Edit.
• Verify the existing SSH rule has Source type set to My IP.
• Click Add security group rule.
• For Type, select HTTP.
• For Source type, select Anywhere. This allows HTTP traffic on port 80 from any IP address so Route 53 health checks can reach the server.

What is a security group?

A security group acts as a virtual firewall for your EC2 instance. The SSH rule lets you connect remotely from your machine. The HTTP rule lets web browsers (and Route 53 health checks) reach your web server on port 80.

The last step is to add a startup script that installs a web server when the instance first boots.

• Scroll down and expand the Advanced details section.
• Scroll to the User data text box at the bottom and paste this startup script:

#!/bin/bash
# Update all installed packages
dnf update -y
# Install Apache web server
dnf install -y httpd
# Start Apache immediately
systemctl start httpd
# Enable Apache to restart automatically after reboot
systemctl enable httpd
# Create a simple web page identifying this region
echo "Hello from us-east-1 (PRIMARY)" > /var/www/html/index.html

What does this script do?

This is a User Data script. EC2 runs it automatically as the root user when the instance first launches.

It installs Apache (httpd), starts the web server, enables it to survive reboots, and writes a simple HTML page to /var/www/html/index.html. Anyone visiting the instance's public IP on port 80 sees that page.

• In the Summary panel on the right, review your settings and click Launch instance.
• Click the instance ID link in the success notification to open the Instances page.
• Wait for the Instance state column to show Running. This takes about 30 to 60 seconds.

Instance stuck in Pending?

Refresh the Instances page. If the instance moves to Terminated instead of Running, check the launch details for an error. A common cause is selecting an instance type not available in your Availability Zone.

help me troubleshoot my EC2 instance that is not reaching Running state in us-east-1

Launch the secondary EC2 instance in us-west-2

Now repeat the process for the secondary region. The setup is almost identical, with a different region and a different message on the web page.

• In the top-right corner of the console, click the region dropdown and select US West (Oregon) us-west-2.
• Click Launch instance.
• Under Name and tags, for Name, enter secondary-server.
• Under Application and OS Images (Amazon Machine Image), choose Amazon Linux and select an Amazon Linux 2023 AMI marked Free tier eligible.
• Under Instance type, select the same free-tier-eligible type you used before (t2.micro or t3.micro).
• Under Key pair (login), create a new key pair for this region (e.g. failover-key-west), or select one if you already have a key pair in us-west-2.
• Under Network settings, click Edit. Verify the SSH rule has Source type set to My IP.
• Click Add security group rule, set Type to HTTP, and set Source type to Anywhere.

Why a new key pair?

Key pairs are region-specific in AWS. A key pair created in us-east-1 does not exist in us-west-2. You need a separate key pair in each region to SSH into the instances there.

• Expand Advanced details and scroll down to the User data text box. Paste this startup script:

#!/bin/bash
# Update all installed packages
dnf update -y
# Install Apache web server
dnf install -y httpd
# Start Apache immediately
systemctl start httpd
# Enable Apache to restart automatically after reboot
systemctl enable httpd
# Create a simple web page identifying this region
echo "Hello from us-west-2 (SECONDARY)" > /var/www/html/index.html

What changed in this script?

The only difference from the primary script is the last line. It writes Hello from us-west-2 (SECONDARY) instead of Hello from us-east-1 (PRIMARY). This is how you will tell which region is serving traffic after failover.

• Click Launch instance.
• Click the instance ID link and wait for the Instance state to show Running.

Confirm both web servers respond

Both instances are running, but you have not yet confirmed that the web servers actually started and are serving pages.

• On the Instances page for us-west-2, click your secondary-server instance.
• In the instance details, find and copy the Public IPv4 address. Write it down somewhere you can reference later.
• Switch the region back to US East (N. Virginia) us-east-1 using the dropdown.
• Click your primary-server instance and copy its Public IPv4 address as well.

Before you open these IPs in your browser, what do you expect to see on each page?

• Open a new browser tab and navigate to http:// followed by the primary instance's public IP address.
• Open another tab and navigate to http:// followed by the secondary instance's public IP address.

You should see Hello from us-east-1 (PRIMARY) on the first page and Hello from us-west-2 (SECONDARY) on the second.

Seeing a connection timeout or refused?

• Make sure you are using http:// and not https://. The server only listens on port 80 (HTTP).
• Check that the security group has an inbound rule for HTTP on port 80 with source 0.0.0.0/0.
• The User Data script can take 1 to 2 minutes to finish after the instance reaches Running state. Wait a moment and refresh the page.
• If you still see nothing, SSH into the instance and check the Apache status by running systemctl status httpd.

help me figure out why my EC2 instance is not serving a web page on port 80

Both servers are live and responding from separate AWS regions. Next up, you will create a Route 53 hosted zone and set up DNS so you can point a domain name at these servers instead of using raw IP addresses.

Create a Route 53 Hosted Zone and Test Without Failover

You now have two web servers running in separate AWS regions, each displaying which region is serving the request.

Your servers are reachable by IP address, but IP addresses are hard to remember and impossible to redirect automatically. DNS solves this by mapping a human-readable domain name to an IP address. In this step, you will create a Route 53 private hosted zone and test basic DNS resolution so you understand how it works before adding failover logic.

In this step, get ready to:

• Create a Route 53 private hosted zone and associate it with both VPCs.
• Create a basic A record and test DNS resolution from an EC2 instance.
• Delete the basic record to prepare for failover routing.

Create a Route 53 private hosted zone

A hosted zone is a container that tells Route 53 how to respond to DNS queries for a domain. A private hosted zone keeps that domain internal to your VPCs, so you do not need to buy or register a real domain name.

• In the AWS Management Console, search for Route 53 in the top search bar and open the service.
• In the left navigation pane, click Hosted zones.
• Click Create hosted zone.
• In the Domain name field, enter failover.internal.
• For Type, select Private hosted zone.
• In the VPC ID section, select the Region us-east-1 and choose the default VPC for that region.
• Click Create hosted zone.

You should see your new hosted zone listed with the domain failover.internal and a type of Private. Two default records (NS and SOA) are created automatically.

What is a private hosted zone?

A public hosted zone requires you to own a registered domain name. A private hosted zone lets you invent any domain name (like failover.internal) and resolve it only inside your VPCs.

This keeps the project free and avoids domain registration costs.

Right now the hosted zone is only associated with the us-east-1 VPC. You need to add the us-west-2 VPC so both regions can resolve the domain.

• On the Hosted zones page, select the radio button next to failover.internal.
• Click Edit.
• Click Add VPC.
• Select the Region us-west-2 and choose the default VPC for that region.
• Click Save changes.

Your hosted zone is now associated with both VPCs. EC2 instances in either region can resolve names under failover.internal.

Don't see the us-west-2 VPC in the dropdown?

Make sure you select the correct Region (us-west-2) from the Region dropdown before choosing a VPC ID. Each VPC is region-specific, so you will only see VPCs for the region you select.

Also confirm that the VPC has DNS resolution and DNS hostnames enabled. These settings are on by default for the default VPC, but if they were changed, the hosted zone cannot resolve names inside that VPC.

Still stuck? Help me associate a second VPC with my Route 53 private hosted zone.

Create a basic A record and test DNS resolution

Before wiring up failover, you need to confirm that DNS resolution works at all inside your VPCs. You will create a simple A record that maps app.failover.internal to the primary instance's IP address.

• Click the name of your failover.internal hosted zone to open it.
• Click Create record.
• In the Record name field, enter app (this creates app.failover.internal).
• For Record type, keep the default A.
• In the Value field, enter the public IPv4 address of your us-east-1 (primary) EC2 instance.
• Leave Routing policy as Simple routing.
• Click Create records.

What is an A record?

An A record maps a domain name to an IPv4 address. When something queries app.failover.internal, Route 53 responds with the IP address you entered.

This basic record has no routing policy and no health check. It always returns the same IP, even if that server is down.

Now test that DNS resolution works by querying the record from inside one of your VPCs. You will SSH into one of your EC2 instances and run nslookup.

• Open a terminal on your local machine.
• Connect to one of your EC2 instances (either region works) by running:

ssh -i [[KEYPAIR_PATH="path to your .pem key file"]] ec2-user@[[EC2_PUBLIC_IP="your EC2 instance public IP"]]

Which instance should I SSH into?

Either instance works because both VPCs are associated with the hosted zone. Use whichever instance's key pair you have handy.

• If prompted about the host fingerprint, type yes and press Enter.
• Before you run the next command, think about what you expect to see. What IP address should app.failover.internal resolve to?
• Run this command to test DNS resolution:

nslookup app.failover.internal

You should see output that includes the public IP address of your us-east-1 (primary) instance. This confirms that Route 53 is resolving your private domain name correctly inside the VPC.

Getting a 'can't find app.failover.internal' error?

Make sure the VPC you are SSH'd into is associated with the failover.internal hosted zone. If you only associated us-east-1 and you SSH'd into the us-west-2 instance, go back and add the us-west-2 VPC.

DNS propagation can take up to 60 seconds. Wait a minute and try again.

Still stuck? Help me troubleshoot nslookup failing for my Route 53 private hosted zone.

Delete the basic record and understand the gap

DNS resolution works, but this basic A record is static. It always points to the primary server's IP address no matter what. If the primary goes down, DNS keeps sending traffic to a dead endpoint.

You need to delete this basic record so you can replace it with failover routing records in the next step.

• Return to the Route 53 console in your browser.
• Click the name of your failover.internal hosted zone to view its records.
• Select the row for the app.failover.internal A record (do not select the NS or SOA records).
• Click Delete.
• Click Delete again to confirm.

Your hosted zone now has only the default NS and SOA records. DNS resolution for app.failover.internal no longer works.

Why delete the record you just created?

A basic A record with simple routing cannot do failover. It has no health check and no awareness of whether the server is alive.

In the next step, you will create failover routing records that automatically switch traffic between your primary and secondary servers based on health check status. Those records replace this basic one.

You have confirmed that DNS resolution works inside your VPCs and you understand the limitation of a static record. Next up, you will add health checks and failover routing to make DNS respond to outages automatically.

Add Health Checks and Failover Routing

In the previous step, you proved that DNS resolution works inside your private hosted zone. You created a basic A record, tested it with nslookup, and then deleted it.

The problem is clear: a basic record blindly points to one IP address. If that server dies, DNS keeps sending traffic to a dead endpoint. Health checks solve this by constantly probing each server and telling Route 53 when one is down. Paired with failover routing records, Route 53 can automatically switch traffic to the healthy server.

In this step, get ready to:

• Create health checks that monitor each EC2 instance.
• Create failover routing records that automatically switch traffic when the primary is unhealthy.
• Verify that DNS resolves to the primary instance while both servers are healthy.

Create two Route 53 health checks

Route 53 health checks are automated probes that continuously ping an endpoint over HTTP, HTTPS, or TCP. When an endpoint fails a configurable number of consecutive checks, Route 53 marks it unhealthy and stops routing traffic to it.

You will create one health check for each EC2 instance so Route 53 knows the status of both regions at all times.

• In the AWS Management Console, open the Route 53 console.
• In the left navigation pane, click Health Checks.
• Click Create health check.
• In the Name field, enter primary-us-east-1.
• For What to monitor, select Endpoint.
• For Specify endpoint by, select IP address.
• In the protocol dropdown, select HTTP.
• In the IP address field, enter the public IPv4 address of your us-east-1 EC2 instance.
• Set Port to 80.
• Set Path to /.
• Leave all other settings at their defaults and click Create health check.

What does a health check actually do?

Route 53 health checkers are servers distributed across multiple AWS data centers worldwide. Each one sends an HTTP request to your instance's public IP on port 80 every 30 seconds (the default interval).

If an endpoint fails 3 consecutive checks (the default failure threshold), Route 53 marks it unhealthy. That takes roughly 90 seconds (3 checks at 30-second intervals).

You should see your new health check listed with a Status column. Route 53 considers a new health check healthy by default until enough data is gathered to determine actual status.

• Now create the second health check. Click Create health check again.
• In the Name field, enter secondary-us-west-2.
• For What to monitor, select Endpoint.
• For Specify endpoint by, select IP address.
• Select HTTP as the protocol.
• In the IP address field, enter the public IPv4 address of your us-west-2 EC2 instance.
• Set Port to 80.
• Set Path to /.
• Leave all other settings at their defaults and click Create health check.

• Wait 1-2 minutes, then refresh the health checks list.
• Confirm that both health checks show Healthy in the Status column.

Health check stuck on Unhealthy?

The most common cause is a security group that blocks inbound HTTP traffic. Go to the EC2 console for the affected region and confirm the instance's security group allows inbound TCP on port 80 from 0.0.0.0/0. Route 53 health checkers come from many different IP addresses, so the rule must be open to all sources.

Also verify the instance is in the running state and that you can reach its public IP in your browser on port 80.

Still stuck? Help me debug why my Route 53 health check shows unhealthy.

💡 Why are health checks on AWS endpoints free?

Route 53 health checks that monitor AWS endpoints (like EC2 instances within the same account) are free for up to 50 health checks. Health checks on non-AWS endpoints cost $0.75 per month each.

Create failover routing records

With health checks in place, Route 53 knows the status of each server. Now you need failover routing records that tell Route 53 what to do with that information: serve the primary IP when it is healthy, and automatically fall back to the secondary IP when it is not.

• In the Route 53 console, click Hosted zones in the left navigation pane.
• Click the failover.internal hosted zone name to open it.
• Click Create record.

You will create the primary failover record first. This record tells Route 53 to serve the us-east-1 IP as long as it is healthy.

• In the Record name field, enter app (so the full record name becomes app.failover.internal).
• For Record type, select A.
• For Routing policy, select Failover.

The form now shows failover-specific fields.

• For Failover record type, select Primary.
• For Value/Route traffic to, choose IP address or another value depending on the record type.
• Enter the public IPv4 address of your us-east-1 EC2 instance.
• Set TTL (seconds) to 60.
• For Health check, select the primary-us-east-1 health check you just created.
• For Record ID, enter primary.
• Click Create records.

Why set the TTL to 60 seconds?

TTL (Time to Live) tells DNS resolvers how long to cache a record before asking Route 53 again. A short TTL of 60 seconds means clients pick up changes quickly when a failover happens.

AWS recommends a TTL of 60 seconds or less when associating a record with a health check so clients respond quickly to changes in health status.

Now create the secondary failover record. This record tells Route 53 to serve the us-west-2 IP when the primary is unhealthy.

• Click Create record again.
• In the Record name field, enter app (same name as the primary record).
• For Record type, select A.
• For Routing policy, select Failover.
• For Failover record type, select Secondary.
• For Value/Route traffic to, choose IP address or another value depending on the record type.
• Enter the public IPv4 address of your us-west-2 EC2 instance.
• Set TTL (seconds) to 60.
• For Health check, select the secondary-us-west-2 health check.
• For Record ID, enter secondary.
• Click Create records.

You should now see two failover A records for app.failover.internal in your hosted zone alongside the default NS and SOA records.

Record creation failed?

Make sure both records use the exact same Record name (app) and the same Record type (A). Failover routing requires one Primary and one Secondary record with matching names and types.

Also verify you selected a different Failover record type for each record. You cannot have two Primary or two Secondary records for the same name.

Need more help? Help me fix my Route 53 failover record creation error.

Verify DNS resolves to the primary instance

Both health checks are healthy, and your failover records are in place. Time to verify that Route 53 is actually routing app.failover.internal to the primary instance.

Before you run this, what IP address do you expect nslookup to return? Since both servers are healthy, Route 53 should prefer the primary.

• SSH into one of your EC2 instances (either region works since both VPCs are associated with the hosted zone).
• Run this command to test DNS resolution:

nslookup app.failover.internal

The returned IP address should be the public IPv4 address of your us-east-1 (primary) instance.

• Switch back to the Route 53 console and click Health Checks in the left navigation pane.
• Confirm both health checks still show Healthy in the Status column.

What would happen if you stopped the primary right now?

Route 53 would detect the primary health check failing after about 3 consecutive missed checks (roughly 90 seconds with the default 30-second interval). Once marked unhealthy, Route 53 stops returning the primary IP and starts returning the secondary IP instead.

Add the 60-second TTL window for clients to pick up the change, and total failover time is roughly 1-3 minutes. You will see this happen for real in the next step.

🙋‍♀️ nslookup returns the wrong IP?

If the returned IP matches your us-west-2 (secondary) instance, check that your primary health check is healthy. Route 53 only serves the primary record when its associated health check is healthy.

If nslookup fails entirely, make sure you are running it from an EC2 instance inside a VPC that is associated with the failover.internal private hosted zone. Private hosted zones only resolve from within associated VPCs.

Still stuck? Help me debug why nslookup is not resolving to my primary instance.

Your failover system is fully wired. Health checks are monitoring both servers, and failover records will automatically reroute traffic if the primary goes down. Next up, you will deliberately break the primary and watch Route 53 heal itself in real time.

Simulate a Regional Outage and Observe Failover

Your failover routing is fully wired. Route 53 health checks are monitoring both EC2 instances, and the failover records are ready to redirect traffic if the primary goes down.

But the whole point of failover is automatic recovery. Until you break something and watch it heal, the concept stays theoretical. In this step, you will deliberately kill the primary server and watch Route 53 reroute traffic to the secondary in real time.

In this step, get ready to:

• Stop the primary EC2 instance to simulate a regional outage.
• Watch the Route 53 health check detect the failure and flip to unhealthy.
• Verify that DNS queries automatically resolve to the secondary instance.

Stop the primary EC2 instance

Stopping your primary instance simulates what happens during a real regional outage. Port 80 becomes unreachable, which is exactly what the Route 53 health check probes.

• In the AWS Management Console, use the region selector in the top-right corner to switch to us-east-1 (N. Virginia).
• Search for EC2 in the top search bar and select it.
• Click Instances in the left navigation pane.
• Select your primary instance by clicking its checkbox.
• Click Instance state at the top of the page.
• Click Stop instance.
• In the confirmation dialog, click Stop.

The Instance state column changes to Stopping, then Stopped within about 30 seconds.

Why stop instead of terminate?

Stopping an EC2 instance shuts it down but keeps it around so you can restart it later. Terminating permanently deletes it. You need this instance back in the next step to observe failback.

🙋‍♀️ Instance stuck in Stopping?

It can take a few minutes for the instance to fully stop. Wait up to 2 minutes before trying anything else. If it remains stuck, you can use Instance state > Force stop instance to force the shutdown.

My instance is stuck in the Stopping state. What should I do?

Watch the health check detect the failure

• Return to the Route 53 console by searching for Route 53 in the top search bar.
• Click Health checks in the left navigation pane.
• Find the health check for your primary (us-east-1) instance in the list.

Route 53 health checkers probe your endpoint every 30 seconds. After 3 consecutive failures, the health check flips from Healthy to Unhealthy. This takes about 90 seconds after the instance stopped.

• Refresh the health checks page every 30 seconds or so until the primary health check's Status column shows Unhealthy. The secondary health check should still show Healthy.

What happens behind the scenes?

Once the primary health check turns unhealthy, Route 53 stops returning the us-east-1 IP for DNS queries to app.failover.internal. It returns the secondary (us-west-2) IP instead.

This is completely automatic. No manual intervention, no scripts to run. Route 53 handles the switchover the moment it confirms the primary is down.

Confirm DNS now routes to the secondary

The health check has flipped. Now it is time to prove that DNS actually changed and traffic is flowing to the backup region.

• SSH into your us-west-2 instance. If you still have a session open from the previous step, switch back to it. Otherwise, connect by running this command in your local terminal:

ssh -i [[PEMPATH="path to your .pem key file"]] ec2-user@[[EC2_WEST_IP="your us-west-2 instance public IP"]]

Connection refused or timed out?

Make sure you are connecting to the us-west-2 instance, not the stopped us-east-1 instance. Double-check that the IP address matches the public IPv4 of your Oregon instance.

I cannot SSH into my us-west-2 EC2 instance. How do I troubleshoot the connection?

Before you run the next command, think about what you expect. The primary instance is stopped and Route 53 has marked it unhealthy. What IP address do you think nslookup app.failover.internal will return?

• Test DNS resolution by running this command from inside the us-west-2 instance:

nslookup app.failover.internal

The returned IP address should now be your us-west-2 instance's public IP. Route 53 has automatically switched DNS to point to the healthy secondary.

• Confirm the secondary is actually serving traffic by running:

curl app.failover.internal

You should see Hello from us-west-2 (SECONDARY). Traffic is now being served entirely by the backup region.

What just happened?

You just witnessed automatic multi-region failover. Route 53 detected that your primary server was down, updated its DNS responses, and redirected all traffic to the secondary.

In a real production environment, users would experience a brief interruption of 1-3 minutes before traffic seamlessly shifts to the backup region. The health check takes about 90 seconds to confirm the failure (3 checks at 30-second intervals), plus up to 60 seconds for DNS caches to expire based on the TTL you set.

🙋‍♀️ Still seeing the primary IP?

DNS responses are cached for the duration of the TTL (60 seconds in your configuration). Wait 60 seconds and run nslookup again.

If the health check still shows Healthy in the Route 53 console, wait another 30-60 seconds. It takes about 90 seconds (3 failures at 30-second intervals) for the check to flip.

Why is nslookup still returning my primary IP after I stopped the instance?

You have just witnessed automatic failover in action. But real disasters end eventually. In the next step, you will restart the primary and watch Route 53 automatically route traffic back.

Restore the Primary and Observe Failback

In the last step, you stopped the primary server and watched Route 53 automatically redirect traffic to the secondary region. That proved your failover architecture works under pressure.

Real disasters end. When they do, traffic should return to the primary region without anyone flipping a switch. In this step, you will restart the primary server and watch Route 53 detect the recovery and route traffic back automatically.

In this step, get ready to:

• Restart the primary EC2 instance and verify the web server is serving again.
• Update Route 53 with the new IP address and watch the health check flip back to healthy.
• Confirm DNS resolves to the primary IP again by running nslookup and curl.

Restart the primary EC2 instance

The primary instance is stopped, not terminated. That means all its storage and configuration are intact. You just need to start it back up.

• Return to the EC2 console and make sure the region selector in the top-right corner shows US East (N. Virginia) us-east-1.
• Click Instances in the left sidebar.
• Select the checkbox next to your stopped primary instance.
• Click Instance state in the top menu bar.
• Click Start instance.
• Wait for the Instance state column to show Running. You may need to click the refresh icon a few times.

Why does the public IP change?

When you stop and start an EC2 instance, AWS releases the old public IP address and assigns a new one from the pool. Only an Elastic IP stays the same across stop/start cycles.

This means your Route 53 health check and failover record still point to the old IP. You will update both after confirming the web server is back up.

• Once the instance is running, note the new Public IPv4 address from the instance details panel at the bottom of the page.
• Copy the new public IP and open it in a new browser tab to check if the web server is responding.

✔️ I see 'Hello from us-east-1 (PRIMARY)'

The httpd service auto-started on boot because the User Data script enabled it with systemctl enable. Your web server is back up. Continue to the next substep.

ⓧ The page won't load

The User Data script only runs on the first boot. After a stop/start cycle, the httpd service may not have restarted. You need to start it manually via SSH.

• SSH into the primary instance using the new public IP by running this command:

ssh -i [[PEMPATH="path to your .pem key file"]] ec2-user@[[EC2_PRIMARY_IP="your us-east-1 instance new public IPv4 address"]]

• Start the HTTP server by running:

sudo systemctl start httpd

• Verify the service is running by running:

sudo systemctl status httpd

• You should see active (running) in the output.
• Refresh the browser tab with the public IP. You should now see Hello from us-east-1 (PRIMARY).

Still can't reach the page?

• Make sure you are using the new public IP, not the old one from before you stopped the instance.
• Check the security group still allows inbound HTTP on port 80 from 0.0.0.0/0. Security groups persist across stop/start cycles, but verify just in case.
• Wait 1-2 minutes after the instance shows "Running" for the OS to fully boot before trying again.

Why can't I reach my restarted EC2 instance on port 80?

Update Route 53 and monitor recovery

Your web server is back, but Route 53 does not know that yet. The health check and failover record still point to the old IP address. You need to update both so Route 53 can detect the recovery.

• Switch back to the Route 53 console.
• Click Health checks in the left sidebar.
• Click the linked ID of the primary health check (the one monitoring the us-east-1 IP).
• Click Edit.
• Replace the old IP address in the IP address field with the new public IPv4 address of your us-east-1 instance.
• Click Save.

The health check status will still show Unhealthy for now. It needs to pass 3 consecutive checks (at 30-second intervals) before flipping back. Next, update the failover record.

• Click Hosted zones in the left sidebar.
• Click failover.internal to open your hosted zone.
• Select the row for the Primary failover A record (app.failover.internal with failover type Primary).
• Click Edit record.
• Replace the old IP in the Value field with the same new public IPv4 address you used for the health check.
• Click Save.

Why update both the health check and the record?

The health check monitors whether the server is alive. The failover record tells DNS which IP to return. If you only update the health check, Route 53 would detect the server is healthy but still return the old (now invalid) IP to DNS queries.

In production, you would use an Elastic IP to avoid this problem. An Elastic IP stays fixed across stop/start cycles, so the health check and record never go stale.

• Navigate back to Health checks in the Route 53 sidebar.
• Watch the primary health check status. Refresh the page every 30 seconds. Within 1-2 minutes, the status should flip from Unhealthy to Healthy.

Both health checks should now show Healthy. This tells Route 53 the primary is alive again and failover routing can send traffic back to it.

Primary health check still showing Unhealthy?

• Double-check that the IP address in the health check matches the new public IP of your us-east-1 instance exactly.
• Confirm the web server is actually responding by loading the new IP directly in your browser. If the page does not load, SSH in and run sudo systemctl start httpd.
• Give it a full 2 minutes. The health check needs 3 consecutive successes at 30-second intervals before flipping to healthy.

Why won't my Route 53 health check flip back to Healthy after restarting my EC2 instance?

Confirm DNS has failed back to the primary

The health check says the primary is healthy, so Route 53 should be directing DNS queries back to the us-east-1 IP. Time to prove it.

• Return to your SSH session on the us-west-2 instance. If you disconnected, reconnect by running:

ssh -i [[PEMPATH="path to your .pem key file"]] ec2-user@[[EC2_SECONDARY_IP="your us-west-2 instance public IPv4 address"]]

Before you run the next command, think about what you expect to see. The primary health check is healthy again. What IP address do you think nslookup will return for app.failover.internal?

• Check which IP address DNS returns by running:

nslookup app.failover.internal

The output should show the new public IP of your us-east-1 (primary) instance. Route 53 detected the recovery and switched DNS back.

Still seeing the us-west-2 IP?

DNS records have a TTL (time to live) of 60 seconds. If you run nslookup too soon after the health check flips, cached results may still return the secondary IP. Wait 60 seconds and try again.

• Confirm the actual response by running:

curl app.failover.internal

You should see Hello from us-east-1 (PRIMARY). Failback is complete. Traffic is back on the primary.

What just happened?

You have now witnessed the complete disaster recovery lifecycle. The primary went down, Route 53 detected the failure and redirected traffic to the secondary. The primary came back, Route 53 detected the recovery and redirected traffic back.

This is the active-passive failover pattern that underpins every multi-region SaaS platform. No manual DNS changes. No code deploys. The health checks and routing records handle it automatically.

🙋‍♀️ Seeing an error from curl?

• If curl times out, the DNS may still be resolving to the old us-east-1 IP. Wait 60 seconds for the TTL to expire and try again.
• If nslookup returns the correct new IP but curl still fails, confirm the web server is running by loading the new us-east-1 public IP directly in your browser.
• Make sure you updated the failover A record value (not just the health check) to the new IP address.

Why is curl failing after my primary instance restarted?

You have proven that your multi-region architecture can break and heal itself automatically. In the Secret Mission, you will take this a step further by wiring up CloudWatch alarms that send you an email the instant a failover event occurs.

Secret mission

Add CloudWatch Alarms for Failover Notifications

In production, you cannot sit and watch the Route 53 console waiting for a health check to flip. Your team needs to know the instant a failover happens. In this secret mission, you will wire up CloudWatch alarms and SNS email notifications so failover events land in your inbox automatically.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The Route 53 hosted zone costs $0.50 per month if kept past 12 hours from creation, and EC2 instances consume free tier hours while running.

Cost warning

The Route 53 private hosted zone costs $0.50 per month if not deleted within 12 hours of creation. EC2 instances are free tier eligible, but they consume your monthly free tier allocation while running. If you are done with the project, delete everything to avoid charges.

Resources you used:

• 2 EC2 instances (us-east-1 and us-west-2).
• 1 Route 53 private hosted zone (failover.internal).
• 2 failover A records for app.failover.internal.
• 2 Route 53 health checks.
• 1 CloudWatch alarm (Secret Mission).
• 1 SNS topic FailoverAlerts (Secret Mission).

Keep everything running

No action needed. Choose this if you are still actively experimenting with your failover system.

Both EC2 instances will continue consuming free tier hours while running. If you are within the free tier, this costs $0 per month for compute.

The Route 53 hosted zone costs $0.50 per month if kept past 12 hours from creation. Health checks on AWS endpoints are free for up to 50 per account. The CloudWatch alarm and SNS topic are covered under the free tier.

Pause - I'll come back to this later

Stop both EC2 instances to free up compute resources while keeping your configuration intact.

• In the EC2 console for the us-east-1 region, select the primary instance.
• Click Instance state, then Stop instance.
• Switch to the us-west-2 region and stop the secondary instance the same way.

Your Route 53 health checks will show as unhealthy while the instances are stopped. Health checks on AWS endpoints are free, so there is no cost for them. The hosted zone continues to cost $0.50 per month if kept past 12 hours.

Delete - I don't want to use this again

Remove all project resources and start fresh. Complete these steps in order, since Route 53 requires you to delete records before the hosted zone can be removed.

• Open the Route 53 console. In the navigation pane, choose Hosted zones, then click failover.internal.
• Select both failover A records for app.failover.internal and choose Delete records. Confirm the deletion.
• In the Route 53 navigation pane, choose Health checks. Select both health checks and choose Delete health checks. Confirm the deletion.
• Return to Hosted zones. Select the row for failover.internal and choose Delete. Type the confirmation key and choose Delete. Complete this within 12 hours of creation to avoid the $0.50 monthly charge.
• Open the EC2 console in the us-east-1 region. Select the primary instance, click Instance state, then Terminate instance. Confirm the termination.
• Switch to the us-west-2 region. Select the secondary instance and terminate it the same way.
• (Secret Mission) Open the CloudWatch console. In the navigation pane, choose Alarms. Select the checkbox next to your failover alarm, choose Actions, then Delete. Confirm by choosing Delete.
• (Secret Mission) Open the Amazon SNS console. In the navigation pane, choose Topics. Select FailoverAlerts and choose Delete. Enter delete me in the confirmation dialog and choose Delete.

Nice Work!

Nice Work!

You just built a multi-region failover system on AWS and proved it works by breaking it yourself. You watched traffic reroute automatically when a region went down, and then watched it heal when the region came back.

You've learned how to:

• Deploy EC2 web servers across two AWS regions (us-east-1 and us-west-2), each serving content that identifies which region is handling the request.
• Wire together Route 53 health checks and failover routing records so DNS automatically redirects traffic from an unhealthy primary to a healthy secondary.
• Simulate a real regional outage by stopping the primary instance and observe both automatic failover and failback, completing the full disaster recovery lifecycle.
• Secret Mission: Add CloudWatch alarms and SNS notifications to receive real-time email alerts when a failover event occurs.

Ready to quiz yourself?