Build a Self-Healing EC2 Monitor

Automate EC2 failure detection and recovery using EventBridge and Lambda.

Introduction

30 Second Summary

When a web server crashes at 3 AM, someone usually has to wake up, diagnose the problem, and manually restart it. What if the system could detect the failure and fix itself before anyone even noticed?

In this project, you will build an event-driven self-healing pipeline on AWS. You'll deploy a real nginx web server and connect it to automated monitoring that detects failures, restarts the service, and sends you a notification.

What You'll Build

A fully automated self-healing infrastructure on AWS that detects web server failures and fixes them without human intervention.

By the end of this project, you'll have:

A live nginx web server on EC2 that you can visit in your browser.
An automated monitoring pipeline using CloudWatch alarms and EventBridge rules to detect failures and trigger remediation.
A Lambda function that uses Systems Manager to restart crashed services and sends SNS email notifications so you know the system healed itself.
A second event-driven automation that catches stopped EC2 instances and automatically restarts them (Secret Mission).

Are there any prerequisites?

You'll need an AWS account with Free Tier access and the AWS CLI installed and configured. Python 3.x should also be installed locally.

Launch EC2 with nginx and Configure IAM

Every self-healing system needs something to heal. In this project, that something is a real web server you can watch go down and automatically come back online.

You'll launch an EC2 instance running nginx and configure an IAM role that gives AWS Systems Manager remote access to the instance. This is the foundation your automated remediation pipeline will build on.

In this step, get ready to:

Create an IAM role that allows Systems Manager to manage your EC2 instance.
Launch a t2.micro EC2 instance with nginx installed via a startup script.
Verify the web server is live and Systems Manager can reach the instance.

Create an IAM role for your EC2 instance

Your EC2 instance needs permission to communicate with AWS Systems Manager. An IAM role grants these permissions so the SSM Agent on the instance can receive and execute remote commands.

Sign in to the AWS Management Console at console.aws.amazon.com.
Type IAM in the search bar at the top of the console.
Select IAM from the results.
In the left sidebar, click Roles.
Click Create role.
Under Trusted entity type, select AWS service.
Under Use case, select EC2.
Click Next.
In the permissions search bar, type AmazonSSMManagedInstanceCore.
Select the checkbox next to AmazonSSMManagedInstanceCore.
Click Next.
In the Role name field, enter SelfHealingEC2Role.
Click Create role.

You should see a green success banner confirming that the role SelfHealingEC2Role was created.

What is Systems Manager?

Systems Manager (SSM) is an AWS service that lets you manage EC2 instances remotely without SSH keys. It can run shell commands directly on your instances.

The AmazonSSMManagedInstanceCore policy gives the pre-installed SSM Agent permission to communicate with the Systems Manager service. Later, your Lambda function will use this connection to restart nginx when something goes wrong.

Launch your EC2 instance with nginx

Now you'll launch the actual instance that runs your web server. You'll use a startup script called User data to install and start nginx automatically when the instance boots.

Type EC2 in the console search bar at the top.
Select EC2 from the results.
Click Launch instance.
Under Name and tags, enter SelfHealingDemo in the Name field.
Under Application and OS Images (Amazon Machine Image), confirm that Amazon Linux 2023 AMI is selected. It should be the default option under Quick Start.
Under Instance type, confirm that t2.micro is selected. This is Free Tier eligible.
Under Key pair (login), select Proceed without a key pair (Not recommended) from the dropdown. You'll use Session Manager instead of SSH to connect to the instance.

Now configure the security group so your web server is accessible from your browser.

Under Network settings, click Edit.
Click Add security group rule.
For the new rule's Type, select HTTP.
For Source type, select Anywhere. This allows traffic on port 80 from any IP address.

Next, attach the IAM role and add the startup script that installs nginx.

Scroll down to Advanced details and click to expand the section.
Under IAM instance profile, select SelfHealingEC2Role from the dropdown.
Scroll to the bottom and find the User data text box. Paste the following startup script:

#!/bin/bash
sudo dnf install -y nginx
sudo systemctl start nginx
sudo systemctl enable nginx

What does this script do?

This script runs automatically when your EC2 instance boots for the first time.

#!/bin/bash tells the instance to run this as a shell script.
sudo dnf install -y nginx installs the nginx web server using the Amazon Linux 2023 package manager.
sudo systemctl start nginx starts the web server immediately.
sudo systemctl enable nginx ensures nginx restarts automatically if the instance reboots.

Click Launch instance.

You should see a success message confirming your instance is launching. The startup script takes about 1-2 minutes to install and start nginx.

Don't see SelfHealingEC2Role in the dropdown?

If the IAM instance profile dropdown doesn't show SelfHealingEC2Role, go back to the IAM console and confirm the role was created with the trusted entity type set to EC2. If you selected a different trusted entity, the role won't appear as an instance profile.

help me figure out why my IAM role isn't showing in the EC2 launch wizard

Verify nginx and Systems Manager connectivity

Your instance is booting up and running the startup script. You need to confirm two things: the nginx web page is accessible and Systems Manager can reach the instance.

Click View all instances on the launch success page, or navigate to Instances in the EC2 left sidebar.
Wait until the Instance state column shows Running and the Status check column shows 2/2 checks passed. This can take 1-2 minutes.
Select your SelfHealingDemo instance by clicking its checkbox.
In the details panel below, find the Public IPv4 address field and copy the IP address.
Open a new browser tab and paste the IP address into the address bar.

You should see the nginx welcome page. This confirms your web server is running and the security group is allowing HTTP traffic.

Not seeing the nginx welcome page?

Make sure your browser URL starts with http:// and not https://. Some browsers auto-upgrade to HTTPS, which won't work here.
Verify your security group has an inbound rule for HTTP (port 80) with source 0.0.0.0/0. Click your instance, then the Security tab to check.
If both look correct, the startup script may still be running. Wait another minute and refresh the page.

help me troubleshoot why my nginx page isn't loading

Now confirm that Systems Manager can reach your instance. This is critical because your self-healing Lambda function will use Systems Manager to restart nginx later.

Type Systems Manager in the console search bar.
Select Systems Manager from the results.
In the left sidebar, click Fleet Manager under Node Management.
Look for your SelfHealingDemo instance in the list. Its SSM Agent ping status should show Online.

Instance not showing in Fleet Manager?

The SSM Agent can take 2-3 minutes to register after the instance boots. Wait a moment and click the refresh button in Fleet Manager.
Confirm the SelfHealingEC2Role IAM role is attached to your instance. In the EC2 console, select your instance, click the Security tab, and check the IAM Role field.
Make sure the IAM role has the AmazonSSMManagedInstanceCore policy attached. Without it, the SSM Agent cannot register.

help me get my EC2 instance to appear in Fleet Manager

Your web server is live and Systems Manager is connected. Next up, you'll set up the monitoring that detects when something goes wrong.

Create the SNS Topic and CloudWatch Alarm

Your EC2 instance is live, nginx is serving a web page, and the IAM role lets Systems Manager communicate with the instance. Now you need a way to detect when things go wrong and a channel to send alerts through.

In this step, you'll set up two key pieces of the self-healing pipeline. An SNS topic acts as a notification channel that emails you when the system takes action. A CloudWatch alarm continuously monitors your instance's CPU usage and fires when it crosses a threshold.

In this step, get ready to:

Create an SNS topic and email subscription for remediation alerts.
Set up a CloudWatch alarm monitoring CPU utilization on your EC2 instance.
Configure the alarm to send notifications when CPU spikes above 80%.

Create the SNS topic and email subscription

SNS (Simple Notification Service) is a messaging service that sends notifications to subscribers. You'll create a topic, which is a named channel, and subscribe your email to it. Whenever the self-healing pipeline takes action, it publishes a message to this topic and you receive an email.

In the AWS Console, type SNS in the top search bar.
Click Simple Notification Service.
In the left sidebar, click Topics.
Click Create topic.
Select Standard as the type.
Enter SelfHealingAlerts in the Name field.
Scroll down and click Create topic.

You should see the topic details page with the SelfHealingAlerts topic created. Now create an email subscription for this topic.

On the topic details page, click Create subscription.
Under Protocol, select Email.
Enter your email address in the Endpoint field.
Click Create subscription.

The subscription status will show Pending confirmation. You need to confirm it from your email before any messages can be delivered.

Check your email inbox for a message from AWS Notifications.
Click the Confirm subscription link in the email.
Return to the SNS subscription page in the AWS Console and refresh the page.

The Status column should now show Confirmed. This means your email is ready to receive notifications from the pipeline.

Why do I need to confirm the subscription?

AWS requires email confirmation for SNS subscriptions to prevent spam. No messages will be delivered to your inbox until you click the confirmation link.

Don't see the confirmation email?

Check your spam or junk folder. AWS notification emails sometimes end up there.
Make sure you entered the correct email address in the Endpoint field. You can delete the subscription and create a new one if needed.
I'm not receiving the SNS subscription confirmation email. How do I fix this?

Set up the CloudWatch alarm

CloudWatch monitors AWS resources and collects metrics like CPU usage, network traffic, and disk I/O. You'll create an alarm that watches your EC2 instance's CPU utilization and triggers when it exceeds 80%.

In the AWS Console, type CloudWatch in the top search bar.
Click CloudWatch.
In the left sidebar, click Alarms, then click All alarms.
Click Create alarm.
Click Select metric.

The metric browser opens. You need to find the CPU utilization metric for your specific instance.

Click EC2.
Click Per-Instance Metrics.
Find your SelfHealingDemo instance in the list. You can search for it by pasting your instance ID in the search bar.
Select the checkbox next to the CPUUtilization metric for your instance.
Click Select metric.

Now configure the alarm conditions. These settings control when the alarm fires.

Under Metric, set the Period to 1 minute.
Under Conditions, select Greater as the threshold type.
Enter 80 as the threshold value.
Expand Additional configuration.
Set Datapoints to alarm to 1 out of 1.
Click Next.

What do these alarm settings mean?

Period of 1 minute means CloudWatch evaluates the CPU metric every 60 seconds.
Greater than 80% is the threshold. If the instance's CPU usage exceeds this, the alarm fires.
1 out of 1 datapoints means a single high reading triggers the alarm immediately. No need to wait for multiple consecutive high readings.

Configure the notification action and create the alarm

You're now on the Configure actions step of the alarm wizard. This is where you connect the alarm to your SNS topic so you receive an email whenever the alarm fires.

Under Alarm state trigger, select In alarm.
Under Send a notification to the following SNS topic, select Select an existing SNS topic.
Choose your SelfHealingAlerts topic from the dropdown.
Click Next.
Enter HighCPU-SelfHealingDemo in the Alarm name field.
Click Next.
Review your alarm configuration on the preview page.
Click Create alarm.

Why send notifications in addition to automated remediation?

Later in this project, you'll build a Lambda function that automatically fixes the problem when this alarm fires. The SNS notification gives you visibility into what the system is doing. Even with full automation, you want to know when incidents happen so you can spot patterns and investigate root causes.

You should see the HighCPU-SelfHealingDemo alarm in your CloudWatch Alarms list. The alarm state will show Insufficient data initially.

Why does my alarm say Insufficient data?

This is completely normal for a new alarm. CloudWatch needs at least one evaluation period (1 minute) to collect data before it can determine the alarm state. After a couple of minutes, it will transition to OK because your instance's CPU utilization should be well below 80%.

Don't see your alarm in the list?

Make sure you clicked Create alarm on the final review page. If you only clicked Next, the alarm was not created.
Check that you are in the correct AWS region. Your alarm must be in the same region as your EC2 instance.
My CloudWatch alarm isn't showing up after creation. What should I check?

Your monitoring and notification system is in place. CloudWatch is watching your instance's CPU, and SNS is ready to deliver alerts to your inbox. Next up, you'll build the Lambda function that actually fixes the problem when the alarm fires.

Build the Lambda Remediation Function

Your monitoring layer is ready. You have an SNS topic that can send you emails and a CloudWatch alarm watching for CPU spikes on your EC2 instance.

Right now, your alarm fires and sends a notification. That is useful, but someone still has to manually log in and fix the problem. In this step, you will build an AWS Lambda function that handles the fix automatically. When EventBridge delivers an alarm event to this function, it will restart nginx on the affected instance and notify you that the system healed itself.

In this step, get ready to:

Create an IAM role that gives your Lambda function permission to restart services and send notifications.
Write the Python remediation function that extracts event data, sends a restart command, and publishes a notification.
Test the function with a sample alarm event to verify it works before wiring it into the live pipeline.

Create the Lambda IAM role

Your Lambda function needs permission to do two things: send commands to your EC2 instance through Systems Manager, and publish notifications to your SNS topic. You will create a dedicated IAM role with only these permissions.

Open the AWS Management Console and search for IAM in the top search bar.
Click Roles in the left sidebar.
Click Create role.
Under Trusted entity type, select AWS service.
Under Use case, select Lambda.
Click Next.
Search for AWSLambdaBasicExecutionRole in the search bar.
Check the box next to AWSLambdaBasicExecutionRole and click Next.
Enter SelfHealingLambdaRole as the role name.
Click Create role.

What is AWSLambdaBasicExecutionRole?

This is a managed policy that gives your Lambda function permission to write logs to CloudWatch Logs. Every Lambda function needs this so you can see execution output and debug errors.

The role exists, but it only has logging permissions so far. Now you need to add an inline policy that grants access to Systems Manager and SNS.

In the IAM Roles list, click on SelfHealingLambdaRole to open it.
Click the Permissions tab.
Click Add permissions, then select Create inline policy.
Click the JSON tab in the policy editor.
Before pasting the policy, you need your SNS topic ARN. Open the SNS console in a new tab, click Topics, click SelfHealingAlerts, and copy the ARN shown at the top of the page.
Replace the default policy JSON with the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:SendCommand",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "sns:Publish",
            "Resource": "[[SNS_TOPIC_ARN=\"arn:aws:sns:us-east-1:123456789012:SelfHealingAlerts\"]]"
        }
    ]
}

What does this policy do?

ssm:SendCommand lets your Lambda function send shell commands to EC2 instances through Systems Manager. The wildcard resource (*) allows it to target any instance.
sns:Publish is scoped to your specific SelfHealingAlerts topic. This follows the least-privilege principle by restricting notifications to just this one topic.

Click Next.
Enter SelfHealingRemediationPolicy as the policy name.
Click Create policy.

You should see SelfHealingRemediationPolicy appear alongside AWSLambdaBasicExecutionRole in the role's permissions list. Your Lambda role now has exactly the permissions it needs.

Policy not showing up?

Make sure you clicked into SelfHealingLambdaRole before adding the inline policy. If you created the policy at the account level instead of on the role, delete it and start the inline policy steps again.
Check that your JSON is valid. A missing comma or bracket breaks the policy. Compare your JSON carefully against the code block above.
Confirm you replaced the SNS topic ARN placeholder with your actual ARN from the SNS console.

Help me debug my inline IAM policy

Create the Lambda function and write the code

With the IAM role ready, you can now create the Lambda function itself. This function receives an alarm event, extracts the instance ID, sends a restart command through SSM, and publishes a notification.

Search for Lambda in the AWS console search bar and open the Lambda service.
Click Create function.
Select Author from scratch.
Enter SelfHealingRemediation as the function name.
Select Python 3.12 as the runtime.
Expand Change default execution role.
Select Use an existing role and choose SelfHealingLambdaRole from the dropdown.
Click Create function.

The default Lambda timeout is 3 seconds, which is too short for SSM commands to complete. You need to increase it before writing code.

Scroll down to the Configuration tab.
Click General configuration in the left sidebar.
Click Edit.
Change the Timeout to 30 seconds.
Click Save.

Now scroll back up to the Code tab. You will see a default lambda_function.py file open in the inline editor. Select all the default code and delete it. You will replace it with your remediation logic in two chunks.

Start by pasting the imports, client setup, and SNS topic ARN at the top of the file:

import boto3
import json
import datetime

# Create AWS service clients
ssm = boto3.client('ssm')
sns = boto3.client('sns')

# Replace with your actual SNS topic ARN from the SNS console
SNS_TOPIC_ARN = '[[SNS_TOPIC_ARN="arn:aws:sns:us-east-1:123456789012:SelfHealingAlerts"]]'

What does this code do?

boto3 is the AWS SDK for Python. It is pre-installed in the Lambda runtime, so you do not need to install anything.
ssm and sns are service clients. Creating them outside the handler function means they are reused across invocations, which is faster.
SNS_TOPIC_ARN tells the function which SNS topic to publish remediation alerts to.

You will not see a change yet. The next chunk adds the main handler function that uses these clients.

Below the SNS_TOPIC_ARN line, paste the handler function:

def lambda_handler(event, context):
    # Extract alarm name from the EventBridge event
    alarm_name = event['detail']['alarmName']

    # Extract instance ID from the alarm's metric dimensions
    instance_id = event['detail']['configuration']['metrics'][0]['metricStat']['metric']['dimensions']['InstanceId']

    # Send SSM Run Command to restart nginx on the affected instance
    ssm.send_command(
        InstanceIds=[instance_id],
        DocumentName='AWS-RunShellScript',
        Parameters={'commands': ['sudo systemctl restart nginx']}
    )

    # Build a remediation summary with timestamp
    timestamp = datetime.datetime.now().isoformat()
    message = (
        f"Self-Healing Remediation Executed\n"
        f"Alarm: {alarm_name}\n"
        f"Instance: {instance_id}\n"
        f"Action: Restarted nginx via SSM Run Command\n"
        f"Timestamp: {timestamp}"
    )

    # Publish the summary to SNS
    sns.publish(
        TopicArn=SNS_TOPIC_ARN,
        Subject=f'Self-Healing Alert: {alarm_name}',
        Message=message
    )

    return {
        'statusCode': 200,
        'body': json.dumps('Remediation complete')
    }

What does this function do?

This is the core of your self-healing pipeline. The function does three things in sequence:

Extract event data. It reads the alarm name and instance ID from the EventBridge event payload. The instance ID is nested deep inside the alarm's metric configuration.
Restart nginx. It sends an SSM Run Command using the AWS-RunShellScript document to run sudo systemctl restart nginx on the affected instance. This is a surgical fix that restarts only the failed service, not the entire server.
Notify you. It publishes a summary to your SNS topic with the alarm name, instance ID, action taken, and timestamp so you know what happened.

Why SSM Run Command instead of rebooting the instance?

A full EC2 reboot is heavy-handed. It causes downtime for everything running on the instance and takes much longer. SSM Run Command lets you restart just the specific service that failed. This is how production self-healing systems work.

Click Deploy above the code editor.

You should see a green "Changes deployed" banner appear at the top of the code editor. Your function is now saved and ready to run.

Deploy button grayed out or error after deploying?

If Deploy is grayed out, click inside the code editor first and make sure you have deleted all the default code before pasting.
Check for Python syntax errors. Make sure the def lambda_handler(event, context): line has a colon at the end, and all the code inside the function is indented.
Confirm your SNS topic ARN is wrapped in quotes and follows the format arn:aws:sns:region:account-id:SelfHealingAlerts.

Help me fix my Lambda function deployment error

Awesome, I've got everything!

Double check that you have saved your function by clicking Deploy. The green banner should confirm your changes are deployed.

I'd like to double check the full code

import boto3
import json
import datetime

# Create AWS service clients
ssm = boto3.client('ssm')
sns = boto3.client('sns')

# Replace with your actual SNS topic ARN from the SNS console
SNS_TOPIC_ARN = '[[SNS_TOPIC_ARN="arn:aws:sns:us-east-1:123456789012:SelfHealingAlerts"]]'

def lambda_handler(event, context):
    # Extract alarm name from the EventBridge event
    alarm_name = event['detail']['alarmName']

    # Extract instance ID from the alarm's metric dimensions
    instance_id = event['detail']['configuration']['metrics'][0]['metricStat']['metric']['dimensions']['InstanceId']

    # Send SSM Run Command to restart nginx on the affected instance
    ssm.send_command(
        InstanceIds=[instance_id],
        DocumentName='AWS-RunShellScript',
        Parameters={'commands': ['sudo systemctl restart nginx']}
    )

    # Build a remediation summary with timestamp
    timestamp = datetime.datetime.now().isoformat()
    message = (
        f"Self-Healing Remediation Executed\n"
        f"Alarm: {alarm_name}\n"
        f"Instance: {instance_id}\n"
        f"Action: Restarted nginx via SSM Run Command\n"
        f"Timestamp: {timestamp}"
    )

    # Publish the summary to SNS
    sns.publish(
        TopicArn=SNS_TOPIC_ARN,
        Subject=f'Self-Healing Alert: {alarm_name}',
        Message=message
    )

    return {
        'statusCode': 200,
        'body': json.dumps('Remediation complete')
    }

Test the Lambda with a sample event

Your function is deployed, but you do not know if it actually works yet. The Lambda console lets you invoke your function with a test event that simulates what EventBridge would send when your CloudWatch alarm fires.

Before creating the test event, grab your EC2 instance ID. Open the EC2 console in a new tab, click Instances, and copy the Instance ID for SelfHealingDemo (it starts with i-).
Back in the Lambda console, click the Test tab.
Select Create new event.
Enter AlarmStateChange as the event name.
Replace the default JSON with the following test event:

{
  "source": "aws.cloudwatch",
  "detail-type": "CloudWatch Alarm State Change",
  "detail": {
    "alarmName": "HighCPU-SelfHealingDemo",
    "state": {
      "value": "ALARM",
      "reason": "Threshold Crossed: 1 out of 1 datapoints were greater than the threshold (80.0).",
      "timestamp": "2026-04-25T15:00:00.000+0000"
    },
    "previousState": {
      "value": "OK",
      "timestamp": "2026-04-25T14:55:00.000+0000"
    },
    "configuration": {
      "metrics": [
        {
          "metricStat": {
            "metric": {
              "namespace": "AWS/EC2",
              "name": "CPUUtilization",
              "dimensions": {
                "InstanceId": "[[EC2_INSTANCE_ID=\"i-0abc1234def567890\"]]"
              }
            }
          }
        }
      ]
    }
  }
}

What is this test event?

This JSON mimics the exact event that EventBridge sends when a CloudWatch alarm transitions to the ALARM state. The structure matches what AWS produces in production.

source and detail-type identify this as a CloudWatch alarm event.
detail.alarmName matches the alarm you created in Step 2.
detail.configuration.metrics[0]...InstanceId contains your real instance ID so the SSM command targets the correct EC2 instance.

Click Save to save the test event.
Click Test to invoke the function.

After a few seconds, you should see a green "Execution result: succeeded" banner. The response body shows "Remediation complete".

Expand the Details section under the execution result to see the full response and log output.
Check your email inbox. You should receive an email from AWS Notifications with the subject "Self-Healing Alert: HighCPU-SelfHealingDemo" and a message containing your instance ID and timestamp.

What just happened behind the scenes?

Your Lambda function successfully processed the fake alarm event, sent a restart command to your real EC2 instance through Systems Manager, and emailed you a notification. The nginx service was already running, so the restart command had no visible effect. In Step 5, you will simulate a real failure and watch the full pipeline fix it.

Seeing an execution error?

AccessDeniedException on ssm:SendCommand means your inline policy is missing or not attached to the role. Go back to IAM, open SelfHealingLambdaRole, and verify both policies are listed.
InvalidInstanceId means the instance ID in your test event does not match a running instance, or the SSM Agent is not registered. Confirm your instance ID is correct and that your instance shows as Online in Systems Manager > Fleet Manager.
Task timed out after 3.00 seconds means you skipped the timeout change. Go to Configuration > General configuration and set the timeout to 30 seconds.
KeyError on any of the event fields means your test event JSON is malformed. Compare it against the code block above and make sure you replaced the instance ID placeholder with your actual instance ID.

Help me debug my Lambda test execution error

Your Lambda function works. It can extract event data, restart nginx, and notify you. Next up, you will create the EventBridge rule that connects your CloudWatch alarm to this function so the whole pipeline runs automatically.

Wire Up EventBridge to Connect Everything

In the last step, you built a Lambda function that can restart nginx and send you notifications. Right now, nothing triggers that function when your alarm fires.

Amazon EventBridge connects AWS services by routing events based on rules you define. In this step, you'll create a rule that catches your CloudWatch alarm and sends the event straight to your Lambda function.

Why EventBridge instead of direct alarm actions?

CloudWatch alarms can directly perform EC2 actions and send SNS notifications. However, routing through EventBridge lets you filter events with custom patterns, route to multiple targets, and build the kind of event-driven automation that real DevOps teams use at scale.

In this step, get ready to:

Create an EventBridge rule named HighCPU-Remediation-Rule.
Define a custom event pattern that filters for your specific CloudWatch alarm.
Connect the rule to your Lambda function as the target.

Create a new EventBridge rule

In the AWS Management Console, search for EventBridge in the top search bar and select it.
In the left sidebar, click Rules.
Click Create rule.
Enter HighCPU-Remediation-Rule in the Name field.
For Event bus, keep default selected.
For Rule type, select Rule with an event pattern.
Click Next.

Build the event pattern

The event pattern is a JSON filter that tells EventBridge which events to care about. You want to match only the events where your specific alarm transitions to the ALARM state.

Scroll down to the Event pattern section.
Select Custom patterns (JSON editor).
Paste the following pattern into the editor:

{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"],
  "detail": {
    "alarmName": ["HighCPU-SelfHealingDemo"],
    "state": {
      "value": ["ALARM"]
    }
  }
}

What does this event pattern do?

"source": ["aws.cloudwatch"] matches events coming from CloudWatch.
"detail-type": ["CloudWatch Alarm State Change"] narrows it to alarm state changes, ignoring other CloudWatch events.
"alarmName": ["HighCPU-SelfHealingDemo"] targets your specific alarm so other alarms don't trigger this rule.
"state": {"value": ["ALARM"]} fires only when the alarm enters the ALARM state, not when it returns to OK.

Click Next to proceed to the target configuration page.

Seeing a validation error?

Check for missing commas, extra trailing commas, or mismatched curly brackets in your JSON.
Make sure all string values are wrapped in square brackets (e.g., ["ALARM"] not "ALARM"). EventBridge patterns require array syntax for values.
Help me fix my EventBridge event pattern JSON.

Set the target and create the rule

Now you'll tell EventBridge where to send matching events. The target is your Lambda function that handles the remediation.

Under Target 1, select AWS service for Target type.
For Select a target, choose Lambda function from the dropdown.
For Function, select SelfHealingRemediation.
Click Next.
On the Tags page, click Next.
Review the rule details and click Create rule.

What happens when EventBridge connects to Lambda?

When you set a Lambda function as an EventBridge target, EventBridge automatically adds a resource-based policy to your Lambda function. This policy grants EventBridge permission to invoke the function, so you don't need to configure this manually.

Check that HighCPU-Remediation-Rule appears on the EventBridge Rules page with a status of Enabled.

Rule not showing as Enabled?

Confirm that you clicked Create rule on the final review page. If you navigated away before creating, you'll need to start from the beginning.
If the rule appears but shows as Disabled, click the rule name to open it, then click Enable in the top right.
Make sure you selected SelfHealingRemediation as the target. If a different function or no function was selected, edit the rule and update the target.
Help me troubleshoot my EventBridge rule setup.

Your self-healing pipeline is fully wired. Next, you'll break things on purpose and watch the system heal itself.

Test the Self-Healing Pipeline End-to-End

Your EventBridge rule is wired up and ready to route alarm events to your Lambda function. Now it is time to put the whole system to the test.

You will connect to your EC2 instance through Session Manager and simulate a real failure. The question is: when nginx crashes and CPU spikes, will your pipeline detect it, fix it, and notify you automatically?

In this step, get ready to:

Simulate a web server failure by stopping nginx and spiking CPU.
Watch CloudWatch and EventBridge detect and route the failure event.
Verify the system automatically restores nginx and sends you a notification.

Simulate a web server failure

Before breaking anything, confirm that your web server is still running. Then you will crash nginx and spike the CPU to trigger your CloudWatch alarm.

Open a new browser tab and navigate to your EC2 instance's public IP address.
Verify you see the nginx welcome page. This confirms the web server is running before you crash it.

Now connect to your instance through Systems Manager so you can run commands directly on it.

In the AWS Console, search for Systems Manager and open it.
In the left sidebar, click Session Manager.
Click Start session.
Select your SelfHealingDemo instance.
Click Start session.

A terminal opens in your browser, connected directly to your EC2 instance.

Can't see your instance in Session Manager?

Make sure your EC2 instance is in the Running state. A stopped instance cannot accept sessions.
Verify the SelfHealingEC2Role IAM role is attached to your instance. Without this role, the SSM Agent cannot communicate with Systems Manager.
Wait 2-3 minutes after launch. The SSM Agent needs time to register with Systems Manager.
Help me troubleshoot why my EC2 instance is not showing up in Session Manager.

Stop the nginx web server to simulate a crash by running:

sudo systemctl stop nginx

Install stress-ng, a CPU load-testing tool, by running:

sudo dnf install -y stress-ng

You should see Complete! at the end of the install output.

Install failed?

If you see No match for argument: stress-ng, your instance may not be running Amazon Linux 2023. Check your instance's AMI in the EC2 console.

Help me fix a failed stress-ng install on my EC2 instance.

Spike the CPU above 80% to trigger the CloudWatch alarm by running:

stress-ng --cpu 2 --timeout 300

What do these commands do?

sudo systemctl stop nginx immediately stops the nginx web server. This simulates what happens when a service crashes in production.
stress-ng --cpu 2 --timeout 300 creates two CPU-intensive worker processes that run for 5 minutes. This pushes CPU utilization above the 80% threshold that triggers your CloudWatch alarm.
stress-ng automatically stops after 5 minutes. You do not need to kill it manually.

This command runs in the foreground and keeps your Session Manager terminal busy. Leave it running and open a new browser tab to continue.

Watch the alarm trigger

With nginx stopped and CPU spiking, your CloudWatch alarm should detect the anomaly within a couple of minutes. Your job now is to watch it happen in real time.

Refresh the nginx page in your browser (the tab with your EC2 public IP).

The page should show an error or timeout. This confirms the web server is down.

In a new browser tab, search for CloudWatch in the AWS Console.
In the left sidebar, click Alarms, then All alarms.
Find your HighCPU-SelfHealingDemo alarm.

Wait 1-2 minutes for CloudWatch to evaluate the CPU spike. The alarm state transitions from OK to In alarm with a red indicator.

In another browser tab, navigate to EventBridge.
Click Rules in the left sidebar.
Select your HighCPU-Remediation-Rule.
Click the Monitoring tab to confirm the rule was triggered.

Pro tip: refresh the alarms page

The CloudWatch alarms page does not auto-refresh. Click the refresh icon in the alarms table every 30 seconds to see the latest state.

Alarm not triggering?

Wait at least 2 full minutes after starting stress-ng. CloudWatch evaluates metrics at the 1-minute period you configured, so it needs at least one full evaluation cycle.
Check your Session Manager tab to confirm stress-ng is still running. If it already finished, run it again.
Verify your alarm period is set to 1 minute with an evaluation period of 1 datapoint. A longer period delays detection.
Help me figure out why my CloudWatch alarm is not transitioning to ALARM state.

Verify the full self-healing chain

The alarm fired and EventBridge routed the event to your Lambda function. Now verify that every link in the chain did its job.

In the CloudWatch console, click Logs in the left sidebar, then Log groups.
Open the /aws/lambda/SelfHealingRemediation log group.
Click the most recent log stream.
Look for log entries confirming the SSM command was sent successfully.

What should I see in the logs?

Your Lambda function logs the alarm name, instance ID, and the SSM command response. If you see these values in the log stream, your function executed correctly.

Check your email inbox for an SNS notification from AWS with the remediation details.

The email includes the alarm name, instance ID, and timestamp of the remediation action.

In the AWS Console, navigate to Systems Manager.
In the left sidebar, click Run Command, then Command history.
Look for a command with document name AWS-RunShellScript showing a Success status.

This confirms that Lambda successfully sent the restart command to your EC2 instance through SSM.

Refresh the nginx page in your browser (the tab with your EC2 public IP).

The nginx welcome page is back. Your self-healing pipeline detected the failure and restored the web server automatically.

nginx still down after the alarm fired?

Check the CloudWatch Logs for /aws/lambda/SelfHealingRemediation. Look for error messages in the most recent log stream.
Verify your Lambda function's timeout is set to 30 seconds. The default 3 seconds is too short for SSM commands to complete.
Check the Run Command history in Systems Manager. If the command shows Failed, the EC2 instance may have lost its SSM connection.
Verify that the SelfHealingLambdaRole has the ssm:SendCommand permission in its inline policy.
Help me debug why my Lambda function executed but nginx did not restart.

Secret mission

Auto-Restart Stopped Instances

Your self-healing system can restart a crashed service. But what happens if someone stops the entire EC2 instance? Build a second automation loop that detects stopped instances and automatically restarts them, turning your monitoring system into a truly resilient infrastructure.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses AWS Free Tier services, but the EC2 instance consumes Free Tier compute hours while running.

Cost warning

The EC2 t2.micro instance is free for 750 hours/month within the 12-month AWS Free Tier. If your Free Tier has expired, this instance costs approximately $0.0116/hr (about $8.50/month). CloudWatch alarms, EventBridge rules, Lambda, SNS, and Systems Manager are all within the Always Free tier for this project's usage.

If you are outside the Free Tier period, stop or terminate the EC2 instance to avoid charges.

Resources you used:

EC2 instance (SelfHealingDemo, t2.micro).
IAM roles (SelfHealingEC2Role and SelfHealingLambdaRole).
CloudWatch alarm (HighCPU-SelfHealingDemo).
EventBridge rule (HighCPU-Remediation-Rule).
Lambda function (SelfHealingRemediation).
SNS topic and subscription (SelfHealingAlerts).
Security group allowing HTTP on port 80.
Secret Mission only: Lambda function (AutoRestartInstance) and EventBridge rule (EC2-AutoRestart-Rule).

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue experimenting with your self-healing pipeline.

Your EC2 instance will continue consuming Free Tier compute hours (750 hours/month shared across all t2.micro instances in your account).
CloudWatch, EventBridge, Lambda, SNS, and Systems Manager usage stays within the Always Free tier.

Pause - I'll come back to this later

Stop the EC2 instance to save Free Tier compute hours while keeping all your alarms, rules, and functions configured.

If you completed the Secret Mission, disable the EC2-AutoRestart-Rule EventBridge rule first. Otherwise, the auto-restart rule will restart your instance as soon as you stop it.
Navigate to EC2 > Instances, select your SelfHealingDemo instance, then choose Instance state > Stop instance.
A stopped instance does not incur compute charges. Your CloudWatch alarm, EventBridge rules, Lambda functions, and SNS topic remain configured at no cost.
When you come back, start the instance again and your public IP address may change. Update your browser bookmark accordingly.

Delete - I don't want to use this again

Remove all project resources to return your AWS account to a clean state.

Terminate the EC2 instance: go to EC2 > Instances, select SelfHealingDemo, then choose Instance state > Terminate instance.
Delete the CloudWatch alarm: go to CloudWatch > Alarms, select HighCPU-SelfHealingDemo, and click Delete.
Delete the EventBridge rules: go to EventBridge > Rules, select HighCPU-Remediation-Rule and click Delete. If you completed the Secret Mission, also delete EC2-AutoRestart-Rule.
Delete the Lambda functions: go to Lambda > Functions, select SelfHealingRemediation and click Delete. If you completed the Secret Mission, also delete AutoRestartInstance.
Delete the SNS topic: go to SNS > Topics, select SelfHealingAlerts, and click Delete. This also removes the email subscription.
Delete the IAM roles: go to IAM > Roles, search for and delete SelfHealingEC2Role and SelfHealingLambdaRole.
Delete the security group: go to EC2 > Security Groups, find the security group you created for this project, and delete it. You can only delete it after the EC2 instance is terminated.

Mission accomplished

Nice Work!

You just built a self-healing infrastructure on AWS with a real web server. Your system automatically detects when nginx crashes, restarts the service without any human intervention, and notifies you that it happened.

Here's what you learned:

Deployed a live nginx web server on EC2 and created a CloudWatch alarm that monitors CPU utilization and triggers on high usage.
Built an EventBridge rule with a custom JSON event pattern that routes alarm events to a Lambda function, which automatically restarts nginx via SSM Run Command and sends you an SNS email notification.
Tested the full self-healing pipeline end-to-end by crashing nginx, watching the system detect and fix the problem, and verifying the web server came back online automatically.
Built a second event-driven automation that detects stopped EC2 instances and automatically restarts them.

Ready to quiz yourself?