Build a Self-Healing Linux Fleet

Automate failure recovery across EC2 instances using Systems Manager.

Introduction

30 Second Summary

When a web server crashes at 3am, someone gets paged, logs in remotely, restarts the service, and goes back to sleep. Now imagine your infrastructure could detect the failure and fix itself without any human involvement.

In this project, you will build a self-healing Linux fleet on AWS that detects service failures and automatically restarts them. You'll manage EC2 instances through AWS Systems Manager without SSH, collect custom metrics with CloudWatch, and wire up an automated remediation pipeline using EventBridge and SSM Automation.

What You'll Build

A production-style self-healing infrastructure with two Amazon Linux EC2 instances managed entirely through Systems Manager, monitored by CloudWatch custom metrics, and automatically remediated when services fail.

By the end of this project, you'll have:

  • A fleet of EC2 instances managed entirely through Systems Manager with no SSH keys or bastion hosts.
  • Custom CloudWatch metrics and alarms showing memory, disk, and process health data that EC2 does not provide by default.
  • A fully automated self-healing pipeline where a failed web server is detected and restarted without human intervention.
  • Secret Mission: A composite alarm that only triggers remediation when multiple conditions are true, reducing false positives.

Are there any prerequisites?

You'll need an AWS account with admin access and the AWS CLI installed and configured on your machine. This project assumes you already have both set up.

Create the IAM Role for Systems Manager and CloudWatch

Traditional server management relies on SSH keys and bastion hosts to access your machines. AWS Systems Manager eliminates all of that by letting you manage instances through IAM permissions instead.

Before you can launch any instances, you need an IAM role that grants them permission to communicate with Systems Manager and CloudWatch. In this step, you will create that role and package it into an instance profile that EC2 instances can use.

In this step, get ready to:
  • Create an IAM role with Systems Manager and CloudWatch permissions.
  • Create an instance profile and attach the role to it.
  • Verify the configuration is correct.
Open your terminal
  • Press the Windows key to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
Create the IAM role with Systems Manager and CloudWatch permissions

EC2 instances need an IAM role to call AWS services on your behalf. You will create a role that trusts EC2 as its principal and attach two managed policies to it.

  • Create a trust policy file that allows EC2 to assume the role by running this command:
@'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
'@ | Set-Content trust-policy.json

What does this command do?

This creates a JSON file called trust-policy.json on your Desktop. The file defines a trust relationship that tells AWS which service is allowed to use this role.

By setting the principal to ec2.amazonaws.com, you are saying only EC2 instances can assume this role.

  • Create the IAM role named SSMFleetRole using the trust policy by running:
aws iam create-role --role-name SSMFleetRole --assume-role-policy-document file://trust-policy.json
  • You should see a JSON response containing the role's ARN, creation date, and the trust policy you defined.
  • Attach the AmazonSSMManagedInstanceCore policy (grants Systems Manager access) by running:
aws iam attach-role-policy --role-name SSMFleetRole --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
  • Attach the CloudWatchAgentServerPolicy policy (grants permission to publish metrics) by running:
aws iam attach-role-policy --role-name SSMFleetRole --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Why two separate policies?

AmazonSSMManagedInstanceCore lets Systems Manager communicate with the instance for Session Manager, Run Command, and other fleet operations.

CloudWatchAgentServerPolicy lets the CloudWatch agent running on the instance publish custom metrics like memory and disk usage. These metrics are not available by default.

Seeing an access denied error?

Make sure your AWS CLI is configured with credentials that have IAM admin permissions. Run aws sts get-caller-identity to confirm which account and user you are operating as.

If you see a "malformed policy document" error, check that trust-policy.json was created in the same directory your terminal is currently in. Run Get-Content trust-policy.json to verify the file contents.

help me fix an access denied or malformed policy error when creating my IAM role

Create the instance profile and attach the role

An instance profile is the container that lets you pass an IAM role to an EC2 instance at launch. You need to create one and link your role to it.

  • Create the instance profile by running:
aws iam create-instance-profile --instance-profile-name SSMFleetRole
  • Attach the SSMFleetRole role to the instance profile by running:
aws iam add-role-to-instance-profile --instance-profile-name SSMFleetRole --role-name SSMFleetRole

Why is the instance profile named the same as the role?

Using the same name for both the role and instance profile makes it easier to track which role is associated with which profile. When you launch EC2 instances later, you will reference this instance profile name.

  • You should see a JSON response for the first command showing the instance profile ARN. The second command produces no output on success.

Getting an EntityAlreadyExists error?

This means a role or instance profile with the name SSMFleetRole already exists in your account. If you are retrying this step, delete the existing resources first and start again.

help me resolve an EntityAlreadyExists error for my SSMFleetRole

Verify the role and instance profile are correctly configured
  • Check that the instance profile has the role attached by running:
aws iam get-instance-profile --instance-profile-name SSMFleetRole
  • In the JSON output, look for the Roles array. You should see SSMFleetRole listed with its ARN and the trust policy you defined earlier.

What should I see?

The output contains InstanceProfileName, InstanceProfileId, and a Roles list. The Roles list should have exactly one entry with RoleName set to SSMFleetRole. If the Roles array is empty, the add-role-to-instance-profile command did not succeed.

Roles array is empty?

Run the add-role-to-instance-profile command again. Make sure both --instance-profile-name and --role-name are set to SSMFleetRole with matching capitalization.

help me figure out why my instance profile has no roles attached

Your IAM role and instance profile are ready. Next up, you will launch your EC2 fleet and watch both instances register with Systems Manager automatically.

Launch the EC2 Fleet

With your IAM role ready, it is time to launch the actual fleet. You will create two EC2 instances that are fully managed through Systems Manager without any SSH keys or open inbound ports.

Session Manager provides secure instance management without inbound ports, bastion hosts, or SSH keys. The SSM Agent on the instance initiates an outbound connection to the Systems Manager service, so your instances need zero inbound security group rules.

In this step, get ready to:
  • Launch 2 Amazon Linux 2023 instances with no SSH key pair and no inbound rules.
  • Verify both instances register as managed nodes in Fleet Manager.
  • Connect to an instance via Session Manager to prove SSH-free access works.
Create a security group with no inbound rules

Traditional EC2 setups require opening port 22 for SSH access. With Systems Manager, the SSM Agent communicates outbound to AWS service endpoints instead, so no inbound ports are needed at all.

  • Create a security group named FleetSG by running this command:
aws ec2 create-security-group --group-name FleetSG --description "Security group for SSM-managed fleet with no inbound rules"

What does this command do?

This creates a new security group called FleetSG in your default VPC. By default, a new security group has no inbound rules (blocking all incoming traffic) and allows all outbound traffic.

That outbound access is what the SSM Agent needs to reach AWS endpoints. You intentionally leave inbound empty to prove SSH is unnecessary.

  • Copy the GroupId value from the output (it looks like sg-0abc123def456789). You will need it in the next command.

Why no inbound rules at all?

Session Manager provides secure instance management without inbound ports, bastion hosts, or SSH keys. The SSM Agent on each instance initiates an outbound connection to the Systems Manager service.

This eliminates the attack surface of open SSH ports entirely.

Seeing an error about VPC?

If the command fails, make sure your AWS CLI default region has a default VPC. You can check with:

aws ec2 describe-vpcs --filters Name=is-default,Values=true

Help me troubleshoot my security group creation error

Launch two Amazon Linux 2023 instances

You will launch both instances in a single command using the SSMFleetRole instance profile from Step 1. Amazon Linux 2023 comes with SSM Agent pre-installed, so the instances will automatically register with Systems Manager.

  • Launch 2 tagged instances by running this command (replace the security group ID with your own):
aws ec2 run-instances --image-id resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 --instance-type t3.micro --count 2 --security-group-ids [[FLEET_SG_ID="your FleetSG security group ID e.g. sg-0abc123def456789"]] --iam-instance-profile Name=SSMFleetRole --tag-specifications "ResourceType=instance,Tags=[{Key=Fleet,Value=WebServers},{Key=Name,Value=WebServer}]"

What does this command do?

  • resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 automatically resolves to the latest Amazon Linux 2023 AMI ID in your region.
  • --count 2 launches two instances at once.
  • --iam-instance-profile Name=SSMFleetRole attaches the instance profile so both instances can communicate with Systems Manager and CloudWatch.
  • --tag-specifications tags both instances with Fleet=WebServers for fleet targeting later, plus a Name tag for easy identification in the console.
  • No --key-name flag is included. This means no SSH key pair is attached, proving you do not need SSH.
  • The command returns a large JSON response. Look for the two InstanceId values (they look like i-0abc123def456789). Note them down for later.

Seeing an error about the image ID?

If the resolve:ssm: syntax is not recognized, your AWS CLI version may be too old. Run aws --version and ensure you are on a recent version.

If you see a region-related error, make sure your AWS CLI region is set correctly. The SSM parameter resolves to the latest AMI in your configured region.

Help me troubleshoot my EC2 instance launch error

Verify both instances appear in Fleet Manager

SSM Agent on Amazon Linux 2023 instances automatically registers with Systems Manager once the instance is running and the IAM role is attached. This registration takes 2-3 minutes.

  • Wait 2-3 minutes for the instances to fully start and register with Systems Manager.
  • In the AWS Management Console, search for and open Systems Manager.
  • In the left navigation pane, click Fleet Manager.
  • Confirm that you see two managed nodes listed with an Online ping status.

What should I see?

You should see two entries in the Fleet Manager node list. Each shows an instance ID, the platform type (Linux), and a ping status of Online. If you only see one or none, wait another minute and refresh the page.

Instances not showing up in Fleet Manager?

Make sure you are viewing the same AWS Region where you launched the instances. You can switch regions using the dropdown in the top-right corner of the console.

If the instances still do not appear after 5 minutes, verify the SSMFleetRole instance profile is attached by running:

aws ec2 describe-instances --filters "Name=tag:Fleet,Values=WebServers" --query "Reservations[].Instances[].{ID:InstanceId,Profile:IamInstanceProfile.Arn,State:State.Name}"

Help me figure out why my instances are not showing up in Systems Manager Fleet Manager

Now for the moment of truth. You will connect to one of your instances without SSH, without a key pair, and without any inbound security group rules.

  • In Fleet Manager, click the button next to one of your instances.
  • Open the Node actions dropdown menu.
  • Select Start terminal session.
  • Click Connect.

A terminal session opens directly in your browser. You now have shell access to your instance with zero SSH configuration.

  • Verify you are connected by typing this command in the Session Manager terminal:
hostname
  • You should see the instance's private hostname in the output. This confirms you have full shell access without SSH.
  • Close the session by clicking Terminate in the top right of the terminal panel.

Why is this a big deal?

You just accessed a Linux server with no SSH keys, no bastion host, and no open ports. In traditional setups, managing SSH keys across a fleet is a security and operational burden.

Session Manager eliminates all of that while providing full audit logging of every command you run.

Session Manager connection failing?

The most common cause is the instance not having outbound internet access. Your security group allows all outbound traffic by default, but if your VPC has a restrictive route table or no internet gateway, the SSM Agent cannot reach the Systems Manager endpoints.

Also verify the instance state is running (not pending) and that the agent has had time to register.

Help me troubleshoot my Session Manager connection to my EC2 instance

Your fleet is live and fully managed through Systems Manager. Next up, you will deploy software across both instances simultaneously using Run Command.

Deploy Software Across the Fleet with Run Command

Your two-instance fleet is live and manageable through Systems Manager. You proved that Session Manager provides shell access without SSH keys or open ports.

Now imagine you need to install a web server on both instances. Without Run Command, you would connect to each instance individually and run the same commands twice. Run Command lets you execute commands across your entire fleet simultaneously, targeting instances by tag.

In this step, get ready to:
  • Install Apache HTTP Server on both instances using a single Run Command execution.
  • Verify the command completed successfully on both instances.
  • Confirm the web server is actively running on the fleet.
Install httpd across the fleet using Run Command

Run Command uses SSM documents to define what commands to execute. The AWS-RunShellScript document lets you run any shell commands on Linux instances. You'll use it to install and start Apache HTTP Server on both instances at once by targeting the Fleet=WebServers tag.

  • In the AWS Management Console, search for and open Systems Manager
  • In the left navigation pane, click Run Command (under Node Management).
  • Click the Run command button in the top right.
  • In the Command document search box, type AWS-RunShellScript and select it from the results.
  • Scroll down to the Command parameters section.
  • In the Commands text box, paste the following commands:
sudo yum install httpd -y
sudo systemctl start httpd
sudo systemctl enable httpd
  • Scroll down to the Targets section.
  • Select Specify instance tags.
  • For the tag key, enter Fleet.
  • For the tag value, enter WebServers.
  • Click Add.
  • Leave all other settings at their defaults and scroll to the bottom of the page.
  • Click Run.

What do these commands do?

  • sudo yum install httpd -y installs Apache HTTP Server without prompting for confirmation.
  • sudo systemctl start httpd starts the web server immediately.
  • sudo systemctl enable httpd ensures httpd starts automatically if the instance reboots.

By targeting the Fleet=WebServers tag, Run Command automatically finds both instances and executes these commands on them simultaneously.

Why Run Command instead of SSH?

Run Command lets you execute operations across your entire fleet without opening any inbound ports or managing SSH keys. Every execution is logged and auditable, which is exactly what production environments require.

Run Command is offered at no additional cost for EC2 instances managed by Systems Manager.

Verify the command executed successfully on both instances

After clicking Run, the console takes you to a command status page. This page shows whether the command succeeded or failed on each targeted instance.

  • Wait for the Overall status to show Success.
  • Under Targets and outputs, confirm that both instances show a status of Success.
  • Click on one of the instance IDs to expand its output.
  • You should see the yum installation output showing httpd packages were downloaded and installed.

Command showing Failed?

  • Check that both instances are still in a Running state in the EC2 console.
  • Verify the tag Fleet=WebServers matches exactly on both instances. Tags are case-sensitive.
  • Make sure the instances have outbound internet access to download packages from yum repositories. The default VPC allows outbound traffic, but a custom VPC might not.

Help me troubleshoot why my Run Command failed on one or both instances.

Confirm httpd is running on both instances

A successful install does not always mean the service is actively running. You'll run a second command across the fleet to check the service status and confirm the web server is alive on both machines.

  • Return to the Run Command page in Systems Manager (click Run Command in the left navigation).
  • Click Run command again.
  • Search for and select AWS-RunShellScript.
  • In the Commands text box, paste the following:
systemctl status httpd
  • Under Targets, select Specify instance tags.
  • Enter Fleet for the tag key and WebServers for the tag value.
  • Click Add.
  • Click Run.
  • Wait for Success status, then click on each instance ID to view the output.
  • You should see Active: active (running) in the output for both instances.

What does this output tell you?

The systemctl status httpd command reports the current state of the httpd service. Seeing active (running) confirms the web server is alive and serving requests. This is the same process your CloudWatch alarm will monitor in a later step.

Seeing inactive (dead) instead?

  • The httpd service may have failed to start. Go back to the first Run Command execution and check its output for errors.
  • Run another command with sudo systemctl start httpd using the same Run Command process to try starting it again.
  • If the install command output mentions missing packages, confirm your instances have outbound internet connectivity through the VPC.

Help me figure out why httpd is not running on my EC2 instance after installation.

Both instances in your fleet are now running Apache HTTP Server, deployed with a single command. Next, you'll install the CloudWatch agent to collect custom metrics like memory, disk, and process health that EC2 does not provide by default.

Install and Configure the CloudWatch Agent

Your fleet is running Apache on both instances, but you have no way to know if memory is maxing out, disk is filling up, or if the httpd process silently dies.

Standard EC2 metrics only cover CPU and network. They don't include memory, disk utilization, or process-level health. The CloudWatch agent fills this gap by collecting custom OS-level metrics and shipping them to CloudWatch.

In this step, get ready to:
  • Store a CloudWatch agent configuration in SSM Parameter Store.
  • Install the CloudWatch agent on both instances using Run Command.
  • Start the agent with your custom configuration to begin collecting metrics.
Create the CloudWatch agent configuration and store it in Parameter Store

The CloudWatch agent needs a JSON configuration file that tells it which metrics to collect. Instead of copying this file to each instance manually, you'll store it in SSM Parameter Store. The agent can then fetch the config directly from Parameter Store when it starts.

  • Create a file called cw-agent-config.json on your local machine with the following configuration:
{
  "metrics": {
    "metrics_collected": {
      "mem": {
        "measurement": [
          "mem_used_percent"
        ]
      },
      "disk": {
        "measurement": [
          "disk_used_percent"
        ],
        "resources": [
          "/"
        ]
      },
      "procstat": [
        {
          "exe": "httpd",
          "measurement": [
            "pid_count"
          ]
        }
      ]
    }
  }
}

What does this configuration do?

This JSON tells the CloudWatch agent exactly which OS-level metrics to collect.

  • mem_used_percent tracks what percentage of memory is in use.
  • disk_used_percent tracks how full the root / partition is.
  • procstat with exe: "httpd" monitors the httpd process by name and reports pid_count (how many matching processes are running).

These metrics are not available from EC2 by default. The CloudWatch agent publishes them to the CWAgent namespace in CloudWatch.

  • Store this configuration in SSM Parameter Store by running this command:
aws ssm put-parameter --name "/fleet/cloudwatch-agent-config" --type "String" --value file://cw-agent-config.json

What does this command do?

aws ssm put-parameter creates a new parameter in SSM Parameter Store. The --name flag uses a hierarchical path /fleet/cloudwatch-agent-config to organize it. The file:// prefix reads the value from your local JSON file.

  • You should see output showing "Version": 1 confirming the parameter was created.
  • Verify the parameter is stored correctly by running:
aws ssm get-parameter --name "/fleet/cloudwatch-agent-config" --query "Parameter.Value" --output text
  • You should see the JSON configuration you just stored printed back in the terminal.

Seeing a ParameterAlreadyExists error?

If you need to update an existing parameter, add --overwrite to the put-parameter command.

If you see a permissions error, confirm your AWS CLI is configured with admin credentials by running aws sts get-caller-identity.

Help me troubleshoot my SSM put-parameter command. I'm trying to store a CloudWatch agent config in Parameter Store.

Install the CloudWatch agent on both instances via Run Command

With the configuration stored centrally, you can now install the CloudWatch agent on both instances simultaneously. You'll use the same Run Command approach from the previous step to target both instances by their tag.

  • Install the CloudWatch agent on both fleet instances by running this command:
aws ssm send-command --document-name "AWS-RunShellScript" --targets "Key=tag:Fleet,Values=WebServers" --parameters "commands=[\"sudo yum install amazon-cloudwatch-agent -y\"]" --comment "Install CloudWatch agent"

What does this command do?

This uses the AWS-RunShellScript SSM document to run a shell command across all instances tagged Fleet=WebServers. The command sudo yum install amazon-cloudwatch-agent -y installs the CloudWatch agent package from the Amazon Linux repositories on both instances at the same time.

  • Copy the CommandId from the output.
  • Check the command completed successfully on both instances by running:
aws ssm list-command-invocations --command-id "[[COMMAND_ID="your CommandId from the previous output"]]" --details
  • Wait about 30 seconds, then confirm both invocations show "Status": "Success" in the output.

Seeing InProgress or Failed status?

If the status is still InProgress, wait another 30 seconds and run the list-command-invocations command again. The yum install command can take up to a minute to complete.

If the status is Failed, check the StandardErrorContent field in the output. The most common cause is the instance not having outbound internet access to reach the yum repositories.

Help me debug a failed Run Command that was trying to install the CloudWatch agent on my EC2 instances.

Start the CloudWatch agent with your configuration

The agent is installed but not running yet. You need to start it and point it to the configuration stored in Parameter Store. The agent control tool (amazon-cloudwatch-agent-ctl) fetches the config from Parameter Store and begins collecting metrics immediately.

  • Start the CloudWatch agent on both instances by running this command:
aws ssm send-command --document-name "AWS-RunShellScript" --targets "Key=tag:Fleet,Values=WebServers" --parameters "commands=[\"sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:/fleet/cloudwatch-agent-config\"]" --comment "Start CloudWatch agent"

What do the flags mean?

  • -a fetch-config tells the agent to fetch and apply a new configuration.
  • -m ec2 specifies the agent is running on an EC2 instance.
  • -s starts the agent after loading the config.
  • -c ssm:/fleet/cloudwatch-agent-config tells the agent to read its configuration from SSM Parameter Store at that path.

The agent pulls the JSON config, translates it into its internal format, and immediately begins collecting memory, disk, and process metrics from both instances.

  • Copy the CommandId from the output.
  • Verify the command succeeded on both instances by running:
aws ssm list-command-invocations --command-id "[[COMMAND_ID_2="your CommandId from the start agent output"]]" --details
  • Confirm both invocations show "Status": "Success".
  • Optionally, verify the agent is actively running by executing a status check across the fleet:
aws ssm send-command --document-name "AWS-RunShellScript" --targets "Key=tag:Fleet,Values=WebServers" --parameters "commands=[\"sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status\"]" --comment "Check CloudWatch agent status"
  • Check the output of that command using list-command-invocations with the new CommandId. You should see "status": "running" in the response for both instances.

Agent status showing stopped?

If the agent is not running, the most common cause is a malformed JSON configuration. Verify your parameter value is valid JSON by running:

aws ssm get-parameter --name "/fleet/cloudwatch-agent-config" --query "Parameter.Value" --output text

Check for missing commas, extra brackets, or mismatched quotes in the output. Fix the JSON and re-run the put-parameter command with --overwrite, then restart the agent.

Help me fix my CloudWatch agent that won't start on my EC2 instances.

Your fleet is now publishing custom OS metrics to CloudWatch. Next, you'll set up alarms that watch these metrics and trigger automated responses when something goes wrong.

Create the CloudWatch Alarm and SNS Notification

Your fleet is now reporting custom metrics that EC2 doesn't provide by default. Memory usage, disk consumption, and process health data are all flowing into CloudWatch under the CWAgent namespace.

Collecting metrics is only half the story. You need CloudWatch to watch those metrics and alert you when something goes wrong. In this step, you'll create an SNS topic for email notifications, then wire up a CloudWatch alarm that fires when the httpd process disappears from one of your instances.

In this step, get ready to:
  • Create an SNS topic and subscribe your email to it.
  • Verify that the procstat metrics are appearing in CloudWatch.
  • Create a CloudWatch alarm that detects when httpd stops running.
Create the SNS topic and email subscription

Before your alarm can notify anyone, it needs a delivery channel. Amazon SNS (Simple Notification Service) acts as a message router. You'll create a topic that your alarm publishes to, and subscribe your email so you receive those messages.

  • Create the SNS topic by running this command:
aws sns create-topic --name FleetAlerts

What does this command do?

This creates an SNS topic named FleetAlerts. The response includes a TopicArn value that looks like arn:aws:sns:us-east-1:123456789012:FleetAlerts. Copy this ARN because you'll need it in the next commands.

  • Subscribe your email address to the topic by running this command:
aws sns subscribe \
  --topic-arn [[TOPIC_ARN="your FleetAlerts topic ARN from above"]] \
  --protocol email \
  --notification-endpoint [[YOUR_EMAIL="your-email@example.com"]]
  • Check your email inbox for a message from AWS Notifications.
  • Click the Confirm subscription link in the email.

Why do I need to confirm?

SNS requires email subscribers to confirm they want to receive messages. This prevents anyone from subscribing someone else's email address to a topic without their consent. You should see a browser page confirming your subscription was successful.

  • Verify your subscription is active by running:
aws sns list-subscriptions-by-topic --topic-arn [[TOPIC_ARN="your FleetAlerts topic ARN"]]

You should see your email listed with a SubscriptionArn value (not "pending confirmation"). This confirms notifications will reach you.

Still seeing pending confirmation?

Check your spam or junk folder for the AWS Notifications email. The confirmation link expires after 2 days. If you can't find it, run the subscribe command again to trigger a new confirmation email.

help me confirm my SNS email subscription

Verify the procstat metrics in CloudWatch

The CloudWatch agent needs a few minutes after starting to begin reporting procstat metrics. Before creating the alarm, you need to confirm the procstat_lookup_pid_count metric is actually appearing in CloudWatch.

  • Wait 3-5 minutes from when you started the CloudWatch agent in the previous step, then check if the metric is reporting by running:
aws cloudwatch list-metrics \
  --namespace CWAgent \
  --metric-name procstat_lookup_pid_count

What should I see?

You should see output listing the procstat_lookup_pid_count metric with dimensions including your instance IDs. The dimensions will include values like exe (set to httpd), pid_finder (set to native), and InstanceId.

Take note of the exact dimension names and values from the output. You'll need them for the alarm in the next substep.

No metrics showing up?

If the command returns empty results, give it a few more minutes. The CloudWatch agent collects on a 60-second interval and needs time to publish. If it still shows nothing after 5 minutes, connect to one instance via Session Manager and check the agent logs:

sudo cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log | tail -20

help me debug why my procstat metrics aren't appearing in CloudWatch

Create the CloudWatch alarm

Now you'll create a CloudWatch alarm that watches the procstat_lookup_pid_count metric for the httpd process. When the count drops below 1 (meaning the process is gone), the alarm triggers and publishes to your FleetAlerts SNS topic.

  • Create the alarm by running this command, replacing the dimension values with what you noted from the previous step:
aws cloudwatch put-metric-alarm \
  --alarm-name HttpdProcessDown \
  --alarm-description "Alarm when httpd process is not running" \
  --namespace CWAgent \
  --metric-name procstat_lookup_pid_count \
  --dimensions Name=exe,Value=httpd Name=pid_finder,Value=native Name=InstanceId,Value=[[INSTANCE_ID="your instance ID from list-metrics output"]] \
  --statistic Average \
  --period 60 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator LessThanThreshold \
  --alarm-actions [[TOPIC_ARN="your FleetAlerts topic ARN"]] \
  --treat-missing-data breaching

What does this alarm configuration do?

  • --metric-name procstat_lookup_pid_count watches how many httpd processes are running.
  • --threshold 1 with --comparison-operator LessThanThreshold means the alarm fires when the count drops below 1 (zero processes).
  • --period 60 and --evaluation-periods 1 mean it only takes one 60-second window with no process to trigger.
  • --treat-missing-data breaching is important. When httpd dies, the procstat plugin may stop reporting the metric entirely rather than reporting zero. This setting ensures missing data is treated as a breach.
  • --alarm-actions publishes to your FleetAlerts SNS topic, which sends you the email notification.
  • Verify the alarm was created and is in the correct state by running:
aws cloudwatch describe-alarms --alarm-names HttpdProcessDown

You should see your alarm listed with a StateValue of OK (since httpd is currently running) or INSUFFICIENT_DATA (if the alarm is still evaluating its first data point). Either state is expected at this point.

Alarm stuck in INSUFFICIENT_DATA?

This is normal for the first 1-2 minutes after creation. The alarm needs at least one evaluation period (60 seconds) of data to transition to OK. If it stays in INSUFFICIENT_DATA for more than 5 minutes, double check that your --dimensions values match exactly what aws cloudwatch list-metrics showed. Dimension names and values are case-sensitive.

help me fix my CloudWatch alarm dimensions for procstat

Why not monitor all instances in one alarm?

A CloudWatch alarm monitors one specific set of dimensions (one instance). For a production fleet, you would create one alarm per instance, or use metric math to aggregate across instances. In this project, you're setting up the alarm for one instance to prove the self-healing loop works. The same pattern scales to any fleet size.

Your alarm is now watching httpd on your instance. In the next step, you'll build the automation that responds when this alarm fires, automatically restarting the failed service without any human intervention.

Build the Self-Healing Automation

Your monitoring infrastructure can now detect when httpd stops running. The alarm fires, you get an email, but the service stays down until someone manually fixes it.

In this step, you'll close that gap by building an automated remediation pipeline. When the alarm fires, EventBridge detects the state change and triggers an SSM Automation runbook that restarts the service automatically.

In this step, get ready to:
  • Create an SSM Automation runbook that restarts httpd on a target instance.
  • Create an EventBridge rule that detects when the HttpdProcessDown alarm enters ALARM state.
  • Create an IAM role that allows EventBridge to invoke SSM Automation.
Create the SSM Automation runbook

An SSM Automation document (runbook) defines a sequence of actions that Systems Manager performs on your behalf. Your runbook will accept an instance ID as input and use the aws:runCommand action to restart httpd on that instance.

  • Create the SSM Automation document by running this command:
aws ssm create-document \
  --name "RestartHttpd" \
  --document-type "Automation" \
  --document-format "YAML" \
  --content '{"schemaVersion":"0.3","description":"Restart httpd on a target instance","parameters":{"InstanceId":{"type":"String","description":"The ID of the instance to restart httpd on"}},"mainSteps":[{"name":"restartHttpd","action":"aws:runCommand","inputs":{"DocumentName":"AWS-RunShellScript","InstanceIds":["{{InstanceId}}"],"Parameters":{"commands":["systemctl restart httpd"]}}}]}'

What does this document do?

The runbook uses schema version 0.3 (required for Automation documents). It declares one input parameter called InstanceId so the caller can specify which instance to target.

The single step restartHttpd uses the aws:runCommand action to execute systemctl restart httpd on the specified instance via the built-in AWS-RunShellScript document.

  • Verify the document was created successfully by running:
aws ssm describe-document --name "RestartHttpd" --query "Document.[Name,DocumentType,Status]"

You should see the name RestartHttpd, type Automation, and status Active in the output.

Seeing a DocumentAlreadyExists error?

If you ran the command before, the document already exists. You can delete it with aws ssm delete-document --name "RestartHttpd" and re-run the create command.

help me fix a DocumentAlreadyExists error when creating my SSM Automation document

Create the IAM role for EventBridge

EventBridge needs an IAM role that grants it permission to start SSM Automation executions on your behalf. You'll create a role named EventBridgeSSMRole with EventBridge as the trusted entity.

  • Create the IAM role with EventBridge as the trusted service by running:
aws iam create-role \
  --role-name EventBridgeSSMRole \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"events.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
  • Attach an inline policy granting the role permission to start SSM Automation and pass roles by running:
aws iam put-role-policy \
  --role-name EventBridgeSSMRole \
  --policy-name SSMAutomationPolicy \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ssm:StartAutomationExecution","iam:PassRole"],"Resource":"*"}]}'

Why does EventBridge need iam:PassRole?

When EventBridge triggers an SSM Automation execution, it may need to pass a role to the automation itself. The iam:PassRole permission allows EventBridge to delegate permissions to the automation runbook.

The ssm:StartAutomationExecution permission allows EventBridge to kick off the runbook when the alarm fires.

  • Verify the role was created by running:
aws iam get-role --role-name EventBridgeSSMRole --query "Role.Arn"

Copy the ARN from the output. You'll need it in the next substep when attaching the EventBridge target.

Getting an EntityAlreadyExists error?

The role already exists from a previous attempt. You can continue to the next substep since the role is already in place.

help me fix an IAM role creation error for my EventBridge SSM integration

Create the EventBridge rule and target

Now you'll wire everything together. The EventBridge rule listens for alarm state changes from CloudWatch, and when it detects the HttpdProcessDown alarm going into ALARM state, it triggers your RestartHttpd runbook.

  • Create the EventBridge rule that matches CloudWatch alarm state change events by running:
aws events put-rule \
  --name "HttpdAlarmToSSM" \
  --event-pattern '{"source":["aws.cloudwatch"],"detail-type":["CloudWatch Alarm State Change"],"detail":{"alarmName":["HttpdProcessDown"],"state":{"value":["ALARM"]}}}'

What does this event pattern match?

The pattern filters events from aws.cloudwatch with detail-type CloudWatch Alarm State Change. It further narrows to only events where the alarm name is HttpdProcessDown and the new state value is ALARM.

CloudWatch guarantees delivery of alarm state change events to EventBridge, so your rule will always fire when the alarm triggers.

  • Now attach the SSM Automation target to the rule. Replace your-account-id and your-region with your actual values by running:
aws events put-targets \
  --rule "HttpdAlarmToSSM" \
  --targets '[{"Id":"RestartHttpdTarget","Arn":"arn:aws:ssm:[[AWS_REGION="your-region"]]:[[AWS_ACCOUNT_ID="your-account-id"]]:automation-definition/RestartHttpd","RoleArn":"arn:aws:iam::[[AWS_ACCOUNT_ID="your-account-id"]]:role/EventBridgeSSMRole","InputTransformer":{"InputPathsMap":{"instance":"$.detail.configuration.metrics[0].metricStat.metric.dimensions.InstanceId"},"InputTemplate":"{\"InstanceId\":[<instance>]}"}}]'

How does the instance ID get passed?

The InputTransformer extracts the instance ID from the CloudWatch alarm event. The alarm event includes metric dimensions containing the InstanceId of the affected instance.

The InputTemplate maps the extracted value into the format your runbook expects: a JSON object with an InstanceId parameter.

  • Verify the rule and target are configured by running:
aws events list-targets-by-rule --rule "HttpdAlarmToSSM"

You should see your target with the RestartHttpd automation document ARN and the EventBridgeSSMRole role ARN listed.

Target shows FailedEntryCount?

Double-check that the EventBridgeSSMRole ARN is correct and that the RestartHttpd document exists. Use aws iam get-role --role-name EventBridgeSSMRole and aws ssm describe-document --name RestartHttpd to confirm both resources are in place.

help me troubleshoot my EventBridge put-targets command failing for SSM Automation

Your self-healing pipeline is fully wired. Next up, you'll test it by intentionally killing httpd and watching the automation bring it back to life.

Test the Self-Healing Loop

You have built every piece of the self-healing pipeline: the alarm that watches for a dead process, the EventBridge rule that catches the alarm, and the SSM Automation runbook that restarts the service.

Now it is time to prove the pipeline works end-to-end. You will deliberately kill the web server on one instance and watch the entire chain fire automatically.

In this step, get ready to:
  • Simulate a real service failure by stopping httpd on one instance.
  • Watch the CloudWatch alarm transition to ALARM and receive an SNS email notification.
  • Verify that SSM Automation restarted the service without any human intervention.
Simulate a service failure via Session Manager

In a production environment, services crash for all sorts of reasons. You will replicate this scenario by connecting to one instance through Session Manager and manually stopping the httpd process.

  • Open the Amazon EC2 console.
  • In the navigation pane, click Instances.
  • Select the instance named WebServer-1 (either one works, but pick just one for this test).
  • Click Connect at the top of the page.
  • Choose the Session Manager tab.
  • Click Connect to open a browser-based shell session.

A terminal session opens in your browser. You are now logged into the EC2 instance without SSH.

  • Stop the Apache web server by running this command in the Session Manager terminal:
sudo systemctl stop httpd

What does this command do?

`sudo systemctl stop httpd` tells the system to stop the Apache HTTP Server process immediately. This simulates a real crash where the web server dies unexpectedly.

Once the process count drops to zero, the CloudWatch agent's procstat plugin reports 0 for procstat_lookup_pid_count. This is what triggers the alarm.

  • You can verify httpd is actually stopped by running:
systemctl status httpd

You should see inactive (dead) in the output. The failure simulation is now complete.

Leave this Session Manager tab open

Keep the browser tab with your Session Manager session open. You will return to it later to confirm the service was automatically restarted.

Watch the CloudWatch alarm trigger and receive the SNS notification

The HttpdProcessDown alarm evaluates the procstat_lookup_pid_count metric every 60 seconds. Once it sees the value drop below 1 for one evaluation period, it transitions from OK to ALARM.

  • Open the CloudWatch console in a new browser tab.
  • In the navigation pane, click Alarms, then All alarms.
  • Find the HttpdProcessDown alarm in the list.
  • Wait 1-2 minutes. Refresh the page periodically until the alarm state changes from OK to In alarm.

Why does it take 1-2 minutes?

The CloudWatch agent reports metrics at the interval configured in the agent config (typically 60 seconds). The alarm then needs one full evaluation period of 60 seconds to confirm the metric is below the threshold.

In total, it can take up to 2 minutes from when you stopped httpd for the alarm to fire.

  • Check your email inbox for a message from AWS Notifications.
  • Open the email. It confirms that the HttpdProcessDown alarm transitioned to ALARM state.

Alarm not transitioning to ALARM?

Make sure you actually ran sudo systemctl stop httpd and not just systemctl stop httpd (without sudo). Without sudo, the command may silently fail due to permissions.

If the alarm still shows OK after 3 minutes, check that the CloudWatch agent is still running on the instance. In your Session Manager terminal, run sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status to confirm its status.

help me troubleshoot why my CloudWatch alarm isn't triggering after I stopped httpd

Verify the automation executed and httpd was restarted

When the alarm fired, EventBridge caught the state change event and triggered the RestartHttpd SSM Automation runbook. Now you will confirm the automation ran successfully.

  • Return to the AWS Systems Manager console.
  • In the navigation pane, click Automation.
  • Look for a recent execution of the RestartHttpd document in the execution list.
  • Click on the execution ID to view the details.
  • Confirm the Overall status shows Success.

What should I see in the execution details?

The execution details page shows the runbook name (RestartHttpd), the instance ID it targeted, and the status of each step. You should see one step with aws:runCommand action showing a status of Success.

The Executed by field will show the EventBridge role ARN, confirming this was triggered automatically and not manually.

Now confirm the service is actually running again on the instance.

  • In the Systems Manager navigation pane, click Run Command.
  • Click Run command.
  • Search for and select the AWS-RunShellScript document.
  • In the Command parameters section, enter this command:
systemctl status httpd
  • In the Target selection section, choose Choose instances manually and select WebServer-1 (the instance where you stopped httpd).
  • Click Run.
  • Wait for the command to complete, then click the instance ID in the Targets and outputs section to view the output.

You should see active (running) in the output. The automation successfully restarted httpd without any human intervention.

Automation execution not showing up?

If you do not see a RestartHttpd execution in the Automation history, the EventBridge rule may not have triggered. Check the EventBridge console and look at the HttpdAlarmToSSM rule's monitoring tab to see if it received any events.

Also verify the EventBridgeSSMRole has the correct ssm:StartAutomationExecution permission and that the trust policy allows the EventBridge service to assume it.

help me figure out why my EventBridge rule didn't trigger SSM Automation

What just happened end-to-end?

Here is the full chain that fired automatically: httpd stopped, the CloudWatch agent reported 0 processes, the HttpdProcessDown alarm entered ALARM state, CloudWatch emitted an event to EventBridge, EventBridge matched the event and invoked SSM Automation, and the RestartHttpd runbook ran systemctl restart httpd on the affected instance.

This is exactly how production fleets handle service recovery. No one was paged. No one manually restarted anything. The system healed itself.

Secret mission

Composite Alarm for Smarter Remediation

Your self-healing loop works, but it fires on any process failure. What about momentary blips where httpd restarts on its own? Or high CPU during expected deployments? Build a composite alarm that only triggers remediation when BOTH the process is down AND CPU is spiking, reducing false positives like a production-grade operations team would.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Your two EC2 instances cost approximately $0.02/hour total while running, and CloudWatch custom metrics cost $0.30/metric/month.

Cost warning

Your two t3.micro instances will keep charging while running. Custom CloudWatch metrics also incur monthly charges regardless of instance state. If you're done with this project, choose Pause or Delete to avoid unexpected costs on your AWS bill.

Resources you used:

  • 2 EC2 instances (WebServer-1, WebServer-2).
  • 1 EC2 security group (FleetSG).
  • 1 IAM role (SSMFleetRole) + instance profile.
  • 1 IAM role (EventBridgeSSMRole).
  • 1 SNS topic (FleetAlerts).
  • 1 SSM Parameter Store parameter (/fleet/cloudwatch-agent-config).
  • 1 SSM Automation document (RestartHttpd).
  • 1 EventBridge rule (HttpdAlarmToSSM).
  • 3 CloudWatch alarms (HttpdProcessDown, HighCPU, HttpdCritical).

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment further with your self-healing fleet.

  • Your two t3.micro instances will continue running at approximately $0.01/hour each.
  • CloudWatch custom metrics will continue at $0.30/metric/month.
  • SNS, EventBridge, Run Command, and Session Manager remain free within normal usage.

Pause - I'll come back to this later

Stop your instances to halt compute charges while keeping all configuration intact. CloudWatch metric storage charges will continue at a minimal rate.

  • Open the EC2 console and navigate to Instances.
  • Select both WebServer-1 and WebServer-2.
  • Click Instance state, then Stop instance.
  • Confirm by clicking Stop.

When you're ready to resume, select both instances and choose Instance state then Start instance. The SSM Agent will re-register automatically within a few minutes.

Delete - I don't want to use this again

Remove all project resources completely. Follow the steps below in order, since some resources depend on others being removed first.

Terminate EC2 instances
  • Terminate both instances by running:
aws ec2 terminate-instances --instance-ids [[INSTANCE_ID_1="your first instance ID"]] [[INSTANCE_ID_2="your second instance ID"]]
  • Wait for the instances to reach the terminated state before continuing. You can check with:
aws ec2 describe-instances --instance-ids [[INSTANCE_ID_1="your first instance ID"]] [[INSTANCE_ID_2="your second instance ID"]] --query "Reservations[*].Instances[*].State.Name"
Delete EventBridge rule
  • Remove the target from the rule first, then delete the rule:
aws events remove-targets --rule "HttpdAlarmToSSM" --ids "1"
aws events delete-rule --name "HttpdAlarmToSSM"
Delete CloudWatch alarms
  • Delete the composite alarm first (it depends on the other two), then delete the individual alarms:
aws cloudwatch delete-alarms --alarm-names "HttpdCritical"
aws cloudwatch delete-alarms --alarm-names "HttpdProcessDown"
aws cloudwatch delete-alarms --alarm-names "HighCPU"
Delete the SNS topic
  • Delete the FleetAlerts topic by running:
aws sns delete-topic --topic-arn [[SNS_TOPIC_ARN="arn:aws:sns:us-east-1:123456789012:FleetAlerts"]]
Delete SSM resources
  • Delete the automation document and the parameter:
aws ssm delete-document --name "RestartHttpd"
aws ssm delete-parameter --name "/fleet/cloudwatch-agent-config"
Delete IAM roles and instance profile
  • Remove the role from the instance profile, delete the instance profile, detach managed policies, then delete the SSMFleetRole:
aws iam remove-role-from-instance-profile --instance-profile-name SSMFleetRole --role-name SSMFleetRole
aws iam delete-instance-profile --instance-profile-name SSMFleetRole
aws iam detach-role-policy --role-name SSMFleetRole --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam detach-role-policy --role-name SSMFleetRole --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws iam delete-role --role-name SSMFleetRole
  • Delete the EventBridgeSSMRole (delete inline policy first, then the role):
aws iam delete-role-policy --role-name EventBridgeSSMRole --policy-name [[INLINE_POLICY_NAME="your inline policy name"]]
aws iam delete-role --role-name EventBridgeSSMRole
Delete the security group
  • Once the instances are fully terminated, delete the security group:
aws ec2 delete-security-group --group-id [[SECURITY_GROUP_ID="your FleetSG security group ID"]]

All project resources are now removed from your AWS account.

That's a Wrap!

That's a Wrap!

Nice work! You just built production-grade self-healing infrastructure from scratch.

You've learned how to:

  • Manage a Linux fleet entirely through Systems Manager with zero SSH access, deploying software across multiple instances simultaneously using Run Command with tag-based targeting.
  • Collect custom OS-level metrics like memory, disk, and process health via the CloudWatch agent, filling the monitoring gaps that EC2 doesn't provide by default.
  • Build an event-driven remediation pipeline where a CloudWatch alarm triggers EventBridge, which invokes SSM Automation to restart a failed service and send an SNS notification automatically.
  • Secret Mission: Created composite alarms for intelligent, low-noise alerting that only triggers remediation when multiple conditions are true simultaneously.

This is exactly how production fleets operate at scale. You now understand the modern approach to Linux fleet management that cloud engineer interviewers expect.

Ready to quiz yourself?