Self-Healing Infrastructure Platform

Introduction

30 Second Summary

When a website goes down at 3 AM, someone gets paged to restart it manually. What if your infrastructure could detect the problem and fix itself before anyone even wakes up?

In this project, you will build a self-healing infrastructure platform on AWS that automatically detects failures, applies graduated fixes (try a soft restart first, then a full reboot, then escalate to a human), and exposes a live API you can query to show incident history in real-time JSON.

What You'll Build

A production-grade event-driven auto-remediation system using Lambda, EventBridge, CloudWatch, and Terraform that demonstrates senior-level SRE operational thinking.

By the end of this project, you'll have:

• A graduated remediation engine that detects failures and applies escalating fixes automatically, tracking attempt history to avoid repeated failed actions.
• A live incident API endpoint you can query with a single curl command during interviews to show healing timelines as JSON. No AWS console needed.
• The entire platform deployed as Infrastructure as Code with remote state, composite alarms for correlated failure detection, and a CI/CD pipeline via GitHub Actions.
• Secret Mission: Build a full Game Day automation that runs all chaos scenarios in sequence, validates healing for each, and generates an SLO availability report with MTTR stats.

Are there any prerequisites?

You need basic Linux terminal comfort and a general understanding of what EC2 is. You should also have an AWS account with free tier available.

No prior experience with Terraform, Lambda, or chaos engineering is required.

Install Tools and Configure AWS

Before you can build a self-healing infrastructure, you need the right tools on your machine.

This step gets your environment ready with four key tools: the AWS CLI for managing cloud resources, Terraform for infrastructure as code, Python 3 for your Lambda functions, and Git for version control.

In this step, get ready to:

• Install and configure the AWS CLI with your credentials.
• Install Terraform and verify it runs.
• Install Python 3, pip, boto3, and Git.

Install and configure AWS CLI v2

The AWS CLI is your command-line interface to all AWS services. You need version 2 to use the latest features and interact with services like Lambda, EventBridge, and SSM.

• Check if you already have the AWS CLI installed by running this command:

aws --version

✔️ I see aws-cli/2.x.x

Your AWS CLI v2 is already installed. Skip ahead to the configuration step below.

ⓧ I see aws-cli/1.x.x

You have version 1 installed. You need version 2 for this project.

• Download and install AWS CLI v2 by running these commands:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install --update

• Verify the upgrade succeeded by running:

aws --version

You should see aws-cli/2.x.x in the output.

ⓧ Command not found

The AWS CLI is not installed yet.

• Download and install AWS CLI v2 by running these commands:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

• Verify the installation succeeded by running:

aws --version

You should see aws-cli/2.x.x in the output.

Still seeing command not found?

Make sure /usr/local/bin is in your PATH. Run echo $PATH to check.

If it is not there, add it by running:

export PATH=/usr/local/bin:$PATH

Then try aws --version again. Help me fix my AWS CLI PATH issue

Now you need to configure the CLI with your AWS credentials. You will need an IAM user with AdministratorAccess and its access keys.

What is an IAM user?

An IAM user is an identity within your AWS account that has specific permissions. For this project, you need an IAM user with AdministratorAccess so it can create and manage all the resources (EC2, Lambda, DynamoDB, etc.).

If you do not already have one, go to the IAM console in AWS, create a new user, attach the AdministratorAccess policy, and generate an access key pair.

• Configure AWS CLI with your credentials by running:

aws configure

What will aws configure ask?

The CLI will prompt you for four values:

• AWS Access Key ID - paste the access key from your IAM user.
• AWS Secret Access Key - paste the secret key from your IAM user.
• Default region name - enter ap-south-1 (Mumbai region).
• Default output format - enter json.

• Verify your credentials are working by running:

aws sts get-caller-identity

You should see a JSON response containing your account ID, user ARN, and user ID.

Seeing an InvalidClientTokenId error?

Double-check that you pasted the correct access key and secret key. Run aws configure again and re-enter them carefully.

Also confirm your IAM user has the AdministratorAccess policy attached in the IAM console. Help me debug my AWS credentials

Install Terraform

Terraform lets you define all your AWS infrastructure as code in .tf files. This project requires Terraform version 1.6 or higher.

• Check if you already have Terraform installed by running:

terraform version

✔️ I see version 1.6 or higher

Your Terraform version meets the requirement. Move on to the next substep.

ⓧ I see an older version

You need version 1.6 or higher for this project.

• Download the latest Terraform binary and install it by running:

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

• Verify the upgrade by running:

terraform version

You should see Terraform v1.6.x or higher.

Not using Ubuntu/Debian?

Visit the official install page at developer.hashicorp.com/terraform/install for instructions specific to your Linux distribution (CentOS, Fedora, Amazon Linux).

ⓧ Command not found

Terraform is not installed yet.

• Install Terraform using the HashiCorp package repository by running:

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

• Verify the installation by running:

terraform version

You should see Terraform v1.6.x or higher.

Not using Ubuntu/Debian?

Visit the official install page at developer.hashicorp.com/terraform/install for instructions specific to your Linux distribution (CentOS, Fedora, Amazon Linux).

Getting a GPG or repository error?

Make sure you have wget, gpg, and lsb-release installed. Run sudo apt install wget gpg lsb-release first, then retry the Terraform install commands. Help me fix my Terraform install

Install Python 3, pip, boto3, and Git

Your Lambda functions are written in Python, and the boto3 library is used for local testing. Git tracks your code and enables the CI/CD pipeline later.

• Check if Python 3 is installed by running:

python3 --version

✔️ I see Python 3.x

Python 3 is ready. Continue below to install the remaining tools.

ⓧ Command not found

Python 3 is not installed.

• Install Python 3 and pip by running:

sudo apt update && sudo apt install python3 python3-pip -y

• Verify Python is installed by running:

python3 --version

You should see a version number like Python 3.10.x or higher.

• Install the boto3 library (the AWS SDK for Python) by running:

pip3 install boto3

What is boto3?

boto3 is the official AWS SDK for Python. It lets you interact with AWS services programmatically. While the Lambda runtime includes boto3 automatically, installing it locally lets you test and develop your functions before deploying.

• Check if Git is installed by running:

git --version

✔️ I see a version number

Git is ready. You have all the tools you need.

ⓧ Command not found

Git is not installed.

• Install Git by running:

sudo apt install git -y

• Verify Git is installed by running:

git --version

You should see a version number in the output.

Finally, open your preferred code editor. This project uses VS Code, but any editor with file-tree navigation works.

pip3 command not found?

On some systems, pip is bundled differently. Try python3 -m pip install boto3 as an alternative. Help me install pip on my system

Your environment is ready. Next up, you will create the project structure and set up Terraform remote state so your infrastructure code is production-grade from the start.

Set Up Project Structure and Remote State

Your tools are installed and your AWS credentials are configured. Now you need a place for all of your code to live, plus a safe location for Terraform to store its state file.

In production environments, teams store Terraform state remotely so that multiple engineers can collaborate without overwriting each other's changes. You'll set up this same pattern using an S3 bucket for the state file and a DynamoDB table for state locking.

In this step, get ready to:

• Create the project directory structure with terraform/, lambda/, chaos/, and .github/workflows/ directories.
• Create the Terraform remote state backend (S3 bucket and DynamoDB lock table).
• Write the Terraform provider and variable configuration files and initialize the project.

Create the project directory structure

Every part of your self-healing platform lives in a dedicated folder. The terraform/ directory holds infrastructure code, lambda/ holds Python function code, chaos/ holds fault injection scripts, and .github/workflows/ holds CI/CD pipeline definitions.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder, subdirectories, and initialize a Git repository by running these commands:

mkdir -p self-healing-infra/{terraform,lambda,chaos,.github/workflows}
cd self-healing-infra
git init

• Confirm the directories exist by running:

ls -la

You should see terraform, lambda, chaos, and .github listed in the output, along with the .git directory confirming Git is initialized.

Don't see all the folders?

Make sure you typed the mkdir -p command exactly as shown. The -p flag creates parent directories as needed, including the nested .github/workflows path.

Help me troubleshoot my folder structure.

Create the Terraform remote state backend

Terraform needs to know the current state of your infrastructure before making changes. Storing this state file in S3 with a DynamoDB lock table prevents two operations from modifying your infrastructure simultaneously.

Why remote state instead of local?

By default, Terraform stores state in a local file called terraform.tfstate. This works for solo projects, but breaks the moment a second person (or a CI/CD pipeline) tries to apply changes.

Remote state in S3 gives you versioning (roll back bad state), encryption at rest, and the DynamoDB lock table prevents concurrent modifications. This is the production pattern every team uses.

You need to create the S3 bucket and DynamoDB table before Terraform can use them. This is a chicken-and-egg problem: you cannot use Terraform to create its own backend.

• First, get your AWS account ID (you will need it for a unique bucket name) by running:

aws sts get-caller-identity --query Account --output text

This outputs your 12-digit account ID. Note it down.

• Create the S3 bucket for Terraform state by running this command (replace the account ID placeholder with yours):

aws s3api create-bucket \
    --bucket self-healing-infra-tfstate-[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]] \
    --region ap-south-1 \
    --create-bucket-configuration LocationConstraint=ap-south-1

What does this command do?

• --bucket sets a globally unique name for your state bucket.
• --region ap-south-1 targets Mumbai.
• --create-bucket-configuration LocationConstraint=ap-south-1 is required for any region outside us-east-1.

• Enable versioning on the state bucket by running:

aws s3api put-bucket-versioning \
    --bucket self-healing-infra-tfstate-[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]] \
    --versioning-configuration Status=Enabled

• Create the DynamoDB table for state locking by running:

aws dynamodb create-table \
    --table-name terraform-locks \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST \
    --region ap-south-1

What does this command do?

• --table-name terraform-locks matches the name Terraform expects in the backend config.
• AttributeName=LockID,KeyType=HASH creates the partition key that Terraform uses to acquire and release locks.
• --billing-mode PAY_PER_REQUEST means you pay nothing when no locks are active. This table will rarely be used.

You should see a JSON response showing "TableStatus": "CREATING" in the output. The table becomes ACTIVE within a few seconds.

Seeing a BucketAlreadyExists error?

S3 bucket names are globally unique across all AWS accounts. Make sure you appended your actual account ID to the bucket name. If someone else already used that exact name, add extra characters to make it unique.

Help me fix my bucket name.

Write the Terraform configuration files

With the backend resources ready, you can now write the Terraform configuration that points to them. You will create two files: providers.tf (backend and provider settings) and variables.tf (project-wide variables).

• Create providers.tf inside the terraform/ directory. Open it in your code editor and paste the following code:

terraform {
  required_version = ">= 1.6"

  backend "s3" {
    bucket         = "self-healing-infra-tfstate"
    key            = "prod/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

What does this code do?

• The backend "s3" block tells Terraform to store its state file in your S3 bucket with DynamoDB locking.
• required_version = ">= 1.6" ensures anyone running this code has a recent Terraform version.
• The aws provider (pinned to ~> 5.0) manages all AWS resources. The archive provider (pinned to ~> 2.0) creates zip files for Lambda deployments later.
• encrypt = true ensures the state file is encrypted at rest in S3.

Update the bucket name

The bucket field in the backend block must match the exact bucket name you created earlier (including your account ID suffix). Update "self-healing-infra-tfstate" to your full bucket name before running terraform init.

• Create variables.tf inside the terraform/ directory and paste:

variable "aws_region" {
  description = "AWS region to deploy resources"
  type        = string
  default     = "ap-south-1"
}

variable "instance_type" {
  description = "EC2 instance type"
  type        = string
  default     = "t3.micro"
}

variable "notification_email" {
  description = "Email address for healing notifications"
  type        = string
}

variable "project_name" {
  description = "Project name used for resource naming"
  type        = string
  default     = "self-healing-infra"
}

What does this code do?

• aws_region defaults to Mumbai (ap-south-1) where all your resources will deploy.
• instance_type defaults to t3.micro which is covered by the AWS Free Tier.
• notification_email has no default because it is unique to you. Terraform will prompt you for it at apply time.
• project_name is used as a prefix for all resource names, keeping everything organized.

• Save both files.
• Initialize Terraform against the remote backend by running:

cd terraform
terraform init

You should see output ending with Terraform has been successfully initialized! confirming that the S3 backend is connected and providers are downloaded.

Seeing a backend configuration error?

The most common cause is a mismatch between the bucket name in providers.tf and the actual bucket you created. Double-check that the bucket name in the backend "s3" block matches your real bucket name (with your account ID suffix).

Also confirm that your AWS CLI region is set to ap-south-1 by running aws configure get region.

Help me debug my backend configuration.

✔️ Awesome, I've got everything!

Great. Double check you have saved both providers.tf and variables.tf before moving on.

ⓧ I'd like to double check the full code

Here is the complete terraform/providers.tf:

terraform {
  required_version = ">= 1.6"

  backend "s3" {
    bucket         = "self-healing-infra-tfstate"
    key            = "prod/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

And here is the complete terraform/variables.tf:

variable "aws_region" {
  description = "AWS region to deploy resources"
  type        = string
  default     = "ap-south-1"
}

variable "instance_type" {
  description = "EC2 instance type"
  type        = string
  default     = "t3.micro"
}

variable "notification_email" {
  description = "Email address for healing notifications"
  type        = string
}

variable "project_name" {
  description = "Project name used for resource naming"
  type        = string
  default     = "self-healing-infra"
}

Your project skeleton is ready and Terraform is connected to remote state. Next up, you'll write the EC2 instance configuration with CloudWatch Agent monitoring so your platform has something to heal.

Deploy EC2 with CloudWatch Agent

Your project structure and remote state are ready. Now it is time to deploy the actual infrastructure your self-healing system will protect.

In this step you will launch an EC2 instance running Apache httpd, configure the CloudWatch Agent to publish process-level metrics, and verify that everything is connected and reporting.

In this step, get ready to:

• Write the Terraform configuration for the EC2 instance, security group, and IAM permissions.
• Configure the CloudWatch Agent to monitor httpd process health and disk usage.
• Deploy with Terraform and verify the instance is running with custom metrics publishing.

Write the security group and IAM configuration

The EC2 instance needs a security group for network access and an IAM role so it can communicate with SSM (for remote command execution) and CloudWatch (for publishing metrics).

• Create a new file called ec2.tf inside your terraform/ folder.
• Add the data sources that look up the latest Amazon Linux 2023 AMI and the default VPC by pasting the following code:

data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

data "aws_vpc" "default" {
  default = true
}

data "aws_subnets" "default" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.default.id]
  }
}

What do these data sources do?

Data sources let Terraform look up existing AWS resources without creating them. aws_ami finds the latest Amazon Linux 2023 AMI so your instance always launches on a current image.

aws_vpc and aws_subnets reference your account's default VPC. This avoids creating custom networking and keeps the focus on the self-healing logic.

• Below the data sources, add the security group that allows SSH and HTTP traffic:

resource "aws_security_group" "healing_sg" {
  name        = "${var.project_name}-sg"
  description = "Allow SSH and HTTP for self-healing demo"
  vpc_id      = data.aws_vpc.default.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name    = "${var.project_name}-sg"
    Project = var.project_name
  }
}

What does this security group allow?

Port 22 (SSH) lets you connect to the instance for manual inspection. Port 80 (HTTP) lets you verify httpd is serving traffic. The egress rule allows all outbound traffic so the instance can reach AWS APIs for SSM and CloudWatch.

• Next, add the IAM role and instance profile that grant SSM and CloudWatch permissions:

resource "aws_iam_role" "ec2_role" {
  name = "${var.project_name}-ec2-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })

  tags = {
    Project = var.project_name
  }
}

resource "aws_iam_role_policy_attachment" "ec2_ssm" {
  role       = aws_iam_role.ec2_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

resource "aws_iam_role_policy_attachment" "ec2_cloudwatch" {
  role       = aws_iam_role.ec2_role.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}

resource "aws_iam_instance_profile" "ec2_profile" {
  name = "${var.project_name}-ec2-profile"
  role = aws_iam_role.ec2_role.name
}

Why two managed policies?

AmazonSSMManagedInstanceCore registers the instance with Systems Manager so Lambda can run commands on it remotely. CloudWatchAgentServerPolicy lets the CloudWatch Agent publish custom metrics to your account.

The instance profile wraps the role so EC2 can assume it at launch.

• Save ec2.tf so far.

Create the EC2 instance with CloudWatch Agent monitoring

The EC2 instance uses a user_data script that runs on first boot. This script installs httpd (the web server your healing system will protect), stress-ng (a chaos tool), and the CloudWatch Agent configured to watch the httpd process.

• At the bottom of ec2.tf, add the EC2 instance resource:

resource "aws_instance" "target" {
  ami                    = data.aws_ami.amazon_linux.id
  instance_type          = var.instance_type
  iam_instance_profile   = aws_iam_instance_profile.ec2_profile.name
  vpc_security_group_ids = [aws_security_group.healing_sg.id]
  subnet_id              = data.aws_subnets.default.ids[0]

  user_data = <<-EOF
    #!/bin/bash
    yum update -y
    yum install -y httpd stress-ng
    systemctl start httpd
    systemctl enable httpd
    echo "<h1>Self-Healing Infrastructure - Target Instance (Mumbai)</h1>" > /var/www/html/index.html

    # Install CloudWatch Agent
    yum install -y amazon-cloudwatch-agent

    # Write CloudWatch Agent config
    cat > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json << 'CWCONFIG'
    {
      "metrics": {
        "namespace": "SelfHealingInfra",
        "metrics_collected": {
          "procstat": [
            {
              "pattern": "httpd",
              "measurement": ["pid_count"],
              "metrics_collection_interval": 30
            }
          ],
          "disk": {
            "measurement": ["used_percent"],
            "metrics_collection_interval": 60,
            "resources": ["/"]
          }
        }
      }
    }
    CWCONFIG

    # Start CloudWatch Agent
    /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
      -a fetch-config \
      -m ec2 \
      -s \
      -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
  EOF

  tags = {
    Name    = "${var.project_name}-target"
    Project = var.project_name
  }
}

What does the CloudWatch Agent config do?

The procstat plugin watches for processes matching the pattern httpd. Every 30 seconds it publishes procstat_lookup_pid_count to the SelfHealingInfra namespace. When httpd stops, this metric drops to 0, which is what triggers the alarm in the next step.

The disk section publishes used_percent every 60 seconds. This enables the disk-fill chaos scenario later.

Why SSM instead of SSH for remediation?

SSH requires managing key pairs and opening port 22 to automation. SSM Run Command uses IAM permissions and the SSM Agent already installed on Amazon Linux 2023. Lambda can execute commands without network-level access.

This is the production pattern. Real SRE teams use SSM for automated remediation because it leaves a full audit trail in CloudTrail.

• Save ec2.tf.

✔️ Awesome, I've got everything!

Great. Double check you have saved ec2.tf with all four sections: data sources, security group, IAM role, and EC2 instance.

ⓧ I'd like to double check the full code

data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

data "aws_vpc" "default" {
  default = true
}

data "aws_subnets" "default" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.default.id]
  }
}

resource "aws_security_group" "healing_sg" {
  name        = "${var.project_name}-sg"
  description = "Allow SSH and HTTP for self-healing demo"
  vpc_id      = data.aws_vpc.default.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name    = "${var.project_name}-sg"
    Project = var.project_name
  }
}

resource "aws_iam_role" "ec2_role" {
  name = "${var.project_name}-ec2-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })

  tags = {
    Project = var.project_name
  }
}

resource "aws_iam_role_policy_attachment" "ec2_ssm" {
  role       = aws_iam_role.ec2_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

resource "aws_iam_role_policy_attachment" "ec2_cloudwatch" {
  role       = aws_iam_role.ec2_role.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}

resource "aws_iam_instance_profile" "ec2_profile" {
  name = "${var.project_name}-ec2-profile"
  role = aws_iam_role.ec2_role.name
}

resource "aws_instance" "target" {
  ami                    = data.aws_ami.amazon_linux.id
  instance_type          = var.instance_type
  iam_instance_profile   = aws_iam_instance_profile.ec2_profile.name
  vpc_security_group_ids = [aws_security_group.healing_sg.id]
  subnet_id              = data.aws_subnets.default.ids[0]

  user_data = <<-EOF
    #!/bin/bash
    yum update -y
    yum install -y httpd stress-ng
    systemctl start httpd
    systemctl enable httpd
    echo "<h1>Self-Healing Infrastructure - Target Instance (Mumbai)</h1>" > /var/www/html/index.html

    # Install CloudWatch Agent
    yum install -y amazon-cloudwatch-agent

    # Write CloudWatch Agent config
    cat > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json << 'CWCONFIG'
    {
      "metrics": {
        "namespace": "SelfHealingInfra",
        "metrics_collected": {
          "procstat": [
            {
              "pattern": "httpd",
              "measurement": ["pid_count"],
              "metrics_collection_interval": 30
            }
          ],
          "disk": {
            "measurement": ["used_percent"],
            "metrics_collection_interval": 60,
            "resources": ["/"]
          }
        }
      }
    }
    CWCONFIG

    # Start CloudWatch Agent
    /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
      -a fetch-config \
      -m ec2 \
      -s \
      -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
  EOF

  tags = {
    Name    = "${var.project_name}-target"
    Project = var.project_name
  }
}

Deploy and verify the infrastructure

With the configuration complete, deploy the infrastructure and confirm that httpd, SSM, and the CloudWatch Agent are all working.

• Deploy the infrastructure by running this command from your terraform/ directory:

terraform apply -var="notification_email=[[NOTIFICATION_EMAIL="your email address"]]"

What happens during apply?

Terraform will show a plan of 7 resources to create (AMI data, VPC data, subnets data, security group, IAM role, instance profile, and the EC2 instance). Review the plan and type yes when prompted.

The EC2 instance takes 2-3 minutes to fully boot and run the user_data script. Wait for Terraform to report Apply complete! before continuing.

• After terraform apply completes, get the instance's public IP by running:

terraform output instance_public_ip

No output defined yet?

If this command fails, the outputs.tf file is created in Step 5. For now, find the public IP in the AWS EC2 console or run terraform show and look for public_ip in the aws_instance.target output.

• Verify httpd is serving traffic by running:

curl http://[[EC2_PUBLIC_IP="your EC2 public IP address"]]

• You should see the HTML response: <h1>Self-Healing Infrastructure - Target Instance (Mumbai)</h1>.
• Verify the instance is registered with SSM by running:

aws ssm describe-instance-information

• You should see your instance in the output with "PingStatus": "Online". This confirms SSM Agent is running and Lambda will be able to execute commands on the instance.

Instance not showing in SSM?

The SSM Agent takes 1-2 minutes to register after boot. Wait a minute and try again. If it still does not appear, verify the IAM role self-healing-infra-ec2-role has the AmazonSSMManagedInstanceCore policy attached in the IAM console.

Also confirm the security group allows outbound traffic (the egress rule). SSM Agent communicates outbound to the SSM endpoint.

Why isn't my instance registering with SSM?

• Finally, verify that custom metrics are publishing. Wait 2-3 minutes after the instance launches, then check CloudWatch:

aws cloudwatch list-metrics --namespace SelfHealingInfra

• You should see metrics listed including procstat_lookup_pid_count with the dimension pattern: httpd. This is the metric your alarm will monitor in the next step.

No metrics showing up?

The CloudWatch Agent takes 2-3 minutes to publish its first data point. If you do not see any metrics after 5 minutes, SSH into the instance and check the agent log:

Look in /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log for error messages. Common issues include the agent config file not being found or the IAM role missing the CloudWatchAgentServerPolicy.

Why aren't my CloudWatch Agent metrics appearing in the SelfHealingInfra namespace?

Your target instance is live, serving traffic, and reporting health metrics every 30 seconds. Next up, you will build the Lambda function that watches these metrics and automatically heals the instance when something breaks.

Build the Graduated Healing Engine

Your EC2 instance is now monitored. The CloudWatch Agent publishes process and disk metrics every 30-60 seconds. But monitoring alone is just observing the problem. Without automation, you still need a human to notice an alert, SSH in, diagnose, and restart services manually.

In this step, you will build the brain of the platform: a graduated healing engine. Instead of blindly restarting everything, it applies escalating remediation. First attempt gets a soft fix (restart the service). If the same failure recurs within 10 minutes, it escalates to a hard fix (reboot the instance). If it fails again, it escalates to a human. This is the difference between a junior script and senior-level operational thinking.

In this step, get ready to:

• Write the graduated remediation Lambda function with attempt tracking and MTTR calculation.
• Define the Lambda infrastructure with Terraform including IAM permissions, packaging, and environment variables.
• Create CloudWatch Alarms, a Composite Alarm, and EventBridge routing to wire everything together.

Write the graduated healing Lambda function

The healing engine is a single Python file that receives alarm events, classifies the failure, checks recent attempt history, and applies the appropriate level of remediation.

• Create the file lambda/heal_handler.py in your project's lambda/ directory.
• Start with the imports, constants, and the main handler function by pasting this code:

import json
import os
import time
import uuid
from datetime import datetime, timezone, timedelta

import boto3
from boto3.dynamodb.conditions import Key

# Initialize AWS service clients
ssm_client = boto3.client("ssm")
ec2_client = boto3.client("ec2")
dynamodb = boto3.resource("dynamodb")
sns_client = boto3.client("sns")
cloudwatch_client = boto3.client("cloudwatch")

# Read environment variables set by Terraform
INSTANCE_ID = os.environ["INSTANCE_ID"]
TABLE_NAME = os.environ["TABLE_NAME"]
SNS_TOPIC_ARN = os.environ["SNS_TOPIC_ARN"]
PROJECT_NAME = os.environ["PROJECT_NAME"]

# Escalation window: if same failure recurs within 10 min, escalate
ESCALATION_WINDOW_MINUTES = 10
MAX_ATTEMPTS_BEFORE_ESCALATE = 3


def lambda_handler(event, context):
    print(f"Received event: {json.dumps(event)}")

    # Extract alarm details from the EventBridge event
    alarm_name = event["detail"]["alarmName"]
    timestamp = datetime.now(timezone.utc)
    incident_id = str(uuid.uuid4())

    # Classify the failure and check escalation state
    failure_type = determine_failure_type(alarm_name)
    severity = determine_severity(alarm_name)
    attempt_count = get_recent_attempt_count(failure_type, timestamp)

    print(f"Failure: {failure_type} | Severity: {severity} | Attempt: {attempt_count + 1}")

    # Mark service as down
    publish_availability_metric(0)

    # Apply graduated remediation based on attempt history
    remediation_result, action_taken = execute_graduated_remediation(
        failure_type, attempt_count
    )

    # Mark service as recovered if fix succeeded
    if remediation_result == "success":
        publish_availability_metric(1)

    # Calculate time to resolve
    mttr = calculate_mttr(timestamp)

    # Log the full incident to DynamoDB
    log_incident(
        incident_id, timestamp, failure_type, alarm_name,
        remediation_result, action_taken, attempt_count + 1, severity, mttr
    )

    # Notify operations team via SNS
    send_notification(
        failure_type, remediation_result, action_taken,
        timestamp, incident_id, severity, attempt_count + 1
    )

    # Publish healing metrics for dashboard tracking
    publish_healing_metric(failure_type, remediation_result)

    return {
        "statusCode": 200,
        "body": json.dumps({
            "incident_id": incident_id,
            "failure_type": failure_type,
            "action": action_taken,
            "result": remediation_result,
            "attempt": attempt_count + 1
        })
    }

What does this code do?

This is the orchestrator. When EventBridge delivers a CloudWatch alarm event, lambda_handler runs through a fixed sequence: classify the failure, check how many times it has happened recently, pick the right level of fix, log the result, and notify the team.

The ESCALATION_WINDOW_MINUTES constant defines the recurrence window. If the same failure type fires again within 10 minutes, the engine assumes the previous fix did not hold and escalates.

• Below the lambda_handler function, add the classification and attempt-tracking functions:

def determine_failure_type(alarm_name):
    """Map alarm name to a canonical failure type."""
    if "httpd-down" in alarm_name:
        return "process_down"
    elif "cpu-high" in alarm_name:
        return "cpu_spike"
    elif "cascading-failure" in alarm_name:
        return "cascading_failure"
    else:
        return "unknown"


def determine_severity(alarm_name):
    """Assign severity based on alarm source."""
    if "cascading" in alarm_name:
        return "CRITICAL"
    elif "httpd-down" in alarm_name:
        return "HIGH"
    elif "cpu-high" in alarm_name:
        return "WARNING"
    return "INFO"


def get_recent_attempt_count(failure_type, current_time):
    """Query DynamoDB for recent incidents of the same failure type."""
    table = dynamodb.Table(TABLE_NAME)
    window_start = (current_time - timedelta(minutes=ESCALATION_WINDOW_MINUTES)).isoformat()

    response = table.query(
        IndexName="failure-type-index",
        KeyConditionExpression=Key("failure_type").eq(failure_type) & Key("timestamp").gte(window_start)
    )

    return response["Count"]

What does this code do?

These three functions handle classification and state lookup. determine_failure_type maps alarm names to canonical failure categories. determine_severity assigns a severity level that controls notification urgency.

The key function is get_recent_attempt_count. It queries the DynamoDB Global Secondary Index for incidents of the same failure type within the last 10 minutes. This count drives the escalation decision.

• Below the classification functions, add the graduated remediation execution logic:

def execute_graduated_remediation(failure_type, attempt_count):
    """Choose remediation level based on attempt history."""
    if failure_type == "unknown":
        return "escalated", "none"

    # After max attempts, stop automating and escalate to human
    if attempt_count >= MAX_ATTEMPTS_BEFORE_ESCALATE:
        return "escalated", "human_escalation"

    # Graduated response: soft fix -> hard fix -> reboot
    if attempt_count == 0:
        return execute_soft_fix(failure_type), "soft_fix"
    elif attempt_count == 1:
        return execute_hard_fix(failure_type), "hard_fix"
    else:
        return execute_reboot(), "reboot"


def execute_soft_fix(failure_type):
    """First-level fix: restart the affected service."""
    commands_map = {
        "process_down": ["systemctl restart httpd", "sleep 2", "systemctl is-active httpd"],
        "cpu_spike": ["pkill -f stress-ng || true", "sleep 2", "top -bn1 | head -5"],
        "cascading_failure": [
            "pkill -f stress-ng || true",
            "systemctl restart httpd",
            "sleep 2",
            "systemctl is-active httpd"
        ]
    }

    commands = commands_map.get(failure_type, [])
    if not commands:
        return "no_action"

    return run_ssm_command(commands, f"soft_fix for {failure_type}")


def execute_hard_fix(failure_type):
    """Second-level fix: forceful process kill and restart."""
    commands = [
        "systemctl stop httpd || true",
        "pkill -9 -f stress-ng || true",
        "pkill -9 -f httpd || true",
        "sleep 3",
        "systemctl start httpd",
        "systemctl is-active httpd"
    ]
    return run_ssm_command(commands, f"hard_fix for {failure_type}")


def execute_reboot():
    """Third-level fix: reboot the entire instance."""
    try:
        ec2_client.reboot_instances(InstanceIds=[INSTANCE_ID])
        print(f"Instance {INSTANCE_ID} reboot initiated")
        return "success"
    except Exception as e:
        print(f"Reboot failed: {str(e)}")
        return "failed"


def run_ssm_command(commands, comment):
    """Execute commands on the EC2 instance via SSM Run Command."""
    try:
        response = ssm_client.send_command(
            InstanceIds=[INSTANCE_ID],
            DocumentName="AWS-RunShellScript",
            Parameters={"commands": commands},
            TimeoutSeconds=30,
            Comment=comment
        )
        command_id = response["Command"]["CommandId"]
        print(f"SSM Command sent: {command_id}")

        # Wait for command to complete
        time.sleep(5)

        result = ssm_client.get_command_invocation(
            CommandId=command_id,
            InstanceId=INSTANCE_ID
        )

        if result["Status"] == "Success":
            print(f"Remediation successful: {result['StandardOutputContent']}")
            return "success"
        else:
            print(f"Remediation failed: {result['StandardErrorContent']}")
            return "failed"
    except Exception as e:
        print(f"SSM error: {str(e)}")
        return "error"

What does this code do?

This is the core graduated escalation logic. execute_graduated_remediation decides what level of fix to apply based on how many recent attempts have already failed.

• Attempt 0 (first time): execute_soft_fix restarts the service via SSM Run Command.
• Attempt 1 (recurred within 10 min): execute_hard_fix kills all related processes forcefully and restarts.
• Attempt 2+: execute_reboot reboots the entire instance via the EC2 API.
• 3+ attempts: stops automating entirely and escalates to a human operator.

The run_ssm_command function sends shell commands to the EC2 instance using the AWS-RunShellScript SSM document, waits for completion, and reports back.

• Finally, add the observability functions at the bottom of the file:

def calculate_mttr(detection_time):
    """Calculate Mean Time To Repair from detection to now."""
    elapsed = (datetime.now(timezone.utc) - detection_time).total_seconds()
    return round(elapsed, 2)


def log_incident(incident_id, timestamp, failure_type, alarm_name,
                 result, action, attempt_count, severity, mttr):
    """Write the full incident record to DynamoDB."""
    table = dynamodb.Table(TABLE_NAME)
    table.put_item(
        Item={
            "incident_id": incident_id,
            "timestamp": timestamp.isoformat(),
            "failure_type": failure_type,
            "alarm_name": alarm_name,
            "remediation_result": result,
            "action_taken": action,
            "attempt_count": attempt_count,
            "severity": severity,
            "mttr_seconds": str(mttr),
            "instance_id": INSTANCE_ID,
            "ttl": int(time.time()) + (30 * 24 * 60 * 60)
        }
    )
    print(f"Incident logged: {incident_id}")


def send_notification(failure_type, result, action, timestamp,
                      incident_id, severity, attempt_count):
    """Send SNS notification with severity-based subject."""
    if result == "escalated":
        subject = f"[{severity}] ESCALATION: {failure_type} requires human intervention"
        message = (
            f"Automated remediation exhausted after {attempt_count} attempts.\n\n"
            f"Incident ID: {incident_id}\n"
            f"Failure Type: {failure_type}\n"
            f"Severity: {severity}\n"
            f"Attempts: {attempt_count}\n"
            f"Timestamp: {timestamp.isoformat()}\n"
            f"Instance: {INSTANCE_ID}\n\n"
            f"ACTION REQUIRED: Manual investigation needed."
        )
    else:
        subject = f"[{severity}] Self-Healed: {failure_type} ({action})"
        message = (
            f"Automated remediation completed.\n\n"
            f"Incident ID: {incident_id}\n"
            f"Failure Type: {failure_type}\n"
            f"Action: {action}\n"
            f"Result: {result}\n"
            f"Attempt: {attempt_count}\n"
            f"Severity: {severity}\n"
            f"Timestamp: {timestamp.isoformat()}\n"
            f"Instance: {INSTANCE_ID}"
        )

    sns_client.publish(
        TopicArn=SNS_TOPIC_ARN,
        Subject=subject[:100],
        Message=message
    )


def publish_availability_metric(value):
    """Publish SLO availability signal (1=up, 0=down)."""
    cloudwatch_client.put_metric_data(
        Namespace="SelfHealingInfra",
        MetricData=[{
            "MetricName": "ServiceAvailability",
            "Dimensions": [{"Name": "Service", "Value": "httpd"}],
            "Value": value,
            "Unit": "None"
        }]
    )


def publish_healing_metric(failure_type, result):
    """Track healing events for dashboard visualization."""
    cloudwatch_client.put_metric_data(
        Namespace="SelfHealingInfra",
        MetricData=[{
            "MetricName": "HealingEvents",
            "Dimensions": [
                {"Name": "FailureType", "Value": failure_type},
                {"Name": "Result", "Value": result}
            ],
            "Value": 1,
            "Unit": "Count"
        }]
    )

What does this code do?

These functions handle the aftermath of every healing attempt.

• log_incident writes a full record to DynamoDB with a 30-day TTL for automatic cleanup.
• send_notification formats different messages for successful heals vs. escalations, with the severity in the subject line.
• publish_availability_metric publishes a 0/1 signal used to calculate SLO availability on the dashboard.
• publish_healing_metric tracks healing event counts by failure type and result.

• Save lambda/heal_handler.py.

Seeing a syntax error?

Make sure all four code sections are pasted in order with no duplicate function definitions. The file should start with the imports and end with publish_healing_metric.

help me check my heal_handler.py for syntax errors

✔️ Awesome, I've got everything!

Great. Double check you have saved lambda/heal_handler.py.

ⓧ I'd like to double check the full code

import json
import os
import time
import uuid
from datetime import datetime, timezone, timedelta

import boto3
from boto3.dynamodb.conditions import Key

ssm_client = boto3.client("ssm")
ec2_client = boto3.client("ec2")
dynamodb = boto3.resource("dynamodb")
sns_client = boto3.client("sns")
cloudwatch_client = boto3.client("cloudwatch")

INSTANCE_ID = os.environ["INSTANCE_ID"]
TABLE_NAME = os.environ["TABLE_NAME"]
SNS_TOPIC_ARN = os.environ["SNS_TOPIC_ARN"]
PROJECT_NAME = os.environ["PROJECT_NAME"]

ESCALATION_WINDOW_MINUTES = 10
MAX_ATTEMPTS_BEFORE_ESCALATE = 3


def lambda_handler(event, context):
    print(f"Received event: {json.dumps(event)}")

    alarm_name = event["detail"]["alarmName"]
    timestamp = datetime.now(timezone.utc)
    incident_id = str(uuid.uuid4())

    failure_type = determine_failure_type(alarm_name)
    severity = determine_severity(alarm_name)
    attempt_count = get_recent_attempt_count(failure_type, timestamp)

    print(f"Failure: {failure_type} | Severity: {severity} | Attempt: {attempt_count + 1}")

    publish_availability_metric(0)

    remediation_result, action_taken = execute_graduated_remediation(
        failure_type, attempt_count
    )

    if remediation_result == "success":
        publish_availability_metric(1)

    mttr = calculate_mttr(timestamp)

    log_incident(
        incident_id, timestamp, failure_type, alarm_name,
        remediation_result, action_taken, attempt_count + 1, severity, mttr
    )

    send_notification(
        failure_type, remediation_result, action_taken,
        timestamp, incident_id, severity, attempt_count + 1
    )

    publish_healing_metric(failure_type, remediation_result)

    return {
        "statusCode": 200,
        "body": json.dumps({
            "incident_id": incident_id,
            "failure_type": failure_type,
            "action": action_taken,
            "result": remediation_result,
            "attempt": attempt_count + 1
        })
    }


def determine_failure_type(alarm_name):
    if "httpd-down" in alarm_name:
        return "process_down"
    elif "cpu-high" in alarm_name:
        return "cpu_spike"
    elif "cascading-failure" in alarm_name:
        return "cascading_failure"
    else:
        return "unknown"


def determine_severity(alarm_name):
    if "cascading" in alarm_name:
        return "CRITICAL"
    elif "httpd-down" in alarm_name:
        return "HIGH"
    elif "cpu-high" in alarm_name:
        return "WARNING"
    return "INFO"


def get_recent_attempt_count(failure_type, current_time):
    table = dynamodb.Table(TABLE_NAME)
    window_start = (current_time - timedelta(minutes=ESCALATION_WINDOW_MINUTES)).isoformat()

    response = table.query(
        IndexName="failure-type-index",
        KeyConditionExpression=Key("failure_type").eq(failure_type) & Key("timestamp").gte(window_start)
    )

    return response["Count"]


def execute_graduated_remediation(failure_type, attempt_count):
    if failure_type == "unknown":
        return "escalated", "none"

    if attempt_count >= MAX_ATTEMPTS_BEFORE_ESCALATE:
        return "escalated", "human_escalation"

    if attempt_count == 0:
        return execute_soft_fix(failure_type), "soft_fix"
    elif attempt_count == 1:
        return execute_hard_fix(failure_type), "hard_fix"
    else:
        return execute_reboot(), "reboot"


def execute_soft_fix(failure_type):
    commands_map = {
        "process_down": ["systemctl restart httpd", "sleep 2", "systemctl is-active httpd"],
        "cpu_spike": ["pkill -f stress-ng || true", "sleep 2", "top -bn1 | head -5"],
        "cascading_failure": [
            "pkill -f stress-ng || true",
            "systemctl restart httpd",
            "sleep 2",
            "systemctl is-active httpd"
        ]
    }

    commands = commands_map.get(failure_type, [])
    if not commands:
        return "no_action"

    return run_ssm_command(commands, f"soft_fix for {failure_type}")


def execute_hard_fix(failure_type):
    commands = [
        "systemctl stop httpd || true",
        "pkill -9 -f stress-ng || true",
        "pkill -9 -f httpd || true",
        "sleep 3",
        "systemctl start httpd",
        "systemctl is-active httpd"
    ]
    return run_ssm_command(commands, f"hard_fix for {failure_type}")


def execute_reboot():
    try:
        ec2_client.reboot_instances(InstanceIds=[INSTANCE_ID])
        print(f"Instance {INSTANCE_ID} reboot initiated")
        return "success"
    except Exception as e:
        print(f"Reboot failed: {str(e)}")
        return "failed"


def run_ssm_command(commands, comment):
    try:
        response = ssm_client.send_command(
            InstanceIds=[INSTANCE_ID],
            DocumentName="AWS-RunShellScript",
            Parameters={"commands": commands},
            TimeoutSeconds=30,
            Comment=comment
        )
        command_id = response["Command"]["CommandId"]
        print(f"SSM Command sent: {command_id}")

        time.sleep(5)

        result = ssm_client.get_command_invocation(
            CommandId=command_id,
            InstanceId=INSTANCE_ID
        )

        if result["Status"] == "Success":
            print(f"Remediation successful: {result['StandardOutputContent']}")
            return "success"
        else:
            print(f"Remediation failed: {result['StandardErrorContent']}")
            return "failed"
    except Exception as e:
        print(f"SSM error: {str(e)}")
        return "error"


def calculate_mttr(detection_time):
    elapsed = (datetime.now(timezone.utc) - detection_time).total_seconds()
    return round(elapsed, 2)


def log_incident(incident_id, timestamp, failure_type, alarm_name,
                 result, action, attempt_count, severity, mttr):
    table = dynamodb.Table(TABLE_NAME)
    table.put_item(
        Item={
            "incident_id": incident_id,
            "timestamp": timestamp.isoformat(),
            "failure_type": failure_type,
            "alarm_name": alarm_name,
            "remediation_result": result,
            "action_taken": action,
            "attempt_count": attempt_count,
            "severity": severity,
            "mttr_seconds": str(mttr),
            "instance_id": INSTANCE_ID,
            "ttl": int(time.time()) + (30 * 24 * 60 * 60)
        }
    )
    print(f"Incident logged: {incident_id}")


def send_notification(failure_type, result, action, timestamp,
                      incident_id, severity, attempt_count):
    if result == "escalated":
        subject = f"[{severity}] ESCALATION: {failure_type} requires human intervention"
        message = (
            f"Automated remediation exhausted after {attempt_count} attempts.\n\n"
            f"Incident ID: {incident_id}\n"
            f"Failure Type: {failure_type}\n"
            f"Severity: {severity}\n"
            f"Attempts: {attempt_count}\n"
            f"Timestamp: {timestamp.isoformat()}\n"
            f"Instance: {INSTANCE_ID}\n\n"
            f"ACTION REQUIRED: Manual investigation needed."
        )
    else:
        subject = f"[{severity}] Self-Healed: {failure_type} ({action})"
        message = (
            f"Automated remediation completed.\n\n"
            f"Incident ID: {incident_id}\n"
            f"Failure Type: {failure_type}\n"
            f"Action: {action}\n"
            f"Result: {result}\n"
            f"Attempt: {attempt_count}\n"
            f"Severity: {severity}\n"
            f"Timestamp: {timestamp.isoformat()}\n"
            f"Instance: {INSTANCE_ID}"
        )

    sns_client.publish(
        TopicArn=SNS_TOPIC_ARN,
        Subject=subject[:100],
        Message=message
    )


def publish_availability_metric(value):
    cloudwatch_client.put_metric_data(
        Namespace="SelfHealingInfra",
        MetricData=[{
            "MetricName": "ServiceAvailability",
            "Dimensions": [{"Name": "Service", "Value": "httpd"}],
            "Value": value,
            "Unit": "None"
        }]
    )


def publish_healing_metric(failure_type, result):
    cloudwatch_client.put_metric_data(
        Namespace="SelfHealingInfra",
        MetricData=[{
            "MetricName": "HealingEvents",
            "Dimensions": [
                {"Name": "FailureType", "Value": failure_type},
                {"Name": "Result", "Value": result}
            ],
            "Value": 1,
            "Unit": "Count"
        }]
    )

Define the Lambda infrastructure with Terraform

The Lambda function needs several supporting resources before Terraform can deploy it: a DynamoDB table to store incidents, an SNS topic for notifications, and IAM permissions for all the AWS services it calls.

• Create terraform/monitoring.tf with the DynamoDB table and SNS topic that the Lambda references:

resource "aws_dynamodb_table" "incidents" {
  name         = "${var.project_name}-incidents"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "incident_id"
  range_key    = "timestamp"

  attribute {
    name = "incident_id"
    type = "S"
  }

  attribute {
    name = "timestamp"
    type = "S"
  }

  attribute {
    name = "failure_type"
    type = "S"
  }

  # GSI enables efficient queries by failure type
  global_secondary_index {
    name            = "failure-type-index"
    hash_key        = "failure_type"
    range_key       = "timestamp"
    projection_type = "ALL"
  }

  tags = {
    Project = var.project_name
  }
}

resource "aws_sns_topic" "healing_notifications" {
  name = "${var.project_name}-notifications"

  tags = {
    Project = var.project_name
  }
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.healing_notifications.arn
  protocol  = "email"
  endpoint  = var.notification_email
}

What does this code do?

The DynamoDB table uses incident_id as the partition key and timestamp as the sort key. The Global Secondary Index on failure_type allows the Lambda to efficiently query "how many process_down incidents happened in the last 10 minutes" without scanning the entire table.

The SNS topic subscription sends email notifications. You will need to confirm the subscription link in your inbox after deploying.

• Create a placeholder lambda/api_handler.py so Terraform can package it. You will replace this with the full API handler in the next step:

import json


def lambda_handler(event, context):
    """Placeholder - replaced in next step with full API logic."""
    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps({"message": "API not yet configured"})
    }

• Create terraform/api.tf with a minimal API Gateway resource. The full endpoint gets built in the next step, but the Lambda permission references this resource:

resource "aws_api_gateway_rest_api" "incidents_api" {
  name        = "${var.project_name}-api"
  description = "Self-Healing Infrastructure Incident API"

  tags = {
    Project = var.project_name
  }
}

• Now create terraform/lambda.tf with the IAM role, Lambda functions, and permissions:

resource "aws_iam_role" "lambda_role" {
  name = "${var.project_name}-lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })

  tags = {
    Project = var.project_name
  }
}

resource "aws_iam_role_policy" "lambda_policy" {
  name = "${var.project_name}-lambda-policy"
  role = aws_iam_role.lambda_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ssm:SendCommand",
          "ssm:GetCommandInvocation"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "ec2:RebootInstances",
          "ec2:DescribeInstances"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "dynamodb:PutItem",
          "dynamodb:Query",
          "dynamodb:Scan"
        ]
        Resource = [
          aws_dynamodb_table.incidents.arn,
          "${aws_dynamodb_table.incidents.arn}/index/*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "sns:Publish"
        ]
        Resource = aws_sns_topic.healing_notifications.arn
      },
      {
        Effect = "Allow"
        Action = [
          "cloudwatch:PutMetricData"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

# Package Lambda source files into zip archives
data "archive_file" "heal_handler_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/heal_handler.py"
  output_path = "${path.module}/../lambda/heal_handler.zip"
}

data "archive_file" "api_handler_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/api_handler.py"
  output_path = "${path.module}/../lambda/api_handler.zip"
}

# Healing engine Lambda - processes alarm events
resource "aws_lambda_function" "healer" {
  filename         = data.archive_file.heal_handler_zip.output_path
  function_name    = "${var.project_name}-healer"
  role             = aws_iam_role.lambda_role.arn
  handler          = "heal_handler.lambda_handler"
  source_code_hash = data.archive_file.heal_handler_zip.output_base64sha256
  runtime          = "python3.12"
  timeout          = 60
  memory_size      = 256

  environment {
    variables = {
      INSTANCE_ID   = aws_instance.target.id
      TABLE_NAME    = aws_dynamodb_table.incidents.name
      SNS_TOPIC_ARN = aws_sns_topic.healing_notifications.arn
      PROJECT_NAME  = var.project_name
    }
  }

  tags = {
    Project = var.project_name
  }
}

# API Lambda - serves incident history (placeholder for now)
resource "aws_lambda_function" "api" {
  filename         = data.archive_file.api_handler_zip.output_path
  function_name    = "${var.project_name}-api"
  role             = aws_iam_role.lambda_role.arn
  handler          = "api_handler.lambda_handler"
  source_code_hash = data.archive_file.api_handler_zip.output_base64sha256
  runtime          = "python3.12"
  timeout          = 10
  memory_size      = 128

  environment {
    variables = {
      TABLE_NAME   = aws_dynamodb_table.incidents.name
      INSTANCE_ID  = aws_instance.target.id
      PROJECT_NAME = var.project_name
    }
  }

  tags = {
    Project = var.project_name
  }
}

# Allow EventBridge to invoke the healer Lambda
resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.healer.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.alarm_state_change.arn
}

# Allow API Gateway to invoke the API Lambda
resource "aws_lambda_permission" "allow_apigw" {
  statement_id  = "AllowAPIGatewayInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.api.function_name
  principal     = "apigateway.amazonaws.com"
  source_arn    = "${aws_api_gateway_rest_api.incidents_api.execution_arn}/*/*"
}

What does this code do?

This file defines the Lambda infrastructure. The IAM policy grants precisely the permissions each function needs: SSM for running commands on the instance, EC2 for reboots, DynamoDB for incident logging, SNS for notifications, CloudWatch for metrics, and CloudWatch Logs for function output.

The healer Lambda gets 256MB memory and a 60-second timeout to accommodate the SSM command round-trip. The API Lambda is lighter at 128MB with a 10-second timeout since it only queries DynamoDB.

The archive_file data sources zip each Python file for upload. The source_code_hash ensures Terraform re-deploys the Lambda whenever the code changes.

• Save terraform/monitoring.tf, terraform/lambda.tf, terraform/api.tf, and lambda/api_handler.py.

Seeing a reference error in Terraform?

If terraform plan reports a reference to an undeclared resource, check that monitoring.tf defines aws_dynamodb_table.incidents and aws_sns_topic.healing_notifications. Also check that api.tf defines aws_api_gateway_rest_api.incidents_api.

help me fix Terraform reference errors in my lambda.tf file

✔️ Awesome, I've got everything!

Great. Double check you have saved all four files.

ⓧ I'd like to double check the full code

resource "aws_iam_role" "lambda_role" {
  name = "${var.project_name}-lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })

  tags = {
    Project = var.project_name
  }
}

resource "aws_iam_role_policy" "lambda_policy" {
  name = "${var.project_name}-lambda-policy"
  role = aws_iam_role.lambda_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ssm:SendCommand",
          "ssm:GetCommandInvocation"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "ec2:RebootInstances",
          "ec2:DescribeInstances"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "dynamodb:PutItem",
          "dynamodb:Query",
          "dynamodb:Scan"
        ]
        Resource = [
          aws_dynamodb_table.incidents.arn,
          "${aws_dynamodb_table.incidents.arn}/index/*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "sns:Publish"
        ]
        Resource = aws_sns_topic.healing_notifications.arn
      },
      {
        Effect = "Allow"
        Action = [
          "cloudwatch:PutMetricData"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

data "archive_file" "heal_handler_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/heal_handler.py"
  output_path = "${path.module}/../lambda/heal_handler.zip"
}

data "archive_file" "api_handler_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/api_handler.py"
  output_path = "${path.module}/../lambda/api_handler.zip"
}

resource "aws_lambda_function" "healer" {
  filename         = data.archive_file.heal_handler_zip.output_path
  function_name    = "${var.project_name}-healer"
  role             = aws_iam_role.lambda_role.arn
  handler          = "heal_handler.lambda_handler"
  source_code_hash = data.archive_file.heal_handler_zip.output_base64sha256
  runtime          = "python3.12"
  timeout          = 60
  memory_size      = 256

  environment {
    variables = {
      INSTANCE_ID   = aws_instance.target.id
      TABLE_NAME    = aws_dynamodb_table.incidents.name
      SNS_TOPIC_ARN = aws_sns_topic.healing_notifications.arn
      PROJECT_NAME  = var.project_name
    }
  }

  tags = {
    Project = var.project_name
  }
}

resource "aws_lambda_function" "api" {
  filename         = data.archive_file.api_handler_zip.output_path
  function_name    = "${var.project_name}-api"
  role             = aws_iam_role.lambda_role.arn
  handler          = "api_handler.lambda_handler"
  source_code_hash = data.archive_file.api_handler_zip.output_base64sha256
  runtime          = "python3.12"
  timeout          = 10
  memory_size      = 128

  environment {
    variables = {
      TABLE_NAME   = aws_dynamodb_table.incidents.name
      INSTANCE_ID  = aws_instance.target.id
      PROJECT_NAME = var.project_name
    }
  }

  tags = {
    Project = var.project_name
  }
}

resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.healer.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.alarm_state_change.arn
}

resource "aws_lambda_permission" "allow_apigw" {
  statement_id  = "AllowAPIGatewayInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.api.function_name
  principal     = "apigateway.amazonaws.com"
  source_arn    = "${aws_api_gateway_rest_api.incidents_api.execution_arn}/*/*"
}

Create alarms and event routing

The Lambda function exists, but nothing triggers it yet. You need CloudWatch Alarms to detect failures and EventBridge rules to route alarm state changes to the Lambda.

• Create terraform/eventbridge.tf with the alarm definitions and event routing:

# Alarm: httpd process count drops below 1
resource "aws_cloudwatch_metric_alarm" "httpd_process" {
  alarm_name          = "${var.project_name}-httpd-down"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 1
  metric_name         = "procstat_lookup_pid_count"
  namespace           = "SelfHealingInfra"
  period              = 60
  statistic           = "Average"
  threshold           = 1
  alarm_description   = "Triggers when httpd process count drops below 1"
  treat_missing_data  = "breaching"

  dimensions = {
    pattern    = "httpd"
    InstanceId = aws_instance.target.id
  }

  tags = {
    Project = var.project_name
  }
}

# Alarm: CPU exceeds 80% for 2 consecutive minutes
resource "aws_cloudwatch_metric_alarm" "cpu_high" {
  alarm_name          = "${var.project_name}-cpu-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 60
  statistic           = "Average"
  threshold           = 80
  alarm_description   = "Triggers when CPU exceeds 80% for 2 minutes"

  dimensions = {
    InstanceId = aws_instance.target.id
  }

  tags = {
    Project = var.project_name
  }
}

What does this code do?

The httpd alarm monitors procstat_lookup_pid_count in the SelfHealingInfra namespace. When this metric drops below 1, httpd has crashed. The treat_missing_data = "breaching" setting means if the CloudWatch Agent stops sending data, the alarm also triggers.

The CPU alarm uses the standard AWS/EC2 namespace. It requires 2 consecutive evaluation periods above 80% before firing, reducing false positives from brief spikes.

• Below the metric alarms in the same file, add the Composite Alarm and EventBridge routing:

# Composite Alarm: fires when BOTH httpd is down AND CPU is high
resource "aws_cloudwatch_composite_alarm" "cascading_failure" {
  alarm_name = "${var.project_name}-cascading-failure"
  alarm_rule = "ALARM(${aws_cloudwatch_metric_alarm.httpd_process.alarm_name}) AND ALARM(${aws_cloudwatch_metric_alarm.cpu_high.alarm_name})"
  alarm_description = "Cascading failure: both httpd down AND CPU high indicates systemic issue"

  tags = {
    Project = var.project_name
  }
}

# EventBridge rule: match alarm state changes to ALARM
resource "aws_cloudwatch_event_rule" "alarm_state_change" {
  name        = "${var.project_name}-alarm-trigger"
  description = "Routes CloudWatch alarm state changes to healing Lambda"

  event_pattern = jsonencode({
    source      = ["aws.cloudwatch"]
    detail-type = ["CloudWatch Alarm State Change"]
    detail = {
      state = {
        value = ["ALARM"]
      }
      alarmName = [
        "${var.project_name}-httpd-down",
        "${var.project_name}-cpu-high",
        "${var.project_name}-cascading-failure"
      ]
    }
  })

  tags = {
    Project = var.project_name
  }
}

# Route matching events to the healing Lambda
resource "aws_cloudwatch_event_target" "lambda_target" {
  rule      = aws_cloudwatch_event_rule.alarm_state_change.name
  target_id = "healing-lambda"
  arn       = aws_lambda_function.healer.arn
}

Why use a Composite Alarm?

A Composite Alarm correlates multiple signals. If httpd crashes AND CPU is spiking simultaneously, that is a fundamentally different failure mode than just a process crash. It likely means the system is under heavy load and the process crash is a symptom, not the root cause.

The composite alarm fires only when both conditions are true. The Lambda's determine_failure_type function classifies this as cascading_failure and applies a broader fix that kills the stress process AND restarts httpd.

How does the EventBridge event pattern work?

CloudWatch sends events to EventBridge whenever an alarm changes state. The detail-type field matches the exact string CloudWatch Alarm State Change. The pattern filters for only the ALARM state (not OK or INSUFFICIENT_DATA) and only for your three specific alarm names.

• Save terraform/eventbridge.tf.
• Deploy all the new infrastructure by running:

cd terraform
terraform apply -var="notification_email=your-email@example.com"

• Review the plan output. Terraform should show new resources for the DynamoDB table, SNS topic, Lambda functions, CloudWatch alarms, and EventBridge rule.
• Type yes when prompted to confirm.

• Verify the Lambda deployed by running:

aws lambda get-function --function-name self-healing-infra-healer --region ap-south-1 --query 'Configuration.[FunctionName,Runtime,Timeout,MemorySize]'

You should see the function name, python3.12 runtime, 60-second timeout, and 256MB memory in the output.

Terraform apply failed?

• If you see a "source file does not exist" error, make sure both lambda/heal_handler.py and lambda/api_handler.py exist in your project. The archive_file data source needs both files to create the zip packages.
• If you see a timeout error, the Lambda might be trying to create before the IAM role propagates. Run terraform apply again. IAM propagation typically resolves within a few seconds.

help me debug my terraform apply failure for the self-healing Lambda infrastructure

✔️ Awesome, I've got everything!

Great. Double check you have saved terraform/eventbridge.tf.

ⓧ I'd like to double check the full code

resource "aws_cloudwatch_metric_alarm" "httpd_process" {
  alarm_name          = "${var.project_name}-httpd-down"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 1
  metric_name         = "procstat_lookup_pid_count"
  namespace           = "SelfHealingInfra"
  period              = 60
  statistic           = "Average"
  threshold           = 1
  alarm_description   = "Triggers when httpd process count drops below 1"
  treat_missing_data  = "breaching"

  dimensions = {
    pattern    = "httpd"
    InstanceId = aws_instance.target.id
  }

  tags = {
    Project = var.project_name
  }
}

resource "aws_cloudwatch_metric_alarm" "cpu_high" {
  alarm_name          = "${var.project_name}-cpu-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 60
  statistic           = "Average"
  threshold           = 80
  alarm_description   = "Triggers when CPU exceeds 80% for 2 minutes"

  dimensions = {
    InstanceId = aws_instance.target.id
  }

  tags = {
    Project = var.project_name
  }
}

resource "aws_cloudwatch_composite_alarm" "cascading_failure" {
  alarm_name = "${var.project_name}-cascading-failure"
  alarm_rule = "ALARM(${aws_cloudwatch_metric_alarm.httpd_process.alarm_name}) AND ALARM(${aws_cloudwatch_metric_alarm.cpu_high.alarm_name})"
  alarm_description = "Cascading failure: both httpd down AND CPU high indicates systemic issue"

  tags = {
    Project = var.project_name
  }
}

resource "aws_cloudwatch_event_rule" "alarm_state_change" {
  name        = "${var.project_name}-alarm-trigger"
  description = "Routes CloudWatch alarm state changes to healing Lambda"

  event_pattern = jsonencode({
    source      = ["aws.cloudwatch"]
    detail-type = ["CloudWatch Alarm State Change"]
    detail = {
      state = {
        value = ["ALARM"]
      }
      alarmName = [
        "${var.project_name}-httpd-down",
        "${var.project_name}-cpu-high",
        "${var.project_name}-cascading-failure"
      ]
    }
  })

  tags = {
    Project = var.project_name
  }
}

resource "aws_cloudwatch_event_target" "lambda_target" {
  rule      = aws_cloudwatch_event_rule.alarm_state_change.name
  target_id = "healing-lambda"
  arn       = aws_lambda_function.healer.arn
}

Your healing engine is wired up. The full event-driven chain is now live: CloudWatch detects a failure, EventBridge routes the event, and Lambda applies graduated remediation. Next, you will build the incident API that makes this system instantly demo-able with a single curl command.

Build the Incident API and Notifications

Your healing engine can detect failures and fix them automatically. But without a way to see what happened, you have no proof it works.

In this step, you will build a live API Gateway endpoint that serves incident history as JSON. You will also create the DynamoDB table that stores incidents, SNS notifications for alerts, and a CloudWatch dashboard for visual monitoring. The result is a single curl command that proves your platform works.

In this step, get ready to:

• Write a Lambda function that serves incident data as a JSON API.
• Create the API Gateway, DynamoDB table, SNS topic, and CloudWatch dashboard via Terraform.
• Deploy everything and verify the live API endpoint responds.

Write the incident API handler

The healer Lambda logs incidents to DynamoDB. Now you need a second Lambda function that reads from that table and serves the data as JSON over HTTP.

• Create the API handler file by running:

touch lambda/api_handler.py

• Open lambda/api_handler.py in your editor and add the imports, globals, helper class, and main handler function:

import json
import os
from decimal import Decimal

import boto3
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource("dynamodb")

TABLE_NAME = os.environ["TABLE_NAME"]
INSTANCE_ID = os.environ["INSTANCE_ID"]
PROJECT_NAME = os.environ["PROJECT_NAME"]


class DecimalEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Decimal):
            return float(obj)
        return super().default(obj)


def lambda_handler(event, context):
    # Extract query parameters from the API Gateway event
    params = event.get("queryStringParameters") or {}
    limit = int(params.get("last", "10"))
    failure_filter = params.get("type", None)

    table = dynamodb.Table(TABLE_NAME)

    # Query by failure type if filter is provided, otherwise scan
    if failure_filter:
        response = table.query(
            IndexName="failure-type-index",
            KeyConditionExpression=Key("failure_type").eq(failure_filter),
            ScanIndexForward=False,
            Limit=limit
        )
    else:
        response = table.scan(Limit=limit)

    incidents = response.get("Items", [])

    # Sort by timestamp descending and apply limit
    incidents.sort(key=lambda x: x.get("timestamp", ""), reverse=True)
    incidents = incidents[:limit]

    stats = calculate_stats(incidents)

    # Build the JSON response body
    body = {
        "project": PROJECT_NAME,
        "instance_id": INSTANCE_ID,
        "stats": stats,
        "incidents": incidents
    }

    return {
        "statusCode": 200,
        "headers": {
            "Content-Type": "application/json",
            "Access-Control-Allow-Origin": "*"
        },
        "body": json.dumps(body, cls=DecimalEncoder)
    }

What does this code do?

This is the entry point for your incident API. When someone hits your endpoint, API Gateway forwards the request to this function.

• It reads ?last=N and ?type=<failure_type> query parameters to filter results.
• If a type filter is provided, it queries the Global Secondary Index for efficiency. Otherwise it scans the whole table.
• The DecimalEncoder class handles DynamoDB's Decimal type, which is not JSON-serializable by default.
• The Access-Control-Allow-Origin header enables CORS. For Lambda proxy integrations, your backend must return CORS headers directly.

• Below the lambda_handler function, add the calculate_stats helper that computes summary metrics from the incidents:

def calculate_stats(incidents):
    # Return empty stats if no incidents exist
    if not incidents:
        return {
            "total_incidents": 0,
            "success_rate": "N/A",
            "avg_mttr_seconds": "N/A",
            "availability_percent": "100.0"
        }

    total = len(incidents)
    successes = sum(1 for i in incidents if i.get("remediation_result") == "success")

    # Collect MTTR values, skipping any non-numeric entries
    mttr_values = []
    for i in incidents:
        mttr = i.get("mttr_seconds")
        if mttr and mttr != "N/A":
            try:
                mttr_values.append(float(mttr))
            except (ValueError, TypeError):
                pass

    avg_mttr = round(sum(mttr_values) / len(mttr_values), 2) if mttr_values else "N/A"

    success_rate = round((successes / total) * 100, 1) if total > 0 else 0

    return {
        "total_incidents": total,
        "success_rate": f"{success_rate}%",
        "avg_mttr_seconds": avg_mttr,
        "auto_healed": successes,
        "escalated": total - successes
    }

What does calculate_stats do?

This function crunches the numbers that make your API response interview-ready. It calculates how many incidents healed automatically, the average Mean Time To Recovery, and a success rate percentage.

When you curl your endpoint later, these stats appear at the top of the JSON response. An interviewer sees your platform's effectiveness at a glance.

• Save lambda/api_handler.py.

Seeing a syntax error?

Make sure the calculate_stats function is at the top level of the file (no indentation), not nested inside lambda_handler. Also check that the f-string f"{success_rate}%" has matching curly braces.

Help me debug my api_handler.py file

✔️ Awesome, I've got everything!

Great. Double check you have saved lambda/api_handler.py before moving on.

ⓧ I'd like to double check the full code

import json
import os
from decimal import Decimal

import boto3
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource("dynamodb")

TABLE_NAME = os.environ["TABLE_NAME"]
INSTANCE_ID = os.environ["INSTANCE_ID"]
PROJECT_NAME = os.environ["PROJECT_NAME"]


class DecimalEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Decimal):
            return float(obj)
        return super().default(obj)


def lambda_handler(event, context):
    params = event.get("queryStringParameters") or {}
    limit = int(params.get("last", "10"))
    failure_filter = params.get("type", None)

    table = dynamodb.Table(TABLE_NAME)

    if failure_filter:
        response = table.query(
            IndexName="failure-type-index",
            KeyConditionExpression=Key("failure_type").eq(failure_filter),
            ScanIndexForward=False,
            Limit=limit
        )
    else:
        response = table.scan(Limit=limit)

    incidents = response.get("Items", [])

    incidents.sort(key=lambda x: x.get("timestamp", ""), reverse=True)
    incidents = incidents[:limit]

    stats = calculate_stats(incidents)

    body = {
        "project": PROJECT_NAME,
        "instance_id": INSTANCE_ID,
        "stats": stats,
        "incidents": incidents
    }

    return {
        "statusCode": 200,
        "headers": {
            "Content-Type": "application/json",
            "Access-Control-Allow-Origin": "*"
        },
        "body": json.dumps(body, cls=DecimalEncoder)
    }


def calculate_stats(incidents):
    if not incidents:
        return {
            "total_incidents": 0,
            "success_rate": "N/A",
            "avg_mttr_seconds": "N/A",
            "availability_percent": "100.0"
        }

    total = len(incidents)
    successes = sum(1 for i in incidents if i.get("remediation_result") == "success")

    mttr_values = []
    for i in incidents:
        mttr = i.get("mttr_seconds")
        if mttr and mttr != "N/A":
            try:
                mttr_values.append(float(mttr))
            except (ValueError, TypeError):
                pass

    avg_mttr = round(sum(mttr_values) / len(mttr_values), 2) if mttr_values else "N/A"

    success_rate = round((successes / total) * 100, 1) if total > 0 else 0

    return {
        "total_incidents": total,
        "success_rate": f"{success_rate}%",
        "avg_mttr_seconds": avg_mttr,
        "auto_healed": successes,
        "escalated": total - successes
    }

Create the API Gateway

API Gateway gives your Lambda function a public URL. You will use a Lambda proxy integration, which means API Gateway passes the entire HTTP request straight to your function and returns whatever your function sends back.

• Create terraform/api.tf and add the following configuration:

resource "aws_api_gateway_rest_api" "incidents_api" {
  name        = "${var.project_name}-api"
  description = "Self-Healing Infrastructure Incident API"

  tags = {
    Project = var.project_name
  }
}

resource "aws_api_gateway_resource" "incidents" {
  rest_api_id = aws_api_gateway_rest_api.incidents_api.id
  parent_id   = aws_api_gateway_rest_api.incidents_api.root_resource_id
  path_part   = "incidents"
}

resource "aws_api_gateway_method" "get_incidents" {
  rest_api_id   = aws_api_gateway_rest_api.incidents_api.id
  resource_id   = aws_api_gateway_resource.incidents.id
  http_method   = "GET"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "lambda_integration" {
  rest_api_id             = aws_api_gateway_rest_api.incidents_api.id
  resource_id             = aws_api_gateway_resource.incidents.id
  http_method             = aws_api_gateway_method.get_incidents.http_method
  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  uri                     = aws_lambda_function.api.invoke_arn
}

resource "aws_api_gateway_deployment" "prod" {
  depends_on  = [aws_api_gateway_integration.lambda_integration]
  rest_api_id = aws_api_gateway_rest_api.incidents_api.id
  stage_name  = "prod"
}

What does this Terraform do?

• Creates a REST API named self-healing-infra-api.
• Adds a /incidents resource with a public GET method (no authorization required).
• The integration type AWS_PROXY forwards the raw HTTP request to your Lambda. The integration_http_method is POST because Lambda invocations always use POST regardless of the client's HTTP method.
• The deployment creates a prod stage, giving you a live URL immediately.

• Save terraform/api.tf.

Seeing a reference error for aws_lambda_function.api?

The aws_lambda_function.api resource is defined in lambda.tf from the previous step. Make sure that file is saved and that the resource name matches exactly.

Help me fix a Terraform reference error in api.tf

Add monitoring infrastructure and deploy

The healing Lambda references a DynamoDB table and an SNS topic that do not exist yet. You also need a CloudWatch dashboard to visualize your platform's health. This is where you define all three.

• Create terraform/monitoring.tf and add the DynamoDB table and SNS resources:

resource "aws_dynamodb_table" "incidents" {
  name         = "${var.project_name}-incidents"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "incident_id"
  range_key    = "timestamp"

  attribute {
    name = "incident_id"
    type = "S"
  }

  attribute {
    name = "timestamp"
    type = "S"
  }

  attribute {
    name = "failure_type"
    type = "S"
  }

  global_secondary_index {
    name            = "failure-type-index"
    hash_key        = "failure_type"
    range_key       = "timestamp"
    projection_type = "ALL"
  }

  tags = {
    Project = var.project_name
  }
}

resource "aws_sns_topic" "healing_notifications" {
  name = "${var.project_name}-notifications"

  tags = {
    Project = var.project_name
  }
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.healing_notifications.arn
  protocol  = "email"
  endpoint  = var.notification_email
}

Why a Global Secondary Index?

The GSI on failure_type lets your API query incidents by type without scanning the entire table. When someone calls ?type=process_down, DynamoDB uses the index to return results instantly instead of reading every row.

The PAY_PER_REQUEST billing mode means you pay nothing when the table is idle. This stays well within the DynamoDB free tier of 25 GB storage.

• Below the SNS subscription in the same file, add the CloudWatch dashboard:

resource "aws_cloudwatch_dashboard" "healing" {
  dashboard_name = "${var.project_name}-dashboard"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x      = 0
        y      = 0
        width  = 12
        height = 6
        properties = {
          title   = "EC2 CPU Utilization"
          metrics = [["AWS/EC2", "CPUUtilization", "InstanceId", aws_instance.target.id]]
          period  = 60
          stat    = "Average"
          region  = var.aws_region
          view    = "timeSeries"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 0
        width  = 12
        height = 6
        properties = {
          title   = "httpd Process Count"
          metrics = [["SelfHealingInfra", "procstat_lookup_pid_count", "pattern", "httpd", "InstanceId", aws_instance.target.id]]
          period  = 60
          stat    = "Average"
          region  = var.aws_region
          view    = "timeSeries"
        }
      },
      {
        type   = "metric"
        x      = 0
        y      = 6
        width  = 12
        height = 6
        properties = {
          title   = "Healing Events (Lambda)"
          metrics = [
            ["AWS/Lambda", "Invocations", "FunctionName", "${var.project_name}-healer"],
            ["AWS/Lambda", "Errors", "FunctionName", "${var.project_name}-healer"]
          ]
          period = 300
          stat   = "Sum"
          region = var.aws_region
          view   = "bar"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 6
        width  = 12
        height = 6
        properties = {
          title  = "Service Availability (SLO)"
          metrics = [
            ["SelfHealingInfra", "ServiceAvailability", "Service", "httpd"]
          ]
          period = 300
          stat   = "Average"
          region = var.aws_region
          view   = "timeSeries"
          yAxis = {
            left = { min = 0, max = 1 }
          }
        }
      }
    ]
  })
}

What do the dashboard widgets show?

• CPU Utilization: real-time EC2 load. Spikes here trigger the cpu-high alarm.
• httpd Process Count: drops to 0 when the process crashes. Fires the httpd-down alarm.
• Healing Events: shows how often your Lambda runs and whether it errors.
• Service Availability (SLO): tracks uptime as a 0-to-1 metric published by your healer Lambda on each incident.

• Save terraform/monitoring.tf.

• Create terraform/outputs.tf to expose the key values you will need for testing and demos:

output "instance_public_ip" {
  description = "Public IP of the target EC2 instance"
  value       = aws_instance.target.public_ip
}

output "instance_id" {
  description = "Instance ID of the target EC2 instance"
  value       = aws_instance.target.id
}

output "api_endpoint" {
  description = "URL of the incidents API"
  value       = "${aws_api_gateway_deployment.prod.invoke_url}/incidents"
}

output "lambda_function_name" {
  description = "Name of the healing Lambda function"
  value       = aws_lambda_function.healer.function_name
}

output "dynamodb_table_name" {
  description = "Name of the incidents DynamoDB table"
  value       = aws_dynamodb_table.incidents.name
}

output "dashboard_url" {
  description = "URL to the CloudWatch dashboard"
  value       = "https://${var.aws_region}.console.aws.amazon.com/cloudwatch/home?region=${var.aws_region}#dashboards:name=${var.project_name}-dashboard"
}

Why outputs matter

Terraform outputs let you retrieve deployed values without digging through the AWS console. The api_endpoint output is the URL you will curl during interviews.

• Save terraform/outputs.tf.
• Deploy all new resources by running:

cd terraform
terraform apply -var="notification_email=your-email@example.com" -auto-approve

What should I see?

Terraform will show resources being created: the DynamoDB table, SNS topic, API Gateway resources, CloudWatch dashboard, and the deployment. At the end, it prints your outputs including the api_endpoint URL.

• Verify your API is live by running:

curl $(terraform output -raw api_endpoint)

• You should see a JSON response with "total_incidents": 0 and an empty incidents array. This confirms the full chain works: API Gateway forwards to Lambda, Lambda queries DynamoDB, and returns JSON.

Getting a 500 Internal Server Error?

• Check that lambda/api_handler.py is saved and that Terraform packaged it correctly. Run terraform apply again to force a re-zip.
• Verify the Lambda has the correct environment variables by checking the TABLE_NAME references the table that was just created.
• Check CloudWatch Logs for the self-healing-infra-api function to see the exact error message.

Help me debug my API Gateway returning a 500 error

• Check your email inbox and confirm the SNS subscription by clicking the confirmation link. You will receive healing notifications at this address.

Why confirm the subscription?

AWS requires email confirmation to prevent spam. Until you click the link, your healing notifications will not be delivered. The confirmation email comes from no-reply@sns.amazonaws.com.

✔️ Awesome, I've got everything!

Great. Make sure all files are saved and the terraform apply completed without errors.

ⓧ I'd like to double check the full code

Here is the full terraform/monitoring.tf:

resource "aws_dynamodb_table" "incidents" {
  name         = "${var.project_name}-incidents"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "incident_id"
  range_key    = "timestamp"

  attribute {
    name = "incident_id"
    type = "S"
  }

  attribute {
    name = "timestamp"
    type = "S"
  }

  attribute {
    name = "failure_type"
    type = "S"
  }

  global_secondary_index {
    name            = "failure-type-index"
    hash_key        = "failure_type"
    range_key       = "timestamp"
    projection_type = "ALL"
  }

  tags = {
    Project = var.project_name
  }
}

resource "aws_sns_topic" "healing_notifications" {
  name = "${var.project_name}-notifications"

  tags = {
    Project = var.project_name
  }
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.healing_notifications.arn
  protocol  = "email"
  endpoint  = var.notification_email
}

resource "aws_cloudwatch_dashboard" "healing" {
  dashboard_name = "${var.project_name}-dashboard"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x      = 0
        y      = 0
        width  = 12
        height = 6
        properties = {
          title   = "EC2 CPU Utilization"
          metrics = [["AWS/EC2", "CPUUtilization", "InstanceId", aws_instance.target.id]]
          period  = 60
          stat    = "Average"
          region  = var.aws_region
          view    = "timeSeries"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 0
        width  = 12
        height = 6
        properties = {
          title   = "httpd Process Count"
          metrics = [["SelfHealingInfra", "procstat_lookup_pid_count", "pattern", "httpd", "InstanceId", aws_instance.target.id]]
          period  = 60
          stat    = "Average"
          region  = var.aws_region
          view    = "timeSeries"
        }
      },
      {
        type   = "metric"
        x      = 0
        y      = 6
        width  = 12
        height = 6
        properties = {
          title   = "Healing Events (Lambda)"
          metrics = [
            ["AWS/Lambda", "Invocations", "FunctionName", "${var.project_name}-healer"],
            ["AWS/Lambda", "Errors", "FunctionName", "${var.project_name}-healer"]
          ]
          period = 300
          stat   = "Sum"
          region = var.aws_region
          view   = "bar"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 6
        width  = 12
        height = 6
        properties = {
          title  = "Service Availability (SLO)"
          metrics = [
            ["SelfHealingInfra", "ServiceAvailability", "Service", "httpd"]
          ]
          period = 300
          stat   = "Average"
          region = var.aws_region
          view   = "timeSeries"
          yAxis = {
            left = { min = 0, max = 1 }
          }
        }
      }
    ]
  })
}

Here is the full terraform/outputs.tf:

output "instance_public_ip" {
  description = "Public IP of the target EC2 instance"
  value       = aws_instance.target.public_ip
}

output "instance_id" {
  description = "Instance ID of the target EC2 instance"
  value       = aws_instance.target.id
}

output "api_endpoint" {
  description = "URL of the incidents API"
  value       = "${aws_api_gateway_deployment.prod.invoke_url}/incidents"
}

output "lambda_function_name" {
  description = "Name of the healing Lambda function"
  value       = aws_lambda_function.healer.function_name
}

output "dynamodb_table_name" {
  description = "Name of the incidents DynamoDB table"
  value       = aws_dynamodb_table.incidents.name
}

output "dashboard_url" {
  description = "URL to the CloudWatch dashboard"
  value       = "https://${var.aws_region}.console.aws.amazon.com/cloudwatch/home?region=${var.aws_region}#dashboards:name=${var.project_name}-dashboard"
}

Your incident API is live and returning data. Next, you will put the entire self-healing loop to the test by simulating a real failure and watching the system detect, heal, and log it automatically.

Test the Self-Healing Loop

Everything is deployed. Your alarms, Lambda, EventBridge rules, DynamoDB table, and API endpoint are all wired up and waiting.

But deployed infrastructure that has never been tested is a liability, not an asset. In this step, you will simulate a real process crash and watch the entire healing chain fire end-to-end.

In this step, get ready to:

• Confirm your SNS email subscription so healing notifications reach your inbox.
• Simulate a process crash by stopping httpd on your EC2 instance.
• Verify the full healing chain via the live API endpoint.

Confirm your SNS subscription and inject failure

Before triggering any incidents, you need to confirm the SNS email subscription. Without confirmation, your healing notifications will never arrive.

• Open your email inbox (the address you provided as notification_email in your Terraform variables).
• Find the email from AWS Notifications with the subject line containing "AWS Notification - Subscription Confirmation".
• Click the Confirm subscription link inside the email.
• You should see a confirmation page in your browser stating that your subscription is confirmed.

Don't see the confirmation email?

Check your spam or junk folder. The email comes from no-reply@sns.amazonaws.com and may be filtered automatically.

If you still cannot find it, the subscription may have expired. Run terraform apply again to re-trigger the subscription email.

Help me find or re-send my SNS confirmation email.

Now you will simulate a real failure. The CloudWatch Agent on your EC2 instance monitors the httpd process count every 30 seconds. When that count drops to zero, the alarm fires.

• First, grab your instance ID from Terraform by running:

terraform -chdir=terraform output -raw instance_id

• Copy the instance ID that appears (it looks like i-0abc123def456...).
• Stop the httpd process remotely via SSM Run Command by running:

aws ssm send-command \
    --instance-ids "$(terraform -chdir=terraform output -raw instance_id)" \
    --document-name "AWS-RunShellScript" \
    --parameters '{"commands":["sudo systemctl stop httpd"]}' \
    --comment "Manual chaos: stop httpd" \
    --region ap-south-1

What does this command do?

SSM Run Command lets you execute shell commands on your EC2 instance without needing SSH keys. The AWS-RunShellScript document runs sudo systemctl stop httpd directly on the instance.

This is the same mechanism your healing Lambda uses to restart services. You are using the attacker's tool to simulate the attack.

• You should see JSON output containing a CommandId field. This confirms the stop command was sent successfully.

Seeing an InvalidInstanceId error?

This means SSM cannot reach your instance. Verify the instance is running and the SSM agent is connected by running:

aws ssm describe-instance-information --region ap-south-1 --query "InstanceInformationList[].{Id:InstanceId,Status:PingStatus}"

You should see your instance ID with a PingStatus of Online. If it shows nothing, the SSM agent has not started yet. Wait 2-3 minutes after instance launch for it to register.

Help me debug my SSM connection to EC2.

Watch the graduated healing chain

The alarm evaluates every 60 seconds with 1 evaluation period. After httpd stops, the CloudWatch Agent reports a process count of zero on its next 30-second cycle. Give the system 1-3 minutes to detect and heal.

• Wait 2 minutes for the alarm to fire and the Lambda to execute.
• Check the alarm state by running:

aws cloudwatch describe-alarms \
    --alarm-names "self-healing-infra-httpd-down" \
    --region ap-south-1 \
    --query "MetricAlarms[0].StateValue"

• If healing already completed, you will see "OK" (the alarm returned to normal after the Lambda restarted httpd). If you are fast enough, you might catch it in "ALARM" state.
• Check your email inbox for the SNS notification. You should see a message with a subject line starting with [HIGH] Self-Healed: process_down.

What happened behind the scenes?

Here is the full chain that fired automatically:

• CloudWatch Agent reported httpd process count = 0.
• The self-healing-infra-httpd-down alarm transitioned to ALARM state.
• EventBridge matched the alarm state change event and invoked the healer Lambda.
• The Lambda queried DynamoDB for recent attempts (found 0), so it chose the soft fix path.
• SSM Run Command executed systemctl restart httpd on the instance.
• The Lambda logged the incident to DynamoDB with attempt_count=1, action_taken=soft_fix, and calculated MTTR.
• SNS sent a notification email with severity HIGH.

No email notification after 3 minutes?

Confirm your SNS subscription is active (not pending) by checking the AWS SNS console under Subscriptions. The status should show Confirmed.

If the subscription is confirmed but no email arrives, check the Lambda logs in CloudWatch Logs. Look for the log group /aws/lambda/self-healing-infra-healer to see if the function was invoked.

Help me figure out why my healing Lambda didn't fire.

Verify healing via your live API

The healing happened. The email confirmed it. Now prove it with data by querying your incident API Gateway endpoint.

• Get your API endpoint URL by running:

terraform -chdir=terraform output -raw api_endpoint

• Query the incidents endpoint by running:

curl $(terraform -chdir=terraform output -raw api_endpoint)

• You should see a JSON response containing your incident record. Look for these key fields:

{
  "project": "self-healing-infra",
  "instance_id": "i-0abc...",
  "stats": {
    "total_incidents": 1,
    "success_rate": "100.0%",
    "avg_mttr_seconds": 7.5,
    "auto_healed": 1,
    "escalated": 0
  },
  "incidents": [
    {
      "failure_type": "process_down",
      "action_taken": "soft_fix",
      "attempt_count": 1,
      "remediation_result": "success",
      "severity": "HIGH",
      "mttr_seconds": "7.5"
    }
  ]
}

What do these fields tell you?

The attempt_count: 1 confirms this was the first occurrence. The Lambda chose soft_fix (restart the service) because there were no recent failed attempts.

The mttr_seconds value shows how many seconds elapsed between detection and successful remediation. This is your Mean Time To Recovery. In a real interview, this single curl command proves your system heals itself without human intervention.

Seeing an empty incidents array?

If the response shows "total_incidents": 0, the Lambda either did not fire or failed before writing to DynamoDB. Check the Lambda logs in CloudWatch.

Also confirm the alarm actually transitioned to ALARM state. If the CloudWatch Agent had not yet reported the zero process count before the 60-second evaluation period ended, wait another minute and check again.

My API returns no incidents after stopping httpd.

Your self-healing loop is proven. You injected a failure, watched it heal automatically, and verified the result with a single API call. Next up, you will build a full chaos engineering suite that automates these injections across multiple failure scenarios.

Run the Chaos Engineering Suite

Your platform healed a single failure in the previous step. That proved the end-to-end loop works. But real production systems face varied, repeated, and correlated failures.

Chaos engineering means intentionally breaking things to prove resilience. In this step, you'll build a reusable injection script that simulates four failure scenarios and use it to validate your platform's graduated escalation logic and composite alarm detection.

In this step, get ready to:

• Create a chaos injection script supporting four failure scenarios.
• Validate graduated escalation by triggering repeated failures.
• Test cascading failure detection with the composite alarm.

Create the chaos injection script

A proper chaos engineering tool is scriptable, repeatable, and documents its own usage. Your injection script will use SSM Run Command to execute fault-injection commands directly on the EC2 instance without needing SSH access.

• Create chaos/inject.sh by running:

touch chaos/inject.sh

• Open chaos/inject.sh in your editor and paste the following code:

#!/bin/bash
set -euo pipefail

INSTANCE_ID="${1:-}"
SCENARIO="${2:-}"

if [ -z "$INSTANCE_ID" ] || [ -z "$SCENARIO" ]; then
    echo "Usage: ./inject.sh <instance-id> <scenario>"
    echo ""
    echo "Scenarios:"
    echo "  process-kill  - Kill httpd process"
    echo "  cpu-stress    - Spike CPU to 100% for 120 seconds"
    echo "  disk-fill     - Fill /tmp with 900MB"
    echo "  cascading     - Kill httpd AND spike CPU (triggers composite alarm)"
    echo ""
    exit 1
fi

echo "=== CHAOS INJECTION ==="
echo "Instance: $INSTANCE_ID"
echo "Scenario: $SCENARIO"
echo "Time: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo ""

case "$SCENARIO" in
    process-kill)
        echo "Injecting: Killing httpd process..."
        COMMANDS='["sudo systemctl stop httpd","echo httpd stopped at $(date)"]'
        ;;
    cpu-stress)
        echo "Injecting: CPU stress for 120 seconds..."
        COMMANDS='["nohup stress-ng --cpu 2 --timeout 120 &>/dev/null &","echo CPU stress started at $(date)"]'
        ;;
    disk-fill)
        echo "Injecting: Filling /tmp with 900MB..."
        COMMANDS='["dd if=/dev/zero of=/tmp/chaos-fill bs=1M count=900","echo Disk filled at $(date)"]'
        ;;
    cascading)
        echo "Injecting: Cascading failure (process kill + CPU stress)..."
        COMMANDS='["sudo systemctl stop httpd","nohup stress-ng --cpu 2 --timeout 120 &>/dev/null &","echo Cascading failure injected at $(date)"]'
        ;;
    *)
        echo "Unknown scenario: $SCENARIO"
        exit 1
        ;;
esac

COMMAND_ID=$(aws ssm send-command \
    --instance-ids "$INSTANCE_ID" \
    --document-name "AWS-RunShellScript" \
    --parameters "{\"commands\":$COMMANDS}" \
    --comment "Chaos injection: $SCENARIO" \
    --region ap-south-1 \
    --output text \
    --query "Command.CommandId")

echo "SSM Command ID: $COMMAND_ID"
echo ""
echo "Chaos injected. Watch for healing in 1-3 minutes."
echo ""
echo "Monitor with:"
echo "  curl \$(terraform -chdir=terraform output -raw api_endpoint)"

What does this script do?

The script accepts an instance ID and a scenario name as arguments. Based on the scenario, it builds a JSON array of shell commands to execute on the target instance.

• process-kill stops the httpd service, simulating a process crash.
• cpu-stress uses stress-ng to spike CPU to 100% for 120 seconds.
• disk-fill writes 900MB of zeros to /tmp, simulating a disk space issue.
• cascading combines process-kill and cpu-stress to trigger both alarms simultaneously, activating the composite alarm.

The actual injection happens through aws ssm send-command using the AWS-RunShellScript document. No SSH key required.

• Save chaos/inject.sh.
• Make the script executable by running:

chmod +x chaos/inject.sh

• Verify the script is runnable by executing it with no arguments:

./chaos/inject.sh

You should see the usage message listing all four scenarios.

Seeing permission denied?

• Make sure you ran chmod +x chaos/inject.sh before executing it.
• Confirm you're in the project root directory (the folder containing the chaos/ folder).

help me fix permission issues with my chaos injection script

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

#!/bin/bash
set -euo pipefail

INSTANCE_ID="${1:-}"
SCENARIO="${2:-}"

if [ -z "$INSTANCE_ID" ] || [ -z "$SCENARIO" ]; then
    echo "Usage: ./inject.sh <instance-id> <scenario>"
    echo ""
    echo "Scenarios:"
    echo "  process-kill  - Kill httpd process"
    echo "  cpu-stress    - Spike CPU to 100% for 120 seconds"
    echo "  disk-fill     - Fill /tmp with 900MB"
    echo "  cascading     - Kill httpd AND spike CPU (triggers composite alarm)"
    echo ""
    exit 1
fi

echo "=== CHAOS INJECTION ==="
echo "Instance: $INSTANCE_ID"
echo "Scenario: $SCENARIO"
echo "Time: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo ""

case "$SCENARIO" in
    process-kill)
        echo "Injecting: Killing httpd process..."
        COMMANDS='["sudo systemctl stop httpd","echo httpd stopped at $(date)"]'
        ;;
    cpu-stress)
        echo "Injecting: CPU stress for 120 seconds..."
        COMMANDS='["nohup stress-ng --cpu 2 --timeout 120 &>/dev/null &","echo CPU stress started at $(date)"]'
        ;;
    disk-fill)
        echo "Injecting: Filling /tmp with 900MB..."
        COMMANDS='["dd if=/dev/zero of=/tmp/chaos-fill bs=1M count=900","echo Disk filled at $(date)"]'
        ;;
    cascading)
        echo "Injecting: Cascading failure (process kill + CPU stress)..."
        COMMANDS='["sudo systemctl stop httpd","nohup stress-ng --cpu 2 --timeout 120 &>/dev/null &","echo Cascading failure injected at $(date)"]'
        ;;
    *)
        echo "Unknown scenario: $SCENARIO"
        exit 1
        ;;
esac

COMMAND_ID=$(aws ssm send-command \
    --instance-ids "$INSTANCE_ID" \
    --document-name "AWS-RunShellScript" \
    --parameters "{\"commands\":$COMMANDS}" \
    --comment "Chaos injection: $SCENARIO" \
    --region ap-south-1 \
    --output text \
    --query "Command.CommandId")

echo "SSM Command ID: $COMMAND_ID"
echo ""
echo "Chaos injected. Watch for healing in 1-3 minutes."
echo ""
echo "Monitor with:"
echo "  curl \$(terraform -chdir=terraform output -raw api_endpoint)"

Test graduated escalation

Your healing Lambda uses attempt history to decide how aggressively to fix a problem. A first-time failure gets a soft fix (service restart). A repeated failure within 10 minutes gets escalated to a hard fix (force-kill and restart). You'll prove this works by injecting the same failure twice in quick succession.

• Get your EC2 instance ID by running:

terraform -chdir=terraform output -raw instance_id

Copy the instance ID from the output. You'll use it in the next commands.

• Inject the first process-kill failure by running:

./chaos/inject.sh [[INSTANCE_ID="your EC2 instance ID"]] process-kill

You should see the SSM Command ID and the "Chaos injected" confirmation message.

What happens next?

The CloudWatch Alarm self-healing-infra-httpd-down detects that the httpd process count dropped to zero. It transitions to ALARM state, which triggers the EventBridge rule. EventBridge invokes your healing Lambda.

Since this is the first occurrence within the 10-minute escalation window, the Lambda performs a soft fix (systemctl restart httpd). This whole cycle takes 1-3 minutes.

• Wait 3 minutes for the healing cycle to complete. Then verify the first incident was healed with a soft fix by running:

curl "$(terraform -chdir=terraform output -raw api_endpoint)?last=1&type=process_down"

You should see JSON with attempt_count of 1 and action_taken of soft_fix.

• Now inject the same failure again within 10 minutes to trigger escalation:

./chaos/inject.sh [[INSTANCE_ID="your EC2 instance ID"]] process-kill

• Wait another 3 minutes for healing. Then check the API for the escalated incident:

curl "$(terraform -chdir=terraform output -raw api_endpoint)?last=2&type=process_down"

The most recent incident should show attempt_count of 2 and action_taken of hard_fix. This proves the Lambda detected the repeated failure and escalated its response.

Why hard_fix instead of just restarting again?

A soft fix that didn't hold suggests a deeper issue. Blindly repeating the same restart wastes time and leaves the service broken. Your Lambda's graduated logic force-kills all related processes before restarting, which clears corrupted state that a simple restart might not.

If this also fails, the third attempt escalates to a full instance reboot. After that, the system stops trying and pages a human. This prevents infinite restart loops.

Not seeing the escalated action?

• Make sure both injections happened within 10 minutes of each other. The escalation window in heal_handler.py is controlled by ESCALATION_WINDOW_MINUTES = 10.
• Check that the first healing completed successfully before injecting the second time. If the alarm is still in ALARM state, the second injection won't trigger a new event.
• Look at your Lambda's CloudWatch Logs for the detailed decision log showing the attempt count.

help me debug why my graduated escalation isn't working

Run the cascading failure scenario

A cascading failure is when multiple things break at once. Your composite alarm self-healing-infra-cascading-failure only fires when BOTH the httpd process is down AND CPU is critically high. This distinguishes a systemic meltdown from a single process crash.

• Wait at least 10 minutes after your last injection so the escalation window resets. Then inject the cascading scenario:

./chaos/inject.sh [[INSTANCE_ID="your EC2 instance ID"]] cascading

What triggers differently here?

The cascading scenario kills httpd AND spikes CPU simultaneously. This triggers both the self-healing-infra-httpd-down and self-healing-infra-cpu-high alarms. When both are in ALARM state, the composite alarm self-healing-infra-cascading-failure fires.

Your Lambda assigns severity CRITICAL to cascading failures instead of the WARNING or HIGH severity for single failures. The SNS notification subject will reflect this elevated severity.

• Wait 3 minutes for healing. Then check your email for an SNS notification with [CRITICAL] in the subject line.
• Verify the full incident list by querying the API:

curl "$(terraform -chdir=terraform output -raw api_endpoint)?last=5"

You should see multiple incidents in the JSON response. Look for the cascading_failure entry with severity CRITICAL alongside your earlier process_down entries.

What should the stats section show?

The API response includes a stats object with total_incidents, success_rate, and avg_mttr_seconds. With 3+ incidents and successful healing, your success rate should be above 80%. The average MTTR shows how quickly your platform resolves issues.

Composite alarm not firing?

• The CPU alarm requires 2 consecutive evaluation periods of 60 seconds each (2 minutes above 80%). The cascading scenario runs stress-ng for 120 seconds, so it should trigger. Give it an extra minute.
• Check that both individual alarms (self-healing-infra-httpd-down and self-healing-infra-cpu-high) are in ALARM state simultaneously. The composite alarm needs both at the same time.
• Verify the composite alarm exists in the CloudWatch console under All alarms.

help me troubleshoot why my composite alarm isn't triggering

Your chaos engineering suite proves that the platform doesn't just restart blindly. It remembers, escalates, and distinguishes single failures from cascading meltdowns. Next up, you'll build an observability dashboard with SLO tracking to visualize all of this data.

Build the Observability Dashboard and SLO Tracking

Your self-healing system now detects failures, applies graduated remediation, and logs every incident to DynamoDB. But operational systems need visibility.

In this step, you will review the CloudWatch Dashboard that Terraform already deployed, confirm that SLO availability metrics are flowing end-to-end, and populate the dashboard with real chaos data so every widget shows live signals.

In this step, get ready to:

• Understand the four CloudWatch Dashboard widgets tracking your infrastructure.
• Confirm SLO availability metric publishing in the healing Lambda.
• Trigger chaos scenarios and verify all widgets populate with real data.

Review the CloudWatch Dashboard configuration

Your monitoring.tf file already defines a CloudWatch Dashboard with four metric widgets. Each widget gives you a different angle on system health.

• Switch back to monitoring.tf in your editor and find the aws_cloudwatch_dashboard resource block.
• Review the dashboard JSON body containing the four widgets:

resource "aws_cloudwatch_dashboard" "healing" {
  dashboard_name = "${var.project_name}-dashboard"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x      = 0
        y      = 0
        width  = 12
        height = 6
        properties = {
          title   = "EC2 CPU Utilization"
          metrics = [["AWS/EC2", "CPUUtilization", "InstanceId", aws_instance.target.id]]
          period  = 60
          stat    = "Average"
          region  = var.aws_region
          view    = "timeSeries"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 0
        width  = 12
        height = 6
        properties = {
          title   = "httpd Process Count"
          metrics = [["SelfHealingInfra", "procstat_lookup_pid_count", "pattern", "httpd", "InstanceId", aws_instance.target.id]]
          period  = 60
          stat    = "Average"
          region  = var.aws_region
          view    = "timeSeries"
        }
      },
      {
        type   = "metric"
        x      = 0
        y      = 6
        width  = 12
        height = 6
        properties = {
          title   = "Healing Events (Lambda)"
          metrics = [
            ["AWS/Lambda", "Invocations", "FunctionName", "${var.project_name}-healer"],
            ["AWS/Lambda", "Errors", "FunctionName", "${var.project_name}-healer"]
          ]
          period = 300
          stat   = "Sum"
          region = var.aws_region
          view   = "bar"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 6
        width  = 12
        height = 6
        properties = {
          title  = "Service Availability (SLO)"
          metrics = [
            ["SelfHealingInfra", "ServiceAvailability", "Service", "httpd"]
          ]
          period = 300
          stat   = "Average"
          region = var.aws_region
          view   = "timeSeries"
          yAxis = {
            left = { min = 0, max = 1 }
          }
        }
      }
    ]
  })
}

What does each widget show?

• EC2 CPU Utilization tracks average CPU percentage every 60 seconds. Spikes here indicate a cpu-stress chaos injection or resource contention.
• httpd Process Count uses your custom SelfHealingInfra namespace metric. When this drops to 0, your process-kill alarm fires.
• Healing Events (Lambda) displays invocations and errors as a bar chart. Each bar represents a 5-minute window showing how often your healer Lambda ran.
• Service Availability (SLO) is the key SRE metric. The Y-axis is pinned between 0 and 1. A value of 1 means the service is healthy. A dip to 0 means an incident was detected.

Confirm SLO tracking in the healing Lambda

The SLO widget only works if your Lambda publishes the ServiceAvailability metric at the right moments. Your heal_handler.py already contains this logic.

• Switch back to lambda/heal_handler.py and find the publish_availability_metric function:

def publish_availability_metric(value):
    cloudwatch_client.put_metric_data(
        Namespace="SelfHealingInfra",
        MetricData=[{
            "MetricName": "ServiceAvailability",
            "Dimensions": [{"Name": "Service", "Value": "httpd"}],
            "Value": value,
            "Unit": "None"
        }]
    )

How does this drive the SLO widget?

Inside the lambda_handler function, two calls publish this metric at different stages of an incident:

• publish_availability_metric(0) fires immediately when an alarm event arrives. This marks the service as "down" in the SelfHealingInfra namespace.
• publish_availability_metric(1) fires after a successful remediation. This marks the service as "up" again.

The dashboard averages these values over 5-minute windows. A sustained 1.0 means 100% availability. Any dip shows exactly when and how long a service disruption lasted.

• Confirm these two calls exist in your lambda_handler function. Find the line publish_availability_metric(0) called before remediation, and publish_availability_metric(1) called after a successful result.

Why track availability as 0 and 1?

This binary encoding is the standard SRE pattern for SLO tracking. When you average a stream of 0s and 1s over time, the result IS your availability percentage. An average of 0.997 over a month means 99.7% availability.

CloudWatch can then apply math expressions or alarms on this metric to detect SLO breaches before they become critical.

Trigger chaos scenarios and verify the dashboard

Your dashboard is deployed and the metrics are wired. Now you need real data flowing through all four widgets to confirm the end-to-end loop works.

• Get your instance ID by running:

cd ~/Desktop/self-healing-infra/terraform
terraform output instance_id

• Inject a process-kill scenario by running:

cd ~/Desktop/self-healing-infra/chaos
./inject.sh [[INSTANCE_ID="your EC2 instance ID"]] process-kill

• Wait 2-3 minutes for the alarm to fire, the Lambda to heal, and metrics to publish.
• Inject a cpu-stress scenario by running:

./inject.sh [[INSTANCE_ID="your EC2 instance ID"]] cpu-stress

• Wait another 2-3 minutes for the CPU alarm to fire and the healer to respond.

Why two different scenarios?

The process-kill scenario populates the httpd Process Count widget (showing the drop to 0 and recovery) and the SLO widget (showing the 0-to-1 transition). The cpu-stress scenario populates the CPU Utilization widget with a visible spike. Together they ensure all four widgets receive data.

• Get your dashboard URL by running:

cd ~/Desktop/self-healing-infra/terraform
terraform output dashboard_url

• Open the URL in your browser. This takes you directly to the self-healing-infra-dashboard in the CloudWatch console.
• Set the time range to 1h using the predefined range at the top of the dashboard.

You should see all four widgets populated with data from your chaos injections.

Dashboard widgets showing no data?

• Custom metrics take 2-3 minutes to appear after the first publish. Wait a few minutes and refresh the dashboard.
• Check the time range. If it is set to a narrow window that ended before your chaos injection, expand it to 1h or 3h.
• Verify your Lambda executed successfully by checking CloudWatch Logs for the self-healing-infra-healer log group.

Help me debug why my CloudWatch Dashboard widgets are empty after running chaos scenarios.

• Verify the incident API also reflects the new incidents by running:

curl $(terraform output -raw api_endpoint)?last=5

You should see JSON with at least two new incident records showing the process-kill and cpu-stress scenarios with their remediation results and MTTR values.

• Take a screenshot of your CloudWatch Dashboard showing all four widgets populated. Save this for your GitHub README.

Best time range for your screenshot

Use the 1h time range for the cleanest screenshot. This shows the chaos injection spikes without too much empty space on either side. The CPU spike, process count drop, Lambda invocation bar, and SLO dip should all be clearly visible.

✔️ Awesome, I've got everything!

Great. Double check you have saved monitoring.tf and heal_handler.py and that your dashboard screenshot shows all four widgets with data.

ⓧ I'd like to double check the full code

Here is the full terraform/monitoring.tf for comparison:

resource "aws_dynamodb_table" "incidents" {
  name         = "${var.project_name}-incidents"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "incident_id"
  range_key    = "timestamp"

  attribute {
    name = "incident_id"
    type = "S"
  }

  attribute {
    name = "timestamp"
    type = "S"
  }

  attribute {
    name = "failure_type"
    type = "S"
  }

  global_secondary_index {
    name            = "failure-type-index"
    hash_key        = "failure_type"
    range_key       = "timestamp"
    projection_type = "ALL"
  }

  tags = {
    Project = var.project_name
  }
}

resource "aws_sns_topic" "healing_notifications" {
  name = "${var.project_name}-notifications"

  tags = {
    Project = var.project_name
  }
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.healing_notifications.arn
  protocol  = "email"
  endpoint  = var.notification_email
}

resource "aws_cloudwatch_dashboard" "healing" {
  dashboard_name = "${var.project_name}-dashboard"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x      = 0
        y      = 0
        width  = 12
        height = 6
        properties = {
          title   = "EC2 CPU Utilization"
          metrics = [["AWS/EC2", "CPUUtilization", "InstanceId", aws_instance.target.id]]
          period  = 60
          stat    = "Average"
          region  = var.aws_region
          view    = "timeSeries"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 0
        width  = 12
        height = 6
        properties = {
          title   = "httpd Process Count"
          metrics = [["SelfHealingInfra", "procstat_lookup_pid_count", "pattern", "httpd", "InstanceId", aws_instance.target.id]]
          period  = 60
          stat    = "Average"
          region  = var.aws_region
          view    = "timeSeries"
        }
      },
      {
        type   = "metric"
        x      = 0
        y      = 6
        width  = 12
        height = 6
        properties = {
          title   = "Healing Events (Lambda)"
          metrics = [
            ["AWS/Lambda", "Invocations", "FunctionName", "${var.project_name}-healer"],
            ["AWS/Lambda", "Errors", "FunctionName", "${var.project_name}-healer"]
          ]
          period = 300
          stat   = "Sum"
          region = var.aws_region
          view   = "bar"
        }
      },
      {
        type   = "metric"
        x      = 12
        y      = 6
        width  = 12
        height = 6
        properties = {
          title  = "Service Availability (SLO)"
          metrics = [
            ["SelfHealingInfra", "ServiceAvailability", "Service", "httpd"]
          ]
          period = 300
          stat   = "Average"
          region = var.aws_region
          view   = "timeSeries"
          yAxis = {
            left = { min = 0, max = 1 }
          }
        }
      }
    ]
  })
}

Your observability stack is now complete. You have a single dashboard that shows CPU health, process monitoring, healing activity, and SLO availability in one view. Next up, you will wire this entire platform into a CI/CD pipeline so deployments happen automatically on every push.

Set Up the CI/CD Pipeline

Your self-healing platform works. You can inject chaos, watch it heal, and query the API for proof.

But right now, every infrastructure change requires you to run terraform apply manually from your laptop. A production team would never do this. They use a CI/CD pipeline to validate, plan, and apply changes automatically on every code push.

In this step, get ready to:

• Write a GitHub Actions workflow with unit tests and Terraform deployment.
• Push your code to GitHub and configure secrets.
• Trigger the pipeline and verify end-to-end deployment.

Write the CI/CD workflow file

GitHub Actions runs your pipeline as a YAML workflow file inside your repository. Each push to main triggers the workflow, and pull requests get a plan preview without applying changes.

Your workflow has two jobs: a test job that validates your Lambda logic with unit tests, and a Terraform job that lints, validates, and deploys your infrastructure.

• Create the test job section of your workflow by adding the following code to .github/workflows/deploy.yml:

name: Deploy Self-Healing Infrastructure

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  AWS_REGION: ap-south-1
  TF_WORKING_DIR: terraform

jobs:
  test:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install boto3 moto pytest

      - name: Run tests
        run: pytest tests/ -v

What does this code do?

• The on: block triggers the workflow on pushes to main and on pull requests targeting main.
• The env: block sets shared variables so both jobs know the AWS region and Terraform directory.
• The test job checks out the code, installs Python 3.12, installs boto3, moto, and pytest, then runs your unit tests.

• Add the Terraform deployment job below the test job in the same file:

  terraform:
    name: Terraform Plan and Apply
    runs-on: ubuntu-latest
    needs: test

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Terraform Init
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform init

      - name: Terraform Format Check
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform fmt -check

      - name: Terraform Validate
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform validate

      - name: Terraform Plan
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform plan -var="notification_email=${{ secrets.NOTIFICATION_EMAIL }}" -no-color

      - name: Terraform Apply
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform apply -var="notification_email=${{ secrets.NOTIFICATION_EMAIL }}" -auto-approve

What does this code do?

The terraform job only runs after test passes (needs: test). It configures AWS credentials from repository secrets, installs Terraform, then runs the standard validation steps.

The if: condition on the Apply step means it only deploys when code is pushed directly to main. Pull requests get a plan output for review but never apply changes.

• Save .github/workflows/deploy.yml.

YAML indentation errors?

YAML is indent-sensitive. The terraform: key must be indented at the same level as test: (two spaces under jobs:). If your editor auto-indents differently, use the full file below to compare.

Help me fix YAML indentation in my GitHub Actions workflow

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before continuing.

ⓧ I'd like to double check the full code

name: Deploy Self-Healing Infrastructure

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  AWS_REGION: ap-south-1
  TF_WORKING_DIR: terraform

jobs:
  test:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install boto3 moto pytest

      - name: Run tests
        run: pytest tests/ -v

  terraform:
    name: Terraform Plan and Apply
    runs-on: ubuntu-latest
    needs: test

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Terraform Init
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform init

      - name: Terraform Format Check
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform fmt -check

      - name: Terraform Validate
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform validate

      - name: Terraform Plan
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform plan -var="notification_email=${{ secrets.NOTIFICATION_EMAIL }}" -no-color

      - name: Terraform Apply
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        working-directory: ${{ env.TF_WORKING_DIR }}
        run: terraform apply -var="notification_email=${{ secrets.NOTIFICATION_EMAIL }}" -auto-approve

Add a unit test for the healing logic

The workflow runs pytest tests/ -v, so you need at least one test file. You will test the pure logic functions in your Lambda (determine_failure_type and the graduated escalation boundaries) without making real AWS calls.

• Create the tests/ directory by running:

mkdir tests

• Create tests/test_heal_handler.py with the following test code:

import os
import sys

# Set required environment variables before importing the handler
os.environ["INSTANCE_ID"] = "i-test123"
os.environ["TABLE_NAME"] = "test-incidents"
os.environ["SNS_TOPIC_ARN"] = "arn:aws:sns:ap-south-1:123456789:test"
os.environ["PROJECT_NAME"] = "test-project"

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lambda"))

from heal_handler import determine_failure_type, determine_severity


def test_determine_failure_type_httpd():
    assert determine_failure_type("self-healing-infra-httpd-down") == "process_down"


def test_determine_failure_type_cpu():
    assert determine_failure_type("self-healing-infra-cpu-high") == "cpu_spike"


def test_determine_failure_type_cascading():
    assert determine_failure_type("self-healing-infra-cascading-failure") == "cascading_failure"


def test_determine_failure_type_unknown():
    assert determine_failure_type("random-alarm") == "unknown"


def test_determine_severity_cascading():
    assert determine_severity("cascading-failure-alarm") == "CRITICAL"


def test_determine_severity_httpd():
    assert determine_severity("httpd-down") == "HIGH"


def test_determine_severity_cpu():
    assert determine_severity("cpu-high") == "WARNING"

What does this test file do?

The environment variables are set before importing heal_handler because the module reads them at import time. Without them, the import would crash.

The tests verify the pure logic functions (determine_failure_type and determine_severity) that map alarm names to failure categories and severity levels. These functions never call AWS, so they run instantly without mocks.

• Verify the tests pass locally by running:

pip install boto3 moto pytest
pytest tests/ -v

• You should see all 7 tests passing with green PASSED markers.

Tests failing on import?

If you see ModuleNotFoundError: No module named 'boto3', make sure you ran pip install boto3 in the same Python environment you are running pytest from.

Help me debug why my heal_handler tests are failing on import

Push to GitHub and configure secrets

Your code is ready. Now you need to get it onto GitHub so the Actions workflow can run.

• Stage and commit all your project files by running:

git add .
git commit -m "Add CI/CD pipeline and unit tests"

• Create a new GitHub repository and push your code by running:

gh repo create self-healing-infra --public --source=. --remote=origin --push

What does this command do?

The gh repo create command creates a new repository on GitHub, sets origin as the remote name, and pushes all local commits in a single step. If prompted to authenticate, follow the on-screen instructions.

gh command not found?

If you do not have the GitHub CLI installed, create the repository manually at github.com/new (do not initialize with a README), then run these commands instead:

git remote add origin https://github.com/YOUR-USERNAME/self-healing-infra.git
git push -u origin main

Help me push my local git repository to GitHub

Now add the three secrets your workflow needs. These let GitHub Actions authenticate with AWS and pass the notification email to Terraform.

• Open your repository on GitHub in your browser.
• Click Settings in the top navigation bar.
• In the left sidebar under Security, select Secrets and variables, then click Actions.
• Click New repository secret.
• Add a secret with name AWS_ACCESS_KEY_ID and paste your IAM access key ID as the value. Click Add secret.
• Repeat for AWS_SECRET_ACCESS_KEY with your IAM secret access key.
• Repeat for NOTIFICATION_EMAIL with the email address you used for SNS notifications.

• Verify you see all three secrets listed on the Actions secrets page.

Why use repository secrets?

Secrets are encrypted and never exposed in logs. Your workflow references them with ${{ secrets.AWS_ACCESS_KEY_ID }} syntax, and GitHub masks any accidental echo of their values.

Don't see the Settings tab?

You must be the repository owner to access Settings. If you created the repo with gh repo create under your account, you already have access. If the tab is hidden, look for the three-dot dropdown menu next to the tab bar.

Help me find the Settings tab in my GitHub repository

Trigger the pipeline and verify deployment

Your workflow triggers on pushes to main. Since you already pushed, a pipeline run should already be in progress.

• In your GitHub repository, click the Actions tab.
• Click the latest workflow run (named after your commit message).
• Watch both jobs: Unit Tests should pass first, then Terraform Plan and Apply runs.
• Expand the Terraform Apply step to confirm it completed without errors.

What should I see?

The Unit Tests job shows 7 tests passing. The Terraform Plan and Apply job shows terraform init, terraform fmt -check, terraform validate, terraform plan, and terraform apply all completing with green checkmarks. The apply step reports no changes (or minor drift) since your infrastructure is already deployed.

Pipeline failing on Terraform steps?

• If terraform fmt -check fails, run terraform fmt locally in your terraform/ directory, commit the formatting changes, and push again.
• If terraform init fails with an access denied error, double-check that your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY secrets are correct and the IAM user has AdministratorAccess.
• If the test job fails, check that your tests/test_heal_handler.py file sets all four environment variables before importing the handler.

Help me debug my GitHub Actions pipeline failure

Your infrastructure is now fully automated. Every push to main runs tests, validates Terraform, and deploys changes without you touching the AWS console. Next up, you will wrap everything together with a demo-ready chaos validation.

Secret mission

Full Game Day Automation with SLO Report

Your self-healing platform works for individual failures. But can you prove it handles everything in sequence under pressure? Build a full Game Day automation that runs all chaos scenarios back-to-back, validates healing via the API, measures MTTR per scenario, and generates a markdown report with SLO availability stats.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The EC2 instance is the main cost driver outside of free tier. CloudWatch custom metrics add a small ongoing charge.

Cost warning

If your AWS account is outside the EC2 free tier, the t3.micro instance costs approximately $0.0104/hour (around $7.50/month or 63/day). CloudWatch custom metrics cost approximately $0.30/metric/month for the 3 metrics in the SelfHealingInfra namespace, totalling around $0.90/month.

If you are not actively using this project for interview demos, stop or terminate the EC2 instance to avoid charges.

Resources you used:

• EC2 t3.micro instance in ap-south-1 (self-healing-infra-target).
• 2 Lambda functions (healer and API).
• CloudWatch Alarms, Composite Alarm, and EventBridge rule.
• API Gateway REST API (incidents endpoint).
• DynamoDB incidents table (self-healing-infra-incidents).
• DynamoDB state-lock table (terraform-locks), created manually.
• SNS topic with email subscription.
• S3 bucket for Terraform remote state (self-healing-infra-tfstate-<account-id>), created manually.
• CloudWatch Dashboard (self-healing-infra-dashboard).
• GitHub repository with CI/CD pipeline.
• Local project files on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively demoing this project in interviews or building on it further.

• Your EC2 instance stays running and the self-healing loop remains active.
• The API endpoint stays live. You can run curl against it at any time to show incident history.
• Estimated monthly cost: approximately $8-10/month if outside free tier (EC2 + CloudWatch custom metrics).
• AWS CLI, Terraform, Python 3, and Git remain installed on your machine.

Pause - I'll come back to this later

Stop the EC2 instance to eliminate the per-hour compute charge. The API endpoint still works and returns historical incident data from DynamoDB.

• Stop the EC2 instance by running this command (replace the instance ID with yours):
• You can find your instance ID in terraform/outputs.tf or by running terraform -chdir=terraform output instance_id.

aws ec2 stop-instances --instance-ids <your-instance-id> --region ap-south-1

• When you are ready to demo again, restart the instance:

aws ec2 start-instances --instance-ids <your-instance-id> --region ap-south-1

• After restarting, wait 2-3 minutes for the CloudWatch Agent and httpd to come back online before running chaos tests.
• CloudWatch custom metrics still incur a small charge (~$0.90/month) while paused.

Delete - I don't want to use this again

Remove all project resources from AWS and clean up local files. Your code stays safe in GitHub.

• Navigate to your terraform directory and destroy all Terraform-managed resources:

cd ~/Desktop/self-healing-infra/terraform
terraform destroy -var="notification_email=your-email@example.com" -auto-approve

• This removes the EC2 instance, Lambda functions, CloudWatch Alarms, EventBridge rule, API Gateway, DynamoDB incidents table, SNS topic, and CloudWatch Dashboard.
• Delete the S3 state bucket (created manually, not managed by Terraform):

aws s3 rb s3://self-healing-infra-tfstate-<your-account-id> --force --region ap-south-1

• Delete the DynamoDB state-lock table (also created manually):

aws dynamodb delete-table --table-name terraform-locks --region ap-south-1

• Delete the local project files:

cd ~/Desktop
rm -rf self-healing-infra

• Your GitHub repository remains intact with all your code and pipeline configuration. You can re-deploy at any time by cloning and running terraform apply again.

Nice Work!

Nice Work!

Nice work! You built a production-grade self-healing infrastructure platform on AWS that demonstrates senior-level operational thinking.

This isn't a toy project. The live API endpoint, graduated remediation logic, and chaos validation suite prove you understand how production systems stay reliable at scale.

You've learned how to:

• Build an event-driven auto-remediation system with graduated escalation (restart, reboot, escalate to human) using Lambda, EventBridge, and SSM Run Command.
• Create a live incident API with API Gateway that's demo-able with a single curl command during interviews.
• Deploy the entire platform as Infrastructure as Code with Terraform remote state, composite alarms for correlated failure detection, and a GitHub Actions CI/CD pipeline with automated testing.
• Build a Game Day automation that runs all chaos scenarios in sequence, measures MTTR per scenario, calculates SLO availability, and generates a markdown incident report.

Ready to quiz yourself?