Build an SRE Lab on Amazon ECS

Build an ECS service and measure ALB availability with an SLO and error budget.

Introduction

30 Second Summary

A service can recover after a failure while its users still experience an outage. Reliability becomes useful when you measure that gap from the user's side.

In this project, you will deploy a public web service on Amazon ECS behind an Application Load Balancer. You will test a measurable availability SLO with controlled failures.

What You'll Build

You'll watch your SRE Reliability Lab move from healthy traffic to a controlled outage while CloudWatch Application Signals records the reliability impact.

By the end of this project, you'll have:

  • A public reliability lab you can open in a browser to confirm the containerized service is healthy.
  • A self-healing demonstration where stopping one task shows Amazon ECS restore the desired count.
  • A measured failure drill that shows how HTTP responses with status code 503 spend the SLO's error budget.
  • Secret Mission: Add task-level redundancy so one task keeps serving HTTP 200 responses while Amazon ECS replaces the other.

Are there any prerequisites?

You'll need a Linux computer with AWS CLI, Visual Studio Code, curl, plus an AWS account.

Your AWS identity needs permission to create the services listed in the project prerequisites. Step 1 verifies your workstation before any billable resources are created.

Before We Start

Before any resources exist, this is your moment to commit to measuring what users experience through HTTP responses and load balancer telemetry. Restoring the desired task count in Amazon ECS proves recovery, while uninterrupted availability requires evidence from the service itself.

Verify Your Linux Workstation

A useful SRE failure drill starts with a known local environment. An unverified command-line identity could make later results ambiguous.

This step confirms your AWS CLI identity in us-east-1. You will also prepare the files that Bash and Visual Studio Code use throughout the lab.

In this step, get ready to:
  • Verify AWS CLI v2, Visual Studio Code, and curl on your Linux workstation.
  • Create the complete ecs-sre-lab workspace in Visual Studio Code.
  • Make the scripts executable and confirm your AWS identity in us-east-1.
Check your command-line tools

The lab depends on curl for its HTTP probes. Any supported curl build that recognizes the script options can perform this job.

  • Check curl by running this command:
curl --version

What does this check prove?

A version banner confirms that your terminal can find curl. The first line identifies the installed build.

✔️ I see a curl version

Curl is ready to probe the load balancer later in the lab.

ⓧ Command not found

Install the curl package for your Linux distribution. Choose the tab that matches your system.

Ubuntu

  • Install curl on Ubuntu by running this command:
sudo apt install curl
What does this command do?

The package manager installs curl from Ubuntu's configured repositories. It may ask for your Linux password before continuing.

  • Confirm the installation by running this command:
curl --version
What should you see?

The terminal should now print the installed curl version.

Fedora Or RHEL

  • Install curl on Fedora or a RHEL-family system by running this command:
sudo dnf install curl
What does this command do?

The package manager installs curl from the configured repositories. It may ask you to approve the package before continuing.

  • Confirm the installation by running this command:
curl --version
What should you see?

The terminal should now print the installed curl version.

Still missing curl?

Confirm that your package installation finished without a repository or permission error.

Use this prompt if curl still cannot run. Help me install curl on my Linux distribution.

The project scripts use AWS CLI v2. Its version banner begins with aws-cli/2.

  • Check your AWS CLI major version by running this command:
aws --version

How do you read the result?

The text before the first space contains the AWS CLI version. This project needs output beginning with aws-cli/2.

✔️ Output begins with aws-cli/2

Your AWS CLI major version is ready for the lab scripts.

ⓧ Output begins with aws-cli/1

Your terminal is finding AWS CLI v1. Install AWS CLI v2 with the official Linux installer.

  • Install AWS CLI v2 for your current user by running this command:
curl -fsSL https://awscli.amazonaws.com/v2/install.sh | bash

What does this installer do?

The official installer downloads and verifies AWS CLI v2. It installs the command for your current Linux user.

  • Start a fresh terminal session after the installer finishes.
  • Confirm the new major version by running this command:
aws --version

What should you see?

The new version banner should begin with aws-cli/2.

ⓧ Command not found

Install AWS CLI v2 with the official Linux installer. Curl is available from the check you completed above.

  • Install AWS CLI v2 for your current user by running this command:
curl -fsSL https://awscli.amazonaws.com/v2/install.sh | bash

What does this installer do?

The official installer downloads and verifies AWS CLI v2. It creates the command-line installation for your current Linux user.

  • Start a fresh terminal session after the installer finishes.
  • Confirm the installation by running this command:
aws --version

What should you see?

The version banner should begin with aws-cli/2.

Still seeing the wrong AWS CLI version?

A previous installation may appear earlier in your shell path. Start a new terminal before checking again.

Use this prompt if the old version remains. Help me find why Linux still runs the wrong AWS CLI version.

  • Check the Visual Studio Code command by running this command:
code --version

What does this check prove?

A version number confirms that the Visual Studio Code command is available from your Linux terminal. You will use it to open the lab folder as a workspace.

Is the code command missing?

Install Visual Studio Code with the official Linux setup guide. Run the version check again after the installation finishes.

Use this prompt if Visual Studio Code is installed but its command is unavailable. Help me make the code command available in my Linux terminal.

Create the lab workspace

The workspace holds the infrastructure template plus the scripts for deployment, traffic, failure drills, reporting, and cleanup. Keeping them together lets every script resolve template.yaml from the same folder.

The last command below opens Visual Studio Code. You may see a Workspace Trust prompt after the window opens.

  • Create the ecs-sre-lab folder on your Desktop by running these commands:
cd ~/Desktop
mkdir ecs-sre-lab
cd ecs-sre-lab
code .

What do these commands do?

  • The first command moves your terminal to the Desktop.
  • The next command creates the ecs-sre-lab folder.
  • The final two commands enter the folder and open it as a Visual Studio Code workspace.
  • Select Yes, I trust the authors if the Workspace Trust prompt appears.
  • Confirm that ecs-sre-lab appears at the top of the Explorer sidebar.
  • Open the integrated terminal from the View menu by selecting Terminal.
  • Create the eight project files by running this command in the integrated terminal:
touch template.yaml deploy.sh traffic.sh replace-task.sh slo-report.sh outage-drill.sh cleanup.sh sample-sla.md

What does this command do?

The command creates each empty file inside ecs-sre-lab. Existing files with these names remain in place.

  • Confirm that all eight files exist by running this command:
ls

What should you see?

The output should list template.yaml, six executable script files, and sample-sla.md.

Use the full-code tab below to fill every file. Save each file before moving to the next one.

✔️ I added the complete project files

Great. Confirm that all eight files are saved in the ecs-sre-lab workspace.

ⓧ I'd like to double check the full code

  • Open template.yaml from the Explorer sidebar.
  • Replace its contents with this complete template:
AWSTemplateFormatVersion: '2010-09-09'
Description: EC2-backed ECS reliability lab with an ALB and optional Application Signals SLO.

Parameters:
  ECSAMI:
    Type: AWS::EC2::Image::Id
    Description: Recommended Amazon Linux 2023 ECS-optimized AMI ID.
  CreateSLO:
    Type: String
    Default: 'false'
    AllowedValues:
      - 'true'
      - 'false'

Conditions:
  CreateSLOCondition: !Equals [!Ref CreateSLO, 'true']

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.20.0.0/16
      EnableDnsHostnames: true
      EnableDnsSupport: true
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}-vpc'

  InternetGateway:
    Type: AWS::EC2::InternetGateway

  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC

  PublicSubnetA:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [0, !GetAZs '']
      CidrBlock: 10.20.1.0/24
      MapPublicIpOnLaunch: true

  PublicSubnetB:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [1, !GetAZs '']
      CidrBlock: 10.20.2.0/24
      MapPublicIpOnLaunch: true

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC

  DefaultPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PublicSubnetARouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnetA

  PublicSubnetBRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnetB

  ALBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow public HTTP traffic to the lab ALB.
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0

  ECSHostSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow ALB traffic to dynamically mapped ECS host ports.
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 32768
          ToPort: 65535
          SourceSecurityGroupId: !Ref ALBSecurityGroup
      SecurityGroupEgress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0

  ECSCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Sub '${AWS::StackName}-cluster'

  ECSInstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role

  ECSInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Roles:
        - !Ref ECSInstanceRole

  ECSLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      LaunchTemplateData:
        ImageId: !Ref ECSAMI
        InstanceType: t3.small
        IamInstanceProfile:
          Name: !Ref ECSInstanceProfile
        SecurityGroupIds:
          - !Ref ECSHostSecurityGroup
        MetadataOptions:
          HttpEndpoint: enabled
          HttpTokens: required
        UserData:
          Fn::Base64: !Sub |
            #!/bin/bash
            echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config

  ECSAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      VPCZoneIdentifier:
        - !Ref PublicSubnetA
        - !Ref PublicSubnetB
      LaunchTemplate:
        LaunchTemplateId: !Ref ECSLaunchTemplate
        Version: !GetAtt ECSLaunchTemplate.LatestVersionNumber
      MinSize: '1'
      MaxSize: '1'
      DesiredCapacity: '1'

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/ecs/${AWS::StackName}'
      RetentionInDays: 7

  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub '${AWS::StackName}-task'
      NetworkMode: bridge
      RequiresCompatibilities:
        - EC2
      ContainerDefinitions:
        - Name: web
          Image: httpd:2.4.69-alpine
          Cpu: 128
          MemoryReservation: 128
          Essential: true
          EntryPoint:
            - sh
            - -c
          Command:
            - |
              echo '<!doctype html><html><head><title>SRE Reliability Lab</title></head><body><h1>SRE Reliability Lab</h1><p>Status: healthy</p><p>This response came through an Application Load Balancer to an ECS task.</p></body></html>' > /usr/local/apache2/htdocs/index.html && exec httpd-foreground
          PortMappings:
            - ContainerPort: 80
              HostPort: 0
              Protocol: tcp
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref LogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: ecs

  ApplicationLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Scheme: internet-facing
      Type: application
      Subnets:
        - !Ref PublicSubnetA
        - !Ref PublicSubnetB
      SecurityGroups:
        - !Ref ALBSecurityGroup

  ALBTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      VpcId: !Ref VPC
      Port: 80
      Protocol: HTTP
      TargetType: instance
      HealthCheckProtocol: HTTP
      HealthCheckPath: /
      HealthCheckIntervalSeconds: 10
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2

  ALBListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref ApplicationLoadBalancer
      Port: 80
      Protocol: HTTP
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref ALBTargetGroup

  ECSService:
    Type: AWS::ECS::Service
    DependsOn:
      - ALBListener
      - ECSAutoScalingGroup
    Properties:
      ServiceName: !Sub '${AWS::StackName}-service'
      Cluster: !Ref ECSCluster
      TaskDefinition: !Ref TaskDefinition
      DesiredCount: 1
      LaunchType: EC2
      SchedulingStrategy: REPLICA
      HealthCheckGracePeriodSeconds: 30
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 0
        DeploymentCircuitBreaker:
          Enable: true
          Rollback: true
      LoadBalancers:
        - ContainerName: web
          ContainerPort: 80
          TargetGroupArn: !Ref ALBTargetGroup

  AvailabilitySLO:
    Type: AWS::ApplicationSignals::ServiceLevelObjective
    Condition: CreateSLOCondition
    DependsOn: ECSService
    Properties:
      Name: !Sub '${AWS::StackName}-availability'
      Description: Availability of ALB routing based on successful target selection and ALB-generated 5XX responses.
      Sli:
        SliMetric:
          MetricDataQueries:
            - Id: requests
              MetricStat:
                Metric:
                  Namespace: AWS/ApplicationELB
                  MetricName: RequestCount
                  Dimensions:
                    - Name: LoadBalancer
                      Value: !GetAtt ApplicationLoadBalancer.LoadBalancerFullName
                Period: 60
                Stat: Sum
              ReturnData: false
            - Id: errors
              MetricStat:
                Metric:
                  Namespace: AWS/ApplicationELB
                  MetricName: HTTPCode_ELB_5XX_Count
                  Dimensions:
                    - Name: LoadBalancer
                      Value: !GetAtt ApplicationLoadBalancer.LoadBalancerFullName
                Period: 60
                Stat: Sum
              ReturnData: false
            - Id: availability
              Expression: '100 * requests / (requests + FILL(errors, 0))'
              ReturnData: true
          MetricThreshold: 99.0
          ComparisonOperator: GreaterThanOrEqualTo
      Goal:
        Interval:
          RollingInterval:
            DurationUnit: DAY
            Duration: 1
        AttainmentGoal: 99.9
        WarningThreshold: 50.0

Outputs:
  LoadBalancerURL:
    Description: Public URL for the lab service.
    Value: !Sub 'http://${ApplicationLoadBalancer.DNSName}'
  ClusterName:
    Value: !Ref ECSCluster
  ServiceName:
    Value: !GetAtt ECSService.Name
  AutoScalingGroupName:
    Value: !Ref ECSAutoScalingGroup
  LoadBalancerFullName:
    Value: !GetAtt ApplicationLoadBalancer.LoadBalancerFullName
  SLOArn:
    Value: !If [CreateSLOCondition, !Ref AvailabilitySLO, 'Not created']

What does this template define?

The template describes the complete EC2-backed ECS environment. Its CreateSLO condition allows the deployment script to add the availability SLO later.

  • Save template.yaml.

The saved file should begin with AWSTemplateFormatVersion and end with the SLOArn output.

  • Open deploy.sh from the Explorer sidebar.
  • Replace its contents with this deployment script:
#!/usr/bin/env bash
set -euo pipefail

REGION="${AWS_REGION:-us-east-1}"
STACK_NAME="${STACK_NAME:-ecs-sre-lab}"
CREATE_SLO="${1:-false}"

if [[ "$CREATE_SLO" != "true" && "$CREATE_SLO" != "false" ]]; then
  echo "Usage: ./deploy.sh [true|false]"
  exit 1
fi

ECS_AMI=$(aws ssm get-parameters \
  --names /aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id \
  --region "$REGION" \
  --query "Parameters[0].Value" \
  --output text)

aws cloudformation validate-template \
  --template-body file://template.yaml \
  --region "$REGION" >/dev/null

aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name "$STACK_NAME" \
  --parameter-overrides ECSAMI="$ECS_AMI" CreateSLO="$CREATE_SLO" \
  --capabilities CAPABILITY_IAM \
  --region "$REGION"

aws cloudformation describe-stacks \
  --stack-name "$STACK_NAME" \
  --region "$REGION" \
  --query 'Stacks[0].Outputs' \
  --output table

What does this script prepare?

The script retrieves the recommended ECS-optimized AMI. It validates and deploys template.yaml with the requested SLO state.

  • Save deploy.sh.

The saved script should begin with the Bash shebang and end with the CloudFormation output query.

  • Open traffic.sh from the Explorer sidebar.
  • Replace its contents with this traffic script:
#!/usr/bin/env bash
set -euo pipefail

REGION="${AWS_REGION:-us-east-1}"
STACK_NAME="${STACK_NAME:-ecs-sre-lab}"

ALB_URL=$(aws cloudformation describe-stacks \
  --stack-name "$STACK_NAME" \
  --region "$REGION" \
  --query 'Stacks[0].Outputs[?OutputKey==`LoadBalancerURL`].OutputValue' \
  --output text)

echo "Sending one request per second to $ALB_URL"
echo "Press Ctrl+C to stop."

while true; do
  HTTP_CODE=$(curl --max-time 5 --silent --output /dev/null --write-out '%{http_code}' "$ALB_URL" || true)
  if [[ -z "$HTTP_CODE" ]]; then
    HTTP_CODE=000
  fi
  printf '%s HTTP %s\n' "$(date -u +%H:%M:%S)" "$HTTP_CODE"
  sleep 1
done

What does this script prepare?

The script reads the load balancer URL from the stack outputs. It sends one request each second and prints the resulting HTTP status code.

  • Save traffic.sh.

The saved script should contain the continuous while true loop near the bottom.

  • Open replace-task.sh from the Explorer sidebar.
  • Replace its contents with this replacement drill:
#!/usr/bin/env bash
set -euo pipefail

REGION="${AWS_REGION:-us-east-1}"
STACK_NAME="${STACK_NAME:-ecs-sre-lab}"

output_value() {
  aws cloudformation describe-stacks \
    --stack-name "$STACK_NAME" \
    --region "$REGION" \
    --query "Stacks[0].Outputs[?OutputKey==\`$1\`].OutputValue" \
    --output text
}

CLUSTER=$(output_value ClusterName)
SERVICE=$(output_value ServiceName)
OLD_TASK=$(aws ecs list-tasks \
  --cluster "$CLUSTER" \
  --service-name "$SERVICE" \
  --region "$REGION" \
  --query 'taskArns[0]' \
  --output text)

echo "Stopping task: $OLD_TASK"
aws ecs stop-task \
  --cluster "$CLUSTER" \
  --task "$OLD_TASK" \
  --reason "SRE reliability lab task interruption" \
  --region "$REGION" >/dev/null

aws ecs wait services-stable \
  --cluster "$CLUSTER" \
  --services "$SERVICE" \
  --region "$REGION"

NEW_TASK=$(aws ecs list-tasks \
  --cluster "$CLUSTER" \
  --service-name "$SERVICE" \
  --region "$REGION" \
  --query 'taskArns[0]' \
  --output text)

echo "Old task: $OLD_TASK"
echo "New task: $NEW_TASK"

aws ecs describe-services \
  --cluster "$CLUSTER" \
  --services "$SERVICE" \
  --region "$REGION" \
  --query 'services[0].{desired:desiredCount,running:runningCount,recentEvents:events[0:3].message}' \
  --output json

What does this script prepare?

The script stops the current service task and waits for ECS stability. It then compares task ARNs and reports the restored service counts.

  • Save replace-task.sh.

The saved file should include the failure-drill reason SRE reliability lab task interruption.

  • Open slo-report.sh from the Explorer sidebar.
  • Replace its contents with this SLO reporting script:
#!/usr/bin/env bash
set -euo pipefail

REGION="${AWS_REGION:-us-east-1}"
STACK_NAME="${STACK_NAME:-ecs-sre-lab}"

SLO_ARN=$(aws cloudformation describe-stacks \
  --stack-name "$STACK_NAME" \
  --region "$REGION" \
  --query 'Stacks[0].Outputs[?OutputKey==`SLOArn`].OutputValue' \
  --output text)

if [[ "$SLO_ARN" == "Not created" || -z "$SLO_ARN" ]]; then
  echo "The SLO is not created. Run ./deploy.sh true first."
  exit 1
fi

aws application-signals batch-get-service-level-objective-budget-report \
  --timestamp "$(date +%s)" \
  --slo-ids "$SLO_ARN" \
  --region "$REGION" \
  --query 'Reports[0].{Name:Name,BudgetStatus:BudgetStatus,Attainment:Attainment,TotalBudgetSeconds:TotalBudgetSeconds,BudgetSecondsRemaining:BudgetSecondsRemaining}' \
  --output table

What does this script prepare?

The script reads the SLO ARN from the stack outputs. It requests attainment plus the budget status and remaining budget seconds.

  • Save slo-report.sh.

The saved file should end with the budget report query and table output setting.

  • Open outage-drill.sh from the Explorer sidebar.
  • Replace its contents with this controlled outage script:
#!/usr/bin/env bash
set -euo pipefail

REGION="${AWS_REGION:-us-east-1}"
STACK_NAME="${STACK_NAME:-ecs-sre-lab}"

output_value() {
  aws cloudformation describe-stacks \
    --stack-name "$STACK_NAME" \
    --region "$REGION" \
    --query "Stacks[0].Outputs[?OutputKey==\`$1\`].OutputValue" \
    --output text
}

CLUSTER=$(output_value ClusterName)
SERVICE=$(output_value ServiceName)

restore_service() {
  aws ecs update-service \
    --cluster "$CLUSTER" \
    --service "$SERVICE" \
    --desired-count 1 \
    --region "$REGION" >/dev/null || true
}
trap restore_service EXIT

echo "Setting desired count to zero for a controlled outage."
aws ecs update-service \
  --cluster "$CLUSTER" \
  --service "$SERVICE" \
  --desired-count 0 \
  --region "$REGION" >/dev/null

aws ecs wait services-stable \
  --cluster "$CLUSTER" \
  --services "$SERVICE" \
  --region "$REGION"

sleep 75

echo "Restoring desired count to one."
restore_service
aws ecs wait services-stable \
  --cluster "$CLUSTER" \
  --services "$SERVICE" \
  --region "$REGION"
trap - EXIT

echo "Service restored. Leave traffic.sh running while metrics are published."

What does this script prepare?

The script creates a controlled zero-capacity interval lasting 75 seconds. Its exit trap restores one desired task if the drill is interrupted.

  • Save outage-drill.sh.

The saved file should include trap restore_service EXIT before the outage begins.

  • Open cleanup.sh from the Explorer sidebar.
  • Replace its contents with this cleanup script:
#!/usr/bin/env bash
set -euo pipefail

REGION="${AWS_REGION:-us-east-1}"
STACK_NAME="${STACK_NAME:-ecs-sre-lab}"

aws cloudformation delete-stack \
  --stack-name "$STACK_NAME" \
  --region "$REGION"

aws cloudformation wait stack-delete-complete \
  --stack-name "$STACK_NAME" \
  --region "$REGION"

echo "Stack deletion complete."

What does this script prepare?

The script deletes the CloudFormation stack and waits until deletion finishes. Its final message gives you a concrete cleanup confirmation.

  • Save cleanup.sh.

The saved file should end with Stack deletion complete.

  • Open sample-sla.md from the Explorer sidebar.
  • Replace its contents with this sample agreement:
# Sample Availability SLA

Status: Non-binding learning example

## Covered service

The public HTTP endpoint identified as the SRE Reliability Lab Application Load Balancer URL.

## Service commitment

The provider targets 99.5% monthly availability for the covered endpoint.

## Measurement

Availability is measured from provider-controlled load balancer telemetry. A minute is unavailable when the lab's availability SLI is below its threshold. Production agreements should also account for target-generated failures and independent external checks.

## Exclusions

- Customer network or client failures
- Scheduled maintenance announced in advance
- Force majeure events
- Actions performed by the customer that remove or misconfigure capacity

## Reporting

Monthly reports include measured availability, qualifying incidents, exclusions, and corrective actions.

## Remedy

If the commitment is missed, the customer may request a service credit under the production agreement's claim process and credit schedule.

## Relationship to this lab

The lab's SLI is the measurement. The 99.9% rolling one-day SLO is the stricter internal engineering target. This sample 99.5% monthly SLA is the external promise and is intentionally non-binding.

What does this document define?

The draft separates the customer-facing monthly commitment from the lab's stricter internal target. Its exclusions and remedy framing keep the learning example explicitly non-binding.

  • Save sample-sla.md.

The saved document should begin with Sample Availability SLA and end with the non-binding relationship statement.

Are any files still empty?

Look for a white dot beside a file tab. That dot means the file has unsaved changes.

Use this prompt if a file does not match the reference. Help me compare my ECS reliability lab files with the expected versions.

Set permissions and verify your AWS identity

Linux requires an executable permission before a script can run with its ./script-name form. This change applies only to the six Bash scripts.

  • Make the six lab scripts executable by running this command in the integrated terminal:
chmod +x deploy.sh traffic.sh replace-task.sh slo-report.sh outage-drill.sh cleanup.sh

What does this command do?

The executable bit allows your Linux shell to launch each script through its Bash shebang. The infrastructure template and SLA document remain regular files.

  • Inspect the script permissions by running this command:
ls -l deploy.sh traffic.sh replace-task.sh slo-report.sh outage-drill.sh cleanup.sh

What should you see?

Each permissions string should contain x characters. Those characters confirm that the scripts are executable.

Do the scripts still lack executable permission?

Confirm that the integrated terminal is inside the ecs-sre-lab folder. The permission command only affects files in the current folder.

Use this prompt if the permission change fails. Help me make my lab scripts executable on Linux.

Your final check asks AWS which authenticated identity is active. Before you run it, do you expect the command to return your intended account and role?

  • Verify your AWS identity and HTTP probe by running these commands:
aws sts get-caller-identity --region us-east-1
curl --version

What does the result prove?

The identity response names the AWS account and principal used by later deployments. The curl banner confirms that your HTTP probe remains available in the same terminal environment.

You should see an AWS identity response followed by a curl version banner. That confirms your local tools are ready before any billable AWS resources are created.

Did the identity check fail?

Confirm that your AWS credentials are configured for the account you intend to use. An expired session must be refreshed before the deployment step.

Use this prompt to diagnose the current session. Help me troubleshoot my AWS CLI identity check.

Your workstation is ready with verified tools, executable scripts, and an authenticated AWS identity. Next, you will deploy the load-balanced ECS service and see the lab page respond.

Deploy the Load-Balanced ECS Service

Your verified AWS session is ready in us-east-1. The reliability lab still needs a healthy baseline before you can test recovery or measure user impact.

In this step, AWS CloudFormation deploys the network foundation. An Amazon ECS service runs on an Amazon EC2 container instance behind an Application Load Balancer.

The first visible win is the public lab page. Once it loads, you know the request path works before any failure experiment begins.

In this step, get ready to:
  • Inspect the infrastructure defined in template.yaml.
  • Deploy ecs-sre-lab while keeping the service level objective (SLO) disabled.
  • Confirm the public endpoint displays the healthy lab page.
Inspect the infrastructure plan

The template.yaml file is the infrastructure blueprint for this lab. Its resources show how public traffic reaches one container task running on one EC2 host.

  • Switch to template.yaml in Visual Studio Code.
  • Locate the Parameters section near the top of the file.
  • Confirm CreateSLO defaults to false.
  • Trace the public network path through VPC, PublicSubnetA, PublicSubnetB, InternetGateway, PublicRouteTable, ALBSecurityGroup, plus ECSHostSecurityGroup.
  • Trace the compute path from ECSLaunchTemplate through ECSAutoScalingGroup, ECSCluster, TaskDefinition, plus ECSService.
  • Trace the request path from ApplicationLoadBalancer through ALBListener, ALBTargetGroup, plus ECSService.
  • Confirm AvailabilitySLO uses CreateSLOCondition.
  • Review the Outputs section to identify the values that the deployment script prints.

Why use EC2-backed ECS?

EC2-backed Amazon ECS keeps host capacity visible as part of the reliability experiment. You can connect container scheduling decisions to the capacity available on the host.

The task definition uses bridge networking with dynamic host port mapping. This lets the load balancer reach a task through an automatically assigned host port.

Deploy the stack without the SLO

The deploy.sh script retrieves the recommended Amazon Linux 2023 ECS-optimized AMI. It validates template.yaml before deploying the stack.

Passing false keeps the optional SLO out of this first deployment. This gives you a healthy application baseline before telemetry becomes part of the lab.

AWS starts billing here

This deployment creates billable AWS resources. The lab uses one t3.small instance plus one Application Load Balancer.

The cleanup section removes the stack when you finish. Prompt deletion keeps the experiment bounded.

  • Return to the Linux terminal from the previous step.

CloudFormation creates dependent resources in sequence, so the deployment can stay active for several minutes. A quiet terminal during that wait does not mean the command has failed.

Before you run the deployment, predict the value that SLOArn will show.

  • Deploy the stack with the SLO disabled by running:
./deploy.sh false

What does this command do?

  • The false argument becomes the CreateSLO parameter value. The CreateSLOCondition condition therefore prevents AvailabilitySLO from being created.
  • The script retrieves the recommended Amazon Linux 2023 ECS-optimized AMI ID from Systems Manager.
  • The script validates template.yaml before provisioning any resources from it.
  • The script deploys the ecs-sre-lab stack. It prints the stack outputs after the deployment completes.
  • Keep the terminal open until the command returns to your prompt.
  • Confirm the output table lists LoadBalancerURL, ClusterName, ServiceName, AutoScalingGroupName, LoadBalancerFullName, plus SLOArn.
  • Confirm SLOArn shows Not created.

Because the deployment returned its output table, the new stack has reached CREATE_COMPLETE. That is the biggest setup hurdle cleared: the lab now has a reachable infrastructure baseline.

Deployment stopped before the outputs?

  • Look at the first failed CloudFormation resource in the terminal output. Later failures can be consequences of that first problem.
  • Check that your authenticated AWS identity can create the resource type named in the failure.
  • Keep template.yaml unchanged so the deployment uses the validated project configuration.
  • Help me diagnose my failed ECS CloudFormation deployment.
Open the healthy service

The LoadBalancerURL output is the public entry point to your lab. The load balancer accepts HTTP requests on port 80.

Its target group sends each request to the healthy ECS task. Loading the page checks the complete path from your browser to the container.

Before you load the endpoint, predict whether the request path will reach a healthy task.

  • Record the LoadBalancerURL value here: your LoadBalancerURL.
  • Paste your LoadBalancerURL into your browser's address bar.
  • Press Enter to load the endpoint.

You'll see SRE Reliability Lab with Status: healthy. This proves the public load balancer can route a request to the running container.

  • Refresh the page twice.

Each refresh should return the same healthy page. Your single-task service is responding through the Application Load Balancer.

Page not loading?

  • Compare the browser address with your LoadBalancerURL to catch a missing character.
  • Confirm the address starts with http://.
  • Confirm the deployment command completed before retrying the page.
  • Help me diagnose why my ECS lab page is not loading.

Your ECS service is live behind its load balancer. Next, you'll interrupt the running task to see exactly what ECS self-healing guarantees.

Prove ECS Self-Healing

Your Application Load Balancer is serving the healthy lab page from one Amazon ECS task. That response proves the service starts from a healthy state.

Now you need evidence of what happens when that only task stops. The ECS service scheduler should restore the service's desired count after the interruption.

A successful replacement proves recovery. It does not prove that every request remained available during the replacement gap.

In this step, get ready to:
  • Interrupt the service's current task with the existing replacement drill.
  • Compare the original task identity with the replacement task identity.
  • Inspect the recovered service counts and recent ECS events.
Test the recovery contract

The service scheduler treats the configured task count as a target to maintain. This drill stops the only running task so ECS has to restore that target.

The waiter may remain quiet while the replacement starts. That pause means ECS is still working toward a stable service.

Before you run the drill, do you expect ECS to reuse the stopped task identity or launch a task with a different identity?

  • Start the replacement drill from the ecs-sre-lab terminal by running this command:
./replace-task.sh

What does this drill do?

  • The OLD_TASK variable captures the running task before the interruption.
  • The aws ecs stop-task command stops that task with the reason SRE reliability lab task interruption.
  • The aws ecs wait services-stable command pauses the script until the running count matches the desired count.
  • The NEW_TASK variable captures the task running after the service becomes stable.

The terminal first prints Stopping task:. After the waiter finishes, it prints Old task: and New task:.

Did the replacement drill stop early?

  • Confirm the terminal is using the existing ecs-sre-lab folder so it can find replace-task.sh.
  • Confirm the ecs-sre-lab stack remains CREATE_COMPLETE in us-east-1.
  • Confirm your AWS session can still read the stack outputs and manage the ECS service.
  • Help me diagnose why the ECS replacement drill stopped before the service became stable.
Compare the task identities

Each ECS task has its own Amazon Resource Name, or ARN. Different ARN values show that the scheduler launched a new task after the original task stopped.

  • Find the value printed after Old task: in the terminal.
  • Find the value printed after New task: in the terminal.
  • Compare the two ARN values.
  • Read the messages listed under recentEvents in the final service report.

You should see two different task ARNs. The recent events should describe the service reaching a stable state with its replacement task.

What has self-healing proved?

The scheduler noticed that the service had fewer running tasks than its desired count. It launched a replacement to close that gap.

This evidence establishes replacement-only recovery. It leaves any request failures during the replacement window unmeasured.

Confirm the recovered service

The task identities prove a replacement occurred. The service counts confirm that ECS restored the intended capacity.

Before you inspect the final report, what values would demonstrate that the one-task service has fully recovered?

  • Locate the final JSON object in the terminal output.
  • Confirm desired is 1.
  • Confirm running is 1.
  • Switch back to the browser tab showing the LoadBalancerURL from earlier.
  • Refresh the reliability lab page.

You should see SRE Reliability Lab with Status: healthy again. The Application Load Balancer is routing requests to the healthy replacement task.

Has the lab page not recovered?

  • Allow the target group health checks to mark the replacement task as healthy before refreshing again.
  • Confirm the terminal report still shows desired as 1.
  • Confirm the terminal report still shows running as 1.
  • Help me diagnose why the replacement task is running but the reliability lab page is not healthy.

You have concrete proof that ECS replaces a stopped task and restores the configured capacity. Next, you will generate load balancer telemetry so reliability can be measured from the request path.

Turn Load Balancer Signals into an SLO

Your replacement drill proved that Amazon ECS can restore desired count after a task stops. That result describes infrastructure recovery.

Users experience the service through the Application Load Balancer. Task counts cannot measure that experience.

In this step, you'll generate request telemetry for an availability SLI. CloudWatch Application Signals will evaluate it against an SLO with an error budget. You'll also compare that internal target with a sample SLA.

In this step, get ready to:
  • Generate continuous HTTP requests through the load balancer.
  • Create a period-based SLO from ALB telemetry.
  • Compare the SLI, SLO, and sample SLA.
Generate continuous ALB traffic

ALB metrics are request-driven. Elastic Load Balancing publishes them in 60-second intervals while requests flow.

The probe creates one request each second. This gives the SLO complete periods to evaluate.

  • Use the split-terminal control in the Visual Studio Code terminal panel to create a second terminal in the same ecs-sre-lab folder.
  • Start the continuous probe in the second terminal by running this command:
./traffic.sh

What Does This Probe Measure?

  • The traffic.sh script reads LoadBalancerURL from the CloudFormation stack.
  • The script uses curl to send one request per second.
  • Each printed status records what a user received through the ALB.

You should see a new line every second with HTTP 200. That steady stream confirms the ALB can route requests to the healthy ECS task.

  • Leave traffic.sh running in the second terminal.

Not Seeing Repeated HTTP 200 Responses?

  • Confirm the second terminal is inside the ecs-sre-lab folder.
  • Confirm the stack remains deployed in us-east-1.
  • Confirm the reliability lab page still loads through the ALB URL.
  • Help me diagnose unexpected responses from traffic.sh.
Create the availability SLO

The SLI evaluates the expression 100 * requests / (requests + FILL(errors, 0)) for each one-minute period. A period meets the metric threshold when availability reaches 99.0.

The SLO targets 99.9% attainment over a rolling one-day interval. Failed periods reduce the available budget within that interval.

Before You Enable the SLO

Application Signals SLO usage is billable. This lab is designed to remain under $10 when run in us-east-1 and deleted promptly.

The stack update can take several minutes while CloudFormation adds the SLO. A quiet terminal during part of that wait is normal.

  • Keep traffic.sh running until at least two complete one-minute periods have passed.
  • Switch back to the first terminal from earlier.
  • Enable the SLO by updating the existing stack with this command:
./deploy.sh true

What Does This Update Do?

  • The true argument sets CreateSLO to true.
  • The CreateSLOCondition condition enables the existing AvailabilitySLO resource in template.yaml.
  • CloudFormation retains the healthy ECS service while adding ecs-sre-lab-availability.
  • The deployment output returns the new identifier through SLOArn.

You should see a successful stack update. The SLOArn row now contains the identifier for ecs-sre-lab-availability instead of Not created.

Good progress. Your existing service stayed healthy while the stack gained a measurable reliability target.

Did the Stack Update Fail?

  • Read the deployment output for the resource that prevented the update.
  • Confirm your AWS identity can create Application Signals resources.
  • Confirm traffic.sh remains active in the second terminal.
  • Help me diagnose the failed SLO stack update.

Before you request the first report, which budget status do you expect while the earliest one-minute periods arrive?

  • Retrieve the SLO attainment and error budget by running this command:
./slo-report.sh

What Does This Report Show?

  • The Name field identifies the SLO as ecs-sre-lab-availability.
  • The BudgetStatus field can report OK, WARNING, BREACHED, or INSUFFICIENT_DATA.
  • The Attainment field reports the percentage of good periods.
  • The TotalBudgetSeconds field defines the available budget for the interval.
  • The BudgetSecondsRemaining field shows how much of that budget remains.

Your first report should name ecs-sre-lab-availability. Early data can produce OK or INSUFFICIENT_DATA while the first periods arrive.

If Data Is Still Arriving

  • Keep the traffic probe running through two more complete one-minute periods.
  • Rerun the slo-report.sh script after those periods finish.
Compare the SLI, SLO, and SLA

An SLO guides internal reliability decisions. An SLA defines the scope of an external commitment.

The sample agreement is explicitly non-binding. Its target is 99.5% monthly availability for the covered endpoint.

  • Switch back to the sample-sla.md tab in Visual Studio Code.
  • Read the Covered service section.
  • Locate the 99.5% monthly target in Service commitment.
  • Review how the Measurement section defines an unavailable minute.
  • Review the customer-controlled conditions in Exclusions.
  • Review the evidence described in Reporting.
  • Review the service-credit language in Remedy.

How Do the Three Layers Differ?

The SLI is the measured ALB availability percentage. It uses RequestCount with HTTPCode_ELB_5XX_Count.

The error metric includes HTTP 5XX responses generated by the ALB. It excludes failures generated by application targets.

The SLO is the stricter internal target of 99.9% over a rolling day. Its error budget turns failed periods into a measurable reliability cost.

The sample SLA is the non-binding external commitment of 99.5% monthly availability. It adds exclusions plus reporting and remedy terms.

Before you run the final check, do you expect the SLO to have enough periods for a populated budget report?

  • Return to the first terminal from earlier.
  • Run the final SLO check with this command:
./slo-report.sh

What Does the Final Check Prove?

  • A named report confirms that Application Signals can read the ALB-derived SLI.
  • A populated BudgetStatus confirms that the SLO is evaluating incoming periods.
  • The attainment and budget fields connect live request outcomes to the internal reliability target.

You should see ecs-sre-lab-availability with its attainment and budget fields. The status may remain INSUFFICIENT_DATA until more complete periods arrive.

  • Return to the second terminal from earlier.

You should still see repeated HTTP 200 responses. Those requests continue feeding the one-minute ALB metrics.

  • Leave traffic.sh running for the next reliability drill.

That closes the measurement loop. Live requests now feed an SLO report with a visible error budget.

Next, you'll test the current one-task topology with a controlled capacity change.

Spend the Error Budget

The earlier replacement drill showed that Amazon ECS can restore a stopped task. That service-level success did not measure the user experience while healthy capacity was unavailable.

Now you will run a controlled zero-capacity drill. The error budget turns the resulting Application Load Balancer telemetry into evidence against the 99.9% reliability target.

In this step, get ready to:
  • Run a controlled drill that temporarily removes all ECS task capacity.
  • Observe the user-facing response stream during the drill.
  • Measure the outage's effect on the SLO report.
Run the controlled outage

An ECS service's desired count defines how many task copies the scheduler tries to keep running. Setting it to zero deliberately empties the target group long enough to measure user impact.

Taking the service to zero is intentionally disruptive. The script's cleanup trap restores one task when the drill exits.

  • Return to the first terminal from earlier.
  • Leave traffic.sh running in the second terminal.

The script holds zero capacity for 75 seconds. It also waits for ECS to stabilize, so the quiet stretch in the first terminal is expected.

Before you run the drill, predict whether the continuous probe can stay successful while the target group has no capacity.

  • Start the controlled outage by running:
./outage-drill.sh

What Does This Drill Do?

  • The first service update sets the desired count to 0.
  • The script holds zero capacity for 75 seconds.
  • The restore_service() function returns the desired count to 1.
  • The services-stable waiter keeps the script active until the restored service reaches its desired count.
  • Watch the second terminal while the drill runs.
  • Wait for the first terminal to print Service restored. Leave traffic.sh running while metrics are published..

During zero capacity, you'll see repeated HTTP 503 responses in the traffic terminal. Once a healthy task registers, the newest lines return to HTTP 200.

That outage is the intended shortfall. ECS restored the desired count only after users saw failed requests.

Confirm the service recovery

The final waiter returns when the running task count matches the desired count. A completed drill therefore confirms that the service recovered to one running task.

  • Return to the first terminal after the traffic responses recover.
  • Confirm the final line reports that the service was restored.
  • Switch back to the second terminal.
  • Confirm the newest responses show HTTP 200.

You brought the service back safely. The traffic history now preserves clear evidence that replacement-only recovery still allowed failed requests.

Measure the budget impact

Load balancer request metrics publish in 60-second intervals. For a period-based SLO, attainment is the number of good periods divided by the number of total periods.

Each failed period reduces the error budget within the rolling interval. Amazon CloudWatch Application Signals exposes that effect through the budget report.

This pause lasts at least two minutes because the metrics arrive in 60-second periods. That publication delay is expected.

  • Keep traffic.sh active in the second terminal.
  • Wait through two complete 60-second publication periods.

Before you request the report, predict which field will reveal the clearest change from the failed period.

  • Request the post-outage budget report by running:
./slo-report.sh

What Does This Report Show?

  • The Name field identifies the SLO being evaluated.
  • The Attainment field reports the percentage of good periods in the rolling interval.
  • The BudgetStatus field classifies the current budget as OK, WARNING, BREACHED, or INSUFFICIENT_DATA.
  • The TotalBudgetSeconds field shows the full error allowance for the interval.
  • The BudgetSecondsRemaining field shows how much of that allowance remains.

You'll see a table for ecs-sre-lab-availability containing all five report fields. At least one of Attainment, BudgetStatus, or BudgetSecondsRemaining now reflects the controlled outage.

Still Seeing Insufficient Data?

Request-driven load balancer metrics only appear after requests flow through complete 60-second periods. Give the report more publication time before checking again.

  • Leave traffic.sh active for two additional periods.
  • Return to the first terminal.
  • Rerun slo-report.sh.
  • Get help diagnosing the delayed SLO report with this troubleshooting prompt.

You have turned a controlled outage into user-visible evidence and measurable SLO impact. The lab now connects infrastructure recovery directly to reliability budget consumption.

Secret mission

Add Task-Level Redundancy

Replacement restores capacity after a task stops. This mission tests whether a second healthy task can preserve HTTP availability during that interruption while both tasks share one EC2 host.

Clean Up Your Resources

Clean Up Your Resources

Your Amazon Web Services stack remains active, so its billable resources can keep accruing charges. Choose whether to keep the lab running, pause its compute capacity, or delete every project resource.

Cost warning

  • Amazon ECS orchestration adds no extra charge for the EC2 launch type.
  • The Amazon EC2 instance and Amazon EBS storage can continue accruing charges.
  • The Application Load Balancer and its public IPv4 addresses can continue accruing charges. An ALB partial hour is billed as a full hour.
  • Amazon CloudWatch Logs and the Application Signals SLO can continue generating usage.
  • Choose the Delete option when you finish experimenting. Prompt deletion is the recommended way to stay within the project's $10 limit.

Resources you used:

  • The ecs-sre-lab AWS CloudFormation stack in us-east-1.
  • The stack's Amazon VPC, internet gateway, two public subnets, route table, and security groups.
  • The stack's IAM role, instance profile, launch template, one-instance EC2 Auto Scaling group, ECS cluster, task definition, and service.
  • The stack's CloudWatch log group, ALB, target group, listener, and availability SLO.
  • The active traffic.sh probe in the second terminal.
  • The local ecs-sre-lab project folder containing your template, scripts, and SLA draft.

Keep everything running

No action is needed. Choose this option if you plan to run more failure experiments soon and accept the ongoing AWS charges.

  • Leave the ecs-sre-lab stack active with one healthy ECS task.
  • Leave the traffic.sh terminal active if you still need continuous ALB telemetry.
  • Monitor your AWS charges while the EC2 instance, EBS volume, ALB, public IPv4 addresses, logs, and SLO remain active.
  • Keep the ecs-sre-lab project folder so you can repeat the drills or extend the lab.

Pause - I'll come back to this later

Pause the task and EC2 compute capacity while keeping the stack available for later. The ALB, public IPv4 addresses, logs, and SLO remain active, so this option can still incur charges.

  • Switch back to the terminal running traffic.sh from earlier.
  • Press Ctrl+C to stop the continuous probe.
  • Switch back to the project terminal from earlier.
  • Pause the ECS task and EC2 capacity by running:
REGION="${AWS_REGION:-us-east-1}"
STACK_NAME="${STACK_NAME:-ecs-sre-lab}"

CLUSTER="${STACK_NAME}-cluster"
SERVICE="${STACK_NAME}-service"
ASG_NAME=$(aws cloudformation describe-stacks \
  --stack-name "$STACK_NAME" \
  --region "$REGION" \
  --query 'Stacks[0].Outputs[?OutputKey==`AutoScalingGroupName`].OutputValue' \
  --output text)

aws ecs update-service \
  --cluster "$CLUSTER" \
  --service "$SERVICE" \
  --desired-count 0 \
  --region "$REGION" >/dev/null

aws ecs wait services-stable \
  --cluster "$CLUSTER" \
  --services "$SERVICE" \
  --region "$REGION"

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name "$ASG_NAME" \
  --min-size 0 \
  --desired-capacity 0 \
  --region "$REGION"

What does this pause command do?

  • The update-service command reduces the ECS desired count to 0.
  • The ECS waiter confirms that the service reaches its new stable state.
  • The stack output identifies the generated Auto Scaling group.
  • The update-auto-scaling-group command reduces its minimum and desired capacity to 0.

The command returns after the ECS service reaches zero desired tasks and the Auto Scaling group receives its zero-capacity configuration. Your stack remains deployed with its non-compute resources.

Compute capacity not pausing?

  • Confirm that your AWS CLI session is still authenticated in us-east-1.
  • Check that your identity can update the ECS service and EC2 Auto Scaling group.
  • Help me diagnose why my ECS service or Auto Scaling group did not pause.

When you return, restore the EC2 capacity before asking ECS to run a task. The waiter keeps the command active until the service is stable again.

  • Restore one EC2 instance and one ECS task by running:
REGION="${AWS_REGION:-us-east-1}"
STACK_NAME="${STACK_NAME:-ecs-sre-lab}"

CLUSTER="${STACK_NAME}-cluster"
SERVICE="${STACK_NAME}-service"
ASG_NAME=$(aws cloudformation describe-stacks \
  --stack-name "$STACK_NAME" \
  --region "$REGION" \
  --query 'Stacks[0].Outputs[?OutputKey==`AutoScalingGroupName`].OutputValue' \
  --output text)

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name "$ASG_NAME" \
  --min-size 1 \
  --desired-capacity 1 \
  --region "$REGION"

aws ecs update-service \
  --cluster "$CLUSTER" \
  --service "$SERVICE" \
  --desired-count 1 \
  --region "$REGION" >/dev/null

aws ecs wait services-stable \
  --cluster "$CLUSTER" \
  --services "$SERVICE" \
  --region "$REGION"

What does this restore command do?

  • The Auto Scaling update restores one EC2 container instance.
  • The ECS update restores the service's desired count to 1.
  • The waiter returns after ECS has one stable running task.

The restore can take time while the EC2 instance starts and joins the ECS cluster. Keep the terminal open until the waiter returns.

Service not becoming stable?

  • Confirm that the Auto Scaling group has a minimum and desired capacity of 1.
  • Confirm that the ECS service has a desired count of 1.
  • Help me diagnose why the restored ECS service is not becoming stable.
  • Restart the continuous probe in the second terminal by running:
./traffic.sh

What does this probe confirm?

The script sends one request per second through the ALB. Its response codes show when the restored task is serving traffic.

You should see repeated HTTP 200 responses. Your paused lab is active again.

Probe not returning HTTP 200?

  • Wait for the ECS service waiter to finish before restarting the probe.
  • Confirm that the service has one running task and one healthy ALB target.
  • Help me diagnose why traffic.sh is not returning HTTP 200 after the lab was restored.

Delete - I don't want to use this again

Remove the AWS stack to stop its cloud resources from accruing charges. This action deletes the lab infrastructure while leaving your local files available until you remove them separately.

  • Switch back to the terminal running traffic.sh from earlier.
  • Press Ctrl+C to stop the continuous probe.
  • Switch back to the project terminal from earlier.

Deleting the stack is final, but your local scripts remain untouched during this command. The script waits for CloudFormation to finish before confirming success.

  • Delete the ecs-sre-lab stack by running:
./cleanup.sh

What does this cleanup command do?

  • The script requests deletion of the ecs-sre-lab CloudFormation stack in us-east-1.
  • The CloudFormation waiter keeps the script active until stack deletion finishes.
  • The deletion removes the stack-managed networking, compute, load-balancing, logging, and SLO resources.

Keep the terminal open while CloudFormation removes the stack. You will see Stack deletion complete. when every stack-managed resource is gone.

Stack deletion not completing?

  • Confirm that your AWS CLI session is still authenticated in us-east-1.
  • Check that your identity can delete CloudFormation resources.
  • Help me diagnose why the ecs-sre-lab stack is not deleting.

The final local cleanup permanently removes your lab files. Your AWS resources are already gone at this point, so the command only targets the ecs-sre-lab folder.

  • Move to the folder above ecs-sre-lab and remove the project by running:
cd ..
rm -rf ecs-sre-lab

What does this local cleanup do?

The first command leaves the project folder. The second command permanently removes the ecs-sre-lab folder with its template, scripts, and SLA draft.

  • Inspect the parent directory in your Linux file manager.

You should no longer see the ecs-sre-lab folder. Your local files and AWS lab resources are now cleaned up.

Nice Work!

Nice Work!

You did it! Your Amazon ECS reliability lab now serves a public page from an Amazon EC2 host through an Application Load Balancer.

Your failure drills connect infrastructure recovery to user-visible HTTP responses. The SLO report turns that traffic into evidence about the remaining error budget.

You've learned how to:

  • Deploy an EC2-backed ECS service with AWS CloudFormation. Route public requests through an internet-facing load balancer.
  • Test replacement-only recovery by stopping the sole task. Expose the zero-capacity outage through HTTP 503 responses.
  • Convert ALB telemetry into an availability SLI with a period-based SLO. Use CloudWatch Application Signals to inspect attainment plus remaining budget seconds. Separate that internal target from the non-binding sample SLA.
  • Complete the Secret Mission by proving task-level redundancy can preserve HTTP 200 responses during one task interruption. Recognize that both tasks still share one EC2 host. Recognize that the experiment does not provide Availability Zone redundancy.

Ready to quiz yourself?