Monitor a SaaS App with Prometheus on AWS

Build an end-to-end SRE monitoring pipeline with Grafana and alerting on AWS.

Introduction

30 Second Summary

When a production service starts failing, how quickly your team detects and responds determines whether it's a minor blip or a full-blown outage. Enterprise SRE teams solve this with monitoring pipelines that automatically detect degradation, fire alerts, and guide investigation.

In this project, you will build an end-to-end monitoring pipeline for a containerized SaaS application on AWS. You'll instrument a Flask API with Prometheus metrics, visualize them in Grafana dashboards, configure alert rules, and practice the full incident response lifecycle by triggering and resolving a simulated outage.

What You'll Build

A production-style observability pipeline that instruments a Flask API on ECS Fargate, scrapes metrics via an ADOT sidecar into Amazon Managed Prometheus, visualizes Golden Signals in Amazon Managed Grafana, and routes severity-tiered alerts through SNS.

By the end of this project, you'll have:

  • A Terraform project that provisions the entire monitoring stack on AWS, including ECS Fargate, Amazon Managed Prometheus, Amazon Managed Grafana, and SNS.
  • A Dockerized Flask API instrumented with Prometheus RED metrics, scraped by an ADOT sidecar and visualized in Golden Signals dashboards.
  • A completed incident response exercise where you generate load, trigger a failure, receive an alert notification, investigate with PromQL, and resolve the issue.
  • Secret Mission: Add multi-window burn-rate SLO alerts and a dead man's switch to monitor the monitoring.

Are there any prerequisites?

This project assumes you already have an AWS account with the AWS CLI configured, Terraform installed, and Docker installed. Step 1 will verify each of these before you begin building.

Set Up the Terraform Project and Build the App

Before you can monitor anything, you need something to monitor. In this step, you will provision the AWS infrastructure and build the Flask application that simulates a production SaaS API.

The app will expose RED metrics (Rate, Errors, Duration) via a Prometheus endpoint. You will also set up the container registry and cluster where the app will eventually run.

In this step, get ready to:
  • Provision an ECR repository, VPC, and ECS cluster using Terraform.
  • Write a Flask API instrumented with Prometheus RED metrics.
  • Containerize the app and push the image to ECR.
Create the Terraform project and provision infrastructure

Terraform lets you define cloud infrastructure as code. You will create a project that provisions an ECR repository (for your Docker image), a VPC (for networking), and an ECS cluster (for running containers).

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir saas-monitor
cd saas-monitor
  • Confirm the folder exists by running:
ls

You should see an empty directory since you just created it.

  • Create a file called variables.tf in the saas-monitor folder, then paste in the following code:
variable "region" {
  default = "us-east-1"
}

variable "project_name" {
  default = "saas-monitor"
}

What does this code do?

This defines two input variables for your Terraform project.

  • region controls which AWS region your resources deploy to. The default is us-east-1.
  • project_name is a prefix used in resource names so everything stays organized.
  • Save variables.tf.
  • Create a file called main.tf in the same saas-monitor folder, then paste in the following code:
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  backend "local" {}
}

provider "aws" {
  region = var.region
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"
  name    = "${var.project_name}-vpc"
  cidr    = "10.0.0.0/16"
  azs             = ["${var.region}a", "${var.region}b"]
  public_subnets  = ["10.0.1.0/24", "10.0.2.0/24"]
  enable_nat_gateway = false
}

resource "aws_ecr_repository" "app" {
  name = "${var.project_name}-app"
}

resource "aws_ecs_cluster" "main" {
  name = "${var.project_name}-cluster"
}

What does this code do?

This is your infrastructure definition. It provisions everything you need to run a containerized app on AWS.

  • The terraform block pins the AWS provider to version 5.x and uses a local state backend (no remote state for this project).
  • The vpc module creates a network with two public subnets across two availability zones. Your ECS tasks will run here.
  • aws_ecr_repository creates a container registry where you will push your Docker image.
  • aws_ecs_cluster creates the ECS cluster that will host your running containers in a later step.
  • Save main.tf.

Not sure which editor to use?

You can use any text editor (VS Code, Cursor, Sublime, vim) to create and edit files. Just make sure you save both files inside the saas-monitor folder on your Desktop.

Still stuck? Help me open and edit files in my code editor.

  • Initialize the Terraform project by running this command in your terminal (make sure you are inside the saas-monitor folder):
terraform init

You should see Terraform has been successfully initialized! in the output. This downloads the AWS provider and the VPC module.

Seeing a provider or module error?

Make sure your terminal's working directory is the saas-monitor folder (run pwd to check). Both main.tf and variables.tf must be in this folder.

Help me debug my terraform init error.

  • Provision the infrastructure by running:
terraform apply
  • Terraform will show you a plan of what it intends to create. Type yes and press Enter to confirm.

The apply takes 1-2 minutes. When it finishes, you should see Apply complete! followed by a count of resources added.

Terraform apply failed?

The most common cause is AWS credentials not being configured. Run aws sts get-caller-identity to check if your CLI can reach AWS. If that fails, reconfigure with aws configure.

Help me fix my terraform apply error.

✔️ Awesome, I've got everything!

Great. Make sure both files are saved and your terraform apply succeeded before moving on.

ⓧ I'd like to double check the full code

variable "region" {
  default = "us-east-1"
}

variable "project_name" {
  default = "saas-monitor"
}
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  backend "local" {}
}

provider "aws" {
  region = var.region
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"
  name    = "${var.project_name}-vpc"
  cidr    = "10.0.0.0/16"
  azs             = ["${var.region}a", "${var.region}b"]
  public_subnets  = ["10.0.1.0/24", "10.0.2.0/24"]
  enable_nat_gateway = false
}

resource "aws_ecr_repository" "app" {
  name = "${var.project_name}-app"
}

resource "aws_ecs_cluster" "main" {
  name = "${var.project_name}-cluster"
}
Write the Flask application with Prometheus metrics

Your monitoring pipeline needs an application that produces real metrics. You will build a Flask API that simulates a SaaS service with realistic behavior: variable latency, a baseline error rate, and the ability to trigger an incident on demand.

  • Create a file called app.py in your saas-monitor folder. Start by pasting the setup code at the top:
import time
import random
from flask import Flask
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app)

incident_mode = False

What does this code do?

  • PrometheusMetrics(app) from prometheus-flask-exporter automatically instruments every Flask route. It tracks request counts, durations, and error rates without any manual instrumentation.
  • incident_mode is a global flag that controls whether the app simulates a production incident (high error rate). You will toggle this later to trigger alerts.
  • Below the setup code, add the two main business endpoints:
@app.route('/api/orders')
def orders():
    time.sleep(random.uniform(0.05, 0.5))
    if incident_mode and random.random() < 0.5:
        return 'Internal Server Error', 500
    if random.random() < 0.05:
        return 'Internal Server Error', 500
    return {'orders': []}, 200

@app.route('/api/users')
def users():
    time.sleep(random.uniform(0.05, 0.5))
    if incident_mode and random.random() < 0.5:
        return 'Internal Server Error', 500
    if random.random() < 0.05:
        return 'Internal Server Error', 500
    return {'users': []}, 200

What does this code do?

These are your SaaS API endpoints. Each one simulates production-like behavior:

  • random.uniform(0.05, 0.5) adds a 50-500ms random delay, simulating real-world variable latency.
  • The baseline 5% error rate (random.random() < 0.05) creates a steady trickle of errors that your dashboard will show.
  • When incident_mode is True, the error rate spikes to 50%. This is what triggers your alerts in a later step.
  • Add the health check, incident simulation endpoints, and the main block at the bottom of app.py:
@app.route('/api/health')
def health():
    return {'status': 'ok'}, 200

@app.route('/simulate/incident')
def simulate_incident():
    global incident_mode
    incident_mode = True
    return {'incident': True}, 200

@app.route('/simulate/resolve')
def simulate_resolve():
    global incident_mode
    incident_mode = False
    return {'incident': False}, 200

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

What does this code do?

  • /api/health is a simple health check endpoint that always returns 200. Monitoring systems use this to confirm the service is alive.
  • /simulate/incident flips the error rate from 5% to 50%, simulating a production failure.
  • /simulate/resolve returns the error rate to baseline, simulating an incident fix.
  • The app.run(host='0.0.0.0', port=5000) line makes the app accessible on port 5000 from outside the container.
  • Save app.py.

✔️ Awesome, I've got everything!

Great. Make sure the file is saved before moving on to the next substep.

ⓧ I'd like to double check the full code

import time
import random
from flask import Flask
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app)

incident_mode = False

@app.route('/api/orders')
def orders():
    time.sleep(random.uniform(0.05, 0.5))
    if incident_mode and random.random() < 0.5:
        return 'Internal Server Error', 500
    if random.random() < 0.05:
        return 'Internal Server Error', 500
    return {'orders': []}, 200

@app.route('/api/users')
def users():
    time.sleep(random.uniform(0.05, 0.5))
    if incident_mode and random.random() < 0.5:
        return 'Internal Server Error', 500
    if random.random() < 0.05:
        return 'Internal Server Error', 500
    return {'users': []}, 200

@app.route('/api/health')
def health():
    return {'status': 'ok'}, 200

@app.route('/simulate/incident')
def simulate_incident():
    global incident_mode
    incident_mode = True
    return {'incident': True}, 200

@app.route('/simulate/resolve')
def simulate_resolve():
    global incident_mode
    incident_mode = False
    return {'incident': False}, 200

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)
Containerize the app and push to ECR

Your Flask app needs to run inside a Docker container so ECS can manage it. You will write a Dockerfile, build the image, test it locally, then push it to the ECR repository you just created.

  • Create a file called requirements.txt in your saas-monitor folder and paste the following:
flask
prometheus-flask-exporter
  • Create a file called Dockerfile (no extension) in the same folder and paste the following:
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY app.py .
EXPOSE 5000
CMD ["python", "app.py"]

What does this Dockerfile do?

  • FROM python:3.11-slim starts from a lightweight Python image.
  • It copies requirements.txt first and installs dependencies before copying the app code. This caching strategy speeds up rebuilds.
  • EXPOSE 5000 documents the port the app listens on. The ADOT sidecar will scrape metrics from this port.
  • Save both requirements.txt and Dockerfile.
  • Build the Docker image by running:
docker build -t saas-monitor-app .

You should see each Dockerfile step execute, ending with a message confirming the image was tagged as saas-monitor-app:latest.

Docker build failed?

Make sure Docker Desktop is running. Also confirm that Dockerfile, requirements.txt, and app.py are all in the same saas-monitor folder.

Help me fix my Docker build error.

  • Test the app locally to verify the metrics endpoint works. Run the container:
docker run -p 5000:5000 saas-monitor-app
  • Open a new terminal window and check the Prometheus metrics endpoint by running:
curl http://localhost:5000/metrics

You should see Prometheus exposition format output including lines containing flask_http_request_duration_seconds and flask_http_request_total. This confirms the instrumentation is working.

Why is the /metrics endpoint important?

This is the endpoint that the ADOT sidecar will scrape every 15 seconds in production. The flask_http_request_total counter tracks Rate and Errors. The flask_http_request_duration_seconds histogram tracks Duration.

Together, these give you the RED method metrics your Golden Signals dashboard will visualize.

  • Stop the running container by pressing Ctrl+C in the terminal where it is running.
  • Authenticate Docker to your ECR registry by running:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com

Find your AWS account ID

Run aws sts get-caller-identity to see your 12-digit account ID in the output. Copy the Account field value.

You should see Login Succeeded in the output.

  • Tag the image for your ECR repository by running:
docker tag saas-monitor-app:latest [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/saas-monitor-app:latest
  • Push the image to ECR by running:
docker push [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/saas-monitor-app:latest

You should see each layer being pushed, ending with a digest hash and the latest tag confirmed.

Docker push returned an authorization error?

The ECR login token expires after 12 hours. If you waited a while between login and push, re-run the aws ecr get-login-password command above. Also double-check that your account ID has no typos.

Help me fix my Docker push error.

✔️ Awesome, I've got everything!

Great. Your image is now in ECR, ready for ECS to pull in the next step.

ⓧ I'd like to double check the full code

flask
prometheus-flask-exporter
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY app.py .
EXPOSE 5000
CMD ["python", "app.py"]

Your infrastructure is provisioned and your instrumented app image is waiting in ECR. Next up, you will deploy the app on ECS Fargate with an ADOT sidecar that scrapes your metrics and ships them to Amazon Managed Prometheus.

Deploy ECS Fargate with ADOT Sidecar and AMP

Your Flask app is containerized and pushed to ECR. Now you need a way to collect the /metrics data your app exposes and ship it to a managed storage backend.

In this step, you will deploy your app on ECS Fargate alongside an ADOT sidecar container that scrapes Prometheus metrics and writes them to Amazon Managed Service for Prometheus (AMP).

In this step, get ready to:
  • Provision an AMP workspace, IAM roles, and an ADOT collector configuration via Terraform.
  • Deploy an ECS task definition with a Flask app container and an ADOT sidecar container.
  • Verify that metrics flow from ECS through ADOT into AMP.
Add the AMP workspace and IAM roles

Before your app can ship metrics, you need three things: a place to store them (AMP workspace), permission for ECS to write to AMP (task role), and permission for ECS to pull container images (execution role).

  • In main.tf, add the following code below the aws_ecs_cluster resource:
resource "aws_prometheus_workspace" "main" {
  alias = "${var.project_name}-amp"
}

resource "aws_iam_role" "ecs_task_role" {
  name = "${var.project_name}-task-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "task_prometheus" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess"
}

resource "aws_iam_role" "ecs_execution_role" {
  name = "${var.project_name}-execution-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "execution_policy" {
  role       = aws_iam_role.ecs_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

What does this code do?

The aws_prometheus_workspace resource creates a managed Prometheus workspace where your metrics will be stored. No servers to manage.

The task role gets the AmazonPrometheusRemoteWriteAccess policy, which allows the ADOT sidecar to push metrics into AMP using the remote_write API.

The execution role gets AmazonECSTaskExecutionRolePolicy. ECS itself uses this role to pull your Docker image from ECR and write container logs.

  • Save main.tf.

Seeing a syntax error in your editor?

Make sure you placed the new code after the closing } of the aws_ecs_cluster resource, not inside it. Each resource block must be at the top level of the file.

help me fix a Terraform syntax error in my IAM role definitions

Define the ADOT collector configuration in SSM

The ADOT sidecar needs a YAML configuration that tells it where to scrape metrics and where to ship them. You will store this config in AWS Systems Manager Parameter Store so the sidecar can read it at startup.

  • In main.tf, add the following code below the execution_policy attachment:
resource "aws_ssm_parameter" "adot_config" {
  name  = "/${var.project_name}/adot-config"
  type  = "String"
  value = <<-EOT
    extensions:
      health_check:
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'flask-app'
              scrape_interval: 15s
              static_configs:
                - targets: ['localhost:5000']
      awsecscontainermetrics:
    exporters:
      prometheusremotewrite:
        endpoint: "${aws_prometheus_workspace.main.prometheus_endpoint}api/v1/remote_write"
        auth:
          authenticator: sigv4auth
    extensions:
      sigv4auth:
        region: "${var.region}"
        service: "aps"
    service:
      extensions: [health_check, sigv4auth]
      pipelines:
        metrics:
          receivers: [prometheus, awsecscontainermetrics]
          exporters: [prometheusremotewrite]
  EOT
}

What is this ADOT config doing?

This YAML defines the ADOT collector pipeline. The prometheus receiver scrapes your Flask app at localhost:5000 every 15 seconds. The awsecscontainermetrics receiver captures task-level CPU and memory metrics from ECS.

The prometheusremotewrite exporter pushes all collected metrics to your AMP workspace. The sigv4auth extension signs requests with AWS credentials so AMP accepts them.

Because the sidecar runs in the same ECS task as your Flask app, localhost:5000 resolves to the app container. This is the sidecar pattern in action.

  • Save main.tf.

Editor showing HCL heredoc errors?

Ensure the closing EOT is indented with spaces, not tabs. Terraform's <<-EOT syntax strips leading tabs but fails on mixed indentation.

help me fix an HCL heredoc formatting issue in my ADOT config

Define the ECS task definition and service

Now you will define how ECS runs your containers. The task definition bundles the Flask app and the ADOT sidecar into a single unit. The service keeps one instance of that task running on Fargate.

  • In main.tf, add the security group, task definition, and service below the aws_ssm_parameter resource:
resource "aws_security_group" "ecs_task" {
  name   = "${var.project_name}-ecs-sg"
  vpc_id = module.vpc.vpc_id
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_ecs_task_definition" "app" {
  family                   = "${var.project_name}-task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "256"
  memory                   = "512"
  task_role_arn            = aws_iam_role.ecs_task_role.arn
  execution_role_arn       = aws_iam_role.ecs_execution_role.arn
  container_definitions = jsonencode([
    {
      name      = "flask-app"
      image     = "${aws_ecr_repository.app.repository_url}:latest"
      cpu       = 128
      memory    = 256
      portMappings = [{ containerPort = 5000 }]
    },
    {
      name      = "adot-collector"
      image     = "public.ecr.aws/aws-observability/aws-otel-collector:v0.48.0"
      cpu       = 128
      memory    = 256
      environment = [{
        name  = "AOT_CONFIG_CONTENT"
        value = aws_ssm_parameter.adot_config.value
      }]
    }
  ])
}

resource "aws_ecs_service" "app" {
  name            = "${var.project_name}-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 1
  launch_type     = "FARGATE"
  network_configuration {
    subnets          = module.vpc.public_subnets
    security_groups  = [aws_security_group.ecs_task.id]
    assign_public_ip = true
  }
}

What does this code do?

The security group allows all outbound traffic so both containers can reach AMP and ECR. No inbound rules are needed yet since the app is not exposed to the internet directly in this step.

The task definition creates a Fargate task with 256 CPU units and 512 MiB of memory split evenly between two containers. The flask-app container runs your instrumented API on port 5000. The adot-collector container (public.ecr.aws/aws-observability/aws-otel-collector:v0.48.0) reads its config from the AOT_CONFIG_CONTENT environment variable.

The ECS service ensures one copy of this task is always running in your public subnets with a public IP so it can reach AWS endpoints.

  • Save main.tf.

Seeing a reference to an unknown resource?

If Terraform complains about aws_iam_role.ecs_task_role or aws_ssm_parameter.adot_config not being found, confirm you added the IAM and SSM resources from the previous substeps above this code.

help me resolve a Terraform unknown resource reference error in my ECS task definition

✔️ Awesome, I've got everything!

Great. Double check you have saved main.tf with all the new resources.

ⓧ I'd like to double check the full code

Here is the complete main.tf after adding all resources:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  backend "local" {}
}

provider "aws" {
  region = var.region
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"
  name    = "${var.project_name}-vpc"
  cidr    = "10.0.0.0/16"
  azs             = ["${var.region}a", "${var.region}b"]
  public_subnets  = ["10.0.1.0/24", "10.0.2.0/24"]
  enable_nat_gateway = false
}

resource "aws_ecr_repository" "app" {
  name = "${var.project_name}-app"
}

resource "aws_ecs_cluster" "main" {
  name = "${var.project_name}-cluster"
}

resource "aws_prometheus_workspace" "main" {
  alias = "${var.project_name}-amp"
}

resource "aws_iam_role" "ecs_task_role" {
  name = "${var.project_name}-task-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "task_prometheus" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess"
}

resource "aws_iam_role" "ecs_execution_role" {
  name = "${var.project_name}-execution-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "execution_policy" {
  role       = aws_iam_role.ecs_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_ssm_parameter" "adot_config" {
  name  = "/${var.project_name}/adot-config"
  type  = "String"
  value = <<-EOT
    extensions:
      health_check:
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'flask-app'
              scrape_interval: 15s
              static_configs:
                - targets: ['localhost:5000']
      awsecscontainermetrics:
    exporters:
      prometheusremotewrite:
        endpoint: "${aws_prometheus_workspace.main.prometheus_endpoint}api/v1/remote_write"
        auth:
          authenticator: sigv4auth
    extensions:
      sigv4auth:
        region: "${var.region}"
        service: "aps"
    service:
      extensions: [health_check, sigv4auth]
      pipelines:
        metrics:
          receivers: [prometheus, awsecscontainermetrics]
          exporters: [prometheusremotewrite]
  EOT
}

resource "aws_security_group" "ecs_task" {
  name   = "${var.project_name}-ecs-sg"
  vpc_id = module.vpc.vpc_id
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_ecs_task_definition" "app" {
  family                   = "${var.project_name}-task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "256"
  memory                   = "512"
  task_role_arn            = aws_iam_role.ecs_task_role.arn
  execution_role_arn       = aws_iam_role.ecs_execution_role.arn
  container_definitions = jsonencode([
    {
      name      = "flask-app"
      image     = "${aws_ecr_repository.app.repository_url}:latest"
      cpu       = 128
      memory    = 256
      portMappings = [{ containerPort = 5000 }]
    },
    {
      name      = "adot-collector"
      image     = "public.ecr.aws/aws-observability/aws-otel-collector:v0.48.0"
      cpu       = 128
      memory    = 256
      environment = [{
        name  = "AOT_CONFIG_CONTENT"
        value = aws_ssm_parameter.adot_config.value
      }]
    }
  ])
}

resource "aws_ecs_service" "app" {
  name            = "${var.project_name}-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 1
  launch_type     = "FARGATE"
  network_configuration {
    subnets          = module.vpc.public_subnets
    security_groups  = [aws_security_group.ecs_task.id]
    assign_public_ip = true
  }
}
Apply Terraform and verify metrics in AMP

Time to provision everything and confirm the end-to-end pipeline is working. You will apply the Terraform changes, check that the ECS task reaches RUNNING status, and then query AMP directly to confirm metrics are arriving.

  • Apply your Terraform changes by running:
terraform apply -auto-approve

What does this command do?

This provisions the AMP workspace, IAM roles, SSM parameter, security group, ECS task definition, and ECS service all at once. Terraform creates resources in dependency order automatically. This may take 2-3 minutes.

  • Wait for the apply to complete, then verify your ECS task reached RUNNING status by running:
aws ecs list-tasks --cluster saas-monitor-cluster --query "taskArns[0]" --output text
  • Copy the task ARN from the output, then check its status by running:
aws ecs describe-tasks --cluster saas-monitor-cluster --tasks [[TASK_ARN="your task ARN from the previous command"]] --query "tasks[0].lastStatus" --output text
  • You should see RUNNING in the output. If you see PENDING, wait 30 seconds and run the command again.

Why might the task stay PENDING?

Fargate tasks transition from PROVISIONING to PENDING to RUNNING. The image pull from ECR takes a few seconds. If it stays in PENDING for more than a minute, check that your ECR image was pushed successfully in the previous step.

  • Install awscurl so you can query AMP with signed requests:
pip install awscurl
  • Get your AMP workspace ID by running:
aws amp list-workspaces --query "workspaces[?alias==\`saas-monitor-amp\`].workspaceId" --output text
  • Query AMP to confirm metrics are arriving. Replace the workspace ID in the endpoint URL:
awscurl --service="aps" --region=us-east-1 "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/[[AMP_WORKSPACE_ID="your workspace ID from the previous command"]]/api/v1/query?query=flask_http_request_total"
  • You should see a JSON response containing "status":"success" and metric data with flask_http_request_total samples.

What does the awscurl response tell you?

A successful response with metric samples proves the entire pipeline is working. Your Flask app is generating Prometheus metrics, the ADOT sidecar is scraping them every 15 seconds, and they are being written to AMP.

If the result array is empty, wait 30-60 seconds for the first scrape cycle to complete and try again.

Getting an empty result or connection error?

  • If the response shows an empty result array, the ADOT collector may not have completed its first scrape yet. Wait 60 seconds and retry.
  • If you get an authentication error, verify your AWS CLI credentials are configured for the same region as your AMP workspace.
  • If the ECS task stopped, check the stopped reason with aws ecs describe-tasks and look for image pull failures or missing permissions.

help me debug why my ADOT sidecar is not sending metrics to AMP

Your metrics pipeline is live. The Flask app generates RED metrics, the ADOT sidecar scrapes and ships them, and AMP stores them. Next up, you will visualize this data in Grafana dashboards using the Golden Signals methodology.

Build Golden Signals Dashboards in Amazon Managed Grafana

Your ECS task is running and the ADOT sidecar is shipping metrics into Amazon Managed Prometheus. Metrics sitting in storage are only useful if you can see them.

In this step, you will provision Amazon Managed Grafana (AMG) and build a Golden Signals dashboard that visualizes the four signals every SRE team watches: Rate, Errors, Duration, and Saturation.

In this step, get ready to:
  • Provision an AMG workspace via Terraform with IAM Identity Center authentication.
  • Connect your AMP workspace as a Prometheus data source in Grafana.
  • Build a four-panel Golden Signals dashboard using PromQL.
Provision the AMG workspace with Terraform

AMG needs an IAM role so it can query your AMP workspace on your behalf. You will create the role first, then the workspace resource that references it.

  • In main.tf, add the Grafana IAM role and policy attachment at the end of the file by pasting this code:
# IAM role that allows the AMG workspace to query AMP
resource "aws_iam_role" "grafana_role" {
  name = "${var.project_name}-grafana-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "grafana.amazonaws.com" }
    }]
  })
}

# Grants the Grafana role permission to read metrics from AMP
resource "aws_iam_role_policy_attachment" "grafana_prometheus" {
  role       = aws_iam_role.grafana_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess"
}

What does this code do?

The grafana_role resource creates an IAM role that the AMG service (grafana.amazonaws.com) can assume.

The AmazonPrometheusQueryAccess managed policy gives that role permission to run PromQL queries against your AMP workspace.

  • Below the policy attachment, add the AMG workspace resource:
# The Amazon Managed Grafana workspace itself
resource "aws_grafana_workspace" "main" {
  name                     = "${var.project_name}-amg"
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["AWS_SSO"]
  permission_type          = "SERVICE_MANAGED"
  data_sources             = ["PROMETHEUS"]
  role_arn                 = aws_iam_role.grafana_role.arn
}

What does this code do?

This creates an AMG workspace that uses IAM Identity Center (formerly AWS SSO) for user authentication. SERVICE_MANAGED means AWS automatically provisions the IAM policies needed for data source access.

The data_sources = ["PROMETHEUS"] setting pre-configures permissions for Prometheus data sources. The role_arn links the workspace to the Grafana IAM role you just created.

AMG requires IAM Identity Center to be enabled in your account before Terraform can create the workspace.

  • In the AWS Console, search for IAM Identity Center and open the service.

✔️ I see the IAM Identity Center dashboard

IAM Identity Center is already enabled. You are ready to proceed.

ⓧ I see an Enable button

  • Click Enable.
  • If prompted to create an AWS Organization, accept the defaults.
  • Wait until the dashboard loads, confirming IAM Identity Center is active.

Why does AMG need IAM Identity Center?

AMG uses IAM Identity Center to manage which users can sign in to the Grafana workspace. This is a one-time enablement. It does not change how your other AWS services authenticate.

  • Save main.tf.
  • Apply the new Terraform resources by running:
terraform apply -auto-approve
  • Wait for the apply to complete. Terraform will output the new resources created.

What should I see?

Terraform reports 3 resources added (the IAM role, the policy attachment, and the AMG workspace). The workspace creation may take 1-2 minutes.

Seeing an IAM Identity Center error?

If Terraform returns an error about IAM Identity Center not being enabled, confirm you completed the enablement step above and that the dashboard shows an active state. Re-run terraform apply -auto-approve after enabling it.

help me fix this IAM Identity Center Terraform error for my AMG workspace

Now you need to assign yourself as an admin user in the workspace so you can sign in to Grafana.

  • In the AWS Console, search for Amazon Managed Grafana and open the service.
  • Click on your workspace name (saas-monitor-amg).
  • Select the Authentication tab.
  • Click Assign new user or group.
  • Select the checkbox next to your IAM Identity Center user, then click Assign user.
  • Select the checkbox next to your user again, then choose Make admin from the Actions dropdown.

No users listed?

If you do not see any users, you need to create one in IAM Identity Center first. Go to the IAM Identity Center console, choose Users, click Add user, fill in your email and name, then return to the AMG workspace to assign them.

  • Back in the workspace overview, click the Grafana workspace URL link to open Grafana.
  • Sign in with your IAM Identity Center credentials.

You should see the Grafana home screen with a welcome message and navigation sidebar.

Can't sign in to Grafana?

Make sure you confirmed the IAM Identity Center email invitation and set a password. Also confirm you assigned yourself as Admin (not just Viewer) in the workspace authentication settings.

help me troubleshoot signing in to my Amazon Managed Grafana workspace

Add AMP as a Prometheus data source

Grafana needs to know where your metrics live. You will connect your AMP workspace as a data source so Grafana can query it using PromQL.

  • In the Grafana left sidebar, click the AWS icon (the AWS logo).
  • Select AWS services, then Prometheus.
  • In the Region dropdown, select the region where you deployed your stack (us-east-1 if you used the default).
  • Your AMP workspace should appear in the list. Select it and click Add data source.

Why use the AWS data source configuration?

This method automatically discovers your AMP workspaces and configures SigV4 authentication. You do not need to manually paste the AMP endpoint URL or configure credentials.

  • Verify the data source is working by navigating to Explore in the left sidebar (the compass icon).
  • Make sure your new AMP data source is selected in the data source dropdown at the top.
  • In the query editor, type the following PromQL query and press Shift+Enter or click Run query:
up{}

You should see at least one result with a value of 1, confirming that Grafana can reach your AMP workspace and that your Flask app target is being scraped successfully.

Query returns no data?

Check that your ECS task is still in RUNNING status. The ADOT sidecar must be scraping and writing metrics for data to appear. Also confirm you selected the correct region in the data source configuration.

help me debug why my Grafana Explore query returns no data from AMP

Create the Golden Signals dashboard

The Golden Signals are four metrics that Google's SRE book identifies as the most important indicators of service health: Rate, Errors, Duration, and Saturation. You will create a dashboard with one panel for each signal.

  • In the Grafana left sidebar, click Dashboards (the four-squares icon).
  • Click New, then New Dashboard.
  • Click Add visualization.
  • Select your AMP data source from the list.

You are now in the panel editor. You will create the first panel for Request Rate.

  • In the query editor at the bottom, paste this PromQL query:
sum(rate(flask_http_request_total[5m]))
  • In the Panel title field on the right side, type Rate - Requests/sec.
  • Click Apply in the top right to save the panel.

What does this query measure?

The rate() function calculates the per-second increase of the flask_http_request_total counter over a 5-minute window. The sum() aggregates across all endpoints and status codes, giving you the total request throughput.

  • Click Add (the + icon at the top), then Visualization to create the second panel.
  • Select your AMP data source.
  • Paste this PromQL query for the error rate percentage:
sum(rate(flask_http_request_total{status=~"5.."}[5m])) / sum(rate(flask_http_request_total[5m])) * 100
  • Set the panel title to Errors - Error Rate %.
  • Click Apply.

What does this query measure?

The numerator counts only requests with 5xx status codes (server errors). Dividing by total requests and multiplying by 100 gives you the error rate as a percentage. Your baseline should show approximately 5% since the app returns 500 on 5% of requests by default.

  • Add a third visualization panel with your AMP data source.
  • Paste this PromQL query for p95 latency:
histogram_quantile(0.95, sum(rate(flask_http_request_duration_seconds_bucket[5m])) by (le))
  • Set the panel title to Duration - p95 Latency.
  • Click Apply.

What does this query measure?

The histogram_quantile(0.95, ...) function returns the latency value below which 95% of requests complete. This is more useful than average latency because it reveals the experience of your slowest users.

  • Add a fourth visualization panel with your AMP data source.
  • Paste this PromQL query for CPU saturation:
ecs_task_cpu_utilized / ecs_task_cpu_reserved * 100
  • Set the panel title to Saturation - CPU %.
  • Click Apply.

What does this query measure?

This divides the actual CPU used by your ECS task by the CPU reserved for it, expressed as a percentage. A value near 100% means the task is CPU-bound and may need more resources.

  • Click the Save dashboard icon (floppy disk) in the top right.
  • Name the dashboard Golden Signals and click Save.

All four panels should now be rendering data. The Rate panel shows requests per second, the Errors panel shows your baseline error rate around 5%, the Duration panel shows p95 latency, and the Saturation panel shows CPU utilization.

Some panels show No data?

The Saturation panel (ecs_task_cpu_utilized) requires the ADOT awsecscontainermetrics receiver to be active. Wait a few minutes for ECS container metrics to appear.

For the Rate and Errors panels, your app needs to be receiving traffic. Open the Flask app's public IP in a browser at /api/orders a few times to generate some requests.

help me debug why my Grafana dashboard panels show no data

✔️ Awesome, I've got everything!

Great. Make sure you have saved your dashboard in Grafana and your main.tf file.

ⓧ I'd like to double check the full code

Here is the complete main.tf file as it should look at the end of this step:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  backend "local" {}
}

provider "aws" {
  region = var.region
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"
  name    = "${var.project_name}-vpc"
  cidr    = "10.0.0.0/16"
  azs             = ["${var.region}a", "${var.region}b"]
  public_subnets  = ["10.0.1.0/24", "10.0.2.0/24"]
  enable_nat_gateway = false
}

resource "aws_ecr_repository" "app" {
  name = "${var.project_name}-app"
}

resource "aws_ecs_cluster" "main" {
  name = "${var.project_name}-cluster"
}

resource "aws_prometheus_workspace" "main" {
  alias = "${var.project_name}-amp"
}

resource "aws_iam_role" "ecs_task_role" {
  name = "${var.project_name}-task-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "task_prometheus" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess"
}

resource "aws_iam_role" "ecs_execution_role" {
  name = "${var.project_name}-execution-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "execution_policy" {
  role       = aws_iam_role.ecs_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_ssm_parameter" "adot_config" {
  name  = "/${var.project_name}/adot-config"
  type  = "String"
  value = <<-EOT
    extensions:
      health_check:
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'flask-app'
              scrape_interval: 15s
              static_configs:
                - targets: ['localhost:5000']
      awsecscontainermetrics:
    exporters:
      prometheusremotewrite:
        endpoint: "${aws_prometheus_workspace.main.prometheus_endpoint}api/v1/remote_write"
        auth:
          authenticator: sigv4auth
    extensions:
      sigv4auth:
        region: "${var.region}"
        service: "aps"
    service:
      extensions: [health_check, sigv4auth]
      pipelines:
        metrics:
          receivers: [prometheus, awsecscontainermetrics]
          exporters: [prometheusremotewrite]
  EOT
}

resource "aws_security_group" "ecs_task" {
  name   = "${var.project_name}-ecs-sg"
  vpc_id = module.vpc.vpc_id
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_ecs_task_definition" "app" {
  family                   = "${var.project_name}-task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "256"
  memory                   = "512"
  task_role_arn            = aws_iam_role.ecs_task_role.arn
  execution_role_arn       = aws_iam_role.ecs_execution_role.arn
  container_definitions = jsonencode([
    {
      name      = "flask-app"
      image     = "${aws_ecr_repository.app.repository_url}:latest"
      cpu       = 128
      memory    = 256
      portMappings = [{ containerPort = 5000 }]
    },
    {
      name      = "adot-collector"
      image     = "public.ecr.aws/aws-observability/aws-otel-collector:v0.48.0"
      cpu       = 128
      memory    = 256
      environment = [{
        name  = "AOT_CONFIG_CONTENT"
        value = aws_ssm_parameter.adot_config.value
      }]
    }
  ])
}

resource "aws_ecs_service" "app" {
  name            = "${var.project_name}-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 1
  launch_type     = "FARGATE"
  network_configuration {
    subnets          = module.vpc.public_subnets
    security_groups  = [aws_security_group.ecs_task.id]
    assign_public_ip = true
  }
}

resource "aws_iam_role" "grafana_role" {
  name = "${var.project_name}-grafana-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "grafana.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "grafana_prometheus" {
  role       = aws_iam_role.grafana_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess"
}

resource "aws_grafana_workspace" "main" {
  name                     = "${var.project_name}-amg"
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["AWS_SSO"]
  permission_type          = "SERVICE_MANAGED"
  data_sources             = ["PROMETHEUS"]
  role_arn                 = aws_iam_role.grafana_role.arn
}

Your monitoring pipeline now has a visual layer. You can see your service's health at a glance. Next up, you will configure alert rules that automatically notify you when these signals cross dangerous thresholds.

Configure Alert Rules and SNS Notification Routing

Your Golden Signals dashboard is live, but dashboards only help when someone is watching them. In a real production environment, you need the system to notify you automatically when something breaks.

In this step, you will define PromQL alert rules that continuously evaluate your metrics. When a threshold is breached, Alertmanager routes the notification through Amazon SNS to your email inbox.

In this step, get ready to:
  • Create an SNS topic and email subscription for alert delivery.
  • Define PromQL alert rules for high error rate and high latency.
  • Configure Alertmanager routing with severity-based escalation.
Create the SNS topic and alert rules in Terraform

Before your monitoring system can send notifications, it needs a destination. An SNS topic acts as a message bus between AWS services and subscribers like email addresses.

  • In variables.tf, add a new variable for your alert email address by copying this code at the bottom of the file:
variable "alert_email" {
  description = "Email address for alert notifications"
}

What does this variable do?

This variable has no default value, which means Terraform will prompt you to enter your email address when you run terraform apply. This keeps your personal email out of version-controlled files.

  • Save variables.tf.
  • In main.tf, add the SNS topic and email subscription resources at the bottom of the file, after the aws_iam_role_policy_attachment.grafana_prometheus resource:
resource "aws_sns_topic" "alerts" {
  name = "${var.project_name}-alerts"
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

What does this code do?

The aws_sns_topic creates a named topic that acts as a message channel. The aws_sns_topic_subscription attaches your email address to that topic.

When an alert fires, AMP pushes a message to the topic, and SNS forwards it to every confirmed subscriber.

  • Save main.tf.

Now you need to tell AMP what conditions should trigger an alert. You do this by defining alert rules using PromQL expressions.

  • Still in main.tf, add the rule group namespace resource directly below the SNS subscription you just added:
resource "aws_prometheus_rule_group_namespace" "alerts" {
  name         = "${var.project_name}-rules"
  workspace_id = aws_prometheus_workspace.main.id
  data         = <<-EOT
    groups:
      - name: saas-alerts
        rules:
          - alert: HighErrorRate
            expr: sum(rate(flask_http_request_total{status=~"5.."}[2m])) / sum(rate(flask_http_request_total[2m])) * 100 > 10
            for: 2m
            labels:
              severity: critical
            annotations:
              summary: "High error rate detected"
          - alert: HighLatency
            expr: histogram_quantile(0.95, sum(rate(flask_http_request_duration_seconds_bucket[5m])) by (le)) > 1
            for: 3m
            labels:
              severity: warning
            annotations:
              summary: "High p95 latency detected"
  EOT
}

What do these alert rules do?

HighErrorRate calculates the percentage of 5xx responses over a 2-minute window. If it exceeds 10% for 2 consecutive minutes, the alert fires with severity critical. The threshold is set above the app's baseline ~5% error rate so it only triggers during genuine degradation.

HighLatency measures the 95th percentile response time. If p95 latency exceeds 1 second for 3 minutes, it fires with severity warning. This catches gradual performance degradation before users notice.

  • Save main.tf.

Seeing YAML indentation errors?

The data field uses a heredoc (<<-EOT). Make sure the YAML inside is indented consistently with spaces, not tabs. The closing EOT must be on its own line with only leading spaces.

Help me fix YAML indentation in my Terraform heredoc

Configure the Alertmanager routing definition

Alert rules define when to fire. Alertmanager defines where to send the notification and how to group related alerts together. Routing by severity ensures critical issues get faster, louder notifications.

  • In main.tf, add the Alertmanager definition resource below the rule group namespace you just added:
resource "aws_prometheus_alert_manager_definition" "main" {
  workspace_id = aws_prometheus_workspace.main.id
  definition   = <<-EOT
    alertmanager_config: |
      route:
        group_by: ['alertname', 'severity']
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 10m
        receiver: 'warning-receiver'
        routes:
          - match:
              severity: critical
            receiver: 'critical-receiver'
            group_wait: 10s
            repeat_interval: 5m
      receivers:
        - name: 'critical-receiver'
          sns_configs:
            - topic_arn: '${aws_sns_topic.alerts.arn}'
              sigv4:
                region: '${var.region}'
        - name: 'warning-receiver'
          sns_configs:
            - topic_arn: '${aws_sns_topic.alerts.arn}'
              sigv4:
                region: '${var.region}'
      inhibit_rules:
        - source_match:
            severity: critical
          target_match:
            severity: warning
          equal: ['alertname']
  EOT
}

What does this routing configuration do?

The default route catches warning-level alerts with a 30-second group wait and 10-minute repeat interval. A child route matches severity: critical alerts and routes them with a faster 10-second group wait and 5-minute repeat.

The inhibit_rules section prevents alert fatigue. When a critical alert fires for a given alertname, any warning-level alert with the same name is silenced. This way you only receive the highest-severity notification.

Both receivers point to the same SNS topic here, but in a production setup you could route critical alerts to a PagerDuty integration and warnings to a Slack channel.

  • Save main.tf.

✔️ Awesome, I've got everything!

Great. Double check you have saved both variables.tf and main.tf.

ⓧ I'd like to double check the full code

Here is the complete variables.tf at this point:

variable "region" {
  default = "us-east-1"
}

variable "project_name" {
  default = "saas-monitor"
}

variable "alert_email" {
  description = "Email address for alert notifications"
}

Here is the complete main.tf at this point:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  backend "local" {}
}

provider "aws" {
  region = var.region
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"
  name    = "${var.project_name}-vpc"
  cidr    = "10.0.0.0/16"
  azs             = ["${var.region}a", "${var.region}b"]
  public_subnets  = ["10.0.1.0/24", "10.0.2.0/24"]
  enable_nat_gateway = false
}

resource "aws_ecr_repository" "app" {
  name = "${var.project_name}-app"
}

resource "aws_ecs_cluster" "main" {
  name = "${var.project_name}-cluster"
}

resource "aws_prometheus_workspace" "main" {
  alias = "${var.project_name}-amp"
}

resource "aws_iam_role" "ecs_task_role" {
  name = "${var.project_name}-task-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "task_prometheus" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess"
}

resource "aws_iam_role" "ecs_execution_role" {
  name = "${var.project_name}-execution-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "execution_policy" {
  role       = aws_iam_role.ecs_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_ssm_parameter" "adot_config" {
  name  = "/${var.project_name}/adot-config"
  type  = "String"
  value = <<-EOT
    extensions:
      health_check:
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'flask-app'
              scrape_interval: 15s
              static_configs:
                - targets: ['localhost:5000']
      awsecscontainermetrics:
    exporters:
      prometheusremotewrite:
        endpoint: "${aws_prometheus_workspace.main.prometheus_endpoint}api/v1/remote_write"
        auth:
          authenticator: sigv4auth
    extensions:
      sigv4auth:
        region: "${var.region}"
        service: "aps"
    service:
      extensions: [health_check, sigv4auth]
      pipelines:
        metrics:
          receivers: [prometheus, awsecscontainermetrics]
          exporters: [prometheusremotewrite]
  EOT
}

resource "aws_security_group" "ecs_task" {
  name   = "${var.project_name}-ecs-sg"
  vpc_id = module.vpc.vpc_id
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_ecs_task_definition" "app" {
  family                   = "${var.project_name}-task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "256"
  memory                   = "512"
  task_role_arn            = aws_iam_role.ecs_task_role.arn
  execution_role_arn       = aws_iam_role.ecs_execution_role.arn
  container_definitions = jsonencode([
    {
      name      = "flask-app"
      image     = "${aws_ecr_repository.app.repository_url}:latest"
      cpu       = 128
      memory    = 256
      portMappings = [{ containerPort = 5000 }]
    },
    {
      name      = "adot-collector"
      image     = "public.ecr.aws/aws-observability/aws-otel-collector:v0.48.0"
      cpu       = 128
      memory    = 256
      environment = [{
        name  = "AOT_CONFIG_CONTENT"
        value = aws_ssm_parameter.adot_config.value
      }]
    }
  ])
}

resource "aws_ecs_service" "app" {
  name            = "${var.project_name}-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 1
  launch_type     = "FARGATE"
  network_configuration {
    subnets          = module.vpc.public_subnets
    security_groups  = [aws_security_group.ecs_task.id]
    assign_public_ip = true
  }
}

resource "aws_grafana_workspace" "main" {
  name                     = "${var.project_name}-amg"
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["AWS_SSO"]
  permission_type          = "SERVICE_MANAGED"
  data_sources             = ["PROMETHEUS"]
  role_arn                 = aws_iam_role.grafana_role.arn
}

resource "aws_iam_role" "grafana_role" {
  name = "${var.project_name}-grafana-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "grafana.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "grafana_prometheus" {
  role       = aws_iam_role.grafana_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess"
}

resource "aws_sns_topic" "alerts" {
  name = "${var.project_name}-alerts"
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

resource "aws_prometheus_rule_group_namespace" "alerts" {
  name         = "${var.project_name}-rules"
  workspace_id = aws_prometheus_workspace.main.id
  data         = <<-EOT
    groups:
      - name: saas-alerts
        rules:
          - alert: HighErrorRate
            expr: sum(rate(flask_http_request_total{status=~"5.."}[2m])) / sum(rate(flask_http_request_total[2m])) * 100 > 10
            for: 2m
            labels:
              severity: critical
            annotations:
              summary: "High error rate detected"
          - alert: HighLatency
            expr: histogram_quantile(0.95, sum(rate(flask_http_request_duration_seconds_bucket[5m])) by (le)) > 1
            for: 3m
            labels:
              severity: warning
            annotations:
              summary: "High p95 latency detected"
  EOT
}

resource "aws_prometheus_alert_manager_definition" "main" {
  workspace_id = aws_prometheus_workspace.main.id
  definition   = <<-EOT
    alertmanager_config: |
      route:
        group_by: ['alertname', 'severity']
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 10m
        receiver: 'warning-receiver'
        routes:
          - match:
              severity: critical
            receiver: 'critical-receiver'
            group_wait: 10s
            repeat_interval: 5m
      receivers:
        - name: 'critical-receiver'
          sns_configs:
            - topic_arn: '${aws_sns_topic.alerts.arn}'
              sigv4:
                region: '${var.region}'
        - name: 'warning-receiver'
          sns_configs:
            - topic_arn: '${aws_sns_topic.alerts.arn}'
              sigv4:
                region: '${var.region}'
      inhibit_rules:
        - source_match:
            severity: critical
          target_match:
            severity: warning
          equal: ['alertname']
  EOT
}
Apply Terraform and verify in Grafana
  • Apply the new resources by running:
terraform apply

What happens during this apply?

Terraform will prompt you to enter a value for alert_email. Type the email address where you want to receive alert notifications, then type yes to confirm the plan.

Terraform creates four new resources: the SNS topic, the email subscription, the rule group namespace, and the Alertmanager definition.

  • Check your email inbox for a message from AWS Notifications with the subject "AWS Notification - Subscription Confirmation".
  • Click the Confirm subscription link in the email.

Check your spam folder

The confirmation email sometimes lands in spam or promotions. If you do not see it within 2 minutes, check those folders. The subscription will not deliver alerts until you confirm it.

Now verify that the alert rules loaded correctly in Grafana.

  • Switch back to your Grafana workspace in the browser.
  • In the left sidebar, click the Alerting icon (bell icon).
  • Click Alert rules.

You should see both HighErrorRate and HighLatency listed with a state of Normal or Inactive. This is correct because the baseline error rate (~5%) is below the 10% threshold.

Alert rules not appearing in Grafana?

If the Alerting view is empty, wait 1-2 minutes and refresh. AMP needs time to load the rule group after creation. If they still do not appear, check that your AMP data source is still connected by navigating to Connections then Data Sources and running a test query.

Help me troubleshoot why my AMP alert rules are not showing in Grafana

Your monitoring pipeline can now detect and notify you about problems automatically. Next up, you will generate real traffic and simulate a production incident to watch the entire alerting pipeline fire end-to-end.

Load Test and Respond to an Incident

Your monitoring pipeline is wired up and ready to prove itself. Dashboards are showing data, alert rules are loaded, and SNS is waiting to deliver notifications.

Now it is time to put the system under realistic load and simulate a production incident. You will generate sustained traffic, trigger a failure, watch your alerts fire, investigate with PromQL, and resolve the issue. This is the SRE incident lifecycle that on-call engineers practice every day: Detection, Notification, Investigation, Resolution.

In this step, get ready to:
  • Write and run a Python load generation script against your ECS service.
  • Trigger an incident and watch alerts fire with an SNS notification.
  • Investigate with PromQL and resolve the incident end-to-end.
Find your ECS task's public IP and open the security group

Before you can send traffic to your Flask app from your local machine, you need two things: the task's public IP address, and an inbound security group rule that allows traffic on port 5000.

  • Get the task ARN by running this command:
aws ecs list-tasks --cluster saas-monitor-cluster --query "taskArns[0]" --output text
  • Copy the task ARN from the output. Use it to find the ENI (elastic network interface) ID by running:
aws ecs describe-tasks --cluster saas-monitor-cluster --tasks [[TASK_ARN="your task ARN from the previous command"]] --query "tasks[0].attachments[0].details[?name=='networkInterfaceId'].value" --output text
  • Copy the ENI ID from the output (it starts with eni-). Retrieve the public IP by running:
aws ec2 describe-network-interfaces --network-interface-ids [[ENI_ID="your ENI ID starting with eni-"]] --query "NetworkInterfaces[0].Association.PublicIp" --output text
  • Note the public IP address from the output. You will use it in the load test script.
  • Open the AWS Console and navigate to VPC > Security Groups.
  • Find the security group named saas-monitor-ecs-sg and click its ID.
  • Click Edit inbound rules.
  • Click Add rule.
  • Set Type to Custom TCP, Port range to 5000, and Source to My IP.
  • Click Save rules.

Why add an inbound rule?

Your ECS task has a public IP, but the security group only allows outbound traffic by default. Without an inbound rule on port 5000, your load test requests from your local machine would be blocked before reaching the Flask app.

  • Verify connectivity by running a quick test from your terminal:
curl http://[[ECS_PUBLIC_IP="your ECS task public IP"]]:5000/api/health

You should see {"status": "ok"} in the output.

Not getting a response?

  • Confirm the security group inbound rule saved correctly. Check that port 5000 is open and the source is set to your IP.
  • Confirm the ECS task is still in RUNNING status. If it restarted, the public IP may have changed.
  • Make sure you are using the public IP (not the private IP starting with 10.0).

Can't reach my ECS task on port 5000

Write and run the load test script

A monitoring pipeline is only useful when there is traffic flowing through it. You need sustained, concurrent requests to make the Grafana dashboard come alive and create a baseline for your alert rules to measure against.

  • Install the requests library locally by running:
pip install requests
  • Create a new file called load_test.py in your project folder. Add the following code:
import requests
import concurrent.futures
import time

# Target your ECS task's public IP on port 5000
BASE_URL = "http://[[ECS_PUBLIC_IP="your ECS task public IP"]]:5000"
ENDPOINTS = ["/api/orders", "/api/users"]

def send_request(endpoint):
    # Send a GET request and silently handle failures
    try:
        requests.get(f"{BASE_URL}{endpoint}", timeout=5)
    except Exception:
        pass

def run_load_test():
    # Use 10 concurrent workers to simulate sustained traffic
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        while True:
            futures = [executor.submit(send_request, ep) for ep in ENDPOINTS * 5]
            concurrent.futures.wait(futures)

if __name__ == "__main__":
    run_load_test()

What does this code do?

  • A ThreadPoolExecutor with 10 workers sends requests in parallel, simulating multiple users hitting the API simultaneously.
  • Each loop iteration submits 10 requests (5 copies of each endpoint) and waits for them to complete before sending the next batch.
  • The while True loop runs indefinitely so you have sustained traffic while you trigger and investigate the incident. You will stop it manually with Ctrl+C when finished.
  • Save load_test.py.
  • Start the load test by running:
python load_test.py

The script runs continuously with no output unless there is an error. Leave it running and switch to your Grafana dashboard.

  • Return to your Golden Signals dashboard in Grafana.
  • Watch for 1-2 minutes. The Rate panel should climb as requests flow in. The Duration panel should show p95 latency values. The Errors panel should settle around the 5% baseline.

Dashboard not showing traffic?

  • Make sure load_test.py is still running in your terminal (no traceback or error output).
  • Confirm the BASE_URL in your script matches your ECS task's current public IP.
  • Check that the ADOT sidecar is scraping metrics by looking at the ADOT container logs in the ECS console. Look for successful remote_write messages.
  • In Grafana, set the dashboard time range to Last 5 minutes and enable auto-refresh.

My Grafana dashboard is not showing load test traffic

Trigger the incident and receive the alert

With baseline traffic established, it is time to simulate a production failure. Calling the /simulate/incident endpoint spikes the error rate from 5% to 50%. Your HighErrorRate alert rule fires when errors exceed 10% for 2 minutes.

  • Open a new terminal tab (keep the load test running in the first one).
  • Trigger the incident by running:
curl http://[[ECS_PUBLIC_IP="your ECS task public IP"]]:5000/simulate/incident

You should see {"incident": true} confirming the error rate is now spiked to 50%.

  • Switch to your Grafana dashboard. Within 30 seconds, the Errors panel should spike dramatically from ~5% to ~50%.
  • Wait approximately 2 minutes. The HighErrorRate alert needs to see errors above 10% for 2 consecutive minutes before firing.
  • In Grafana, navigate to Alerting in the left sidebar. Watch the HighErrorRate alert transition from Normal to Pending to Firing.
  • Check your email inbox. You should receive an SNS notification with the alert details. This is Detection and Notification complete.

What is the alert lifecycle?

Prometheus alerts have three states. Inactive means the condition is not met. Pending means the condition is met but the for duration has not elapsed yet. Firing means the condition has been met continuously for the full for period and notifications are sent.

The for: 2m clause prevents brief traffic spikes from triggering false alarms. Only sustained degradation fires the alert.

Alert not firing after 2 minutes?

  • Confirm the load test is still running. Without sustained traffic, there are not enough data points for the rate() function to calculate a meaningful error percentage.
  • Verify the incident endpoint responded with {"incident": true}. If it returned an error, the app may have restarted.
  • Check the Errors panel in your dashboard. If it is not showing 40-50%, the incident mode may not have activated correctly.

My HighErrorRate alert is stuck in Pending and won't fire

Investigate with PromQL and resolve the incident

Detection and notification are only half the story. An SRE must investigate to confirm the root cause, then resolve the issue. Grafana's Explore tab lets you run ad-hoc PromQL queries against your AMP data.

  • In Grafana, click Explore in the left sidebar (the compass icon).
  • Make sure your AMP data source is selected at the top.
  • Enter this PromQL query to see the error rate spike:
rate(flask_http_request_total{status=~"5.."}[1m])
  • Click Run query. You should see a sharp increase in 5xx responses correlating with when you called the incident endpoint.
  • Now check if latency correlated with the error spike. Enter this query:
histogram_quantile(0.95, sum(rate(flask_http_request_duration_seconds_bucket[5m])) by (le))
  • Click Run query. This shows your p95 latency over time. You can confirm whether the failures also increased response times.

What are these PromQL queries doing?

  • The first query calculates the per-second rate of 5xx responses over a 1-minute window. A narrow window like [1m] gives you fine-grained visibility into exactly when the spike occurred.
  • The second query uses histogram_quantile to calculate the 95th percentile latency from the histogram buckets. This tells you whether 95% of requests completed within an acceptable time.
  • Together, these two queries answer the core SRE questions: "What is failing?" and "Is performance also degraded?"

Now that you have confirmed the root cause (elevated error rate, correlating with the simulated incident), it is time to resolve it.

  • Resolve the incident by running this in your second terminal:
curl http://[[ECS_PUBLIC_IP="your ECS task public IP"]]:5000/simulate/resolve

You should see {"incident": false} confirming the error rate returns to the 5% baseline.

  • Switch back to your Grafana dashboard. The Errors panel should drop back to ~5% within 30 seconds.
  • Navigate to Alerting in the left sidebar. Watch the HighErrorRate alert transition from Firing back to Normal/Inactive.
  • Stop the load test in your first terminal by pressing Ctrl+C.

What just happened?

You completed the full SRE incident lifecycle. Detection happened when the HighErrorRate alert transitioned to Firing. Notification came via the SNS email. Investigation used PromQL in Grafana Explore to confirm the error spike and check for latency correlation. Resolution was calling the resolve endpoint and confirming the system recovered.

In a real production environment, each of these phases would be logged in an incident management tool and followed by a post-incident review. The patterns are identical.

✔️ Awesome, I've got everything!

Great work. Double check you have saved load_test.py and that your load test has been stopped with Ctrl+C.

ⓧ I'd like to double check the full code

Here is the complete load_test.py file:

import requests
import concurrent.futures
import time

BASE_URL = "http://<ECS_PUBLIC_IP>:5000"
ENDPOINTS = ["/api/orders", "/api/users"]

def send_request(endpoint):
    try:
        requests.get(f"{BASE_URL}{endpoint}", timeout=5)
    except Exception:
        pass

def run_load_test():
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        while True:
            futures = [executor.submit(send_request, ep) for ep in ENDPOINTS * 5]
            concurrent.futures.wait(futures)

if __name__ == "__main__":
    run_load_test()

You just exercised the complete monitoring pipeline from end to end. You proved that your instrumentation, collection, storage, visualization, alerting, and notification layers all work together. Next up, take on the Secret Mission to implement SLO-based burn-rate alerts that reduce alert fatigue while catching real incidents faster.

Secret mission

Add SLO Burn-Rate Alerts and a Dead Man's Switch

Your static threshold alerts work, but production SRE teams use a more sophisticated approach. In this secret mission, you will implement multi-window burn-rate SLO alerting (the technique recommended by the Google SRE Workbook) and add meta-monitoring that watches the monitoring system itself.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The ECS Fargate task costs approximately $0.25/day while running, and the AMG workspace will begin charging $9/editor/month after the 90-day free trial ends.

Cost warning

Your ECS Fargate task is actively running and incurring charges (~$0.01/hour). If you are not actively using the monitoring pipeline, pause or delete your resources to avoid unnecessary costs.

AMP stays within free tier (40M samples/month) for this project's scale. AMG is free for 90 days but charges $9/editor/month after that.

Resources you used:

  • ECS Fargate task (running the Flask app and ADOT sidecar).
  • AMP workspace (storing Prometheus metrics).
  • AMG workspace (Grafana dashboards and alerting views).
  • SNS topics (alerts and watchdog notifications).
  • ECR repository (Docker image storage).
  • VPC, subnets, security group, and IAM roles.
  • SSM Parameter (ADOT collector configuration).
  • CloudWatch Log Groups (created automatically by ECS tasks).
  • Local project files on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively exploring the Grafana dashboards, running PromQL queries, or experimenting with load tests.

  • Your ECS task continues running and sending metrics to AMP (~$0.25/day).
  • AMP ingestion stays within the free tier at this project's traffic volume.
  • AMG is free for 90 days from when you created the workspace.
  • SNS email notifications remain active within the free tier (1,000 emails/month).

Pause - I'll come back to this later

Shut down the running ECS task to stop incurring Fargate charges. Your Terraform configuration, AMP workspace, and AMG workspace remain intact so you can resume later.

  • Open main.tf and find the aws_ecs_service resource.
  • Change desired_count = 1 to desired_count = 0.
  • Run terraform apply and type yes to confirm.
  • This stops the Fargate task (no more compute charges) while preserving AMP metrics history and your Grafana dashboards.
  • To resume later, set desired_count back to 1 and run terraform apply again.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your AMP metrics, Grafana dashboards, alert rules, and all AWS infrastructure created by Terraform.

  • Stop the load test script if it is still running (press Ctrl+C in the terminal).
  • Make sure your terminal is in the project directory (saas-monitor on your Desktop).
  • Destroy all AWS resources by running:
terraform destroy
  • Terraform will show a plan listing all resources to be destroyed. Type yes when prompted to confirm.
  • After the destroy completes, check for any CloudWatch Log Groups that ECS created automatically. List them by running:
aws logs describe-log-groups --log-group-name-prefix /ecs/saas-monitor --query 'logGroups[].logGroupName' --output text
  • If any log groups appear in the output, delete each one by running:
aws logs delete-log-group --log-group-name /ecs/saas-monitor-task
  • Replace the log group name with the actual name from the previous command's output.
  • Delete the local project files from your Desktop.

macOS

rm -rf ~/Desktop/saas-monitor

Windows

rmdir /s /q %USERPROFILE%\Desktop\saas-monitor

What about IAM Identity Center?

IAM Identity Center (which AMG required) is a free service and remains enabled in your account. It does not incur any charges, so there is no need to disable it.

Nice Work!

Nice Work!

You just built a production-grade monitoring pipeline from scratch and used it to detect, investigate, and resolve a real incident. That is the exact loop SRE teams run every day in production.

You've learned how to:

  • Deploy a Prometheus-instrumented Flask API on ECS Fargate with an ADOT sidecar scraping and shipping metrics to Amazon Managed Prometheus.
  • Build Golden Signals dashboards in Amazon Managed Grafana and configure PromQL alert rules with severity-based routing through Alertmanager to SNS.
  • Load test a service and practice the full incident lifecycle: detection, notification, investigation, and resolution.
  • Secret Mission: Implement multi-window burn-rate SLO alerts and a dead man's switch for meta-monitoring.

Ready to quiz yourself?