SRE Observability Stack for a Financial API

Build a monitored Flask API with Prometheus, Grafana, and SLO-based alerting.

Introduction

30 Second Summary

When a payment system goes down, every second without visibility costs money and trust. The difference between a team scrambling in the dark and one that responds in minutes is observability.

In this project, you will build a local observability stack that monitors a simulated financial API. You will instrument the API with metrics, visualize them on a live dashboard, inject a failure, and watch an automated alert fire.

What You'll Build

You will demo the full SRE incident detection loop: trigger a failure in your payment API, watch your dashboard cross the SLO threshold line in real time, and see Alertmanager fire an alert within seconds.

By the end of this project, you'll have:

  • A payment status API you can hit with curl and watch request counters and latency histograms update live on a /metrics endpoint scraped by Prometheus.
  • A Grafana dashboard where you can see availability drop below the 99.5% SLO line and latency spike past 500ms in real time.
  • A failure injection trigger that degrades the service, burns through your error budget, and causes Alertmanager to fire an alert automatically.
  • Secret Mission: A burn-rate alert that detects fast SLO consumption before the error budget is fully exhausted.

Are there any prerequisites?

This project assumes you have Docker installed via WSL2, comfort with the Linux command line, and basic Python knowledge. You do not need any cloud accounts or paid services.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Observability is the foundation of Site Reliability Engineering. Without it, you're guessing whether your service is healthy. This project gives you hands-on experience building the feedback loop that turns guessing into knowing.

Set Up Your Project and Docker Compose Stack

Before you can observe anything about your service, you need the monitoring infrastructure in place. Right now you have an empty project directory and no way to collect metrics, visualize them, or get alerted when something goes wrong.

Docker Compose lets you define your entire monitoring stack as code in a single file. One command spins up Prometheus (metrics collection), Grafana (dashboards), and Alertmanager (notifications) on a shared network.

In this step, get ready to:
  • Write configuration files for Docker Compose, Prometheus, and Alertmanager.
  • Start the monitoring stack with a single command.
  • Verify all three services are accessible in your browser.
Create the Docker Compose file

Docker Compose reads a docker-compose.yml file that declares every container your stack needs, along with their ports, volumes, and network connections.

  • In your project directory, create a new file called docker-compose.yml.
  • Paste the following code that defines the Prometheus and Grafana services:
services:
  prometheus:
    image: prom/prometheus:v3.13.1
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_rules.yml:/etc/prometheus/prometheus_rules.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-lifecycle'
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:13.1.0
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    networks:
      - monitoring

What does this code do?

You are defining two services that will run as containers:

  • prometheus pulls the official Prometheus image, exposes port 9090, mounts your local config files into the container, and enables a reload endpoint so you can update rules without restarting.
  • grafana pulls the Grafana image, exposes port 3000, and sets the admin password to admin so you can log in immediately.
  • Below the grafana service block, add the Alertmanager service and the shared network definition:
  alertmanager:
    image: prom/alertmanager:v0.28.1
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

What does this section add?

alertmanager handles alert notifications. It receives firing alerts from Prometheus and routes them to the appropriate receiver (in this project, a webhook).

The networks block creates a shared bridge network called monitoring. All three services join this network so they can communicate by container name.

  • Save docker-compose.yml.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

Your docker-compose.yml should contain two sections you added above, in this order:

  • The services block starting with prometheus and grafana (first chunk).
  • The alertmanager service followed by the networks block (second chunk).

Scroll up and confirm both chunks are present and indented correctly under services:.

Write the Prometheus and Alertmanager configuration files

Docker Compose mounts local config files into each container. You need to create those files now so the containers have something to read on startup.

  • Create a new file called prometheus.yml in your project directory and paste this configuration:
global:
  scrape_interval: 5s
  evaluation_interval: 5s

rule_files:
  - "prometheus_rules.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

scrape_configs:
  - job_name: "payment-api"
    static_configs:
      - targets: ["host.docker.internal:5000"]

What does this configure?

  • scrape_interval: 5s tells Prometheus to pull metrics from your API every 5 seconds.
  • rule_files points to where alert rules live (you will populate this later).
  • alertmanagers tells Prometheus where to send firing alerts.
  • host.docker.internal:5000 is how a Docker container reaches a service running on your host machine (your Flask API in a later step).
  • Save prometheus.yml.
  • Create another file called prometheus_rules.yml and paste this placeholder:
groups:
  - name: slo_alerts
    rules: []
  • Save prometheus_rules.yml. This file is intentionally empty for now. You will add alert rules in a later step.
  • Create a third file called alertmanager.yml and paste this configuration:
global:
  resolve_timeout: 5m

route:
  receiver: "default"
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h

receivers:
  - name: "default"
    webhook_configs:
      - url: "http://host.docker.internal:5001/webhook"
        send_resolved: true

What does this configure?

Alertmanager decides what to do with alerts it receives from Prometheus.

  • route sends all alerts to a receiver named default.
  • The webhook_configs receiver would forward alerts to an external endpoint. The URL points to a service you have not built yet, and that is fine. The alert still fires and appears in the Alertmanager UI regardless.
  • Save alertmanager.yml.
  • Confirm all four configuration files exist by running:
ls

You should see docker-compose.yml, prometheus.yml, prometheus_rules.yml, and alertmanager.yml listed in the output.

Missing a file?

Double-check that you created each file in the same directory as docker-compose.yml. If a file is missing, create it now using the code above.

help me figure out why my config files are not showing up in ls

Start the stack and verify all services are accessible

All four config files are in place. Time to bring the monitoring stack online.

  • Start all three containers in detached mode by running:
docker compose up -d

What does this command do?

docker compose up reads your docker-compose.yml, pulls any images you do not already have, creates the monitoring network, and starts all three containers.

The -d flag runs them in the background so your terminal stays free.

The first run downloads the container images, which may take a minute or two depending on your connection.

  • Once the command finishes, verify the containers are running:
docker compose ps

You should see three containers listed with a STATUS of Up.

Before you open each service, what do you think you will see at each URL? Will they show data, or empty dashboards waiting for input?

  • Open http://localhost:9090 in your browser to confirm Prometheus is running.
  • Open http://localhost:3000 to confirm Grafana is running. Log in with username admin and password admin.
  • Open http://localhost:9093 to confirm Alertmanager is running.

All three UIs load, but they are empty. Prometheus has no targets to scrape, Grafana has no data source configured, and Alertmanager has no alerts. That is exactly right for now.

A service is not loading?

  • Run docker compose logs <service-name> (replace <service-name> with prometheus, grafana, or alertmanager) to check for error messages.
  • If you see a YAML parsing error, check your indentation. YAML is whitespace-sensitive and requires spaces, not tabs.
  • If a port is already in use, stop whatever process occupies it or change the host port in docker-compose.yml.

help me troubleshoot why my Docker Compose services are not starting

Your monitoring infrastructure is live. Next up, you will build the Flask API that produces the metrics these tools will collect and visualize.

Build and Run the Instrumented Flask API

Your monitoring stack is running. Prometheus, Grafana, and Alertmanager are all healthy and waiting for data. But right now, Prometheus has nothing to scrape.

In this step, you'll build a Flask API that simulates a financial payment service. You'll instrument it with Prometheus metrics so that every request generates data about request count, error rate, and latency. Then you'll generate traffic and discover the gap: raw metrics exist, but without a dashboard, you have zero visibility into your service's health.

In this step, get ready to:
  • Write a Flask API instrumented with Prometheus Counter and Histogram metrics.
  • Run the app and confirm Prometheus scrapes its /metrics endpoint.
  • Generate baseline traffic and observe the observability blind spot.
Write the Flask application with Prometheus instrumentation

The API needs to expose metrics that Prometheus understands. You'll use two metric types: a Counter that tracks the total number of requests (labeled by success or error), and a Histogram that records how long each request takes.

  • Create a new file called app.py in your project directory.
  • Add the imports, app setup, and metric definitions by copying this code:
import time
import random
from flask import Flask, jsonify
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST

app = Flask(__name__)

# Chaos state
chaos_enabled = False
chaos_error_rate = 0.5

# Prometheus metrics
REQUEST_COUNT = Counter(
    'payment_requests_total',
    'Total payment status requests',
    labelnames=['method', 'endpoint', 'status']
)

REQUEST_LATENCY = Histogram(
    'payment_request_duration_seconds',
    'Payment request latency in seconds',
    labelnames=['method', 'endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

What does this code do?

This sets up the foundation your API needs to speak Prometheus's language.

  • REQUEST_COUNT is a Counter that increments every time a request completes. The status label distinguishes successes from errors.
  • REQUEST_LATENCY is a Histogram that records request duration in seconds. The buckets array defines the latency ranges Prometheus uses to calculate percentiles.
  • chaos_enabled and chaos_error_rate are flags you'll use later to simulate service degradation.
  • Below the metric definitions in app.py, add the main API route that handles payment status requests:
@app.route('/payment/status')
def payment_status():
    start_time = time.time()

    # Simulate variable latency
    if chaos_enabled:
        time.sleep(random.uniform(0.3, 1.5))
    else:
        time.sleep(random.uniform(0.01, 0.1))

    # Simulate errors
    if chaos_enabled and random.random() < chaos_error_rate:
        duration = time.time() - start_time
        REQUEST_LATENCY.labels(method='GET', endpoint='/payment/status').observe(duration)
        REQUEST_COUNT.labels(method='GET', endpoint='/payment/status', status='error').inc()
        return jsonify({'error': 'Service degraded'}), 500

    duration = time.time() - start_time
    REQUEST_LATENCY.labels(method='GET', endpoint='/payment/status').observe(duration)
    REQUEST_COUNT.labels(method='GET', endpoint='/payment/status', status='success').inc()

    payment_id = f"PAY-{random.randint(1000, 9999)}"
    return jsonify({
        'payment_id': payment_id,
        'status': 'completed',
        'amount': round(random.uniform(10, 5000), 2),
        'currency': 'USD'
    })

What does this route do?

This is the core endpoint that simulates a payment status check. Every request records both its duration (via the Histogram) and its outcome (via the Counter).

  • When chaos is disabled, requests complete in 10-100ms with no errors.
  • When chaos is enabled, latency jumps to 300-1500ms and the configured error rate kicks in.
  • The .labels() method tags each metric with the HTTP method and endpoint so Prometheus can filter and aggregate them.
  • Save app.py and verify the file has no syntax errors by running:
python -c "import ast; ast.parse(open('app.py').read()); print('Syntax OK')"

You should see Syntax OK printed. This confirms your code so far is valid Python.

Seeing a SyntaxError?

Check that you have matching parentheses on the Counter() and Histogram() constructors. Each opening parenthesis needs a closing one on its own line.

Also check the f-string on the payment_id line uses curly braces, not parentheses.

help me debug a SyntaxError in my Flask app

  • Now add the chaos control endpoints and the metrics endpoint below the payment_status function in app.py:
@app.route('/chaos/enable')
def enable_chaos():
    global chaos_enabled, chaos_error_rate
    chaos_enabled = True
    chaos_error_rate = 0.5
    return jsonify({'chaos': 'enabled', 'error_rate': chaos_error_rate})


@app.route('/chaos/enable/moderate')
def enable_moderate_chaos():
    global chaos_enabled, chaos_error_rate
    chaos_enabled = True
    chaos_error_rate = 0.1
    return jsonify({'chaos': 'enabled', 'error_rate': chaos_error_rate})


@app.route('/chaos/disable')
def disable_chaos():
    global chaos_enabled
    chaos_enabled = False
    return jsonify({'chaos': 'disabled'})


@app.route('/metrics')
def metrics():
    return generate_latest(), 200, {'Content-Type': CONTENT_TYPE_LATEST}


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

What do these endpoints do?

  • /chaos/enable and /chaos/disable let you toggle failure injection on and off without restarting the app.
  • /chaos/enable/moderate sets a lower error rate (10%) for testing subtle degradation.
  • /metrics is the endpoint Prometheus scrapes. It returns all metric values in Prometheus text format using generate_latest().
  • The if __name__ == '__main__' block starts the Flask server on port 5000, accessible from all network interfaces.
  • Save app.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file and move on to the next substep.

ⓧ I'd like to double check the full code

Your app.py should contain these sections in order. Scroll up and confirm each one is present:

  • Imports (time, random, Flask, prometheus_client).
  • App initialization and chaos state variables.
  • REQUEST_COUNT Counter and REQUEST_LATENCY Histogram definitions.
  • The payment_status() route with latency simulation, error simulation, and metric recording.
  • Three chaos endpoints (/chaos/enable, /chaos/enable/moderate, /chaos/disable).
  • The /metrics endpoint and if __name__ == '__main__' block.
Create requirements.txt and run the app

Your app imports Flask and prometheus-client, but neither is installed yet. You need a requirements.txt to pin the exact versions.

  • Create a file called requirements.txt in your project directory with these pinned dependencies:
Flask==3.1.3
prometheus-client==0.25.0
  • Install the dependencies into your virtual environment by running:
pip install -r requirements.txt

You should see pip downloading and installing both packages along with their dependencies.

Seeing a 'No matching distribution' error?

Make sure your virtual environment is activated. You should see the environment name in parentheses at the start of your terminal prompt.

Also confirm that requirements.txt uses == (double equals) not = (single equals).

help me fix a pip install error for Flask and prometheus-client

  • Start the Flask app by running:
python app.py

You should see output indicating the server is running on http://0.0.0.0:5000. Leave this terminal running.

  • Open a new terminal tab in the same directory (with your virtual environment activated).
  • Test the API by sending a request with curl:
curl http://localhost:5000/payment/status

You should see a JSON response with a payment_id, status, amount, and currency field.

  • Now check the metrics endpoint to see Prometheus-formatted data:
curl http://localhost:5000/metrics

You should see lines like payment_requests_total and payment_request_duration_seconds_bucket in the output. These are the raw metrics Prometheus will scrape.

Before you check Prometheus targets, do you think Prometheus has already found your app? It was configured to scrape host.docker.internal:5000 before the app existed.

  • Open http://localhost:9090/targets in your browser to confirm Prometheus is scraping the Flask app.

You should see the payment-api job with state UP. Prometheus discovered your app the moment it started responding on port 5000.

Target showing as DOWN?

Make sure your Flask app is still running in the other terminal tab. If you see connection refused errors, the app process may have crashed.

Also confirm your prometheus.yml has host.docker.internal:5000 as the target. This is how Docker containers reach services on your host machine.

help me debug why Prometheus can't scrape my Flask app target

Generate baseline traffic and observe the blind spot

Your API is running and Prometheus is scraping it. Time to simulate real traffic and see what you can learn from the raw data alone.

  • Send 50 requests to the API to generate baseline traffic by running:
for i in $(seq 1 50); do curl -s http://localhost:5000/payment/status > /dev/null; done
echo "Done - 50 requests sent"

You should see Done - 50 requests sent after a few seconds.

  • Check the metrics endpoint again to see the updated counters:
curl -s http://localhost:5000/metrics | grep payment_requests_total

You'll see the counter has incremented. But here's the problem: can you tell what percentage of those 50 requests failed? Can you see whether latency is trending up or down?

Why is this a problem?

You have raw metrics, but no way to answer the questions that matter: Is this service reliable enough? Is it getting worse? Right now, you would need to manually run PromQL queries and do mental math every time you want to check health.

In production, nobody watches raw metric endpoints. SREs build dashboards with SLIs (what to measure) and SLOs (the target) so that a single glance tells you whether the service is meeting its reliability goals. That's what you'll build next.

Your API is live, instrumented, and generating data. But you're flying blind. You have no dashboard, no SLO lines, and no alerts. Next up, you'll build a Grafana dashboard that turns these raw metrics into meaningful reliability signals.

Build Your SLI/SLO Dashboard in Grafana

In the previous step, you sent traffic to your API and saw the responses come back. But you have no idea what percentage of those requests actually succeeded, or how fast they were.

Raw counters are meaningless without context. SREs define Service Level Indicators (SLIs) and Service Level Objectives (SLOs) to answer one question: is this service reliable enough? This step transforms your raw payment_requests_total counter into an availability percentage, and your payment_request_duration_seconds histogram into a P95 latency number. Then you draw the SLO threshold line so you can instantly see if you are in violation.

In this step, get ready to:
  • Add Prometheus as a data source in Grafana.
  • Create an Availability SLI panel with a 99.5% SLO threshold line.
  • Create a P95 Latency SLI panel with a 500ms SLO threshold line.
Add Prometheus as a data source in Grafana

Grafana needs to know where to pull metrics from. You will connect it to your running Prometheus instance so it can query the time-series data your Flask app is producing.

  • Open your browser and navigate to http://localhost:3000.
  • Log in with username admin and password admin.
  • Click Connections in the left-side menu.
  • Click Add new connection.
  • Enter Prometheus in the search bar.
  • Click Prometheus data source.
  • Click Add new data source in the upper right.

Grafana opens the Settings tab for your new data source.

  • In the Prometheus server URL field, enter http://prometheus:9090.
  • Scroll to the bottom of the page and click Save & test.

Why http://prometheus:9090 and not localhost?

Grafana runs inside a Docker container. From inside that container, localhost refers to the container itself, not your host machine. The service name prometheus resolves to the Prometheus container because both services share the monitoring Docker network you defined in docker-compose.yml.

You should see a green success message confirming that Grafana can reach Prometheus.

Connection test failed?

  • Double-check the URL is exactly http://prometheus:9090 with no trailing slash.
  • Confirm your Docker Compose stack is still running by checking docker compose ps in your terminal. All three containers should show a healthy or running status.
  • If you restarted your machine, run docker compose up -d again from your project folder.

Still stuck? Help me troubleshoot my Grafana to Prometheus connection.

Create the Availability SLI panel

Your first panel answers the most critical SRE question: what percentage of requests are succeeding? This is your availability SLI. You will write a PromQL query that divides successful requests by total requests, giving you a ratio between 0 and 1.

  • Click Dashboards in the left-side menu.
  • Click New and select New Dashboard.
  • Click the blue + icon (Add new element) and drag a panel onto the dashboard.
  • Select Configure visualization.

This opens the panel editor. You will see a query input area at the bottom of the screen.

  • Confirm the data source is set to Prometheus in the data source dropdown.
  • Switch to the code editor by clicking the Code toggle (if not already active).
  • Paste the following PromQL query into the query field:
1 - (
  sum(rate(payment_requests_total{status="error"}[5m]))
  /
  sum(rate(payment_requests_total[5m]))
)

What does this query calculate?

  • rate(payment_requests_total{status="error"}[5m]) calculates the per-second error rate over the last 5 minutes.
  • rate(payment_requests_total[5m]) calculates the per-second total request rate over the same window.
  • Dividing errors by total gives the error ratio. Subtracting from 1 converts it to a success ratio (your availability SLI).
  • A result of 0.995 means 99.5% of requests succeeded.
  • Click Run queries to confirm data appears in the panel preview.

You should see a line hovering near 1.0 (since all your baseline traffic from earlier was successful). Now add the SLO threshold line so you can see when availability drops below your target.

  • In the panel editor pane on the right, scroll down to the Thresholds section (or type thresholds in the search bar at the top of the pane).
  • Click + Add threshold.
  • Enter 0.995 as the threshold value (this represents 99.5%).
  • Under Show thresholds, select As lines.
  • In the Panel options section at the top of the pane, set the Title to Availability SLI (99.5% SLO).
  • Click Back to dashboard in the top-left corner.

Your first panel now shows a time series line with a horizontal threshold line at the 99.5% mark. Any time availability dips below that line, you know your SLO is being violated.

Panel shows No data?

  • Make sure your Flask app is still running. If it stopped, restart it with python app.py from your project folder.
  • Verify Prometheus is scraping successfully by visiting http://localhost:9090/targets and confirming the payment-api target shows state UP.
  • Send a few more requests to generate fresh data:
for i in $(seq 1 10); do curl -s http://localhost:5000/payment/status > /dev/null; done

Still stuck? Help me debug my Grafana panel showing no data.

Create the P95 Latency SLI panel

Your second panel answers: how fast is this service? Specifically, you want the 95th percentile latency. If P95 is under 500ms, 95% of your users are getting a response within half a second.

  • On your dashboard, click the blue + icon again and drag a new panel onto the dashboard.
  • Select Configure visualization.
  • Confirm the data source is Prometheus.
  • Switch to the Code editor and paste this PromQL query:
histogram_quantile(0.95, sum(rate(payment_request_duration_seconds_bucket[5m])) by (le))

What does this query calculate?

  • histogram_quantile(0.95, ...) extracts the 95th percentile from a histogram.
  • rate(payment_request_duration_seconds_bucket[5m]) calculates the per-second rate of observations falling into each latency bucket over 5 minutes.
  • by (le) groups the result by the upper boundary of each bucket, which is required for histogram_quantile to interpolate correctly.
  • The result is in seconds. A value of 0.5 means the P95 latency is 500ms.
  • Click Run queries to confirm data appears.
  • Scroll down to the Thresholds section in the panel editor pane.
  • Click + Add threshold.
  • Enter 0.5 as the threshold value (0.5 seconds = 500ms).
  • Under Show thresholds, select As lines.
  • In Panel options, set the Title to P95 Latency SLI (500ms SLO).
  • Click Back to dashboard.

You now have two panels side by side. Save the dashboard so your work persists.

  • Click the Save icon (floppy disk) in the top-right corner of the dashboard.
  • Enter Payment API SLIs as the dashboard name.
  • Click Save.

Before you look at the result, what do you expect to see on the availability panel? Should it be above or below the SLO line?

Since all traffic so far has been healthy (no chaos mode), your availability line should sit at 1.0, well above the 99.5% SLO threshold. The latency line should be well below the 500ms threshold since normal mode uses random.uniform(0.01, 0.1) second delays.

What are SLIs and SLOs?

An SLI (Service Level Indicator) is a measured quantity that tells you how well a service is performing. Common SLIs include availability (percentage of successful requests) and latency (how fast responses are).

An SLO (Service Level Objective) is a target you set for an SLI. For example, 99.5% availability means you tolerate up to 0.5% of requests failing. The gap between 100% and your SLO is your error budget.

Your dashboard now transforms raw counters into the two signals that matter most: is the service up, and is it fast? In the next step, you will inject a failure that burns through your error budget and watch these panels turn from green to red in real time.

Inject Failure and Watch Your SLO Burn

Your Grafana dashboard currently shows a flat line above the SLO threshold. Everything looks healthy. That is the problem.

A dashboard that always shows green teaches you nothing about what degradation actually looks like in your metrics. In this step, you will inject a failure into your payment API and watch availability and latency blow past your SLO lines in real time.

In this step, get ready to:
  • Activate the chaos injection endpoints built into your API.
  • Generate traffic while the service is degraded.
  • Watch your Grafana dashboard breach both SLO lines in real time.
Test the failure injection endpoints

Your Flask app already has chaos engineering endpoints built in. The /chaos/enable endpoint switches the API into a degraded mode with a 50% error rate and elevated latency. The /chaos/disable endpoint recovers it.

  • In your terminal, test the chaos enable endpoint by running:
curl http://localhost:5000/chaos/enable

You should see a JSON response confirming chaos is enabled with a 0.5 error rate.

  • Immediately disable it for now by running:
curl http://localhost:5000/chaos/disable

What do these endpoints do?

When chaos is enabled, every request to /payment/status has a 50% chance of returning a 500 error. Latency also jumps from the normal 10-100ms range up to 300-1500ms.

This simulates the kind of degradation a real payment service might experience during an infrastructure issue. The API still responds, but half the requests fail and everything is slow.

Seeing "Connection refused"?

Make sure your Flask app is still running in another terminal. If it stopped, restart it with python app.py from your project folder.

Help me troubleshoot why my Flask app isn't responding to curl requests.

Trigger failure and generate load

Now you will flip the switch for real. You need to enable chaos mode and then flood the API with requests so Prometheus collects enough degraded data points to show the breach on your dashboard.

  • Enable chaos mode by running:
curl http://localhost:5000/chaos/enable
  • Generate a burst of traffic against the degraded API by running this loop:
for i in $(seq 1 100); do curl -s -o /dev/null -w "%{http_code}\n" http://localhost:5000/payment/status; done

What does this loop do?

This sends 100 requests to your payment API. The -s flag silences progress output. The -o /dev/null discards the response body. The -w "%{http_code}\n" prints just the HTTP status code for each request.

You should see a mix of 200 and 500 codes scrolling by. Roughly half will be 500 errors. That is the chaos in action.

You should see a stream of status codes, with roughly half showing 500 (errors) and half showing 200 (successes).

Before you check your Grafana dashboard, take a moment to predict: what do you think happened to your availability SLI? Is it still above the 99.5% SLO line, or has it dropped below?

  • Switch back to your Grafana dashboard in the browser at http://localhost:3000.
  • Navigate to your SLI/SLO dashboard.

Your Availability SLI panel should have plummeted well below the 99.5% SLO threshold line. The P95 Latency panel should be spiking above the 500ms SLO line. Both SLOs are now in violation.

Why this matters for SRE

You just experienced what a real service degradation looks like through the lens of SLIs and SLOs. The raw request counts alone would not have told you whether the service was "reliable enough". The SLO threshold lines give you an instant binary answer: you are in violation.

But here is the critical gap. You had to open Grafana and look at the dashboard at exactly the right moment to notice the breach. If you had been away from your screen, the SLO violation would have gone completely undetected. That is the problem the next step solves.

Dashboard still showing flat lines?

Check that the time range in Grafana is set to "Last 5 minutes" or "Last 15 minutes" using the time picker in the top right corner. A wider time range can flatten the spike visually.

Also confirm that Prometheus is still scraping your app. Visit http://localhost:9090/targets and check that the payment-api target shows state UP.

Help me figure out why my Grafana panels aren't showing the expected SLO breach after injecting chaos.

You now know what it looks like when a service burns through its error budget. The SLO lines give you instant clarity on whether the service is reliable enough. But relying on someone watching a dashboard is not a real incident detection strategy. Next up, you will close this gap by setting up automated alerts that fire the moment your SLO is violated.

Set Up Alerting for SLO Violations

In the last step, you saw the dashboard turn red as availability dropped below your SLO line. But you only noticed because you were actively watching it at that exact moment.

Watching a dashboard manually does not scale. The whole point of SRE alerting is to be notified automatically when reliability degrades past the SLO. In this step, you will write a Prometheus alert rule that evaluates your error rate continuously. When the threshold is crossed, Alertmanager fires the alert for you.

In this step, get ready to:
  • Write a Prometheus alert rule that fires when error rate exceeds the SLO.
  • Walk through the full incident detection loop from healthy to firing.
  • Recover the service and confirm the alert resolves.
Write the Prometheus alert rule

Your prometheus_rules.yml file currently has an empty rules list. You need to add a rule that calculates the 5-minute error rate and fires when it exceeds 0.5% (the inverse of your 99.5% SLO).

  • Open prometheus_rules.yml in your editor.
  • Replace the entire file contents with the following alert rule:
groups:
  - name: slo_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(payment_requests_total{status="error"}[5m]))
            /
            sum(rate(payment_requests_total[5m]))
          ) > 0.005
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Payment API error rate exceeds SLO"
          description: "Error rate is {{ $value | humanizePercentage }} (SLO: 0.5%)"

What does this rule do?

The expr calculates the ratio of error requests to total requests over a 5-minute sliding window. If that ratio exceeds 0.005 (0.5%), the condition is met.

The for: 1m clause tells Prometheus to wait 1 minute with the condition continuously true before actually firing the alert. This prevents brief spikes from triggering false alarms.

  • Save prometheus_rules.yml.
  • Reload the Prometheus configuration without restarting the container by running:
curl -X POST http://localhost:9090/-/reload

Why not restart the container?

The --web.enable-lifecycle flag in your docker-compose.yml enables the /-/reload endpoint. This lets Prometheus pick up configuration changes on the fly without losing its in-memory metric data.

  • Verify the rule loaded by visiting http://localhost:9090/alerts in your browser.
  • You should see a HighErrorRate alert listed in an inactive (green) state.

Don't see the HighErrorRate alert?

  • Make sure the curl reload command returned successfully. If you see a connection error, confirm Prometheus is still running with docker compose ps.
  • Check your YAML indentation. The rules: key must be nested under name: slo_alerts and alert: must be a list item under rules:.
  • help me debug why my Prometheus alert rule is not loading after a reload

✔️ Awesome, I've got everything!

Great. Double check you have saved prometheus_rules.yml and you can see HighErrorRate in the Prometheus Alerts page.

ⓧ I'd like to double check the full code

groups:
  - name: slo_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(payment_requests_total{status="error"}[5m]))
            /
            sum(rate(payment_requests_total[5m]))
          ) > 0.005
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Payment API error rate exceeds SLO"
          description: "Error rate is {{ $value | humanizePercentage }} (SLO: 0.5%)"
Trigger the full incident detection loop

Right now, chaos mode is still enabled from the previous step. To demonstrate the full loop cleanly, you will start from a healthy state, then degrade the service and watch the alert transition through its lifecycle.

  • Disable chaos mode to return to a healthy baseline by running:
curl http://localhost:5000/chaos/disable
  • Generate some healthy traffic so the 5-minute rate window fills with successful requests:
for i in $(seq 1 50); do curl -s http://localhost:5000/payment/status > /dev/null; done
  • Refresh http://localhost:9090/alerts in your browser.
  • Confirm the HighErrorRate alert shows as inactive (green). This is your healthy baseline.

Now you will inject failure and watch the alert come alive.

  • Re-enable chaos mode by running:
curl http://localhost:5000/chaos/enable
  • Generate a burst of traffic under degraded conditions by running:
for i in $(seq 1 100); do curl -s http://localhost:5000/payment/status > /dev/null; done

Before you check the alerts page, what state do you think the alert is in right now? Remember the rule has a for: 1m clause.

  • Refresh http://localhost:9090/alerts immediately after the traffic loop finishes.
  • You should see HighErrorRate in a pending (yellow) state. Prometheus detected the threshold breach but is waiting the full 1 minute before firing.
  • Wait about 60 seconds, then refresh again.
  • The alert transitions to firing (red). This means Prometheus has confirmed the breach is sustained and has sent the alert to Alertmanager.

What are inactive, pending, and firing?

Prometheus alerts have three states. Inactive means the condition is not met. Pending means the condition is met but the for duration has not elapsed yet. Firing means the condition has been true for the required duration and Alertmanager has been notified.

  • Now confirm the alert arrived in Alertmanager by visiting http://localhost:9093 in your browser.
  • You should see an active HighErrorRate alert with the severity label critical and the summary annotation you defined.

Alert stuck in pending or not appearing in Alertmanager?

  • If the alert stays pending, you may need to generate more traffic. Run the 100-request loop again and wait another 60 seconds.
  • If Alertmanager at http://localhost:9093 shows no alerts, confirm the Alertmanager container is running with docker compose ps.
  • help me debug why my Prometheus alert is not transitioning from pending to firing
Recover and confirm the alert resolves

A complete incident loop includes recovery. You need to confirm that when the service returns to healthy, the alert clears itself automatically.

  • Disable chaos mode to recover the service by running:
curl http://localhost:5000/chaos/disable
  • Generate healthy traffic to flush the 5-minute rate window by running:
for i in $(seq 1 100); do curl -s http://localhost:5000/payment/status > /dev/null; done

What do you think will happen to the alert once the 5-minute window fills with only successful requests?

  • Wait about 2 minutes for the rate window to fully roll over with healthy data.
  • Refresh http://localhost:9090/alerts.
  • The HighErrorRate alert should now show as inactive (green) again.
  • Refresh http://localhost:9093.
  • The alert should no longer appear in the active alerts list in Alertmanager.

The full SRE feedback loop

You just walked through the complete cycle that SREs rely on in production: service is healthy, degradation occurs, metrics breach the SLO, alert fires automatically, team responds, service recovers, alert resolves. No one needed to be staring at a dashboard.

Alert not resolving after recovery?

  • The 5-minute rate window needs time to fully roll over. Generate another batch of healthy traffic and wait 2-3 more minutes.
  • Confirm chaos is truly disabled by running curl http://localhost:5000/chaos/disable again and checking the response says {"chaos": "disabled"}.
  • help me understand why my Prometheus alert is not resolving after I disabled chaos mode

You now have a complete observability pipeline. Your service is instrumented, your dashboard visualizes SLIs against SLOs, and your alerting automatically detects and notifies when reliability drops below the target. The Secret Mission takes this further with burn-rate alerts that catch slow degradation before the budget is fully exhausted.

Secret mission

Implement a Burn-Rate Alert

Your HighErrorRate alert only fires after the SLO is already violated. In production, SREs use burn-rate alerts to detect when the error budget is being consumed faster than expected. Can you implement a multi-window burn-rate alert that catches slow degradation before the budget is fully exhausted?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker containers and a Python virtual environment. There are no ongoing cloud costs or subscription charges.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers: Prometheus, Grafana, and Alertmanager.
  • Python virtual environment with Flask and prometheus-client packages.
  • Project files: docker-compose.yml, app.py, prometheus.yml, prometheus_rules.yml, alertmanager.yml, and requirements.txt.
  • Grafana dashboard and data source configuration (stored inside the Grafana container volume).

Keep everything running

No action needed. Choose this if you are still actively experimenting with the observability stack.

  • Your Docker containers continue running in the background. They consume minimal system resources.
  • Your Flask app remains running in its terminal window. Keep that terminal open.
  • Grafana stays accessible at http://localhost:3000, Prometheus at http://localhost:9090, and Alertmanager at http://localhost:9093.
  • Everything is local. There are no costs accumulating.

Pause - I'll come back to this later

Shut down running processes to free up memory and CPU. Your project files and container data stay on disk so you can restart later.

  • Stop the Flask app by pressing Ctrl+C in the terminal where it is running.
  • Stop the Docker containers (without removing them) by running this command from your project folder:
docker compose stop

When you are ready to resume, restart everything with these two steps:

  • Start the Docker containers again by running:
docker compose start
  • Activate your Python virtual environment and start the Flask app again by running:
source venv/bin/activate
python app.py

Delete - I don't want to use this again

Remove all containers, volumes, and project files from your machine. This is a complete teardown.

  • Stop the Flask app by pressing Ctrl+C in the terminal where it is running.
  • Stop and remove all Docker containers and volumes by running this command from your project folder:
docker compose down -v
  • Confirm the containers are gone by running:
docker ps -a

You should see no containers related to Prometheus, Grafana, or Alertmanager in the output.

  • Delete the entire project folder (the folder containing docker-compose.yml and app.py) from your Desktop. On macOS or Linux, navigate to your Desktop and remove it:
cd ~/Desktop
rm -rf sre-observability-stack

Folder name differs?

Replace sre-observability-stack with whatever you named your project folder in Step 1. You can check the name by running ls ~/Desktop first.

  • On Windows (if using File Explorer), navigate to your Desktop and delete the project folder.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete SRE observability pipeline from scratch, from instrumenting a financial API with Prometheus metrics to detecting an SLO violation and firing an automated alert.

You've learned how to:

  • Instrument a Flask API with Prometheus counters and histograms, and deploy a full monitoring stack (Prometheus, Grafana, Alertmanager) using Docker Compose.
  • Create an SLI/SLO dashboard that translates raw metrics into reliability signals with threshold lines showing exactly when your service is in violation.
  • Run the full SRE feedback loop: inject failure into a running service, watch availability drop below the SLO line in real time, and confirm an automated alert fires in Alertmanager.
  • Implement a burn-rate alert that catches slow degradation before the error budget is fully exhausted, proving its advantage over a simple threshold alert (Secret Mission).

Ready to quiz yourself?